Utilizing Historical Data for Neural Bandits with Domain Shift

Integrating historical data into multi-armed bandits is a critical challenge, as indiscriminately incorporating biased offline data can lead to unwanted negative transfer. Motivated by high-stakes clinical settings where skewed training distributions have historically induced algorithmic bias (Kallus et al., 2020), we investigate how to safely bridge the offline-to-online gap. We extend the Artificial Replay meta-algorithm (Banerjee et al., 2022) to contextual neural bandits, accommodating the continuous context space by introducing closeness heuristics that enforce conservative adaptation by restricting historical replays to contexts with high spatial proximity. Empirically, we show that under severe domain shift, both full pre-training and unconstrained replay perform worse than learning entirely from scratch. Conversely, our spatial filtering approach effectively extracts the utility of biased historical data while avoiding negative transfer, consistently outperforming pure online baselines, full historical starts, and unrestricted Artificial Replay.