Testing For Distribution Shifts with Conditional Conformal Test Martingales

We propose a sequential test for distribution-shift detection that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptotic power one and bounded expected detection delay. Empirically, our method detects shifts faster than standard CTMs, providing a powerful and reliable distribution-shift detector.

When machine learning models are deployed in the real world, they often fail when encountering unfamiliar data, a problem known as a distribution shift. Standard detection methods evaluate new data by constantly updating their baseline with everything they see, meaning a "contaminated'' baseline will mask the severity of the change and make the system slow to react. To fix this contamination, we develop a monitoring method that compares incoming data against a fixed reference set. We designed new mathematical tools to safely account for the statistical uncertainty of using this fixed reference set without causing false alarms. Our approach detects distribution shifts faster and more reliably than existing methods. This allows developers to catch degrading performance early, ensuring safer and more trustworthy deployments in dynamic environments.