Design-Based Anytime-Valid Inference for Randomized Experiments with Delayed Outcomes and Staggered Entry

Delayed outcomes are ubiquitous in online experimentation: treatment can affect whether an outcome occurs, when it occurs, and its realized value. To accommodate staggered entry while remaining robust to environmental nonstationarity and unit-level heterogeneity, we adopt a design-based perspective and target the sample cumulative reward in each arm as a function of calendar time. Our confidence sequences allow practitioners to continuously monitor the counterfactual incremental reward, such as revenue, that would have been realized by calendar time $t$ had all entered units been assigned to treatment rather than control. The main technical challenge is the choice of design-based filtration, complicated by the presence of asynchronous potential outcome times. We show that the IPW treatment-effect estimation error is not a martingale with respect to any filtration, while each arm-specific IPW estimation error is a martingale with respect to a carefully chosen arm-specific event-time filtration. We therefore construct a confidence sequence for the treatment effect by combining two arm-level confidence sequences with a union bound, and further demonstrate that this can outperform the traditional design-based variance upper bound. Finally, we characterize the class of augmentations for which the per-arm AIPW estimation error remains a martingale.

Real-world A/B tests are often more complicated than they first appear. A marketing intervention, for example, might make some customers purchase sooner while reducing the fraction who eventually purchase at all. At short time scales the treatment may therefore look beneficial, while at longer time scales it may be harmful. Revenue adds another layer: a treatment might increase the number of purchases and accelerate when they occur, but steer customers toward cheaper products and reduce total revenue. This creates a challenge for continuous monitoring. Although the eventual outcome is often the natural target, estimating it during an ongoing experiment requires assumptions that may be hard to justify. In practice, units enter experiments over time rather than all at once, and users who enter at different times may differ substantially. They may also experience different product versions, market conditions, or external environments. We therefore ask what can be monitored continuously without relying on such assumptions. Our answer is the cumulative reward, such as total revenue, that would have been observed by each calendar time had all entered units been assigned to treatment versus control. We construct confidence sequences, uncertainty bands that remain valid under continuous monitoring, for this treatment-control difference. The resulting quantity is directly interpretable: it compares the total reward generated under treatment and control as the experiment unfolds, without requiring a probability model for a hypothetical population.