Tempora: Characterising the Time-Contingent Utility of Online Test-Time Adaptation

Test-time adaptation (TTA) offers a compelling remedy for machine learning (ML) models that degrade under domain shifts, improving generalisation on-the-fly with only unlabelled samples. This flexibility suits real deployments, yet conventional evaluations unrealistically assume unbounded processing time, overlooking the accuracy-latency trade-off. As ML increasingly underpins latency-sensitive and user-facing use-cases, temporal pressure constrains the viability of adaptable inference; predictions arriving too late to act on are futile. We introduce Tempora, a framework for evaluating TTA under this pressure. It consists of temporal scenarios that model deployment constraints, evaluation protocols that operationalise measurement, and time-contingent utility metrics that quantify the accuracy-latency trade-off. We instantiate the framework with three such metrics: (1) discrete utility for asynchronous streams with hard deadlines, (2) continuous utility for interactive settings where value decays with latency, and (3) amortised utility for budget-constrained deployments. By applying Tempora to 11 TTA methods, we find that rank instability persists across 750+ temporal evaluations spanning diverse datasets, models, and hardware platforms; i.e., conventional rankings do not predict rankings under temporal pressure. The highest-utility method varies with the shift and temporal pressure, with no clear winner. By enabling systematic evaluation across diverse temporal constraints for the first time, Tempora reveals when and why rankings change, offering practitioners a lens for method selection and researchers a target for deployable adaptation. Code: https://github.com/sudotensor/tempora.

A deployed machine learning model often encounters data dissimilar to what it saw during training. These out-of-distribution datapoints can degrade model quality. Test-time adaptation methods aim to fix this on the fly, but existing evaluations crucially ignore time pressure; delayed predictions may harm practical utility. Tempora is a framework that evaluates these methods under diverse deployment constraints and finds that conventional benchmark rankings break down far more broadly than previously observed; no single method wins across all settings. It also helps reveal why a method failed in a given setting and offers concrete guidance on the design of future methods to be more suitable in practice.