TiME: Test-Time Mixture-of-Experts Routing via Asymmetric CO-Optimal Transport for Continual Test-Time Adaptation

Large language models usually face continuous domain shifts during testing, which degrade performance on unseen shifting domains. So, researchers propose continual test-time adaptation (CTTA) to adapt to evolving testing domains while preserving knowledge of previous domains, making adaptability-stability (A-S) balance. Existing CTTA methods are constrained by dense base models that encode knowledge from all domains into a global model, hardly achieving the A-S balance. We observe that the model sparsity of mixture-of-experts (MoE) models is better for achieving A–S balance than dense models. In CTTA, however, MoE faces difficulty in (1) correctly routing samples from unseen shifting domains and (2) capturing domain-level shifts. In this paper, we propose test-time mixture-of-experts routing (TiME) via asymmetric co-optimal transport (As-COOT): we model MoE routing in CTTA as a test-time allocation problem via COOT. To ensure reliable routing, we propose a semantic space alignment to align sample-expert distributions via bidirectional contrastive learning. To address COOT’s limitations in CTTA, we propose As-COOT, relaxing sample-side constraints while enforcing expert-side constraints to ensure noise robustness and balance expert load. Experiments show TiME outperforms baselines.

Large language models (LLMs) often struggle when the domain of input they see changes over time. For example, a model trained on general text may perform poorly when it is continuously exposed to new domains such as medicine or law. Existing adaptation methods try to learn from these changing environments, but they often either forget previously learned knowledge while adapting to new domains or preserve old knowledge too strongly and fail to adapt effectively. The balance is difficult to achieve. We found that mixture-of-experts (MoE) LLMs are naturally better at balancing adaptation and stability because different expert modules can specialize in different domains. However, their routing decisions are usually fixed after training and may fail when testing domains continuously change. To address this, we propose TiME, a new method that dynamically reallocates inputs to experts based on the current testing domain. To achieve this, we use an optimization theory inspired by transportation planning, allowing the model to better allocate inputs with suitable experts while filtering noisy inputs. Our method helps LLMs adapt more effectively to continuously changing environments without severely forgetting previous knowledge. Experiments on text understanding and medical reasoning task streams show that TiME consistently outperforms existing continual test-time adaptation methods, improving both robustness and long-term stability.