Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction

A range of methods aim to enhance the performance of vision-language models (VLMs) at test time. Among them, transduction has emerged as a promising paradigm due to its strong compatibility and efficiency. However, realistic evaluations often involve highly imbalanced class distributions, which cause performance degradation or even collapse. In this work, we systematically revisit transduction from the perspective of penalized likelihood estimation (PLE), showing that PLE with a KL-divergence anchor term naturally yields an adaptive shrinkage behavior between prior anchors and empirical estimates. From this viewpoint, the brittleness of transductive methods can be attributed to the absence of anchoring mechanism and static modeling of the shrinkage strength. Therefore, we propose Mixture of Von Mises-Fisher Models with Dynamic Shrinkage (MOON). MOON is built upon a mixture of von Mises-Fisher distributions to model feature representations on the unit hypersphere. To handle imbalance, MOON dynamically adjusts the shrinkage strength using zero-shot priors at both instance and class levels. Thus, it suppresses unreliable assignments and prevents harmful updates from outlier classes, thereby mitigating negative transfer. MOON is model-agnostic, training-free, and requires no task-specific hyperparameter tuning. Extensive experiments further validate the advantage of MOON in both performance and efficiency.

Vision-language models such as CLIP can recognize images without retraining, but real-world test data is often imbalanced: some categories appear much more frequently than others, or similar images arrive together in a stream. Existing test-time adaptation methods can become unreliable in these settings because they may overfit to the limited test data they observe. We introduce MOON, a simple training-free method that improves prediction stability under such realistic conditions. MOON combines the model’s original predictions with information shared among related test examples, while automatically reducing the influence of uncertain samples and unlikely classes. It requires no additional training, model updates, or task-specific tuning. Experiments on many datasets and vision-language models show that MOON consistently improves accuracy and efficiency, especially under imbalanced test distributions.