MODEL SOUPS NEED ONLY ONE INGREDIENT

Fine-tuning large pre-trained models on a target distribution often improves in-distribution (ID) accuracy, but at the cost of out-of-distribution (OOD) robustness as representations specialize to the fine-tuning data. Weight-space ensembling methods, such as Model Soups, mitigate this effect by averaging multiple checkpoints, but they are computationally prohibitive, requiring the training and storage of dozens of fine-tuned models. In this paper, we introduce MonoSoup, a simple, data-free, hyperparameter-free, post-hoc method that achieves a strong ID–OOD balance using only a single checkpoint. Our method applies Singular Value Decomposition (SVD) to each layer’s update and decomposes it into high-energy directions that capture task-specific adaptation and low-energy directions that introduce noise but may still encode residual signals useful for robustness. MonoSoup then uses entropy-based effective rank to automatically re-weigh these components with layer-wise coefficients that account for the spectral and geometric structure of the model. Experiments on CLIP models fine-tuned on ImageNet and evaluated under natural distribution shifts, as well as on Qwen language models tested on mathematical reasoning and multiple-choice benchmarks, show that this plug-and-play approach is a practical and effective alternative to multi-checkpoint methods, retaining much of their benefits without their computational overhead.

Modern AI systems are usually built in two steps. First, a large model is trained on broad, general data to learn a wide range of skills. Then it is "fine-tuned" on a narrower task, such as solving math problems or recognizing specific objects, to make it better at that one job. Unfortunately, this specialization comes at a cost: as the model gets better at its target task, it often becomes worse at handling inputs that look even slightly different from what it was fine-tuned on, such as the same math problem written in another language. This loss of flexibility is a major obstacle to deploying AI reliably in the real world. One popular fix is to train many copies of the model with slightly different settings and then average them together, an approach known as "model soups." This works well, but it is enormously expensive, since it requires training and storing dozens of separate models. We introduce a method called MonoSoup that achieves a similar benefit using only a single fine-tuned model, with no extra training and no access to the original training data. Our key insight is that the changes fine-tuning makes to a model can be mathematically separated into two parts: strong changes that capture the new skill, and faint changes that quietly preserve the model's original, more general knowledge. Earlier work often discarded these faint changes as noise, but we show that they actually hold information crucial for staying robust. MonoSoup automatically rebalances these two parts so the model keeps its new skill without forgetting its general abilities. Across both image and language models, MonoSoup restores much of the lost robustness in seconds, turning what used to require training many models into a quick, one-time adjustment of a single one. This makes reliable AI more accessible to people and organizations without large computing budgets.