Self-Soupervision: Cooking Model Soups without Labels

Model soups are strange and strangely effective combinations of parameters. They take a model (the stock), fine-tune it into multiple models (the ingredients), and then mix their parameters back into one model (the soup) to improve predictions. While all known soups require supervised learning, and optimize the same loss on labeled data, our recipes for Self-Soupervision generalize soups to self-supervised learning (SSL). Our Self-Souping lets us flavor ingredients on new data sources, e.g. from unlabeled data from a task for transfer or from a shift for robustness. We show that Self-Souping on corrupted test data, then fine-tuning back on uncorrupted train data, boosts robustness by +3.5% (ImageNet-C) and +7% (LAION-C). Self-Soupervision also unlocks countless SSL algorithms to cook the diverse ingredients needed for more robust soups. We show for the first time that ingredients can differ in their SSL hyperparameters---and more surprisingly, in their SSL algorithms. We cook soups of MAE, MoCoV3, MMCR, and LeJEPA ingredients that are more accurate than any single SSL ingredient.

We create machine-learning models by training them to recognize things in images by giving them many image-annotation pairs. The learned algorithm, which does the recognizing, is a set of weights that perform mathematical operations on the input image to make an output-annotation prediction. Combining the weights of many models into a single model is called a model soup, which can perform better than any of the models that went into it—called ingredients—without increasing the cost of making predictions because the number of mathematical operations is not changed. In our work, we make model soups by combining ingredient models that we train without annotations. We make these ingredients without annotations in different ways, for example, training a model to predict hidden patches or to represent the same image in the same way regardless of its color or orientation. Our work makes model soups more general since we no longer need human-annotated datasets for which to create ingredients and soups.