Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws

This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting, named model steering. While ad-hoc methods have been used in various contexts, including the training of large foundation models, its underlying principles remain insufficiently understood, leading to sub-optimal performance. In this work, we propose a theory-driven framework for model steering called DRRho risk minimization, which is rooted in Distributionally Robust Optimization (DRO). Through a generalization analysis, we provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model. To the best of our knowledge, this is the first time such theoretical insights are provided for the new learning paradigm, which significantly enhance our understanding and practice of model steering. Building on these insights and the connection between contrastive learning and DRO, we introduce a novel method for Contrastive Language-Image Pretraining (CLIP) with a reference model, termed DRRho-CLIP. Extensive experiments validate the theoretical insights, reveal a superior scaling law compared to CLIP without a reference model, and demonstrate its strength over existing heuristic approaches. Code is released at github.com/Optimization-AI/DRRho-CLIP

Training powerful AI models often requires massive amounts of data and computing power. In this paper, we try to find an approach for the following question: How can AI models learn more efficiently by following the guidance of an existing AI model?First we developed a novel framework, named DRRho risk minimization, to show how the guidance from an existing AI model helps the new AI learn more effectively and perform better on new tasks. We provide theoretical results showing that our framework helps new AI model learn better than existing frameworks.Then we proposed DRRho-CLIP, a practical application of our framework for AIs that help them better understand both images and text. This approach allows the new AI to learn with significantly less data, sometimes even outperform its guiding AI, and improve more rapidly as computing resources increase.