Contrastive Weak-to-Strong Generalization

Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose \textbf{Contrastive Weak-to-Strong Generalization (ConG)}, a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI. Our code is available at: \url{https://github.com/jianghoucheng/ConG}

Large language models are becoming increasingly capable, but improving them safely often depends on costly human feedback. A promising alternative is to let a smaller, already improved model help train a larger one. However, smaller models can also produce noisy, biased, or low-quality answers, and these problems may be passed on to the larger model. This paper proposes a simple way to make such training signals cleaner. Instead of directly using the smaller model’s answers, we compare how the smaller model behaves before and after it has been improved. This comparison helps identify answers that better reflect the desired behavior while filtering out some of the weaker model’s noise. We then use these higher-quality answers to guide the larger model. Experiments on different model families show that this method consistently improves larger models’ ability to follow instructions, while largely preserving their general abilities on other tasks. The results suggest that better use of weak model supervision can make model improvement more reliable and less dependent on expensive human feedback, though careful evaluation is still needed because the method may inherit remaining biases from the weaker models.