Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning

Supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) are two widely used post-training paradigms for improving the reasoning ability of large language models (LLMs). Recent methods attempt to integrate SFT and RLVR in a single stage by reweighting or scheduling their objectives. However, such coupling can be counterproductive because supervised updates are not uniformly beneficial for reward optimization. To address this, we propose BRIDGE, a scalable framework in which SFT learns to supervise RL by selectively transferring knowledge that improves reward optimization. Specifically, BRIDGE alternates two updates at each meta-training step: a base-model update that fuses the SFT and RL gradients, and an update to a lightweight low-rank adapter (LoRA) that coordinates the two objectives by maximizing a cooperative-gain signal, defined as the reward of joint SFT–RL training over an RL-only baseline. Across five mathematical reasoning benchmarks, BRIDGE consistently outperforms two-stage cold start, naive mixing, and representative single-stage integration baselines, yielding over three points average absolute improvement and more stable training dynamics. We further show that BRIDGE extends to logical reasoning and generalizes out-of-distribution to code and science without additional training, while staying robust under noisy rewards.

When teaching an AI to reason — say, to solve math problems — there are two common approaches. One shows the AI worked-out examples to imitate, like a student copying solved homework. The other lets the AI try problems on its own and rewards it when the final answer is correct, like learning by trial and error. Each has weaknesses, so people often combine them, but the two methods can pull the AI in conflicting directions: imitating an expert's steps doesn't always help the AI get more answers right on its own. We developed BRIDGE, a method that lets the "imitation" teacher learn when its guidance actually helps the AI succeed, rather than forcing the AI to copy at every step. The teacher only passes along the lessons that improve the AI's own problem-solving. Across several math benchmarks, BRIDGE outperformed standard ways of combining the two approaches and trained more stably. It also extended to logic puzzles, and the math-trained model generalized to code and science problems it had never been trained on — holding up even when the rewards were sometimes wrong.