DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization

Large language models are increasingly deployed in multi-turn interactive settings where users or environments can iteratively provide lightweight feedback. Unfortunately, optimizing such behavior presents a sharp dilemma in practice: online reinforcement learning is able to effectively address multi-turn dynamics but is prohibitively expensive due to the cost of generating full correction trajectories at every update, whereas offline supervised fine-tuning (SFT) is efficient but suffers from distribution shift and behavioral collapse. To this end, we novelly propose DRIFT (Decoupled Rollouts and Importance-Weighted Fine-Tuning), a framework that operationalizes the theoretical insight that the KL-regularized RL objective is equivalent to importance-weighted supervised learning. DRIFT decouples rollout from optimization by sampling offline interaction trajectories from a fixed reference policy, deriving return-based importance weights, and optimizing the policy via weighted SFT on the resulting dataset. Empirically, we demonstrate that DRIFT matches or exceeds the performance of multi-turn reinforcement learning baselines while maintaining the training efficiency and simplicity of standard supervised fine-tuning. Code is available at https://github.com/2020-qqtcg/DRIFT.

As large language models are used more like interactive assistants, they need to do more than answer once: when a user points out a mistake, they should rethink and improve rather than repeat the same error. Training this behavior is difficult. One approach, reinforcement learning, rewards the model through trial and error and can work well, but it is costly because the model must repeatedly generate full back-and-forth conversations during training. A cheaper approach trains on fixed examples, but it often fails to teach the model how to recover after feedback. We propose DRIFT, a method that first collects example conversations from a fixed model, scores them by whether they reach the right answer and how quickly they do so, and then gives more training influence to the better conversations. This lets the model learn from simple feedback such as “Incorrect, please try again” without generating new conversations at every training step. Across math and general reasoning tasks, DRIFT learns stronger correction behavior while keeping training much closer in cost and simplicity to standard example-based training.