Expanding the Capabilities of Reinforcement Learning via Text Feedback

The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other extreme, distillation offers dense supervision but requires demonstrations, which are costly and difficult to scale. We study natural language feedback as an intermediate signal: richer than scalar rewards, yet cheaper than complete demonstrations. Textual feedback is a natural mode of human interaction and is already abundant in many real-world settings, where users, tools, and automated judges routinely critique LLM outputs. Towards leveraging text feedback at scale, we formalize a multi-turn RL setup where text feedback is available during training but not at inference. Therefore, models must learn to internalize the feedback in order to improve their test-time single-turn performance. To do this, we propose two methods: Self Distillation, which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling, which predicts the feedback as an auxiliary objective. We provide theoretical analysis on both methods, and empirically evaluate on reasoning puzzles, competition math, and creative writing tasks. Our results show that both methods consistently outperform strong baselines across benchmarks, highlighting the potential of RL with an additional source of rich supervision at scale.

When training AI language models to improve their responses, the standard approach gives the model only a simple thumbs-up or thumbs-down after each attempt — a single bit of information that says nothing about what went wrong or how to fix it. At the other extreme, providing full corrected solutions is expensive and hard to scale. We study a middle ground: training with written critiques — the kind of feedback a teacher might give a student, such as "you made an error in step 3" or "your argument missed a key constraint." This text feedback is richer than a simple score, yet far cheaper than writing out a complete solution. The challenge is that such feedback is available during training but typically not when the model is actually used. We develop two methods that help the model internalize critiques so it learns to avoid mistakes on its own, without needing feedback at test time. One method treats the model's own corrected responses as demonstrations to learn from; the other trains the model to predict what a critic would say, sharpening its internal error awareness. Across reasoning, math, and creative writing tasks, both methods substantially outperform standard training approaches.