RAGEN-2: Reasoning Collapse in Agentic RL

RL training of multi-turn LLM agents is unstable, and reasoning quality drives task performance. Entropy, the standard reasoning-stability monitor, only measures within-input diversity and misses whether reasoning depends on the input. We identify template collapse: stable entropy alongside input-agnostic boilerplate, invisible to entropy and existing metrics. We diagnose it via a mutual-information (MI) proxy that scores cross-input distinguishability online; across tasks, MI correlates with final performance far more strongly than entropy. We then explain collapse via a signal-to-noise ratio (SNR) mechanism: low within-input reward variance weakens task gradients, letting input-agnostic regularization dominate and erase cross-input differences. We mitigate this with SNR-Aware Filtering, prioritizing high-variance prompts each iteration. Across planning, math reasoning, web navigation, and code execution, the method consistently improves input dependence and task performance.

When AI systems learn through trial and error to complete tasks like writing code or browsing the web, they can develop a hidden failure: they start producing the same generic responses regardless of what they are actually asked. Existing monitoring tools miss this because they only check whether responses within a single conversation are varied — not whether responses actually change across different situations. We call this template collapse — like a student who memorizes a fill-in-the-blank essay structure and applies it to every question. The responses look diverse word-by-word but are effectively interchangeable. We show that standard monitoring tools are blind to this by design: they measure variety, not relevance. To catch collapse, we developed a new diagnostic that measures whether the AI's reasoning actually shifts when the input shifts. This signal predicts task performance far better than existing tools. We traced the root cause to training situations where the reward signal is too weak, letting the AI fall back on input-agnostic habits. The fix is simple: prioritize training examples where the reward signal is strong, so the model has a real reason to pay attention to the input. Across coding, math, web navigation, and planning, this consistently prevents collapse and improves performance. An AI agent that doesn't actually respond to its situation — however fluent it appears — is unreliable in deployment. This work gives practitioners a way to detect and fix that before it causes harm.