Coupled Variational Reinforcement Learning for Language Model General Reasoning

While reinforcement learning has achieved impressive progress in language model reasoning, it is constrained by the requirement for verifiable rewards. Recent verifier-free RL methods address this limitation by utilizing the probabilities that LLMs generate reference answers as reward signals. However, these approaches typically sample reasoning traces conditioned only on the question. This design decouples reasoning-trace sampling from answer information, leading to inefficient exploration and incoherence between traces and final answers. In this paper, we propose \textit{\b{Co}upled \b{V}ariational \b{R}einforcement \b{L}earning} (CoVRL), which bridges variational inference and reinforcement learning by coupling prior and posterior distributions through a hybrid sampling strategy. By constructing and optimizing a composite distribution that integrates these two distributions, CoVRL enables efficient exploration while preserving strong thought-answer coherence. Extensive experiments on mathematical and general reasoning benchmarks show that CoVRL improves performance by 12.4\% over the base model and achieves an additional 2.3\% improvement over state-of-the-art verifier-free RL baselines, providing a principled framework for enhancing the general reasoning capabilities of language models.

Large language models can often solve reasoning problems better when they are trained through trial and error, but this usually requires an automatic way to check whether each answer is correct. Such checkers are easy to build for some math problems, but much harder for many more general questions where correct answers may be expressed in different ways. This paper introduces CoVRL, a training method that helps language models improve their reasoning without relying on an external answer checker. During training, the model learns in two complementary ways: sometimes it tries to solve a question normally, as it would at test time; other times it is also shown the correct answer and learns to produce a reasoning process that supports it. By combining these two forms of practice, the model explores better reasoning paths while still learning skills that can be used when the answer is not available. Experiments on a wide range of math and general reasoning benchmarks show that CoVRL consistently improves model performance over the original model and over previous methods that also avoid external checkers.