Distributional Alignment Games for Answer-Level Fine-Tuning

We focus on the problem of \emph{Answer-Level Fine-Tuning} (ALFT), where the goal is to optimize a language model based on the correctness or properties of its final answers, rather than the specific reasoning traces used to produce them. Directly optimizing answer-level objectives is computationally intractable due to the need to marginalize over the vast space of latent reasoning paths. To overcome this, we propose a general game-theoretical framework that lifts the problem to a Distributional Alignment Game. We formulate ALFT as a two-player game between a Policy (the generator) and a Target (an auxiliary distribution). We prove that the Nash Equilibrium of this game corresponds exactly to the solution of the original answer-level optimization problem. This variational perspective transforms the intractable marginalization problem into a tractable projection problem. We demonstrate that this framework unifies recent approaches to diversity and self-improvement (coherence) and provide efficient algorithms compatible with Group Relative Policy Optimization (GRPO), such as Coherence-GRPO, yielding significant complexity gains in mathematical reasoning tasks.

Large language models solve problems by writing out reasoning steps before producing a final answer. When training these models, we ultimately care about the quality of that final answer, not the particular steps used to reach it. However, optimizing directly for answer quality is extremely difficult because countless reasoning paths can lead to the same answer, and examining them all is computationally infeasible. We propose a new approach that recasts this training problem as a two-player game. One player is the language model, which generates reasoning. The other is a "target" representing the ideal distribution of answers. We prove that the solution of this game exactly recovers the best possible model, turning an intractable problem into a manageable one. Different training goals (encouraging diverse solutions, improving self-consistency, or ensuring safe outputs) all arise naturally as different strategies within the same game, unifying techniques that were previously designed separately. We develop practical algorithms from this framework and show that models improve their math accuracy by up to 9 percentage points and question-answering performance by up to 42%, without requiring any human-provided correct answers.