How Reasoning Evolves from Post-Training Data: An Empirical Study Using Chess

We study how reasoning evolves in a language model -- from supervised fine-tuning (SFT) to reinforcement learning (RL) -- by analyzing how a set of theoretically-inspired datasets impacts language model performance in chess. We find that fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance -- however, the RL stage elicits \textit{unfaithful} reasoning (reasoning inconsistent with the chosen move). Alternatively, training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. We show that RL induces a substantial positive shift in the distribution of move quality and reduces hallucination rates as a side effect. Finally, we find several SFT-checkpoint metrics -- metrics spanning evaluation performance, hallucination rates, and reasoning quality -- to be predictive of post-RL model performance. We release checkpoints and final models as well as training data, evaluations, and code that allowed us to surpass leading open-source reasoning models in chess with a 7B-parameter model.

\texit{What makes for an effective language model checkpoint prior to reinforcement learning (RL) post-training}? We choose a domain where our base model (Qwen2.5 7B) struggles in -- chess -- and create several theoretically-inspired supervised fine-tuning (SFT) datasets. Following SFT, we conduct RL on custom verifiable chess tasks. Beyond showing a formula for training a language model to perform well in chess (from a very poor baseline), we study several aspects of model behavior before and after the RL stage including reasoning strategies used, hallucination rates, quantitative performance, and reasoning faithfulness. We find that the data you train on influences reasoning behavior in ways that are unintuitive. For example, by training a model to predict multiple moves (a 'line') during SFT, the model's reasoning remains faithful (true to the final answer); in contrast, training the model to only predict the best move leads to unfaithful reasoning. Additionally, we run statistical tests to show several aspects of an SFT checkpoint are predictive of downstream final RL model performance -- we also conduct analyses on our datasets to gain a proxy for information density of each dataset. We publish all datasets, various code, and scaled models openly.