The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoning

Knowledge distillation from powerful reasoning models is widely used to improve Small Language Models (SLMs) on mathematical reasoning, often assuming that traces with higher reward model scores provide more useful supervision. We identify a counterintuitive Quality-Utility Paradox in mathematical reasoning distillation. Data refined or synthesized by a stronger Oracle obtains higher perceived quality according to reward models, yet consistently underperforms traces generated by the SLM itself and selected through rejection sampling across Qwen2.5, LLaMA-3, and DeepSeek families. Our analysis shows that Oracle refinement couples logical repair with distributional drift away from the SLM's native reasoning distribution. This drift increases the learner's adaptation cost and can outweigh the benefit of improved reasoning logic. To test this mechanism, we introduce Style-Aligned Refinement, which preserves the native trajectory of the SLM while retaining logical repair from the Oracle. This intervention lowers adaptation cost and restores downstream utility, allowing distilled SLMs to match or surpass baselines generated by the SLMs themselves. These findings suggest that effective mathematical reasoning distillation should optimize perceived quality together with compatibility between learner and data. The datasets and code are available at https://github.com/Dracoqhl/Quality-Utility-Paradox.

This paper studies how to choose training examples for smaller AI models that solve math problems. A common assumption is that answers judged as higher quality by a stronger AI model should make better training data. We find that this is not always true. In our experiments, polished answers from a stronger model often receive higher quality scores, but they can make smaller models perform worse than answers generated by the smaller model itself. The reason is that the stronger model may express solutions in a style that is harder for the smaller model to learn, even when the logic is improved. We show that keeping the smaller model's natural problem-solving style while correcting its mistakes leads to better learning. These results suggest that useful training data should be judged not only by how good it looks, but also by how well it matches the model that will learn from it.