Uncovering the Gradient Geometry of Long CoT: A Spectral-guided Approach to Reasoning Distillation

Large reasoning models (LRMs) achieve remarkable reasoning performance by generating long chains-of-thought (CoT). However, standard supervised fine-tuning (SFT) treats all tokens uniformly, indiscriminately minimizing loss across both essential reasoning steps and those that are noisy, redundant, or instance-specific. This often leads student models to memorize superficial patterns rather than acquire generalizable reasoning capabilities. To better understand this limitation, we introduce \textit{Loss Subspace Attribution}, a gradient decomposition analysis approach that uncovers a striking geometric structure: Gradients corresponding to effective reasoning predominantly lie within a low-rank consensus subspace, while conflicting or unstructured signals dominate the residual subspace. Guided by this insight, we propose \textbf{\textit{Spectral-guided Learning}}, a step-level distillation strategy that uses spectral strength to identify reasoning steps aligned with the consensus subspace and prioritizes their contribution to parameter updates, while suppressing gradients from the residual subspace. Experiments across various LRMs and diverse complex reasoning tasks consistently demonstrate that focusing optimization on the consensus subspace yields more robust and generalizable student models.

Large artificial intelligence models have recently become much better at solving complex math and science problems by thinking out loud, creating a step-by-step "chain of thought" before giving an answer. To build smaller, faster, and cheaper AI models, scientists usually have them learn by copying these long thinking processes from larger models. However, these long explanations often contain a lot of clutter, such as repetitive phrases, filler words, or distracting side notes that are only relevant to one specific example. When smaller models try to copy everything blindly, they end up memorizing these superficial text patterns instead of learning how to actually reason. In this paper, we look under the hood at the mathematical signals (gradients) that guide an AI's learning process. We discovered a very clean geometric structure: the signals that represent genuine, reusable logic naturally line up and reinforce one another across different problems. Meanwhile, the signals from filler words and unhelpful steps clash and cancel each other out like random noise. Based on this discovery, we created a new training method called Spectral-guided Learning. Instead of forcing the AI to memorize every single word of a long explanation, our method automatically acts like a smart highlighter. It identifies the core logical steps that contain generalizable reasoning and tells the model to focus its learning entirely on them, while safely ignoring the confusing background noise. When tested on difficult academic and math competitions, our method allowed smaller AI models to learn much more effectively, significantly boosting their accuracy while saving training time and computing power.