Attention Illuminates LLM Reasoning: The Uncovered Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

The reasoning patterns of large language models (LLMs) remain opaque, and Reinforcement learning (RL) typically assigns uniform credit across an entire generation, blurring the distinction between pivotal and routine steps. This work treats attention as a natural substrate for interpreting LLM reasoning and a window for aligning optimization with its internal dynamics. We first distinguish attention heads between locally and globally focused information processing and reveal that locally focused heads produce a sawtooth pattern near the diagonal indicating phrasal chunks, while globally focused heads expose tokens that exert broad downstream influence over future tokens. We quantify these with two metrics measuring the extent of backward attention within a clipped window and the average attention a token receives from subsequent tokens, respectively. Taken together, these signals indicate a recurring preplan-and-anchor regularity, where the model first performs a long-range contextual reference to generate an introductory token, which is immediately followed by or coincides with a semantic anchor token that organizes subsequent reasoning. Leveraging these insights, we introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes (preplan tokens, anchor tokens, and their temporal coupling) and show consistent performance gains across various reasoning tasks.

A central question in AI research is whether we can strengthen the reasoning abilities of large language models (LLMs) more effectively by first gaining a deeper understanding of how they reason internally. This work takes a step toward answering that question. LLMs generate text one word at a time, and buried inside each generation is an "attention" mechanism that records what the model focuses on as it thinks. By carefully analyzing these attention patterns, we uncover a recurring rhythm in how models reason through hard problems. Some tokens act as planners, where the model reaches back into earlier context to set up a new reasoning step. Others act as anchors, tokens that the model repeatedly revisits to keep its reasoning on track. These two roles appear together in a consistent plan-then-anchor pattern across a wide range of problems and model sizes. We then use this understanding to improve how models are trained. Standard training methods spread learning signals uniformly across every token in a response, making no distinction between a pivotal reasoning step and a routine filler word. Instead, we dynamically identify the planner and anchor tokens through attention and assign them stronger training credit. This focuses learning on the moments that actually matter. The result is a training approach that is more efficient, reaches higher accuracy on challenging mathematical reasoning benchmarks, and works across multiple model families, all with negligible added computational cost. Understanding the internal rhythm of model reasoning, it turns out, opens a practical path to making that reasoning stronger.