DenseSteer: Steering Small Language Models towards Dense Math Reasoning

Large language models (LLMs) demonstrate strong chain-of-thought (CoT) reasoning abilities, while smaller models ($\leq$ 3B parameters) significantly underperform on multi-step reasoning tasks. Based on empirical analyses of the Qwen-2.5 model family on math reasoning benchmarks, we find that more proficient reasoning is associated with fewer reasoning steps but higher information density per step, a property we term *Dense Reasoning*. Motivated by this observation, we propose **DenseSteer**, a training-free inference-time steering framework that enhances small-model reasoning by modulating internal representations toward dense reasoning patterns. Experiments show that our method yields consistent accuracy improvements without increasing token-level Negative Log-Likelihood, highlighting dense reasoning as an effective structural approach to mathematical problem solving.

Small language models are cheaper and easier to run than very large models, but they often struggle with multi-step math problems. One reason is that their explanations can become long, fragmented, or repetitive, which creates more chances for mistakes. We study how stronger models organize their reasoning and find that successful solutions often use fewer, more informative steps. Based on this observation, we introduce DenseSteer, a method that guides a small model while it is generating an answer so that its reasoning becomes more compact and accurate. DenseSteer does not require retraining and uses only 50 examples, making it much cheaper than many training-based approaches. Across several math benchmarks, this guidance improves the accuracy of small models. Our results suggest that changing how a model organizes its reasoning can be an efficient way to make smaller language models more capable.