A Narrowing Geometry in Contaminated Reasoning

Despite the advancing reasoning capabilities of large language models (LLMs), many reasoning evaluations are increasingly compromised by data contamination, which induces unreliable contaminated reasoning on leaked inputs. While this phenomenon is widely observed, its underlying mechanism remains poorly understood, hindering the ability to distinguish generalization from memorization and to develop effective solutions. In this work, we first identify a distinctive signal of contaminated reasoning, namely the mutual information decay between representations and gradients. Our mechanistic analysis reveals that contaminated models exhibit pronounced eigenspectrum concentration in their representations, leading to a low-dimensional computation regime. Under leaked inputs, this mechanism weakens the linear coupling between representations and gradients, manifested as a structural decay of the singular values in the whitened space. We show that this narrowing geometry mathematically implies a reduction in mutual information, and further demonstrate the practical utility of our analysis by successfully restoring the reasoning behavior of contaminated models, achieving up to an 11.03% absolute improvement in average consistency with the base model over the strongest baseline. Our code is available at https://github.com/jiakuan929/ContamReasoning.

Large language models can solve many difficult reasoning problems, but their performance may be misleading when similar questions or solution patterns have appeared in their training data. This issue, known as data contamination, makes it hard to tell whether a model is genuinely solving a new problem or relying on patterns it has already seen. In this work, we study what happens inside a model during contaminated reasoning. We find that the model's internal computation becomes concentrated in a much lower-dimensional space, suggesting that it relies on a narrower set of patterns rather than using more flexible reasoning. Based on this observation, we develop a method to identify and reduce the effects of contaminated reasoning. Our work helps researchers make language model evaluations more accurate and avoid overestimating a model's reasoning ability.