Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal “overthinking,” leading to performance degradation. In this work, we quantify inference-time effort by identifying *deep-thinking tokens*—tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that *deep-thinking ratio* (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@$n$, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@$n$ matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.

AI models tackle hard problems by “thinking out loud,” writing long chains of reasoning before answering. A common assumption is that the more an AI writes, the better it thinks. But recent research suggests that longer isn’t always smarter—sometimes it’s just overthinking, leading to wrong answers while wasting energy. So how can we tell whether an AI is genuinely thinking hard rather than just being wordy? We looked inside the model. An AI generates each word by passing it through a series of internal processing stages, like an idea rising through the floors of a building. For easy words, it makes up its mind on the ground floor; for difficult ones, it keeps revising all the way to the top. We measured the fraction of these deeply processed words, what we call “deep-thinking tokens”, and found that it tracks correct answers far more reliably than output length does. We also put this insight to work. By peeking at this signal early in a response, we can identify promising answers and discard unpromising ones. This enables efficient performance that matches or exceeds the standard approach.