Reasoning Structure of Large Language Models

Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count. However, identical scores on these metrics can hide fundamentally different reasoning structures. To address this limitation, we introduce a scalable LRM benchmark of logic puzzles and a pipeline that converts unstructured traces into verifiable reasoning graphs of claims and dependencies. This turns reasoning into a structured, measurable object whose topology can be quantitatively analyzed. Building on this, we define a reasoning efficiency metric that quantifies how concentrated the model's logical flow is. Our analysis on open-source reasoning models shows that structural measurements separate behaviors that token count and accuracy conflate, providing a practical tool for diagnosing failure modes and comparing how reasoning scales with puzzle difficulty.

When a student solves a problem, you can grade the final answer, count the scratch work, or read the reasoning. Today's AI reasoning models are usually graded the first two ways, but identical scores can hide different ways of thinking. Two models can reach the same answer with one walking a clean path and another wandering through dead ends. We built a benchmark of 21 logic puzzles across four difficulty levels and a pipeline that turns model reasoning into a map of claims and their connections. Each claim is checked against the rules, so we can see where the logic goes off track, not just the final answer. From this map we compute an efficiency score that captures how focused the model's logic is on the solution. Across four reasoning models, longer outputs did not mean better reasoning, with extra text mostly spent on checking rather than new steps. Our score separates models that look identical on accuracy and length, revealing real differences in thinking. On the hardest puzzles, no model succeeds however long it thinks, suggesting current systems hit a wall. We hope this gives a richer way to compare and diagnose reasoning models beyond leaderboards.