From Growing to Looping: A Unified View of Iterative Computation in LLMs

Looping, reusing a block of layers across depth, and depth growing, training shallow-to-deep models by duplicating middle layers, have both been linked to stronger reasoning, but their relationship remains unclear. We provide a mechanistic unification: looped and depth-grown models exhibit convergent depth-wise signatures, including increased reliance on late layers and recurring patterns aligned with the looped or grown block. These shared signatures support the view that their gains stem from a common form of iterative computation. Building on this connection, we show that the two techniques are adaptable and composable: applying inference-time looping to the middle blocks of a depth-grown model improves accuracy on some reasoning primitives by up to $2\times$, despite the model never being trained to loop. Both approaches also adapt better than the baseline when given more in-context examples or additional supervised fine-tuning data. Additionally, depth-grown models achieve the largest reasoning gains when using higher-quality, math-heavy cooldown mixtures, which can be further boosted by adapting a middle block to loop. Overall, our results position depth growth and looping as complementary, practical methods for inducing and scaling iterative computation to improve reasoning.

Large language models excel at many tasks, but getting them to reason through multi-step problems, like math or logic puzzles, remains hard. Two training techniques have each been shown to help: looping (recycling the same set of network layers multiple times in sequence) and depth growing (starting with a small model and progressively duplicating its middle layers until full size). Despite their different appearances, it was unclear whether they work for the same reason. We show they do: both push the model to refine its understanding step by step, like a student who rechecks their work, rather than solving everything in one pass. Measurements of how much each layer contributes to the final answer confirm that both approaches shift critical computation toward later layers in the same periodic, structured way. Building on this, we show the two techniques can be combined: looping the middle section of a depth-grown model at test time, without any extra training, can double accuracy on certain reasoning tasks. This unification suggests a simple recipe "grow first, loop later" for building more capable reasoning models without proportionally increasing training or inference cost.