The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models

Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a topological manifold where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model, which frames arithmetic errors as topological slippages caused by internal neural noise pushing a continuous, latent carry potential across quantization thresholds. This geometric framework further elucidates probe versatility, explaining how lightweight probes can disentangle conflicting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference.

Large language models often perform well on complex reasoning tasks, but they can still make simple mistakes when adding long numbers. This paper investigates why this happens by examining the model’s internal states while it produces each digit of an addition answer. We find that the model stores arithmetic information in a highly organized way. Its internal states reflect not only the digit it is about to output, but also important information such as column sums and carries. Many mistakes occur when the model’s internal state lies near the boundary between two possible digits, leading it to choose the wrong neighboring digit. Using this insight, we propose a method that checks whether different internal arithmetic signals agree with each other. This allows us to detect and correct many errors while the model is generating its answer. These findings provide a clearer explanation of how language models handle arithmetic and why they sometimes fail even when useful information is present internally.