AlgoTrace: Algorithmic Primitives and Compositional Geometry of Reasoning in Language Models

How do inference time and latent computations enable large language models (LLMs) to solve multi-step reasoning problems? We introduce AlgoTrace, a framework for tracing and steering algorithmic operations in the model latent space for multi-step reasoning. We operationalize primitives by clustering latent activations of the model when solving four benchmarks: Traveling Salesperson Problem (TSP), 3SAT, AIME, and Graph Navigation. We annotate the clusters using their corresponding tokens in the reasoning trace. We then apply function vector methods to extract primitive vectors as reusable compositional building blocks of reasoning. We find that a) injecting a primitive vector into models (Phi, Qwen, Llama) elicits the associated algorithmic operation in the reasoning trace, b) injecting primitives can steer behavior across tasks, c) primitive vectors can be composed through algebraic operations, revealing a geometric logic in activation space, and d) a fine-tuned model exhibits improved composition of primitives (Phi-4-Reasoning vs. Phi-4). These findings demonstrate that LLM reasoning can be understood as a walk through algorithmic primitives in the latent space governed by compositional geometry. These primitives transfer across tasks, and reasoning finetuning strengthens algorithmic generalization and composition across domains.

How do large language models accomplish reasoning on new tasks? Here we define reasoning as a walk through the internal processes of language models, which in turn generates the solution as a natural language sequence in the output. We propose an interpretability framework where given a task such as finding the shortest path from A to B, we identify algorithmic reasoning steps that we call primitives (e.g., proposing or comparing potential paths), and characterize how they compose together. We identify potential algorithmic primitives by identifying clusters of activations that are particularly similar. We find that these clusters play highly coherent roles in the natural language sequence. They allow us to inject these algorithmic primitives into the model layers to causally induce specific reasoning behaviors and to compare which algorithmic primitives are used by different models and how they are composed. By identifying and demonstrating the causal roles of algorithmic primitives, we provide a bridge between the internal computations of LLMs and their external reasoning traces. This framework could help us to better predict what problems language models will succeed or fail on and can suggest future methods for improving their training.