Learning a Generative Meta-Model of LLM Activations

Existing approaches for analyzing neural network activations, such as PCA and sparse autoencoders, rely on strong structural assumptions. Generative models offer an alternative: they can uncover structure without such assumptions and act as priors that improve intervention fidelity. We explore this direction by training diffusion models on one billion residual stream activations, creating "meta-models" that learn the distribution of a network's internal states. We find that diffusion loss decreases smoothly with compute and reliably predicts downstream utility. In particular, applying the meta-model's learned prior to steering interventions improves fluency, with larger gains as loss decreases. Moreover, the meta-model's neurons increasingly isolate concepts into individual units, with sparse probing scores that scale as loss decreases. These results suggest generative meta-models offer a scalable path toward interpretability without restrictive structural assumptions.

As a large language model (LLM) reads and writes text, it produces a stream of internal signals, similar to neural activity in the human brain. Researchers study these signals to understand and steer how the LLM behaves. However, today’s tools require us to assume in advance how these signals are organized, which may not match reality. We build a new tool without such assumptions. We take a billion snapshots of the LLM’s internal signals and train a second model, or “meta-model,” on them. We train the meta-model using a popular technique from image generation, called “diffusion.” The meta-model has many practical uses. It can help steer an LLM’s behavior more precisely, without the usual tradeoff in writing quality. It can also help us understand the LLM’s signals, as it holds cleaner versions of those signals that respond more sharply to specific concepts. Finally, the meta-model reliably improves with more computational resources, in a predictable way.