The Truth Lies Somewhere in the Middle (of the Generated Tokens)

How should the sequence of hidden states produced during autoregressive generation be compressed into a representation that reflects the model’s internal state? We study representations derived from generated tokens and compare them to grounded embeddings across several domains. We find that pooling embeddings across tokens produces more informative representations than any individual token. This observation is consistent with semantic information being distributed across generated tokens rather than localized to a single position. In this setting, alignment provides a way to study how a model’s internal representations evolve and pooling offers a more reliable summary of the model's state across generation.

This paper studies how to extract semantic representations from hidden states produced during autoregressive generation. Across vision-language, reasoning, and protein-language settings, we find that mean pooling over generated-token hidden states yields representations that align better with reference spaces than any individual token representation. This improvement is not explained by averaging alone: convex mixtures of different token segments, decoding seeds, and semantic views also improve alignment, suggesting that semantic information is distributed across the generation trajectory. Generated-token representations also outperform prompt-token representations, whose mean pooling does not show the same benefit under causal masking. Finally, tokenwise alignment reveals interpretable phases of generation, such as recall and task-specific response, and these dynamics depend on the model embedding its own outputs. Overall, the results suggest that generated hidden states should be treated as trajectory-level representations rather than isolated token states.