LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States

Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hidden states, which are optimized for next-token prediction and thus often fail to capture global, sentence-level semantics. This paper introduces a novel perspective, demonstrating that attention value vectors capture sentence semantics more effectively than hidden states. We propose *Value Aggregation (VA)*, a simple method that pools token values across multiple layers and token indices. In a training-free setting, VA outperforms other LLM-based embeddings, even matches or surpasses the ensemble-based MetaEOL. Furthermore, we demonstrate that when paired with suitable prompts, the layer attention outputs can be interpreted as aligned weighted value vectors. Specifically, the attention scores of the last token function as the weights, while the output projection matrix ($W_O$) aligns these weighted value vectors with the common space of the LLM residual stream. This refined method, termed *Aligned Weighted VA (AlignedWVA)*, achieves state-of-the-art performance among training-free LLM-based embeddings, outperforming the high-cost MetaEOL by a substantial margin. Finally, we highlight the potential of obtaining strong LLM embedding models through fine-tuning Value Aggregation.

Many artificial intelligence systems need to compare pieces of text by meaning, such as finding relevant documents, grouping similar articles, or matching a question with a useful answer. A common way to do this with large language models is to use the model’s final internal representation from the last layer as a sentence “fingerprint.” However, this representation is mainly trained to help the model guess the next word, so it may focus too much on what comes immediately after the sentence rather than on the full meaning of the sentence. In this paper, we study which internal signals are better for creating sentence fingerprints. We find that attention values, another type of information already computed inside the model, provide a clearer signal of sentence meaning than the commonly used final representation. Based on this finding, we introduce a simple method that combines attention values from different words and model layers, without extra training. This method performs better than many existing no-training methods for text search, similarity, grouping, and related tasks. We also show that a prompted version can further improve performance while avoiding the high cost of methods that require many repeated prompts. These findings suggest a simple way to make existing large language models more useful for search and other text understanding tasks.