InstEmb: Instruction-Following Embeddings through Glimpses of the Future

Recent advances have empowered large language models (LLMs) with remarkable fine-grained instruction-following capabilities in text generation tasks. However, embedding methods typically rely solely on the hidden state of the input's last token, limiting their ability to capture complete semantic signals distributed across the full output tokens. Moreover, existing discrete-to-continuous re-encoding approaches introduce semantic discontinuity. To address these limitations, we propose $\textbf{InstEmb}$, a novel instruction following embedding framework. InstEmb jointly optimizes two key aspects: (1) Input-Intrinsic semantic information, achieved by employing contrastive learning focused on the representation of the last input token, and (2) Output-Aware semantic information, captured through representation self-distillation leveraging learnable look-ahead tokens without introducing additional decoding latency. Additionally, we introduce $\textbf{Dual-Anchor Alignment Pooling (DAAP)}$, explicitly aligned with our dual training objectives. Extensive experiments demonstrate that InstEmb achieves state-of-the-art performance across multiple instruction following benchmarks without benchmark-specific supervised data.

Here’s a lay summary of the paper based on the abstract: --- Lay Summary Large language models (LLMs) are very good at following detailed instructions when generating text. However, when we try to convert text into numerical representations called embeddings (which help computers understand meaning), current methods have two big problems. First, they usually only look at the very last word of the input, missing important meaning that might be spread across the entire output. Second, when they try to turn discrete words into continuous embeddings, some meaning gets lost. To fix this, the researchers propose a new framework called InstEmb. It learns two kinds of information at the same time: 1. Input-intrinsic meaning – using a technique that focuses on the last input token while comparing different examples. 2. Output-aware meaning – using a clever trick with "look‑ahead" tokens that anticipate future outputs, without slowing down the system. They also introduce a new method called Dual‑Anchor Alignment Pooling (DAAP) that aligns perfectly with these two goals. Experiments show that InstEmb works better than existing methods on several instruction‑following tests – and it does this without needing any special training data for those specific tests. In simple terms: InstEmb helps computers understand instructions more fully, by paying attention to both the input and the output, leading to more accurate and meaningful text embeddings.