Attention Projection Mixing with Exogenous Anchors

Cross-layer reuse of early attention projections can improve optimization and data efficiency, but it creates a structural conflict: the first layer must simultaneously act as a stable, reusable anchor for all deeper layers and as an effective computational block. We demonstrate that this tension constrains the performance of internal-anchor designs. We propose ExoFormer, which resolves the conflict by learning exogenous anchor projections outside the sequential layer stack. We introduce a unified normalized mixing framework that mixes queries, keys, values, and gate logits using learnable coefficients (exploring coefficient granularities: elementwise, headwise, and scalar), and we show that normalizing anchor sources is key to stable reuse. ExoFormer variants consistently outperform their internal-anchor counterparts, and the dynamic variant yields 1.5x downstream accuracy points while matching validation loss using 1.5x fewer tokens than Gated Attention. We explain this efficacy via an Offloading Hypothesis: external anchors preserve essential token identity, allowing layers to specialize exclusively in feature transformation. We release code and models to facilitate future research.

Large language models process text through dozens of sequential layers, but a persistent challenge is that deeper layers often blur individual word identities, a phenomenon called oversmoothing. Existing solutions try to preserve early information by recycling the first layer as a reference point for all deeper layers, yet this forces that layer to perform two conflicting roles: acting as a stable anchor while also doing its own computation. We introduce ExoFormer, which resolves this tension by creating dedicated external anchor projections that sit outside the normal layer sequence. These anchors preserve essential word identity and are carefully blended into every layer, freeing the sequential layers to focus entirely on refining and transforming features.