The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path

Large Language Models (LLMs) are fundamentally limited by representation collapse, a bottleneck that severely degrades long-context performance. We identify that existing approaches risk drifting into one of two pathological extremes: homogenization collapse (e.g., attention sinks causing rank deficiency) and isolation collapse (e.g., local attention causing context disconnection). Through spectral analysis of attention dynamics, we derive an intrinsic trade-off between mixing efficiency (spectral gap) and information capacity (effective rank) that standard mechanisms struggle to balance. To resolve this dilemma, we propose the Topologically Regularized Side-Path (TRSP), a non-invasive architectural intervention that achieves spectral balance. TRSP employs a parameter-free Triangular Box mechanism, scaled by a lightweight, length-aware gate, to regularize the token interaction topology. By integrating proximal coupling to preserve effective rank and distal propagation to support non-degenerate mixing, TRSP promotes a geometrically healthier transition operator without altering core attention. Experiments show significant improvements across general capabilities and long-context benchmarks. Notably, on NoLiMa at $8\times$ the training length, TRSP retains $83\%$ accuracy and surpasses the Differential Transformer and Gated Attention by approximately 30 and 50 percentage points, respectively. Code available at: https://github.com/Eziotao-tyd/TRSP.

Large language models often struggle to process very long documents. When handling large amounts of text, the internal representations of words tend to either blur together into indistinguishable noise or become completely isolated from each other. Existing methods fail to solve both issues at the same time. We address this problem by introducing a new architectural component called the Topologically Regularized Side Path. This lightweight addition acts as a structural guide for the model. It uses a mathematical filtering mechanism to control how information flows between words. This ensures that nearby words share context effectively while distant words maintain strong connections throughout the text. Our method significantly improves the ability of artificial intelligence models to understand and reason over extremely long sequences of text. It outperforms current leading approaches by a large margin on long document evaluation benchmarks. This advancement allows models to process extensive information more reliably without requiring massive computational resources.