Dynamic Linear Attention

The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost. To improve representation capacity under long contexts, recent approaches organize memory in a multi-state manner. However, existing multi-state linear attention methods rely on fixed state merging policies that cannot adapt to dynamically varying token importance, irreversibly obscuring critical tokens and causing severe error accumulation over long sequences. To address this limitation, we propose DLA, a dynamic memory modeling framework for multi-state linear attention. DLA introduces (i) Information-Aware Dynamic State Merging, which adaptively determines state boundaries based on token-level information variation, preserving high-resolution representations around semantic transitions while aggressively summarizing stable regions, and (ii) Capacity-Bounded Memory Modeling, which maintains a fixed-size, chronologically ordered state cache by selectively merging adjacent low-information states to control memory growth with minimal information loss. We pre-train DLA on two different linear attention models and evaluate on 10 datasets from three different aspects. Experimental results demonstrate the superiority of DLA over state-of-the-art.

Large language models often need to process long inputs, but standard attention becomes very expensive as inputs grow. Linear attention is more efficient, but it may lose important details by compressing long histories too aggressively. We propose Dynamic Linear Attention (DLA), which adaptively decides what information to preserve and what to compress. DLA keeps more detailed memory around important content changes while summarizing stable regions more compactly. It also bounds the number of memory states to maintain efficient inference. Experiments show that DLA improves both accuracy and efficiency on reasoning, retrieval, and long-context tasks.