Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations

Softmax attention is the cornerstone of modern large language models but is hindered by its linear memory and quadratic compute requirements. Linear recurrent models, such as linear attention and state space models, have become widely studied as alternatives due to their linear compute and constant memory requirements. While these sub-quadratic methods achieve promising efficiency gains and competitive results on a wide range of benchmarks, current linear recurrent models still lag behind on tasks that require long-context retrieval or in-context learning, resulting in a growing body of work on hybrid architectures. In this work, we propose Oryx, which explores a new axis of developing hybrid models: across the sequence, the model has the ability to operate with different mixers, e.g., quadratic attention and linear recurrences, rather than fixing a single mixer throughout. Oryx ties more than 90% of its parameters across mixer types, enabling attention and recurrent modes to operate over shared internal representations. We validate our design with Mamba-2 and Gated DeltaNet variants up to the 1.4B model scale. Under fixed token budgets and a mixed-training strategy, all modes of the 1.4B Oryx outperform their respective baselines by at least 0.7 percentage points on average on downstream language modeling evaluations. On retrieval tasks, Oryx achieves performance comparable to the Transformer baseline even when processing only a tiny fraction (<10%) of the tokens in the attention mode. These results suggest that attention and linear recurrent models can share internal representations, and motivate sequence-axis hybridization as a promising direction for efficient long-context language models.