Effective Distillation to Hybrid xLSTM Architectures

There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless distillation, which we define in terms of tolerance-corrected Win-and-Tie rates between student and teacher on sets of tasks. To this end, we introduce an effective distillation pipeline for xLSTM-based students. We propose an additional merging stage, where individually linearized experts are combined into a single model. We show the effectiveness of this pipeline by distilling base and instruction-tuned models from the Llama, Qwen, and Olmo families. In many settings, our xLSTM-based students recover most of the teacher's performance, and even exceed it on some downstream tasks. Our contributions are an important step towards more energy-efficient and cost-effective replacements for transformer-based LLMs.

Large language models can answer questions, write code, and solve problems, but they are expensive to run because a key part of their design becomes slower and more memory-hungry as the text gets longer. This makes them costly to deploy, especially when many users or long documents are involved. In this work, we study whether these models can be replaced by more efficient models without losing much of their ability. We start from existing large language models and transfer their behavior into a different kind of model based on xLSTM, a design that can process long text more efficiently. Our method combines a short-range memory for recent words with a long-range recurrent memory, and then trains the new model to imitate the original one. We also train specialists for skills such as math, coding, and instruction following, then merge them into one final model. The resulting models often match, and sometimes exceed, their original teachers on many benchmarks, while being faster and more memory-efficient during generation. This suggests a practical path toward cheaper and more energy-efficient language models.