Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression

The deployment of Large Language Models is constrained by the memory and bandwidth demands of static weights and dynamic Key-Value cache. SVD-based compression provides a hardware-friendly solution to reduce these costs. However, existing methods suffer from two key limitations: some are suboptimal in reconstruction error, while others are theoretically optimal but practically inefficient. In this paper, we propose Swift-SVD, an activation-aware, closed-form compression framework that simultaneously guarantees theoretical optimum, practical efficiency and numerical stability. Swift-SVD incrementally aggregates covariance of output activations given a batch of inputs and performs a single eigenvalue decomposition after aggregation, enabling training-free, fast, and optimal layer-wise low-rank approximation. We employ effective rank to analyze local layer-wise compressibility and design a dynamic rank allocation strategy that jointly accounts for local reconstruction loss and end-to-end layer importance. Extensive experiments across six LLMs and eight datasets demonstrate that Swift-SVD outperforms state-of-the-art baselines, achieving optimal compression accuracy while delivering 3–70$\times$ speedups in end-to-end compression time. Our code is available at https://github.com/hiahei/Swift-SVD.

Why This Matters AI assistants have become part of daily life—but running them is surprisingly expensive. These models contain billions of numerical parameters, and every time a user sends a message, the system must load and process enormous amounts of data at high speed. This makes deploying AI costly in both money and energy, limiting who can afford to build and use these systems. A Smarter Way to Slim Down AI One promising solution is compression: mathematically simplifying the model so it runs on less memory without losing much capability—similar to how image formats like JPEG reduce file size while preserving visual quality. We developed Swift-SVD, a new compression method with two advantages over prior approaches. First, it is theoretically optimal: using a mathematical technique called matrix decomposition, it finds the best possible simplified representation of each part of the model in a single pass over data, rather than requiring slow, repeated optimization. Second, it is smart about where to compress: not all parts of an AI model are equally important or equally compressible. Swift-SVD automatically gives more compression to redundant parts and protects the parts the model relies on most—like a careful editor who cuts filler paragraphs but preserves the core argument. What We Found Testing across six AI models and eight tasks, Swift-SVD consistently produced more accurate compressed models than existing methods—and compresses a model up to 70 times faster than prior approaches. This brings practical, efficient AI compression closer to real-world deployment—helping reduce the cost and energy footprint of large AI systems.