Fine-Tuning of Transformer models with Frames

Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scales with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling highly efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, strongly reducing the memory footprint and parameter count. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames, give sizable compute benefits. Our technical analysis shows that FrameFT allows obtaining formal convergence results. We evaluate our method across a suite of supervised fine-tuning benchmarks, primarily focusing on language tasks, but also report applicability to vision models. Our empirical evaluations show that FrameFT achieves performance on par with or exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters and less memory.

AI models are incredibly powerful, but adapting them for specific tasks requires a process called "fine-tuning," which is often slow and expensive. While researchers have found ways to update only a fraction of a model to save costs, these methods still consume massive amounts of memory when companies try to host thousands of custom AI assistants at once. To solve this, we propose FrameFT. Instead of making heavy, broad updates to the model weights, FrameFT uses a geometric concept called "Fusion Frames" to pinpoint a highly organized, sparse network of essential connections. Think of it like using a precise set of overlapping stencils to paint only the exact details needed for a new job, rather than repainting the entire canvas. By sharing these structural "stencils" across the model, we only need to adjust a tiny fraction of the model weights. The result is an approach that matches or beats the top fine-tuning methods while using up to 30 times fewer trainable parts. By drastically shrinking the storage footprint, FrameFT makes it remarkably cheaper, faster, and more sustainable to run thousands of specialized AI models simultaneously.