SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks

This paper presents a theoretical framework that explains why fine-tuning small, randomly selected subnetworks (slices) within pre-trained models is sufficient for downstream adaptation. We establish that pretrained networks exhibit a universal winning slice property, arising from two phenomena: (1) spectral balance— the eigenspectra of different weight matrix slices are remarkably similar—and (2) high task energy—their backbone representations (pretrained weights) retain rich, task-relevant features. This leads to the Universal Winning Slice Hypothesis, which provides a theoretical foundation for parameter-efficient fine-tuning (PEFT) in large-scale models. Inspired by this, we propose SliceFine, a PEFT method that uses this inherent redundancy by updating only selected slices of the origi- nal weights—introducing zero new parameters, unlike adapter-based approaches. Empirically, SliceFine matches the performance of SOTA PEFT methods across various language and vision tasks, while significantly improving training speed, memory efficiency, and model compactness. Our work bridges theory and prac- tice, offering a theoretically grounded alternative to existing PEFT techniques.

Modern AI systems like large language models and vision models are trained on huge amounts of data, creating "pretrained" models with billions of internal parameters. Adapting these models to a new task — say, teaching a general language model to answer medical questions — usually requires updating only a small portion of those parameters, a practice called parameter-efficient fine-tuning. But it has remained a puzzle why updating such a tiny slice of a giant model works at all, and the most popular methods rely on adding extra components to the model rather than reusing what's already there. This paper shows that the answer lies in the structure pretrained models already have. We find that the internal building blocks of these models are remarkably balanced: no single piece is much stronger or weaker than the rest, and the knowledge needed for new tasks is already concentrated in a small number of directions inside the model. Together, these properties mean that almost any small piece of a pretrained model can be successfully adapted to a new task. Based on this insight, we develop SliceFine, a method that fine-tunes models by simply training tiny strips of the existing weights — adding no new parameters. SliceFine matches the accuracy of leading methods while being faster, using less memory, and producing smaller models, making AI adaptation more efficient and accessible.