Low-cost Full Fine-tuning: Learning What to Update for LLMs

While Large language models (LLMs) have strong abilities, they generally rely on fine-tuning to supplement downstream task-specific knowledge. Due to the prohibitive memory overhead of full fine-tuning (FT), existing parameter-efficient fine-tuning techniques, e.g., LoRA and Adapters, update parameters only in low-rank or restricted subspaces. However, they fail to approximate FT---the performative fine-tuner---and risk performance degradation in tough tasks. Therefore, we naturally raise a Low-cost Full Fine-tuning question: Can we approach standard full fine-tuning in theory, yet with much lower costs in practice? Our key insight is that performing selective updates at each step can, theoretically, recover FT asymptotically, while being cost-effective and ignoring no parameter direction. This motivates a new general fine-tuning paradigm (called Think-Touch): we first predict potentials of parameter groups (think) and then update only the selected (touch) in one step. Theoretically, we show that under a very weak sufficient condition---divergence of the cumulative coverage of the expected gradient norm---any selection strategy can converge in the full-parameter space to a stationary point at which the FT admits no further first-order improvement. Besides, we further derive the general convergence rate for our paradigm and identify a post-hoc greedy strategy that is rate-optimal. Unfortunately, this strategy cannot be directly applied in practice due to its reliance on full and accurate gradient information. Thus, we propose a bandit-based method to online approximate this ideal strategy in the long run with a rigorous regret guarantee. Extensive experimental results on various tasks demonstrate the potential of our paradigm, including much lower space overheads against FT and better performance than LoRAs.

Large language models can be very capable, but adapting them to a new task often requires changing many internal model values. The most powerful way to do this can be expensive because it needs a large amount of computer memory. Cheaper alternatives usually change only a limited part of the model, which can save memory but may reduce performance on difficult tasks. This paper studies how to get closer to the benefits of full model adaptation while keeping the cost much lower. The main idea is simple: instead of changing every part of the model at every training step, the method first estimates which parts are likely to matter most, and then updates only those parts. Over time, different parts of the model can still be updated, so the method is not permanently restricted to a small fixed subset. We also provide mathematical analysis explaining when this kind of selective updating can approach the behavior of full adaptation. Experiments on language, math, and coding tasks show that the method can achieve strong performance while using much less memory and less time per training step than full adaptation.