Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs

Zeroth-order optimizers have recently emerged as an attractive approach for fine-tuning large language models (LLMs), as they avoid backpropagation and can substantially reduce memory overhead relative to standard first-order training. However, existing zeroth-order methods rely on hand-crafted, static sampling strategies that are not adaptable to model-specific structures. To address this, we propose ZO-Finetuner, a learning-based zeroth-order optimizer for LLMs that automatically learns efficient perturbation strategies through a compact and memory-efficient design. Motivated by the fact that a small set of base LLMs is repeatedly fine-tuned across tasks, ZO-Finetuner supports one-time per-model training and reuse across downstream tasks with minimal overhead. Therefore, learning the optimizer once for a given LLM and reusing it across diverse downstream tasks is both feasible and highly desirable. Accordingly, ZO-Finetuner is designed to scale learning to learn (L2L) to the foundation-model era by supporting one-time per-model training with minimal overhead. Experiments on 4 LLMs and 7 datasets show that ZO-Finetuner outperforms prior zeroth-order baselines in 82.1\% of task-model combinations, thereby demonstrating strong performance and scalability for efficient LLM fine-tuning.

Fine-tuning large language models can require a large amount of GPU memory, because standard training methods need to store many intermediate values during learning. A recent alternative, called zeroth-order fine-tuning, avoids this expensive step by updating the model using only a few measurements of how the loss changes when the model parameters are slightly perturbed. However, existing zeroth-order methods usually choose these perturbations by hand, which may not match the structure of each language model. We introduce ZO Fine-tuner, a learned optimizer that automatically decides how strongly to perturb different parts of a language model during fine-tuning. Importantly, the learned optimizer can be trained once for a base model and then reused across many downstream tasks and related model checkpoints. Across four language models and seven datasets, ZO Fine-tuner improves over prior zeroth-order baselines in most settings, while adding very little memory or runtime overhead. This suggests a practical path toward making language-model fine-tuning more accessible on limited hardware.