$\texttt{MetaDistill}$: Unlocking the Performance Ceiling for Pretrained Optimizers

Meta Black-Box Optimization (MetaBBO) has emerged as a promising paradigm by employing meta learning to automatically optimize the configurations of low-level black-box optimizers. Despite its potential, the generalization of MetaBBO remains significantly constrained when facing unseen, complex objective landscapes. We identify that this bottleneck stems from a restricted performance upper bound inherent in current training mechanisms: by learning from scratch in a self-supervised or unsupervised manner, low-level learnable optimizers are never exposed to advanced, high-quality optimization behaviors, forcing them to converge on suboptimal strategies. In this paper, we propose \texttt{MetaDistill}, a general MetaBBO training framework designed to lift the strategy ceiling through pretraining and test-time fine-tuning. In the pretraining stage, we represent high-quality strategies from classical algorithms as expert optimization trajectories and utilize them for diversity-preserving distillation, enabling the learnable optimizer to internalize advanced optimization behaviors. In the optional fine-tuning stage, we perform self-supervised fine-tuning as a warm-start procedure to further refine the distilled knowledge on unseen tasks. We evaluate our \texttt{MetaDistill} framework on the BBOB test suite and three control tasks. The results demonstrate that \texttt{MetaDistill} significantly improves the generalization ability of various learnable optimizers compared to their original training paradigms. Our code is available at \href{https://github.com/Roki-Xing/MetaDistill}{here}.

Meta black-box optimization aims to train optimizers that can automatically solve a wide range of optimization problems, but existing methods often struggle to generalize to new and complex tasks. We found that a key reason is that these optimizers are typically trained from scratch without access to strong optimization behaviors, which limits the quality of the strategies they can learn. To address this issue, we developed MetaDistill, a training framework that distills optimization knowledge from expert trajectories generated by high-performing classical algorithms into learnable optimizers, thus enables them to internalize advanced search strategies that are difficult to discover through learning from scratch alone. Our work helps to elevate the performance ceiling of learnable optimizers, offering a pathway toward more robust and generalizable optimization systems.