Landmark-Guided Policy Optimization for Multi-Objective Language Model Selection

Selecting a pretrained large language model (LLM) to fine-tune for a task-specific dataset can be time-consuming and costly. With several candidate models available to choose from, varying in size, architecture, and pretraining data, finding the best model for a specific task often involves extensive trial and error. In addition, the "best" model may not necessarily be the one with the lowest test loss, as practical considerations such as deployment costs, inference throughput, and limited search budgets might also play crucial roles. To address this, we introduce LAMPS (LAnguage Model Pareto Selection), a novel and open-source multi-objective AutoML framework that meta-learns a resource allocation policy to efficiently identify (or approximate) the Pareto front of candidate LLMs for a task-specific dataset. It is based on two key ideas: (1) landmark fine-tuning, which generates early performance indicators of the candidate models, and (2) meta-learning via reinforcement learning, which learns an effective selection policy from historical performance data (a meta-dataset). Our results show that, on held-out datasets, LAMPS reduces search time by an average of 73% compared to exhaustive search, while still covering more than 99% of the optimal target space hypervolume.

Choosing the right language model for a new task can be costly because users may need to train and compare many options. The best choice is also not always the largest model, since smaller models can be faster, cheaper, or even more accurate for certain tasks. This paper introduces LAMPS, a method that searches for good model choices more efficiently by first collecting early signs of how each model is likely to perform, then using experience from past tasks to decide which models deserve more training. Across several tasks, LAMPS greatly reduced search time while still finding nearly the same best trade-offs as a much more expensive exhaustive search, making language model selection more practical and resource-efficient.