TuneAhead: Predicting Fine-tuning Performance Before Training Begins

Fine-tuning large language models (LLMs) is compute-intensive and error-prone: model performance depends sensitively on data quality and hyperparameter choices, and naïve runs can even degrade model performance. This raises a practical question:can we predict fine-tuning performance before committing to a full training run? We present TUNEAHEAD, a lightweight framework for pre-hoc prediction of fine-tuning performance. TUNEAHEAD encodes each candidate run as a meta-feature vector that combines static dataset descriptors with dynamic probe features from a short standardized probe. A predictor maps these features to performance estimates, while SHAP-based attributions provide interpretable diagnostics that reveal which specific features drive the prediction. Across 1,300+ fine-tuning runs on Qwen2.5-7B-Instruct, TUNEAHEAD consistently outperforms strong baselines such as Early-Stop Extrapolation and ProxyLM. On a held-out test set of 370 runs, TUNEAHEAD achieves an RMSE of 1.47 percentage points and places 95.1% of predictions within ±3 percentage points of the true score. These accurate continuous predictions support practical go/no-go screening policies that can reduce unnecessary full fine-tuning while retaining most promising runs.

Fine-tuning large language models can be costly, and not every fine-tuning attempt improves performance. In practice, researchers and engineers often need to decide whether a dataset and training setup are worth the full computational cost, but this is usually known only after training is finished. This paper introduces TuneAhead, a lightweight method that predicts fine-tuning performance before full training begins. TuneAhead looks at two kinds of low-cost signals: properties of the dataset itself, such as size, diversity, and possible repetition, and signals from a short early training probe, such as how stable the loss and gradients are. These signals are used to estimate the final performance of a candidate fine-tuning run. The method also explains its prediction by showing which dataset or training factors contributed most to a likely success or failure. By identifying unpromising runs early, TuneAhead can help avoid wasting computation on fine-tuning attempts that are unlikely to work. It can also guide users toward targeted fixes, such as cleaning duplicated data or adjusting unstable training settings.