A Risk Decomposition Framework for Pre-hoc Fine-tuning Prediction

The high cost of fine-tuning LLMs poses a significant economic barrier; pre-hoc performance prediction offers a critical solution to substantially reduce this expense. However, the theoretical limits of pre-hoc performance prediction remain unexplored. We formulate it as a stochastic estimation problem under information constraints, decomposing prediction risk into two components: an \textbf{intrinsic limit} (static data-model compatibility) and a \textbf{reducible optimization variance}. We prove that optimization variance admits a necessary lower bound on its decay rate, implying fundamental constraints on how quickly uncertainty dissipates, regardless of the predictor used. Based on these dynamics, we derive a budget-optimal probing principle and introduce a predictability phase diagram that organizes tasks into three distinct regimes: Static-Sufficient, Dynamic-Critical, and Noise-Dominant. Extensive experiments on synthetic and real-world benchmarks validate these theoretical regimes and demonstrate the efficiency of our probing strategy.

Large language models are often adapted to new tasks through fine-tuning, but running a full fine-tuning process can be expensive and uncertain. In many cases, practitioners do not know in advance whether a dataset, model, or training setup will lead to useful improvements, so they may spend substantial computation on trials that ultimately fail. This paper studies whether we can better understand what makes fine-tuning outcomes predictable before full training is completed. Instead of treating prediction error as a single black-box quantity, we separate it into two parts: uncertainty that cannot be removed even with more computation, and uncertainty that can be reduced by briefly observing the early training process. We show that this reducible uncertainty cannot disappear arbitrarily fast, which means that different tasks require different amounts of early probing. Our framework explains why some fine-tuning tasks can be predicted almost immediately, why others need longer observation, and why some remain inherently difficult to predict. This can help researchers and practitioners decide when early probing is worthwhile and avoid unnecessary fine-tuning costs.