Data Difficulty and the Generalization–Extrapolation Tradeoff in LLM Fine-Tuning

Data selection during supervised fine-tuning (SFT) can critically change the behavior of large language models (LLMs). Although existing work has studied the effect of selecting data based on heuristics such as perplexity, difficulty, or length, the reported findings are often inconsistent or context-dependent. In this work, we systematically study the role of data difficulty in fine-tuning from both empirical and theoretical perspectives, and find that there is no universally optimal difficulty level; rather, its effectiveness depends on the dataset size. We show that for a fixed data budget, there exists an optimal data difficulty for SFT, and that this optimal difficulty shifts toward harder data as the data budget increases. To explain this phenomenon, we conduct controlled synthetic experiments that reveal a simple underlying mechanism: the interplay between the (in-distribution) generalization gap and the extrapolation gap. We further support this mechanism through a theoretical analysis using PAC-Bayesian generalization bounds. Overall, our results clarify how data size and difficulty jointly affect the trade-off between generalization and extrapolation in SFT, providing guidance for difficulty-based data selection under certain model and data conditions.

When fine-tuning large language models, people often need to decide whether to train on easier or harder examples. Existing studies give conflicting recommendations: some suggest that hard examples are more informative, while others argue that easier examples lead to better generalization. In this work, we show that neither view is universally correct. Instead, the best data difficulty depends strongly on the amount of available training data. With limited data, easier examples are often more effective because the model can learn them reliably. As the training dataset grows, harder examples become increasingly beneficial because they improve the model’s ability to handle challenging tasks. Through both real-world experiments and controlled synthetic studies, we explain this phenomenon as a trade-off between generalization and extrapolation. Our findings provide practical guidance for selecting training data for supervised fine-tuning and help clarify why previous studies reached seemingly contradictory conclusions.