D$^3$: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training

Training data plays a central role in large language models (LLMs) optimization, motivating extensive research on data scheduling strategies. Most existing approaches concentrate on adjusting the overall data distribution but neglect the underlying interactions between samples during training. However, we argue that such interactions cannot be overlooked, as real-world data samples frequently exhibit directional influences on each other, making the training order crucial. Intuitively, we can prioritize train-units with greater influence to improves learning efficiency. In this work, we propose $D^3$, a **D**ynamic, **D**irectional graph-constrained **D**ata scheduling framework. $D^3$ formulates the complex interactions among train-units as a dynamic influence graph, where edges represent loss-based dependencies. It then solves a constrained optimization problem over this graph to derive the training order, which ensures that the data sequence respects the evolving information flow throughout training. Our approach is theoretically motivated and yields consistent improvements over existing data scheduling methods across both pre-training and post-training phases. Furthermore, for scalability, $D^3$ also employs an efficient approximation algorithm that keeps the additional computational overhead within a manageable range. For future research, the code is available at https://github.com/xuyj233/D3.

Training data is central to optimizing large language models (LLMs), yet most existing scheduling strategies focus solely on adjusting the overall data distribution while ignoring how individual samples interact during training. We argue that these interactions matter: real-world data samples frequently exert directional influence on one another, making the order in which they are presented crucial. Intuitively, prioritizing training units with greater downstream influence should improve learning efficiency. We propose D³, a Dynamic, Directional graph-constrained Data scheduling framework. D³ models the complex interactions among training units as a dynamic influence graph, where edges capture loss-based dependencies between samples. It then solves a constrained optimization problem over this graph to determine the training order, ensuring the data sequence respects the evolving information flow throughout training. Theoretically motivated, D³ achieves consistent improvements over existing scheduling methods across both pre-training and post-training phases. For scalability, D³ employs an efficient approximation algorithm that keeps computational overhead manageable. Code: https://github.com/xuyj233/D3.