Group-wise Data Ordering: Enhancing Instruction Tuning of Large Language Models via Embedding Proximity

Instruction tuning (IT) is a central mechanism for aligning large language models (LLMs) with user intent. In practice, randomly shuffling the training set is a simple yet surprisingly strong baseline. However, it overlooks latent structure, such as domain and reasoning depth, and thus interleaves heterogeneous objectives, which can induce gradient conflicts and diminish effective optimization progress. To this end, we propose EP-Order, an embedding-proximity-based data-ordering paradigm for IT of LLMs. Unlike previous paradigms that derive order from per-example scores, EP-Order explicitly accounts for inter-sample correlations by operating in representation space. EP-Order trains a warm-up model on a small subset of data (e.g., 10%), embeds all training samples for clustering, and ranks these clusters according to embedding proximity. To handle sharp gradient changes at cluster transitions and alleviate catastrophic forgetting under cluster-based training, we introduce mixed regions that interleave samples from the previous, current, and next clusters, thereby stabilizing learning. Extensive experiments on 14 benchmarks spanning vision-language, text-only, and hybrid thinking/no-thinking scenarios show that EP-Order achieves broadly consistent improvements over random shuffling. These results suggest that exploiting embedding-level data structure offers an effective and general direction for improving IT on complex, high-conflict training data.

Large language models learn from many training examples. Usually, these examples are shown in a random order. This is simple, but it can mix very different examples together, such as image questions, math problems, and open-ended writing tasks. As a result, the model may learn less smoothly. We propose EP-Order, a method that puts similar examples closer together during training. The model first gets a rough sense of the data, then groups examples that look similar to it. It also uses transition and replay examples to make training smoother and reduce forgetting. Our results show that EP-Order improves training across multimodal, text-only, and hybrid thinking text-only settings. This suggests that data order matters more than people often assume. Since EP-Order only changes the order of examples, it can help models learn better without making them larger or slower to use.