The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order

Sequential learning is order-dependent: from Pile-style next-token domain adaptation to instruction-SFT and DPO, $N$ candidate sources induce $N!$ possible curricula. We show that the local order effect is governed by a computable geometric quantity, the Lie-bracket commutator of gradient update fields, yielding a pairwise score for whether $A \to B$ or $B \to A$ is better for a target domain. The pairwise bracket primitive also defines a *Lie-Bracket Tournament*: with a shared $\theta_0$ target-gradient reference, Hessian symmetry gives Borda/row-sum scores from one Hessian-vector product per source, $O(N)$ dot products, and an $O(N\log N)$ sort, without materializing the $O(N^2)$ edge matrix. Empirically, the planner reaches 98.1%/98.9% pairwise accuracy at $k=1$ for instruction-SFT/DPO, remains at 73.1%/72.2% at $k=20$, and preserves the original pretraining-domain evidence with 82.4–92.0% accuracy across four LLMs and 91.1% on diffusion. At curriculum scale, it recovers the best of all $3!$ schedules in 87.5% of trials, ranks 85 Stack programming-language source domains for a Python target in the 99th sampled percentile, and reaches the 99.0–99.6th sampled percentile on 56 MMLU subjects, sharply above the reported descending gradient-norm baseline. These results reframe sequential learning as a geometric tournament problem: commutators provide both local pairwise order information and a scalable primitive for many-domain schedules.

AI models are often improved by training them on several datasets in sequence, such as general text, instruction data, and human preference data. The order of these datasets can change the final model, but trying every possible order quickly becomes impossible: with many datasets, the number of possible schedules becomes enormous. This paper introduces a way to predict good training orders without actually training the model in every order. The method takes a small number of measurements near the starting model and uses them to estimate which dataset should come before another, then combines those pairwise comparisons into a full schedule. Across language-model adaptation, instruction tuning, preference tuning, and many-dataset scheduling, the method often chooses orders that perform close to the best schedules found by much more expensive searches. This can make multi-stage model training cheaper and more systematic, while also helping practitioners choose orders that preserve important capabilities rather than relying on trial and error.