GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry

Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific target task. In practice, influence is often measured through the effect of an example on parameter updates. To make selection scalable, many approaches leverage optimizer statistics (e.g., Adam states) as an axis-aligned surrogate for update geometry (i.e., diagonal precondition), implicitly treating parameters as coordinate-wise independent. We show that this assumption breaks down in parameter-efficient fine-tuning (PEFT) methods such as LoRA. In this setting, the induced optimization geometry exhibits strong cross-parameter coupling with non-trivial off-diagonal interactions, while the task-relevant update directions are confined to a low-dimensional subspace. Motivated by this mismatch, we propose GIST (Gradient Isometric Subspace Transformation), a simple yet principled alternative that replaces axis-aligned scaling with robust subspace alignment. GIST recovers a task-specific subspace from validation gradients via singular value decomposition (SVD), projects training gradients into this coupled subspace, and scores examples by their alignment with target directions. Extensive experiments have demonstrated that GIST matches or outperforms the state-of-the-art baseline with only 0.29% of the storage and 25% of the computational time under the same selection budget.

Training large language models often requires huge collections of examples, but not every example is useful for every goal. If we want a model to improve on a specific task, unrelated training examples can waste time, increase cost, or even make the model worse. We developed GIST, a method that chooses a small set of training examples tailored to the target task. The key idea is to look at how the target task “wants” to change the model, then select examples that push the model in similar directions. Instead of treating every model parameter separately, GIST looks for the main shared directions of change, which helps it avoid noisy or misleading examples. Across several language model benchmarks, GIST matched or outperformed strong existing selection methods while using much less computation and storage. This makes fine-tuning language models cheaper and faster, especially when users care about improving performance on a specific task or domain.