TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuning

Instruction tuning is essential for aligning large language models (LLMs) to downstream tasks and commonly relies on large, diverse corpora. However, small, high-quality subsets, known as coresets, can deliver comparable or superior results, though curating them remains challenging. Existing methods often rely on coarse, sample-level signals like gradients, an approach that is computationally expensive and overlooks fine-grained features. To address this, we introduce TRIM (Token Relevance via Interpretable Multi-layer Attention), a forward-only, token-centric framework. Instead of using gradients, TRIM operates by matching underlying representational patterns identified via attention-based "fingerprints" from a handful of target samples. Such an approach makes TRIM highly efficient and uniquely sensitive to the structural features that define a task. Coresets selected by our method consistently outperform state-of-the-art baselines by up to 9% on downstream tasks and even surpass the performance of full-data fine-tuning in some settings. By avoiding expensive backward passes, TRIM achieves this at a fraction of the computational cost. These findings establish TRIM as a scalable and efficient alternative for building high-quality instruction-tuning datasets.

Large language models improve by training on large collections of examples, but not every example is equally useful. Training on all available data can be expensive, slow, and sometimes unnecessary. TRIM addresses this problem by helping choose a much smaller set of training examples that are especially relevant to the task we care about. Instead of looking only at whole examples, TRIM examines how the model pays attention to individual words and phrases inside each example. After a short warm-up stage, it compares these internal signals with the target task and selects examples whose important tokens best match what the model needs to learn. This makes data selection more fine-grained and efficient: rather than treating every training sample as a single unit, TRIM identifies which samples contain the most useful information for the model. Our results show that language models can often be fine-tuned effectively using only a small fraction of the original data, reducing training cost while preserving or improving performance on downstream tasks.