Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducing ***N**euron-Activated Graph Ranking* (NAG-based Ranking), a training-free and interpretable framework for target pretraining data selection. Rather than using black-box representations, our approach directly characterizes each target input by a sparse set of high-impact neurons in any off-the-shelf LLMs. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact Neuron-Activated Graph (NAG), and rank candidate data by NAG similarity to target examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9\% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3\% accuracy on HellaSwag. It also remains effective under a more applicable multi-target setting, where our best setup surpasses two baselines by 1.1\% and 4.1\%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g., deactivating NAG-selected neurons (only 0.12\% of all) causes a 23.5\% performance collapse, and restricting NAG to the final layer incurs a 4.1\% average drop, indicating that NAG captures a sparse ``functional backbone'' for learning target features. We release the code at https://github.com/asillycat/NAG.

Training a language model to excel at a specific skill — like medical reasoning or math — requires choosing the right training data from an ocean of text. But how do you know which data will actually teach the skill you want? Current approaches either rely on vague notions of "quality" or use opaque scoring systems that are hard to understand or improve. We take a different approach: we look inside an existing model to see which internal components activate when it processes examples of the target skill. We then search through candidate training data and select texts that trigger the same internal components — much like finding books that exercise the same mental muscles. Our method, called NAG, requires no extra training and is fully transparent: we can point to exactly which components matter. Models trained on NAG-selected data consistently outperform those trained on randomly chosen or generically filtered data across six reasoning benchmarks. We also show that the components NAG identifies are genuinely critical — disabling just 0.12% of them causes performance to collapse by 23.5%.