Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

The diversity of post-training data is critical for effective downstream performance in large language models (LLMs). Many existing approaches to constructing post-training data quantify diversity using text-based metrics that capture linguistic variation, but such metrics provide only weak signals for the task-relevant features that determine downstream performance. In this work, we introduce Feature Activation Coverage (FAC) which measures data diversity in an interpretable feature space. Building upon this metric, we further propose a diversity-driven data synthesis framework, named FAC Synthesis, that first uses a sparse autoencoder to identify missing features from a seed dataset, and then generates synthetic samples that explicitly reflect these features. Experiments show that our approach consistently improves both data diversity and downstream performance on various tasks, including instruction following, toxicity detection, reward modeling, and behavior steering. Interestingly, we identify a shared, interpretable feature space across model families (i.e., LLaMA, Mistral, and Qwen), enabling cross-model knowledge transfer. Our work provides a solid and practical methodology for exploring data-centric optimization of LLMs.

Large language models often need additional training data to perform well on specific tasks, but simply collecting or generating more data is expensive and may not improve performance if the data is repetitive or misses important cases. This paper proposes a method for measuring data diversity inside the model’s own feature space, rather than only looking at surface-level text differences. Using sparse autoencoders, our method identifies which task-relevant features are missing from an initial dataset and then generates new synthetic examples to cover those missing features. Experiments across instruction following, toxicity detection, reward modeling, and behavior steering show that this approach improves model performance while using far less synthetic data than prior methods. The results suggest that carefully targeted data can be more useful than simply scaling up dataset size.