SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection

While large language models (LLMs) are pretrained on massive amounts of data, their knowledge coverage remains incomplete in specialized, data-scarce domains, motivating extensive efforts to study synthetic data generation for knowledge injection. We propose $\textbf{SPA}$ ($\textbf{S}$caling $\textbf{P}$rompt-engineered $\textbf{A}$ugmentation), a simple but tough-to-beat baseline that uses a small set of carefully designed prompts to generate large-scale synthetic data for knowledge injection. Through systematic comparisons, we find that $\textbf{SPA}$ outperforms several strong baselines. Furthermore, we identify two key limitations of prior approaches: (1) while RL-based methods may improve the token efficiency of LLM-based data augmentation at small scale, they suffer from diversity collapse as data scales, leading to diminishing returns; and (2) while multi-stage prompting may outperform simple augmentation methods, their advantages can disappear after careful prompt tuning. Our results suggest that, for knowledge injection, careful prompt design combined with straightforward large-scale augmentation can be surprisingly effective, and we hope SPA can serve as a strong baseline for future studies in this area. Our code is available at https://github.com/Tangkexian/SPA.

Large language models are trained on massive amounts of internet text, yet their knowledge remains incomplete in specialized domains such as medicine, finance, and law, where data is scarce. Directly training these models on limited domain data often leads to overfitting rather than robust knowledge acquisition. A popular solution is to use AI to expand the original data into a much larger synthetic dataset, but existing methods often require complex multi-stage pipelines or expensive reinforcement learning procedures. We propose SPA (Scaling Prompt-engineered Augmentation), a simple method that uses seven carefully designed prompt templates to rewrite domain texts into diverse synthetic training data at scale. These templates are inspired by how humans learn, covering concept learning, critical thinking, and generative learning. Despite its simplicity, SPA consistently matches or outperforms substantially more complex methods across benchmarks spanning question answering, document comprehension, and multi-hop reasoning. Our results reveal two key insights: reinforcement learning methods can suffer from diversity collapse as data scales, and adding more stages to a generation pipeline may not help more than improving the quality of the prompts themselves. We hope SPA serves as a strong baseline for future research on teaching language models new knowledge.