Cure-SFT: Diagnostic-Guided Data Curation for Instruction Tuning

Instruction data curation is central to improving the instruction-following ability of large language models. However, existing approaches often struggle to simultaneously maintain data quality, diversity, and distributional consistency, largely because they do not explicitly distinguish semantic redundancy from quality defects and rely on coarse-grained modeling of instruction data quality. To address this issue, we propose Cure-SFT, a coarse-to-fine, diagnostic-guided method for instruction data curation that explicitly disentangles semantic redundancy from quality defects. Specifically, Cure-SFT removes redundant samples via stratified semantic-geometric sampling, applies teacher models for diagnostic triage, and performs targeted defect remediation on fixable samples. Our experiments show that Cure-SFT can surpass full-data instruction tuning using only 10% of the data budget. Moreover, Cure-SFT consistently outperforms strong selection-based and rewriting-based baselines across data budgets, supporting the effectiveness of diagnostic-guided data curation.

Training large language models to follow instructions relies on high-quality examples, but not all examples are equally helpful. Some are repetitive, while others contain mistakes or inconsistencies. We wanted to find a better way to curate instruction data so that models can learn efficiently without being misled by poor examples. To fix this, we created a new method called Cure-SFT that acts like a careful editor. First, it removes repetitive examples to ensure the AI learns from diverse topics. Then, instead of just keeping or tossing the remaining examples, it diagnoses them—sorting them into high-quality, completely useless, or "fixable" categories. For the fixable ones, it performs targeted repairs to correct specific flaws while keeping the valid, original information intact. Our approach proves that repairing data is much more effective than simply discarding or blindly rewriting it. We found that by using this diagnostic method, we could train AI models to perform better than if we had used the entire original dataset, while only needing 10% of the data.