Navigating the Pareto Frontier of Alignment: Spectrum-Adaptive Fine-Tuning for LLMs

Supervised Fine-Tuning with Negative Log-Likelihood (NLL) remains the standard post-training paradigm for Large Language Models, yet it imposes a disproportionately aggressive update force on low-probability target tokens. This focus forces the model to prioritize minimizing the loss of difficult samples over optimizing the overall quality of the generation, often leading to unwarranted overconfidence. On the other hand, alternatives like Dynamic Fine-Tuning suffer from vanishing gradients on these tokens, which severely hinders the acquisition of new concepts. To bridge this gap, we propose Spectrum-Adaptive Fine-Tuning (SAFT), a unified framework that interpolates between the aggressive learning signal of NLL and the robust nature of probability-weighted optimization. By adaptively balancing these objectives, SAFT effectively mitigates outlier sensitivity without sacrificing learning efficiency. Empirically, our method achieves state-of-the-art performance on mathematical reasoning benchmarks, demonstrating superior generalization on out-of-distribution tasks. Furthermore, evaluations on general conversational alignment validate SAFT's broad adaptability across diverse data regimes. Our code is available at https://github.com/sjtu-scx/SAFT.

When teaching Large Language Models new skills, we typically penalize them heavily for their mistakes. However, if the training data contains human errors or messy information, this strict punishment backfires, forcing the AI to memorize the bad data. Conversely, if we make the training too forgiving to ignore these errors, the AI learns complex, valid reasoning much too slowly. We face a strict dilemma: learn fast but become fragile, or be robust but learn slowly. To solve this, we created SAFT (Spectrum-Adaptive Fine-Tuning). Think of it as a smart teacher that adapts its feedback based on the AI's confidence. Instead of a rigid, one-size-fits-all rule, SAFT dynamically adjusts the learning intensity. It automatically filters out messy, erroneous data to prevent bad memorization, while maintaining a strong, focused learning signal for genuinely difficult yet correct logical steps. This approach gives us the best of both worlds. Our method achieves top-tier performance in complex mathematical reasoning—where training data is notoriously noisy—and also excels in general conversational tasks. Ultimately, SAFT provides a reliable, adaptable recipe for training smarter and more robust AI systems in the real world.