Don't Force the Fit: Bounded Log-Likelihood Loss for Enhanced Reasoning in Large Language Models

Supervised fine-tuning (SFT) is central to aligning large language models (LLMs) with instruction following and task-specific reasoning. Despite its success, SFT optimizes token-level likelihoods under the implicit assumption that strictly fitting all tokens in expert demonstrations induces the desired downstream behavior. However, in reasoning tasks where correctness is defined by logical validity or final outcomes rather than exact token realizations, this assumption can lead to optimization misalignment. We empirically observe that low-probability tokens in reasoning demonstrations often correspond to realization-specific or stylistic variations, and that reducing their influence during training consistently improves generalization on reasoning benchmarks. Motivated by this insight, we propose the Bounded Log-Likelihood Loss (BLL-Loss), a simple and parameter-free alternative to standard likelihood training that bounds gradient contributions from low-probability tokens while preserving conventional optimization behavior. We provide theoretical insights and extensive empirical results demonstrating that BLL-Loss improves reasoning generalization across diverse model scales and challenging benchmarks.

Large language models (LLMs) acquire strong reasoning and logic skills during their initial, massive-scale training. However, when developers later "fine-tune" these models to follow specific instructions, the standard training methods have a significant blind spot: they rely on strict, word-for-word matching.Imagine a student who solves a complex math problem using a perfectly correct, alternative method, but receives a failing grade simply because their steps don't exactly match the teacher's answer key. Current AI training does exactly this. If an AI model generates a logically valid reasoning path that happens to be worded differently than the training example, it is severely penalized as if it had made a complete logical error.Our paper addresses this fundamental flaw. We argue that fine-tuning should not force AI to blindly memorize specific paths, but rather elicit the reasoning skills it already possesses. We introduce a new training approach that stops penalizing the model for surface-level phrasing differences, explicitly protecting its valid alternative reasoning paths and allowing its underlying logic to naturally emerge. Ultimately, our method allows AI models to think more flexibly and become more reliable problem solvers without suppressing their valid internal logic.