Sample Margin-Aware Recalibration of Temperature Scaling

Deep neural networks frequently exhibit overconfidence, undermining reliability in safety-critical applications. Existing adaptive methods rely on indirectly learned proxies of sample difficulty. We establish the logit margin as a direct and principled hardness indicator. We prove that margin constrains the feasible temperature range for a target confidence. Empirically, margin strongly correlates with decision boundary proximity and reveals systematic calibration patterns across difficulty levels. We further identify a fundamental flaw in NLL-based optimization: minimizing NLL can paradoxically worsen calibration. To address this, we introduce Charbonnier-SoftECE, a smooth objective that provably upper-bounds the smooth calibration error (smCE). Building on these insights, we propose SMART (Sample Margin-Aware Recalibration of Temperature), a lightweight method that learns a sample-wise margin-to-temperature mapping guided by our calibration-centric objective. Experiments demonstrate state-of-the-art calibration across CNNs and ViTs on standard, long-tailed, and distribution-shifted benchmarks, with minimal inference-time overhead. Code is available at: https://github.com/Misakaaaaaz/ICML2026-SMART.

AI systems often make predictions with a confidence score, but these scores can be misleading: a model may say it is 99% sure and still be wrong much more often than that. This is especially concerning in settings such as medical or autonomous decision-making, where people may rely on the model’s confidence. Our work asks how to make a trained model’s confidence more trustworthy without retraining the whole model. We found that a simple clue, the gap between the model’s first and second choices, tells us a lot about how hard a case is. Based on this clue, we built SMART, a lightweight method that adjusts each prediction’s confidence after the model has already been trained. SMART keeps the model’s predicted answer the same, but makes the confidence score better reflect how often the answer is actually correct. Across many image datasets and model types, SMART produced more reliable confidence estimates with very little extra computation. This makes existing AI systems easier to trust and evaluate in practice.