CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction

Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's {\em behavioral robustness} to irrelevant or misleading information. In this paper, we argue that a model's true confidence should reflect its stability under cognitive pressure. We introduce \textsc{CaliDist}, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction. \textsc{CaliDist} quantifies how an LLM's predictions and uncertainty change when its input prompt is perturbed with semantic \textit{distractors}. This stability (or lack thereof) signal is then used to adaptively scale the model's initial confidence score. Our extensive experiments on seven Natural Language Understanding classification benchmarks using six distinct LLMs show that \textsc{CaliDist} consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23\% to 7\% on average—a relative improvement of 70\%—demonstrating that behavioral stability is a powerful signal for calibration. We make our code and datasets available at \url{github.com/m-anas-j/CaliDist}.

When we interact with Large Language Models (LLMs), they routinely deliver answers with high confidence, making it incredibly difficult for everyday users to distinguish a genuinely correct response from a convincing but incorrect answer. Traditional methods for fixing this rely on complex statistical adjustments that require deep access to the model's internal workings, making them difficult to apply to popular, closed-source systems. To solve this, we drew inspiration from human psychology: people with a strong understanding of a topic tend to remain stable when challenged, while those with a weaker grasp are more easily swayed by misleading information. We developed CaliDist, a post-hoc method that acts as a cognitive "stress test" for LLMs. By injecting distracting or misleading information into a prompt, our system measures how easily the model's answer or confidence changes. If the model becomes unstable, CaliDist lowers its confidence score to better reflect its reliability. Our experiments show that this simple behavioral check reduces LLM overconfidence by an average of 70%. Crucially, because CaliDist only observes how a model behaves rather than how it is built, it works equally well on closed, proprietary systems from OpenAI and Google — making trustworthy AI more accessible wherever it is deployed.