Confidence is Not Universal: Task-Dependent Calibration and Emergent Behavior in LLMs

Large language models (LLMs) increasingly support human decision-making, rendering human-interpretable confidence essential. However, it remains unclear whether verbalized confidence calibration generalizes across heterogeneous tasks without degrading accuracy. We show that universal confidence calibration fails. Across diverse benchmarks, we identify two incompatible task families with distinct confidence semantics. In reasoning-centric tasks, confidence supervision transfers within the family, often improving calibration while preserving or even improving accuracy, and induces emergent behaviors such as confidence-dependent reasoning length and self-verification. Retrieval- and copy-oriented tasks also exhibit within-family transfer, but fail to generalize to reasoning tasks, with cross-family supervision degrading both calibration and accuracy. Motivated by this finding, we disentangle confidence into reasoning uncertainty and evidence localization uncertainty. This simple decomposition restores cross-family generalization using supervised fine-tuning alone, suggesting that effective confidence alignment requires task-aware semantics rather than a universal scalar notion.

AI language models are increasingly used to help people make important decisions, from solving math problems to answering questions about long documents. For these tools to be trustworthy, they should not only give an answer but also report how confident they are, so that people know when to double check. Many researchers have assumed that a model could be trained once to report its confidence reliably across every kind of task. We show that this assumption does not hold. Confidence turns out to carry two different meanings. In some cases a model is unsure because its step by step reasoning might be wrong, as in a math problem. In other cases it is unsure because it cannot locate the right piece of information inside a long text. Training a model to express one kind of confidence makes it worse at the other. We also observed that when a model was trained to judge its confidence on reasoning problems, it began to think longer and check its own work whenever it was unsure, which led to more accurate answers. Our solution is simple. We let the model report two separate confidence scores, one for its reasoning and one for finding information. This restores reliable confidence across all task types, without any complex training.