How do LLMs Compute Verbal Confidence?

Verbal confidence—prompting LLMs to state their confidence as a number or category—is widely used to extract uncertainty estimates from black-box models. However, how LLMs internally generate such scores remains unknown. We address two questions: first, when confidence is computed -- just-in-time when requested, or automatically during answer generation and cached for later retrieval; and second, what verbal confidence represents -- token log-probabilities, or a richer evaluation of answer quality? Focusing on Gemma 3 27B (across TriviaQA, BigMath, and MMLU), Qwen 2.5 7B, and the reasoning model Magistral Small 24B, we provide convergent evidence for cached retrieval. Activation steering, patching, noising, and swap experiments reveal that confidence representations emerge at answer-adjacent positions before appearing at the verbalization site. Attention blocking pinpoints the information flow: confidence is gathered from answer tokens, cached at the first post-answer position, then retrieved for output. Critically, linear probing and variance partitioning reveal that these cached representations explain substantial variance in verbal confidence beyond token log-probabilities, suggesting a richer answer-quality evaluation rather than a simple fluency readout. These findings demonstrate that verbal confidence reflects automatic, sophisticated self-evaluation—not post-hoc reconstruction—with implications for understanding metacognition in LLMs and improving calibration.

Large language models like Gemini are increasingly asked to tell us how confident they are in their answers — saying things like "I'm 90% sure" or "almost certain." But how do they actually generate these confidence ratings? Do they compute confidence on the spot when asked, or do they secretly track how well the answer is going as they produce it, storing this internal "feeling" for later use? By carefully intervening in the inner workings of several large language models, we found that confidence is computed automatically during answer generation and stored at a specific location — like a sticky note left on the desk just after writing an answer. When the model is later asked how confident it is, it simply reads this note rather than working it out from scratch. We also found that the model's confidence reflects a genuine evaluation of how well the answer fits the question, not just how easily the words rolled off the tongue. This matters because it suggests language models possess a basic form of self-evaluation — an internal sense of when they might be wrong. Understanding this mechanism is an important step toward making AI more reliable, helping users know when to trust an answer and when to double-check.