Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth

As LLMs generate increasingly long outputs, effective uncertainty estimation must identify errors at fine-grained levels rather than discard entire responses. While such methods exist, evaluating uncertainty at any resolution (token to an entire generation) is challenging and highly sensitive to label imperfections, making zero-noise benchmarks essential; yet, long-form generation benchmarks tend to rely on fallible labels rather than deterministic ground truth. We introduce Single-answer Atomic Long-form Target (SALT), a benchmark of six procedurally generated tasks with single deterministic long textual ground truths, enabling unit-level evaluation of correctness, calibration, and ranking without external judges. Equipped with SALT, our analysis of 50+ LLMs reveals key insights: We identify which confidence functions dominate each uncertainty aspect and show that confidence ranking largely breaks at atomic resolution, even when clearer separability emerges at coarser line-level units. SALT further enables controlled atom-level interventions throughout generation, revealing two separable drivers of future errors: propagation from corrupted prefixes, dominated by global context correctness, and bounded degradation from increasing answer-context length. Finally, we demonstrate that reasoning, via Chain-of-Thought prompting or internalized through training, introduces a trade-off, improving accuracy while degrading confidence ranking. These findings directly impact risk-critical applications requiring reliable error identification and mitigation. We release the Code at https://github.com/IdoAmit198/SALT.

Large language models can now write long answers, but they still make mistakes, so users need to know which parts are reliable. This is hard to evaluate because long answers often require human or AI judges, whose labels can also be wrong. We introduce SALT, a benchmark for testing long-form model outputs using tasks with exact, deterministic answers. SALT lets us check each small part of an answer directly, without relying on subjective judgment. We use it to evaluate more than 50 language models and study whether their confidence scores identify mistakes. Our results reveal a key limitation: model confidence can rank large chunks of an answer reasonably well, but often breaks down on smaller, high-resolution parts. A model may know that one line is riskier than another, yet fail to pinpoint the exact wrong step or item. We also find that step-by-step reasoning can improve accuracy while hurting confidence ranking, and that earlier mistakes can strongly increase the chance of later mistakes. These findings highlight the need for better methods to detect specific errors in long model outputs.