Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning

Large language models (LLMs) using chain-of-thought reasoning often waste substantial compute by producing long, incorrect responses. Abstention can mitigate this by withholding outputs unlikely to be correct. While most abstention methods decide to withhold outputs before or after generation, dynamic mid-generation abstention considers early termination of unpromising reasoning traces at each token position. Prior work has explored empirical variants of this idea, but principled guidance for the abstention rule remains lacking. We present a formal analysis of dynamic abstention for LLMs, modeling abstention as an explicit action within a regularized reinforcement learning framework. An abstention reward parameter controls the trade-off between compute and information. We show that abstaining when the value function falls below this reward strictly outperforms natural baselines under general conditions. We further derive a principled and efficient method to approximate the value function. Empirical results on mathematical reasoning tasks support our theory and demonstrate improved selective accuracy over existing methods.

Large language models that "think step by step" often produce long chains of reasoning that end in the wrong answer — and wrong answers tend to be longer than right ones, wasting compute on responses that fail. Existing fixes either predict failure from the question alone (too soon to tell) or check the answer after generation (too late to save effort). We give the model a way to recognize mid-thought when its reasoning is unlikely to pan out, and to stop early and hand off to a person or a stronger system. A small "confidence meter" reads the model's internal state at each step and estimates the chance the trace will end correctly; when this drops below a chosen bar, the model quits. We prove this stop-when-stuck rule beats fixed-time alternatives and is optimal under idealized assumptions. On grade-school and Olympiad-level math, the method nearly doubles accuracy on the hardest problems among the answers the model commits to, while saving most of the wasted compute. The same idea also helps the model avoid producing toxic text. Teaching models to know when to quit could make AI cheaper, greener, and more trustworthy in settings where confidently wrong answers cause real harm.