Risk-Controlled Commitment in Instruction-Following Agents: A Controlled Diagnostic
Abstract
Before executing an LLM-parsed goal, an instruction-following agent must decide whether the instruction is specific enough to act on. We study this decision as commitment uncertainty in instruction-following control: should an agent commit to an LLM-parsed goal, abstain, or spend more computation on step-level LLM control? We evaluate this decision in a controlled grid-world testbed using normal coverage, wrong-commit rate, abstention behavior, and token cost. In a symbolic grid world with normal, ambiguous, conflicting, invalid, and empty instructions, we compare one-shot parser+planner, confidence-gated parser+planner, and a step-level controller across three local LLMs. To avoid choosing thresholds by inspection, we use a calibration-set screening rule over a finite grid: select the highest-normal-coverage gate whose binomial upper confidence bound on uncertain-input wrong-commit rate is below a target. Parser+planner is reliable on normal instructions at low token cost, but uncertain inputs expose model-dependent wrong commitments. The selected gate is empirically below the 5% target on held-out seeds for Gemma and Qwen, but not for Llama. In this testbed, a step-level controller, without a parser-like abstention rule or verifier, spends orders of magnitude more tokens but does not improve the commit/abstain decision. These results motivate coverage-risk-cost evaluation for agentic instruction following.