Uncertainty Quantification for LLM Agents via Semantic Abstraction Trajectories

Quantifying uncertainty in agentic LLM outputs is essential for safe deployment, yet existing approaches require access to model internals or ignore the reasoning trace entirely. We propose a black-box uncertainty signal derived from \emph{structured probabilistic inference over an agent's reasoning trace}. We define Semantic Level of Detail (SLoD), a continuous macro-to-micro abstraction axis, and recover per-chunk abstraction scores via a Plackett--Luce ranking model fit to pairwise LLM judgments -- a latent-variable inference step that requires only a non-reasoning judge. The resulting SLoD trajectory is a structured time series whose shape serves as a selective-prediction signal. A LightGBM classifier over interpretable trajectory-shape detectors predicts SWE-agent failures at ROC-AUC $0.89\pm0.02$ and, used as an out-of-fold scorer to abstain from low-quality attempts, lifts Pass@1 on FrontierScience Olympiad from $58.2\%$ to $68.0\%$ over five DeepSeek-V3.2 attempts (recovering $39.5\%$ of the Pass@5 oracle gap); the coverage--accuracy curve dominates a length-only baseline at every operating point. The same features classify hallucination categories on AgentHallu/OpenManus at macro-AUC $0.71$. We also report negative results on GPQA-Diamond and Qwen3-30B traces, delineating the conditions under which the signal is informative.