TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoning

Estimating uncertainty for AI agents in real-world multi-turn tool-using interaction with humans is difficult because failures are often triggered by sparse critical episodes (e.g., looping, incoherent tool use, or user-agent miscoordination) even when local generation appears confident. Existing uncertainty proxies focus on single-shot text generation and therefore miss these trajectory-level breakdown signals. We introduce TRACER, a trajectory-level uncertainty metric for dual-control Tool-Agent-User interaction. TRACER combines content-aware surprisal with situational-awareness signals, semantic and lexical repetition, and tool-grounded coherence gaps, and aggregates them using a tail-focused risk functional with a MAX-composite step risk to surface decisive anomalies. We evaluate TRACER on $\tau^2$-bench (Barres et al., 2025) by predicting task failure and selective task execution. To this end, TRACER improves AUROC by up to 37.1\% and AUARC by up to 55\% over baselines, enabling earlier and more accurate detection of uncertainty in complex conversational tool-use settings. Our codebase is available at: https://github.com/sinatayebati/agent-tracer.

When AI assistants interact with humans and use tools—like booking a flight or searching a database—it can be hard to tell when they are confused. Existing methods for measuring this "uncertainty" only check one AI response at a time. They miss bigger, conversation-wide problems, like the AI getting stuck in a loop or failing to coordinate with the user. To fix this, we created TRACER, a new way to measure uncertainty across the entire conversation. TRACER acts like a safety monitor, looking for specific warning signs: unexpected responses, repetitive looping, and illogical tool use. It highlights the most severe mistakes to figure out if the overall task is going off track. We tested TRACER on a challenging dataset to see if it could predict AI failures. We found that it significantly outperformed existing methods, enabling much earlier and more accurate detection of when an AI is struggling in complex, real-world tasks.