E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing

\textit{Agentic} AI systems execute a sequence of actions, such as reasoning, coding or tool calls, in response to user prompts. To evaluate the success of their trajectories, researchers have developed verifiers, such as process-reward models, to score the quality of each action in an agent's trajectory. Although these heuristic scores can be informative, there are no guarantees of correctness when used to decide whether an agent will yield a successful output. Here, we introduce \textit{e-valuator}, a method to convert any black-box verifier score into a decision rule with provable control of false alarm rates. We frame the problem of distinguishing a successful trajectory---that is, a sequence of actions that will lead to a correct response to the user's prompt---from an unsuccessful trajectory as a sequential hypothesis testing problem. \textit{E-valuator} develops a sequential hypothesis test that remains valid at every step of an agent's trajectory, enabling online monitoring of agents. Empirically, we demonstrate that \textit{e-valuator} provides greater statistical power and better false alarm rate control than other strategies across six datasets and three agents. We additionally show that \textit{e-valuator} can quickly terminate unsuccessful trajectories to save tokens. Together, \textit{e-valuator} provides a lightweight framework that converts verifier heuristics into decision rules with statistical guarantees, enabling the deployment of reliable agents.