Failure-Driven Workflow Refinement

Workflow optimization for tool-using LLM agents is often cast as global search over candidate graphs, scored by a scalar metric. This collapses rich, multi-step failure traces into binary outcomes, obscuring recurring failure structure and making refinement inefficient. We reframe optimization as \emph{distributional refinement}: each workflow induces a density over a \textbf{Failure Signature Space} $\mathcal{F}$, and the goal is to minimize its \textbf{Expected Failure Mass}. We propose \textbf{CE-Graph}, which maintains a counterexample pool, estimates dense failure modes, and applies operator-constrained graph edits via a \textbf{Propose-and-Verify} loop with a convergence-aware stopping rule. Across math, code, and QA benchmarks, CE-Graph improves robustness while reducing optimization cost compared to strong workflow-search baselines, suggesting reliability emerges from learning and reshaping failure landscapes rather than merely maximizing aggregate success rates.

Many AI systems now solve difficult problems by breaking them into several steps, using tools, and checking their own answers. However, when these systems make mistakes, most existing improvement methods only record whether the final answer was right or wrong. This loses valuable information about where the mistake happened and why it happened, making improvement slow and expensive. This paper introduces CE-Graph, a method that helps AI systems improve by carefully studying their failures. Instead of treating mistakes as random, CE-Graph collects failed cases, groups similar failures together, and identifies the most common problems. It then makes targeted changes to the system’s workflow and tests whether those changes actually fix the problem. Across tasks involving math reasoning, code generation, and question answering, CE-Graph makes AI workflows more reliable while using fewer resources than previous search-based methods. The main message of this work is that robust AI systems should not only chase higher scores, but should also learn systematically from their recurring mistakes.