Fantastic Adaptive Taxonomies and How to Use Them

Many LLM-based methods improve agent systems without changing model weights: a verifier selects the best trajectory from N attempts, a prompt or workflow optimizer rewrites the agent after failed runs, and a runtime reflection module asks the agent to reconsider before acting. These methods typically operate by asking an LLM to read execution traces directly. We show that they can be improved by first constructing a failure taxonomy for the target domain and agent system, and then passing this taxonomy to the LLM-based improvement procedure. We present ATLAS (Automatic Taxonomy Learning for Agent Systems), a framework that creates Adaptive Taxonomies: 15–30 failure codes induced from execution traces, with no hand-authored codes and no per-trace annotation. We evaluate ATLAS’s adaptive taxonomies in three downstream procedures. (1) As best-of-N judges on Terminal-Bench 2.0, ATLAS based LLM-Judge agents set a new high score (89.9% accuracy, +15 pp over Pass@1) and outperform judges with fixed taxonomies or no taxonomies. (2) As mutation feedback for evolutionary agent-system optimization, taxonomy-coded diagnoses outperform free-form LLM reflection across competitive programming, math, STEM QA, and discrete reasoning. (3) As runtime feedback for SWE-agent on SWE-bench-Verified-Mini, adaptive taxonomies improve over free-form reflection in both in-prompt and external-judge integrations. On TRAIL, induced taxonomies align with expert annotations at Cohen’s κ=0.725.