AutoJudge: Automatic Generation of LLM-Based Judges from Execution Traces

Evaluating multi-agent systems from LLM execution traces is challenging due to long-context inputs, diverse failure modes, and the lack of adaptable evaluation pipelines. Existing methods rely on fixed, benchmark-specific judges or manual prompt engineering, limiting generality and scalability. We introduce AutoJudge, a framework that automatically constructs LLM-based evaluation pipelines from execution traces. Given a trace, an evaluation taxonomy, and an output schema, a MetaAgent synthesizes a set of intermediate judges and a final aggregation judge, enabling input-adaptive evaluation without task-specific training. To handle long traces, we propose a dual strategy combining full-trace evaluation with a summary-based mode and controlled step-level access to the original trace. The system uses a hierarchical model design that separates pipeline construction from large-scale judging, improving both efficiency and robustness. Experiments across multiple benchmarks show that AutoJudge matches or outperforms prior fixed-judge baselines, while generalizing across heterogeneous evaluation taxonomies and maintaining strong performance under both full-trace and summarized settings. These results suggest that evaluation of complex agent systems can be formulated as a dynamic, inference-time pipeline generation problem. Our code is available at https://anonymous.4open.science/r/AutoJudge-6644/