ADK Arena: Surfacing Failure Modes in Agent Development Kits via LLM-as-a-Developer

Agent Development Kits (ADKs), framework-level libraries for building LLM-powered agents, have proliferated rapidly, yet their failure modes remain poorly understood. We propose LLM-as-a-Developer, a methodology that deploys an LLM coding agent as a controlled proxy developer to systematically build agents with each framework, surface failures through a three-level validation pipeline, and benchmark the resulting agents on real tasks. The pipeline produces both a failure diagnostic (what breaks during development and why) and an effectiveness measure (how well generated agents perform), creating an automated feedback loop for ADK improvement. We implement this in ADK Arena, evaluating 51 popular Python ADK frameworks across 4 benchmarks (204 agent-benchmark pairs). We find that: (1) 23% of generation attempts fail despite iterative repair, with failures clustering into 9 categories and 70+ error signatures, where the top three (API configuration, async mismatches, breaking changes) account for 63% of all failures; (2) generation cost varies 7.7× across frameworks ($4 to $32 per agent), directly reflecting API complexity; (3) ADK-generated agents resolve only 2-14% of tasks vs. 32-82% for frontier coding agents, quantifying a 3-5× gap that exposes where framework abstractions fall short; (4) documentation quality is the critical fragility axis: removing docs drops generation success from 77% to 32% for less-popular frameworks. These findings provide actionable, per-framework diagnostics that enable ADK developers to identify and fix the failure surfaces that most impede agent reliability.