Coverage-Aware Test Generation for Conversational AI Agents

Testing multi-capability conversational AI agents requires diverse, high-coverage test scenarios spanning the agent’s full intent space. Existing synthetic data tools generate scenarios from context documents in a single pass, causing intent collapse—the generated set concentrates on the most prominent topic, leaving the majority of agent capabilities untested. We present Scenario Studio (SS), a config-driven pipeline that prevents intent collapse through coverage-aware orchestration: a PolicyGraph models the agent’s full capability space, a CoverageTracker monitors real-time intent coverage, and gap-directed generation steers each round toward uncovered areas. SS also serves as a pluggable orchestration layer—any open-source generation or red-teaming tool can be integrated as a backend module, immediately benefiting from SS’s compiled context and coverage steering. We benchmark SS against DeepEval Synthesizer and two GPT-prompting baselines across two production agents: Agent-A, a diverse general-support assistant spanning 7+ intent families (127 ground-truth queries), and Agent-B, a narrow ride-booking B2B agent (106 queries). SS achieves 86.6% ground-truth coverage on the diverse agent versus 15.7% for DeepEval—a 5.5 improvement with non-overlapping 95% bootstrap confidence intervals (p<0.05). With a comparable frontier model, SS outperforms premium GPT-5.2 baselines by +14.9pp, demonstrating that architecture matters more than model capability. A dual-judge study confirms 89.0–95.3% per-item agreement with identical tool rankings, and all gains on the diverse agent are statistically significant under both judges. SS also generates adversarial scenarios across 12+ attack categories via hybrid red-teaming backends (native + DeepTeam + PromptFoo) and supports consolidation that removes 31–37% of scenarios while slightly improving coverage.