Oral Thu, Jul 9, 2026 • 12:15 AM – 12:30 AM PDT HALL B2

Measuring Agents in Production

Melissa Pan ⋅ Negar Arabzadeh ⋅ Riccardo Cogo ⋅ Yuxuan Zhu ⋅ Alexander Xiong ⋅ Lakshya A Agrawal ⋅ Huanzhi Mao ⋅ Emma Shen ⋅ Sid Pallerla ⋅ Liana Patel ⋅ Shu Liu ⋅ Tianneng Shi ⋅ Xiaoyuan Liu ⋅ Jared Davis ⋅ Emmanuele Lacavalla ⋅ Alessandro Basile ⋅ Shuyi Yang ⋅ Paul Castro ⋅ Daniel Kang ⋅ Koushik Sen ⋅ Dawn Song ⋅ Joseph E Gonzalez ⋅ Ion Stoica ⋅ Matei Zaharia ⋅ Marquita Ellis

Abstract

LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first systematic study of Measuring Agents in Production, MAP, using first-hand data from agent developers. We conducted 20 case studies via in-depth interviews and surveyed 86 deployed systems practitioners across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and their top development challenges. Our study finds that production agents are built using simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability (consistent correct behavior over time) remains the top development challenge, which practitioners currently address through systems-level design. MAP documents the current state of production agents, providing the research community with visibility into deployment realities and underexplored research avenues.

Lay Summary

AI "agents"—programs designed to complete multi-step tasks autonomously—are entering industries from healthcare to finance. However, many fail in practice, and the secrets behind the successful ones remain hidden inside tech companies. We conducted the first large-scale study of AI agents operating in the real world. By surveying the teams behind 86 deployed agents and interviewing 20 tech organizations, we discovered a surprising trend: the best agents are deliberately simple. Instead of deploying the highly complex, fully independent systems common in research, these teams limit how many steps an agent can take alone and ensure a human reviews the work. Reliability is the biggest hurdle, which developers solve through careful, restrictive system design rather than smarter artificial intelligence. By revealing these closely guarded industry practices, we provide a clear roadmap for the scientific community, highlighting exactly which real-world problems need to be solved next.