AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

Agentic Web is an emerging paradigm where autonomous agents help users use online information. As the paradigm develops, content providers are also deploying agents to manage their data and serve it through controlled interfaces. This shift moves information access from centralized retrieval to decentralized coordination. To study this setting, we introduce AgentWebBench, a benchmark that evaluates how well a user agent synthesizes answers by interacting with website-specific content agents. We evaluate four tasks that cover common web information needs, spanning ranked retrieval (web search, web recommendation) and open-ended synthesis (question answering, deep research). Across seven advanced LLMs and three coordination strategies, multi-agent coordination generally lags behind centralized retrieval as expected, because user agent cannot directly access the corpus, but the gap shrinks with model scale and can even outperform centralized retrieval on question answering. This benchmark also enables us to study properties of the emerging paradigm of the digital world. We find that decentralized access concentrates traffic toward a small set of websites, test time scaling improves both interaction reliability and task performance, and strong results require sufficient interactions guided by careful planning. Finally, our failure analysis suggests that user agents need better planning and answer synthesis, while content agents need more reliable retrieval and evidence quality. Code, data, and APIs are released on https://github.com/cxcscmu/AgentWebBench.

The web is changing: instead of people typing into a search engine, AI assistants are starting to browse and gather information for us. Meanwhile, many websites are locking their content away, preferring to answer through their own AI helpers rather than let anyone freely copy their pages. Information-gathering becomes a negotiation: your assistant must coax answers from many website "gatekeeper" AIs, each guarding its own slice of the internet, and we don't know how well today's AI copes. We built AgentWebBench, a testing ground that recreates this future web: 100 websites holding 18 million documents, each managed by its own gatekeeper AI. Using seven AI models, we measured how well a user's assistant handles four everyday tasks (searching, getting recommendations, answering questions, and writing research reports) by cooperating with these gatekeepers. We find this scattered web is harder to navigate than today's single search engine, though smarter models close much of the gap and even answer questions better. We also see traffic funneling toward a few popular sites, which could quietly shrink the diversity of voices people see online. Our benchmark gives researchers a shared yardstick for building more capable web assistants and for keeping the open web fair.