Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

Large language models have advanced web agents, yet current agents lack personalization capabilities. Since users rarely specify every detail of their intent, practical web agents must be able to interpret ambiguous queries by inferring user preferences and contexts. To address this challenge, we present Persona2Web, the first benchmark for evaluating personalized web agents on the real open web, built upon the clarify-to-personalize principle, which requires agents to resolve ambiguity based on user history rather than relying on explicit instructions. Persona2Web consists of: (1) user histories that reveal preferences implicitly over long time spans, (2) ambiguous queries that require agents to infer implicit user preferences, and (3) a reasoning-aware evaluation framework that enables fine-grained assessment of personalization. We conduct extensive experiments across various agent architectures, backbone models, history access schemes, and queries with varying ambiguity levels, revealing key challenges in personalized web agent behavior. For reproducibility, our codes and datasets are publicly available at https://serin-kimm.github.io/Persona2Web/

AI assistants are increasingly able to act on our behalf on the web — booking restaurants, shopping, or scheduling appointments. Yet people rarely spell out every detail when asking for help: someone saying "find a good place for dinner tonight" expects the assistant to already know their dietary needs, budget, and favorite neighborhoods. Today's web agents struggle with this everyday vagueness because they depend on users stating every preference explicitly. We built Persona2Web, the first benchmark for measuring how well AI web agents personalize their actions on the real, open web. It pairs simulated users — each with months of past activity that reveals preferences only indirectly — with deliberately ambiguous requests. To succeed, an agent must read between the lines, inferring what a user likely wants from their history rather than asking them to repeat known preferences. Our evaluation also inspects the agent's reasoning, not just its final action, to check whether it truly understood the user.