InteractComp: Evaluating Search Agents With Ambiguous Queries

Language agents have demonstrated remarkable potential in web search and information retrieval. However, many search-agent benchmarks assume that user queries are complete and unambiguous. This assumption leaves under-tested a practical failure mode: agents may face ambiguous requests where the intended target cannot be identified without clarification. Yet most agents lack interactive mechanisms during the search process, and existing benchmarks cannot assess this capability. To address this gap, we introduce InteractComp, a benchmark designed to evaluate whether search agents can recognize query ambiguity and actively interact to resolve it during search. Following the principle of easy to verify, interact to disambiguate, we construct 210 expert-curated questions across 9 domains through a target-distractor methodology that creates controlled ambiguity resolvable only through interaction. Evaluation of 17 models reveals striking failure: the best model achieves only 13.73% accuracy despite 71.50% with complete context, exposing systematic overconfidence rather than reasoning deficits. Forced interaction produces dramatic gains, demonstrating latent capability current strategies fail to engage. Longitudinal analysis shows interaction capabilities stagnated over 15 months while search performance improved seven-fold, revealing a critical blind spot. This stagnation, coupled with the immediate feedback inherent to search tasks, makes InteractComp a valuable resource for both evaluating and training interaction capabilities in search agents. The code is available at https://github.com/FoundationAgents/InteractComp

AI search assistants are becoming better at finding information on the web, but they usually assume that users ask clear and complete questions. In real life, people often ask vague or incomplete questions. For example, they may describe something they only partly remember, and the assistant must ask follow-up questions before it can know what the user really means. In this paper, we study whether current AI search agents can handle such ambiguous questions. We build a benchmark called InteractComp, where each question has several plausible interpretations and can only be answered correctly if the agent asks useful clarification questions. The final answers are short and easy to check, so the benchmark can clearly measure whether the agent resolved the ambiguity. We test 17 strong AI models and find that even the best model performs poorly when it must decide by itself whether to ask questions. However, when the missing context is directly provided, the same models perform much better. This suggests that the main problem is not that models cannot reason about the answer, but that they are overconfident and often fail to ask for the information they need. Our results show that interaction is still a major weakness of today’s search agents. InteractComp provides a way to evaluate and improve future AI systems so that they can better recognize uncertainty, ask helpful questions, and work with users more reliably.