LiveNewsBench: Evaluating Web Search Agents with Freshly Curated News

Large Language Models (LLMs) with agentic web search capabilities show strong potential for tasks requiring real-time information access and complex fact retrieval, yet evaluating such systems remains challenging. We introduce LiveNewsBench, a rigorous and regularly updated benchmark designed to assess the agentic web search abilities of LLMs. LiveNewsBench automatically generates fresh question-answer pairs from recent news articles, ensuring that questions require information beyond an LLM's training data and enabling clear separation between internal knowledge and search capability. The benchmark features intentionally difficult questions requiring multi-hop search queries, page visits, and reasoning, making it well-suited for evaluating agentic search behavior. Our automated data curation and question generation pipeline enables frequent benchmark updates and supports construction of a large-scale training dataset for agentic web search models, addressing the scarcity of such data in the research community. To ensure reliable evaluation, we include a subset of human-verified samples in the test set. We evaluate a broad range of systems using LiveNewsBench, including commercial and open-weight LLMs as well as LLM-based web search APIs. The leaderboard, datasets, and code are publicly available at \url{livenewsbench.com}.

When you ask an AI assistant a question, how do you know whether it actually searched the internet for the answer, or just recalled something it memorized during training? This distinction matters a lot, but it turns out it's surprisingly hard to test. To address this, we built LiveNewsBench: a regularly updated quiz that tests whether AI systems can genuinely find fresh information online. The trick is that every question is based on recent news events — ones that happened after the AI finished its training — so the only way to answer correctly is to actually go out and search the web.