LitReview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform

Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspects of research utility depend on expert judgment rather than reference-overlap metrics. We introduce \textbf{LitReview Arena}, a battle-style evaluation platform with a structured protocol tailored to literature review quality: domain experts with AI paper-writing experience compare anonymized drafts, are matched to topics within their expertise, and provide dimension-wise outcomes over five literature-review-specific criteria. From this protocol, we collect 3k expert judgments, each containing five dimension-wise outcomes, and show that even the strongest current systems win only 23.0\% of decisive matches against human drafts on overall utility, while agentic LLMs such as Sonar Deep Research substantially outperform base language models by over 60\%. We further find that existing LLM-as-a-judge methods are substantially misaligned with human experts (Spearman's $\rho \approx 0.467$), especially on synthesis-heavy criteria such as paper structure and research suggestions. Using the collected preference data, we provide an expert-calibrated evaluator, LitJudge, which improves alignment to $\rho \approx 0.78$, comparable to inter-expert consistency; code and data are publicly available at https://github.com/VanellopeAsher/LitReview-Arena.

Scientific literature reviews help researchers understand a field, compare existing work, and identify questions that are worth studying next. As AI systems become better at writing such reviews, we need to know whether their outputs are useful to researchers, rather than only checking whether they include the right keywords or citations. We built LitReview Arena, a platform where domain experts compare anonymized literature review drafts side by side and judge their coverage, evidence, organization, research suggestions, and overall usefulness. Based on these expert comparisons, we find that current AI systems still lag far behind human-written drafts, especially in parts of the review that require judgment and synthesis. We also find that automatic evaluators often disagree with expert researchers, so we develop LitJudge, an evaluator designed to better match expert preferences. Our work releases a public dataset and an evaluation process to help future AI systems produce scientific reviews that are more reliable and more useful to researchers.