InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

The rapid evolution of Large Language Models has catalyzed a surge in scientific idea production, yet this leap has not been accompanied by a matching advance in idea evaluation. The fundamental nature of scientific evaluation needs knowledgeable grounding, collective deliberation, and multi-criteria decision-making. However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation dimensions, and the inherent bias in LLM-as-a-Judge. To address these, we regard idea evaluation as a knowledge-grounded, multi-perspective reasoning problem and introduce InnoEval, a deep innovation evaluation framework designed to emulate human-level idea assessment. We apply a heterogeneous deep knowledge search engine that retrieves and grounds dynamic evidence from diverse online sources. We further achieve review consensus with an innovation review board containing reviewers with distinct academic backgrounds, enabling a multi-dimensional decoupled evaluation across multiple metrics. We construct comprehensive datasets derived from authoritative peer-reviewed submissions to benchmark InnoEval. Experiments demonstrate that InnoEval can consistently outperform baselines in point-wise, pair-wise, and group-wise evaluation tasks, exhibiting judgment patterns and consensus highly aligned with human experts.

Large language models can now generate many research ideas, but deciding which ones are promising still depends on scarce expert reviewers. This creates a bottleneck: weak ideas may consume time, while unusual but valuable ideas may be missed. We study research idea evaluation as a task that needs evidence, several expert viewpoints, and multiple criteria rather than one simple score. We introduce InnoEval, a system that searches recent papers, websites, and code repositories to gather relevant background knowledge. It then simulates a review board of researchers with different expertise and asks them to judge each idea on clarity, novelty, feasibility, validity, and significance. The system combines these views into a report with scores, explanations, supporting evidence, a recommendation, and suggestions for improvement. We test InnoEval on ideas extracted from real conference papers already reviewed by experts and compare it with existing methods. InnoEval better matches expert judgments when judging single ideas, comparing pairs, and ranking groups. Our goal is not to replace human reviewers, but to provide a careful assistant that helps researchers screen, understand, and improve ideas before investing major effort.