SciNet: Evaluating AI Agents in Relation-Aware Scientific Literature Retrieval

AI agents have seen widespread adoption in information retrieval for scientific research, giving rise to tools such as Deep Research. However, existing retrieval agents mainly rely on keyword- or embedding-based methods. While effective at capturing content-level similarities, they struggle to understand complex relational networks among scientific papers, such as identifying corroborating or conflicting studies and tracing technological lineages. This fundamental limitation often results in fragmented knowledge structures, misinterpreted research sentiment, and ineffective modeling of collective scientific progress. To address this limitation, we introduce SciNet, the first Scientific Network relation-aware dataset for information retrieval agents. Built on a meta-database of 269 million papers across 7 disciplines and containing 8,940 carefully designed tasks, SciNet systematically captures three levels of relational understanding: ego-centric retrieval of papers with novel knowledge structures, pairwise identification of scholarly relationships, and path-wise reconstruction of scientific evolution. Extensive evaluation of three categories of retrieval agents shows that their accuracy on relation-aware tasks often falls below 20%, highlighting a fundamental shortcoming of current retrieval paradigms. Importantly, in a downstream literature review application, agents empowered with SciNet achieve a 25.3% improvement in review quality, highlighting the critical value of relation-aware retrieval for deepening scientific insights. We publicly release SciNet at https://github.com/tsinghua-fib-lab/SciNet to support future research.

Modern AI systems are increasingly used to help researchers search for scientific papers, summarize related work, and understand new research fields. However, most current systems mainly look for papers with similar words or similar topics. This is often not enough for real scientific research. Researchers also need to know how papers are connected: which papers introduced truly new ideas, which studies support or challenge each other, and how a research area has developed over time. In this paper, we introduce SciNet, a new benchmark designed to test whether AI systems can understand these relationships among scientific papers. SciNet is built from a very large collection of scientific publications and contains thousands of tasks across multiple scientific fields. These tasks ask AI systems to retrieve papers based on three kinds of relationships: the importance or novelty of individual papers, the relationship between pairs of papers, and the chain of papers that shows how an idea evolved. Our experiments show that today’s AI retrieval systems, including advanced search agents, still struggle with these relation-based tasks. They can often find papers that look topically relevant, but they frequently miss papers that are important because of their role in the broader scientific network. We also show that when AI systems are given better information about these paper relationships, they can produce more useful literature reviews. Overall, SciNet highlights an important gap in current AI research tools and provides a benchmark for building future systems that can support deeper and more reliable scientific discovery.