AgentExpt: Automating AI Experiment Design with LLM-based Resource Retrieval Agent

In modern AI research, baseline and dataset selection is a high-stakes decision in experimental design. It operationalizes a research idea into a concrete evaluation protocol and largely determines the validity and comparability of empirical conclusions. However, making appropriate choices is increasingly difficult as baselines and datasets proliferate, while suitability is inherently context-dependent and rarely captured by baseline and dataset metadata. To address these challenges, we present \textbf{AgentExpt}, a comprehensive framework for baseline and dataset recommendation. We first curate a large-scale, high-quality knowledge base that links 108{,}825 accepted papers to their used baselines and datasets. Based on this resource, we design a \textit{collective perception-enhanced retriever} that represents each baseline or dataset by integrating first-person self-descriptions with third-person citation contexts, thereby effectively positioning them within the scholarly network. We further design a \textit{reasoning-augmented reranker} that encodes baseline-dataset interaction chains as a reasoning prior to fine-tune an LLM, producing refined rankings with interpretable justifications. Experiments show that our framework outperforms the strongest baseline, with average gains of +5.85\% in Recall@20 and +7.90\% in HitRate@10, and ablation studies confirm the effectiveness of our designed components. Overall, AgentExpt advances the efficient and reliable automation of experimental design. Our code is available at \url{https://anonymous.4open.science/r/Agentexpt-DD3E}.

When artificial intelligence researchers invent a new method, they must rigorously test it by comparing it against existing tools (baselines) using standard information (datasets). However, choosing the right combination is incredibly difficult. There are thousands of options, and simply searching their basic descriptions often fails to reveal which ones are truly appropriate for a specific test. To solve this, we created AgentExpt. We built an AI assistant trained on over 100,000 published AI papers to learn how scientists actually conduct experiments in the real world. Instead of just matching keywords, our tool looks at "collective perception"—how the scientific community actually talks about and uses these datasets and baselines in practice. It then connects the dots between past papers to recommend the perfect experimental setup for a new research idea, even explaining the reasoning behind its choices. This tool automates a complex, time-consuming part of scientific discovery. By recommending reliable experimental designs, it helps researchers test their ideas more rigorously, ensures fairer comparisons across the field, and makes it easier for newcomers to navigate the fast-paced world of AI research.