REPA: : Reproducibility Evaluation via an Autonomous Pipeline Architecture

We present REPA, a framework to autonomously reproduce text classification experiments from paper descriptions, without access to reference code or repositories. Unlike prior AI~Scientist systems, REPA targets the reproduction problem directly by deconstructing it as a four-stage process incorporating protocol extraction, input preparation, experiment generation, and evaluation. On a favorable set of ten well-documented text classification papers, REPA replicates eight studies when instantiated with a GPT-4o backend, and five with Qwen3-Coder-30B. Compared to a replication rate of zero via direct prompting, our results with REPA establish a performance ceiling for current LLM automation as of mid-2026 and the importance of template scaffolding in scientific reproduction success.