DSGym: A Standardized and Holistic Framework for Evaluating and Training Data Science Agents

Data science agents promise to accelerate discovery and insight-generation by turning data into executable analyses and findings. Yet existing data science benchmarks fall short due to fragmented evaluation interfaces that make cross-benchmark comparison difficult, narrow task coverage and a lack of rigorous data grounding. In particular, we show that a substantial portion of tasks in current benchmarks can be solved without using the actual data. To address these limitations, we introduce DSGym, a standardized framework for evaluating and training data science agents in self-contained execution environments. Unlike static benchmarks, DSGym provides a modular architecture that makes it easy to add tasks, agent scaffolds, and tools, positioning it as a live, extensible testbed. We curate DSGym-Tasks, a holistic task suite that standardizes and refines existing benchmarks via quality and shortcut solvability filtering. We further expand coverage with (1) DSBio: expert-derived bioinformatics tasks grounded in literature and (2) DSPredict: challenging prediction tasks spanning domains such as computer vision, molecular prediction, and single-cell perturbation. Beyond evaluation, DSGym enables agent training via execution-verified data synthesis pipeline. As a case study, we build a 2,000-example training set in DSGym that substantially improves a 4B mode on standardized analysis benchmarks. Overall, DSGym enables rigorous end-to-end measurement of whether agents can plan, implement, and validate data analyses in realistic scientific context.

AI systems are increasingly being asked to act like data scientists: inspect datasets, write code, run analyses, and turn results into answers. But it is hard to tell whether these systems are genuinely using the data, because existing tests use different formats and some questions can be answered from memory or common patterns without opening the files. We introduce DSGym, a standardized testing environment where data-science agents must work inside controlled computing environments, interact with real datasets, and submit answers that can be checked automatically. DSGym also audits existing tasks and removes many “shortcut” questions that models can solve without data access. To broaden what is tested, we add tasks from expert-designed biomedical analyses and recent prediction challenges, where agents must understand scientific context and build working modeling pipelines. Our experiments show that even strong AI models still struggle with scientific concepts, specialized data formats, and careful verification, and often stop after producing a runnable but weak solution. We also show that the same environment can generate execution-checked training examples, improving a small open model on data-analysis tasks. Overall, DSGym helps researchers evaluate and improve AI data scientists in a more realistic, reproducible, and data-grounded way.