Investigating Component Contributions in Multi-Agent ML Systems

Autonomous agents for machine learning engineering have advanced rapidly, yet comparing their effectiveness remains difficult. Existing systems combine different techniques---multi-agent decomposition, iterative refinement, memory management, and planning---in varying configurations, making it unclear which components actually drive performance. Complicating evaluation, existing benchmarks rely on historical competitions whose data likely contaminates LLM training corpora and whose static baselines reflect outdated human performance. To address this, we conduct over 4,000 controlled experiments systematically ablating architectural components, alongside K-live, a new benchmark of 25 active competitions that provides a contamination-free, dynamic evaluation environment. Our findings challenge common design assumptions: iterative feedback contributes more than architectural complexity, and multi-agent coordination can hurt as often as it helps. These results provide concrete guidance for practitioners building ML engineering agents.

LLM-based agents are increasingly used to automate machine learning engineering—reading a dataset, writing training code, and producing competitive predictions. To improve them, systems layer on architectural components: multi-agent decomposition, structured memory, and explicit planning. But because each system bundles these differently, end-to-end comparisons reveal little about which components actually drive performance. We built a single modular agent in which each component can be independently toggled, then ran roughly 4,000 controlled ablations across 100 Kaggle competitions. To separate genuine problem-solving from memorization of contaminated training data, we also introduce K-LIVE, a benchmark of 25 active competitions whose data postdates current model knowledge cutoffs. The dominant factor by a wide margin was iterative refinement—execute, observe, revise—which mostly buys robustness against the buggy first drafts LLMs tend to write. Fixed-role multi-agent coordination consistently underperformed a single-agent baseline, with failures traced to coordination overhead, while memory and planning were near-neutral.