Towards Execution-Grounded Automated AI Research

Automated AI research holds great potential to accelerate scientific discovery. However, current LLMs often generate plausible-looking but ineffective ideas. Execution grounding may help, but it is unclear whether automated execution is feasible and whether LLMs can learn from the execution feedback. To investigate these, we first build an automated executor to implement ideas and launch large-scale parallel GPU experiments to verify their effectiveness. We then convert two realistic research problems -- LLM pre-training and post-training -- into execution environments and demonstrate that our automated executor can implement a large fraction of the ideas sampled from frontier LLMs. We analyze two methods to learn from the execution feedback: evolutionary search and reinforcement learning. Execution-guided evolutionary search is sample-efficient: it finds a method that significantly outperforms the GRPO baseline on post-training, and finds a pre-training recipe that outperforms the nanoGPT baseline on pre-training, all within just ten search epochs. Frontier LLMs often generate meaningful algorithmic ideas during search, but they tend to saturate early and only occasionally exhibit scaling trends. Reinforcement learning from execution reward, on the other hand, suffers from mode collapse. It successfully improves the average reward of the ideator model but not the upper-bound, due to models converging on simple ideas. We thoroughly analyze the executed ideas and training dynamics to facilitate future efforts.

A core weakness of automated AI research is that LLMs tend to generate ideas that sound plausible but fail in practice. Here, we ask whether closing the loop — having the AI execute its own ideas as real experiments and learn from the results — can address this. We build a system that automatically implements AI-generated ideas, runs GPU experiments, and measures whether performance actually improves, testing this on two realistic problems: LLM pre-training and post-training. Our evolutionary search approach proves efficient: within ten rounds of search, we identify training methods that substantially outperform standard baselines. However, reinforcement learning struggles with mode collapse — models converge on safe, simple ideas, improving average quality but not peak performance. We conclude that execution-grounded feedback is both feasible and beneficial for automated research, though consistently generating genuinely novel ideas remains an open challenge.