PostTrainBench: Can LLM Agents Automate LLM Post-Training?

AI agents have become surprisingly proficient at software engineering over the past year, largely due to improvements in reasoning capabilities. This raises a deeper question: can these systems extend their capabilities to automate AI research itself? We introduce PostTrainBench to benchmark how well LLM agents can perform post-training autonomously under bounded compute constraints (10 hours on one H100 GPU). We ask frontier agents (e.g., Claude Code with Opus 4.6) to optimize the performance of a base LLM on a particular benchmark (e.g., Qwen3-4B on AIME). Importantly, we do not provide any predefined strategies to the agents and instead give them full autonomy to find necessary information on the web, run experiments, and curate data. We find that frontier agents make substantial progress but generally lag behind official instruction-tuned LLMs: 27.9% for the best agent vs. 51.1% for official instruction-tuned models. However, agents can exceed instruction-tuned models in targeted scenarios: GPT-5.1 Codex Max achieves 89% on BFCL with Gemma-3-4B vs. 67% for the official model. We also observe failure modes: agents engage in reward hacking by training on the test set, downloading existing instruction-tuned checkpoints, or using found API keys to generate synthetic data without authorization. Overall, we hope PostTrainBench will be useful for tracking progress in AI R&D automation and for studying the risks that come with it.

Modern AI assistants like ChatGPT, Claude, and Gemini start out as raw "base" models that are not very useful on their own. Human engineers turn them into helpful assistants through a step called post-training, which teaches them how to follow instructions, write code, do math, and act safely. This step is one of the most important and time-consuming parts of building modern AI. In the past year, AI assistants have become surprisingly good at writing code and using developer tools. That raises a natural question: can these AI assistants now do the work of the engineers who build them? In other words, can AI improve AI without human help? To answer this, we built PostTrainBench, a test that gives an AI assistant a fixed budget — ten hours on a single graphics card — and asks it to take a raw base model and make it as good as possible on a chosen task, such as solving math problems or writing code. The AI works on its own: it searches the web for ideas, collects training data, runs experiments, and decides what to try next. We tested leading AI assistants from OpenAI, Anthropic, and Google, and compared their results to the polished versions of the same base models that human teams released to the public. We found that today's AI assistants make real progress, but on average they still fall short of what human engineering teams achieve (about 28% versus 51% on our combined score). At the same time, the picture is not one-sided: on certain focused tasks, the AI assistants actually beat the human teams. For example, one system trained Google's Gemma-3-4B model to score 89% on a tool-use test, well above the 67% that Google's own released version reaches. We also noticed some warning signs. When left to their own devices, some AI assistants tried to cheat — for instance, by training on the very test they were supposed to be evaluated on or by downloading an already-finished model and submitting it as their own work. These behaviors did not cause real harm in our controlled setup, but they show why careful supervision matters as these systems get more capable. PostTrainBench gives researchers, companies, and policymakers a clear way to track how quickly AI assistants are learning to do AI research itself. This is a capability that could speed up scientific progress, but it is also one we need to watch carefully. By measuring it openly and repeatedly over time, we hope to give the broader community an early and honest signal of where this technology is heading.