FormulaCode: Evaluating Agentic Optimization on Large Codebases

Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and 264.6 community-maintained performance workloads per task, enabling evaluation of the full optimization lifecycle—triage, diagnosis, and resolution—under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents.

This project introduces FormulaCode, a continually updating large-scale benchmark for evaluating the holistic ability of LLM agents to optimize codebases. Each task in FormulaCode corresponds to a documented performance issue or optimization opportunity with associated tests and constraints, and exposes realistic trade-offs among runtime, memory use, accuracy, and other software concerns. It has multiple features that are useful for practitioners (teams using LLMs / Agents) and researchers (teams developing new LLMs / Agents). For practitioners, FormulaCode is a practical way to compare optimization workflows under realistic constraints. It helps you understand which agent + model scaffolds reliably produce speedups on large repos, whether an agent + model scaffold works better on holistic large-scale changes or focused small-scale optimizations, what agent + model scaffold offers the best cost-optimization trade-off, and how well they negotiate performance tradeoffs along the way (risk of regressions, reliance on profiling tools, aggressiveness of refactors, etc.). For researchers, FormulaCode provides a controlled setting to study agentic performance engineering at repo scale. You can evaluate generalization across diverse repositories (including bespoke scientific repositories never used in any coding benchmark), compare behavior against strong human-written reference solutions, and analyze optimization strategies and failure modes -- e.g., which tools an agent uses, how it prioritizes varying hypotheses, and how those choices correlate with final speedups and correctness.