SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce \textsc{SimulCost}, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. \textsc{SimulCost} compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,947 single-round (initial guess) and 1,931 multi-round (adjustment by trial-and-error) tasks across 13 simulators from fluid dynamics, solid mechanics, and plasma physics. Each simulator's cost is analytically defined and platform-independent. Frontier LLMs achieve 46--65\% success rates in single-round mode, dropping to 35--55\% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 72--81\%, but LLMs are 1.5--2.5$\times$ slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source \textsc{SimulCost} as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at \url{https://github.com/Rose-STL-Lab/SimulCost-Bench}.

Scientists use computer simulations to study complex physical systems like plasma behavior or fluid flow, but running these simulations is expensive and time-consuming. AI assistants could potentially help by suggesting good starting configurations, reducing the trial-and-error typically needed. But do they actually save time and resources compared to traditional approaches? We created SimulCost, a benchmark to test whether AI assistants can effectively tune simulation settings while being mindful of computational costs. We tested leading AI systems on nearly 5,000 tasks across 13 physics simulators covering fluid dynamics, solid mechanics, and plasma physics. Our findings reveal a gap between promise and practice. AI assistants struggle with accurate initial guesses (succeeding 46-65% of the time), dropping to just 35-55% when high accuracy is required. While they improve when given feedback (72-81% success), they actually consume 1.5-2.5 times more computational resources than traditional parameter scanning methods. We also examine how related parameters can inform each other (for potential knowledge transfer) and how providing examples or encouraging more reasoning affects performance, offering practical guidance for deployment and improvement. This suggests current AI tools are not yet cost-effective for scientific parameter tuning. We release SimulCost as an open benchmark to help researchers develop more resource-efficient AI assistants for scientific computing.