GameDevBench: Evaluating Agentic Capabilities Through Game Development

Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex---the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only $53.8\%$ of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from $51.4\%$ on gameplay-oriented tasks to $33.0\%$ on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from $41.1\%$ to $52.0\%$ when given visual feedback.

GameDevBench is the first benchmark for game development that focuses on LLM agents. Tasks are created from YouTube tutorials and are relatively simple by design. Despite their simplicity, agents still struggle with game development. We learned that this primarily stems from agents' poor visual understanding.