General Modular Harness for LLM Agents in Multi-Turn Gaming Environments

We introduce a modular harness design for LLM agents that composes of perception, memory, and reasoning components, enabling a single LLM/VLM backbone to tackle a wide spectrum of multi-turn gaming environments without domain-specific engineering. Using classic and modern game suites as low-barrier, high-diversity testbeds, our framework provides a unified workflow for analyzing how each module affects performance across dynamic interactive settings. Extensive experiments demonstrate that the harness lifts gameplay performance consistently over un-harnessed baselines and reveals distinct contribution patterns—for example, memory dominates in long-horizon puzzles while perception is critical in vision-noisy arcades. These findings highlight the effectiveness of our modular harness design in advancing general-purpose agent, given the familiarity and ubiquity of games in everyday human experience.