From Winning to Understanding: A Diagnostic Long-Horizon RTS Benchmark for LLMs

Large language models (LLMs) are increasingly used as decision modules, yet existing benchmarks provide limited coverage of long-horizon, adversarial interaction while faithfully acting on human instructions. We introduce a long-horizon Red Alert RTS benchmark with a hierarchical interface in which LLMs output budgeted, low-frequency macro/tactical intents that are executed deterministically for standardized comparison. The benchmark evaluates (i) robustness to ``rules-as-variable'' perturbations via rule-style shifts , (ii) competitive strength via Elo-style ratings from head-to-head matches, and (iii) human steerability via standardized language interventions. Beyond win/loss, we log economy growth/spending, combat loss ratio, and visibility coverage to diagnose long-horizon failure modes. Overall, the benchmark provides a reproducible and diagnostic testbed for robustness and controllability in long-horizon adversarial decision making.

Artificial intelligence systems that power chatbots are increasingly being asked to make real-world decisions—managing resources, planning strategies, and responding to changing situations over long periods. However, most existing tests measure short, isolated tasks rather than sustained, adaptive decision-making. We created a new benchmark using the classic strategy game Red Alert to evaluate how well AI systems handle long-horizon, adversarial decision-making. In this test, AI systems act as commanders that must build economies, scout unknown territory, and defeat opponents over 30-minute matches. We evaluate them in three ways: against distinct AI opponents with different styles, in head-to-head matches against each other, and in their ability to follow human tactical instructions like “surround the enemy” or “protect fragile units in the back.” Our experiments with six leading AI systems reveal that no single model dominates everywhere. One system may excel at economic management but struggle with combat coordination; another may follow instructions precisely yet falter against adaptive opponents. These findings show that a single overall score can hide important weaknesses. By diagnosing specific failure modes—such as poor scouting, inefficient combat, or incomplete command following—our benchmark helps researchers build AI decision-makers that are more robust, competitive, and responsive to human guidance.