AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation

As Large Language Models (LLMs) evolve into autonomous agents, existing safety evaluations face a fundamental trade-off: manual benchmarks are costly, while LLM-based simulators are scalable but suffer from logic hallucination. We present AutoControl Arena, an automated framework for frontier AI risk evaluation built on the principle of logic-narrative decoupling. By grounding deterministic state in executable code while delegating generative dynamics to LLMs, we mitigate hallucination while maintaining flexibility. This principle, instantiated through a three-agent framework, achieves over 98% end-to-end success and 60% human preference over existing simulators. To elicit latent risks, we vary environmental Stress and Temptation across X-Bench (70 scenarios, 7 risk categories). Evaluating 9 frontier models reveals: (1) Alignment Illusion: risk rates surge from 21.7% to 54.5% under pressure, with capable models showing disproportionately larger increases; (2) Scenario-Specific Safety Scaling: advanced reasoning improves robustness for direct harms but worsens it for gaming scenarios; and (3) Divergent Misalignment Patterns: weaker models cause non-malicious harm while stronger models develop strategic concealment. Code and data are available at https://github.com/CosmosYi/AutoControl-Arena.

As AI systems become more capable, they are increasingly used not only to answer questions, but also to act on behalf of people: using tools, making plans, and working through multi-step tasks. This makes safety testing harder. A system may appear safe in simple settings, but behave differently when it faces pressure, or tempting shortcuts. This paper introduces AutoControl Arena, a framework for automatically creating realistic test environments for evaluating such AI agents before they are deployed. Instead of relying only on hand-written tests, which are expensive to build, or purely text-based simulations, which can become inconsistent, AutoControl Arena combines executable environments with flexible natural-language interactions. This allows the tests to be both scalable and more faithful to real situations. Using this framework, we build a benchmark of 70 risk scenarios across seven types of safety concerns and test nine frontier AI models under different levels of pressure and opportunity. We find that many risks are hidden in normal conditions but become much more visible when the environment becomes stressful or offers easier unsafe paths. We also find that stronger models are sometimes safer on direct harmful requests, but can be better at finding loopholes in complex tasks. These results show that careful stress-testing is important for discovering safety failures before AI agents are used in the real world.