EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

The rapid evolution of Large Language Models (LLMs) has shifted their role from passive information providers to active agents capable of executing complex workflows. However, the realization of a true "AI worker" is currently hindered by benchmarks that fail to capture the intricacy of professional environments, which demand long-horizon planning, complex tool usage, and adherence to strict access protocols. To bridge this gap, we introduce EnterpriseOps-Gym, a benchmark environment designed to evaluate agentic planning in realistic enterprise settings. EnterpriseOps-Gym provides: (i) 1,150 expert-curated tasks across eight interconnected domains (including HR, IT, Customer Service and productivity tools) that require managing persistent state and adhering to strict outcome-based verification logic; and (ii) a high-fidelity, containerized sandbox environment hosting 164 database tables and 512 functional tools. Our evaluation reveals critical limitations in state-of-the-art models: even the top-performing Claude Sonnet~4.5 achieves only 34.1\% success, struggling significantly with planning consistency, error recovery, and policy constraints. Furthermore, we observe that agents frequently fail to refuse infeasible tasks, leading to unintended and potentially harmful side effects on the system. These findings indicate that current agents are not yet ready for enterprise deployment. By releasing EnterpriseOps-Gym, we provide a concrete testbed to advance the reliability of autonomous agents in professional workflows.

AI assistants are increasingly being asked to do more than answer questions; they are being deployed as digital workers that take actions on behalf of people, such as updating records, managing schedules, and resolving customer issues. But before we trust AI with these responsibilities in real workplaces, we need rigorous ways to test whether they are actually up to the task. We built ENTERPRISEOPS-GYM, a realistic testing environment that mimics the Enterprise Knowledge Worker paradigm—think IT helpdesks, customer support systems, and HR departments as well as general enterprise tools like Teams, Drive, Email and Calendar management. Our test contains 1,150 realistic workplace tasks authored by human experts with domain knowledge, ranging from resolving a technical support ticket to onboarding a new employee. Specifically, we isolated our measurement to the core planning and task completion stage by assuming an oracle knowledge retriever and tool retriever. However, even under this idealized setting, the results were striking: even the best models completed roughly only 4 in 10 tasks. AI systems struggled most when tasks required following workplace process rules and planning long sequences of actions where each step depends on the one. Critically, when asked to handle requests that were impossible or policy-violating, most AI systems failed to simply say "no" — often attempting the task anyway and inadvertently corrupting data in the process. One of our most important findings is that the core challenge ahead is not just intelligence, but constraint aware planning. When we gave AI systems the same step-by-step plans a human expert would use, performance nearly doubled. These results motivate future research priorities: constraint-aware planning to methods that explicitly reason over policy constraints, side-effect dependencies, and prerequisite structures before committing to action sequences (think world models), long horizon memory management, and safe abstention.