DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents

Progress in neural combinatorial optimization for Dynamic Flexible Job Shop Scheduling Problem (DFJSP) is currently hindered by a methodological tension: static benchmarks encourage benchmark overfitting, while uncalibrated generators obscure algorithmic capability with stochastic noise. To resolve this, we introduce \textbf{DynaSchedBench}, a diagnostic framework for DFJSP that rigorously controls the instance-generation process. Instead of relying on parameter sampling, our approach utilizes Sequential Event-Space Calibrator (SESC) that computes a novel Schedule Stress Index (SSI) to stratify instances by difficulty. We demonstrate that SESC is substantially more computationally efficient than evolutionary baselines while converging reliably to the target metrics. The framework integrates modular components for instance generation, snapshot-based simulation, agents, evaluation, and visualization, thereby enabling rigorous testing of reactive and lookahead-based policies. Leveraging this calibrated environment, we identify key limitations of LLM-based scheduling agents. Specifically, in step-wise online decision-making for dynamic scheduling, we identify an ``Observability Paradox'': providing agents with oracle access to full structural information can degrade policy performance, underperforming concise information. Furthermore, despite substantial token overhead, tool-augmented and refinement strategies fail to reliably improve performance, and most LLM agents fail to consistently surpass strong dispatching baselines—behaving more like robust heuristic approximators than superior optimizers.

Modern factories must decide, minute by minute, which job should run on which machine, even when new orders arrive, machines break, or processing times change. Researchers want to test whether AI systems, including large language models, can make these decisions, but existing tests are often too fixed or too random, making it hard to tell whether a method is genuinely better or just lucky. We introduce DynaSchedBench, a testing framework that creates factory scheduling problems with controlled levels of difficulty and disruption. It adjusts the actual stream of events, such as job arrivals, due dates, processing times, and machine outages, and scores how stressful each problem is. This lets researchers compare scheduling methods under fairer and more repeatable conditions. Using this framework, we find that giving language models more detailed factory information does not always help: concise summaries can lead to better decisions than richer structural descriptions. We also find that tool use and repeated self-refinement often cost many more tokens without reliably improving schedules. These results suggest that today’s language models are useful scheduling assistants, but they still behave more like safe heuristic followers than truly superior planners.