Constructing Industrial-Scale Optimization Modeling Benchmark

Optimization modeling underpins decision-making in logistics, manufacturing, energy, and finance, yet translating natural-language requirements into correct optimization formulations and solver-executable code remains labor-intensive. Although large language models (LLMs) have been explored for this task, evaluation is still dominated by toy-sized or synthetic benchmarks, masking the difficulty of industrial problems with $10^{3}$--$10^{6}$ (or more) variables and constraints. A key bottleneck is the lack of benchmarks that align natural-language specifications with reference formulations/solver code grounded in real optimization models. To fill in this gap, we introduce MIPLIB-NL, built via a structure-aware reverse construction methodology from real mixed-integer linear programs in MIPLIB~2017. Our pipeline (i) recovers compact, reusable model structure from flat solver formulations, (ii) reverse-generates natural-language specifications explicitly tied to this recovered structure under a unified model--data separation format, and (iii) performs iterative semantic validation through expert review and human--LLM interaction with independent reconstruction checks. This yields 223 one-to-one reconstructions that preserve the mathematical content of the original instances while enabling realistic natural-language-to-optimization evaluation. Experiments show substantial performance degradation on MIPLIB-NL for systems that perform strongly on existing benchmarks, exposing failure modes invisible at toy scale.

Many important decisions, such as how to route shipments, schedule factory work, or allocate limited resources, are made by turning real-world requirements into precise planning problems. To use optimization software, experts usually need to write both a mathematical model and computer code that a solver can run to find the best decision. Current AI systems are often tested only on small, simplified examples, so it is hard to know whether they can help with the larger and messier planning problems that organizations actually face. We introduce MIPLIB-NL, a new test set built from real industrial planning problems that have long been used to evaluate optimization software. For each problem, we create a natural-language description, the corresponding mathematical model, and runnable solver code. We also carefully check that these pieces describe the same problem and lead to the same solver behavior. Our experiments show that AI systems that look strong on existing simple tests perform much worse on MIPLIB-NL. This suggests that today’s systems still struggle with realistic scale, complex structure, careful use of data, and writing reliable solver code. MIPLIB-NL can help researchers build and evaluate more reliable AI tools for real-world decision-making.