OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling

We investigate the capabilities and scalability of Large Language Models (LLMs) in optimization modeling, a domain requiring structured reasoning and precise formulation. To this end, we introduce OPT-ENGINE, an extensible benchmark framework with quantifiable and controllable complexity. OPT-ENGINE spans ten canonical Operations Research problems, systematically scaling from Linear Programming to Mixed-Integer Programming, providing a structured environment to probe the limits of automated problem formulation and solving. Utilizing OPT-Engine, we address three pivotal research questions. First, we examine whether Pure-Text Reasoning (PTR) via classical Chain-of-Thought can efficiently tackle optimization tasks, finding that PTR suffers from a critical robustness gap as task complexity increases. Second, we examine whether integrating external computational tools can mitigate PTR's arithmetic weaknesses and improve performance. Our results show that while such tools aid local calculations, they still fail to adhere to global optimization constraints. Finally, we pinpoint that for the current SOTA paradigm, Solver-integrated Reasoning (SIR), the automated formulation of constraints represents the primary bottleneck. These findings clarify current paradigms' limitations and provide a structured roadmap for developing next-generation LLMs for optimization modeling. We release our code and data to facilitate future research (https://github.com/Cardinal-Operations/OPTEngine).

Large Language Models (LLMs) like those powering chatbots are increasingly being asked to solve complex planning and decision-making problems — for example, scheduling factory jobs, managing investment portfolios, or designing supply chains. In this paper, we ask: how good are LLMs at this kind of structured, optimization-heavy reasoning? We build a testing framework called OPT-ENGINE that lets us systematically measure LLM performance on classic optimization problems, from simple ones to much harder versions involving discrete choices and constraints. We test three approaches. The first asks the LLM to reason purely in text, step by step. The second lets the LLM use external calculators or solvers as tools. The third, called Solver-integrated Reasoning (SIR), lets the LLM work more closely with an optimization solver. Our key findings are: (1) Pure text reasoning becomes unreliable as problems get more complex. (2) Adding external tools helps with arithmetic but often misses the bigger picture of global constraints. (3) Even the best current method, SIR, still struggles most with correctly translating real-world requirements into precise mathematical constraints. These results highlight where current LLMs fall short in optimization tasks and point to clear directions for making them more reliable for real-world decision-making. We publicly release our benchmark and code to support further work in this area.