Draft-and-Audit Reinforcement Learning for Optimization Modeling

Natural language to optimization (NL2Opt) requires translating unstructured text into executable mathematical models. Beyond simple syntax errors, this task suffers from silent modeling failures, where incorrect formulations execute successfully but yield invalid results. We propose \textbf{Draft-and-Audit RL (DA-RL)}, a framework that learns optimization modeling as a two-step iterative workflow. Unlike inference-time scaffolds that rely on intermediate solver feedback to guide repairs, DA-RL optimizes a shared-parameter policy using terminal-only verification: the model is rewarded solely based on the execution of the final audited program. This constraint forces the model to internalize rubric-guided revision as a learned capability and encourages the emergence of cross-turn synergy, where the policy learns to generate drafts that are structurally amenable to self-correction.

Many real-world planning tasks, such as routing vehicles or scheduling jobs, must first be turned into an optimization model before a computer solver can compute a good decision. Today, people increasingly ask large language models to write these models from natural-language descriptions, but a dangerous failure mode is that the generated code may run successfully while still representing the problem incorrectly. We study how to make this process more reliable. Our method, Draft-and-Audit Reinforcement Learning, trains a model to work in two steps: first write a complete draft of the optimization program, then review it against a checklist for missing constraints, wrong numbers, and inappropriate variable types. Unlike systems that repeatedly run the solver and react to error messages, our model receives feedback only on whether the final audited program gives the correct answer. This encourages the model to learn better revision habits and to write drafts that are easier to check. Experiments show that this approach substantially improves correctness. The results suggest that teaching models to audit their own structured code can make AI-assisted decision modeling more dependable.