{"id":"bf38f143-c558-4e28-b81d-fefe7d501cda","arxiv_id":"2608.07040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OptiDSL maps optimization problem descriptions to domain-specific language templates and reports large gains over MILP-based LLM formulation pipelines, with the caveat that its headline benchmark is self-generated.","lead":"OptiDSL uses large language models to turn natural-language descriptions of optimization problems into domain-specific data files, such as VRPLIB format, instead of mixed-integer linear programs. The authors report large gains in optimality rate and modeling time over MILP-based pipelines, but the main benchmark is generated from the same DSL templates the framework targets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 51.66% gain is not yet interpretable because the DSL-to-MILP conversion used for the unified Gurobi evaluation is unspecified, so the accuracy comparison may measure a hidden hand-written modeling layer rather than the LLM's DSL formulation.","rationale":"The reader correctly identifies the template-aligned benchmark as a major threat to external validity. My stress-test focuses on a more internal threat: the evaluation protocol is incomplete because OptiDSL's DSL output cannot be fed to Gurobi without an unspecified conversion step. The DSL templates encode the mathematical model implicitly (e.g., TYPE: CVRP implies an objective and constraints), so the LLM's task is largely data extraction and flag setting. The MILP baselines, by contrast, must produce the full model including constraints. Consequently, the headline 51.66% OR gain may reflect a shifted modeling burden rather than a difference in LLM formulation ability. This concern does not refute the qualitative value of the DSL-centric direction; the external LLMCoSolver results provide partial independent support. However, absent disclosure of the converter and a direct test with independently implemented conversion semantics, the quantified claim is not reproducible or fully interpretable. The conditional verdict already requests disclosure of the conversion pipeline and dataset, so my read does not change the verdict; it sharpens the reason why those disclosures are necessary.","tokens_in":22859,"tokens_out":7314,"duration_ms":70305,"concrete_test":"Freeze the LLM-generated DSL files for a random sample of 100 OptiDSLBench instances, and have an independent implementer who has not seen the paper's code write the converter from each published DSL specification (VRPLIB, OR-Library formats) to a Gurobi model using only the spec's semantics. If the independently implemented converter reproduces the reported OR values to within a few points on the same frozen DSL outputs, the hidden-converter concern is resolved; if not, the OR numbers depend on unpublished modeling choices and the central comparison is not yet established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To compute the headline Optimality Rate, OptiDSL's output must be executable by Gurobi. Yet the paper never specifies how DSL data files (VRPLIB, OR-Library, etc.) are converted to MILP; the Experiments section only states that Gurobi is used as a 'unified downstream solver.' This is a load-bearing gap. The selected DSLs are data formats, not modeling languages: for a CVRP instance, the file contains coordinates, demands, and capacity, while the objective (minimize route length), the decision variables, and the subtour-elimination and capacity constraints are not present—they are implied by the TYPE field and must be supplied by an external converter. For the MILP baselines, the LLM itself must generate all of those mathematical components. The reported 51.66% gain in OR therefore conflates the LLM's data-extraction ability with a per-domain converter that performs the actual mathematical modeling. If that converter is non-trivial (e.g., it decides between DFJ and MTZ subtour elimination), then OptiDSL's 'formulation accuracy' is not solely the LLM's accuracy. The paper must disclose this component and show that the comparison isolates the representation choice rather than shifting modeling work out of the LLM. The template-aligned benchmark described in 'Data Placeholder Substitution' further amplifies this: descriptions are generated from the same DSL templates OptiDSL fills, so the benchmark cannot separate template recovery from genuine formulation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OptiDSL, a framework that uses LLMs to translate natural-language optimization problem descriptions into domain-specific language (DSL) data files (e.g., VRPLIB, OR-Library formats) instead of MILP formulations. The framework includes a two-stage LLM workflow (semantic routing to a DSL template, then instantiation of the template), an adaptive solver-routing mechanism that selects among exact, heuristic, and learning-based solvers, and a new benchmark, OptiDSLBench, spanning 44 COP types across five domains. The authors report that OptiDSL outperforms MILP-based baselines (CoE, ORLM, LLMOPT) in execution rate (ER), optimality rate (OR), and modeling time on OptiDSLBench, and also reports gains on an existing COP benchmark and on MILP-oriented benchmarks.","tokens_in":23134,"tokens_out":2876,"duration_ms":26800,"significance":"If the empirical claims are correct, the paper makes a useful contribution by demonstrating that a DSL-centric intermediate representation can reduce LLM formulation burden and enable solver flexibility for combinatorial optimization, an alternative to the dominant text-to-MILP paradigm. The construction of a large, multi-domain benchmark (4,400 instances, 44 types) is a valuable asset, and the external-benchmark results provide some independent grounding. However, the headline quantitative claims are currently undermined by two load-bearing evaluation issues: the unspecified DSL-to-MILP conversion used for the unified Gurobi evaluation, and the template-aligned construction of OptiDSLBench. The core idea is defensible, but the evidence as presented does not yet support the strong '51.66% gain' and '91.71% decrease' statements.","major_comments":[{"comment":"The experiments report that Gurobi is used as the 'unified downstream solver across all evaluations,' but the paper never specifies how OptiDSL's DSL data files are converted into a Gurobi MILP model. The DSL formats are data formats (coordinates, demands, capacity), not modeling languages; the objective, decision variables, and constraints (e.g., subtour elimination for CVRP) are absent and must be supplied by an external converter. If that converter hard-codes the mathematical model per problem type, then the reported 51.66% OR gain conflates the LLM's data extraction with a pre-specified modeling layer, and the comparison against MILP baselines is not a comparison of formulation accuracy. The paper must disclose the conversion mechanism and either subtract its modeling content or treat it as part of the proposed pipeline and compare end-to-end fairly.","section":"Experiments, Experimental Setup"},{"comment":"OptiDSLBench is constructed by generating natural-language descriptions from the same DSL template pool and placeholder tags (e.g., <item_weight>, <demand>) that OptiDSL is explicitly instructed to fill. High ER and OR on this benchmark therefore partly measure template recovery rather than general optimization-modeling ability. The external LLMCoSolver benchmark covers only CVRP, JSSP, MIS, and MVC, which is too narrow to support the headline claim of superiority across 44 COP types. The paper should either evaluate on independently authored problem descriptions that do not share placeholder/template structure, or report a separate breakdown for template-aligned vs. non-template instances.","section":"COPs Benchmark Evaluation, Data Placeholder Substitution"},{"comment":"The abstract and the main text state that OptiDSL 'significantly surpasses' MILP pipelines, but no variance information or statistical significance tests are reported. All ER/OR values in Table 2 are point estimates over 100 instances without standard deviations, confidence intervals, or paired tests, and some individual rows show OptiDSL below a baseline (e.g., OVRPMBL: OptiDSL ER 0.78 vs. CoE 0.84). A paired test across instances or across problem types is needed to support the use of 'significantly'; otherwise the claim should be reworded to report observed mean differences.","section":"Overall Performance Comparison, Table 2"}],"minor_comments":[{"comment":"The main table shows only partial VRP results with an ellipsis and defers the remaining 24 types to the supplementary material; since the main text cites average VRP gains of 13.05% ER and 68.83% OR, the aggregate VRP row should be included in the main table to make the claim verifiable.","section":"Table 2"},{"comment":"The workflow diagram includes 'COPs Benchmark Evaluation' as a component of the framework, but the benchmark is an evaluation artifact rather than part of the online OptiDSL pipeline; the caption or diagram should clarify this distinction.","section":"Figure 2"},{"comment":"The text attributes the Dantzig-Fulkerson-Johnson formulation to 'Miller, Tucker, and Zemlin 1960' in the Introduction; the DFJ subtour elimination is due to Dantzig, Fulkerson, and Johnson (1954), while Miller-Tucker-Zemlin is a different (MTZ) formulation. The citation and description should be corrected.","section":"Introduction, Figure 1"},{"comment":"The OR values in Table 2 are reported as fractions (e.g., 0.93) while Table 4 reports percentages (e.g., 100.0 (6/6)); the units should be made consistent across tables for readability.","section":"Tables 2 and 4"},{"comment":"The reference for Blackstone et al. contains a typo: 'and, G. L. H.' should be corrected to the proper author formatting for the Handbook of dispatching rules citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable, but the empirical backbone is not yet solid. The unspecified DSL-to-MILP converter is the most serious issue because it directly affects the interpretability of the headline accuracy numbers; the template circularity is fixable with additional experiments or careful re-framing of the benchmark as a 'template recovery' test rather than general formulation ability. I recommend major revision rather than rejection because both issues are addressable within the scope of the manuscript: disclose the converter and re-evaluate on a non-template external corpus, or substantially temper the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. The core idea is genuinely worth engaging with: instead of forcing an LLM to emit a full MILP formulation, OptiDSL has it emit a domain-standard data file (VRPLIB for routing, OR-Library formats for scheduling, packing, knapsack, graph), then routes that file to a specialized solver. That is a real departure from the text-to-MILP pipelines in its own related-work survey, and it directly sidesteps the constraint explosion that sinks LLM-generated MILP for things like CVRP. Second, the headline numbers — 51.66% gain in optimality rate, 91.71% less modeling time — are not yet interpretable.\n\nThe load-bearing problem is the unified Gurobi evaluation. The DSL formats are data files, not modeling languages: a VRPLIB file says where the customers are and what they demand, but the objective, the variables, and the subtour/capacity constraints are implied by the TYPE field. To execute those files in Gurobi, someone must supply a per-domain converter that builds the actual mathematical model. The paper never describes that converter. If it is a hand-written MILP generator per problem type, the comparison is LLM-plus-converter against LLM-alone, and the 51.66% OR gain conflates the converter's modeling ability with the LLM's formulation skill. This must be disclosed before the headline claim means anything, and the stress-test note is right to push on it.\n\nThe benchmark circularity is real but somewhat less damning than it first appears. The generation prompt instructs the LLM to write descriptions using placeholders like <item_weight> that line up directly with DSL fields, so high ER/OR on OptiDSLBench partly measures template recovery. Still, the model must parse the scenario and infer business logic — OPEN_ROUTE from 'without returning to the warehouse' — and the gains persist, in diluted form, on the independently authored LLMCoSolver benchmark across four problem types. The qualitative direction is probably right; the magnitude is unsupported as stated.\n\nSecondary issues: no variance or significance tests on a stochastic pipeline, an underspecified filtered subset for the MILP-benchmark comparison, and a minor citation slip (DFJ attributed to Miller-Tucker-Zemlin in the intro). All fixable.\n\nThe DSL-centric direction deserves a serious referee; I'd send it out, expecting a revise-and-resubmit. The authors need to disclose the converter, validate on independently authored descriptions, and report variability. The framework is thoughtful and the solver pool is broad; this will influence the automated-modeling subfield regardless of how the numbers shake out.","headline":"The DSL-centric framing is a genuine contribution worth refereeing, but the 51.66% accuracy claim cannot be interpreted until the DSL-to-MILP conversion behind the Gurobi evaluation is disclosed and the template-aligned benchmark is supplemented with independent descriptions.","tokens_in":23701,"tokens_out":8004,"would_cite":true,"duration_ms":64722,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that automated optimization modeling should translate natural language into domain-specific data files rather than mixed-integer linear programs, and reports large gains in accuracy and speed from doing so.","keywords":["combinatorial optimization","domain-specific language","MILP","LLM-based modeling","automated formulation","solver selection","OptiDSLBench","vehicle routing"],"falsifier":"Take an independently written set of optimization problems from textbooks, industrial case studies, or community forums, with no overlap with OptiDSL's template pool, and run both OptiDSL and a MILP-based baseline on them; if the DSL pipeline's optimality-rate advantage disappears or reverses on these out-of-template descriptions, the central claim is not supported. A cheaper check is to inspect the existing external benchmark used in the paper: it covers only CVRP, JSSP, MIS, and MVC, so a failure on any one of these domains at scale would already pressure the generalization claim.","tokens_in":22641,"feed_emoji":"🧩","tokens_out":6727,"duration_ms":55747,"temperature":0.7,"pith_summary":"This paper is trying to establish that the standard way of automating optimization modeling—having a large language model translate a problem description into a mixed-integer linear program (MILP) and then solve it with a MILP solver—is the wrong default for many combinatorial optimization problems. The proposed alternative, OptiDSL, makes the intermediate representation a domain-specific data file, the kind of standardized format that a problem family's own solvers already read and write. The central empirical claim is that this DSL-centric pipeline beats MILP-based LLM pipelines on a new 44-problem-type benchmark by 51.66% in formulation accuracy (optimality rate) and 91.71% in modeling time, and by 23.09% on an existing benchmark. The intended significance is that the bottleneck in LLM-based optimization is not solver strength but the choice of modeling target: picking the right domain representation avoids constraint explosion and opens the pipeline to heuristics and learning-based solvers.","feed_headline":"DSL-first LLM modeling beats MILP pipelines by 51.66% accuracy","feed_subtitle":"Encoding problems as domain data files, not MILP, lifts optimality rates and cuts modeling time sharply.","key_machinery":"The load-bearing object is the DSL template pool: a library of standardized data-file formats (VRPLIB-style files for routing, OR-Library-style files for scheduling, packing, and knapsack, and network files for graph problems) that domain solvers already consume. A two-stage LLM workflow first performs semantic routing, matching the problem description to a template using concise meta-descriptions, then instantiates the template, filling fields and deducing flags like OPEN_ROUTE from the wording. An adaptive solver-execution layer profiles solvers offline and routes each DSL file to an exact, heuristic, or learning-based solver according to user preference.","core_discovery":"On the paper's own terms, the discovery is that text-to-MILP is a self-imposed restriction rather than a necessity. For problems such as capacitated vehicle routing, forcing a description into linear constraints produces an explosion of subtour-elimination constraints that overtaxes an LLM, while the same description maps cleanly onto a VRPLIB-style data file. OptiDSL operationalizes this by having an LLM first select a matching DSL template and then fill it with data and deduced logical flags, so formulation is decoupled from execution. The reported results—optimality rates above 80% across five problem domains, with the largest gains in vehicle routing—are taken as evidence that the DSL representation is the more faithful and tractable target for automated modeling.","pith_inferences":["An implication the authors leave implicit is that the measured advantage may be partly an output-economics effect: data files require far fewer output tokens than full mathematical models, so the same model solves a cheaper task; separating this from representational fidelity would require a token-matched comparison.","We infer the template-pool design should transfer to other structured optimization families (e.g., network design, production planning) as long as a domain-standard data format exists; the framework says little about problems without such a standard.","A testable extension is to feed OptiDSL problem descriptions whose data are not given as explicit bracketed lists—for example, textual mentions like 'each customer needs about a third of a truckload'—to see whether the DSL grounding step handles implicit quantities as well as explicit placeholders.","Because the benchmark descriptions are generated from the same template pool the system is instructed to use, the headline numbers likely measure template recovery; external, independently written problem sets would reveal how much of the gain transfers to genuinely novel phrasings."],"forward_implications":["If the central claim is correct, automated optimization pipelines should represent problems in the data format of the target domain rather than forcing them into a universal MILP schema.","The reported gains imply that a large part of LLM modeling error comes from the output representation itself, so the same reasoning model can be much more accurate when given a simpler, structured target.","Because DSL files are solver-native, LLM-based automation can exploit specialized solvers—exact MILP for small instances, heuristics and neural solvers for large ones—selecting per scale and time budget.","The new benchmark of 44 COP types with 4,400 instances gives the community a larger and more diverse testbed for automated modeling than the existing MILP-centric datasets.","Costs drop sharply: shaping output as a compact data file reduces modeling time and token consumption, which matters for deploying such systems in latency-sensitive settings."],"supporting_citations":[{"why":"Supplies the MTZ subtour-elimination formulation cited as the source of constraint explosion in MILP modeling of routing problems.","marker":"Miller, Tucker, and Zemlin 1960"},{"why":"Provides the VRPLIB-style DSL format and the PyVRP solver that OptiDSL uses for VRP instances.","marker":"Wouda, Lan, and Kool 2024"},{"why":"Chain-of-Experts is one of the three MILP-based LLM pipelines used as a baseline.","marker":"Xiao et al. 2024"},{"why":"ORLM is a baseline and represents the data-synthesis approach to text-to-MILP training.","marker":"Huang et al. 2025a"},{"why":"LLMOPT is a baseline and supplies the ER/OR evaluation metrics adopted throughout.","marker":"Jiang et al. 2025a"},{"why":"Provides the existing COP benchmark (CVRP, JSSP, MIS, MVC) used for the external comparison.","marker":"Jiang et al. 2025b"},{"why":"NL4Opt is the original text-to-MILP benchmark and is used in the MILP-benchmark comparison.","marker":"Ramamonjison et al. 2022"},{"why":"RouteFinder is a learning-based VRP solver that consumes DSL-format data, demonstrating the solver-flexibility advantage.","marker":"Berto et al. 2024"},{"why":"Used as the unified downstream solver so that all methods are compared on formulation quality alone.","marker":"Gurobi Optimization 2024"}],"fun_headline_variants":["MILP isn't the only way: DSL framework boosts formulation accuracy by 51.66%","Why forcing COPs into MILP hurts: OptiDSL offers a better path","DSL beats MILP for optimization modeling: 51.66% accuracy gain","Optimization modeling: drop MILP, use DSL for 51.66% accuracy boost","OptiDSL: LLM-driven DSL modeling outperforms MILP pipelines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the natural-language descriptions in the benchmark, generated from the same DSL templates and placeholder tags that OptiDSL is told to fill, are representative of real-world optimization problems, so the reported accuracy gains largely measure template recovery rather than generalization to genuinely new descriptions.","fun_headline_variants_meta":{"raw":{"variants":["MILP isn't the only way: DSL framework boosts formulation accuracy by 51.66%","Why forcing COPs into MILP hurts: OptiDSL offers a better path","DSL beats MILP for optimization modeling: 51.66% accuracy gain","Optimization modeling: drop MILP, use DSL for 51.66% accuracy boost","OptiDSL: LLM-driven DSL modeling outperforms MILP pipelines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001117,"raw_usage":{"total_tokens":4651,"prompt_tokens":944,"completion_tokens":3707,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":3595}},"tokens_in":560,"tokens_out":3707,"duration_ms":21331,"temperature":1.0,"reasoning_tokens":3595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:53:50.499112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an independently written set of optimization problems from textbooks, industrial case studies, or community forums, with no overlap with OptiDSL's template pool, and run both OptiDSL and a MILP-based baseline on them; if the DSL pipeline's optimality-rate advantage disappears or reverses on these out-of-template descriptions, the central claim is not supported. A cheaper check is to inspect the existing external benchmark used in the paper: it covers only CVRP, JSSP, MIS, and MVC, so a failure on any one of these domains at scale would already pressure the generalization claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MTZ subtour-elimination formulation cited as the source of constraint explosion in MILP modeling of routing problems."},{"cited_title":"ICML 2024 Workshop on Foundation Models in the Wild , year=","cited_arxiv_id":null,"evidence_quote":"RouteFinder is a learning-based VRP solver that consumes DSL-format data, demonstrating the solver-flexibility advantage."},{"cited_title":"2024 , url =","cited_arxiv_id":null,"evidence_quote":"Used as the unified downstream solver so that all methods are compared on formulation quality alone."}],"review_version":1}