{"id":"33dc9d36-3c5d-4ab2-965e-c323fdfdb889","arxiv_id":"2608.02641","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A one-call, intermediate-representation-first pipeline achieves competitive optimization formulation accuracy at a fraction of the inference cost of iterative multi-agent systems.","lead":"This paper introduces IR2Solve, a pipeline that uses one call to a large language model to turn a word problem into a structured model description, then compiles it with fixed code into a solver-ready optimization model. It reports accuracy close to much more expensive multi-call systems while spending far fewer tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The deterministic verifier is not conservative: Rules 6/7/3 (Appendix D) change model semantics from ungrounded cue words, and their +11.1 pp gain is measured only on the datasets used to develop them; the general one-call claim depends on this gain transferring.","rationale":"The reader's weakest assumption—that the verifier's hand-written rewrite rules are conservative and portable—is exactly the load-bearing concern for the general claim. The paper states the relevant limitations honestly (ungrounded cues, benchmark heuristic, net-effect measurement), but that does not remove the risk. The decisive observation is that the verifier is not merely canonicalizing syntax: Rule 6 changes variable domains, Rule 7 changes constraint directions, and Rule 3 inserts numerical values. Each of these can alter the feasible set relative to the reference problem. Calling the stage 'deterministic verification' is accurate about execution, but not about semantic preservation. The ablation's R2→R3 contrast is the only evidence for the verifier's value, and its dataset-dependence (0.0 pp on IndustryOR, +15.3 pp on ComplexLP) is exactly the pattern expected from cue-based heuristics tuned to one benchmark's failure modes. Because the final Table 2 configuration includes the verifier on all six benchmarks, the headline accuracy numbers inherit this fragility. I agree with the reader that this warrants a CONDITIONAL verdict: the internal ablation is well-controlled and transparent, and the one-call architecture is a legitimate contribution, but the portability of the verifier gain should be demonstrated before the accuracy-cost operating point is treated as general. The weak R0 baseline is a secondary concern: it inflates the R0→R1 bundle effect, but the main six-benchmark comparison and the CAFA/SAC-Opt cost panel stand independently, so it does not change the verdict. A held-out R2-vs-R3 test on the four non-ablation benchmarks is the most direct way to settle the concern.","tokens_in":17980,"tokens_out":5956,"duration_ms":57754,"concrete_test":"Run the exact R2 (no-verifier) and R3 (with-verifier) prompts, with the same temperature-0 one-call protocol, on the 1,340-instance four-dataset refresh (NL4Opt, EasyLP, NLP4LP, ReSocratic) that the final configuration already processes, and compute the paired McNemar contrast on objective correctness. If R3 is not significantly better than R2, or is worse, on these held-out benchmarks, the +11.1 pp verifier gain in Table 3 is dataset-specific and the one-call generalization claim must be re-scoped. As a secondary check, log every Rule 6/7/3 rewrite and manually adjudicate a random 50-instance sample against reference models to measure the semantic-error rate of the rewrites.","verdict_should_be":"UNCHANGED","load_bearing_attack":"IR2Solve's central claim—a practical one-call accuracy–cost operating point—depends on the deterministic verifier adding reliable accuracy without a second semantic call. That condition is least secure in Appendix D. Rule 6 (integrality sanity) rewrites continuous variables to binary or integer when cue words (assignment, select, choose, take, open, close, build, install, use, activate, visit, route, serve, facility, worker, task, item, node, edge, arc; set cues city, job) appear in LLM-generated names or descriptions. Rule 7 rewrites equality constraints to >= or <= from at-least/at-most cues, and is a no-op if both cue families occur. Rule 3 fills missing square-matrix diagonals with 0.0. These are not conservative syntax repairs; they restrict or otherwise change the feasible set of the model. The paper itself concedes (Appendix D) that rules 6–7 inspect metadata generated in the same semantic call, not independently grounded source spans, and that Rule 3's zero-diagonal assumption is a benchmark heuristic. Determinism (Eq. 4) preserves whatever the rules do; it does not make them sound. The empirical support is a single paired contrast on 153 instances (Table 3): R2→R3 is +11.1 pp pooled, but +15.3 pp on ComplexLP and 0.0 pp on IndustryOR. With no development/holdout split, one cannot distinguish a general safety improvement from a cue list tuned to ComplexLP's integer-heavy failure modes. Since the final Table 2 numbers for ComplexLP and the other four benchmarks include this verifier, the 'competitive without iterative correction' claim would be materially weakened if the cue lists and thresholds do not transfer to other problem distributions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IR2Solve, a one-call LLM autoformulation pipeline in which a single semantic LLM call produces a schema-constrained JSON intermediate representation (ModelIR), followed by two fully deterministic stages: a verification stage that applies fixed guarded rewrites and a compiler that builds a Gurobi model. The evaluation reports objective-correct accuracy on six survey-cleaned benchmarks, a nested R0–R3 ablation on 153 IndustryOR/ComplexLP instances, and a matched COST-10 panel measuring semantic calls and token volume. The central claim is that an explicit IR with deterministic verification and compilation reaches competitive objective accuracy while using one semantic call per instance, in contrast to iterative multi-call systems such as Chain-of-Experts and SAC-Opt. The paper is transparent about the scoped nature of several comparisons and lists limitations in Section 6.","tokens_in":18330,"tokens_out":5916,"duration_ms":58647,"significance":"If the reported results transfer beyond the evaluated datasets, the paper makes a useful contribution to LLM-based autoformulation: it demonstrates an explicit, inspectable representation layer with a deterministic post-processing tail as an accuracy–cost operating point between brittle direct code generation and expensive iterative workflows. The nested ablation design, paired McNemar tests with bootstrap intervals, prompt hashes, and archived metadata are genuine strengths, and the paper is careful to distinguish bundle-level effects from isolated component effects. The main caveats are external validity of the deterministic verifier, the strength of the R0 code-first baseline, and protocol heterogeneity in the six-benchmark comparison; these make the central claim defensible but not yet fully established.","major_comments":[{"comment":"The R2→R3 contrast measures the verifier only on the 153 IndustryOR/ComplexLP instances, while the verifier's active rules include semantic rewrites driven by hand-set cue lists (Rules 4, 6, 7) and a zero-diagonal heuristic (Rule 3). The appendix states that Rules 6–7 inspect metadata generated in the same semantic call rather than independently grounded source spans, and no development/validation split is reported for the rule parameters (free-index symbol set, leader margin of 2, set-size cap, cue lists, zero-diagonal assumption). Because the verifier gain is +15.3 pp on ComplexLP and 0.0 pp on IndustryOR, the pooled +11.1 pp gain may be in-sample rule tuning rather than a portable safety improvement. This is load-bearing for the one-call claim, and the paper should provide a held-out validation split or source-grounded rule design before the verifier improvement is presented as general.","section":"Appendix D and Table 3"},{"comment":"The R0 condition is described as a controlled one-call interface comparison, but its prompt is explicitly not optimized and is the same ModelIR-oriented template with the IR field set to null. The R0→R1 gain of +28.8 pooled points therefore does not cleanly isolate the value of the structured IR interface; it conflates representation change with prompt quality. Since the conclusion attributes the largest sequential increase to the structured-IR bundle, this contrast needs a purpose-built direct-code baseline (or a CAFA-style external comparison in the same ablation), or the claim should be explicitly restricted to the tested R0 prompt.","section":"Section C.4 and Table 3"},{"comment":"The six-column comparison mixes locally generated IR2Solve results with survey-reported values and SAC-Opt/OptiMUS five-run means that differ in backbone, decoding, solver time limit, and run aggregation, and two columns (IndustryOR and ComplexLP) come from a different evaluation refresh than the other four. The paper properly labels the table as a scoped clean-split comparison, but the sentence 'IR2Solve is higher than the survey rows on all six columns' is protocol-sensitive and could be misread as a controlled head-to-head. Please add an explicit comparability table (backbone, temperature/sampling, solver time limit, dataset release, run aggregation) and state clearly which columns support direct comparison.","section":"Table 2 and Section 5.2"}],"minor_comments":[{"comment":"The rendered text contains missing spaces, e.g., 'Thefirstmapinterpretsproblemsemantics;thesecondrealizes', which should be corrected.","section":"Section 3.2"},{"comment":"The pooled McNemar tests combine two heterogeneous datasets; the per-dataset counts are helpful, but the pooled p-values should carry a sentence noting that pooling assigns each instance equal weight and does not by itself establish homogeneous effects.","section":"Table A4"},{"comment":"The paper notes in Appendix H that the Chain-of-Experts token total is a lower bound because the workflow terminated before evaluation; the main text of Section 5.4 should carry this caveat in full rather than only in the appendix.","section":"Section 5.4 and Appendix H"},{"comment":"The accuracy axis is an unweighted mean across six datasets collected under different protocols; the text explains this, but the caption should state that it is a cross-study proxy, not a same-instance Pareto comparison.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically careful, and the core IR-first idea is sound. My main concern is that the deterministic verifier's semantic rewrites are developed and evaluated on the same datasets, and the R0 baseline is deliberately weakened, so the headline ablation story is not yet fully protected against alternative explanations. These issues are addressable with additional experiments and reframing, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a good, unusually transparent systems paper on one-call LLM autoformulation through an explicit intermediate representation. The ablation is the real contribution; the verifier's semantic rewrites are the soft spot, and the paper's own appendix concedes more than the abstract promises.\n\nWhat's actually new: ModelIR (a schema-constrained JSON IR) plus the scalar-constraint expansion convention, with a deterministic verifier and compiler. The nested R0→R3 ablation on 153 instances is well built: R1 removes the scalar instruction, R2 adds it, R3 applies the verifier to the same R2 output, so the verifier contrast adds no extra model call. The paired exact McNemar tests and bootstrap intervals are computed correctly. The cost panel (1 vs 8 vs 39 calls) is useful, and the paper is upfront that the CoE numbers are a pre-termination lower bound.\n\nCredit where it's due: the reporting is unusually open. Exact prompts, schema, verifier rule list, seeds, SHA-256 hashes, unit-tested examples, and a limitations section that names most of a referee's likely complaints. That raises the paper's credibility, and it should count positively in review.\n\nThe soft spots are real but not fatal. The verifier is described as 'conservative', but rules 6/7/3 (Appendix D) are not conservative syntax repairs: they change the feasible set, turning continuous variables into binary/integer based on cue words in LLM-generated names, rewriting equalities as inequalities from at-least/at-most cues, and filling missing diagonals with zeros. The paper admits these cues are not source-grounded and calls the diagonal rule a benchmark heuristic. The measured verifier gain (+11.1pp pooled) is entirely from ComplexLP; on IndustryOR it's 0.0. Without a development/holdout split, it's plausible the cue lists and thresholds are tuned to ComplexLP's failure modes. That weakens the general 'one-call competitive without repair' claim, though it does not touch the larger IR-first effect (+28.8pp), which is the paper's strongest finding.\n\nMinor: Table 2 mixes protocols, though it's flagged as scoped; R0 is deliberately unoptimized, which is fine as a controlled contrast; no public code/data link, only metadata hashes. The formal lemmas are simple and correct.\n\nBottom line: worth a serious referee, likely conditional accept after the verifier question is addressed with a cleaner rule set or a dev/test split. I'd cite it for the ablation design and the representation pattern, and bring it to a reading group for the verifier discussion.","headline":"Solid one-call IR-first autoformulation paper with a genuinely strong ablation; the deterministic verifier's semantic rewrites need a dev/test split before the general claim is secure.","tokens_in":18926,"tokens_out":3728,"would_cite":true,"duration_ms":31739,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One semantic LLM call producing a structured intermediate representation, followed by deterministic verification and compilation, is enough to make optimization autoformulation competitive with multi-agent and iterative systems at a…","keywords":["optimization autoformulation","intermediate representation","mixed-integer linear programming","large language models","deterministic verification","cost-efficient inference","structured generation"],"falsifier":"Run the pipeline with and without the verifier on a held-out set of LP/MILP instances with non-semantic variable names (e.g., x1, x2) and per-index constraints written with generic cues; if the verifier's integrality or direction rules fail to improve objective correctness over the no-verifier condition, or actively decrease it, the assumed portability of the verifier is falsified. A second check is to recompute the verifier gain after removing the leader-margin threshold and set-size cap; if the gain disappears on a broad sample, the measured +11.1 points are an artifact of benchmark-specific tuning.","tokens_in":17747,"feed_emoji":"⚙️","tokens_out":10258,"duration_ms":79748,"temperature":0.7,"pith_summary":"The paper tries to establish that LLM-based optimization autoformulation—turning a natural-language problem description into a solver-ready model—can be made both accurate and cheap by changing the representation interface rather than scaling inference. Its claim is that a single semantic LLM call producing a schema-constrained JSON intermediate representation, followed by deterministic verification and IR-to-solver compilation, matches or beats several surveyed baselines on six cleaned benchmarks. The supporting ablation on 153 instances from two benchmark families shows pooled objective correctness rising from 7.2% with direct code generation to 68.6% with the full pipeline, with the largest gain coming from the structured IR itself. If correct, this gives practitioners a predictable one-call operating point whose artifact is inspectable and replayable, in contrast to multi-agent or iterative-repair systems that use 8–39 calls and 3.3–22.9 times the tokens.","feed_headline":"Structured IR lifts LLM optimization accuracy to 68.6% at one call","feed_subtitle":"IR2Solve uses a schema-constrained JSON model, deterministic verification, and a single semantic call to rival systems spending 3.3–22.9x…","key_machinery":"The central object is ModelIR, a JSON schema with typed fields for sets, params, vars, objective, and constraints, where objective and constraint expressions are restricted Python-like strings that may contain arithmetic, indexing, and sum/quicksum aggregators but no solver code or control flow. The load-bearing convention is scalar-constraint granularity: every finite per-index constraint family is expanded into individual concrete entries, eliminating free indices and implicit 'for all' semantics; this simplifies verification and compilation at the cost of longer output for large index products. After generation, a deterministic verifier applies a fixed composition of guarded rewrite rules $V = T_K \\circ \\cdots \\circ T_1$ covering name and index canonicalization, index-scope and aggregation repair, and variable-domain and direction sanity checks; a formal lemma states that restricting generation to a representable language cannot hurt correctness when the correct formulation is expressible. The deterministic compiler then turns the verified IR into one named solver constraint per entry, making the final model a fixed, replayable function of the IR.","core_discovery":"On its own terms, the discovery is that forcing the LLM to emit a restricted, schema-constrained ModelIR instead of solver code removes most of the failure surface of autoformulation, and that a deterministic verifier can then recover a further slice of errors without any additional semantic call. The paper reports that IR2Solve is higher than the survey-reported rows on all six cleaned datasets (for example 97.4% on EasyLP and 64.3% on IndustryOR), and competitive with, though not uniformly better than, the strongest iterative semantic-correction baseline (higher on two datasets, lower on four). The mechanism evidence is the nested ablation: pooled objective correctness rises from 7.2% with direct code generation to 35.9% with the structured IR, 57.5% with the scalar-constraint instruction, and 68.6% with deterministic verification on the same 153 instances, with paired tests significant for the latter two contrasts. The paper is explicit that these are sequential stage effects, not independent causal contributions, and that the evaluation is a scoped clean-split comparison rather than a fully controlled head-to-head.","pith_inferences":["Editorial inference: the large direct-code-to-IR gain suggests that the dominant source of error in code-first autoformulation is not missing problem semantics but the fragile program scaffolding around the math; richer schemas (e.g., typed index families or named quantifier scopes) may push accuracy further without more inference compute.","Editorial inference: scalar expansion is acceptable at benchmark scale, but for large Cartesian index products the output-length cost could become prohibitive; a compressed schema that still lets a deterministic checker expand scopes might preserve the accuracy gain with less truncation risk.","Editorial inference: because the verifier's integrality and direction rules rely on cue words in LLM-generated names rather than source text, a natural test is whether source-grounded cues would transfer the +11.1-point verifier gain to problem families with non-standard naming.","Editorial inference: the objective-correctness metric allows different feasible regions with the same optimal value to pass; extending evaluation to compare feasible regions or optimal solution sets on small instances would reveal whether the accuracy gains are semantic or just objective-level."],"forward_implications":["A one-call pipeline makes inference cost predictable and auditable, since the number of semantic calls is fixed at one per instance rather than open-ended.","Because each constraint is stored as a concrete scalar entry, practitioners can inspect declared variables, domains, objective direction, and individual constraints before solving, separating semantic modeling errors from program-construction failures.","The representation interface, not additional reasoning, produces the largest observed accuracy gain (+28.8 pooled points for the direct-code-to-IR contrast), implying that better output schemas can substitute for iterative repair on these benchmarks.","Deterministic verification adds +11.1 pooled points but is dataset-dependent (0.0 on one benchmark family, +15.3 on the other), so its benefit should be expected to vary across problem distributions.","On the matched cost panel, IR2Solve uses 16,031 tokens per instance versus 53,459 for a multi-agent workflow and 367,028 for an iterative semantic-correction workflow—3.3x and 22.9x fewer—while using one semantic call instead of eight and 39."],"supporting_citations":[{"why":"Supplies the cleaned benchmark releases and the survey baseline rows used in the six-dataset comparison.","marker":"[Xiao et al., 2025]"},{"why":"Provides the NL4Opt benchmark, one of the six cleaned datasets and the source of the COST-10 cost panel.","marker":"[Ramamonjison et al., 2023]"},{"why":"Provides the MAMO EasyLP and ComplexLP datasets used for the 153-instance ablation.","marker":"[Huang et al., 2025b]"},{"why":"Provides the OptiMUS modular workflow baseline and the NLP4LP benchmark lineage.","marker":"[AhmadiTeshnizi et al., 2024]"},{"why":"Provides the SAC-Opt iterative semantic-correction baseline and its five-run accuracy means.","marker":"[Zhang et al., 2026]"},{"why":"Provides the Chain-of-Experts multi-agent baseline used for accuracy and cost comparison.","marker":"[Xiao et al., 2024]"},{"why":"Provides CAFA, the code-first baseline that anchors the direct-code contrast and the cost panel.","marker":"[Deng et al., 2024]"},{"why":"Supplies the ReSocratic/OptiBench dataset in the cleaned benchmark comparison.","marker":"[Yang et al., 2025]"}],"fun_headline_variants":["IR2Solve: 1 call, 68.6% accuracy via structured IR","Structured IR + deterministic check: 68.6% at one call","IR2Solve: One semantic call, 68.6% objective accuracy","Cut calls 8–39x, keep 68.6% with IR2Solve"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the verifier's hand-written rewrite rules are conservative and portable: in particular, the cue-word rules that convert continuous variables to binary/integer and equality constraints to inequalities rely on LLM-generated names and descriptions that are not independently grounded in the source text, and thresholds such as the free-index leader margin, the 1–50 set-size cap, and the zero-diagonal assumption may have been tuned to the benchmark distribution.","fun_headline_variants_meta":{"raw":{"variants":["IR2Solve: 1 call, 68.6% accuracy via structured IR","Structured IR + deterministic check: 68.6% at one call","IR2Solve: One semantic call, 68.6% objective accuracy","Cut calls 8–39x, keep 68.6% with IR2Solve"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2997,"prompt_tokens":1068,"completion_tokens":1929,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":1841}},"tokens_in":684,"tokens_out":1929,"duration_ms":12563,"temperature":1.0,"reasoning_tokens":1841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:22:36.435884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline with and without the verifier on a held-out set of LP/MILP instances with non-semantic variable names (e.g., x1, x2) and per-index constraints written with generic cues; if the verifier's integrality or direction rules fail to improve objective correctness over the no-verifier condition, or actively decrease it, the assumed portability of the verifier is falsified. A second check is to recompute the verifier gain after removing the leader-margin threshold and set-size cap; if the gain disappears on a broad sample, the measured +11.1 points are an artifact of benchmark-specific tuning.","supporting_citations":[{"cited_title":"A Survey of Optimization Modeling Meets","cited_arxiv_id":null,"evidence_quote":"Supplies the cleaned benchmark releases and the survey baseline rows used in the six-dataset comparison."}],"review_version":1}