{"id":"07019381-0e68-4e09-930f-5d40d00f273b","arxiv_id":"2508.10047","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of LLM-based optimization modeling, plus an audit revealing high error rates in existing benchmarks and a cleaned leaderboard.","lead":"This paper surveys how large language models are being used to turn plain-English descriptions into mathematical optimization models. It also audits popular benchmark datasets, finds many contain errors, and offers a cleaned set and a fresh performance ranking.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Error-rate audit lacks annotation protocol and inter-annotator agreement; cleaned datasets and leaderboard inherit this unverified foundation.","rationale":"The reader's weakest assumption exactly matches the load-bearing concern I identify: the reliability of the human error annotation. The paper's central empirical claims—high benchmark error rates, the need for cleaning, and the validity of the cleaned leaderboard—all depend on this unverified labeling process. The paper reports only the number of experts and the cross-validation threshold, omitting annotation criteria, agreement measures, and dispute resolution. This is a fixable methodological gap, not a fatal flaw, so the CONDITIONAL verdict stands. I did not find a separate concern that would require changing the verdict. The leaderboard fairness issue (mixed base models) is secondary because it can be addressed by additional runs and does not undermine the error-rate finding. The complexity metric is also less central. Therefore, my stress test supports the reader's conditional assessment with the same primary concern.","tokens_in":12728,"tokens_out":2745,"duration_ms":30052,"concrete_test":"Randomly sample 100 instances stratified across the seven benchmarks. Have a new group of at least three OR experts, blinded to the paper's labels, independently annotate errors using a preregistered protocol with explicit error categories (e.g., unboundedness, parameter inconsistency, wrong ground truth) and decision rules for ambiguous cases. Compute pairwise Cohen's kappa and compare the re-estimated error rates to Table 1. If kappa < 0.6 or the error rates differ by more than 10 percentage points, the audit is not robust and the cleaning decisions must be reconsidered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical contribution is Table 1's claim that existing benchmarks have high error rates (≥8.13% to ≥54.0%), motivating the cleaned datasets and new leaderboard. The audit method (Section 3.3) is described in one sentence: '11 human experts manually identify errors... each error case is cross-validated by at least three different experts.' No annotation criteria are given (what counts as a 'logical error'? how are ambiguous cases resolved?), no inter-annotator agreement is reported (e.g., Cohen's kappa), and there is no breakdown of how many instances were flagged by 1, 2, or 3+ experts. Without this, the error rates are not reproducible or verifiable. Furthermore, the subsequent 'manual filter[ing]' of error cases could introduce selection bias if the same subjective criteria are applied. Since the cleaned datasets are then used to produce the leaderboard (Table 2), any unreliability in the error annotation propagates to the headline findings. If the error rates are over- or underestimated, the motivation for cleaning, and the specific instances removed, may be flawed. The paper itself provides no evidence that the experts agreed on what constitutes an error, leaving the entire empirical claim resting on an unvalidated human-labeling process.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey and empirical audit of LLM-based optimization modeling. It organizes the field into a technical stack spanning data synthesis and fine-tuning, inference frameworks, benchmark datasets, and evaluation protocols. Its principal empirical contribution is an analysis of seven existing benchmark datasets (Table 1), reporting high error rates between ≥8.13% (EasyLP) and ≥54.0% (IndustryOR), based on manual review by 11 human experts. The authors then clean these datasets, present a new leaderboard for open-source methods (Table 2) under a supposedly unified evaluation setting, and provide an online portal aggregating datasets, code, and papers. The paper concludes with challenges and future directions such as reasoning models, explainability, domain-knowledge injection, and human-in-the-loop modeling.","tokens_in":13038,"tokens_out":6387,"duration_ms":68798,"significance":"If the error-rate findings are trustworthy, the paper addresses a timely and important problem: benchmark noise can materially distort reported progress in LLM-based optimization modeling. The effort to curate cleaned datasets and to re-evaluate multiple open-source methods under one protocol is a valuable service to the community, and the online portal is a practical contribution. The taxonomy is broad and will be useful as a reference map of the area. However, the empirical core of the paper is not yet adequately supported. The error-rate audit is described in a single sentence, the cleaning procedure is underspecified, and the leaderboard's fairness claim is weakened by the use of different base models across compared methods. These are load-bearing issues because the abstract and introduction explicitly advertise fair comparison and reliable cleaned benchmarks. With additional detail and possibly re-analysis, these concerns are addressable, but the current version does not fully substantiate its central empirical claims.","major_comments":[{"comment":"The error-rate audit is not reproducible as reported. The text states that 11 human experts manually identified errors and that each error case was cross-validated by at least three experts, but it does not define what counts as an 'error,' give annotation criteria, report inter-annotator agreement (e.g., Cohen's kappa), or describe how disagreements were resolved. It also does not report the number of instances flagged by one, two, three, or more experts, nor the exact counts behind the '≥' figures. Because Table 1 motivates the entire cleaning effort and the subsequent leaderboard, this is a load-bearing gap. Please provide an annotation rubric (in main text or appendix), agreement statistics, per-dataset counts, and a clarification of what the '≥' symbol denotes.","section":"Section 3.3, Table 1"},{"comment":"The cleaning step is insufficiently documented. The paper says 'we manually filter all error cases' and 'compile a unified, cleaned collection,' but it does not state how many instances were removed or modified per dataset, what the post-clean sizes are, or how borderline cases were treated. The cleaned datasets are only available through the online portal, not described in the manuscript. Without this information, readers cannot assess selection bias introduced by the same subjective criteria used in the error audit, and the leaderboard in Table 2 cannot be independently checked on the exact same data. Please report the per-dataset cleaning statistics and publish the full list of removed/modified instances with reasons.","section":"Section 3.3, cleaned datasets"},{"comment":"The leaderboard does not establish the claimed fairness with respect to the base LLM. The caption and text say GPT-4o is used for training-free methods, but the ORLM row is a fine-tuned LLaMA-3-8B model. Thus any performance difference between ORLM and the training-free methods confounds the method with the base-model capability. The abstract's claim of 'fair performance evaluation in terms of base LLM model and datasets' is therefore not supported by Table 2. To fix this, either evaluate all methods on a common base model (including fine-tuned methods) or clearly decompose and report the base-model factor, for instance by also running training-free methods on the same LLaMA-3-8B base.","section":"Section 3.4, Table 2"},{"comment":"The 'accuracy' metric used in Table 2 is not defined precisely. The text discusses objective-wise (test-driven) and model-wise evaluation, but it does not state which of these is used for the leaderboard, nor the solver, time limit, tolerance, or handling of infeasible/unbounded formulations and decimal precision. The claim of a 'standardized evaluation method' requires these details to be reproducible and to support cross-method comparisons. Please specify the evaluation protocol for each benchmark and report the relevant hyperparameters.","section":"Section 3.4, evaluation metric"}],"minor_comments":[{"comment":"Typo: 'identity the optimization objective' should be 'identify the optimization objective.'","section":"Section 1"},{"comment":"Grammar: 'A model whose parameters are denoted by mathematical symbols called a abstract model' should be 'is called an abstract model.'","section":"Section 2.2"},{"comment":"Table 1 lists EasyLP and ComplexLP as separate rows; the text should make explicit that these are subsets of MAMO and clarify the relation between their sizes and MAMO's overall test set.","section":"Section 3.3"},{"comment":"The '≥' symbol in the Error Rate column is not explained in the table caption. A sentence defining what the lower bound represents would improve clarity.","section":"Table 1"},{"comment":"The timeline/taxonomy figure is dense and many abbreviations (e.g., CAFA, TTG, EOR) are not expanded in the caption. A short definition list or a more detailed caption would help readers.","section":"Figure 3"},{"comment":"Reference formatting is inconsistent: '[haoxuan deng et al., 2024]' uses lower-case author name and '[JU et al., 2024]' has unusual capitalization; please normalize to the journal style.","section":"References"},{"comment":"Typo: 'There are two primary approaches exist' should be 'There are two primary approaches.'","section":"Section 3.4"},{"comment":"The complexity metric is defined as the number of variables and constraints in a model generated by 'standard prompting.' This is pipeline-dependent; a sentence acknowledging that different prompting strategies may yield different complexity estimates would be appropriate.","section":"Section 3.3, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The survey includes and ranks methods by some of the same authors (notably Chain-of-Experts) as competitive on the new leaderboard. This is not by itself improper, but given the paper also performs the data cleaning, an independent audit of the cleaning rules and evaluation protocol would substantially increase confidence. I would also encourage the editor to make sure that the online portal and cleaned datasets are archived in a stable repository, since the manuscript's empirical contribution depends on access to those artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on LLMs for OR. The survey part is a competent synthesis of the technical stack—data synthesis, fine-tuning, inference frameworks, benchmarks, evaluation—and the historical figure of representative works is genuinely handy for newcomers. But the real contribution is the audit: Table 1 reports error rates from 8.13% (EasyLP) to 54.0% (IndustryOR) across seven benchmarks, and the authors clean the datasets and provide a unified leaderboard. That is a legitimate, useful resource for the community, and the observation that existing benchmarks are error-prone is not just plausible but well-motivated by the internal examples they give (logical errors, poorly defined parameters, wrong ground truth). The cleaned leaderboard also gives three sensible takeaways: CoE and ORLM lead, CoT is not universally helpful, and CAFA behaves like a specialized CoT. I agree with the reader that this is real new empirical work, not just a survey. The soft spots are mostly in the methods section, and they are real but fixable. The error audit is described in one sentence: 11 human experts, each case cross-validated by at least three experts. No annotation criteria, no inter-annotator agreement, no breakdown of how many cases were flagged by one vs. multiple experts, no resolution protocol for disputes. That matters because the whole persuasion of the paper rests on those numbers. If the experts disagreed heavily, the error rates could be off, and the cleaned datasets inherit that. I would not call this a fatal flaw—the claim that there are many bad instances is almost certainly true—but it is under-supported. Second, the leaderboard mixes base models: ORLM is a fine-tuned LLaMA-3 8B while training-free methods use GPT-4o, so the comparison is not fully 'fair' despite the paper's stated goal. Third, the cleaned datasets and annotations are not shipped in the paper; they are on a portal, which makes verification harder. These are the kind of issues a decent referee would flag as major revisions, not grounds for rejection. The survey part holds up fine; the empirical part needs more methodological transparency. Who is this for? Anyone entering the field or needing a quick map of methods and benchmarks. It deserves a serious referee and likely publication after the annotation protocol is documented. I would cite it for the audit numbers and the cleaned benchmark resource.","headline":"A competent survey with a genuinely useful empirical audit of benchmark quality, whose main credibility problem is the under-documented human annotation protocol behind its headline error rates.","tokens_in":13482,"tokens_out":587,"would_cite":true,"duration_ms":7478,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims current LLM-optimization benchmarks contain high error rates—up to 54% on one dataset—and that a cleaned, unified leaderboard offers a fairer basis for comparing methods.","keywords":["optimization modeling","large language models","benchmark quality","data cleaning","leaderboard","evaluation protocol","operations research","chain-of-thought"],"falsifier":"Take a random sample of instances flagged as erroneous by the paper and have independent experts re-annotate them using a written rubric; if the independently measured error rate on a benchmark like IndustryOR falls far below 54% (e.g., below 20%), the claim of high error rates would be refuted.","tokens_in":12658,"feed_emoji":"📊","tokens_out":4397,"duration_ms":41317,"temperature":0.7,"pith_summary":"The paper reviews the field of using large language models for optimization modeling and makes a central empirical claim: existing benchmark datasets are surprisingly unreliable. A manual audit by 11 human experts found error rates of at least 8.13% on EasyLP and as high as 54.0% on IndustryOR, with most datasets exceeding 15%. To address this, the authors cleaned the benchmarks, unified them, and reran all fully open-source methods under a standardized protocol. The resulting leaderboard identifies the most competitive methods and shows that chain-of-thought prompting is not universally beneficial.","feed_headline":"LLM optimization benchmarks hold up to 54% bad cases","feed_subtitle":"After cleaning seven datasets, a unified leaderboard shows trained models beat prompting on hard problems.","key_machinery":"The benchmark-quality audit and the standardized evaluation protocol. The audit quantifies error rates by having 11 human experts flag erroneous instances, with each case cross-validated by at least three experts; the protocol then evaluates all open-source methods on cleaned data using the same base model (gpt-4o-2024-08-06) and the same accuracy metric, eliminating confounds from differing base models, preprocessing, and metrics.","core_discovery":"The paper's central claim is that the benchmark datasets used to evaluate LLM-based optimization modeling are substantially flawed: manual expert audit found error rates of at least 8.13% (EasyLP), 26.4% (NL4Opt), and 54.0% (IndustryOR), with most datasets exceeding 15%. Because ground-truth labels and problem descriptions contain logical errors, poorly defined parameters, or incorrect objectives, published evaluation results are not fully reliable. To fix this, the authors cleaned all erroneous cases, unified the datasets, and reran every fully open-source method under one protocol (gpt-4o as base model, accuracy as metric). The resulting leaderboard shows Chain-of-Experts and ORLM as the c","pith_inferences":["If the reported error rates generalize, some previously reported performance gaps between methods may be artifacts of dirty labels rather than real capability differences.","A natural next step is to make the cleaning process reproducible by publishing annotation guidelines, per-instance flags, and inter-annotator agreement so the community can audit the audit.","The same audit protocol could be applied to newer or larger datasets, especially abstract-modeling benchmarks, as a standard quality gate before release.","Model-wise evaluation is more informative than objective-wise accuracy; combining cleaned benchmarks with graph-edit-distance scoring could yield a finer-grained leaderboard."],"forward_implications":["Published accuracy numbers on the original benchmarks are not directly comparable, since flawed labels can inflate or deflate scores.","The cleaned datasets and unified leaderboard give the community a common testbed for measuring future methods.","Trained open-source models such as ORLM appear stronger than prompt-only methods on harder instances, suggesting fine-tuning pays off as complexity rises.","Chain-of-thought prompting should be applied selectively; on some datasets it lowers accuracy.","Current benchmarks lack hard instances, so progress on truly complex OR problems may be under-measured."],"supporting_citations":[{"why":"Supplies the NL4Opt benchmark and the protocol for extracting canonical formulation matrices; one of the datasets the paper audits.","marker":"[Ramamonjison et al., 2023]"},{"why":"Introduces IndustryOR and ORLM; IndustryOR is the dataset with the highest measured error rate (54.0%) and ORLM is a leading open-source method on the cleaned leaderboard.","marker":"[Tang et al., 2024]"},{"why":"Provides MAMO with EasyLP and ComplexLP subsets; both are audited in Table 1.","marker":"[Huang et al., 2024]"},{"why":"ReSocratic filters erroneous cases and extends the test set; its dataset is also audited and compared.","marker":"[Yang et al., 2025]"},{"why":"OptiMUS/NLP4LP contributes the abstract-modeling benchmark NLP4LP used in the audit.","marker":"[AhmadiTeshnizi et al., 2024]"},{"why":"Chain-of-Experts introduces ComplexOR and the test-driven objective-wise evaluation method that the unified protocol adopts.","marker":"[Xiao et al., 2024b]"},{"why":"OptiBench provides a model-data separation dataset and graph-isomorphism evaluation; relevant to the abstract-modeling category.","marker":"[Wang et al., 2024b]"},{"why":"Supports the paper's finding that chain-of-thought should be applied selectively, since CoT helps mainly on math and symbolic reasoning.","marker":"[Sprague et al., 2024]"}],"fun_headline_variants":["LLM optimization benchmarks up to 54% error","Cleaning datasets upends LLM optimization ranking","Trained models outdo prompting after benchmark cleanup","Hidden flaws found in LLM optimization test sets","New leaderboard corrects LLM optimization evaluations"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The entire quality finding rests on the 11 experts' manual annotations being correct and unbiased; the paper does not report the annotation criteria, the inter-annotator agreement, or how disputes were resolved, so the error rates could shift under a different labeling protocol.","fun_headline_variants_meta":{"raw":{"variants":["LLM optimization benchmarks up to 54% error","Cleaning datasets upends LLM optimization ranking","Trained models outdo prompting after benchmark cleanup","Hidden flaws found in LLM optimization test sets","New leaderboard corrects LLM optimization evaluations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":1984,"prompt_tokens":690,"completion_tokens":1294,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":1222}},"tokens_in":434,"tokens_out":1294,"duration_ms":11072,"temperature":1.0,"reasoning_tokens":1222,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T21:23:53.477109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of instances flagged as erroneous by the paper and have independent experts re-annotate them using a written rubric; if the independently measured error rate on a benchmark like IndustryOR falls far below 54% (e.g., below 20%), the claim of high error rates would be refuted.","supporting_citations":[{"cited_title":"Optimus: Scalable optimization modeling with (MI)LP solvers and large language models","cited_arxiv_id":null,"evidence_quote":"OptiMUS/NLP4LP contributes the abstract-modeling benchmark NLP4LP used in the audit."},{"cited_title":"To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning, 2024","cited_arxiv_id":null,"evidence_quote":"Supports the paper's finding that chain-of-thought should be applied selectively, since CoT helps mainly on math and symbolic reasoning."}],"review_version":1}