Pith. sign in

REVIEW 4 major objections 8 minor 7 cited by

A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions

T0 review · 4 major / 8 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey claims current LLM-optimization benchmarks contain high error rates—up to 54% on one dataset—and that a cleaned, unified leaderboard offers a fairer basis for comparing methods.

desk verdict A competent survey with a genuinely useful empirical audit of benchmark quality, whose main credibility problem is the under-documented human annotation protocol behind its headline error rates. read the letter →

arxiv 2508.10047 v1 pith:46KNRCYB submitted 2025-08-12 cs.AI

Ziyang Xiao , Jingrong Xie , Lilin Xu , Shisi Guan , Jingyan Zhu , Xiongwei Han , Xiaojin Fu , WingYin Yu
show 10 more authors
Han Wu Wei Shi Qingcan Kang Jiahui Duan Tao Zhong Mingxuan Yuan Jia Zeng Yuan Wang Gang Chen Dongxiang Zhang
This is my paper
classification cs.AI
keywords optimizationmodelinglargelanguagemodelsbenchmarkqualitydatacleaningleaderboardevaluationprotocoloperationsresearchchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reviews the field of using large language models for optimization modeling and makes a central empirical claim: existing benchmark datasets are surprisingly unreliable. A manual audit by 11 human experts found error rates of at least 8.13% on EasyLP and as high as 54.0% on IndustryOR, with most datasets exceeding 15%. To address this, the authors cleaned the benchmarks, unified them, and reran all fully open-source methods under a standardized protocol. The resulting leaderboard identifies the most competitive methods and shows that chain-of-thought prompting is not universally beneficial.

What carries the argument

The benchmark-quality audit and the standardized evaluation protocol. The audit quantifies error rates by having 11 human experts flag erroneous instances, with each case cross-validated by at least three experts; the protocol then evaluates all open-source methods on cleaned data using the same base model (gpt-4o-2024-08-06) and the same accuracy metric, eliminating confounds from differing base models, preprocessing, and metrics.

What would settle it

Take a random sample of instances flagged as erroneous by the paper and have independent experts re-annotate them using a written rubric; if the independently measured error rate on a benchmark like IndustryOR falls far below 54% (e.g., below 20%), the claim of high error rates would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that the benchmark datasets used to evaluate LLM-based optimization modeling are substantially flawed: manual expert audit found error rates of at least 8.13% (EasyLP), 26.4% (NL4Opt), and 54.0% (IndustryOR), with most datasets exceeding 15%. Because ground-truth labels and problem descriptions contain logical errors, poorly defined parameters, or incorrect objectives, published evaluation results are not fully reliable. To fix this, the authors cleaned all erroneous cases, unified the datasets, and reran every fully open-source method under one protocol (gpt-4o as base model, accuracy as metric). The resulting leaderboard shows Chain-of-Experts and ORLM as the c

Load-bearing premise

The entire quality finding rests on the 11 experts' manual annotations being correct and unbiased; the paper does not report the annotation criteria, the inter-annotator agreement, or how disputes were resolved, so the error rates could shift under a different labeling protocol.

Editorial extensions

If this is right

  • Published accuracy numbers on the original benchmarks are not directly comparable, since flawed labels can inflate or deflate scores.
  • The cleaned datasets and unified leaderboard give the community a common testbed for measuring future methods.
  • Trained open-source models such as ORLM appear stronger than prompt-only methods on harder instances, suggesting fine-tuning pays off as complexity rises.
  • Chain-of-thought prompting should be applied selectively; on some datasets it lowers accuracy.
  • Current benchmarks lack hard instances, so progress on truly complex OR problems may be under-measured.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported error rates generalize, some previously reported performance gaps between methods may be artifacts of dirty labels rather than real capability differences.
  • A natural next step is to make the cleaning process reproducible by publishing annotation guidelines, per-instance flags, and inter-annotator agreement so the community can audit the audit.
  • The same audit protocol could be applied to newer or larger datasets, especially abstract-modeling benchmarks, as a standard quality gate before release.
  • Model-wise evaluation is more informative than objective-wise accuracy; combining cleaned benchmarks with graph-edit-distance scoring could yield a finer-grained leaderboard.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This manuscript is a survey and empirical audit of LLM-based optimization modeling. It organizes the field into a technical stack spanning data synthesis and fine-tuning, inference frameworks, benchmark datasets, and evaluation protocols. Its principal empirical contribution is an analysis of seven existing benchmark datasets (Table 1), reporting high error rates between ≥8.13% (EasyLP) and ≥54.0% (IndustryOR), based on manual review by 11 human experts. The authors then clean these datasets, present a new leaderboard for open-source methods (Table 2) under a supposedly unified evaluation setting, and provide an online portal aggregating datasets, code, and papers. The paper concludes with challenges and future directions such as reasoning models, explainability, domain-knowledge injection, and human-in-the-loop modeling.

Significance. If the error-rate findings are trustworthy, the paper addresses a timely and important problem: benchmark noise can materially distort reported progress in LLM-based optimization modeling. The effort to curate cleaned datasets and to re-evaluate multiple open-source methods under one protocol is a valuable service to the community, and the online portal is a practical contribution. The taxonomy is broad and will be useful as a reference map of the area. However, the empirical core of the paper is not yet adequately supported. The error-rate audit is described in a single sentence, the cleaning procedure is underspecified, and the leaderboard's fairness claim is weakened by the use of different base models across compared methods. These are load-bearing issues because the abstract and introduction explicitly advertise fair comparison and reliable cleaned benchmarks. With additional detail and possibly re-analysis, these concerns are addressable, but the current version does not fully substantiate its central empirical claims.

major comments (4)
  1. [Section 3.3, Table 1] The error-rate audit is not reproducible as reported. The text states that 11 human experts manually identified errors and that each error case was cross-validated by at least three experts, but it does not define what counts as an 'error,' give annotation criteria, report inter-annotator agreement (e.g., Cohen's kappa), or describe how disagreements were resolved. It also does not report the number of instances flagged by one, two, three, or more experts, nor the exact counts behind the '≥' figures. Because Table 1 motivates the entire cleaning effort and the subsequent leaderboard, this is a load-bearing gap. Please provide an annotation rubric (in main text or appendix), agreement statistics, per-dataset counts, and a clarification of what the '≥' symbol denotes.
  2. [Section 3.3, cleaned datasets] The cleaning step is insufficiently documented. The paper says 'we manually filter all error cases' and 'compile a unified, cleaned collection,' but it does not state how many instances were removed or modified per dataset, what the post-clean sizes are, or how borderline cases were treated. The cleaned datasets are only available through the online portal, not described in the manuscript. Without this information, readers cannot assess selection bias introduced by the same subjective criteria used in the error audit, and the leaderboard in Table 2 cannot be independently checked on the exact same data. Please report the per-dataset cleaning statistics and publish the full list of removed/modified instances with reasons.
  3. [Section 3.4, Table 2] The leaderboard does not establish the claimed fairness with respect to the base LLM. The caption and text say GPT-4o is used for training-free methods, but the ORLM row is a fine-tuned LLaMA-3-8B model. Thus any performance difference between ORLM and the training-free methods confounds the method with the base-model capability. The abstract's claim of 'fair performance evaluation in terms of base LLM model and datasets' is therefore not supported by Table 2. To fix this, either evaluate all methods on a common base model (including fine-tuned methods) or clearly decompose and report the base-model factor, for instance by also running training-free methods on the same LLaMA-3-8B base.
  4. [Section 3.4, evaluation metric] The 'accuracy' metric used in Table 2 is not defined precisely. The text discusses objective-wise (test-driven) and model-wise evaluation, but it does not state which of these is used for the leaderboard, nor the solver, time limit, tolerance, or handling of infeasible/unbounded formulations and decimal precision. The claim of a 'standardized evaluation method' requires these details to be reproducible and to support cross-method comparisons. Please specify the evaluation protocol for each benchmark and report the relevant hyperparameters.
minor comments (8)
  1. [Section 1] Typo: 'identity the optimization objective' should be 'identify the optimization objective.'
  2. [Section 2.2] Grammar: 'A model whose parameters are denoted by mathematical symbols called a abstract model' should be 'is called an abstract model.'
  3. [Section 3.3] Table 1 lists EasyLP and ComplexLP as separate rows; the text should make explicit that these are subsets of MAMO and clarify the relation between their sizes and MAMO's overall test set.
  4. [Table 1] The '≥' symbol in the Error Rate column is not explained in the table caption. A sentence defining what the lower bound represents would improve clarity.
  5. [Figure 3] The timeline/taxonomy figure is dense and many abbreviations (e.g., CAFA, TTG, EOR) are not expanded in the caption. A short definition list or a more detailed caption would help readers.
  6. [References] Reference formatting is inconsistent: '[haoxuan deng et al., 2024]' uses lower-case author name and '[JU et al., 2024]' has unusual capitalization; please normalize to the journal style.
  7. [Section 3.4] Typo: 'There are two primary approaches exist' should be 'There are two primary approaches.'
  8. [Section 3.3, Figure 5] The complexity metric is defined as the number of variables and constraints in a model generated by 'standard prompting.' This is pipeline-dependent; a sentence acknowledging that different prompting strategies may yield different complexity estimates would be appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark-quality audit is an empirical measurement, not a derivation, and self-citations are not load-bearing.

full rationale

This paper is a survey with an empirical benchmark-quality audit, not a derivation chain. The central claim—that existing optimization-modeling benchmarks have high error rates (Table 1)—is an external measurement based on manual expert annotation, not a quantity defined in terms of the paper's own outputs. The cleaned datasets and leaderboard (Table 2) are downstream artifacts whose reliability depends on the annotation process, which is underdocumented in Section 3.3 (no annotation criteria, no inter-annotator agreement, no dispute-resolution protocol). That is a validity/correctness risk, not circularity: the error rates are not equal to any input of the paper by construction. The paper does cite and evaluate the authors' own methods and datasets (Chain-of-Experts, ComplexOR, EOR), and it ranks Chain-of-Experts highly in Table 2, but this is self-citation rather than circular reasoning. ComplexOR is an external benchmark introduced in prior work; evaluating CoE on it involves no fitted parameter or definitional equivalence. The leaderboard uses cleaned versions of external benchmarks, so the evaluation is not self-referential. No step reduces to its own input, and no 'prediction' is a renamed fit. Therefore the paper warrants a circularity score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The empirical part of the paper rests on human annotation reliability, metric choice, benchmark representativeness, and the choice of a single base model. None of these are formally verified or externally anchored, and the fairness claim is weakened by the base-model mismatch.

assumptions (4)
  • domain assumption Human expert annotations are a reliable ground truth for identifying errors in benchmark instances.
    The error-rate claim rests on 11 experts manually labeling errors with cross-validation, but inter-annotator agreement and the annotation checklist are not reported in Section 3.3.
  • domain assumption Accuracy of the final objective value is a sufficient metric for comparing optimization modeling methods.
    The leaderboard uses accuracy as the primary metric, inheriting the limitations of objective-wise evaluation acknowledged in Section 3.4.
  • domain assumption The selected benchmarks are representative of the field.
    Conclusions about error rates and leaderboard are drawn from seven benchmarks; WIQOR and OptiBench are discussed but not included in the cleaned evaluation.
  • ad hoc to paper GPT-4o is a fair and representative base model for all training-free methods.
    The unified evaluation adopts gpt-4o-2024-08-06 for training-free methods, but this choice is not justified as neutral; other base models could change rankings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions." pith.science (2026). https://pith.science/paper/46KNRCYB

@misc{pith2026250810047,
  author       = {Pith},
  title        = {Pith review of: A Survey of Optimization Modeling Meets LLMs: Progress and Future Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46KNRCYB}},
  note         = {Machine review of arXiv:2508.10047}
}
read the original abstract

By virtue of its great utility in solving real-world problems, optimization modeling has been widely employed for optimal decision-making across various sectors, but it requires substantial expertise from operations research professionals. With the advent of large language models (LLMs), new opportunities have emerged to automate the procedure of mathematical modeling. This survey presents a comprehensive and timely review of recent advancements that cover the entire technical stack, including data synthesis and fine-tuning for the base model, inference frameworks, benchmark datasets, and performance evaluation. In addition, we conducted an in-depth analysis on the quality of benchmark datasets, which was found to have a surprisingly high error rate. We cleaned the datasets and constructed a new leaderboard with fair performance evaluation in terms of base LLM model and datasets. We also build an online portal that integrates resources of cleaned datasets, code and paper repository to benefit the community. Finally, we identify limitations in current methodologies and outline future research opportunities.

Figures

Figures reproduced from arXiv: 2508.10047 by the authors.

Figure 1
Figure 1. An example of an optimization modeling task. The orange [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison between concrete and abstract models. The [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Left: Taxonomy of LLMs-based optimization modeling, organized according to the LLMs’ technical stack. Right: Representative [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Three types of inference methods. approach scales up the single LLM to the LLM-based multi￾agent system (referred to as “multi-expert”). Prompt At the advent of ChatGPT, NL4Opt [Ramamon￾jison et al., 2023] pioneers the use of ChatGPT for solving optimization modeling p…
Figure 5
Figure 5. Figure 5: Statistics of complexity distribution for each benchmark [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Generating Robust Portfolios of Optimization Models using Large Language Models

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    An algorithm generates a portfolio of LLM-produced optimization models with guarantees that high-quality candidates are included if either the generator or evaluator aligns with human preferences.

  2. Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    LLMs hit an effective binding limit in text-to-optimization as data complexity grows, but externalizing numeric data via BIND and binding-focused finetuning raises accuracy from 59% to 82%+ and lets small specialists ...

  3. Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization

    cs.LG 2026-05 conditional novelty 6.0 of 10

    A new solver-verified benchmark shows that LLMs solving optimization problems from text fail mainly at transcribing numerical data ('binding'), and file-based data access or binding-specific fine-tuning recovers most ...

  4. Democratizing Large-Scale Re-Optimization with LLM-Guided Model Patches

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    LLM agent translates user prompts into model patches and selects primal-aware re-optimization techniques for large-scale dynamic problems, shown on supply-chain and exam-scheduling cases.

  5. From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modeling

    math.OC 2026-04 unverdicted novelty 6.0 of 10

    Agora-Opt uses decentralized debate among LLM agent teams plus a read-write memory bank to produce more accurate optimization models from text than prior LLM methods.

  6. AutoOR: Scalably Post-training LLMs to Autoformalize Operations Research Problems

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    AutoOR uses synthetic data generation and RL post-training with solver feedback to enable 8B LLMs to autoformalize linear, mixed-integer, and non-linear OR problems, matching larger models on benchmarks.

  7. Democratizing Large-Scale Re-Optimization with LLM-Guided Model Patches

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    An LLM agent converts user prompts into optimization-model patches and selects primal-based re-optimization methods from a toolbox to produce feasible solutions for dynamic supply-chain and exam-scheduling problems.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages · cited by 5 Pith papers

  1. [1]

    Optimus: Scalable optimization modeling with (MI)LP solvers and large language models

    Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. Optimus: Scalable optimization modeling with (MI)LP solvers and large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024

    work page 2024

  2. [2]

    Autoformulation of mathematical optimization models using llms, 2024

    Nicolás Astorga, Tennison Liu, Yuanzhang Xiao, and Mihaela van der Schaar. Autoformulation of mathematical optimization models using llms, 2024

    work page 2024

  3. [3]

    Graph of Thoughts: Solving Elaborate Problems with Large Language Models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Micha Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of Thoughts: Solving Elaborate Problems with Large Language Models . Proceedings of the AAAI Conference on Artificial Intelligence , 38(16):17682--17690, Mar 2024

    work page 2024

  4. [4]

    Constante - Flores, and Can Li

    Hao Chen, Gonzalo E. Constante - Flores, and Can Li. Diagnosing infeasible optimization problems using large language models. CoRR , abs/2308.12923, 2023

    arXiv 2023

  5. [5]

    Navigate through enigmatic labyrinth A survey of chain of thought reasoning: Advances, frontiers and future

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth A survey of chain of thought reasoning: Advances, frontiers and future. pages 1173--1203. Association for Computational Linguistics, 2024

    work page 2024

  6. [6]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR , abs/2110.14168, 2021

    arXiv 2021

  7. [7]

    Cuthbertson

    Richard W. Cuthbertson. The logic of logistics: Theory, algorithms and applications for logistics management. J. Oper. Res. Soc. , 49(9):1016--1017, 1998

    work page 1998

  8. [8]

    Leal de Matos and Richard J

    Paula A. Leal de Matos and Richard J. Ormerod. The application of operational research to european air traffic flow management - understanding the context. Eur. J. Oper. Res. , 123(1):125--144, 2000

    work page 2000

Show all 43 references
  1. [9]

    An equity-based optimization model to solve the location problem for healthcare centers applied to hospital beds and covid-19 vaccination

    Erwin J Delgado, Xavier Cabezas, Carlos Martin-Barreiro, V \' ctor Leiva, and Fernando Rojas. An equity-based optimization model to solve the location problem for healthcare centers applied to hospital beds and covid-19 vaccination. Mathematics , 10(11):1825, 2022

    2022

  2. [10]

    KTO: model alignment as prospect theoretic optimization

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: model alignment as prospect theoretic optimization. CoRR , abs/2402.01306, 2024

    2024 arXiv

  3. [11]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and et al Xiao Bi. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR , abs/2501.12948, 2025

    2025 arXiv

  4. [12]

    2023 state of mathematical optimization report, 2023

    Gurobi Optimization . 2023 state of mathematical optimization report, 2023

    2023

  5. [13]

    CAFA : Coding as auto-formulation can boost large language models in solving linear programming problem

    haoxuan deng, Bohao Zheng, Yirui Jiang, and Trung Hieu Tran. CAFA : Coding as auto-formulation can boost large language models in solving linear programming problem. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24 , 2024

    2024

  6. [14]

    Mamo: a mathematical modeling benchmark with solvers, 2024

    Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang. Mamo: a mathematical modeling benchmark with solvers, 2024

    2024

  7. [15]

    LLMOPT: learning to define and solve general optimization problems from scratch

    Caigao Jiang, Xiang Shu, Hong Qian, Xingyu Lu, Jun Zhou, Aimin Zhou, and Yang Yu. LLMOPT: learning to define and solve general optimization problems from scratch. CoRR , abs/2410.13213, 2024

    2024 arXiv

  8. [16]

    City-leo: Toward transparent city management using LLM with end-to-end optimization

    Zihao Jiao, Mengyi Sha, Haoyu Zhang, Xinyu Jiang, and Wei Qi. City-leo: Toward transparent city management using LLM with end-to-end optimization. CoRR , abs/2406.10958, 2024

    2024 arXiv

  9. [17]

    Kao, Maryam Fazel - Zarandi, and Yuandong Tian

    Da JU, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine T. Kao, Maryam Fazel - Zarandi, and Yuandong Tian. To the globe (TTG): towards language-driven guaranteed travel planning. CoRR , abs/2410.16456, 2024

    2024 arXiv

  10. [18]

    Large language models for supply chain optimization

    Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language models for supply chain optimization. CoRR , abs/2307.03875, 2023

    2023 arXiv

  11. [19]

    Llm-based frameworks for power engineering from routine to novel tasks, 2023

    Ran Li, Chuanqing Pu, Junyi Tao, Canbing Li, Feilong Fan, Yue Xiang, and Sijie Chen. Llm-based frameworks for power engineering from routine to novel tasks, 2023

    2023

  12. [20]

    Towards foundation models for mixed integer linear programming

    Sirui Li, Janardhan Kulkarni, Ishai Menache, Cathy Wu, and Beibin Li. Towards foundation models for mixed integer linear programming. CoRR , abs/2410.08288, 2024

    2024 arXiv

  13. [21]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/, 2024. [Accessed 19-09-2024]

    2024

  14. [22]

    Empowering language models with active inquiry for deeper understanding, 2024

    Jing-Cheng Pang, Heng-Bo Fan, Pengyuan Wang, Jia-Hao Xiao, Nan Tang, Si-Hang Yang, Chengxing Jia, Sheng-Jun Huang, and Yang Yu. Empowering language models with active inquiry for deeper understanding, 2024

    2024

  15. [23]

    WIQOR : A dataset for what-if analysis of operations research problems, 2025

    Aditya Parashar, Natalia Kosilova, and Ari Kobren. WIQOR : A dataset for what-if analysis of operations research problems, 2025

    2025

  16. [24]

    Scaling large-language-model-based multi-agent collaboration, 2024

    Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large-language-model-based multi-agent collaboration, 2024

    2024

  17. [25]

    Rindranirina Ramamonjison, Timothy T. L. Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghaddar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi - Dehkordi, Zirui Zhou, and Yong Zhang. Nl4opt competition: Formulating optimization problems based on their natural language de...

    2023 arXiv

  18. [26]

    Operational Research Literature as a Use Case for the Open Research Knowledge Graph , page 327–334

    Mila Runnwerth, Markus Stocker, and Sören Auer. Operational Research Literature as a Use Case for the Open Research Knowledge Graph , page 327–334. Springer International Publishing, 2020

    2020

  19. [27]

    To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning, 2024

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning, 2024

    2024

  20. [28]

    ORLM: training large language models for optimization modeling

    Zhengyang Tang, Chenyu Huang, Xin Zheng, Shixi Hu, Zizhuo Wang, Dongdong Ge, and Benyou Wang. ORLM: training large language models for optimization modeling. CoRR , abs/2405.17743, 2024

    2024 arXiv

  21. [29]

    Leveraging large language models for solving rare MIP challenges

    Teng Wang, Wing - Yin Yu, Ruifeng She, Wenhan Yang, Taijie Chen, and Jianping Zhang. Leveraging large language models for solving rare MIP challenges. CoRR , abs/2409.04464, 2024

    2024 arXiv

  22. [30]

    Optibench: Benchmarking large language models in optimization modeling with equivalence-detection evaluation, 2024

    Zhuohan Wang, Ziwei Zhu, Yizhou Han, Yufeng Lin, Zhihang Lin, Ruoyu Sun, and Tian Ding. Optibench: Benchmarking large language models in optimization modeling with equivalence-detection evaluation, 2024

    2024

  23. [31]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. 2022

    2022

  24. [32]

    Evo-step: Evolutionary generation and stepwise validation for optimizing LLM s in OR , 2025

    Yang Wu, Yifan Zhang, Yurong Wu, Yuran Wang, Junkai Zhang, and Jian Cheng. Evo-step: Evolutionary generation and stepwise validation for optimizing LLM s in OR , 2025

    2025

  25. [33]

    A deep reinforcement learning agent for geometry online tutoring

    Ziyang Xiao and Dongxiang Zhang. A deep reinforcement learning agent for geometry online tutoring. Knowl. Inf. Syst. , 65(4):1611--1625, 2023

    2023

  26. [34]

    Enhancing LLM reasoning via vision-augmented prompting

    Ziyang Xiao, Dongxiang Zhang, Xiongwei Han, Xiaojin Fu, Wing Yin Yu, Tao Zhong, Sai Wu, Yuan Wang, Jianwei Yin, and Gang Chen. Enhancing LLM reasoning via vision-augmented prompting. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomc...

    2024

  27. [35]

    Chain-of-experts: When llms meet complex operations research problems

    Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, and Gang Chen. Chain-of-experts: When llms meet complex operations research problems. OpenReview.net, 2024

    2024

  28. [36]

    Towards human-aligned evaluation for linear programming word problems

    Linzi Xing, Xinglu Wang, Yuxi Feng, Zhenan Fan, Jing Xiong, Zhijiang Guo, Xiaojin Fu, Rindra Ramamonjison, Mahdi Mostajabdaveh, Xiongwei Han, Zirui Zhou, and Yong Zhang. Towards human-aligned evaluation for linear programming word problems. In LREC/COLING , pages 16550--16556, 2024

    2024

  29. [37]

    Optibench meets resocratic: Measure and improve LLM s for optimization modeling

    Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang. Optibench meets resocratic: Measure and improve LLM s for optimization modeling. In The Thirteenth International Conference on Learning Representa...

    2025

  30. [38]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. 2023

    2023

  31. [39]

    Neural TSP solver with progressive distillation

    Dongxiang Zhang, Ziyang Xiao, Yuan Wang, Mingli Song, and Gang Chen. Neural TSP solver with progressive distillation. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on...

    2023

  32. [40]

    Solving general natural-language-description optimization problems with large language models

    Jihai Zhang, Wei Wang, Siyan Guo, Li Wang, Fangquan Lin, Cheng Yang, and Wotao Yin. Solving general natural-language-description optimization problems with large language models. pages 483--490. Association for Computational Linguistics, 2024

    2024

  33. [41]

    Chain-of-knowledge: Integrating knowledge reasoning into large language models by learning from knowledge graphs, 2024

    Yifei Zhang, Xintao Wang, Jiaqing Liang, Sirui Xia, Lida Chen, and Yanghua Xiao. Chain-of-knowledge: Integrating knowledge reasoning into large language models by learning from knowledge graphs, 2024

    2024

  34. [42]

    Decision information meets large language models: The future of explainable operations research

    Yansen Zhang, Qingcan Kang, Wing Yin YU, HaileiGong, Xiaojin Fu, Xiongwei Han, Tao Zhong, and Chen Ma. Decision information meets large language models: The future of explainable operations research. In The Thirteenth International Conference on Learning Representations , 2025

    2025

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.