{"paper":{"title":"OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Solver-integrated LLMs for optimization modeling are limited primarily by errors in automated constraint formulation as problem complexity scales.","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Cheng cheng, Dongdong Ge, Yinan Sun, Yitian Chen, Zi Ling","submitted_at":"2026-01-09T09:22:33Z","abstract_excerpt":"We investigate the capabilities and scalability of Large Language Models (LLMs) in optimization modeling, a domain requiring structured reasoning and precise formulation. To this end, we introduce OPT-ENGINE, an extensible benchmark framework with quantifiable and controllable complexity. OPT-ENGINE spans ten canonical Operations Research problems, systematically scaling from Linear Programming to Mixed-Integer Programming, providing a structured environment to probe the limits of automated problem formulation and solving. Utilizing OPT-Engine, we address three pivotal research questions. Firs"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"For the current SOTA paradigm, Solver-integrated Reasoning (SIR), the automated formulation of constraints represents the primary bottleneck.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that the ten canonical problems and the chosen complexity scaling metrics (variables, constraints, integrality) sufficiently represent the space of real-world optimization modeling tasks that LLMs would encounter.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"OPT-Engine shows pure-text chain-of-thought reasoning in LLMs loses robustness as optimization complexity grows, external tools fix only local arithmetic, and solver-integrated methods are bottlenecked by automated constraint formulation.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Solver-integrated LLMs for optimization modeling are limited primarily by errors in automated constraint formulation as problem complexity scales.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"9c3fc2f0ceb5095e2a819ba56d33047d581664a6678ab39e105e890684cb7833"},"source":{"id":"2601.19924","kind":"arxiv","version":2},"verdict":{"id":"a9bcce02-5f71-486b-a47b-54af35ebb9b0","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T16:15:21.254376Z","strongest_claim":"For the current SOTA paradigm, Solver-integrated Reasoning (SIR), the automated formulation of constraints represents the primary bottleneck.","one_line_summary":"OPT-Engine shows pure-text chain-of-thought reasoning in LLMs loses robustness as optimization complexity grows, external tools fix only local arithmetic, and solver-integrated methods are bottlenecked by automated constraint formulation.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that the ten canonical problems and the chosen complexity scaling metrics (variables, constraints, integrality) sufficiently represent the space of real-world optimization modeling tasks that LLMs would encounter.","pith_extraction_headline":"Solver-integrated LLMs for optimization modeling are limited primarily by errors in automated constraint formulation as problem complexity scales."},"references":{"count":43,"sample":[{"doi":"","year":2023,"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","ref_index":1,"cited_arxiv_id":"2303.08774","is_internal_anchor":true},{"doi":"","year":2023,"title":"Gemini: A Family of Highly Capable Multimodal Models","work_id":"83f7c85b-3f11-450f-ac0c-64d9745220b2","ref_index":2,"cited_arxiv_id":"2312.11805","is_internal_anchor":true},{"doi":"","year":2024,"title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","work_id":"80e3e977-f1bb-4c83-8d0c-1ab0a0c5c3f1","ref_index":3,"cited_arxiv_id":"2403.05530","is_internal_anchor":true},{"doi":"","year":2024,"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","ref_index":4,"cited_arxiv_id":"2412.19437","is_internal_anchor":true},{"doi":"","year":2025,"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","ref_index":5,"cited_arxiv_id":"2501.12948","is_internal_anchor":true}],"resolved_work":43,"snapshot_sha256":"2ce58239370396c817464bfce08651c77a096182468c96fcae0b0da971e819be","internal_anchors":12},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}