pith. sign in

arxiv: 2605.21751 · v1 · pith:KR7RUNQNnew · submitted 2026-05-20 · 💻 cs.LG

Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization

Zhiqi Gao , Albert Ge , Alexander Berenbeim , Nathaniel D. Bastian , Frederic Sala This is my paper

Pith reviewed 2026-05-22 09:34 UTC · model grok-4.3

classification 💻 cs.LG
keywords text-to-optimizationbinding limitlarge language modelsoptimization modelingstructured groundingText2Opt-Benchinference-time methodsmodel specialization
0
0 comments X

The pith

Text-to-optimization models can choose the right mathematical structure but fail to ground concrete data values as instance size grows, and externalizing data to files largely removes the error.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper separates text-to-optimization into two distinct skills: selecting the optimization formulation and correctly assigning every coefficient, index, and parameter from the given instance data. Experiments across a solver-verified benchmark of twelve problem categories show that accuracy falls sharply once the number of variables and data points increases, even when the underlying structure stays simple. The authors trace the drop to binding failures rather than formulation mistakes and introduce BIND, an inference-time technique that stores numeric data in external structured files. Models then reference the data programmatically instead of transcribing values from the prompt. This change raises accuracy for a small model from 59.1 percent to 82.4 percent and allows a 1.5-billion-parameter model trained only on binding to match a 7-billion-parameter model trained end-to-end.

Core claim

Current models can select appropriate optimization formulations but cannot reliably ground parameters, coefficients, and indices when problem instances contain many data points. This effective binding limit appears consistently across textbook linear programs through stochastic and multi-objective formulations in Text2Opt-Bench. BIND externalizes numeric data to structured files so the model performs binding through code rather than prompt transcription. Finetuning a model exclusively on binding tasks produces specialists that outperform both end-to-end supervised fine-tuning and reinforcement learning, with a 1.5B binding specialist matching a 7B end-to-end baseline.

What carries the argument

The BIND inference-time method, which stores numeric instance data in external structured files so the model binds values programmatically instead of transcribing from the prompt.

If this is right

  • Accuracy on text-to-optimization improves without model retraining when data is moved outside the prompt.
  • Models trained only on binding can match or exceed larger models trained on complete end-to-end tasks.
  • The performance gap between models widens with instance size because of binding rather than formulation errors.
  • Solver-verified benchmarks can separate grounding failures from other sources of error in optimization modeling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Tasks that require grounding large amounts of instance data, such as scheduling or resource allocation, may benefit from the same separation of binding from core reasoning.
  • Future model architectures could include native access to external structured data sources during inference instead of relying on prompt content alone.
  • Training objectives that focus narrowly on data grounding may prove more parameter-efficient than broad end-to-end training on full optimization problems.

Load-bearing premise

The Text2Opt-Bench problems together with solver verification isolate binding difficulty without introducing generation artifacts or other confounds that affect measured performance.

What would settle it

Measure whether accuracy still drops with increasing numbers of variables and constraints on a new collection of optimization problems that have been independently verified by solvers and presented without changes to the data format.

Figures

Figures reproduced from arXiv: 2605.21751 by Albert Ge, Alexander Berenbeim, Frederic Sala, Nathaniel D. Bastian, Zhiqi Gao.

Figure 1
Figure 1. Figure 1: Solution accuracy vs. combined token cost across three model families (550 template problems). BIND significantly improves pass@1 accuracy, and remains competitive with other test-time-compute strategies while using significantly fewer tokens. We compare against oracle feedback, representing an upper bound on iterative refinement, and pass@5 as an upper bound on parallel sampling. that evaluation failures … view at source ↗
Figure 2
Figure 2. Figure 2: Modeling vs. binding on a resource allocation instance. Modeling selects the optimization structure (objective type, variable domains, constraints); binding extracts every numerical coefficient from prose. As instances scale, binding becomes the dominant failure mode. RULER (Hsieh et al., 2024) measures retrieval degradation using controlled tasks. Our experiments (§4.2) show that this retrieval degradatio… view at source ↗
Figure 3
Figure 3. Figure 3: Text2Opt-Bench generation pipeline. Problems are constructed via forward engineering with solver verification, then described in natural language. Template-based insertion decouples linguistic complexity from data scale. representation. Regardless of approach, these capabilities scale differently. Modeling difficulty depends on the structural complexity of the problem and is independent of instance scale. … view at source ↗
Figure 4
Figure 4. Figure 4: (a) Failure composition by model scale on resource allocation (1,012 problems). As model size grows, binding errors increasingly make up a significant proportion of failures. (b) Each model exhibits an effective binding limit beyond which accuracy sharply declines. Curves are smoothed with a Gaussian-weighted rolling average. (e.g. cost matrices) to a JSON file. The model receives: (1) the structural probl… view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy on four RULER binding tasks across Qwen-2.5 sizes (0.5B–32B). Strict exact-match scoring; 200 samples per task per context length. Multi-binding tasks exhibit sharp cliffs as individual retrieval failures compound multiplicatively. Appendix E). We establish two upper bounds: pass@5 and iterative repair with oracle feedback (a verifier with ground-truth objective and model structure provides diagno… view at source ↗
Figure 6
Figure 6. Figure 6: Accuracy heatmaps by problem size (number of variables vs. constraints) for each training approach on resource allocation (248 eval problems). The red line marks the maximum number of variables seen during training; problems above it are out-of-distribution. Yellow = 100% accuracy, purple = 0%. it, indicating that none of the training regimes generalize binding to larger problem sizes. The 7B binding speci… view at source ↗
read the original abstract

Text-to-optimization requires two separable capabilities: modeling -- choosing the right optimization structure -- and binding -- grounding every coefficient, index, and parameter in the concrete problem data. We study this via Text2Opt-Bench, a scalable benchmark of solver-verified optimization problems spanning 12 categories, from textbook linear programs to stochastic and multi-objective formulations with up to thousands of variables. Across 10+ models, we find that accuracy collapses as instance data grows, even when the formulation itself is simple. We call this the effective binding limit. We address this via a simple inference-time approach, BIND, which externalizes numeric data to structured files so the model binds data programmatically rather than transcribing from the prompt. BIND improves GPT-5-Nano from 59.1% to 82.4% accuracy, matching pass@5 (82.0%) at lower token cost than pass@1, and GPT-5 from 86.2% to 95.8%. Furthermore, we validate our hypothesis by finetuning a model exclusively on binding and show that it outperforms end-to-end SFT and RL across three structurally distinct optimization categories, with a 1.5B binding specialist alone matching a 7B end-to-end baseline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that text-to-optimization requires two separable capabilities: modeling the optimization structure and binding concrete coefficients, indices, and parameters from instance data. Using the new Text2Opt-Bench benchmark of solver-verified problems across 12 categories (from simple LPs to stochastic and multi-objective with thousands of variables), the authors show that model accuracy collapses as instance data grows even for simple formulations, which they term the effective binding limit. They propose BIND, an inference-time method that externalizes numeric data to structured files so models bind programmatically rather than transcribe from the prompt; this raises GPT-5-Nano accuracy from 59.1% to 82.4% (matching pass@5 at lower cost) and GPT-5 from 86.2% to 95.8%. They further validate by fine-tuning a 1.5B binding specialist that matches a 7B end-to-end baseline across three categories.

Significance. If the results hold, the work usefully isolates a practical limitation in current LLMs for structured grounding tasks and supplies both a scalable solver-verified benchmark and a low-cost mitigation (BIND) that yields substantial gains. The finding that a small binding specialist can match much larger end-to-end models is a concrete strength, as is the direct empirical comparison on held-out instances rather than self-referential derivations. These elements could inform future system design for optimization and other data-grounding applications.

major comments (2)
  1. [§4] §4 (Text2Opt-Bench scaling experiments): the central claim that accuracy collapse reflects an effective binding limit rather than prompt-length or numeric-density effects would be strengthened by explicit controls that hold formulation fixed while varying only data embedding style (e.g., repeated coefficients vs. summarized data vs. BIND external files); without such isolation the skeptic concern that observed BIND gains partly reflect context-length relief remains open.
  2. [Results tables] Results tables (GPT-5-Nano and GPT-5 rows): the reported jumps (59.1% → 82.4%, 86.2% → 95.8%) are presented without variance, run counts, or error bars, which reduces confidence that the BIND improvement is robust rather than sensitive to particular instance sampling or verification details.
minor comments (1)
  1. [Abstract] Abstract: the phrase '10+ models' should list the exact models and sizes evaluated to support reproducibility claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and have revised the manuscript to incorporate the suggested improvements.

read point-by-point responses

  1. Referee: [§4] §4 (Text2Opt-Bench scaling experiments): the central claim that accuracy collapse reflects an effective binding limit rather than prompt-length or numeric-density effects would be strengthened by explicit controls that hold formulation fixed while varying only data embedding style (e.g., repeated coefficients vs. summarized data vs. BIND external files); without such isolation the skeptic concern that observed BIND gains partly reflect context-length relief remains open.

    Authors: We agree that stronger isolation of the binding effect from context-length and numeric-density factors would improve the central claim. In the revised manuscript we have added a controlled ablation in §4 that holds the optimization formulation fixed while varying only the data embedding style across four conditions: (1) full numeric coefficients embedded in the prompt, (2) summarized data, (3) repeated coefficient values, and (4) BIND external structured files. The new results show that accuracy still collapses under summarized and repeated styles but recovers specifically under BIND, indicating the gains are not attributable to context-length relief alone. These comparisons are reported in a new Table 4 with accompanying analysis. revision: yes

  2. Referee: [Results tables] Results tables (GPT-5-Nano and GPT-5 rows): the reported jumps (59.1% → 82.4%, 86.2% → 95.8%) are presented without variance, run counts, or error bars, which reduces confidence that the BIND improvement is robust rather than sensitive to particular instance sampling or verification details.

    Authors: We appreciate this observation on statistical reporting. The revised results tables and figures now report standard deviations computed over five independent runs (different random seeds for instance sampling and verification), explicitly state the number of held-out instances per category (N=200), and include error bars on all plots. The BIND accuracy improvements remain consistent and statistically significant (paired t-test, p<0.01) across runs. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical results on held-out benchmark instances

full rationale

The paper reports direct empirical measurements of model accuracy on Text2Opt-Bench problems whose correctness is verified by external solvers. The observed accuracy collapse with growing instance size and the gains from BIND (externalizing numeric data) are obtained by running models on held-out instances and comparing pass rates; these quantities are not derived from any fitted parameter, self-referential definition, or self-citation chain. No equations, ansatzes, or uniqueness theorems appear in the provided text, so none of the enumerated circularity patterns apply. The central claim therefore remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Work is empirical and relies on standard assumptions about LLM prompting and solver correctness rather than new theoretical constructs or fitted constants.

axioms (1)
  • domain assumption Solver verification correctly identifies valid optimization formulations without systematic generation bias
    Invoked to establish ground truth for the benchmark problems across 12 categories.

pith-pipeline@v0.9.0 · 5768 in / 1309 out tokens · 72934 ms · 2026-05-22T09:34:19.080156+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, and Gang Chen

    URLhttps://openreview.net/forum?id=KD9F5Ap878. Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, and Gang Chen. Chain-of-experts: When LLMs meet complex operations research problems. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openrev...

    work page 2024

  2. [2]

    URLhttps://arxiv.org/abs/2508.10047. Alex L. Zhang, Tim Kraska, and Omar Khattab. Recursive language models, 2026. URLhttps://arxiv. org/abs/2512.24601. Bowen Zhang, Pengcheng Luo, Genke Yang, Boon-Hee Soong, and Chau Yuen. Or-llm-agent: Automating modeling and solving of operations research optimization problems with reasoning llm, 2025. URL https://arxi...

    work page arXiv 2026

  3. [3]

    15 Models Can Model, But Can’t Bind: Structured Grounding in Text-to-Optimization 2.Anchor Solution (xanchor):We sample a feasible solutionxanchor ≥0

    Matrix Construction (A):We initialize A∈R m×n with random values and apply a sparsity mask to simulate real-world interactions. 15 Models Can Model, But Can’t Bind: Structured Grounding in Text-to-Optimization 2.Anchor Solution (xanchor):We sample a feasible solutionxanchor ≥0

    work page

  4. [4]

    The structured representation is then passed to an LLM (GPT-5) with a prompt, and all numerical coefficients fromA,b, andcare put into a text description

    RHSDerivation( b):Thevector bisderivedvia bi =(Ax anchor)i +s i, ensuringfeasibilitybyconstruction. The structured representation is then passed to an LLM (GPT-5) with a prompt, and all numerical coefficients fromA,b, andcare put into a text description. Example of this process is in § A.3. Algorithm 2Direct Translation Dataset Generation 1:Input:Dimensio...

    work page

  5. [5]

    For example, when generating facility location problems: •Coordinates forNfacilities andMcustomers

    Structured Parameter Generation.Instead of a generic matrixA, we generate domain-specific parame- ters. For example, when generating facility location problems: •Coordinates forNfacilities andMcustomers. •Fixed costsf i, capacitiess i, demandsd j, and transport ratesr. The transport cost matrix is not provided directly; the model must compute it from coor...

    work page

  6. [6]

    business memo

    Template Generation via LLM.The LLM generates a template “ business memo” describing the logic of the problem butexcludingnumerical data. Placeholders such as{CUSTOMER_DEMANDS} are forced to be included

    work page

  7. [7]

    Deterministic Data Insertion.The pipeline programmatically replaces placeholders with formatted generated data, decoupling linguistic complexity from numerical complexity. A.3. Data Embedding Example Example: Data Embedding Transformation We sample a specific variable and constraint to demonstrate the mapping from structured parameters to natural language...

    work page

  8. [8]

    On-Site Retrofit Packages: Each completed package adds6.86in contribution. Each package uses0.8 units from our environmental emissions allowance

    Variable Embedding Input (Structured): •Var_1: TypeInteger, Obj Coeffc 1 =6.86 •Interaction: Consumes0.8of ResourceC 0 Output (Narrative): “On-Site Retrofit Packages: Each completed package adds6.86in contribution. Each package uses0.8 units from our environmental emissions allowance...”

    work page

  9. [9]

    Environmental Emissions Allowance: Total available is8.25allowance units and cannot be exceeded

    Constraint Embedding Input (Structured): •Constraint C0: Sense≤, RHSb 0 =8.25 Output (Narrative): “Environmental Emissions Allowance: Total available is8.25allowance units and cannot be exceeded.” A.4. Note on GPT-5 Contamination GPT-5 is used both to generate benchmark instances and as an evaluated model, raising a potential self- contaminationconcern. F...

    work page 2024

  10. [10]

    source_capacity_{i}

    with greedy decoding (temperature=0 ) and a maximum generation length of 128 tokens. We use strict exact-matchscoring for all tasks with no partial credit. This all-or-nothing criterion is motivated by optimization evaluation, where a single incorrect parameter yields an incorrect result. We did not include closed-source frontier models because these task...

    work page

  11. [11]

    Phase 1 (Binding):A fine-tuned model extracts all decision variables, constraints, and objective function parameters from the natural language problem description into structured JSON

    work page

  12. [12]

    This decomposition isolatesbinding—the mapping from unstructured text to structured mathematical parameters—as the sole task requiring learned reasoning

    Phase 2 (Solve):A deterministic template loads the extracted JSON and constructs a Gurobi optimiza- tion model programmatically—no LLM is needed. This decomposition isolatesbinding—the mapping from unstructured text to structured mathematical parameters—as the sole task requiring learned reasoning. 23 Models Can Model, But Can’t Bind: Structured Grounding...

    work page 2024

  13. [13]

    Solver status(+0.10 ifoptimal; +0.05 if feasible but not optimal): the Gurobi solver reaches a meaningful termination status

    work page

  14. [14]

    25 Models Can Model, But Can’t Bind: Structured Grounding in Text-to-Optimization

    Variable-count match(+0.05): the number of decision variables in the generated model equals the reference. 25 Models Can Model, But Can’t Bind: Structured Grounding in Text-to-Optimization

    work page

  15. [15]

    Constraint satisfaction(+0.20, continuous): the fraction of reference constraints satisfied by the generated solution, evaluated by substituting generated variable values into the ground-truth constraint matrix

    work page

  16. [16]

    An exact solution (objective and all variable values within10−4 of the reference) overrides the partial score and receivesr=1.0

    Objectivecloseness(+0.20, continuous): exp(−α⋅rel_gap) whererel_gap=∣z gen −z ∗∣/(∣z∗∣+10 −6) andα=10, awarding near-full credit for small deviations and decaying smoothly for larger gaps. An exact solution (objective and all variable values within10−4 of the reference) overrides the partial score and receivesr=1.0 . All other hyperparameters (Table 10) r...

    work page