Pith. sign in

REVIEW 3 major objections 6 minor 25 references

LLMs can iteratively rewrite correct constraint models into faster ones when the prompt keeps behaviourally diverse past attempts and the final model is chosen on validation data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 12:44 UTC pith:4HUUNJ3W

load-bearing objection Solid empirical CP/LLM systems paper: whole-model evolutionary reformulation works on held-out CSPLib, diversity-aware retention helps, but headline GMs lean on Def. 1 and a few huge wins. the 3 major comments →

arxiv 2607.28268 v1 pith:4HUUNJ3W submitted 2026-07-30 cs.AI

LLM-Guided Evolutionary Search for Constraint Model Reformulation to Improve Solver Efficiency

Kostis Michailidis , Dimos Tsouros , Nguyen Dang , Tias Guns This is my paper
classification cs.AI
keywords constraint programminglarge language modelsmodel reformulationevolutionary searchautomatic heuristic designsolver efficiencycontext diversityvalidation selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Even a correct constraint model can be slow; modelling choices strongly affect solver time. This paper asks whether a language model can automate performance-oriented reformulation of a whole declarative model, not just append extra constraints. It runs an evolutionary loop: propose a candidate, check that its answers still satisfy the baseline on training instances, measure runtimes, and let a retention strategy decide which prior attempts and feedback enter the next prompt. On eight satisfaction benchmarks the loop often returns held-out speedups, strategies that keep the retained context diverse beat pure recency or fastest-only memory, and picking the final model on a validation set improves every strategy. A reader who builds or deploys solvers should care because modelling skill has long been the bottleneck, and this offers an automated path from a working baseline to a faster formulation without hand-written transformation catalogues.

Core claim

Iterative LLM-guided whole-model reformulation produces substantial held-out speedups over user baselines on constraint satisfaction problems. Retention strategies that keep the prompt context diverse—above all Profile-Diverse Retention, which balances total speed against distance of instance-level runtime vectors—outperform strategies that keep only recent or only fastest attempts. Validation-based final selection improves held-out speedup for every strategy, and whole-model rewrite outperforms an adapted streamliner-generation baseline that only adds constraints.

What carries the argument

Profile-Diverse Retention (PDR): keep the fastest valid attempt, then repeatedly add candidates via Maximal Marginal Relevance on log-normalised per-instance runtime profiles so the next prompt sees attempts that are both fast and behaviourally distinct.

Load-bearing premise

Agreeing with the baseline’s answers on a small stratified split of instances is treated as enough to accept a reformulation—even ones that shrink the solution set—and gains under one language model, one solver setup, and eight satisfaction problems are assumed to transfer.

What would settle it

Repeat the ten strategies on the same problems with a different language model and a different solver configuration; if held-out geometric-mean speedups fall to about 1× and diversity retention loses its edge over sampling and fastest-only retention, the central claim does not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A correct baseline model can be automatically rewritten into faster variants without hand-authored reformulation rules.
  • Prompt memory should favour behaviourally diverse successful attempts, not only the latest or the single fastest.
  • Final model choice should use a held-out validation set rather than the training incumbent alone.
  • Rewriting the whole model is a stronger efficiency lever than only appending streamlining constraints.
  • Classic modelling moves (viewpoint changes, tabulation, symmetry breaking, redundant constraints) can emerge from the search without being hardcoded.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Runtime-profile diversity for prompt retention could transfer to MIP or SAT encoding search wherever per-instance solve times are cheap to collect.
  • Problems that almost never yield valid incumbents (wide incidence matrices, costly lex orderings) suggest the loop may need propagation or conflict signals, not wall-clock totals alone.
  • Because validation is the main filter against training-only streamliner-style restrictions, deployment on industrial instance families would need stronger answer-set or equisatisfiability checks than the small stratified splits used here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper adapts Automatic Heuristic Design to performance-oriented reformulation of declarative CP models: an LLM iteratively proposes whole-model candidates that are solution-validated against a user baseline and timed on training instances; a retention strategy chooses which attempts enter the next prompt; and the final model is selected on a held-out validation set. Ten strategies are compared (sampling, recency, quality, hybrid, full history, EoH ± feedback, and the proposed Profile-Diverse Retention / PDR, which applies MMR to log-normalized per-instance runtime vectors), on eight CSPLib satisfaction problems with a stratified 8/8/16 train/val/test split, single-worker OR-Tools CP-SAT, and DeepSeek V4 Flash. The main empirical claims are that iterative reformulation yields substantial held-out speedups (aggregate GM up to ~2.26× for PDR; 144/240 runs faster on test), that diversity-aware retention outperforms recency- or fastest-only retention, that validation selection improves every strategy, and that whole-model search beats an adapted StreamLLM-single streamliner baseline.

Significance. If the results hold under clearer validity framing, this is a solid and timely contribution at the intersection of LLM-guided search and constraint modelling. Strengths include a carefully operationalized evaluator (solution-level validity, PAR2, training caps), anti-overfit validation selection (usually missing in AHD pipelines), bootstrap and leave-one-problem-out sensitivity, ablations (PDR vs Top-5, EoH ± feedback, warm-start), qualitative cataloguing of recovered modelling patterns against the CP literature, and a fairer single-model StreamLLM comparison. PDR is a concrete, reusable idea for behaviour-aware prompt retention. The work is practically relevant for reducing manual reformulation effort and advances beyond correctness-only LLM modelling benchmarks.

major comments (3)
  1. [§3 Definition 1; §6.1; Table 3; Table 2; Table 4] Definition 1 (§3) accepts any candidate whose returned answer is a baseline solution; it does not enforce equisatisfiability or solution-set preservation, even though the system prompt asks for semantic equivalence. §6.1 and Table 3 explicitly state that the largest reported win (covering arrays, 250.7× under PDR) “fixes a canonical enumeration prefix that restricts the solutions it can return, acting as a streamliner.” Aggregate GMs (Table 2: PDR 2.26×) and claim (ii) on diversity-aware retention are therefore partly carried by validity-weak, solution-restricting models that Def. 1 admits by design. Please either (a) report held-out results stratified by whether the selected model preserves the full solution set / is a pure streamliner vs a viewpoint/global-constraint reformulation, or (b) add a stricter post-hoc filter (e.g., multiple diverse baseline solutions accepted, or dual checks
  2. [Abstract; Figure 2; Table 2; Table 5; §6.2] Problem-level heterogeneity is extreme (Figure 2, Table 5): near-universal large gains on tail assignment / vessel loading / car sequencing, near-floor results on BIBD (2/30) and social golfers (9/30), and PDR’s aggregate lead driven heavily by covering arrays (including the 250.7× outlier). Leave-one-problem-out ranges in Table 2 help, but the abstract and §6.2 still present diversity strategies as generally superior. Please qualify claim (ii) with problem-conditioned analysis (e.g., GM excluding covering arrays; win rates; median-focused summary) and state clearly that strategy ranking is unstable across problems and that a few large wins dominate the headline GM.
  3. [§5; §7; Abstract] The experimental scope is one LLM, one solver configuration (single-worker CP-SAT), eight satisfaction problems, and a small stratified split (8 train instances). Limitations (§7) acknowledge this, but transfer is load-bearing for the automation claim. At minimum, add a short sensitivity discussion or appendix on whether selected models remain faster under a second search configuration (e.g., more workers) or on a modestly perturbed instance pool; if infeasible, narrow the claims to “under this solver/LLM/protocol” in the abstract and conclusion rather than implying general solver-efficiency automation.
minor comments (6)
  1. [Figure 2] Figure 2 caption and axis: geometric standard deviation intervals are useful; consider also marking the fraction of runs returning the baseline so readers can separate “no improvement” from “slowdown.”
  2. [§4.1 PDR; Eq. (5)] Eq. (5) and the profile definition: state the default when max=min on an instance more prominently in the main text (currently brief); λ=0.5 is fixed with no sensitivity—brief λ∈{0.25,0.75} note in appendix would help.
  3. [Table 3] Table 3 “Best model” column mixes strategy and patterns; a count of how many faster models are streamliner-style vs viewpoint/encoding changes would support the qualitative story.
  4. [Abstract; §1] Typos / spacing: “Combinatorialproblems”, “naturallanguage”, “toaddress this gap” and similar missing spaces appear in the abstract and early sections—copy-edit pass needed.
  5. [§6.1; Appendix F] StreamLLM-single adaptation details are thorough in Appendix F; a one-sentence pointer in §6.1 that portfolio reconstruction yields 1.87× vs PDR+base 2.41× would help readers who skip the appendix.
  6. [Figure 4; §6.2] Cost analysis (Figure 4) is valuable; clarify whether USD costs are API list prices at access date and whether cache hits are provider-reported.

Circularity Check

0 steps flagged

No significant circularity: held-out speedups and strategy comparisons are empirical measurements, not results forced by definition or self-citation.

full rationale

This is an empirical systems/AHD paper. The load-bearing claims—(i) iterative LLM reformulation yields held-out speedups, (ii) diversity-aware retention (PDR) beats recency/quality-only retention, (iii) validation selection improves every strategy—are established by measuring PAR2 runtimes of solution-valid models on stratified held-out test instances against a fixed baseline, after selection on a separate validation set (Algorithm 1, §5–§6). Training runtime is the search objective; test GM is an independent evaluation metric on unseen instances, so success is not defined by construction from the fit. Definition 1 (solution-level validity) is a methodological acceptance criterion, not a circular identity that forces speedups. Self-citations to the authors’ prior CP-Bench / solution-level evaluation work supply the validity checker and modelling setup; they do not supply uniqueness theorems or fitted constants that the present speedups reduce to. PDR’s MMR on log-normalized per-instance runtime profiles is a retention heuristic whose benefit is measured comparatively (PDR vs Top-5, etc.), not derived as a tautology. Concerns that Def. 1 admits streamliner-style solution-set restrictions (e.g. the 250.7× covering-array model) affect claim scope and correctness risk, not circularity of the derivation chain. No self-definitional step, fitted-input-as-prediction, load-bearing uniqueness import, or renamed known identity was found.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

Empirical systems paper. Load-bearing commitments are experimental protocol choices and the solution-validity surrogate for full model equivalence, not physical constants or new particles. Free parameters are search hyperparameters and the LLM/solver stack.

free parameters (4)
  • PDR λ (quality–diversity trade-off) = 0.5
    Fixed at 0.5 without sweep; directly controls which attempts enter the prompt and thus which models can be discovered.
  • Retention/population sizes k, K and iteration budget T = k=K=5, T=100
    k=5 for PDR/Top-5, K=5 for EoH, T=100 iterations; chosen by authors and shape search cost and diversity.
  • Training evaluation cap C_B = max(T_B, min(1.5 T_B, 900)) s
    Per-candidate train time bound max(T_B, min(1.5 T_B, 900))s affects which models survive as incumbents.
  • LLM and decoding defaults = DeepSeek V4 Flash, T=1.0
    DeepSeek V4 Flash, temperature/top_p at API defaults 1.0; all proposals depend on this black box.
axioms (4)
  • domain assumption Solution-level validity (candidate answers must be solutions of the baseline after fixing baseline variables) is an adequate acceptance criterion for reformulations, including those that may restrict the solution set.
    Definition 1 and §3 explicitly prefer this over equisatisfiability because full solution-set comparison is hard; the 250.7× covering-array model is accepted under this rule.
  • domain assumption Time-to-first-solution under single-worker OR-Tools CP-SAT is the right performance objective for comparing models.
    §5 fixes one solver configuration; rankings could change under other solvers, multi-thread, or optimization objectives.
  • ad hoc to paper Instance-level log-normalized runtime vectors are a meaningful behavioural diversity signal for prompt retention (PDR/MMR).
    Eqs. (4)–(5); authors note in limitations that propagations/conflicts/tree depth are unused alternatives.
  • standard math Standard CP modelling background (global constraints, viewpoints, symmetry breaking, streamliners) and AHD loop structure are valid prior art to build on.
    Preliminaries and related work; not re-proved here.
invented entities (1)
  • Profile-Diverse Retention (PDR) no independent evidence
    purpose: Select up to k valid attempts that trade total runtime quality against diversity of per-instance runtime profiles via MMR.
    Core methodological proposal; defined by normalized log-runtime profiles and the iterative argmax in Eq. (5).

pith-pipeline@v1.2.0-daily-grok45 · 28069 in / 3245 out tokens · 70217 ms · 2026-07-31T12:44:59.629988+00:00 · methodology

0 comments
read the original abstract

Combinatorial problems appear in numerous industrial applications. A common approach is to formulate these problems as declarative constraint models that can subsequently be compiled to and solved by a range of back-end solvers. Recent work shows that Large Language Models (LLMs) can produce correct models from natural language, but even a correct model can be expensive to solve because performance remains sensitive to modelling choices. In this work, we investigate whether LLMs can automate performance-oriented model reformulation. Inspired by Automatic Heuristic Design (AHD), we use an evolutionary framework in which an LLM proposes candidate reformulations that are verified and benchmarked against the user-defined baseline model. We compare AHD-adapted search strategies that control which prior attempts, instructions, and measured feedback enter each prompt. Existing retention strategies prioritize recency or performance, but do not explicitly diversify the context. To cover this gap, we introduce Profile-Diverse Retention (PDR), which applies Maximal Marginal Relevance (MMR) to instance-level runtime vectors to retain behaviourally diverse attempts. We systematically evaluate the strategies on eight CSPLib problems using validation-based final model selection. The results show that: (i) iterative reformulation can produce substantial held-out speedups; (ii) strategies that keep the retained context diverse outperform those that retain only recent or the fastest attempts; and (iii) validation-based selection improves the held-out speedup of every strategy.

Figures

Figures reproduced from arXiv: 2607.28268 by Dimos Tsouros, Kostis Michailidis, Nguyen Dang, Tias Guns.

Figure 1
Figure 1. Figure 1: Iterative model reformulation. B: baseline; Dtr, Dval, Dtest: training, validation, test sets. At iteration t ≤ T, the generator proposes model mt with rationale ρt, evaluates it on Dtr, and updates history Ht with evaluator feedback Et; strategy σ then selects context Rt+1 from Ht for the next iteration. The final model is chosen on Dval after search terminates. PDR exemplifies the search strategies compa… view at source ↗
Figure 2
Figure 2. Figure 2: Held-out test speedup by problem and strategy, ordered by each problem’s GM across all runs. Each bar is the GM [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Held-out speedup when selecting the final model [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Mean LLM cost per run against speedup. 1.30× over the quality-only Top-5, showing the benefit of retaining a diverse context, and by 1.37× over PDR warm￾start, suggesting that an initial context-free warm-start (di￾versity through randomness) is less beneficial. Additionally, for the EoH strategy we see that performance feedback is beneficial across problems: EoH+feedback improves over EoH, with the larges… view at source ↗
Figure 6
Figure 6. Figure 6: Baseline-runtime distributions and selected instances under the runtime-stratified protocol. Each panel shows the [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of BR-PAR2 test outcomes. Each marker is one search; diamonds mark geometric means and the right [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Incumbent and validation trajectories, repetition 1. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Incumbent and validation trajectories, repetition 2. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Incumbent and validation trajectories, repetition 3. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references

  1. [1]

    Keep the function signature build_model(data) -> ( ,→model, extractor) and the same input contract ,→(data keys and meaning) as the baseline

  2. [2]

    You may change decision variables or viewpoints ,→freely, but extractor() must return the exact ,→same JSON structure as the baseline (same keys ,→and value types); how you map your variables ,→to it is up to you

  3. [3]

    Preserve semantics exactly: the model must not ,→accept invalid solutions and must stay ,→satisfiable for any instance possible (even ,→out of this training set)

  4. [4]

    Prefer ‘cp.sum ,→‘, ‘cp.max‘, ‘cp.min‘, ‘cp.any‘, ‘cp.all‘ over ,→Python built-ins

    Dependencies: ‘import cpmpy as cp‘, numpy, and the ,→Python standard library only. Prefer ‘cp.sum ,→‘, ‘cp.max‘, ‘cp.min‘, ‘cp.any‘, ‘cp.all‘ over ,→Python built-ins

  5. [5]

    Integer modeling only (floats/fractional constants ,→are strictly not allowed); no printing, no ,→file I/O, no extra scripts, no solving

  6. [6]

    ## CPMpy Documentation CPMpy is a Constraint Programming and Modeling ,→library in Python, based on numpy

    Treat prior proposals as explored candidates; do ,→not repeat a previous model with only ,→superficial changes. ## CPMpy Documentation CPMpy is a Constraint Programming and Modeling ,→library in Python, based on numpy. ### Model - ‘cp.Model( *constraints)‘: create a model. - ‘model += constraint‘: add a constraint. - ‘model.minimize(expr)‘ / ‘model.maximi...

  7. [7]

    t"] k = data[

    Do not include additional sections, markdown ,→tables, or extra code blocks. Execution details not given in the main paper.Instance parametervaluesarenotincludedinprompts.Solverseed42 isfixed,andthereturned-answercheckhasa60-secondlimit. The baseline is remeasured before each search. An instance isexcludedfromthatrunifthelivebaselinetimesout,cannot beeval...

  8. [10]

    Use the baseline model, earlier conversation, ,→measured feedback, and partial reasoning as ,→context

  9. [11]

    Output exactly two top-level sections: ‘## ,→Rationale‘ and ‘## Final Model‘

  10. [12]

    Under ‘## Rationale‘, provide a concise visible ,→rationale based on the partial answer

  11. [13]

    Under ‘## Final Model‘, output exactly one fenced ,→‘python‘ code block containing a complete ,→standalone implementation

  12. [14]

    Keep the same function signature: build_model(data ,→) -> (model, extractor)

  13. [15]

    Preserve the same input contract and exact ,→extractor output schema as the baseline

  14. [17]

    ,→Understand the problem being addressed, ,→including its variables, constraints, and ,→optimization goals

    Analyze Content: Read the provided CPMpy code. ,→Understand the problem being addressed, ,→including its variables, constraints, and ,→optimization goals

  15. [18]

    ,→These should offer targeted modifications or ,→restrictions designed to reduce the search ,→space effectively

    Generate additional Constraints: Based on your ,→analysis, create five unique constraints. ,→These should offer targeted modifications or ,→restrictions designed to reduce the search ,→space effectively

  16. [19]

    streamliner_1

    Always return your constraints as a JSON object, ,→adhering to the structure: {"streamliner_1": ,→"<CPMpy constraint expression>", ..., " ,→streamliner_5": "<CPMpy constraint expression ,→>"}. Your final output should exclusively be ,→the JSON object containing the five ,→constraints

  17. [20]

    Some constraints might lead to ,→errors, timeouts, or unsatisfiable instances

    As a response, you will get feedback for each ,→constraint. Some constraints might lead to ,→errors, timeouts, or unsatisfiable instances

  18. [21]

    Use the information provided in the previous step ,→to generate five new, and hopefully better ,→constraints

  19. [22]

    Compliance Rules:

    Repeat steps 3 to 5 multiple times. Compliance Rules:

  20. [23]

    streamliner_1

    Response Format: Your final output should ,→exclusively be the JSON object containing the ,→five constraints, adhering to the structure: ,→{"streamliner_1": "<CPMpy constraint ,→expression>", ..., "streamliner_5": "<CPMpy ,→constraint expression>"}

  21. [24]

    Code Quality: All CPMpy expressions must be ,→syntactically correct and functional

  22. [25]

    CPMpy adaptation: Each value must be a single ,→constraint expression that can be added to the ,→existing model

    Creativity: You’re encouraged to be innovative in ,→proposing constraints, keeping in mind their ,→purpose: to narrow down the search space ,→efficiently without oversimplifying the ,→problem. CPMpy adaptation: Each value must be a single ,→constraint expression that can be added to the ,→existing model. For example, use an ,→expression such as ‘cp.all([c...

  23. [2005]

    InIJCAI,109–116

    Therulesofconstraintmodelling. InIJCAI,109–116. Gent, I. P.; and Walsh, T. 1999. CSPlib: A Benchmark Li- brary for Constraints. In Jaffar, J., ed.,Principles and Prac- tice of Constraint Programming – CP’99, 480–481. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-540- 48085-3. Gestrin, E.; and Seipp, J. 2026. LLM-Evolved Domain- Independent He...

    1999

  24. [2006]

    Hou, Z.; Li, X.; Zhang, Y.; Li, T.; and You, K

    Constraintmodelsforthecoveringtestproblem.Con- straints, 11(2): 199–219. Hou, Z.; Li, X.; Zhang, Y.; Li, T.; and You, K. 2026. LLM4Branch: Large Language Model for Discovering Effi- cientBranchingPoliciesofIntegerPrograms. InForty-third International Conference on Machine Learning. Ishay, A.; Yang, Z.; and Lee, J. 2023. Leveraging Large Language Models to...

    arXiv 2026

  25. [2007]

    Constraints, 12: 21–62

    Global constraint catalogue: Past, present and future. Constraints, 12: 21–62. Bradley, H.; Dai, A.; Teufel, H.; Zhang, J.; Oostermeijer, K.; Bellagente, M.; Clune, J.; Stanley, K.; Schott, G.; and Lehman, J. 2024. Quality-diversity through AI feedback. InInternational Conference on Learning Representations, volume 2024, 21036–21147. Carbonell, J. G.; and...

    2024