{"id":"74b63277-494d-4f89-97bc-f192c0f9d205","arxiv_id":"2608.13333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LLM-generated graph generators let one generic large neighborhood search and one cross-problem algorithm selector handle 20 diverse MiniZinc optimization problems.","lead":"The authors use a large language model to read MiniZinc optimization models and generate a Python program that turns each problem instance into a shared weighted graph, then run a generic local-improvement search on that graph. On 20 MiniZinc competition benchmarks they report a 39.5% average problem-weighted win rate against a one-shot Gurobi baseline, rising to 44.0% after ablations, more than doubling the best fixed configuration.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing random/unweighted SLIM baseline leaves untested whether LLM semantic weights, not generic LNS, drive the reported win rates.","rationale":"The reader's weakest assumption concerns LLM misreading semantics; my concern is broader: even a correct LLM's weights are not demonstrated to matter. The experiments show SLIM+selection beats one-shot Gurobi, but that could be due to the well-known power of LNS. The paper frames the contribution as automated semantic structure extraction, so a random baseline is essential. I agree with the CONDITIONAL verdict because the pipeline is concrete and reproducible in principle, but the missing control is the single most direct test of the central claim. Adding it could either strengthen or refute the paper; until then the causal claim is unverified.","tokens_in":104,"tokens_out":2840,"duration_ms":38645,"concrete_test":"Run SLIM on all 20 problem instances with three graph variants: (a) the LLM-generated semantic graph, (b) an unweighted graph (all node and edge weights set to 1, same topology), and (c) a purely syntactic constraint-incidence graph with no LLM weights. Keep all 30 configurations, budgets, timeouts, seeds, and algorithm-selection training and evaluation identical. Report problem-weighted win rates against one-shot Gurobi for each variant, with per-seed spread. If (b) or (c) matches (a) within a few points, the semantic weights are not the source of the gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is the LLM's semantic graph compilation, but no experiment isolates semantic weights from the generic LNS machinery. Section 4.2 explicitly notes that random extraction simply becomes standard LNS, yet no random or unweighted-graph results are reported. The reported 39.5-44.0% win rates compare against one-shot Gurobi, not against random-selection SLIM. Since the algorithm selector also trains on features extracted from the same semantic graph, if weights carry no signal the features are at best neutral and at worst confounded. Thus the load-bearing claim of Section 3.2—that graph weights encode constraint semantics that guide variable selection—is never causally tested. The paper's own limitation statement concedes the generators remain 'approximations of the true constraint semantics'; without a control that removes the weights, any success could be attributed to the LNS structure (subsolver + neighborhood budget) alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated pipeline in which an LLM, prompted with human-written semantic guidelines, generates a Python graph generator for each MiniZinc problem type. The generator maps any instance of that type to a uniform weighted graph: nodes are decision variables with objective-importance weights and domain sizes, and edges are constraint relations with coupling strengths. A generic structure-based local improvement framework (SLIM) uses this graph for variable selection via BFS-based and LNS-based extraction, and the same graph format supplies 54 features for cross-problem algorithm configuration selection among 30 SLIM configurations. The authors evaluate on 20 MiniZinc competition problem types satisfying an instance filter, reporting that the best single configuration achieves a 19.3% average problem-weighted win rate against a one-shot Gurobi baseline, algorithm selection achieves 39.5%, and configuration/feature ablation boosts this to 44.0%.","tokens_in":10429,"tokens_out":5218,"duration_ms":52200,"significance":"If the central causal claim is supported, this is a significant contribution: it would replace months of problem-specific SLIM engineering with a one-time LLM compilation step and would enable portfolio-style algorithm selection across heterogeneous problem types under a single feature representation. The paper has clear strengths: a held-out evaluation split stratified by problem, problem-weighted metrics that prevent the RCPSP majority from dominating, multiple random seeds, and a Zenodo artifact for reproducibility. The approach also produces deterministic, auditable generators rather than treating the LLM as a runtime oracle. However, the significance depends on whether the LLM-generated semantic weights are actually responsible for the reported gains, and the current experiments do not isolate that mechanism.","major_comments":[{"comment":"Section 4.2 explicitly states that SLIM with Gurobi as the subsolver 'becomes standard LNS when the variable extraction is random and does not utilize the uniform weights,' yet no random-selection or unweighted-graph condition is reported in Figure 2, Table 1, or the supplementary ablation. Because the reported win rates are compared only against one-shot Gurobi, they do not isolate the contribution of the LLM-assigned semantic weights; the gains could be produced entirely by the LNS loop, the subsolver choice, or the budget structure. Please add controls that remove the semantic signal—for example, random variable extraction, uniform (constant) node and edge weights, and a syntactic variable–constraint graph without LLM weights—run with the same 30-configuration grid, and report their per-problem and problem-weighted win rates as well as how the algorithm selector behaves on these controls.","section":"Section 4.2 and Section 3.3.1"},{"comment":"The instance filter in Section 4.1 requires that Gurobi finds a feasible solution within 10 minutes but does not prove optimality within 60 minutes, and that at least five instances qualify per problem. These criteria select exactly the instances where the one-shot baseline is weakest and exclude entire problem types, so the abstract's claim that the pipeline is 'problem-agnostic to all problems in the MiniZinc format' is not supported by the evaluation. Please either restrict the generalization claim to the 20 selected problem types and their qualifying instances, or report results over a larger, less filtered benchmark together with a sensitivity analysis of the filter thresholds.","section":"Section 4.1 and Abstract"},{"comment":"The paper reports 39.5% as the algorithm-selection win rate and 44.0% after configuration and feature ablation, but Section 4.3 states that 'for the best-performing approach on each seed' the win rates are 37.9–40.6%, and the 44.0% comes from greedy backward elimination over configurations and features described in Section 3.4 and the supplementary material. Selecting the best approach per seed and then eliminating configurations/features using cross-validation before reporting a test-set win rate is a repeated model-selection procedure that can inflate the reported average; Table 1 also lacks standard errors or confidence intervals. Please provide nested cross-validation estimates, report per-seed standard errors, and give a precise statement of how the 39.5% and 44.0% headline numbers are aggregated across seeds, problems, and ablation steps.","section":"Section 4.3 and Section 3.4"},{"comment":"Section 5 concedes that the generated graphs 'remain LLM-produced approximations of the true constraint semantics,' but the paper does not describe how generators are validated or how faithful the graphs are to the original MiniZinc models. Since all downstream extraction and feature computation in Sections 3.3 and 3.4 depends on the correctness of node and edge weights, please add a per-problem validation protocol (for example, checks against the model text, statistics on missed constraints or objective terms, and a sensitivity analysis over the semantic guidelines) and report the resulting error rates.","section":"Section 5 and Section 3.2"}],"minor_comments":[{"comment":"Figure 2 says the results are 'averaged across 3 seeds,' but no per-seed spread is shown; please add seed-level error bars or a table with the per-seed values.","section":"Section 4.2, Figure 2"},{"comment":"The edge aggregation formula W=1−∏(1−w_i) is given without an equation number or a definition of the index set; please number it and state explicitly that the product runs over all constraints linking the same pair of variables.","section":"Section 3.2.2"},{"comment":"The budget b is described only as 'a threshold bounded by a domain-size-aware metric'; please give the exact formula or point to a numbered equation in the supplementary material.","section":"Section 3.3.1"},{"comment":"The author block contains formatting errors, including 'V aidyanathan' with an erroneous space and a line break in the name; please fix the LaTeX source.","section":"Title page"},{"comment":"Section 1 reports the best single configuration as 19.3% while Table 1 lists 17.8, 17.7, and 22.3% for the three seeds; please state explicitly that 19.3% is the seed average and reconcile the 44.0% ablation number with the per-seed values in the supplementary material.","section":"Section 1 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The missing random/unweighted baseline is the decisive issue for the paper's central claim. The evaluation pipeline is otherwise transparent and well positioned for reproducibility, and I did not find evidence of train/test leakage in the main split, but the repeated model selection in the ablation and the per-seed selection of 'best approach' need tighter methodology before the headline numbers can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The pipeline is real and the paper reads well. The idea of using an LLM as a one-time semantic compiler that turns a MiniZinc model into a Python graph generator is new, and the move to a uniform weighted graph shared by both a generic SLIM loop and a cross-problem algorithm selector is a concrete step forward. The evaluation is transparent: held-out instances, problem-stratified splits, three seeds, and a Zenodo release. I believe the results are what they say they are.\n\nThe soft spot is exactly what the stress test flags: the semantic weights are never causally isolated. Section 4.2 admits that random extraction without the uniform weights is just standard LNS, but no random-selection or unweighted-graph baseline is reported anywhere. So the 39.5–44.0% win rates against one-shot Gurobi could be coming from the LNS machinery alone—a powerful subsolver plus a neighborhood budget—rather than from the LLM's semantic edge weights. The algorithm-selection result is less vulnerable because it compares configurations against each other, but the features are extracted from the same graph, so if the weights carry no signal the feature set is at least partially confounded. The paper's own limitation paragraph concedes the generators are 'approximations of the true constraint semantics,' which is honest but only makes the missing ablation more conspicuous.\n\nOther concerns are more minor. The instance filter (Gurobi finds a feasible solution in 10 minutes but does not prove optimality in 60) favors problems where local search has room to improve, so the absolute win rates shouldn't be read as a general claim about all MiniZinc instances. The 44.0% headline comes from a post-hoc ablation that is not properly nested or significance-tested, so it should be labeled exploratory. None of this sinks the paper, but it does mean the title-level claim about LLM-guided semantics is not yet backed by the experiments.\n\nOn citation pattern: the related work is relevant and the authors are not ignoring alternatives. The comparison against syntactic variable-constraint graphs is mentioned but not performed; adding it would address the strongest counterhypothesis. Self-citation is fine here because it is the actual SLIM lineage.\n\nWho is this for? People working on automated algorithm design, LNS, and LLM-driven optimization. It deserves a serious referee, not a desk reject. But before acceptance I would require a random-LNS baseline, an unweighted or syntactic-graph control, a documented instance-selection table, and a nested cross-validation of the ablation. The core pipeline is worth publishing; the current evidence just doesn't yet prove the semantic weights are what make it work.","headline":"A genuinely new LLM-to-SLIM pipeline with clear writing and honest evaluation, but the central semantic-weight claim lacks a random-selection control and should be tightened before acceptance.","tokens_in":10964,"tokens_out":1753,"would_cite":true,"duration_ms":20511,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM, guided by semantic rules, can turn any MiniZinc constraint model into a uniform weighted graph that lets a generic local-improvement framework beat a one-shot exact solver on 44.0% of problem-weighted…","keywords":["large neighborhood search","structure-based local improvement","LLM-guided graph generation","weighted constraint graph","MiniZinc","algorithm selection","variable selection","constraint optimization"],"falsifier":"On one problem type, compute a ground-truth variable-importance ranking by perturbing each decision variable individually and measuring the objective change, then compare it with the LLM-assigned node weights and with SLIM's selection frequency; if these do not correlate, the semantic weights are not the active mechanism.","tokens_in":10049,"feed_emoji":"🧩","tokens_out":10742,"duration_ms":101643,"temperature":0.7,"pith_summary":"Large neighborhood search normally needs problem-specific expert knowledge to decide which variables to optimize together. This paper claims that a large language model, prompted with a few semantic guidelines, can read a MiniZinc constraint model and emit a graph generator that maps every instance of that problem to one uniform weighted graph, with nodes for decision variables and edges for constraint couplings. That single graph lets a generic structure-based local-improvement framework (SLIM) select neighborhoods without knowing the problem, and lets one algorithm selector choose among 30 SLIM configurations using the same 54 graph features across all problems. On 20 MiniZinc competition problems, the reported average problem-weighted win rate against a one-shot exact solver is 39.5%, rising to 44.0% after configuration and feature ablation, versus 19.3% for the best fixed configuration. If the claim holds, it replaces hand-built per-problem neighborhood design with a one-time automated semantic compilation step.","feed_headline":"LLM-built graphs lift local search to a 44% win rate","feed_subtitle":"One weighted graph turns any MiniZinc model into structure-aware search and cross-problem configuration.","key_machinery":"The load-bearing object is the uniform weighted graph generated by the LLM's Python program. Nodes represent decision variables with a weight $w\\in[0,1]$ (objective importance) and domain size $d$; edges represent constraint relations with a coupling weight $w\\in[0,1]$, combined across parallel constraints with $W=1-\\prod_i(1-w_i)$ so no weight exceeds 1. Semantic guidelines push objective-related nodes to weights at or above 0.6, assign large global constraints the weight $\\max(0.1,1/n)$, forbid isolated nodes, and bound all weights. This object carries problem structure into two generic consumers: BFS and weighted-random variable extraction for SLIM, and 54 topology/statistics features for configuration selection.","core_discovery":"The paper's central claim is that LLM-produced graph generators, guided by semantic weighting rules, capture enough constraint structure to drive generic local improvement across heterogeneous optimization problems. Each node in the produced graph carries an importance weight in [0,1] and a domain size; each edge carries a coupling strength, aggregated by $W=1-\\prod_i(1-w_i)$ and bounded at 1, with large global constraints down-weighted to $\\max(0.1,1/n)$ to avoid clique domination. Because the representation is uniform, BFS-based and weighted-random extraction methods operate identically on every instance, and the same 54 graph features feed a cross-problem configuration selector trained with problem-weighted samples. The decisive evidence is the win-rate comparison against a one-shot Gurobi baseline: the best single SLIM configuration reaches 19.3%, algorithm selection reaches 39.5%, and ablations raise it to 44.0%.","pith_inferences":["If the central claim holds, the semantic-compilation step could generalize beyond MiniZinc: any constraint language an LLM can parse could be mapped to the same weighted-graph format, extending structure-aware large neighborhood search to satisfaction and hybrid problems.","A direct mechanism test would compare LLM-assigned node and edge weights with an empirical sensitivity ranking obtained by perturbing each variable and measuring objective change; a strong correlation would confirm the semantic weights, not just graph topology, carry the signal.","The paper's closing suggestion of adaptive weight updates during search is an untested extension: if weights change over iterations, the 54 selector features would need to be recomputed or made time-aware, turning the pipeline into a closed-loop optimizer."],"forward_implications":["A new MiniZinc problem type only needs one LLM synthesis pass over its model, typically minutes, before the same generic SLIM framework and selector can be applied to its instances.","Because the graph format is uniform, the same 54 features support algorithm selection across different problem types, so portfolio design no longer requires per-domain feature engineering.","The node and edge weights give a principled, automatically derived ranking of which variables and couplings matter, making structure-aware neighborhood selection available for any declarative optimization model.","Configuration choice matters: the paper's 19.3%-versus-44.0% gap shows that the same generic SLIM with a learned configuration is much stronger than any fixed strategy, making selection a first-class component of the method."],"supporting_citations":[{"why":"Establishes that structure-aware variable selection improves large neighborhood search, the core premise SLIM's graph-guided extraction generalizes.","marker":"(Shaw, 1998)"},{"why":"Surveys LNS variants and adaptive operators, framing the random-selection baseline the paper compares against.","marker":"(Pisinger and Ropke, 2019)"},{"why":"Introduces the structure-based local improvement framework that the paper automates into a problem-agnostic version.","marker":"(Fichte et al., 2017)"},{"why":"Defines portfolio-based algorithm selection, the paradigm the paper's cross-problem configuration selector applies.","marker":"(Xu et al., 2008)"},{"why":"Shows LLM program search can yield working algorithms, the mechanism the paper relies on to synthesize graph generators.","marker":"(Romera-Paredes et al., 2024)"}],"fun_headline_variants":["LLM graphs more than double local search win rate","Problem-agnostic LLM graphs lift local search to 44%","Automatic LLM graph generation yields 44% win rate","From 19.3% to 44%: LLM-built graphs for local search","Weighted graphs from LLM prompts reach 44% wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything downstream depends on the LLM's graph weights and edge strengths faithfully reflecting which variables and constraint relationships actually matter for improving the objective; if the generator misreads the model, the graph drives variable selection toward the wrong parts of the instance.","fun_headline_variants_meta":{"raw":{"variants":["LLM graphs more than double local search win rate","Problem-agnostic LLM graphs lift local search to 44%","Automatic LLM graph generation yields 44% win rate","From 19.3% to 44%: LLM-built graphs for local search","Weighted graphs from LLM prompts reach 44% wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2111,"prompt_tokens":947,"completion_tokens":1164,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1072}},"tokens_in":563,"tokens_out":1164,"duration_ms":10981,"temperature":1.0,"reasoning_tokens":1072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:26:44.167096+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On one problem type, compute a ground-truth variable-importance ranking by perturbing each decision variable individually and measuring the objective change, then compare it with the LLM-assigned node weights and with SLIM's selection frequency; if these do not correlate, the semantic weights are not the active mechanism.","supporting_citations":[{"cited_title":"Using constraint programming and local search methods to solve vehicle routing problems","cited_arxiv_id":null,"evidence_quote":"Establishes that structure-aware variable selection improves large neighborhood search, the core premise SLIM's graph-guided extraction generalizes."}],"review_version":1}