{"id":"756c5753-4834-4bd7-8b01-fbd5eba5f135","arxiv_id":"2505.11526","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Code retrieval from a pre-built MILP library can generate new instances similar to a target, but the paper's evidence is weakened by using the retrieval metric as the evaluation metric.","lead":"The paper proposes a system that, given a hard math optimization problem (a MILP), searches a library of problem generators and reuses the best-matching generator's code to create new similar problems. It is aimed at researchers who need large training datasets for machine-learning-based solvers, but the main evaluation shares the same similarity score used to pick the generator, so the reported gains are not convincing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported similarity gains may be a selection artifact: Section 3.4 retrieves code by MILP-EmbedSim, and the same metric is then used to measure similarity in Tables 1-2, so the library's coverage of unseen classes is never corroborated independently.","rationale":"The reader's formal weakest_assumption is library coverage, while the reader's rationale also identifies the same-metric circularity. I see the self-confirming evaluation as the more load-bearing concern because it directly undermines the headline comparisons and also weakens the evidence for coverage: Figure 7, the main coverage experiment, can be explained by the increasing maximum of a bounded score over a larger candidate pool. The library-coverage assumption matters, but it is provably untestable with the current protocol because retrieval and evaluation use the same learned similarity function. I therefore agree with the REJECT verdict: the central claim that MILP-Retrieval produces instances highly similar to unseen targets and outperforms baselines is not established by the reported experiments. Secondary issues such as GPT-4o few-shot examples being drawn from the test set, the absence of released code and data, and the omission of fixed library-construction costs all reinforce the verdict but are not the main technical defect. A clean independent-metric evaluation could rehabilitate the paper, which is why I recommend UNCHANGED rather than a stronger or weaker verdict.","tokens_in":18619,"tokens_out":6623,"duration_ms":73614,"concrete_test":"Recompute Table 1 and Table 2 using an independent, pre-registered similarity measure that is not used in retrieval: the 11-statistic Jensen-Shannon divergence employed by Geng et al. (2023) and Guo et al. (2024), computed between the generated instance set and the target instance set for FCNF, TSP, GA, VRP, NurseSched, CVS, and IIS. Include a control that executes a randomly selected library code under the same independent measure. If MILP-Retrieval's gap over baselines and over the random-code control disappears or reverses under this metric, the reported advantage is a selection artifact; if the gap survives, the circularity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is supported by a self-confirming evaluation loop. The retrieval rule in Section 3.4 chooses c_k = argmax_k MILP-EmbedSim(p, p_k), and the same MILP-EmbedSim (Eq. 3) is the similarity metric reported in Tables 1-2 and Figures 6-7. Because the retrieved code is selected to maximize exactly the quantity later reported, high similarity scores can reflect selection bias rather than genuine structural or semantic closeness between the generated instances and the target. This is not a purely formal quibble: the only evidence that the 4,000-class Evolve/Train library 'covers' unseen classes (FCNF, TSP, GA, VRP, and MIPLIB problems) is Figure 7, which shows retrieval similarity increasing as the library is enlarged. Taking the maximum over a larger candidate set mechanically increases the expected maximum of any bounded score, even if no relevant generator exists in the library. Thus the load-bearing assumption that a near-match exists for an arbitrary target has never been tested by a metric independent of the retrieval objective. If that assumption fails, MILP-Retrieval returns an unrelated code and the generated instances will not resemble the target, regardless of embedding quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reformulates MILP instance generation as a code generation task: it builds a library of MILP generation codes (plus instances, graphs, and text descriptions) using MILP-Evolve, trains a CLIP-style embedding model that aligns bipartite graph representations of MILP instances with textual descriptions, and defines a similarity metric MILP-EmbedSim as the cosine similarity between normalized embeddings. The proposed pipeline, MILP-Retrieval, retrieves the library code whose instance embedding is most similar to a target instance and executes that code to generate new instances. The authors claim that MILP-Retrieval outperforms two code-generation baselines (few-shot GPT-4o and fine-tuned LLaMA-3-8b) and a learning-based instance-generation baseline (ACM-MILP) on both code generation and instance generation tasks.","tokens_in":1479,"tokens_out":1582,"duration_ms":66060,"significance":"If validated, the reformulation of instance generation as code retrieval would be a useful and practical contribution: it offers training-free generation, interpretable code, and control over instance scale and difficulty, potentially removing the need for per-class model training. The paper also contributes a substantial library of 4,000 MILP classes and 59,033 instances, and a non-trivial embedding model. However, the empirical evidence for the central claim is currently compromised by a circular evaluation and by a test-set leakage in the baseline setup, so the significance of the results as presented is not established.","major_comments":[{"comment":"The evaluation metric is the same function used for retrieval, making the reported similarity scores largely self-confirming. Section 3.4 selects the code by k = argmax_k MILP-EmbedSim(p, p_k), and Section 4.1 states that instance similarity is assessed using the same MILP-EmbedSim. Tables 1 and 2 and Figures 6–7 then report MILP-EmbedSim as the outcome measure. Because the retrieved code is chosen to maximize exactly this quantity, high similarity values can reflect selection bias rather than genuine structural or semantic proximity between the generated instances and the target. This concern is not purely formal: the authors' only evidence that the 4,000-class library covers unseen classes (FCNF, TSP, GA, VRP, MIPLIB) is Figure 7, which shows the retrieval similarity increasing with library size; taking a maximum over a larger candidate set mechanically increases the expected maximum of any bounded score even when no relevant generator exists. The paper should be re-evaluated with an independent similarity measure, e.g., Jensen–Shannon divergence on structural statistics (as in prior work), constraint/variable type distributions, solving-time profiles, or downstream solver performance. Section 4.1 mentions computational hardness and feasible ratio, but these are not reported in the results tables.","section":"§3.4, §4.1, Eq. (3), Tables 1–2"},{"comment":"The GPT-4o baseline uses few-shot examples randomly sampled from the Evolve/Test dataset, the very dataset on which the evaluation is performed. This is test-set leakage into the baseline prompt. Although the leakage favors the baseline (because the model sees examples from the target classes), it invalidates the comparison between MILP-Retrieval and GPT-4o. The few-shot examples should be drawn from a disjoint set, such as Evolve/Train, or from held-out classes not used in evaluation.","section":"Appendix E.1"},{"comment":"The claim that the Evolve/Train library covers unseen classes is supported only by the same MILP-EmbedSim metric that is maximized during retrieval. The paper does not report whether the retrieved code for a TSP, VRP, GA, or FCNF target actually belongs to the target class, nor does it report any class-specific structural test on the generated instances. Such a sanity check is essential: if the retrieved code is from an unrelated class, the generated instances will not resemble the target no matter how high the embedding similarity is, and the library-coverage assumption is then false. The authors should report the identity or class of the retrieved code for each target, and validate equivalence through independent structural metrics.","section":"§3.1, §4.2, Figure 7"}],"minor_comments":[{"comment":"The text refers to Figure D.4 for the accuracy curves, but the figure appears in the main text as Figure 4; the reference should be corrected.","section":"§3.2, Figure 4"},{"comment":"In the description of the embedding model, 'Let xu, xu ∈ R^emb size' should presumably read 'xu, xv ∈ R^emb size'.","section":"Appendix D.1"},{"comment":"The Averaged Similarity for the baselines is computed only over classes for which valid code was generated, while MILP-Retrieval is evaluated on all 50 classes; this difference in evaluation sets should be stated explicitly and ideally a per-class comparison should be provided.","section":"§4.1, Table 1"},{"comment":"The figure reports best results from 10 trials for GPT-4o and 25 trials for fine-tuned LLaMA-3-8b, whereas MILP-Retrieval is deterministic; the best-of-k selection should be acknowledged as a favorable treatment of the baselines.","section":"§4.2, Figure 6"},{"comment":"There are minor language and typographical issues, e.g., 'As illustrate in Figure 3' should be 'As illustrated in Figure 3'.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the library construction is a significant resource, but the current empirical validation is not sound because of the circular use of MILP-EmbedSim as both the retrieval objective and the evaluation metric, and because of the test-set leakage in the GPT-4o baseline. These are fixable with additional independent evaluations, but the revision is substantial; if the authors cannot supply independent evidence of similarity and coverage, the paper would not be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this paper has a genuinely new framing — treat MILP instance generation as retrieving and executing code from a library — and the authors put together a coherent pipeline to do that. The embedding model and MILP-EmbedSim are sensible, and the idea that a single retrieval system can cover multiple problem classes without per-class training is worth taking seriously. That said, the central similarity numbers in Tables 1 and 2 are not evidence, because the retrieval objective and the evaluation metric are the same function. Section 3.4 picks code by argmax of MILP-EmbedSim, then the tables report MILP-EmbedSim as the similarity. This is selection bias, not corroboration. The ablation in Figure 7 has the same problem: taking the max over a larger library mechanically raises the expected maximum even if no relevant code exists. So the claim that the Evolve/Train library covers unseen classes like TSP, VRP, and MIPLIB problems is untested by any independent measure.\n\nThe paper does have real contributions. The task reformulation itself, the pairwise scale-insensitive similarity metric, and the contrastive training setup are all new to MILP instance generation. The code validity numbers are objective and strongly favor the retrieval pipeline — 50/50 versus GPT-4o's 2/50 is a striking gap. But the GPT-4o baseline is handicapped by using few-shot examples drawn from the same test set, which is a leak; a fair baseline would use training set examples or no examples. Also, no code or data are released, and the efficiency comparison omits the four weeks and $50 used to build the library plus the 40 hours of embedding training. Those are fixed costs any user would pay.\n\nSo my verdict: the idea is plausible and the framing is useful, but the current evidence for \"highly similar instances\" does not survive contact with the circularity. This deserves a serious referee, not a desk reject, because the question is real and a revised version with an independent similarity metric (e.g., JS divergence on structural statistics, or solver performance distribution) plus a de-contaminated GPT-4o baseline could be a solid paper. I'd send it to review with the explicit request to fix the evaluation.\n\nReading group: yes — the circular evaluation is a teaching moment. I wouldn't cite the current version in my own work.","headline":"Clever retrieval framing for MILP instance generation, but the headline similarity numbers are circular: the retrieval objective and evaluation metric are the same.","tokens_in":19417,"tokens_out":2713,"would_cite":false,"duration_ms":27970,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C11"],"pacs":[],"model":"deepseek-v4-flash","headline":"MILP instance generation can be done by retrieving executable code, not training a model per class.","keywords":["mixed-integer linear programming","MILP instance generation","code generation","retrieval-augmented generation","contrastive learning","bipartite graph embeddings","instance similarity metric"],"falsifier":"Hold out a class absent from the training library, generate many target instances at varied scales, and for each retrieve the top-1 code. If the retrieved code's true class label (not its embedding score) frequently fails to match the target class, or if same-class pairs at different scales score below cross-class pairs at the same scale on MILP-EmbedSim, the central claim is falsified.","tokens_in":18401,"feed_emoji":"🔍","tokens_out":9184,"duration_ms":87112,"temperature":0.7,"pith_summary":"The paper claims that generating new MILP instances for a target problem class can be reduced to retrieving an existing piece of executable generator code and running it, instead of training a separate generative model for each class. The authors build the claim on an observation that the space of MILP problems is compact enough to be covered by a finite library: they evolve 4,000 generator codes from eight seed classes. To find the right code, they train an embedding model that maps MILP instances and their textual descriptions into one space, then define MILP-EmbedSim as the cosine similarity between normalized instance embeddings, a metric they argue stays meaningful across instance sizes. Given one target instance, MILP-Retrieval selects the library code whose generated instance embedding is closest and executes it. If the claim holds, data generation for learning-based solvers becomes training-free, single-instance, and interpretable, because scale and difficulty are controlled by parameters in the retrieved code.","feed_headline":"Code retrieval replaces per-class training for MILP instance generation","feed_subtitle":"A 4,000-code library and scale-aware embeddings turn a single target instance into a matching training set.","key_machinery":"The load-bearing object is MILP-EmbedSim, a similarity between MILP instances defined as the cosine similarity of their normalized embeddings. The embeddings come from a contrastively trained encoder: a bipartite-graph neural network with a global summary node reads the instance's variables, constraints, and coefficients, and is aligned with a frozen text encoder through a symmetric cross-entropy loss on (instance, textual description) pairs. Because every library class contributes instances of several sizes that share one description, the trained space is pushed to put same-class instances of different scales near each other; retrieval is then literal nearest-neighbor search for a target instance's embedding, and the retrieved code is the matched library entry's generator.","core_discovery":"The central claim is that instance generation and code generation are the same task: the object to produce is not an instance but a program that yields a distribution of instances matching the target class. The paper reports that execution of the retrieved code achieves high average similarity to targets (0.920 on the test library) while producing generally feasible instances, and that this outperforms both few-shot and fine-tuned language-model code-generation baselines, which rarely produced valid code. Against a learned reconstruction baseline, MILP-Retrieval's generated instances receive higher MILP-EmbedSim scores on all seven tested classes, including classes whose baseline training timed out or produced infeasible instances. The method needs only one target instance, no per-class training, and the retrieved code is accompanied by a mathematical formulation, so the generation process is inspectable and adjustable.","pith_inferences":["If library coverage is the true bottleneck, the natural next experiment is to grow the library by targeted evolution around failed retrievals and measure whether top-1 similarity to held-out classes rises; the paper's size-ablation already points that way.","A workflow the paper does not evaluate is downstream solver training: using MILP-Retrieval to generate a training set, then testing whether a learned branching or cut-selection model improves, which would directly validate the motivation.","Because the embedding is trained jointly with textual descriptions, classes with vague or missing descriptions may embed poorly; a testable variant is description-free retrieval, training only on graph-graph pairs.","The retrieved code could be locally tuned to hit a target difficulty profile by adjusting parameters, turning retrieval into retrieval-plus-optimization; the paper mentions the parameter control but does not automate it."],"forward_implications":["New MILP classes can be targeted with a single instance and no model training, so data-scarce solver tasks can be addressed immediately.","Generated instances inherit a readable mathematical formulation, so scale, density, and difficulty can be adjusted through code parameters before solving.","The retrieval pipeline is reusable across classes: one encoder and one library serve all targets, amortizing the expensive embedding pretraining.","MILP-EmbedSim can replace the previous 11-statistic JS-divergence comparisons with pairwise, scale-robust instance similarity scores.","Because instances are produced by executing code rather than iterative structure prediction, generation is substantially faster and cheaper than learned reconstruction."],"supporting_citations":[{"why":"Supplies the evolutionary method and the Evolve/Train and Evolve/Test libraries of codes, instances, and textual descriptions that the whole retrieval pipeline is built on.","marker":"Li et al., 2024a"},{"why":"Defines the lossless bipartite graph representation of MILP instances that the embedding model consumes.","marker":"Gasse et al., 2019"},{"why":"Provides the contrastive image-text alignment paradigm the authors adapt to align MILP graphs with textual descriptions.","marker":"Radford et al., 2021"},{"why":"Supplies the frozen text embedding model used as the text encoder in the contrastive training objective.","marker":"Lee et al., 2024"},{"why":"Defines the prior JS-divergence similarity metric and reconstruction framework that MILP-EmbedSim is designed to improve on.","marker":"Geng et al., 2023"},{"why":"Provides the ACM-MILP baseline against which MILP-Retrieval is compared in the instance generation experiments.","marker":"Guo et al., 2024"}],"fun_headline_variants":["MILP generation via code retrieval outperforms baselines","One target instance, 4,000 codes: MILP generation via retrieval","Retrieve MILP generation code, skip the training","MILP instances by code retrieval, not per-class models","Code retrieval beats fine-tuned models for MILP data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the finite evolved library contains a generator whose output distribution is close enough to the target class that executing the retrieved code yields usefully similar instances; if that premise fails for a class, no embedding or similarity score can repair the match.","fun_headline_variants_meta":{"raw":{"variants":["MILP generation via code retrieval outperforms baselines","One target instance, 4,000 codes: MILP generation via retrieval","Retrieve MILP generation code, skip the training","MILP instances by code retrieval, not per-class models","Code retrieval beats fine-tuned models for MILP data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000995,"raw_usage":{"total_tokens":4192,"prompt_tokens":898,"completion_tokens":3294,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":3209}},"tokens_in":514,"tokens_out":3294,"duration_ms":22391,"temperature":1.0,"reasoning_tokens":3209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:29:34.436136+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a class absent from the training library, generate many target instances at varied scales, and for each retrieve the top-1 code. If the retrieved code's true class label (not its embedding score) frequently fails to match the target class, or if same-class pairs at different scales score below cross-class pairs at the same scale on MILP-EmbedSim, the central claim is falsified.","supporting_citations":[{"cited_title":"A deep instance generative framework for milp solvers under limited data availability","cited_arxiv_id":null,"evidence_quote":"Defines the prior JS-divergence similarity metric and reconstruction framework that MILP-EmbedSim is designed to improve on."},{"cited_title":"Acm-milp: Adaptive constraint modification via grouping and selection for hardness-preserving milp instance generation","cited_arxiv_id":null,"evidence_quote":"Provides the ACM-MILP baseline against which MILP-Retrieval is compared in the instance generation experiments."}],"review_version":1}