{"id":"948d8ec3-4d4e-4cef-a8ea-715aa46cd92d","arxiv_id":"2509.07867","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CP-Model-Zoo retrieves the most relevant MiniZinc model from a pool of 67 problems using text embeddings, with mean reciprocal rank between 0.87 and 1.00 in the paper's experiments.","lead":"This paper presents CP-Model-Zoo, a search engine that retrieves the closest constraint programming model from a database of 67 MiniZinc models based on a user's plain-English problem description. It works by generating simple summaries of expert models with a large language model and ranking matches by vector similarity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic queries embed the ground-truth problem name by instruction (Fig. 2), so the reported MRR may measure title matching rather than semantic understanding of user descriptions.","rationale":"The paper is a fair, honest description of a useful IR tool, with released code and a live prototype, and the CSPLib validation gives some independent evidential value. However, the load-bearing assumption for the headline 'excellent accuracy' claim is that the synthetic queries are representative of genuine user descriptions. The design violates this assumption in a specific, demonstrable way: the Fig. 2 prompts require the LLM to include the problem's name, and the same name appears in the source-code comments and file identifiers that populate the database. The very high code-only MRR (0.9851 for D1 queries) is consistent with title/lexical overlap being the dominant signal. Because no standard IR baseline is reported, one cannot tell whether the embedding model is doing anything beyond keyword matching. This is a construct-validity threat to the central claim, not a mere implementation detail. The reader's weakest_assumption identified exactly this circularity, and my analysis agrees. Under the paper's current evidence, the conditional verdict is appropriate: the method is plausible and the prototype is real, but the accuracy claim should be re-tested with independent, human-generated queries before being taken at face value. I therefore recommend keeping the CONDITIONAL verdict rather than upgrading to acceptance or moving to rejection.","tokens_in":11679,"tokens_out":4738,"duration_ms":44376,"concrete_test":"Recruit at least three annotators with no CP-modeling expertise, show them the 36 CSPLib problem statements (or a random subset of 67 problem descriptions) with all canonical problem names removed, and ask them to describe each problem in their own words as if asking a search engine. Then run CP-Model-Zoo with the SC+D2 embedding configuration and compute MRR under the same leave-one-out protocol. If the MRR remains near the reported 0.9–1.0, the leakage concern is not the main driver; if it drops materially (e.g., below 0.7), the synthetic-query evaluation substantially overstates real-world accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CP-Model-Zoo reliably retrieves the correct model from a user's natural-language description rests on an evaluation in which the queries are generated from the very source code that defines the ground truth, and the prompts in Fig. 2 explicitly instruct the LLM to state 'The name of the problem' (D1 and D2) or 'name:' (D3). The synthetic queries therefore contain the canonical problem title, and the database entries — source code plus LLM descriptions — contain the same title in comments, identifiers, and descriptions. The code-only baseline already reaches MRR 0.9851 for D1 queries, which is what one would expect if lexical/title overlap alone is sufficient. Thus the reported MRR values (0.8634–1.0) measure how well the embedding matches a name/title, not how well the system generalises to real user descriptions that may describe the problem without using its canonical name. The absence of a simple baseline such as BM25 or TF-IDF makes it impossible to rule out that the high accuracy is an artifact of this leakage. The paper itself acknowledges that comments in source code add important semantic information, which further supports this interpretation. The CSPLib human-written queries (36) provide some independent signal, but the reported MRR of 0.8736–0.9051 on those queries also faces the fact that CSPLib descriptions are verbose and often contain distinctive problem-specific vocabulary; moreover, the same database contains the matching MiniZinc file with comments. The core concern is not that the tool performs poorly, but that the experimental design does not yet demonstrate the claimed 'excellent accuracy' for genuinely user-generated queries.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CP-Model-Zoo, an information-retrieval system that takes a natural-language description of a combinatorial problem and returns the most relevant MiniZinc model from a database of 67 expert-written models. The database entries are embedded with a transformer encoder, optionally augmented by LLM-generated descriptions at three purported expertise levels (novice, intermediate, expert), and retrieval is performed by cosine similarity between query and database embeddings. The evaluation reports Mean Reciprocal Rank (MRR) for synthetic queries generated by Llama3-70B from the same source code that defines the ground truth, plus 36 human-written CSPLib descriptions used as an independent validation set. The headline results are MRR values from 0.8634 to 1.0 across configurations, with the best configuration reaching MRR 1.0 for novice-level queries.","tokens_in":11951,"tokens_out":4364,"duration_ms":33319,"significance":"If the reported accuracy were robust, CP-Model-Zoo would be a practical, low-cost tutoring tool for constraint programming: it reuses expert-validated models, avoids direct LLM generation of possibly hallucinated code, and requires no human-labeled training data. The system is open-source, incremental, and accompanied by a running web interface, and the CSPLib human-written queries provide an independent check on the synthetic evaluation. However, the central claim of 'excellent accuracy' is currently supported mainly by an evaluation in which the synthetic queries are generated from the very source code that serves as the retrieval target and are explicitly instructed to include the problem name, so the reported MRR values may largely reflect title/token overlap. The absence of any standard lexical baseline (TF-IDF, BM25) makes it impossible to assess how much the learned embeddings add. With these issues addressed, the contribution could be a solid practical addition to the CP ecosystem.","major_comments":[{"comment":"The synthetic-query protocol leaks the ground-truth problem name into every query. The D1 prompt in Figure 2a explicitly requires the generated description to explain 'The name of the problem', and the D3 prompt in Figure 2c requires a 'name:' field; the query is generated from the same source code that constitutes the target database entry. It is therefore unsurprising that the code-only embedding already achieves MRR 0.9851 on D1 queries (Table 1), and the authors' own observation that comments and variable identifiers add 'important semantic information' supports the interpretation that the high scores come from lexical/name overlap rather than semantic understanding of a user description. Please quantify the overlap (e.g., the fraction of synthetic queries containing the canonical problem name) and rerun the evaluation with the problem name and other obvious identifiers removed from the queries; if the MRR collapses, this limitation must be reported in the abstract and conclusions.","section":"Section 4, Table 1 and Figure 2"},{"comment":"No standard lexical baseline is reported, so the reader cannot determine what the embedding model contributes over simple term matching. The code-only embedding already scores 0.8736 on CSPLib queries, which is close to the range of augmented embeddings (0.8634–0.9051), suggesting that much of the signal is lexical. Please add BM25 and TF-IDF baselines on the same query–database pairs and report whether the augmentation with LLM-generated descriptions yields a statistically significant improvement over these baselines, e.g., via a paired bootstrap over the 67 synthetic and 36 CSPLib queries.","section":"Section 4, Table 1"},{"comment":"The 36 human-written CSPLib descriptions are a useful independent check, but they are long, structured, and often contain highly distinctive vocabulary (see the steel-mill-slab example in Figure 4c), and the associated MiniZinc files may share identifiers and terminology with the descriptions. The MRR of 0.8736 for code-only embeddings on these queries again suggests strong lexical overlap. Please report the full per-query rank lists for the CSPLib queries and analyze how many queries would succeed under a pure title/lexical match; a harder version of this test, for instance using only the first sentence of each CSPLib description or with proper nouns and domain-specific identifiers removed, would more convincingly demonstrate semantic understanding.","section":"Section 4, CSPLib validation set"}],"minor_comments":[{"comment":"The notation 'k arg max_{s_i} sim(e_q, e_i)' is not mathematically well-formed; arg max returns a single element, and the intended meaning is 'return the k items with the highest similarity scores.' Please rewrite this as a top-k ranking rule.","section":"Section 3, retrieval formulation"},{"comment":"The table is hard to read because dashes and numbers are not properly spaced (e.g., '0.9851 -1.00.9925'); please add column separators and a footnote explaining which cells are excluded by the leave-one-out protocol.","section":"Table 1"},{"comment":"The statement that rankings for problems not in the database are 'highly coherent and meaningful' is based on subjective assessment; either add an objective evaluation of these out-of-database queries or explicitly frame them as anecdotal illustrations.","section":"Section 4 and Figure 4"},{"comment":"The sample GUI query 'You need to put classes in a timetable so students don't have two at the same time...' is not evaluated; reporting its actual top-5 retrieval result would give the reader a concrete, verifiable example of the system's behavior.","section":"Section 5"},{"comment":"Reference [7] is cited as 'ACM Communications' without a title or article identifier, and several other references lack stable version identifiers; please complete the reference list for reproducibility.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is well scoped and the system is a reasonable practical contribution, but the main evaluation is currently dominated by an information-leakage artifact. I believe the authors can address this within the manuscript's scope by adding lexical baselines, stripping problem names from synthetic queries, and analyzing the CSPLib subset more carefully. I would not recommend rejection, because the underlying idea and the open-source artifact are valuable, but the abstract's 'excellent accuracy' claim should be withheld until the leakage is quantified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper builds a straightforward embedding-retrieval system for CP models: embed source code plus LLM-generated descriptions, retrieve the nearest model by cosine similarity. That is the whole method, but it works well enough to be useful as a tutoring and decision-support utility. The paper is the first to apply this exact setup to a CP model library, and the authors ship code and a live prototype, which is real value.\n\nWhat it does well: the methodology is transparent, the leave-one-out design is sensible, and including CSPLib human-written descriptions as an independent check is a good instinct. The CSPLib MRR of 0.9051 with the SC+D2 configuration is genuine evidence that the tool can rank relevant models from human text. The qualitative examples in Fig. 4, though subjective, are also fairly convincing. For someone looking to build a CSPLib search front-end, this is a reasonable starting point.\n\nThe soft spots are mostly about the main evaluation. The synthetic queries are generated from the target source code, and the prompts in Fig. 2 explicitly instruct the LLM to state the problem name. That means the queries contain the canonical title, and both the title and the code are in the database entry. The code-only baseline already reaches MRR 0.9851 for D1 queries, which is exactly what you would expect from lexical/title overlap. The authors never compare against TF-IDF or BM25, so we cannot tell how much the embedding model adds over plain keyword matching. That is the load-bearing weakness. The CSPLib results partially rescue the claim, but they are only 36 queries, and those descriptions are often verbose and full of problem-specific jargon.\n\nThe paper itself acknowledges that comments in source code add semantic information, which supports the interpretation that the high scores come partly from the code text, not from the LLM's descriptions. I do not think the tool is bad; I think the abstract's \"excellent accuracy\" is not yet supported for genuinely user-generated descriptions. The fix is standard: add baselines, generate queries without the problem name, and test with real users.\n\nWho is this for? People working on NL interfaces to CP libraries, and the CP community if this gets hosted on CSPLib. It deserves a serious referee, but the evaluation needs revision before acceptance. I would send it to review rather than desk reject.","headline":"A useful, honest tool paper whose headline accuracy claim is weakened by a circular synthetic-query setup and missing baselines; still worth reviewing with revisions.","tokens_in":723,"tokens_out":687,"would_cite":true,"duration_ms":22667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes CP-Model-Zoo, a retrieval system that returns the closest expert-written constraint model for a natural-language problem description, and reports near-perfect ranking accuracy on simulated queries.","keywords":["constraint programming","model retrieval","large language models","natural language processing","information retrieval","text embeddings","NLP4CP"],"falsifier":"Collect queries from people who have not seen the model library and ask them to describe problems without using the canonical name (for example, 'pack items into a bag without exceeding its weight' instead of 'knapsack'), then measure MRR against the same database; a substantial drop below the reported 0.86–1.0 range would show the current scores are inflated by name leakage.","tokens_in":11491,"feed_emoji":"🔍","tokens_out":7592,"duration_ms":62014,"temperature":0.7,"pith_summary":"CP-Model-Zoo is a retrieval system for constraint programming: given a natural-language description of a combinatorial problem, it returns the closest expert-written model from a database instead of asking a large language model to write code. The paper claims this sidesteps two obstacles that block non-experts—the difficulty of mastering modeling languages and the hallucination risk of LLM-generated models—while needing no human-labeled training data. In experiments on 67 source-code models drawn from existing collections, the system achieved Mean Reciprocal Rank values between 0.8634 and 1.0, with the best configuration ranking the correct model first for every simulated novice-level query. If the results hold for real users, the tool would give novices a reliable bridge from a problem description to validated models, and the authors propose it as a tutoring system and community resource.","feed_headline":"Plain-language queries retrieve the right CP model almost every time","feed_subtitle":"It matches problem descriptions to expert-validated models, giving novices working code instead of LLM guesses.","key_machinery":"The load-bearing mechanism is an embedding-and-retrieval pipeline. A frozen text-embedding model (built on a modern bidirectional encoder) maps strings—source code, source code plus synthetic description, or a user query—to 768-dimensional vectors; retrieval selects the top-k candidates by cosine similarity between the query vector and precomputed database vectors. The crucial ingredient is the LLM-generated descriptions appended to source code at three expertise levels, because these supply the vocabulary that plain-language queries share, and the prompts make the descriptions self-contained by asking the LLM to state the problem name, variables, constraints, and objective. This design replaces human labeling with LLM-generated text and makes adding a model incremental: generate a description, embed it, store the vector.","core_discovery":"The paper's central claim is that semantic retrieval, rather than code generation, is a viable route from natural language to constraint programming models. Each database entry is a MiniZinc source file, optionally enriched with an LLM-generated textual description at one of three expertise levels (novice, intermediate, expert); the query is embedded with the same text-embedding model, and cosine similarity ranks the candidates. Reported Mean Reciprocal Rank ranges from 0.8634 to 1.0 across configurations: source-code-only embeddings trail the enriched ones, and embeddings combining source code with intermediate-level descriptions reach MRR 1.0 for novice queries and 0.9051 on human-written descriptions from a public benchmark library. The authors interpret this as evidence that generated descriptions narrow the semantic gap between code and natural language, and that intermediate-level language is the most useful register for retrieval.","pith_inferences":["The reported accuracy is probably an upper bound for real-world use: the synthetic novice queries are generated from the target source code and are instructed to name the problem, so the query already contains the answer; a human-query benchmark with deliberately anonymized descriptions is needed before trusting the MRR numbers.","The retrieval idea extends naturally beyond the 67 models tested: the same embed-and-rank pipeline could index models in other CP formalisms, or be paired with an LLM that adapts the retrieved model to the user's specific variant rather than just displaying it.","The system could be turned into a benchmark generator: by varying the prompt and expertise level, it can produce query sets that stress-test how well embeddings separate semantically similar but distinct problems, such as knapsack variants."],"forward_implications":["A non-expert can type a plain-language problem description and receive expert-validated source-code models ranked by relevance, avoiding LLM code generation and its hallucination risk.","Adding a new model to the zoo is incremental: generate a synthetic description with an LLM, embed it, and store the vector—no human labeling or retraining.","The best configuration (source code plus intermediate-level descriptions) reaches MRR 1.0 for novice queries and outperforms expert-level descriptions on human-written queries, suggesting a sweet spot for description register.","The same retrieval service could be hosted as a public tutoring layer over the community's accumulated models, with the authors explicitly proposing deployment on a shared benchmark library."],"supporting_citations":[{"why":"defines MiniZinc, the modeling language in which the database source-code models are written","marker":"[27]"},{"why":"provides the benchmark library plus human-written problem descriptions used as validation queries with known ground truth","marker":"[13]"},{"why":"is the language model used to generate the synthetic novice, intermediate, and expert descriptions from source code","marker":"[1]"},{"why":"provides the orchestration layer for building the retrieval pipeline and embeddings","marker":"[25]"},{"why":"is the encoder foundation on which the text-embedding model is built","marker":"[36]"},{"why":"supplies the actual embedding model that maps code-plus-description strings and queries to 768-dimensional vectors","marker":"[37]"},{"why":"defines Mean Reciprocal Rank, the metric used to report retrieval accuracy","marker":"[8]"}],"fun_headline_variants":["Plain text retrieves the right CP model nearly every time","No LLM guesses: text search finds expert CP models","Semantic retrieval outperforms generation for CP modeling","Text embeddings rank MiniZinc models with near-perfect accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that LLM-generated descriptions of source code—which are explicitly asked to give the problem's name—are a faithful stand-in for the queries real users will type; because each synthetic query is derived from the very source code that defines the correct answer, the high scores may partly reflect the problem name leaking into the query rather than general semantic understanding.","fun_headline_variants_meta":{"raw":{"variants":["Plain text retrieves the right CP model nearly every time","No LLM guesses: text search finds expert CP models","Semantic retrieval outperforms generation for CP modeling","Text embeddings rank MiniZinc models with near-perfect accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1468,"prompt_tokens":873,"completion_tokens":595,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":529}},"tokens_in":489,"tokens_out":595,"duration_ms":5546,"temperature":1.0,"reasoning_tokens":529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:08:46.830214+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect queries from people who have not seen the model library and ask them to describe problems without using the canonical name (for example, 'pack items into a bag without exceeding its weight' instead of 'knapsack'), then measure MRR against the same database; a substantial drop below the reported 0.86–1.0 range would show the current scores are inflated by name leakage.","supporting_citations":[],"review_version":2}