REVIEW 3 major objections 5 minor 16 references
CP-Model-Zoo: A Natural Language Query System for Constraint Programming Models
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper proposes CP-Model-Zoo, a retrieval system that returns the closest expert-written constraint model for a natural-language problem description, and reports near-perfect ranking accuracy on simulated queries.
desk verdict A useful, honest tool paper whose headline accuracy claim is weakened by a circular synthetic-query setup and missing baselines; still worth reviewing with revisions. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is an embedding-and-retrieval pipeline. A frozen text-embedding model (built on a modern bidirectional encoder) maps strings—source code, source code plus synthetic description, or a user query—to 768-dimensional vectors; retrieval selects the top-k candidates by cosine similarity between the query vector and precomputed database vectors. The crucial ingredient is the LLM-generated descriptions appended to source code at three expertise levels, because these supply the vocabulary that plain-language queries share, and the prompts make the descriptions self-contained by asking the LLM to state the problem name, variables, constraints, and objective. This design replaces human labeling with LLM-generated text and makes adding a model incremental: generate a description, embed it, store the vector.
What would settle it
Collect queries from people who have not seen the model library and ask them to describe problems without using the canonical name (for example, 'pack items into a bag without exceeding its weight' instead of 'knapsack'), then measure MRR against the same database; a substantial drop below the reported 0.86–1.0 range would show the current scores are inflated by name leakage.
Extended reading notes
Core claim
The paper's central claim is that semantic retrieval, rather than code generation, is a viable route from natural language to constraint programming models. Each database entry is a MiniZinc source file, optionally enriched with an LLM-generated textual description at one of three expertise levels (novice, intermediate, expert); the query is embedded with the same text-embedding model, and cosine similarity ranks the candidates. Reported Mean Reciprocal Rank ranges from 0.8634 to 1.0 across configurations: source-code-only embeddings trail the enriched ones, and embeddings combining source code with intermediate-level descriptions reach MRR 1.0 for novice queries and 0.9051 on human-written descriptions from a public benchmark library. The authors interpret this as evidence that generated descriptions narrow the semantic gap between code and natural language, and that intermediate-level language is the most useful register for retrieval.
Load-bearing premise
The evaluation assumes that LLM-generated descriptions of source code—which are explicitly asked to give the problem's name—are a faithful stand-in for the queries real users will type; because each synthetic query is derived from the very source code that defines the correct answer, the high scores may partly reflect the problem name leaking into the query rather than general semantic understanding.
Editorial extensions
If this is right
- A non-expert can type a plain-language problem description and receive expert-validated source-code models ranked by relevance, avoiding LLM code generation and its hallucination risk.
- Adding a new model to the zoo is incremental: generate a synthetic description with an LLM, embed it, and store the vector—no human labeling or retraining.
- The best configuration (source code plus intermediate-level descriptions) reaches MRR 1.0 for novice queries and outperforms expert-level descriptions on human-written queries, suggesting a sweet spot for description register.
- The same retrieval service could be hosted as a public tutoring layer over the community's accumulated models, with the authors explicitly proposing deployment on a shared benchmark library.
Reading between the lines
- The reported accuracy is probably an upper bound for real-world use: the synthetic novice queries are generated from the target source code and are instructed to name the problem, so the query already contains the answer; a human-query benchmark with deliberately anonymized descriptions is needed before trusting the MRR numbers.
- The retrieval idea extends naturally beyond the 67 models tested: the same embed-and-rank pipeline could index models in other CP formalisms, or be paired with an LLM that adapts the retrieved model to the user's specific variant rather than just displaying it.
- The system could be turned into a benchmark generator: by varying the prompt and expertise level, it can produce query sets that stress-test how well embeddings separate semantically similar but distinct problems, such as knapsack variants.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CP-Model-Zoo, an information-retrieval system that takes a natural-language description of a combinatorial problem and returns the most relevant MiniZinc model from a database of 67 expert-written models. The database entries are embedded with a transformer encoder, optionally augmented by LLM-generated descriptions at three purported expertise levels (novice, intermediate, expert), and retrieval is performed by cosine similarity between query and database embeddings. The evaluation reports Mean Reciprocal Rank (MRR) for synthetic queries generated by Llama3-70B from the same source code that defines the ground truth, plus 36 human-written CSPLib descriptions used as an independent validation set. The headline results are MRR values from 0.8634 to 1.0 across configurations, with the best configuration reaching MRR 1.0 for novice-level queries.
Significance. If the reported accuracy were robust, CP-Model-Zoo would be a practical, low-cost tutoring tool for constraint programming: it reuses expert-validated models, avoids direct LLM generation of possibly hallucinated code, and requires no human-labeled training data. The system is open-source, incremental, and accompanied by a running web interface, and the CSPLib human-written queries provide an independent check on the synthetic evaluation. However, the central claim of 'excellent accuracy' is currently supported mainly by an evaluation in which the synthetic queries are generated from the very source code that serves as the retrieval target and are explicitly instructed to include the problem name, so the reported MRR values may largely reflect title/token overlap. The absence of any standard lexical baseline (TF-IDF, BM25) makes it impossible to assess how much the learned embeddings add. With these issues addressed, the contribution could be a solid practical addition to the CP ecosystem.
major comments (3)
- [Section 4, Table 1 and Figure 2] The synthetic-query protocol leaks the ground-truth problem name into every query. The D1 prompt in Figure 2a explicitly requires the generated description to explain 'The name of the problem', and the D3 prompt in Figure 2c requires a 'name:' field; the query is generated from the same source code that constitutes the target database entry. It is therefore unsurprising that the code-only embedding already achieves MRR 0.9851 on D1 queries (Table 1), and the authors' own observation that comments and variable identifiers add 'important semantic information' supports the interpretation that the high scores come from lexical/name overlap rather than semantic understanding of a user description. Please quantify the overlap (e.g., the fraction of synthetic queries containing the canonical problem name) and rerun the evaluation with the problem name and other obvious identifiers removed from the queries; if the MRR collapses, this limitation must be reported in the abstract and conclusions.
- [Section 4, Table 1] No standard lexical baseline is reported, so the reader cannot determine what the embedding model contributes over simple term matching. The code-only embedding already scores 0.8736 on CSPLib queries, which is close to the range of augmented embeddings (0.8634–0.9051), suggesting that much of the signal is lexical. Please add BM25 and TF-IDF baselines on the same query–database pairs and report whether the augmentation with LLM-generated descriptions yields a statistically significant improvement over these baselines, e.g., via a paired bootstrap over the 67 synthetic and 36 CSPLib queries.
- [Section 4, CSPLib validation set] The 36 human-written CSPLib descriptions are a useful independent check, but they are long, structured, and often contain highly distinctive vocabulary (see the steel-mill-slab example in Figure 4c), and the associated MiniZinc files may share identifiers and terminology with the descriptions. The MRR of 0.8736 for code-only embeddings on these queries again suggests strong lexical overlap. Please report the full per-query rank lists for the CSPLib queries and analyze how many queries would succeed under a pure title/lexical match; a harder version of this test, for instance using only the first sentence of each CSPLib description or with proper nouns and domain-specific identifiers removed, would more convincingly demonstrate semantic understanding.
minor comments (5)
- [Section 3, retrieval formulation] The notation 'k arg max_{s_i} sim(e_q, e_i)' is not mathematically well-formed; arg max returns a single element, and the intended meaning is 'return the k items with the highest similarity scores.' Please rewrite this as a top-k ranking rule.
- [Table 1] The table is hard to read because dashes and numbers are not properly spaced (e.g., '0.9851 -1.00.9925'); please add column separators and a footnote explaining which cells are excluded by the leave-one-out protocol.
- [Section 4 and Figure 4] The statement that rankings for problems not in the database are 'highly coherent and meaningful' is based on subjective assessment; either add an objective evaluation of these out-of-database queries or explicitly frame them as anecdotal illustrations.
- [Section 5] The sample GUI query 'You need to put classes in a timetable so students don't have two at the same time...' is not evaluated; reporting its actual top-5 retrieval result would give the reader a concrete, verifiable example of the system's behavior.
- [References] Reference [7] is cited as 'ACM Communications' without a title or article identifier, and several other references lack stable version identifiers; please complete the reference list for reproducibility.
Circularity Check
Synthetic query evaluation is self-referential because each query is generated from the ground-truth source code and forced to include the problem name.
-
self definitional
[Section 4 (Experiments), Figure 2 prompts, Table 1]
""The truth value here is the source code used to generate the description." (Section 4); Figure 2 instructs: "In your answer please explain: The name of the problem.""
For the D1-D3 query sets, each query is an LLM paraphrase of the same MiniZinc file that is the ground-truth target, and the prompts force the canonical problem name into every query. Since the database entries (SC and SC+D*) contain that same source code and the same problem name in comments, identifiers, and LLM descriptions, the reported MRR values (0.9851-1.0 for D1) partly measure title/name overlap rather than retrieval from an independent user description. The code-only SC embedding already reaches 0.9851 MRR for D1 queries, confirming that name/comment matching alone nearly solves the task. This part of the evaluation is self-referential by construction.
full rationale
The paper's retrieval pipeline itself is not circular: it embeds source-code strings and queries, ranks by cosine similarity, and the code is open-source. The central methodological weakness is the synthetic evaluation in Section 4: D1-D3 queries are LLM-generated from the very MiniZinc files that serve as ground truth, and the Figure 2 prompts explicitly require the problem name. This makes the D1-D3 MRR values partly a measure of paraphrase and name matching; the code-only SC embedding reaching 0.9851 MRR on D1 queries shows that title and comment overlap is sufficient. However, the paper also evaluates on 36 independent CSPLib human-written descriptions, whose truth values are the associated MiniZinc files; those queries were not generated from the target code, and MRR remains 0.8634-0.9051. This independent benchmark prevents the entire claim from reducing by construction. No load-bearing self-citation, fitted-parameter-as-prediction, or imported uniqueness theorem was found. Score 4 reflects a genuine self-referential evaluation component without total circularity.
Assumptions & free parameters
free parameters (1)
- embedding configuration selection =
SC+D2 (source code plus intermediate-level LLM descriptions)
assumptions (4)
- domain assumption The source code models in the database are expert-validated and correct.
- domain assumption LLM-generated descriptions from source code are representative of real user queries.
- domain assumption Cosine similarity in the embedding space captures problem-model relevance.
- domain assumption The MiniZinc file associated with a CSPLib description is the unique correct retrieval target.
Cite this review
Pith. "Pith review of CP-Model-Zoo: A Natural Language Query System for Constraint Programming Models." pith.science (2026). https://pith.science/paper/X7FPUPBU
@misc{pith2026250907867,
author = {Pith},
title = {Pith review of: CP-Model-Zoo: A Natural Language Query System for Constraint Programming Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/X7FPUPBU}},
note = {Machine review of arXiv:2509.07867}
}
read the original abstract
Constraint Programming and its high-level modeling languages have long been recognized for their potential to achieve the holy grail of problem-solving. However, the complexity of modeling languages, the large number of global constraints, and the art of creating good models have often hindered non-experts from choosing CP to solve their combinatorial problems. While generating an expert-level model from a natural-language description of a problem would be the dream, we are not yet there. We propose a tutoring system called CP-Model-Zoo, exploiting expert-written models accumulated through the years. CP-Model-Zoo retrieves the closest source code model from a database based on a user's natural language description of a combinatorial problem. It ensures that expert-validated models are presented to the user while eliminating the need for human data labeling. Our experiments show excellent accuracy in retrieving the correct model based on a user-input description of a problem simulated with different levels of expertise.
Reference graph
Works this paper leans on
-
[2]
URL:http://arxiv. org/abs/2312.06908,arXiv:2312.06908[cs],doi:10.48550/arXiv.2312.06908. 23 Christophe Lecoutre. Str2: optimized simple tabular reduction for table constraints.Con- straints, 16:341–371,
-
[10]
Accessed: 2025-04-03. URL:https://github. com/hakank. 22 Connor Lawless, Jakob Schoeffer, Lindy Le, Kael Rowan, Shilad Sen, Cristina St Hill, Jina Suh, and Bahareh Sarrafzadeh. "i want it that way": Enabling interactive decision support using large language models and constraint programming. version:
work page 2025
-
[16]
URL:https://arxiv.org/abs/2412.13663,arXiv:2412.13663. 37 Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Bao- song Yang, Pengjun Xie, Fei Huang, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. InProceedings of the 2024 Conference on Empirical Methods in Nat...
arXiv 2024
-
[1988]
18 Lothar Hotz, Christian Bähnisch, Sebastian Lubos, Alexander Felfernig, Albert Haag, and Johannes Twiefel. Exploiting large language models for the automated genera- tion of constraint satisfaction problems. URL: https://www.semanticscholar.org/ paper/Exploiting-Large-Language-Models-for-the-Automated-Hotz-B%C3%A4hnisch/ f655f067a1d769d72ec48de52942f3ce...
-
[1999]
14 Tias Guns. Increasing modeling language convenience with a universal n-dimensional array, cppy as python-embedded example. InProceedings of the 18th workshop on Constraint Modelling and Reformulation at CP (Modref 2019), volume 19,
work page 2019
-
[2007]
Springer Berlin Heidelberg. A. Crespin et al. 13 28 Yuting Ning, Jiayu Liu, Longhu Qin, Tong Xiao, Shangzi Xue, Zhenya Huang, Qi Liu, En- hong Chen, and Jinze Wu. A novel approach for auto-formulation of optimization prob- lems. URL: http://arxiv.org/abs/2302.04643, arXiv:2302.04643[cs], doi:10.48550/ arXiv.2302.04643. 29 Colin Raffel, Noam Shazeer, Adam ...
-
[2008]
Highlighting named entities in input for auto- formulation of optimization problems
12 Neeraj Gangwar and Nickvash Kani. Highlighting named entities in input for auto- formulation of optimization problems. In Catherine Dubois and Manfred Kerber, ed- itors,Intelligent Computer Mathematics, pages 130–141. Springer Nature Switzerland. doi:10.1007/978-3-031-42753-4_9. 13Ian P Gent and Toby Walsh. Csplib: a benchmark library for constraints. ...
-
[2009]
9 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova
doi:10.1007/978-0-387-39940-9_488. 9 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. URL:http://arxiv.org/abs/ 1810.04805,arXiv:1810.04805[cs],doi:10.48550/arXiv.1810.04805. 10Eugene C Freuder. In pursuit of the holy grail.Constraints, 2(1):57–61,
Show all 16 references
-
[2010]
Xcsp3: an integrated format for benchmarking combinatorial constrained problems.arXiv preprint arXiv:1611.03398,
4 Frédéric Boussemart, Christophe Lecoutre, Gilles Audemard, and Cédric Piette. Xcsp3: an integrated format for benchmarking combinatorial constrained problems.arXiv preprint arXiv:1611.03398,
-
[2011]
BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
24 Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai,...
2020
-
[2014]
MCP-solver: Integrating language models with constraint programming systems
33 Stefan Szeider. MCP-solver: Integrating language models with constraint programming systems. URL: http://arxiv.org/abs/2501.00539, arXiv:2501.00539[cs], doi:10.48550/ arXiv.2501.00539. 34 Dimos Tsouros, Hélène Verhaeghe, Serdar Kadıoğlu, and Tias Guns. Holy grail 2.0: From ...
-
[2019]
Tillman, Niklas Metzger, Julian Siber, and Bernd Finkbeiner
15 Christopher Hahn, Frederik Schmitt, Julia J. Tillman, Niklas Metzger, Julian Siber, and Bernd Finkbeiner. Formal specifications from natural language. URL:http://arxiv.org/abs/2206. 01962,arXiv:2206.01962[cs],doi:10.48550/arXiv.2206.01962. 16 JiangLong He, Mamatha N, Shiv V...
-
[2021]
OptiChat: Bridging optimization models and practitioners with large language models
6 Hao Chen, Gonzalo Esteban Constante-Flores, Krishna Sri Ipsit Mantri, Sai Madhukiran Kompalli, Akshdeep Singh Ahluwalia, and Can Li. OptiChat: Bridging optimization models and practitioners with large language models. URL: http://arxiv.org/abs/2501.08406, arXiv:2501.08406[cs...
-
[2022]
26 Kostis Michailidis, Dimos Tsouros, and Tias Guns
URL:https://github.com/jerryjliu/llama_index, doi: 10.5281/zenodo.1234. 26 Kostis Michailidis, Dimos Tsouros, and Tias Guns. Constraint modelling with LLMs using in-context learning. In Paul Shaw, editor,30th International Conference on Principles and Practice of Constraint Pr...
-
[2023]
URL: https://cacm.acm.org/news/ the-impact-of-ai-on-computer-science-education/
Accessed: 2025-04-03. URL: https://cacm.acm.org/news/ the-impact-of-ai-on-computer-science-education/. 8 Nick Craswell.Mean Reciprocal Rank, pages 1703–1703. Springer US, Boston, MA,
2025
-
[2024]
2 Gilles Audemard, Christophe Lecoutre, and Emmanuel Lonca
URL:https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md. 2 Gilles Audemard, Christophe Lecoutre, and Emmanuel Lonca. Proceedings of the 2024 xcsp3 competition.arXiv preprint arXiv:2412.00117,
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.