{"id":"90626fe8-db78-49c2-a0b1-1a4497689f72","arxiv_id":"2412.00117","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"The 2024 XCSP3 competition proceedings document the benchmark suite, solver descriptions, and final rankings across six constraint-solving tracks.","lead":"This is the official proceedings of the 2024 XCSP3 competition, a yearly benchmark that ranks constraint programming solvers on a fixed set of 34 problems. It describes the selected problems, the participating solvers, and the rankings across six tracks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The documented benchmark is not self-consistent: §4.1 says 200 CSP/250 COP instances but the listed series sum to 195/254, and several printed PyCSP3 models contain syntax errors or undefined names; rankings cannot be tied to the documented benchmark from this paper alone.","rationale":"This is a competition proceedings whose strongest claim is factual: the official rankings. The most load-bearing condition for that claim is that the benchmark described in the paper is the benchmark actually run. The paper itself provides enough detail (models, data series, instance counts, scoring rules) to attempt a reconstruction, and that reconstruction fails on internal evidence: the instance-count totals in §4.1 do not match the sum of the listed series in §2, and several printed PyCSP3 models are not valid Python (syntax errors or undefined names). These are not matters of external consensus; they are discrepancies within the document. The reader's verdict was CONDITIONAL on archive identity; our finding strengthens that condition by showing that the printed description is already self-inconsistent. We do not claim the rankings are false—the organizers presumably ran the archived instances—only that the paper cannot serve as a standalone specification of the executed benchmark. This does not change the reader's CONDITIONAL verdict: the condition is real and now concretely evidenced. The proposed check is cheap and decisive: if the counts reconcile and the printed models compile, the identity concern is resolved; if not, the rankings are unverifiable from the paper alone.","tokens_in":58970,"tokens_out":11849,"duration_ms":100556,"concrete_test":"Write a script that (a) parses §2.1–§2.2 for every 'A series of N instances' statement and data-parameter list, sums the counts, and compares with §4.1's 200/250 totals; and (b) for each printed PyCSP3 model, runs 'python <model>.py -data=<first listed data>' to see whether it compiles and generates an XCSP3 instance. If the sums differ or any model fails to run, then compare the archived model files from the XCSP24 website with the printed listings to quantify the divergence; the identity condition for the rankings is then not satisfied by the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the listed rankings reflect solver performance on the 2024 XCSP3 instance suite. That claim presupposes that the instances described in the proceedings are the instances actually run. The proceedings fail this identity condition internally. First, §4.1 reports 200 CSP and 250 COP instances, but summing the 'series of N instances' statements in §2.1–§2.2 gives 195 CSP and 254 COP instances. Second, several printed models are not executable Python: §2.2.9 MaximumDensityOscillatingLife has 'k in (k1 k2)' (missing comma); §2.2.13 StillLife has an unmatched parenthesis in the Sum call; §2.2.15 TravelingTournament assigns 'istances' but references 'distances'; §2.2.18 Wordpress uses undefined 'lbWP'. The front matter already warns that 'some differences may exist when compiling the models presented in this document and those that can be found in this archive', but the scale of the discrepancy (count mismatch plus non-runnable code) means the archived benchmark cannot be reconstructed from this document. Unless the archive is verified to match the intentions expressed here, the reported rankings cannot be checked against the benchmark as documented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript documents the 2024 XCSP3 constraint-solver competition: it describes the 34 selected problems, gives PyCSP3 models for the instance generators, presents the participating solvers and their configurations, states the hardware and timeout settings and the scoring rules, and lists the official ranking for each of the six tracks. The central claimed result is that the listed rankings, e.g., Picat first in CSP and Fast COP, CPMpy ortools first in COP and parallel COP, and Exchequer first in Mini COP, reflect the solvers' actual performance on the 2024 instance suite under the stated conditions. The paper points to an external archive and website for the exact instances and detailed solver traces.","tokens_in":59204,"tokens_out":8241,"duration_ms":73556,"significance":"If the rankings are accurate, the document is a useful record of the relative performance of a broad set of CP, SAT-based, MIP, and SMT technologies on a fresh, publicly available benchmark suite. The manuscript has real strengths: it states the scoring rule, reports hardware and timeout information, discloses that the instance-selection organizer's solver ACE was run off-competition, and defers to an archive and a website with per-instance traces. These features make the competition more auditable than many solver competitions. The value of the proceedings, however, depends on whether the benchmark described inside the document is the benchmark that was actually run; several internal inconsistencies currently prevent the reader from checking that identity from the manuscript alone.","major_comments":[{"comment":"The timeout information is inconsistent. Table 4.1 lists 40 minutes for CSP, COP, and parallel COP and 4 minutes for Fast COP, and Table 4.2 lists 40 minutes for both mini tracks, but the bullets immediately below state that fast-COP sequential solvers were allocated 3 minutes of CPU time and 4.5 minutes of wall-clock time, and that all other sequential solvers were allocated 30 minutes of CPU time and 45 minutes of wall-clock time. The paper must state which limits were actually enforced, because the rankings are only meaningful relative to the true per-track timeouts.","section":"Section 4.1 (Tables 4.1–4.2)"},{"comment":"The stated totals of 200 CSP instances and 250 COP instances do not match the per-series counts given in the problem sections. Summing the stated series sizes in Sections 2.1.1–2.1.16 gives 195 CSP instances, and summing the COP series in Sections 2.2.1–2.2.18 gives 254 COP instances (counting the BinPacking series as 6 + 3 + 6). The totals or the per-problem series counts must be corrected, and they must be made consistent with the archived instance set, since the rankings are defined over those instances.","section":"Section 4.1 vs Sections 2.1–2.2"},{"comment":"The hardware description is internally contradictory: the cluster nodes are said to have 32 GiB of RAM (24 GiB for jobs), but each solver is said to have been allocated 64 GiB of RAM independently of the track. A per-solver allocation larger than the node memory is not possible as written. Please clarify the node memory and the actual per-job memory allocation; this is part of the run conditions needed to reproduce the results.","section":"Section 4.1, hardware paragraph"},{"comment":"Several printed PyCSP3 models are not valid code as shown. MaximumDensityOscillatingLife contains the expression `(k1 k2)` with a missing comma; StillLife has an unmatched parenthesis in the `Sum` call; TravelingTournament assigns `istances = data` but then references `distances`; and Wordpress uses the undefined identifier `lbWP`. The front matter already warns that the printed models may differ from the archived instances, but the scale of these discrepancies means the benchmark cannot be reconstructed from the document alone. Please either replace the faulty listings with versions that compile, or explicitly label them as illustrative and give a precise pointer to the authoritative archive versions, ideally with version or checksum information.","section":"Sections 2.2.9, 2.2.13, 2.2.15, 2.2.18"}],"minor_comments":[{"comment":"The informal description says that every two vectors must have Hamming distance at most `k`, but the model and the accompanying comment enforce `Hamming(row1, row2) >= k`; the description should be aligned with the intended semantics.","section":"Section 2.1.5"},{"comment":"The file names `WordQuare.py` and `AztecDimaondSym.py` appear to be misspelled for `WordSquare.py` and `AztecDiamondSym.py`; please correct the typos.","section":"Sections 2.1.16 and 2.2.2"},{"comment":"The data paragraph writes `Values of (n, h, s)` but the model description defines the word size as `m`; the notation should be consistent.","section":"Section 2.2.17"},{"comment":"The validation procedure for solver outputs is only mentioned as \"validating solvers and rankings\" without describing how solutions or unsat answers were checked; adding one sentence about the checker used (e.g., XCSP3 Tools) would improve reproducibility.","section":"Section 4.1"},{"comment":"The rankings list only the top three solvers per track without the corresponding scores; since the paper relies on external traces for verification, a brief per-track score table or an explicit pointer to where the full score tables are stored would make the proceedings more self-contained.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"This is a competition proceedings report rather than a methodological paper, and its archival value depends on the public instance archive and website. The internal inconsistencies listed in the major comments are fixable, but they currently stand between the reader and the central claim that the reported rankings pertain to the benchmark as documented. I would encourage the editor to require the authors to reconcile the instance counts, timeouts, memory allocation, and model listings with the archived benchmark before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a useful annual record of the 2024 XCSP3 solver competition, with fresh benchmark instances and credible rankings. It is not a research advance, and the document has a reproducibility gap that you should not ignore: the printed model listings are not all runnable and the instance counts do not match the stated totals, so the on-line archive, not the paper, is the only ground truth.\n\nWhat the paper does well: the 34-problem suite with PyCSP3 models is a real community resource; many problems are new to this edition. The scoring rules are precise, the solver descriptions are informative, and the organizers are honest about the ACE selection bias—they exclude ACE from the rankings on that basis. For someone tracking solver performance year to year, these proceedings give a clear reference.\n\nThe soft spots are mostly about self-containment. Section 4.1 says 200 CSP and 250 COP instances, but adding up the series listed in Sections 2.1–2.2 gives 195 and 254. Several code listings have clear errors: 'k1 k2' in MaximumDensityOscillatingLife, an unmatched parenthesis in StillLife, 'istances' vs 'distances' in TravelingTournament, and an undefined 'lbWP' in Wordpress. The front matter warns that 'some differences may exist' when compiling the printed models versus the archive, but this is not minor—the printed models cannot all be compiled at all. Since the archive link is not given as a direct URL or hash, a reader cannot reconstruct the benchmark from this document alone.\n\nThere's also a small disclosure issue: CoSoCo, ranked third in COP, is authored by one of the proceedings' organizers (Gilles Audemard). It is not as direct a conflict as ACE, since Audemard did not select instances, but a one-line note would have been appropriate.\n\nI don't think any of this invalidates the rankings. The scores come from actual runs, and the scoring rules are explicit. The central claim holds up as a record of what happened in the 2024 competition, provided the archive is what was actually run.\n\nWho this is for: CP researchers and solver developers who want a yearly benchmark and a comparison point, plus anyone organizing solver competitions. It is not a paper that changes theory.\n\nFor peer review, I would send it out. The count and code issues are fixable, and the competition record deserves a citable proceedings.","headline":"A solid annual benchmark record with credible rankings, but the printed models have errors and the instance counts don't add up, so the archive is the only ground truth.","tokens_in":59741,"tokens_out":6799,"would_cite":true,"duration_ms":56805,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The 2024 XCSP3 competition ranks Picat first in the CSP and Fast COP tracks, CPMpy ortools first in the COP and parallel COP tracks, Exchequer first in Mini COP, and CPMpy chuffed first in Mini CSP.","keywords":["XCSP3 competition","constraint programming","constraint satisfaction","constraint optimization","XCSP3-core","PyCSP3","solver benchmarking","solver ranking"],"falsifier":"Re-run the ranked solver configurations on the archived instance files under the stated CPU time limits and apply the published scoring rule; if the top finishers in any track change, the reported rankings do not reproduce.","tokens_in":58756,"feed_emoji":"🏆","tokens_out":12330,"duration_ms":105520,"temperature":0.7,"pith_summary":"These proceedings record the official outcome of the 2024 XCSP3 competition, an annual benchmark in which constraint solvers run on a newly selected set of problems. The organizers chose 34 problems, restricted the input language to the XCSP3-core kernel, and generated 200 CSP plus 250 COP instances for the standard tracks and 150 each for the mini tracks. The claimed result is the ranking: Picat first in the CSP and Fast COP tracks, CPMpy ortools first in the COP and parallel COP tracks, Exchequer first in the Mini COP track, and CPMpy chuffed first in the Mini CSP track. If the ranking is correct, it identifies which solver approaches currently perform best on this benchmark suite under fixed time limits and a published scoring rule.","feed_headline":"Picat, CPMpy and Exchequer split XCSP3 2024 wins","feed_subtitle":"A fresh 34-problem suite and published scoring rank today's constraint solvers.","key_machinery":"The load-bearing objects are the 2024 instance suite and the scoring protocol. XCSP3-core is the restricted input format used throughout: integer variables with a kernel of 24 popular global constraints such as allDifferent, sum, table, cumulative, noOverlap, and binPacking. The 34 selected problems, almost all new, are modelled in the Python library PyCSP3 and generated as series of increasing difficulty, giving 200 CSP and 250 COP instances for the standard tracks and 150 each for the mini tracks. Solvers run on identical cluster nodes with time limits enforced by runsolver, and the ranking algorithm discards off-competition entries, removes less efficient variants of the same solver, and applies the published point rules. This combination of fresh problem selection and fixed protocol gives the rankings their content.","core_discovery":"On the paper's own terms, the discovery is a measured performance ranking rather than a theoretical theorem. The official 2024 XCSP3 competition results put Picat at the top of the CSP and Fast COP tracks, CPMpy ortools at the top of the COP and parallel COP tracks, Exchequer at the top of the Mini COP track, and CPMpy chuffed at the top of the Mini CSP track, with Fun-sCOP, CoSoCo, Choco, toulbar2, miniBTD, Nacre, miniRBO, and CPMpy gurobi also in the top places. The ranking was produced by counting solved instances for satisfaction problems and by awarding points for best-known bounds, with full credit only when optimality is proved, under time limits of 30 CPU-minutes per instance, 3 minutes in the Fast COP track, and 4 CPUs in the parallel track. Because the instance selector was also the author of one solver, that solver was run but kept off the official ranking. The paper asserts these lists accurately reflect solver performance on the 2024 instance suite.","pith_inferences":["Had the selector's solver been allowed to compete, the standings might shift, because the paper reports that its mixed-heuristic variant scores better than its default configuration in several tracks while being excluded from the ranking.","A testable extension of this work would be to reselect the instance suite with a different selector and re-run the same solver configurations, which would reveal how much of the ranking reflects selection bias.","If the warned mismatch between the printed models and the archive affects any problem, then the models in this document should be treated as documentation and only the archive used for exact reproduction.","The half-point rule for shared, unproven best bounds means that a solver which finds good solutions quickly but rarely proves optimality can rank below a slower solver that closes the proof; computing rankings under alternative scoring would show how much this design choice matters."],"forward_implications":["If the rankings hold, Picat's first places show that a SAT-encoding route to XCSP3-core is the strongest submitted approach for satisfaction and for fast optimization on this suite.","If the rankings hold, CPMpy ortools' wins in the COP and parallel COP tracks show that a lazy-clause-generation solver with parallel search leads on the 250 optimization instances.","If the rankings hold, Exchequer's Mini COP win shows that translating XCSP3 into a C program and using a bounded model checker can be competitive in a mini track.","Because the COP scoring gives full credit only when optimality is proved, the optimization podium measures proof capability, not merely the ability to find good solutions.","The reported results are tied to the exact 2024 instance archive; the proceedings' own warning means the printed models should not be assumed to reproduce the runs byte for byte."],"supporting_citations":[{"why":"It supplies the XCSP3 format specification that all competition instances conform to.","marker":"[5]"},{"why":"It defines the XCSP3-core kernel, restricting the competition to integer variables and a fixed set of popular global constraints.","marker":"[6]"},{"why":"It provides the PyCSP3 library used to write the models from which the competition instances were generated.","marker":"[15]"},{"why":"It documents the ACE solver, whose difficulty assessment guided the selection of instances and whose author's role in selection made ACE an off-competition entry.","marker":"[14]"}],"fun_headline_variants":["XCSP3 2024: Picat, CPMpy, Exchequer take top tracks","Who won XCSP3 2024? Picat, CPMpy, Exchequer","XCSP3 2024 winners: Picat, CPMpy, Exchequer","2024 constraint solver champs: Picat, CPMpy, Exchequer","XCSP3 2024 ranks solvers; Picat, CPMpy, Exchequer lead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings are valid only if the archived instance files match the models printed in these proceedings and every solver output was checked against the correct instance; the proceedings themselves warn that the printed models and the archive may differ.","fun_headline_variants_meta":{"raw":{"variants":["XCSP3 2024: Picat, CPMpy, Exchequer take top tracks","Who won XCSP3 2024? Picat, CPMpy, Exchequer","XCSP3 2024 winners: Picat, CPMpy, Exchequer","2024 constraint solver champs: Picat, CPMpy, Exchequer","XCSP3 2024 ranks solvers; Picat, CPMpy, Exchequer lead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3475,"prompt_tokens":784,"completion_tokens":2691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":2569}},"tokens_in":400,"tokens_out":2691,"duration_ms":17476,"temperature":1.0,"reasoning_tokens":2569,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:39:09.793294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ranked solver configurations on the archived instance files under the stated CPU time limits and apply the published scoring rule; if the top finishers in any track change, the reported rankings do not reproduce.","supporting_citations":[],"review_version":1}