{"id":"193fd1c8-0147-42cf-926e-0459dc906003","arxiv_id":"2607.20456","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning small LMs on synthetic and bootstrapped syntax-error corrections lifts MiniZinc execution accuracy from ~0% to 98% in an ensemble, but solution accuracy saturates near 35%.","lead":"Small language models fail almost completely at rare, domain-specific languages like MiniZinc. This paper shows that fine-tuning on a curated error-correction dataset can nearly fix syntax generation (up to 98% execution accuracy), while semantic constraint reasoning remains stuck near 35%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98% execution claim rests on IndustryOr test problems being absent from all 15,649 fine-tuning examples; the paper only asserts this, and the Text2Zinc source listing is inconsistent about whether IndustryOr is included.","rationale":"The central empirical claim is that error-correction fine-tuning teaches small LMs MiniZinc syntax, moving execution accuracy from near zero to 98% in an ensemble. For that claim to mean what it says, the 100-problem IndustryOr test set must be disjoint from the 15,649 fine-tuning examples. The paper asserts this in one sentence, but the manuscript contains a relevant internal inconsistency: §2.2 says Text2Zinc includes IndustryOr problems, while Table 1's source breakdown for the fine-tuning data omits IndustryOr entirely without explaining whether it was held out. No overlap detection, split hashes, or exact problem identifiers are supplied. This is not an accusation of deliberate leakage; it is an auditability gap in the exact condition that makes the headline result interpretable as syntax learning rather than memorization. Other issues, such as missing error bars and the absence of pinned artifact hashes, are secondary and addressable. They would weaken the strength of the conclusion but do not threaten the central claim as directly as potential train/test overlap. Because the reader already assigned CONDITIONAL on this same concern, my read does not change the verdict; it sharpens the specific internal inconsistency and the concrete verification that would resolve it.","tokens_in":20115,"tokens_out":5029,"duration_ms":46235,"concrete_test":"Use the released datasets to compute normalized hashes or MinHash/SimHash signatures for all 100 IndustryOr test problem statements and for the instruction text of every training example: 8,014 base pairs, 4,452 synthetic corruption triples, 2,286 cross-model correction triples, and 8,911 positive generation pairs. Flag any exact or near-duplicate pair above a conservative threshold, and independently check that no IndustryOr problem ID appears in any training source file. If any overlap is detected, re-run Table 8 with all overlapping instances removed; if the reported 98% drops materially, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is train/test separation. Section 2.2 describes Text2Zinc as containing problems from several sources, saying 'the remaining problems ... originate from IndustryOr, Mamo, and Nl4Opt.' Section 3 then says the fine-tuning data starts from 'the verified MiniZinc solutions in Text2Zinc.' If IndustryOr problems are part of Text2Zinc, the 8,014 base instruction pairs could include them unless explicitly withheld. Table 1 lists Text2Zinc sub-sources (Mamo, Nl4Opt, Nlp4Lp, Hakank, ComplexOr, LpWp, CspLib) but has no IndustryOr row, so the reader cannot verify from the manuscript whether IndustryOr was excluded from training. The only protection is the asserted sentence in §4.4: 'It is important to note theses problems are not included in the fine-tuning.' No split hashes, problem-ID overlap lists, or near-duplicate analysis are provided. The augmented dataset then propagates from the base set: synthetic corruptions sample from baseline examples, and cross-model bootstrapping runs on the base dataset. If any IndustryOr problem or near-duplicate entered those pools, the reported 98% execution accuracy and the claim of 'effectively solving the MiniZinc syntax problem' would be inflated by memorization rather than genuine syntax learning. This is the most load-bearing condition for the central claim, and it is currently unverifiable from the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Learn2Zinc, a fine-tuning pipeline for small language models (0.6B–20B parameters) to generate MiniZinc constraint models from natural-language problem descriptions. The authors construct training data from Text2Zinc verified solutions and Or-Instruct problems translated to MiniZinc via GPT-5.2, then augment it with grammar-based synthetic corruptions and cross-model error bootstrapping. They evaluate five SLMs on a 100-problem Text2Zinc-IndustryOr test set and report that augmented fine-tuning improves execution accuracy over direct generation and chain-of-thought variants, that self-reflection further improves execution accuracy, and that a top-down ensemble reaches 98% execution accuracy while solution accuracy remains near 35%.","tokens_in":20409,"tokens_out":5956,"duration_ms":49310,"significance":"If the results are robust, the paper makes a useful contribution: it documents a concrete syntax bottleneck for a rare DSL, shows that targeted fine-tuning can mitigate that bottleneck even with sub-10B models, and releases data, models, and code. The verification protocol—compiling MiniZinc models and checking objective values against ground truth—is a genuine strength, as is the open release of the pipeline. The main scientific value lies in the separation of syntactic learnability from semantic reasoning, which is clearly demonstrated internally. However, the headline 98% execution claim rests entirely on the test set having been excluded from every stage of the fine-tuning data construction, and the manuscript currently provides only an assertion of that exclusion. The statistical presentation also lacks any variance estimates, and the ensemble method is not specified to the level needed for reproducibility.","major_comments":[{"comment":"The load-bearing claim that the 100 Text2Zinc-IndustryOr test problems are not in the fine-tuning data is only asserted, not demonstrated. §2.2 states that Text2Zinc includes problems from IndustryOr, Mamo, and Nl4Opt; §4.4 asserts 'theses problems are not included in the fine-tuning'; Table 1 lists training sources with no IndustryOr row. No split hashes, problem-ID overlap lists, or near-duplicate analyses are provided. Because the 8,014 base pairs, the 4,452 synthetic corruptions, and the 2,286 cross-model corrections are all built from the same Text2Zinc pool, any test problem or near-duplicate in training would directly inflate the 98% execution accuracy. Please provide release-quality split metadata, exact problem IDs, and a deduplication check against all 15,649 fine-tuning examples.","section":"§4.4 / §2.2 / Table 1"},{"comment":"The 98% ensemble result is not reproducible from the written description. The text says models are 'tried in descending order of capability' with retries, but it does not specify how many retries each model receives, whether self-reflection is used at each cascade level, whether a problem counts as successful when any model produces an executable model, how ties are handled, or how the 98% execution accuracy is aggregated across the 100 problems. Please provide algorithm-level pseudocode, per-problem results, and the exact criterion used to compute ensemble execution accuracy.","section":"§6.2 / Table 8"},{"comment":"All results are single-run percentages on 100 test problems, with no confidence intervals, seeds, or repeated fine-tuning/decoding runs. With n=100, the standard error of a proportion near 50% is about 5 percentage points; differences such as 70% vs. 74% or 49% vs. 51% are not meaningful without variance estimates. This matters particularly for the claims that CoT underperforms Base across all models and that augmented fine-tuning consistently outperforms Base. Please report binomial confidence intervals and ideally repeated runs with different seeds.","section":"§4.4 / Tables 2–8"},{"comment":"The augmented data is generated with GPT-5.2 both for Copt-to-MiniZinc translation and for correcting bootstrapped errors, creating a distributional coupling between the training data and the frontier-model outputs used for comparison. This is not circular, but it limits the claim that the pipeline 'teaches syntax' from independent signal. An ablation isolating synthetic corruptions only vs. GPT-5.2-corrected examples would clarify how much of the execution gain is attributable to the proposed cross-model bootstrapping as opposed to generic correction examples.","section":"§5.1 / §3"}],"minor_comments":[{"comment":"The introduction says 'four out of five models... Qwen3, LLaMa, and Gemini' — only three model names are listed, and Gemini is not part of the evaluated model set. The tested models are Qwen3, LLaMa, Gemma, and GPT-OSS.","section":"§1"},{"comment":"The text above Table 6 says 'cross-model error correction bootstrapping instances (6,728)' but Table 6 reports 6,738 and 6,738 + 8,911 = 15,649. The 6,728 figure is a typo.","section":"§5.1 / Table 6"},{"comment":"The text says Qwen3-0.6B jumps to 65% execution accuracy with augmented fine-tuning, but Table 2 reports 64.0%. The narrative and table should be aligned.","section":"§5.2 / Table 2"},{"comment":"The conclusion says solution accuracy saturates at 34%, while the abstract and Table 8 say 35%. Use one consistent figure.","section":"§10 / Abstract"},{"comment":"Typo: 'theses problems' should be 'these problems'. Also, 'opens-source' in the abstract should be 'open-source'.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is plausible but depends on train/test separation and ensemble reproducibility, both of which are currently under-specified. The manuscript also leans heavily on the authors' own prior Text2Zinc and Text2Model papers; this is not disqualifying, but the editor may wish to verify that the IndustryOr split provenance is independently checkable before publicizing the 98% execution claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time if you care about text-to-model translation or fine-tuning for rare DSLs. The paper's real contribution is the dataset construction: BNF-derived corruption rules plus cross-model error bootstrapping, where failures from five small LMs are repaired by GPT-5.2 and fed back as error-correction pairs. That is a sensible, falsifiable idea, and the release of datasets and models makes it reusable. The execution-accuracy numbers are internally consistent across model sizes, and the finding that CoT fine-tuning hurts when syntax knowledge is absent is interesting even if not deeply analyzed.\n\nWhat the paper does not do is solve the harder semantic problem, and it says so plainly: solution accuracy saturates at 35%, with a useful error analysis separating contradictory constraints from phantom variables. That honesty is a strength.\n\nThe soft spot is the one the stress-test note flags: the 98% execution claim rests on Text2Zinc-IndustryOr (100 problems) being absent from all 15,649 fine-tuning examples. Section 4.4 asserts this, and Table 1 does omit IndustryOr from the source breakdown, so the authors likely did exclude it. But the paper provides no split hashes, no overlap check, and no problem-ID list. Given that Text2Zinc as originally described includes IndustryOr problems, the reader cannot verify the exclusion from the manuscript alone. This is not evidence of leakage; it is a lack of auditability. It is fixable by publishing exact split artifacts.\n\nAlso minor but worth noting: all results are single-run percentages over one 100-problem test set with no error bars or repeated seeds. That is acceptable for a systems paper at this stage, but it should temper the \"effectively solving the MiniZinc syntax problem\" claim. A single benchmark, even a hard one, is not a language.\n\nVerdict: this is a solid empirical paper with a novel data-generation method and a clear, honest evaluation. It deserves a serious peer-review process, not a desk reject. The revisions that matter are reproducibility-focused: release split hashes, verify and document non-overlap, report runs over multiple seeds, and pin the exact commit hashes for the model and dataset artifacts. If those are addressed, this becomes a useful reference for anyone working on domain-specific code generation.\n\nBring it to reading group if you discuss data-centric approaches to code generation; otherwise cite it when the topic is fine-tuning for under-resourced languages.","headline":"Useful, credible fine-tuning study for MiniZinc with a clever error-correction data recipe; the main weakness is that the IndustryOr test-set exclusion is asserted but not auditable.","tokens_in":20923,"tokens_out":2343,"would_cite":true,"duration_ms":22576,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning small language models on grammar-corrupted and cross-model error-correction data teaches them to generate executable MiniZinc code, lifting execution accuracy from near zero to 98% in an ensemble, while solution accuracy remains","keywords":["MiniZinc","small language models","fine-tuning","code generation","error correction","constraint programming","text-to-model translation","syntax errors"],"falsifier":"Compute near-duplicate similarity between each of the 100 IndustryOr test problems and the 8,014 base pairs plus 15,649 augmented pairs (character n-grams or semantic embeddings); if any test problem or close variant is present in the training data, the 98% execution result is inflated. Alternatively, re-run the ensemble on a freshly curated held-out MiniZinc set from the same distribution and check whether execution accuracy drops materially from 98%.","tokens_in":19958,"feed_emoji":"🧩","tokens_out":6317,"duration_ms":51276,"temperature":0.7,"pith_summary":"This paper asks whether small language models (0.6B–20B parameters) can be taught to generate code in MiniZinc, a rare domain-specific language for constraint modeling. The authors find that out-of-the-box execution accuracy for these models is 0–6%, with syntax errors dominating, and they argue this is because MiniZinc is nearly absent from pretraining corpora. They build a fine-tuning dataset that combines verified text–model pairs, grammar-based corruptions of MiniZinc code, and cross-model error bootstrapping—real errors produced by one model and corrected by another. After fine-tuning, the best single model reaches 76% execution accuracy, self-reflection pushes it to 89%, and an ensemble of five fine-tuned small models reaches 98% execution accuracy. Solution accuracy, however, saturates near 35%, so the paper's conclusion is that syntax is learnable through targeted fine-tuning while constraint reasoning remains the bottleneck.","feed_headline":"Fine-tuned small models reach 98% execution accuracy on MiniZinc","feed_subtitle":"Error-correction fine-tuning lifts syntax accuracy from near zero; semantic solution accuracy remains at 35%.","key_machinery":"The central mechanism is 'cross-model error bootstrapping': run several fine-tuned small language models at multiple sampling temperatures, collect execution failures, and use a frontier model to produce minimal fixes that pass compilation; these (corrupted code, corrected code) pairs become training examples, teaching the small model to avoid and repair its own syntax errors. This is complemented by a grammar-based corruption taxonomy derived from the MiniZinc BNF (20 rules in three difficulty levels). The error-correction supervision is what differentiates the Augmented strategy from plain generation fine-tuning, and it is what enables self-reflection loops to fix mistakes at inference tim","core_discovery":"The central discovery is that the syntax gap for a rare domain-specific language like MiniZinc can be closed by fine-tuning small language models on an error-correction objective, rather than adding more code examples or chain-of-thought reasoning. The paper demonstrates this by showing that an augmented fine-tuning dataset—8,014 base generation pairs, plus 4,452 synthetic corruptions and 2,286 cross-model bootstrapped corrections—converts near-zero execution accuracy into 98% for an ensemble of five fine-tuned models (Qwen3-0.6B, LLaMA-3.2-1B/3B, Gemma-2-9B, GPT-OSS-20B). The same treatment leaves solution accuracy at 34–35%, which the authors interpret as separating the syntactic problem f","pith_inferences":["A natural extension is to run the same error-bootstrapping recipe on other model-oriented DSLs (e.g., GAMS, CPMpy): if the syntax gap closes there too while semantic accuracy stays low, the syntax/semantics split is general, not MiniZinc-specific.","The two dominant semantic failure modes the paper describes—contradictory constraints and 'phantom' variables that force zero objectives—suggest a concrete next experiment: add training examples that introduce and repair these semantic flaws, and measure whether solution accuracy moves.","Because the reported 98% rests on the test set's disjunction from training data, an independent contamination audit (e.g., n-gram overlap between test and all augmented pairs) is the cheapest check that the result is not an artifact.","The cross-scale bootstrapping pattern—smaller models produce the most corrections and learn from larger models' errors—hints at a distillation loop: repeatedly bootstrap errors from the fine-tuned ensemble to create ever-stronger training data."],"forward_implications":["If the 98% execution result is real, the syntactic barrier for using small models as MiniZinc assistants is effectively removed; the remaining obstacle is semantic correctness.","Fine-tuned small models, when ensembled, can exceed the execution accuracy of a frontier model with agentic prompting on this benchmark, suggesting that domain-specific fine-tuning is a viable alternative to large-scale prompting for rare languages.","The finding that chain-of-thought fine-tuning underperforms base fine-tuning implies that reasoning traces cannot substitute for missing syntax knowledge at small scale; syntax should be taught before reasoning.","Once code compiles, retries do not fix semantic errors, so solution accuracy is governed by modeling decisions, not compilation success—this separates the two problems cleanly."],"fun_headline_variants":["Error-correction bootstrapping lifts MiniZinc execution accuracy to 98%","Syntax errors fixed by fine-tuning: MiniZinc execution hits 98%","Bootstrapped error correction boosts MiniZinc syntax, not reasoning","From 0% to 98% execution: fine-tuned small models solve MiniZinc syntax"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 100-problem Text2Zinc-IndustryOr test set is truly disjoint from all fine-tuning data, including the 15,649 augmented pairs and the bootstrapping runs; the paper asserts this but gives no overlap detection or deduplication evidence, so a hidden near-duplicate would inflate the reported 98% execution accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Error-correction bootstrapping lifts MiniZinc execution accuracy to 98%","Syntax errors fixed by fine-tuning: MiniZinc execution hits 98%","Bootstrapped error correction boosts MiniZinc syntax, not reasoning","From 0% to 98% execution: fine-tuned small models solve MiniZinc syntax"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001016,"raw_usage":{"total_tokens":4147,"prompt_tokens":783,"completion_tokens":3364,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":3277}},"tokens_in":527,"tokens_out":3364,"duration_ms":20786,"temperature":1.0,"reasoning_tokens":3277,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:54:44.845095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute near-duplicate similarity between each of the 100 IndustryOr test problems and the 8,014 base pairs plus 15,649 augmented pairs (character n-grams or semantic embeddings); if any test problem or close variant is present in the training data, the 98% execution result is inflated. Alternatively, re-run the ensemble on a freshly curated held-out MiniZinc set from the same distribution and check whether execution accuracy drops materially from 98%.","supporting_citations":[],"review_version":1}