{"id":"9f56d2e3-64f5-41e0-953b-34c37a53c1a8","arxiv_id":"2508.15180","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A DSL plus SMT solver generates and validates 83,657 logic puzzles, and fine-tuning on them improves a 7B model's scores on several reasoning benchmarks.","lead":"PuzzleClone is a system that turns a few hand-made logic puzzles into more than 83,000 new verified puzzle questions, checked by a math solver. The authors train a small AI model on these puzzles and report better scores on several logic and math tests, though some reported gains are inconsistent across the paper's own tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-variant validity of the 83K puzzles is not demonstrated: Sec. 2.3's reproduction check covers only seed configs, while Sec. 6 admits leakage and solvability are per-spec design risks.","rationale":"The reader's weakest assumption is the same one: per-variant validity is assumed, not demonstrated. I agree. This is the most load-bearing concern because it underpins both the artifact claim ('83K programmatically validated puzzles') and the empirical training results. I do not think this invalidates the central idea; the DSL+Z3 pipeline is plausible, and the seed-reproduction check is real evidence that the encoding is faithful. But the concrete test above is feasible with the promised repository, and it would settle whether the concern lands. Secondary issues (abstract/body numerical mismatches, BBEH-mini drop, lack of error bars) are real but less load-bearing; they can be fixed without changing the framework. The verdict remains CONDITIONAL: accept after per-instance verification, or after the paper clearly qualifies that 'programmatic validation' refers to seed encoding only, and after correcting the reported numbers.","tokens_in":25896,"tokens_out":7300,"duration_ms":81504,"concrete_test":"Using the released repository, regenerate all 83,657 variants from their specs/configs and run an independent per-instance verification: (1) re-execute the Z3 encoding for each variant; (2) enumerate all satisfying models (or up to the generator's max_solution limit) and verify that the published answer is identical across models; (3) for single-choice items, verify that exactly one option is entailed; (4) for open-answer items with ans_assertion, verify the assertion holds. Report the number/fraction of variants that fail, by seed and difficulty stratum. If the failure rate is materially above zero, the 'programmatically validated' claim is false and the Table 3 results should be recomputed on the filtered subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PC-83K is 'programmatically validated' relies on Sec. 2.3, which describes a validation script that reconstructs the seed puzzle from the seed config and compares the computed answer to the known seed answer. This verifies the DSL encoding for the seed, but it does not verify the 83,657 randomized variants. For each variant, the generator uses Z3 to find a model and derive an answer, but a satisfiable model is not the same as a verified, unique, non-leaky answer. A 'must be true' query is valid only if the answer is entailed by all models (or exactly one option is entailed for single-choice items). The paper does not state that every variant is checked for multiple models, multiple entailed options, or answer leakage. The DSL has ans_assertion fields (e.g., 9-vase, 28-exam), but there is no evidence they are applied uniformly. Section 6's discussion of 'Preventing Redundancy in Puzzle Statements' is phrased as a design guideline ('Specifications should explicitly avoid this redundancy'), and the difficulty analysis says harder variants are more likely to be unsolvable. Thus the paper itself treats semantic validity of variants as a risk to be designed against, not as a property verified per instance. If a non-negligible fraction of variants have ambiguous answers, leaked answers, or no unique answer, both the benchmark's integrity and the post-training transfer numbers are called into question. This is the load-bearing assumption behind the central claim that the data are 'programmatically validated.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PuzzleClone, a DSL-based framework for synthesizing large numbers of logical/mathematical puzzles from manually encoded seed puzzles. A seed puzzle is encoded into a formal specification plus a configuration; a generator uses Z3 to produce randomized variants, and a config-based reproduction step validates that the seed puzzle can be regenerated. The authors construct PC-83K, a dataset of 83,657 puzzles, and report post-training (SFT and RL) of Qwen2.5-7B-Instruct on PuzzleClone data. The central empirical claim is that such training improves both in-distribution PuzzleClone accuracy and out-of-distribution logic/mathematics benchmarks. The paper also includes extensive DSL schema details, dataset construction and partitioning, difficulty analysis, and benchmark evaluations of many proprietary and open-source models.","tokens_in":26340,"tokens_out":3055,"duration_ms":36049,"significance":"If the main claims hold, the paper makes a useful contribution: it provides a scalable, Z3-backed pipeline for generating verifiable reasoning data and demonstrates transfer to external benchmarks. The DSL design, the use of SMT solving for answer derivation, the seed-reproduction checker, and the public release of code/data are notable strengths. The construction of 83K diverse, formally derived puzzles is potentially valuable to the LLM reasoning community. However, the empirical inconsistencies and the gap between 'programmatically validated' and per-variant verification materially affect the strength of the central claims. The external-benchmark transfer results, if reliable, are the most compelling evidence, but the reported numbers need correction before the claims can be accepted.","major_comments":[{"comment":"The headline numbers are inconsistent. The abstract reports PuzzleClone average improving from 14.5 to 66.0 and SATBench from 51.6 to 70.0, while the body (Table 3 and surrounding text) reports 14.4 to 56.2 and SATBench 52.8 to 65.0. These are different results, and the abstract cannot be reconciled with the table. This is load-bearing because the central claim is specifically about the magnitude of improvement. The authors must identify and correct the discrepancy.","section":"Abstract vs. §Experiments (Table 3)"},{"comment":"The claim of 'consistent improvements across 7 logic and mathematical benchmarks' is contradicted by the paper's own data. Table 3 shows that SFT lowers BBEH-mini from 11.3 to 9.8. The text acknowledges this but still uses 'consistent improvements' in the abstract and elsewhere. The claim needs qualification or the experiments need to be reconciled. Also, the explanation that duplicated generations caused the BBEH-mini drop is not supported by any experiment.","section":"§Experiments, 'Post Training' and Table 3"},{"comment":"The claim that all 83,657 variants are 'programmatically validated' is not supported. Section 2.3 describes a validation script that only reproduces the seed puzzle from the seed config, not per-variant checks. For multiple-choice items, the DSL has 'cond: any' and 'cond: all' but no demonstrated mechanism that exactly one option is entailed or that no option leaks the answer. Section 6 itself states that redundancy/leakage 'should be designed against' and that harder variants are more likely to be unsolvable. Thus, the paper treats per-variant semantic validity as an assumption, not a verified property. Please provide either per-instance uniqueness/leakage checks, a statistical audit, or a formal argument covering all generated variants.","section":"§2.3, §6, Table 10/Table 11"},{"comment":"The dataset partitioning table is internally inconsistent. The row sums and column interpretation are unclear: 4,300 + 74,354 + 860 = 79,514, not 83,657; the 'Test' column contains values 50,738 and 24,046 that do not sum to the stated test total of 8,443. The relationship between SFT, RL-Train, RL-Val, and Test counts needs a clear and correct presentation, because the paper's post-training numbers depend on these subsets.","section":"Table 1"}],"minor_comments":[{"comment":"The difficulty model is acknowledged as imperfect in §6, and Figure 8 shows hard variants sometimes outperforming normal variants. This is acceptable as a limitation, but the Normal/Hard stratification in Table 2 and Table 3 should be interpreted with this caveat in mind.","section":"§3.4 and Figure 8"},{"comment":"Minor typos and wording issues: 'paramter,' 'curated' vs. 'curated,' 'SMT-solvable problems' etc. A careful proofread is needed.","section":"Throughout"},{"comment":"The checklist contains several placeholder responses such as 'Type your response here'; these should be completed before publication.","section":"Reproducibility Checklist"}],"recommendation":"major_revision","confidential_remarks":"The core idea—using a DSL plus an SMT solver to generate verifiable logical puzzles at scale—is promising, and the external benchmark improvements, if confirmed, would be a meaningful contribution. However, the paper currently contains a direct numerical contradiction between the abstract and the main results, and the central 'programmatic validation' claim is not backed by per-variant evidence. Both issues are fixable within the manuscript's scope. I recommend major revision rather than rejection. The authors should also ensure the dataset partition table is corrected, as it currently undermines the reproducibility of the experimental setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the short version: PuzzleClone is a real, concrete system. The DSL, Z3-based randomizer, config-level deduplication, and seed-reproduction validator form a coherent pipeline, and PC-83K is a new 83K-instance benchmark that current models find hard. The post-training improvements on Qwen2.5-7B are worth taking seriously. The paper deserves a serious referee. It also needs revision before I'd trust the headline claims.\n\nWhat's new: the DSL encodes augmentation-specific constraints — value domains, constraint count ranges, difficulty factors — not just the puzzle logic. The config-based reproduction step is a sensible check on the encoding, and deduplication by config rather than surface text is a good idea. The model zoo is wide and the transfer results to AMC and AIME suggest the generated data carries real signal.\n\nThe soft spots are real. The abstract and body report different numbers: 14.5→66.0 vs 14.4→56.2 on PC-83K, and SATBench 51.6→70.0 vs 52.8→65.0. That changes the headline, not a typo. 'Consistent improvements' is contradicted by Table 3, where BBEH-mini drops from 11.3 to 9.8 after SFT; the paper acknowledges it but doesn't qualify the claim. And the central claim that the 83K variants are 'programmatically validated' overreaches. Section 2.3's reproduction check only re-derives the seed puzzle from the config. It doesn't certify each randomized variant has a unique, non-leaky answer. Section 6 admits leakage must be designed against and that harder variants are more likely to be unsolvable. So per-instance validity is a design risk, not a verified property. Some specs have ans_assertion fields, but we're not shown evidence they're applied uniformly across all seeds. If a non-trivial fraction of variants are ambiguous or leaked, both the benchmark's integrity and the transfer numbers become shaky.\n\nThese are fixable: align abstract/body, add error bars or significance tests, qualify the BBEH drop, and release code/data that actually runs uniqueness checks on all 83K variants.\n\nThe citation pattern is fine — SATBench and SynLogic get their due for prior SMT generation. This is for anyone building synthetic reasoning data or using reasoning benchmarks. Send it to a serious referee; expect major revision.","headline":"A real DSL-driven synthesis pipeline and a useful new benchmark, but validity of individual 83K variants is not demonstrated and the empirical reporting is inconsistent.","tokens_in":26792,"tokens_out":3088,"would_cite":true,"duration_ms":29506,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PuzzleClone claims that DSL-encoded, SMT-solved puzzle generation can produce 83,657 verifiable problems, and training on them lifts a 7B model from 14.4 to 56.2 on the benchmark and up to 12.5 points on external math and logic tests.","keywords":["PuzzleClone","PC-83K","SMT solving","domain-specific language","verifiable data synthesis","logical reasoning benchmark","LLM post-training","data augmentation"],"falsifier":"Take a random sample of variants from the hard subset (difficulty > 0.5), where the paper notes solvability drops as complexity grows, and run the generation script for each variant's own config: check that the solver finds at least one solution, that the number of solutions is compatible with the stored answer (exactly one where the spec asserts uniqueness), and that the rendered question text does not state the answer explicitly. The validator in the paper only reproduces the seed puzzle, so if more than a small fraction of sampled variants fail any of these checks, the claim that all 83,657","tokens_in":25819,"feed_emoji":"🧩","tokens_out":16035,"duration_ms":146223,"temperature":0.7,"pith_summary":"The paper claims that trustworthy reasoning data does not have to be hand-curated and small: a puzzle can be encoded as a logical template plus parameters, and software can then produce tens of thousands of variants whose answers are computed by a solver rather than guessed by a model. Its framework, PuzzleClone, encodes 86 seed puzzles in a purpose-built domain-specific language, randomizes variables and constraints to generate 83,657 deduplicated puzzles (PC-83K), derives ground truth with the Z3 SMT solver, and certifies the encoding by reproducing the original seed puzzles from their configs. The resulting benchmark is hard for current models — Qwen2.5-7B-Instruct scores 14.4 and GPT-4o 27.7 — and the central empirical claim is that training on it transfers: after supervised fine-tuning, the same 7B model reaches 56.2 on PC-83K and improves six of seven external logic and math benchmarks, with AMC2023 rising from 52.5 to 65.0. If this holds, the main bottleneck in reasoning-data creation — trustworthy labels at scale — can be replaced by a formal pipeline whose cost structure permits essentially unbounded expansion.","feed_headline":"83K solver-verified puzzles push LLM math scores up 12 points","feed_subtitle":"SMT-generated training data lifts a 7B model from 14.4 to 56.2 on the test set and to 65.0 on AMC2023.","key_machinery":"The machinery is the specification/config pair (Qs, Qc): a structured DSL description of a puzzle's variables, symbols, conditions, queries, and description template, paired with a config file pinning concrete values. A generator re-randomizes the config; the Z3 SMT solver turns the symbolic constraints into solver-derived answers; and a validation script deterministically reproduces the seed puzzle from its config, certifying the encoding. The load-bearing idea is that the template/parameter split turns puzzle synthesis into a search over a formal space, so ground truth comes from a solver rather than a language model, and seed reproduction substitutes for per-instance human review.","core_discovery":"PuzzleClone's central claim: a puzzle can be split into a universal logical template plus parameters, captured in a purpose-built DSL (a domain-specific language describing variables, symbols, condition templates, queries, and the natural-language wrapper). Randomizing a spec's config generates new puzzles; the Z3 SMT (Satisfiability Modulo Theories) solver computes ground-truth answers from the symbolic constraints; and a config-based validator re-runs the pipeline on the seed config to confirm the encoding reproduces the seed's known answer. Applied to 86 curated seeds this produced PC-83K (83,657 deduplicated puzzles). The paper further claims the data is both hard and instructive: a 7B b","pith_inferences":["The 'verified' claim is only as strong as the seed-reproduction check: per-variant fidelity, answer uniqueness, and non-leakage are assumed from careful spec design, not measured. A per-config validation sweep over all 83,657 instances would convert that assumption into data; the paper does not run it.","The backward generation strategy the paper recommends — synthesize a solution first, then constraints certifying it — is the component most likely to transfer to other NP-complete puzzle families (scheduling, packing, graph problems), where it would guarantee solvability by construction.","If the transfer gains are real, the framework suggests a general recipe: any puzzle family expressible in SMT can be turned into ground-truth-checkable training data, which may matter more for reasoning skill acquisition than the puzzle surface itself.","The dynamic rephrasing feature (same config, new language or scenario) offers a ready test of whether the trained gains survive surface changes; the paper provides the mechanism but reports no experiment on it."],"forward_implications":["SFT on the 4,300-sample SFT subset lifts Qwen2.5-7B-Instruct's PC-83K average from 14.4 to 56.2 (normal 63.5, hard 49.0), showing the harder half remains a real challenge.","The same SFT transfers to external benchmarks: AMC2023 52.5→65.0, SATBench 52.8→65.0, AIME25 6.7→23.3, MATH500 75.2→81.2, OlympiadBench 42.5→46.4, AIME24 16.7→20.0; BBEH-mini drops from 11.3 to 9.8, which the paper attributes to duplicated generations in the SFT set.","Reinforcement learning (GRPO) alone also lifts the baseline — 14.4 to 47.7 — and is the better stage for BBEH-mini (13.3 vs 9.8), but it trails SFT on every math benchmark, so the two stages are not substitutes.","Because randomization and answer derivation are programmatic and LLM-free after encoding, the marginal cost of additional puzzles is low — the framework's route to much larger datasets is different from LLM-based augmentation.","Config-level deduplication plus per-seed, difficulty-stratified splits mean the test set is structurally distinct from training variants, not just reworded copies."],"supporting_citations":[{"why":"Supplies the Z3 SMT solver used to search feasible solutions and programmatically derive ground-truth answers for generated puzzles.","marker":"De Moura and Bjørner 2008"},{"why":"Introduces the GRPO reinforcement-learning algorithm used in the RL post-training stage.","marker":"Shao et al. 2024"},{"why":"Provides SATBench, the logic benchmark on which SFT training raises accuracy from 52.8 to 65.0.","marker":"Wei et al. 2025a"},{"why":"Provides BBEH-mini, the second logic benchmark used to measure post-training transfer.","marker":"Kazemi et al. 2025"},{"why":"Supplies AMC2023, the math benchmark with the headline transfer gain (52.5 to 65.0).","marker":"MAA and users 2023"},{"why":"Supplies MATH500, a mathematical benchmark used to measure post-training gains.","marker":"Hendrycks et al. 2021a"},{"why":"Supplies OlympiadBench, an olympiad-level math benchmark used to measure transfer.","marker":"He et al. 2024"},{"why":"Provides Qwen2.5-7B-Instruct, the base model for all SFT and RL experiments.","marker":"Yang et al. 2024"},{"why":"Contributes the template-plus-parameters view of puzzles that the DSL encoding builds on.","marker":"Pan et al. 2023"},{"why":"Represents the LLM-driven augmentation approach (MathGenie) whose reliability and scalability limitations PuzzleClone targets.","marker":"Lu et al. 2024"}],"fun_headline_variants":["DSL framework synthesizes 83K verifiable puzzles for LLM math","PuzzleClone: solver-verified puzzles boost LLM logic benchmarks","Randomized specs create 83K hard puzzles that lift LLM scores","From 14.5 to 66.0: PuzzleClone trains sharper reasoning models","SMT-verified data synthesis: 83K puzzles for better LLM math"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Every one of the 83,657 generated puzzles is treated as verified because the pipeline successfully reproduces the one seed puzzle from its config; the randomized variants themselves are not individually checked to be faithful, non-leaking, and uniquely answered.","fun_headline_variants_meta":{"raw":{"variants":["DSL framework synthesizes 83K verifiable puzzles for LLM math","PuzzleClone: solver-verified puzzles boost LLM logic benchmarks","Randomized specs create 83K hard puzzles that lift LLM scores","From 14.5 to 66.0: PuzzleClone trains sharper reasoning models","SMT-verified data synthesis: 83K puzzles for better LLM math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1288,"prompt_tokens":806,"completion_tokens":482,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":550,"tokens_out":482,"duration_ms":5439,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:02:58.104117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of variants from the hard subset (difficulty > 0.5), where the paper notes solvability drops as complexity grows, and run the generation script for each variant's own config: check that the solver finds at least one solution, that the number of solutions is compatible with the stored answer (exactly one where the spec asserts uniqueness), and that the rendered question text does not state the answer explicitly. The validator in the paper only reproduces the seed puzzle, so if more than a small fraction of sampled variants fail any of these checks, the claim that all 83,657","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Z3 SMT solver used to search feasible solutions and programmatically derive ground-truth answers for generated puzzles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies AMC2023, the math benchmark with the headline transfer gain (52.5 to 65.0)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the template-plus-parameters view of puzzles that the DSL encoding builds on."}],"review_version":1}