{"id":"792199d3-f69e-4ed6-8617-35ba2b502026","arxiv_id":"2506.04821","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Reinforcement learning on seven verifiable logic puzzles improves LLM accuracy on out-of-distribution math benchmarks, especially mid-difficulty problems.","lead":"The authors fine-tune an LLM with reinforcement learning on seven custom logic puzzles, then test it on math benchmarks. They report that the All-Game model improves average accuracy from 44.51% to 48.17%, with the biggest gains on mid-difficulty problems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control isolates puzzle-specific content from generic RL effects; the All-Game vs single-game comparison also confounds data volume and training compute.","rationale":"I agree with the reader's weakest_assumption. The central claim is causal but the design is purely correlational. The most load-bearing gap is the absence of any control that would rule out generic RL effects (format adherence, longer outputs, more training compute). In particular, the All-Game result is consistent with simply training on more data: if each per-game model is trained on one game's dataset, then the All-Game model's superior average may reflect 7x data rather than 'diverse reasoning skills.' The paper also lacks seeds, error bars, a named base model, and released artifacts, which compounds the attribution problem. A single matched control experiment would discriminate between 'puzzle content matters' and 'any verifiable RL training with enough data helps.' Given that the central claim and the mechanistic story are unsupported, the REJECT verdict is appropriate; however, the paper does contain useful puzzle-generation machinery and an explicit multi-game RL objective, so a revised version with controls and seeds could reasonably be considered. The critique is about experimental design, not integrity.","tokens_in":8452,"tokens_out":4956,"duration_ms":62698,"concrete_test":"Train a control with the identical base model, GRPO hyperparameters, reward functions (format and final-answer), and the same total number of training samples and steps as the All-Game run, but on a synthetic corpus of verifiable arithmetic/algebra word problems generated from templates disjoint from all evaluation benchmarks. Evaluate this control on the same seven benchmarks under the same zero-shot protocol, with at least 5 seeds. If the control average reaches or exceeds 48.17%, the logic-puzzle content is not responsible for the gains; if it falls below by a margin larger than the seed variance, the puzzle-curriculum attribution survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that RL over logic puzzles 'effectively enhances an LLM's compositional reasoning'—requires that the observed benchmark gains are caused by the puzzle curriculum. The experimental design does not establish this. Table 1 compares RL-tuned models only against a zero-shot base model; there is no RL-on-math-data control, no RL-on-random-verifiable-problems control, no SFT-on-puzzles control, and no equal-compute/equal-data control. The All-Game RL model is trained jointly on all seven datasets, so it sees more samples and likely more gradient steps than any single-game model; its 48.17% average could be a pure data/compute effect rather than evidence that a 'diverse curriculum fosters more generalizable inference.' Moreover, all numbers are single-run without seeds or error bars, so the +0.85% to +5.19% differences are not shown to be statistically significant. The paper's statement that gains are 'statistically significant' is unsupported. Equation (4) makes the multi-game objective explicit, but no ablation holds total data constant. Thus the central causal claim is currently unattributed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LogicPuzzleRL, a 'play-to-learn' framework that fine-tunes an LLM via GRPO on seven procedurally generated logic puzzles (Sudoku, Nonogram, Cryptarithm, Magic Square, Zebra, Graph Connectivity, Knights and Knaves) with verifiable rewards and a difficulty curriculum. The authors report that per-game RL fine-tuning yields modest gains on seven math benchmarks, while a joint All-Game curriculum improves average accuracy from 44.51% to 48.17%, and they argue that puzzle-based RLVR enhances compositional reasoning. The paper also discusses category-level and difficulty-level transfer effects, attributing the gains to reusable reasoning routines such as constraint propagation and multi-step deduction.","tokens_in":8691,"tokens_out":6058,"duration_ms":70513,"significance":"If the causal claim were established, the result would be practically significant: cheap, automatically verifiable puzzle curricula would offer a scalable route to improving out-of-distribution mathematical reasoning without task-specific symbolic tools. The design has real strengths: the training puzzles and evaluation benchmarks are cleanly separated, the rewards are verifiable, the puzzle suite covers diverse reasoning archetypes, and a public code link is provided. However, the current evidence does not yet establish the central causal claim, because the experimental design lacks controls for generic RL effects, all results come from single unseeded runs without uncertainty quantification, and key promised analyses (trace-level reasoning, the category/difficulty figures) are absent from the manuscript.","major_comments":[{"comment":"The central attribution claim requires control conditions that isolate puzzle content from generic RL effects. Table 1 compares only against the zero-shot base model; there is no RL-on-math-data control, no RL-on-random-verifiable-problems control, no SFT-on-puzzles control, and no equal-data/equal-compute comparison. Moreover, Eq. (4) shows that the All-Game objective sums over all seven puzzle datasets, so the All-Game run sees far more data and likely more gradient updates than any single-game run; the 48.17% average could be a scale effect rather than evidence for curriculum diversity. Without these controls, the observed benchmark gains are not causally attributed to the logic-puzzle content.","section":"Experiments and Analysis; Table 1"},{"comment":"All reported numbers appear to come from a single unseeded run per condition, with no error bars, seeds, or significance tests. The Introduction's claim of 'statistically significant improvements' is therefore unsupported. For example, the AIME24 gain from 13.33% to 20.00% is a change of two problems out of thirty, which is within normal run-to-run noise; several other benchmark-level differences of 1-3 percentage points are similarly fragile.","section":"Experiments and Analysis; Table 1"},{"comment":"The base model is never named, despite the paper's stated intention to 'detail the choice of base LLM.' The entropy coefficient lambda, discount factor gamma, curriculum thresholds tau_int and tau_final, puzzle difficulty parameters, number of training steps, batch sizes, and learning rates are also not reported. This makes the experiments irreproducible and leaves open the possibility that the results depend on a specific base model or a particular hyperparameter configuration.","section":"Method, Training Procedure"},{"comment":"The Introduction promises a qualitative analysis of reasoning traces, including 'the frequency of reflection, patterns in error correction, and strategies used in hypothesis generation,' but no such analysis appears in the Experiments section. The Conclusion's claim that the gains 'result from improved reasoning ability rather than the accumulation of new domain-specific heuristics' is therefore not supported by any presented evidence.","section":"Introduction; Experiments and Analysis"},{"comment":"The Abstract states that models 'receive binary feedback based on puzzle correctness,' yet Eq. (1) defines a dense composite reward with r_fmt and r_int in [0,1] and only the final term r_final in {0,1}. This inconsistency matters because dense intermediate rewards may be doing much of the learning work; the paper should either reconcile the descriptions or clarify that the binary-feedback statement refers only to the final-answer component.","section":"Abstract; Method, Eq. (1)"},{"comment":"The quantitative category-level and difficulty-level analyses, as well as the in-domain game performance, rest on Figures 1, 2, and 3, but none of these figures is included in the manuscript. Without them, claims such as 'gains peak at mid-level difficulty' and 'Nonogram yielded the largest gains in geometric reasoning' cannot be checked.","section":"Experiments and Analysis; Figures 1-3"}],"minor_comments":[{"comment":"Kakuro is mentioned in the analysis as a puzzle used for constraint propagation, but it is not among the seven constructed puzzle games; the text should either add Kakuro to the dataset description or remove the mentions.","section":"Method; Experiments and Analysis"},{"comment":"There are frequent formatting errors in inline math and references, e.g., 'Erd˝os–R´enyi' is corrupted and Eq. (2) shows 'A(d)int' with missing subscripts; a full proofread is needed.","section":"Throughout"},{"comment":"The final column mixes absolute average accuracy with relative gain in parentheses; clarify the column header and state the number of evaluation problems for each benchmark.","section":"Table 1"},{"comment":"The Conclusion states that puzzle RL yields 'near perfect in-domain performance,' but no in-domain accuracy numbers are reported anywhere in the text; add the game-benchmark results or remove the claim.","section":"Conclusion"},{"comment":"The GitHub link is given, but the manuscript does not include a version, license, or instructions for reproducing the RL runs; add these to improve reproducibility.","section":"Code availability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early preprint: the promised trace analysis and figures are missing, and the experimental protocol is under-specified. The core idea is worth investigating, but the current evidence does not support the strong causal claim in the title and abstract. I would encourage the authors to add control conditions, multi-seed runs, and the missing analyses; if that is not feasible, the manuscript should be reframed as a preliminary study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the transfer result could be real, but the paper doesn't yet show it. The experiment compares RL-on-puzzles to a zero-shot base model only, so the observed gains could come from generic RL effects (format adherence, longer outputs, more careful decoding) rather than the puzzle curriculum. The All-Game model also sees more data and compute than any single-game run, so its lead over the other rows in Table 1 is not evidence that curriculum diversity helps.\n\nWhat is actually new: building an RLVR curriculum from seven custom logic puzzles and evaluating transfer to out-of-distribution math benchmarks is a concrete setup I haven't seen exactly in the cited game-RL or math-RL lines. The puzzle suite is sensible; each puzzle has a verifiable reward and adjustable difficulty, which gives a clean training signal. The paper is well organized and clearly written.\n\nThe soft spots, in order of importance. First, no control condition. No RL on math problems, no RL on random verifiable tasks, no SFT on puzzles, no equal-compute or same-data-volume ablation. That makes the headline claim—puzzle training enhances compositional reasoning—unattributed. The stress-test note gets this exactly right. Second, all numbers are single-run. AIME24 is 30 problems, so +6.67% absolute is two problems, and the 'statistically significant' claim in the text has no support. Third, the base model is unnamed, and the code link is not supported by released data or hyperparameters, so the result is not reproducible. Fourth, the intermediate-step reward depends on 'annotated proofs or partial solutions,' but there are no details on how these were produced or matched; that could be a large hidden engineering effort. Also small but telling: the category analysis mentions Kakuro, which is not one of the seven puzzles.\n\nThe central argument is plausible but not yet demonstrated. It deserves a serious referee because the question is important and the design is testable, but the version I read should not be accepted as is. For a reader deciding what to do: useful as a discussion piece about experimental rigor in RLVR, and as a pointer to a possible curriculum idea. I would not cite it until the control conditions and seeds are added.","headline":"A novel puzzle-RLVR curriculum with a testable transfer claim, but the experiment as run does not isolate puzzle content from generic RL effects—worth reviewing, not yet convincing.","tokens_in":9224,"tokens_out":3148,"would_cite":false,"duration_ms":42588,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that reinforcement learning on seven logic puzzles improves an LLM's out-of-distribution mathematical reasoning, with a joint curriculum raising average benchmark accuracy from 44.51% to 48.17%.","keywords":["reinforcement learning","logic puzzles","LLM reasoning","out-of-distribution generalization","verifiable rewards","GRPO","curriculum learning","mathematical reasoning"],"falsifier":"Train the same base model with reinforcement learning on a matched set of random or non-logical problems (or on math problems directly) with an identical reward structure; if the math benchmark gains match or exceed the +8.22% relative improvement, the puzzle curriculum is not the causal driver.","tokens_in":8283,"feed_emoji":"🧩","tokens_out":2564,"duration_ms":31200,"temperature":0.7,"pith_summary":"This paper tries to establish that fine-tuning an LLM with reinforcement learning on a set of custom logic puzzles, each with a verifiable answer, transfers into better performance on unrelated mathematical benchmarks. The authors argue that puzzle-based RLVR trains general reasoning routines, such as constraint propagation and multi-step deduction, rather than narrow domain heuristics. The central evidence is that models trained on individual puzzles improve modestly on math tasks, while the All-Game model trained on all seven puzzles achieves the highest average accuracy of 48.17%, a +8.22% relative gain over the zero-shot baseline. A sympathetic reader would care because this suggests that cheap, verifiable puzzle games could serve as a general-purpose reasoning curriculum for LLMs, without needing task-specific symbolic tools.","feed_headline":"Logic-puzzle training lifts LLM math scores by 8 percent","feed_subtitle":"RL on seven verifiable puzzles improves out-of-distribution math reasoning, especially on mid-difficulty problems.","key_machinery":"The central mechanism is a reinforcement learning with verifiable reward (RLVR) setup over a suite of seven generated logic puzzles, each satisfying logical consistency, a unique solution, and controllable difficulty. The reward is composite: format reward, intermediate-step reward (validated against annotated partial solutions), and a final binary correctness reward. Training uses GRPO with entropy regularization, plus a dynamic curriculum that advances puzzle difficulty when intermediate-step and final-answer accuracies exceed thresholds. This per-game reward structure enables both single-game and joint All-Game training, and the joint training is the configuration that yields the strongest transfer to mathematics.","core_discovery":"The core discovery is that RL fine-tuning on seven logic puzzles—Sudoku, Nonogram, Cryptarithm, Magic Square, Zebra, Graph Connectivity, and Knights and Knaves—improves out-of-distribution mathematical reasoning in a base LLM. The joint All-Game RL model reaches 48.17% average accuracy on seven math benchmarks, up from 44.51% zero-shot, with the largest gains on mid-difficulty problems (ratings 5 to 7.5) and on categories such as algebra, geometry, and combinatorics. The paper interprets this as evidence that puzzle training reshapes the model's internal reasoning into transferable, compositional routines, while providing limited benefit for rote arithmetic or highly specialized contest tricks.","pith_inferences":["A natural testable extension the paper does not run is a control condition: RL on randomly generated but equally structured puzzles (or RL directly on math problems) with the same reward components, to isolate whether the puzzle content or merely the RL training signal causes the math gains.","If the attribution holds, puzzle RLVR could be combined with math-specific RL to push gains beyond the mid-difficulty plateau, since the two appear to target complementary reasoning regimes.","The paper's difficulty analysis suggests a concrete prediction the authors leave implicit: scaling puzzle diversity and difficulty should shift the benefit window toward harder math problems, a claim that could be checked by training with harder puzzle curricula and measuring ratings 8–9 accuracy.","The category-level trade-offs, such as Sudoku hurting logic-focused math tasks, imply that puzzle curricula should be deliberately matched to target skills; an implicit design rule that future work could formalize."],"forward_implications":["If the central claim is correct, verifiable logic puzzles provide a low-cost RL curriculum that improves out-of-distribution mathematical reasoning in LLMs without task-specific solvers or tools.","The observed mid-difficulty peak implies that puzzle training strengthens chained, multi-step inference while leaving rote computation and niche contest tricks largely untouched.","The All-Game configuration outperforming every single-game model suggests that diversity of puzzle archetypes is itself a driver of transferable reasoning, not just puzzle difficulty or reward density.","Per-game results are category-selective: Nonogram boosts geometry and grid-based tasks, Cryptarithm boosts algebra and number theory, and Zebra transfers poorly to math, indicating that puzzle choice should be aligned with target domains."],"supporting_citations":[{"why":"Supplies the GSM8K benchmark, one of the seven math evaluation sets used to measure out-of-distribution transfer.","marker":"Cobbe et al. 2021"},{"why":"Supplies the MATH benchmark, a central evaluation set spanning multiple difficulty levels and categories.","marker":"Hendrycks et al. 2021"},{"why":"Supplies OlympiadBench, used to assess transfer to olympiad-level problems.","marker":"He et al. 2024"},{"why":"Supplies Minerva-MATH, a benchmark derived from quantitative reasoning problems, included in the evaluation suite.","marker":"Lewkowycz et al. 2022"},{"why":"Describes DeepSeek-R1 and the RL training paradigm that motivates the paper's RLVR approach, including the GRPO algorithm family.","marker":"Guo et al. 2025"},{"why":"Documents LLMs' poor performance on logic puzzles like ZebraLogic, providing the motivation for using logic puzzles as training material.","marker":"Lin et al. 2025"}],"fun_headline_variants":["Logic puzzle RL boosts LLM math reasoning by 8%","Seven logic puzzles train LLMs to think mathematically","RL on logic puzzles sharpens LLM math generalization","Puzzle-solving RL improves LLM math skills","LogicPuzzleRL: RL on puzzles lifts math scores 8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The math benchmark gains are caused by the logic-puzzle reasoning curriculum rather than by generic effects of reinforcement learning, such as better format adherence, longer outputs, or more careful decoding.","fun_headline_variants_meta":{"raw":{"variants":["Logic puzzle RL boosts LLM math reasoning by 8%","Seven logic puzzles train LLMs to think mathematically","RL on logic puzzles sharpens LLM math generalization","Puzzle-solving RL improves LLM math skills","LogicPuzzleRL: RL on puzzles lifts math scores 8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1227,"prompt_tokens":895,"completion_tokens":332,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":252}},"tokens_in":511,"tokens_out":332,"duration_ms":4428,"temperature":1.0,"reasoning_tokens":252,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:32:30.254126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same base model with reinforcement learning on a matched set of random or non-logical problems (or on math problems directly) with an identical reward structure; if the math benchmark gains match or exceed the +8.22% relative improvement, the puzzle curriculum is not the causal driver.","supporting_citations":[],"review_version":1}