{"id":"1eb5b508-a2f8-4544-84ae-edb1905f5e48","arxiv_id":"2507.00726","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training LLMs with dense chess rewards improves puzzle accuracy but plateaus around 25-30%, far below expert level, because the base models lack internal chess knowledge.","lead":"This paper trained language models to solve chess puzzles with trial-and-error learning, giving graded rewards from a chess engine-trained network instead of simple right/wrong feedback. All models plateaued far below expert level, and the authors found evidence that the models lack the basic chess knowledge that reward optimization alone cannot replace.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central negative result depends on a fixed 150-step RL budget and single-run curves; without evidence that training curves have truly converged, the 'RL cannot overcome pretraining deficit' conclusion is under-supported.","rationale":"I agree with the reader's identification of the weakest assumption: the fixed 150-step training budget and single-run curves are the most load-bearing vulnerability. The paper's positive contributions (dense rewards outperform sparse, reasoning SFT produces more coherent traces) are internally consistent and convincing. The central negative claim ('RL alone cannot overcome chess pretraining deficit') requires that the training procedure has actually converged, but the text never demonstrates convergence — it reports only a single evaluation point (step 150) in Table 2's hyperparameter table and in Figs. 2–3. The qualitative examples in Appendix D show models still producing coherent but strategically incomplete reasoning, which is compatible with either a hard plateau or an under-trained model. The paper's own ablations (§C.1) show large sensitivity to prompt format (legal moves, UCI vs. SAN), reinforcing that capability is strongly modulated by the training setup. The reasoning-SFT experiment (§3.2) is the strongest support for the 'knowledge deficit' explanation, since better reasoning traces did not lift the plateau; however, it also used the same 150-step budget, so a non-convergence artifact remains plausible. I would not change the reader's CONDITIONAL verdict: the finding is plausible and valuable, but the causal claim about what RL 'cannot' do is not yet settled. If the requested convergence/multi-seed test shows flat curves beyond step 150, the paper's central claim is materially strengthened and would merit an UNCHANGED verdict with higher confidence.","tokens_in":17731,"tokens_out":1867,"duration_ms":19254,"concrete_test":"Re-run the Qwen2.5-7B dense-reward configuration with 3 seeds and extend training to at least 1,000 GRPO steps (or train until per-step puzzle accuracy is flat for 200 consecutive steps), logging evaluation accuracy every 25 steps. Additionally, run the reasoning-SFT + RL condition for Llama3.1-8B with 3 seeds to check whether Fig. 3's decline reproduces. If any configuration crosses ~35–40% puzzle accuracy or shows a monotonically increasing curve past step 150, the plateau claim falsifies; if all curves remain flat for >200 steps after 150, the plateau claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim (abstract, §3.1, §5) is that RLVR plateaus at 25–30% puzzle accuracy and that this plateau reflects a pretraining deficit that RL alone cannot overcome. The load-bearing evidence is Fig. 2/Fig. 3 with Training Steps = 150 (Table 2), from a single run per configuration (no seeds, no error bars). The reader's weakest_assumption identifies the same vulnerability: 150 GRPO steps and 19.2k samples may simply be an insufficient training budget — RLVR systems in math domains often need thousands of steps or curriculum/data scaling to show continued gains. §3.2's reasoning-SFT result strengthens the 'knowledge deficit' interpretation, but it too is reported at the same 150-step budget and as single runs; the o3-SFT + RL decline for Llama3.1-8B in Fig. 3 could also reflect reward hacking, KL collapse, or optimization instability. The paper's own ablations (legal moves, SAN vs. UCI) demonstrate real sensitivity to training signal, so the plateau could be an artifact of this specific reward/prompt configuration rather than a fundamental limit. A second concern: the authors hypothesize that base LLMs lack chess knowledge, but the comparison 'expert-level chess models achieve 66.5%' refers to a 270M-parameter chess-pretrained action-value network — a different architecture with billions of chess-specific training tokens. The conclusion may be correct, but the evidence as presented does not rule out the simpler explanation that the RL recipe, not the pretrained model, is the bottleneck. The manuscript's own limitations section (§5) explicitly raises this uncertainty, which the paper flags but does not resolve.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether LLMs can develop strategic reasoning through reinforcement learning with verifiable rewards, using chess as a testbed. The authors train Qwen2.5-3B/7B and Llama3.1-8B with GRPO on 19.2k position-action pairs from Lichess puzzles, comparing sparse binary rewards against dense rewards produced by a pretrained chess action-value network. They report three main findings: dense rewards usually outperform sparse rewards; all models plateau at roughly 25-30% puzzle accuracy, far below the 66.5% accuracy of the expert critic; and additional reasoning SFT from OpenAI o3 traces does not lift the plateau. Diagnostic experiments on board-state comprehension and mate detection show near-zero or weak performance, which the authors interpret as evidence that base LLMs lack internal chess knowledge and that RL alone may not overcome this pretraining deficit. The paper also includes ablations on prompt format, move notation, board representation, and reward type.","tokens_in":18022,"tokens_out":1985,"duration_ms":24877,"significance":"If the central claim holds, the paper is a valuable negative result for RLVR: it identifies a domain where reward optimization amplifies existing knowledge but does not create de novo strategic competence, and it provides a concrete diagnostic framework (board-state comprehension, mate detection, legal-move sensitivity) for testing pretraining knowledge. The dense-reward distillation from a chess critic is a useful methodological contribution, and the ablations on SAN vs. UCI notation and on the necessity of legal moves are informative. The paper also ships code and uses held-out ground-truth puzzle evaluation, which strengthens reproducibility. However, the main conclusion is currently under-supported because the evidence is limited to a fixed, short training budget and single runs per configuration, so the significance depends on whether the plateau is a true convergence result rather than an artifact of the training setup.","major_comments":[{"comment":"The central plateau claim rests on exactly 150 GRPO training steps, a single run per model/reward configuration, with no seeds, no error bars, and no convergence analysis. In RLVR settings for mathematical reasoning, performance often continues to improve well beyond 150 steps, so the reader cannot distinguish a genuine ceiling from an undertrained policy. The authors should provide either (i) training curves extended until evaluation accuracy is flat for a sustained number of steps, (ii) multiple seeds with variance estimates, or (iii) an explicit comparison at larger step budgets and larger training datasets, before claiming that RL alone cannot overcome the deficit.","section":"§3.1, Fig. 2, Table 2"},{"comment":"The decline of Llama3.1-8B after o3-SFT followed by RL is interpreted as evidence against the value of reasoning traces, but the same curve is also consistent with optimization instability, KL collapse, or reward hacking in the dense-reward critic. The paper does not report reward curves, KL divergence, format-compliance rates, or the accuracy of the final policy on the SFT corpus. Reporting these quantities would clarify whether the decline reflects a knowledge limit or a training failure.","section":"§3.2, Fig. 3"},{"comment":"The comparison against 'expert-level chess models' achieving 66.5% puzzle accuracy is not apples-to-apples: the 66.5% figure comes from a 270M-parameter action-value network trained for 10M optimization steps on 15B Stockfish-annotated state-action pairs, whereas the LLMs are general-purpose pretrained models fine-tuned for 150 GRPO steps on 19.2k samples. The plateau could therefore reflect the RL recipe, the reward formulation, the prompt interface, or the amount of chess-specific data, rather than a fundamental inability of pretrained LLMs to learn chess. The authors should either provide a matched comparison (e.g., an LLM trained on comparable chess-specific data) or weaken the conclusion to 'RLVR with this budget and this reward formulation does not overcome the deficit in these models.'","section":"§3.3, Table 1"}],"minor_comments":[{"comment":"The evaluation metric 'puzzle accuracy' is defined as exact match of the complete action sequence; since the model outputs only one move per position, it would help to clarify how the full sequence is obtained during evaluation and whether any rollout search or greedy decoding is used.","section":"§2.1, §3.1"},{"comment":"The diagnostic results are reported as single numbers without confidence intervals or sample sizes. Given that some entries are exactly 0.0%, adding the number of evaluation examples and any error bars would make the failure analysis more convincing.","section":"§3.3, Table 1"},{"comment":"The comparison between direct win-rate feedback and normalized rank feedback is useful, but the claim that 'both methods converge to similar final performance' is based on the same single-run issue; please state the number of runs used for these ablations.","section":"Appendix C.2"},{"comment":"The reasoning SFT corpus is only 1,000 examples, and the paper does not report how many o3 traces were filtered for correctness or formatting. A sentence describing trace-quality filtering and the number of rejected traces would help the reader assess the corpus.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a well-scoped, honest empirical study of RLVR on chess, and it deserves a serious referee. The genuinely new piece is the dense-reward setup: using Ruoss et al.'s pretrained 270M action-value network to give graded win-probability feedback to LLMs during GRPO, rather than sparse binary correctness. That is a clean instantiation of reward distillation, and the paper makes it pay off: dense rewards consistently beat sparse, and they also beat SFT. The reasoning-SFT-plus-RL ablation with o3 traces, and the two diagnostic tasks (board-state tracking at 0%, MATE at 13–54%), are thoughtful additions that directly probe the paper's knowledge-deficit hypothesis. The claims are carefully hedged — \"may not,\" \"evidence,\" and an explicit limitations section — and the code is out. That is credit where it is due.\n\nThe soft spots are proportionate, and the stress-test note largely has them right. The central negative result — all models plateau at 25–30% puzzle accuracy and RL alone cannot overcome the pretraining deficit — rests on 150 GRPO steps, 19.2k samples, and single runs with no seeds or error bars. That is a thin foundation for a negative conclusion. If longer training, more data, or a different reward formulation breaks the ceiling, the main claim weakens considerably. And the anchor point for \"expert level\" (66.5%) is a 270M chess-specialized network trained on 15B Stockfish pairs, not an LLM, so it's an apples-to-oranges comparison that may be unreachable for reasons the paper doesn't isolate.\n\nThat said, the diagnostics give the interpretation real support: scoring 0% on board-state comprehension suggests the base models genuinely cannot simulate chess, so the plateau is not obviously a training-signal artifact. The o3-SFT result (no gain, decline for Llama-3.1-8B) is suggestive but could be optimization instability or reward hacking; without seeds, I wouldn't lean on it. The paper itself flags this uncertainty without resolving it.\n\nWho should read it? Anyone thinking about RLVR in new domains and whether pretraining knowledge is a bottleneck. I'd bring it to reading group and would cite it, but the referee should push for multi-seed runs, a longer training budget or an explicit convergence check, and ideally an RL run on a chess-pretrained LLM as a positive control. That would turn a plausible result into a convincing one. Send it to review.","headline":"A clean, honest RLVR-on-chess study with a plausible but under-supported negative result; worth reviewing, but the causal claim needs multi-seed and longer-training evidence.","tokens_in":18600,"tokens_out":3945,"would_cite":true,"duration_ms":41399,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that RL fine-tuning on chess puzzles lifts LLMs only to 25-30% accuracy because the models lack chess knowledge that reinforcement learning cannot create from scratch.","keywords":["LLM chess reasoning","reinforcement learning with verifiable rewards","GRPO","dense reward via action-value network","knowledge distillation","pretraining knowledge deficit","strategic reasoning","Lichess puzzles"],"falsifier":"Run the same dense-reward GRPO recipe on Qwen2.5-7B for several times longer (e.g., 600-1,500 steps, or with process-level rewards along the puzzle line) and measure held-out 10K-puzzle accuracy; if accuracy climbs beyond roughly 35% toward the 66.5% expert mark, the claim that RL cannot overcome the pretraining deficit is falsified. Likewise, a base model scoring near 0% on FEN tracking that reaches expert-level accuracy after pure RL would contradict the diagnosis.","tokens_in":1705,"feed_emoji":"♟️","tokens_out":4291,"duration_ms":104824,"temperature":0.7,"pith_summary":"This paper tests whether reinforcement learning with verifiable rewards (RLVR) can give large language models strategic reasoning, using chess as the testbed. The authors fine-tune Qwen2.5 and Llama3.1 models with GRPO on 19.2k Lichess puzzle positions, rewarding moves with win-probability scores from a chess-pretrained action-value network instead of a binary correct/incorrect signal. Dense rewards beat sparse rewards, but every trained model plateaus at 25-30% puzzle accuracy, well below the 66.5% of expert-level chess models; adding distilled OpenAI o3 reasoning traces before RL does not break the ceiling. The paper concludes that the bottleneck is the base models' sparse internal chess knowledge—RL amplifies what pretraining already supplied and cannot create the missing domain understanding from scratch. A sympathetic reader would care because this sets a clear boundary on what post-training RL can do and redirects the recipe for new domains toward pretraining coverage.","feed_headline":"Chess RL lifts LLMs to ~30% puzzle accuracy, then plateaus","feed_subtitle":"Dense expert rewards beat sparse ones, yet accuracy caps at 25-30%—pretraining never taught the models chess.","key_machinery":"The load-bearing object is a chess-pretrained action-value network $Q_\\theta(s,a)$, a 270M-parameter 16-layer transformer trained on 15B Stockfish-annotated state-action pairs with HL-Gauss loss, which outputs post-move win probability in $[0,1]$. It serves as a dense reward model, converting every candidate move—including suboptimal ones—into graded feedback to the LLM, a form of knowledge distillation. The paper pairs this with a FEN plus legal-moves prompt with <think>/<answer> tags, GRPO rollouts, and two diagnostics (FEN state-transition prediction and MATE binary move choice) that probe whether the base model can internally simulate chess.","core_discovery":"Post-training with expert dense rewards improves LLM chess tactics, but not enough: across Qwen2.5-3B/7B and Llama3.1-8B, GRPO with a 2299-Elo action-value critic as reward reaches only 25-30% held-out puzzle accuracy, relative to 66.5% for a 1800-Elo chess model, with sparse binary rewards failing outright on two of the three base models. Reasoning SFT on 1,000 o3-generated chess analyses produces much more articulate reasoning yet the same plateau, and in Llama3.1-8B it lowers accuracy. On diagnostic tasks, the models score 0% on FEN board-state tracking and 12.7-53.7% on MATE tactic selection, showing they lack faithful internal chess simulators; from this the paper argues that RL alone cannot compensate for chess knowledge missing from pretraining, and that RLVR's success in math is an amplification of knowledge that was already there.","pith_inferences":["If the diagnosis is right, RLVR scale-ups in any domain with thin pretraining coverage—negotiation, novel games, specialized simulations—should show a similar ceiling; a cheap pretest is whether the base model can simulate the domain's state transitions before spending RL compute.","The plateau might partly reflect the 150-step, single-run training budget the paper used; longer training, a larger or more diverse puzzle pool, or process-level rewards is an untested alternative that, if it broke 30%, would falsify the strongest reading of the claim.","A direct testable extension: pretrain or heavily SFT the same LLMs on chess game corpora (as chess-specialized models do) and then run the identical dense-reward RL; if expert-level accuracy appears, the pretraining-deficit account is confirmed."],"forward_implications":["Dense, expert-graded rewards consistently outperform sparse correct/incorrect rewards for chess puzzle accuracy, and sparse rewards can completely fail on some base models.","RL fine-tuning beats supervised fine-tuning for tactical move quality, yet the 25-30% ceiling holds across model families and scales, so expert-level play is not reachable with this recipe.","Distilling high-quality reasoning traces (o3) before RL does not lift the plateau and can degrade Llama3.1-8B, separating reasoning form from chess content.","The plateau's root cause is diagnostic: base LLMs cannot reliably track board states or spot basic tactics, implying RLVR amplifies only what pretraining already encodes.","The contrast with RLVR success in math suggests pretraining domain coverage decides whether post-training RL can elicit expert-level reasoning."],"supporting_citations":[{"why":"Supplies the action-value critic Qθ used as dense reward, the 10K-puzzle evaluation set, and the 66.5% expert-level accuracy baseline.","marker":"Ruoss et al. (2024)"},{"why":"Provides the GRPO algorithm used for all policy updates in the RL fine-tuning.","marker":"Shao et al. (2024)"},{"why":"Establishes RLVR with rule-based rewards and think-tag reasoning on math, the success story the chess results are contrasted against.","marker":"Guo et al. (2025)"},{"why":"Contributes the MATE dataset used as a diagnostic for tactical move selection.","marker":"Wang et al. (2024)"},{"why":"Supports the paper's conclusion that RL post-training amplifies behaviors already learned in pretraining.","marker":"Zhao et al. (2025b)"},{"why":"Backs the claim that reasoning from demonstrations is learned as structure, not content, aligning with the amplifier view.","marker":"Li et al. (2025)"},{"why":"Frames training with reward from a stronger model as knowledge distillation, which motivates the dense-reward setup.","marker":"Zhang et al. (2025c)"}],"fun_headline_variants":["Dense expert rewards help LLM chess, but accuracy caps at 30%","LLM chess plateau: RL amplifies knowledge, can't create it","Sparse rewards fail; LLMs need pretrained chess sense","Chess RL: LLMs reach 30% accuracy, stuck far below expert","RL boosts LLM chess tactics, yet pretraining gap remains"],"cache_read_input_tokens":20608,"weakest_assumption_plain":"The conclusion depends on treating 150 GRPO steps on 19.2k puzzle samples, with every configuration run once, as sufficient evidence that no amount of additional or differently shaped RL can push past the 25-30% plateau.","fun_headline_variants_meta":{"raw":{"variants":["Dense expert rewards help LLM chess, but accuracy caps at 30%","LLM chess plateau: RL amplifies knowledge, can't create it","Sparse rewards fail; LLMs need pretrained chess sense","Chess RL: LLMs reach 30% accuracy, stuck far below expert","RL boosts LLM chess tactics, yet pretraining gap remains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2588,"prompt_tokens":907,"completion_tokens":1681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1595}},"tokens_in":523,"tokens_out":1681,"duration_ms":14542,"temperature":1.0,"reasoning_tokens":1595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:08:45.065220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same dense-reward GRPO recipe on Qwen2.5-7B for several times longer (e.g., 600-1,500 steps, or with process-level rewards along the puzzle line) and measure held-out 10K-puzzle accuracy; if accuracy climbs beyond roughly 35% toward the 66.5% expert mark, the claim that RL cannot overcome the pretraining deficit is falsified. Likewise, a base model scoring near 0% on FEN tracking that reaches expert-level accuracy after pure RL would contradict the diagnosis.","supporting_citations":[{"cited_title":"Amortized planning with large-scale transformers: A case study on chess","cited_arxiv_id":null,"evidence_quote":"Supplies the action-value critic Qθ used as dense reward, the 10K-puzzle evaluation set, and the 66.5% expert-level accuracy baseline."}],"review_version":1}