{"id":"be670592-9e8f-4ab8-8fc8-de39e98d7dc0","arxiv_id":"2502.06773","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A post-training framework called RLSP, decoupling exploration from correctness rewards, induces backtracking and self-correction in LLMs and improves math reasoning performance.","lead":"This paper introduces RLSP, a post-training recipe that combines supervised fine-tuning, an exploration reward that encourages longer reasoning, and reinforcement learning against an answer verifier. The authors report that even a simple 'think longer' reward makes small language models backtrack, verify, and explore alternative solution paths, improving accuracy on math benchmarks like MATH-500 and AIME 2024.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Performance gains are not yet attributable to RLSP's exploration reward: Tables 1–2 lack outcome-only PPO controls, and 'emergent' behaviors are only qualitatively illustrated despite Remark 2 admitting pretraining presence.","rationale":"The paper is a good-faith empirical proposal with a clear framework, public code, and a useful negative result in Section 3: outcome-only PPO does not increase response length on Llama-3.1-8B, while length reward does. That ablation is real evidence that the exploration reward changes behavior in at least some settings. However, the strongest headline claims—the performance boosts and the general emergence of search behavior—depend on two conditions that are not directly tested. First, the exploration reward must improve accuracy beyond what the outcome verifier alone would produce from the same starting model; Tables 1–2 omit that comparison. Second, the 'emergent' behaviors must appear at higher rates after RLSP than in ordinary CoT from the base model; the paper provides examples but no distributions, and Remark 2 concedes the behaviors exist in pretraining data. The self-consistency comparison in Table 3 is not a substitute for these controls, because SC is an inference-time method and does not isolate the training reward. These gaps are addressable with additional experiments, so they support a conditional verdict rather than rejection. The reader's weakest-assumption analysis pointed at the length-reward proxy and reward hacking; my concern is adjacent but more specifically about the missing outcome-only ablation and the lack of quantitative behavior metrics, hence partial agreement.","tokens_in":29064,"tokens_out":6679,"duration_ms":61509,"concrete_test":"Run the missing control arms: (1) PPO with α=1 (outcome-only) starting from the same SFT Llama-3.1-8B model on the same MATH training subset and hyperparameters, then evaluate MATH-500; (2) PPO with α=1 starting from Qwen2.5-32B-Instruct on the AIME 1983–2003 set, then evaluate AIME 2024. If outcome-only accuracy is within noise of 73.4% and 23.3%, the exploration reward is not the driver. Separately, apply a rule-based or human-annotated behavior classifier (backtracking, verification, alternative-method statements) to 100–200 base-CoT versus RLSP-CoT outputs per model; if frequencies are statistically indistinguishable, the 'emergence' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that decoupling exploration reward from correctness reward drives reasoning gains is not backed by the necessary control. Equation (1) defines RLSP as α·1[verifier] + (1−α)·R_ex, but Table 1 compares SFT (70.2%) against SFT+ER-RLSP (73.4%) and Table 2 compares base (13.3% AIME) against Base+ER-RLSP (23.3%), with no arm using PPO with only the outcome verifier (α=1). Without SFT+PPO(α=1) and Base+PPO(α=1), the reported 3.2-point and 10-point gains could be ordinary outcome-reward RL rather than the exploration term; the paper's own Section 3 shows outcome-only PPO can increase response length for Qwen models, so this is a live possibility. The 'emergent behavior' claim is similarly undersupported: it rests on selected traces in Figures 1–3 and Appendix A, while Remark 2 and Section 6 concede these behaviors are 'implicitly present in pretraining data' and 'not surprising.' No quantitative frequencies of backtracking, verification, or alternative-method statements in base versus trained CoT are reported. Section 2.2 also concedes the length reward can be hacked by repetition. These are omitted controls rather than direct contradictions, so the paper remains promising, but the causal role of RLSP's exploration reward is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RLSP (Reinforcement Learning via Self-Play), a three-stage post-training recipe: (1) optional SFT on reasoning traces, (2) an exploration reward independent of answer correctness (e.g., a length bonus), and (3) PPO with an outcome verifier, with the total reward a convex combination of correctness and exploration terms (Eq. (1)). The authors report that on Llama-3.1-8B-Instruct, SFT plus exploration-reward RL reaches 73.4% on MATH-500 versus 70.2% for SFT alone, and on Qwen2.5-32B-Instruct, exploration-reward RL improves AIME 2024 pass@1 from 13.3% to 23.3%. They further claim that even without SFT, the length reward induces emergent search behaviors (backtracking, self-correction, verification, alternative methods), illustrated with qualitative traces in Figures 1–3 and Appendix A. The paper ends with a heuristic argument, grounded in chain-of-thought expressivity results, for why rewarding longer trajectories should create self-play over reasoning traces.","tokens_in":29369,"tokens_out":3363,"duration_ms":30399,"significance":"If the causal role of the exploration reward were established, this would be a useful design principle for reasoning post-training, and the public code plus decontamination efforts are valuable assets. The strengths of the paper are its clear framework, its honesty about limitations (Remark 2, Section 6), and concrete benchmark comparisons. However, the central attribution of performance gains to the exploration term is not yet supported: the headline improvement on Llama is mostly from SFT (50.6→70.2), and the marginal 3.2-point gain from ER-RLSP lacks an outcome-only PPO control and error bars. Similarly, the emergence claim relies on selected traces rather than quantitative behavior measurement, and the length reward partially builds the observed behavior into the objective. These are omitted controls and measurement gaps, not contradictions, so the paper remains promising, but the framework's distinctive contribution requires further evidence.","major_comments":[{"comment":"The performance claims attribute the gains to the exploration reward in Eq. (1), but the comparison arms are SFT vs. SFT+ER-RLSP and Base vs. Base+ER-RLSP. No arm trains with only the outcome verifier (α=1). Because §3 demonstrates that outcome-only PPO can increase response length for Qwen2.5-7B, the 10-point AIME gain for Qwen2.5-32B could plausibly be due to outcome reward alone. Please add SFT+PPO(α=1) and Base+PPO(α=1) controls, and report multiple seeds or confidence intervals, especially since the Llama gain (70.2→73.4) is only 3.2 points and may be within run-to-run noise.","section":"§4.1, §4.2, Tables 1–2; Eq. (1)"},{"comment":"The emergence claim is supported only by qualitatively selected traces. The paper defines emergent as 'the model does not exhibit that behavior in regular CoT without any special prompting' (Remark 2), but no systematic measurement of backtracking, verification, or alternative-method frequencies in the base/SFT model versus the ER-RLSP model is provided. Since Remark 2 and Section 6 concede these behaviors are 'indirectly present in the pretraining data,' the claim that RLSP induces them requires a quantitative behavioral protocol (e.g., code or template matching for search patterns) rather than illustrative examples, one of which the paper itself reports as a failure case.","section":"§1.1, Remark 2, §3, Figures 1–3, Appendix A"},{"comment":"The length reward R_ex = -C/|o| directly incentivizes longer outputs, and longer outputs are then interpreted as 'search behavior.' The paper acknowledges in §2.2 that models can hack this signal by repeating responses. The assumption that output length is a valid proxy for reasoning effort is load-bearing but not validated; the observed behaviors could reflect reward hacking or amplification of pretraining patterns. Please provide evidence that the longer traces contain diverse intermediate steps rather than repetition, or measure search behaviors independently of length.","section":"§2.2, Eq. (1); §3"}],"minor_comments":[{"comment":"The abstract states RLSP 'can boost performance by 23%' on MATH-500 for Llama-3.1-8B, but 22.8 points of the 50.6→73.4 gain come from SFT; the exploration-reward RL contributes 3.2 points. Clarify this decomposition to avoid overstating the novel component.","section":"Abstract, §4.1"},{"comment":"The caption claims the increase in response length is 'necessary for search behavior and better reasoning, even in a strict theoretical sense [MS23].' The cited results show that longer chain-of-thought can increase expressivity, not that length is necessary for reasoning or search; please rephrase to match what the citations actually establish.","section":"Figure 4 caption"},{"comment":"There are typos in the hyperparameters paragraph: 'value clippoing' should be 'value clipping'.","section":"Appendix B.4"},{"comment":"The bibliography entry [MS23] contains the misspelling 'expresssive power'; also consider providing volume/page details for the published version, if available.","section":"References"},{"comment":"The response-length plots are informative, but the section claims 'search behavior' based solely on length; adding example outputs or behavior counts for the pure-RL arms would make the comparison more convincing.","section":"§3, Figures 6–8"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern is valid and lands on the central claim: the missing outcome-only PPO controls leave open the possibility that the reported gains are ordinary outcome-reward RL. The emergence language is also stronger than the evidence. However, the manuscript is honest about several of these gaps, and the framework is clearly described; with the proposed control experiments and quantitative behavioral analysis, the paper could become a solid contribution. I am recommending major_revision rather than rejection because the required additions are within the scope of a revision and the core idea is not invalidated by the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: it has a clean idea — split the RL reward into an outcome verifier term and an explicit exploration term — and it is unusually candid about its own limits. The main performance claim, though, is not yet backed by the right control. Tables 1 and 2 compare SFT vs SFT+ER-RLSP and base vs base+ER-RLSP, but there is no PPO-with-verifier-only arm (α=1) in those tables. Without that arm you cannot attribute the 3.2-point MATH-500 gain or the 10-point AIME gain to the exploration term rather than ordinary outcome-reward RL. Section 3 does show that outcome-only PPO fails to increase response length on Llama-8B and on Qwen-7B coding, which indirectly supports the authors, but it is not a direct accuracy comparison.\n\nCredit where due: the paper is well written, the idea is simple enough to be useful, the ablation in Section 3 is a genuinely informative comparison across model families and domains, they release code, and the qualitative traces in the appendix are honest enough to include a failure case — the self-correction example that still gets the wrong answer. Remark 2 and the conclusion concede that these 'emergent' behaviors are implicitly present in pretraining data, which is the right thing to say. The definition of emergent — absent in regular CoT without special prompting — is defensible but weak, and the paper does not quantify how often backtracking or verification actually occurs in base vs trained models. So the emergence story is illustrative, not measured.\n\nOther soft spots: single runs, no error bars or seeds; the 3.2-point gap over SFT could be noise. The length reward is known to be hackable, and the paper admits this. The theory is informal — an intuition argument resting on CoT expressiveness results, not a derivation. None of these are fatal; they are missing experiments and quantification, not contradictions.\n\nWho is this for? People working on post-training for reasoning models. The decoupled exploration-reward idea is worth engaging with, and the Section 3 result that outcome-only PPO behaves differently across base models is useful. It deserves a serious referee; a competent reviewer could extract a sharp revision that adds the missing control and quantitative behavioral metrics.\n\nRecommend sending to peer review, with the expectation that the authors add the α=1 PPO arms and report multiple seeds. If they can show the exploration term causes the gain beyond the verifier alone, this is a solid contribution.","headline":"A clear, honest proposal for decoupling exploration from correctness in RL post-training; the main tables miss the outcome-only PPO control, so the causal claim is promising but not yet proven.","tokens_in":29910,"tokens_out":2144,"would_cite":true,"duration_ms":19509,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that decoupling an exploration reward from a correctness reward in PPO training induces emergent search behaviors—backtracking, verification, and exploration of alternatives—in LLMs and improves mathematical…","keywords":["reinforcement learning","large reasoning models","emergent search behavior","chain-of-thought","backtracking","self-verification","PPO","mathematical reasoning"],"falsifier":"Train an 8B model with RLSP's length reward and, in parallel, with a control reward that pays for long but repetitive output while keeping the outcome-verifier component identical. If the control model matches RLSP in response length and shows the same backtracking, verification, and accuracy gains, then length rather than search-oriented exploration is the active ingredient and the emergence claim would be falsified. A second check is to evaluate RLSP's MATH-500 gains only on problems whose trajectories contain at least one backtrack or verification event; if the gains are not concentrated there, the proposed causal link between emergent search and improved reasoning is weakened.","tokens_in":28868,"feed_emoji":"🧠","tokens_out":9352,"duration_ms":76950,"temperature":0.7,"pith_summary":"Reinforcement Learning via Self-Play (RLSP) is a post-training recipe that separates the reward for thinking effort from the reward for a correct answer. The paper's central claim is that a minimal exploration reward—even simply rewarding longer responses that spell out intermediate steps—makes search behaviors such as backtracking, verification, and exploring alternative approaches emerge during PPO training, in some cases with no supervised fine-tuning at all. If correct, this offers a scalable path toward 'thinking' language models: instead of engineering search logic, one needs only an outcome verifier and a weak signal that effort is valued. Empirically, RLSP lifts MATH-500 accuracy by 23% for an 8B model and AIME 2024 accuracy by 10% for a 32B model, and matches or beats majority voting under the same token budget.","feed_headline":"Rewarding longer thinking makes LLMs backtrack and self-correct","feed_subtitle":"Without fine-tuning, rewarding longer answers sparks search behavior and lifts MATH-500 accuracy by 23 percent.","key_machinery":"The load-bearing mechanism is the two-component RLSP reward combined with the PPO update that propagates it. The exploration reward $R_{\\mathrm{ex}}(q,o) = -C/|o|$ is the simplest instance: longer responses that explicitly write out intermediate steps receive higher reward, under the assumption—made precise by the chain-of-thought expressivity results the paper cites—that more intermediate steps increase a transformer's computational power. The outcome-verifier component keeps the search tethered to correct answers and, because it carries 80% of the weight, counteracts reward hacking. The KL penalty to the reference policy is also part of the machinery: since the KL term suppresses long responses, the exploration reward partly exists to counteract that suppression so that long reasoning traces can survive training.","core_discovery":"The paper's central discovery is that emergent reasoning behavior can be induced by decoupling exploration from correctness in the reward signal. During PPO training, the total reward is $R(q,o) = \\alpha \\cdot \\mathbf{1}[\\mathrm{Ver}(q,o)=\\mathrm{True}] + (1-\\alpha) R_{\\mathrm{ex}}(q,o)$, with $R_{\\mathrm{ex}}(q,o) = -C/|o|$ in the simplest implementation. With $\\alpha=0.8$, most of the reward still comes from a binary outcome verifier, but the model also receives a dense, correctness-independent incentive to produce longer, more explicit reasoning trajectories. Across two model families and both math and coding domains, models trained this way produced trajectories that abandon a failed approach, re-examine earlier steps, verify an answer through multiple routes, and try several rationales before concluding—behaviors that did not appear under outcome-reward-only PPO except in one model-and-domain combination. The paper treats these behaviors as emergent relative to standard chain-of-thought, while acknowledging that they are implicitly present, less frequently, in pretraining data, and it interprets RLSP as enabling the model to synthesize novel chain-of-thought traces through self-play.","pith_inferences":["A sharper test of the length-reward theory: stratify MATH-500 by problem difficulty and check whether RLSP's accuracy gain grows with the number of search steps the trajectory actually contains; the chain-of-thought expressivity argument predicts exactly that monotonic relationship.","The paper's own Remark 2 suggests a confound: if backtracking is already present in pretraining data, then the observed behaviors could be unmasked rather than newly created. Comparing models pretrained on corpora with different rates of self-correction, while holding the RLSP reward fixed, would separate these explanations.","The decoupling principle extends naturally to other effort signals—diversity of sampled rationales, number of distinct methods tried, or a learned judge of creativity—and the framework predicts each would induce similar search behavior; this is testable by replacing $-C/|o|$ with each signal and measuring emergent backtracking.","If the theory's self-play story is right, RLSP should compound: iterative rounds of RLSP on the model's own successful long trajectories should keep improving accuracy beyond a single round, which the current single-round experiments do not yet test."],"forward_implications":["Even a crude length-based exploration reward, applied without SFT, elicits backtracking, verification, and alternative-approach search across several base models and domains.","Under a fixed token budget, RLSP-trained models beat self-consistency majority voting: 73.4% versus 61.8% on MATH-500 for the 8B model, and 23.3% versus 20% on AIME 2024 for the 32B model.","The recipe needs only an outcome verifier and an exploration signal, so it can transfer to any domain with verifiable answers without process reward models or hand-constructed search trees.","Because the model generates its own chain-of-thought traces during training, the framework predicts continued self-improvement as long as sufficiently diverse new problems are available.","The decoupled reward structure suggests a scaling recipe: with more compute and more problems, the length signal can be refined into richer effort signals for further gains."],"supporting_citations":[{"why":"It supplies the theoretical premise that chain-of-thought provably increases transformer expressivity with more steps, which motivates the length-based exploration reward.","marker":"[MS23]"},{"why":"It extends the expressivity argument by showing that chain-of-thought lets transformers solve serial problems, tying step count to computational power.","marker":"[LLZM24]"},{"why":"It provides the PPO algorithm that is the RL core of the RLSP training loop.","marker":"[SWD+17]"},{"why":"It supplies the RLHF-style PPO setup that RLSP extends with its decoupled exploration and correctness rewards.","marker":"[OWJ+22]"},{"why":"It provides the MATH dataset used as the training and evaluation ground for the math reasoning experiments.","marker":"[HBK+21]"},{"why":"It supplies the long chain-of-thought reasoning traces used to create the SFT dataset for the 8B experiment.","marker":"[Qwe24c]"},{"why":"It provides the distributed RL training implementation used to run the PPO experiments.","marker":"[HWW+24]"},{"why":"It supplies the concept of auxiliary rewards in reinforcement learning, to which the exploration reward is explicitly related.","marker":"[JMC+16]"},{"why":"It provides the judge model used to score exploration effort in the full RLSP experiments.","marker":"[HLG+24]"}],"fun_headline_variants":["Decoupling rewards sparks emergent reasoning in LLMs","Exploration reward alone triggers backtracking and self-check","RLSP: separate correctness from exploration to teach reasoning","Rewarding exploration, not just answers, makes LLMs think"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that response length is a valid proxy for reasoning effort, so rewarding longer outputs teaches genuine search rather than repetition; the paper itself concedes that models can hack this signal by repeating responses, and that the behaviors it calls emergent are already present, less frequently, in pretraining data.","fun_headline_variants_meta":{"raw":{"variants":["Decoupling rewards sparks emergent reasoning in LLMs","Exploration reward alone triggers backtracking and self-check","RLSP: separate correctness from exploration to teach reasoning","Rewarding exploration, not just answers, makes LLMs think"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1544,"prompt_tokens":1175,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":791,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":791,"tokens_out":369,"duration_ms":3689,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:21:39.981352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an 8B model with RLSP's length reward and, in parallel, with a control reward that pays for long but repetitive output while keeping the outcome-verifier component identical. If the control model matches RLSP in response length and shows the same backtracking, verification, and accuracy gains, then length rather than search-oriented exploration is the active ingredient and the emergence claim would be falsified. A second check is to evaluate RLSP's MATH-500 gains only on problems whose trajectories contain at least one backtrack or verification event; if the gains are not concentrated there, the proposed causal link between emergent search and improved reasoning is weakened.","supporting_citations":[],"review_version":1}