{"id":"93b86284-f714-48f8-940f-fac51da6407a","arxiv_id":"2504.14286","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SRPO, a two-stage GRPO variant with history resampling, achieves 50.0 on AIME24 and 41.6 on LiveCodeBench with Qwen2.5-32B, surpassing DeepSeek-R1-Zero-Qwen-32B in about 1/10 of the training steps.","lead":"Researchers at Kuaishou trained a 32-billion-parameter model with a two-stage reinforcement learning recipe on math first, then coding, and say it beats DeepSeek's R1-Zero on math and coding benchmarks using about a tenth of the training steps. The result is relevant because it suggests cheaper ways to get LLMs to reason across domains.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination check means the headline comparison to DeepSeek-R1-Zero-Qwen-32B could reflect training-set leakage rather than learned reasoning; the 3-point AIME gap is also within evaluation noise.","rationale":"I read the paper as claiming that SRPO's staged math-then-code RL plus history resampling beats R1-Zero-Qwen-32B at 1/10 the training steps. I looked for the least secure condition for that claim. It is not the RL mechanics: Equations 1–3 are standard GRPO and the two-stage idea is plausible; Table 1 supports staged over mixed training. The insecure condition is the external validity of the benchmark numbers: no decontamination, no dataset disclosure, no error bars, and an AIME gap of about one question on a 30-problem set. This is the same weakness the reader flagged, so I agree. I would keep the CONDITIONAL verdict: the paper should be required to provide a decontamination audit or an out-of-distribution evaluation before the superiority and efficiency claims are accepted. I do not see an internal inconsistency that would justify rejection, and the released model weights are a genuine asset for independent checking.","tokens_in":7381,"tokens_out":8340,"duration_ms":77552,"concrete_test":"Run the released SRPO checkpoint and DeepSeek-R1-Zero-Qwen-32B with identical sampling and the same harness on AIME25 and on the LiveCodeBench split released after 2025-01 (e.g., 2025-02 onwards), which post-date the reported training data; if SRPO does not maintain a comparable or better margin on these out-of-distribution sets, the claimed superiority on AIME24 and LiveCodeBench is not established as genuine generalization, and the authors should then release the training set for an n-gram overlap audit against AIME24 and LiveCodeBench test prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—SRPO surpasses DeepSeek-R1-Zero-Qwen-32B on AIME24 and LiveCodeBench with roughly 1/10 the training steps—is a comparison against externally published numbers, and its validity depends on the SRPO training data being disjoint from the evaluation sets. The paper never reports a decontamination check, does not name the HuggingFace datasets used, and releases no training data. The difficulty filter in §3.3 makes this omission more acute: samples in which all rollouts are wrong are retained, and AIME-level competition problems are exactly the 'hard' items that would survive repeated epochs. If AIME24 or LiveCodeBench (2024-08–2025-01) prompts appear in the source pools, pass@1 measures retrieval rather than generalization, and the efficiency claim is likewise an artifact. AIME24 has only 30 problems, so the 50.0 vs 47.0 gap is roughly one question; without seed variance or a matched evaluation harness the 'surpasses' wording is already fragile. The model weights are released, which is real support, but the data-side leak risk is unresolved and is the single most load-bearing unchecked assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SRPO, a GRPO-based reinforcement learning recipe for training LLM reasoning across mathematics and coding. The method combines two innovations: a two-stage curriculum that trains on math first and then introduces code, and an epoch-level History Resampling mechanism that removes prompts for which all rollouts are correct while retaining hard all-incorrect prompts. Using Qwen2.5-32B-Base as the initial checkpoint, the authors report 50.0 pass@1 on AIME24 and 41.6 pass@1 on LiveCodeBench (2024-08 to 2025-01), compared with 47.0 and 40.2 for DeepSeek-R1-Zero-Qwen-32B, using roughly 1/10 of the training steps. The paper also describes a data curation pipeline, a rule-based reward design, training dynamics, and qualitative observations of self-reflection and code-assisted verification behaviors.","tokens_in":7627,"tokens_out":4934,"duration_ms":41891,"significance":"If the headline results hold, SRPO is a valuable engineering contribution: it demonstrates that a carefully staged curriculum, data filtering, and history resampling can match or exceed an open R1-Zero result on both math and code benchmarks at a fraction of the training steps, and the model weights are publicly released. The qualitative analysis of emergent reflection patterns and code-assisted verification is also useful for the community. However, the empirical claims currently rest on a single run without variance estimates, on a comparison against externally published baseline numbers, and on training data whose provenance and overlap with the evaluation sets are not reported. These gaps directly affect the strength of the stated superiority and efficiency conclusions, making the current evidence promising but not yet conclusive.","major_comments":[{"comment":"The central claim that SRPO surpasses DeepSeek-R1-Zero-Qwen-32B rests entirely on externally published numbers (47.0 AIME24, 40.2 LiveCodeBench) compared with a single SRPO run. Since AIME24 contains only 30 problems, the 3.0-point gap corresponds to roughly one question, and no seed variance, confidence interval, or statistical test is reported. Please report results over multiple seeds or provide bootstrap/permutation intervals, and state the exact evaluation harness, decoding settings, and LiveCodeBench version and date range used for both models; otherwise the 'surpasses' wording is not statistically supported.","section":"Section 4.2, Table 2"},{"comment":"The paper never names the HuggingFace datasets used for training and reports no decontamination check. Because the data is collected from public sources and the History Resampling rule in Section 3.3 deliberately retains hard all-incorrect samples over multiple epochs, any overlap between the training pool and AIME24 or LiveCodeBench (2024-08 to 2025-01) would inflate pass@1 through memorization rather than learned reasoning. Please name the source datasets, quantify overlap with the evaluation sets (e.g., exact-match and n-gram-match statistics), and release the training data or a decontaminated version; this is load-bearing for both the superiority and the 1/10-step efficiency claims.","section":"Section 3.4 and Section 4.2"},{"comment":"The effectiveness of History Resampling on the final benchmark scores is not demonstrated. Table 1 compares Naive Mixed Training with Staged Training only under the 'w/o HR' condition, while the headline scores in Table 2 come from the full SRPO that includes HR. Add an ablation with and without HR under the same staged-training protocol, reporting AIME24, LiveCodeBench, and total training steps, so the contribution of HR to the final result and to the efficiency claim can be quantified.","section":"Section 3.3 and Table 1"},{"comment":"The efficiency claim of 'about 1/10 of the training steps' is not substantiated in comparable units. The paper reports 840 Stage-1 steps followed by an unspecified number of Stage-2 steps, and compares to DeepSeek-R1-Zero's step count without reporting batch size, tokens processed, or hardware for either run. Please report the total number of optimizer steps, tokens processed, and wall-clock time (or a normalized compute measure) for both SRPO and the DeepSeek baseline, and state clearly whether the 1/10 factor refers to optimizer steps or to overall compute.","section":"Section 4.1"}],"minor_comments":[{"comment":"State whether the '32 samples per question' evaluation is applied identically to the DeepSeek baseline and SRPO, and whether the reported DeepSeek numbers use the same decoding temperature and maximum length.","section":"Figure 1 and Section 4.2"},{"comment":"The reward components R_format and Penalty_mix are described, but the training section does not give the exact numerical combination formula or any weighting; please provide the complete reward expression used in the experiments.","section":"Section 3.5"},{"comment":"The paper says the KL term is removed from the GRPO loss, but Eq. (1) includes a KL penalty with coefficient beta; clarify whether beta was set to 0 or the term was otherwise disabled, and whether Eq. (2) still refers to the reference policy.","section":"Section 4.1"},{"comment":"There are minor presentation issues: 'optimazition' in the Figure 16 caption is a typo, the phrase 'two-Staged' is capitalized inconsistently between the abstract and Section 3.2, and some equations in the preprint render with unusual spacing (e.g., Eq. (3)); a proofreading pass is needed.","section":"Figure 16 and Abstract"},{"comment":"The wording claims to 'fully reproduce DeepSeek-R1's performance,' but the experimental comparison is only to DeepSeek-R1-Zero-Qwen-32B, not to the full DeepSeek-R1 model that includes cold-start SFT; please adjust the claim to match the actual baseline.","section":"Abstract and Contributions"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is closer to a system report than a rigorous scientific study, and its strongest assets are the released weights, the clear recipe, and the useful qualitative observations. The two most serious issues are the absence of any decontamination analysis and the reliance on a single run with no variance estimates; both are addressable with additional experiments and reporting. I would not recommend rejection because the central recipe is plausible and the gaps are fixable, but the authors should be required to provide the missing evidence before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful empirical recipe for cross-domain RL on a 32B model, but the 'surpasses DeepSeek-R1-Zero' headline is built on a single run and an unverified assumption that training data didn't contaminate the eval sets. The contribution is real; the evidence is not yet at the level of the claim.\n\nWhat's actually new: the two-stage math-then-code training paradigm and epoch-level history resampling are simple, well-motivated ideas. The paper does a good job explaining why mixing math and code data from the start hurts both response length and benchmark performance – the length-conflict story is plausible and backed by the Table 1 ablation (44.3 vs 40.5 on AIME24 between staged and mixed training). Releasing the 32B weights on HuggingFace is real support; people can check the model themselves.\n\nSoft spots, in order of severity. First, the central comparison rests on externally published numbers for DeepSeek-R1-Zero-Qwen-32B. AIME24 has only 30 problems, so 50.0 vs 47.0 is roughly one question; LiveCodeBench is a bit more robust but the same concern about evaluation-harness matching applies. There are no error bars, no multiple seeds, no statistical tests. Second, and more load-bearing: the paper never reports a decontamination check, and the training data sources are unnamed. Given that the data pipeline deliberately filters for 'hard' problems and retains all-wrong samples across epochs, AIME-style competition problems could survive in the training pool by construction. If that happened, the pass@1 numbers are retrieval, not generalization, and the 1/10-steps claim collapses. This is a real risk, not a minor quibble. Third, there is no ablation isolating History Resampling's contribution to the final scores; the training dynamics plots are suggestive but not conclusive.\n\nThe reflection-pattern analysis in Section 4.4 is interesting and worth publishing, though it is descriptive rather than causal.\n\nWho this is for: people working on RL for reasoning who want a cheaper recipe than a full R1-Zero-scale run. It deserves a serious referee: the method is plausible, the model is released, and the efficiency claim is important if it holds. But the authors should be pushed to name their data sources, run a decontamination check, report multiple seeds, and ideally release the training data or at least a data-filtering script. I'd accept it for peer review with major revisions required.\n\nBottom line: worth engaging with, but do not yet treat the headline numbers as established.","headline":"A useful empirical recipe for cheaper cross-domain LLM RL, but the headline comparison to DeepSeek-R1-Zero is not yet solid: single-run numbers, no decontamination check, and a gap on AIME24 that is roughly one question.","tokens_in":8248,"tokens_out":1630,"would_cite":true,"duration_ms":15679,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reinforcement-learning recipe that trains a 32B model on math first and code second, skipping samples the model already solves, beats a published R1-Zero baseline on AIME24 and LiveCodeBench while using roughly one-tenth of the training…","keywords":["reinforcement learning","LLM reasoning","GRPO","two-stage training","history resampling","mathematical reasoning","code generation","chain-of-thought"],"falsifier":"Run the exact SRPO data pipeline and training recipe, then measure on a freshly released math and code benchmark that cannot have appeared in training data; if the 50.0 AIME24 / 41.6 LiveCodeBench scores fall to or below the published baseline, the reported lead is explained by training-data overlap rather than by two-stage training and history resampling.","tokens_in":7165,"feed_emoji":"🧠","tokens_out":7759,"duration_ms":64050,"temperature":0.7,"pith_summary":"This paper argues that large-scale RL for reasoning does not need to mix all domains into one training run. By training a 32B base model first on hard mathematical problems and then introducing code, SRPO lets the model develop long chain-of-thought behavior that transfers to programming. An epoch-level history resampling step removes problems the current policy already solves perfectly, keeping reward variance non-zero so gradient updates stay informative. The reported result is 50.0 pass@1 on AIME24 and 41.6 on LiveCodeBench, surpassing the reference R1-Zero baseline of 47.0 and 40.2 with the same base model in about one-tenth of the steps. If the comparison holds, a carefully staged curriculum and data filtering can substitute for much larger RL compute budgets.","feed_headline":"Two-stage RL training cuts steps tenfold and beats baseline","feed_subtitle":"Math-first, code-second training plus resampling passes R1-Zero scores with one-tenth of the steps.","key_machinery":"The load-bearing object is SRPO — two-Staged history-Resampling Policy Optimization — built on GRPO (Group Relative Policy Optimization), where each prompt's G rollouts produce rewards normalized within the group into advantages. The two-stage schedule makes math-only training the first stage, so long chain-of-thought behaviors such as rechecking and backtracking develop before code is introduced; the paper reports that mixed training stagnates in response length while staged training grows. History Resampling is an epoch-level filter that drops samples with all-correct rollouts and keeps mixed or all-wrong samples, maintaining nonzero group variance and aligning with curriculum learning. A rule-based reward (format score, math accuracy score, code test-case score) and removal of the KL term encourage exploration.","core_discovery":"The central claim is that the two difficulties that block vanilla GRPO training on mixed math and code data—conflicting response-length incentives and vanishing advantages—are both addressable by training design rather than by more compute. SRPO stages the curriculum: Stage 1 trains only on math to elicit long reflective chains, and Stage 2 adds code; the paper reports that this staged order clearly beats math-only, code-only, and naive mixed training. History Resampling filters each epoch's dataset by removing questions whose every rollout is correct, so each sampled group retains positive reward variance and a meaningful advantage signal. With these mechanisms, plus a curated difficulty-tiered dataset and a rule-based reward, the final model reaches 50.0 pass@1 on AIME24 and 41.6 on LiveCodeBench, surpassing the published R1-Zero baseline of 47.0 and 40.2 on the same base model after about one-tenth of the baseline's training steps.","pith_inferences":["Editorial: If decontamination checks confirm no leakage, the implication generalizes beyond math and code: any two domains with different response-length incentives might be staged similarly.","Editorial: History Resampling is orthogonal to dynamic rollout-count methods; combining both could further reduce compute while preserving gradient quality.","Editorial: The spontaneous code-to-verify-math behavior suggests RL can induce simple tool use from pure outcome rewards, potentially extending to search or calculators without explicit tool-use training.","Editorial: The report gives no seed variance or error bars, so the true size of the 2.0–3.0 point lead over the baseline is unknown; multi-seed reruns would reveal whether staging and resampling are robust."],"forward_implications":["A math-first, code-second curriculum order can transfer reasoning skills across domains within a single policy, making training order a direct lever for cross-domain RL.","History Resampling turns too-easy data into a recoverable compute saving: removing it keeps gradients informative without dynamically adjusting rollout counts.","Models trained this way spontaneously emit verification behavior, such as writing code to check a math answer, suggesting self-correction can be elicited through rewards rather than hand-crafted prompts.","The same recipe reaches higher benchmark scores on both math and code with roughly one-tenth of the baseline's training steps, making strong reasoning RL more accessible at smaller compute budgets."],"supporting_citations":[{"why":"Sets the baseline scores (47.0 AIME24, 40.2 LiveCodeBench) and the no-CoT training template that SRPO aims to reproduce and beat.","marker":"[3]"},{"why":"Provides the DAPO GRPO-based method that SRPO says it matches on AIME24, setting the comparison to the current best GRPO variant.","marker":"[12]"},{"why":"Defines GRPO, the group-relative advantage estimator whose zero-variance failure mode SRPO targets.","marker":"[13]"},{"why":"Supplies the LiveCodeBench benchmark and version used for the code pass@1 evaluation.","marker":"[14]"},{"why":"Specifies the Qwen2.5-32B base model shared by SRPO and the baseline, making the comparison apples-to-apples.","marker":"[15]"},{"why":"Supplies the curriculum-learning rationale for keeping hard samples and dropping all-correct samples in history resampling.","marker":"[16]"},{"why":"Documents the prior math-only reproduction effort whose domain gap SRPO addresses.","marker":"[11]"}],"fun_headline_variants":["SRPO: 10% steps, beats R1-Zero on AIME and LiveCodeBench","Staged RL + resampling: 1/10 steps, higher scores","Math-first, code-second RL: 10x fewer steps, beats R1-Zero","History resampling + two-stage curriculum: 90% fewer steps, higher scores","Two-stage RL beats R1-Zero with 10% of the training steps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that the published baseline scores were measured under comparable conditions and that SRPO's training data does not overlap the AIME24 or LiveCodeBench test sets; no decontamination check or variance estimate is reported, so leakage or an evaluation mismatch would erase the efficiency and superiority claims.","fun_headline_variants_meta":{"raw":{"variants":["SRPO: 10% steps, beats R1-Zero on AIME and LiveCodeBench","Staged RL + resampling: 1/10 steps, higher scores","Math-first, code-second RL: 10x fewer steps, beats R1-Zero","History resampling + two-stage curriculum: 90% fewer steps, higher scores","Two-stage RL beats R1-Zero with 10% of the training steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000838,"raw_usage":{"total_tokens":3656,"prompt_tokens":950,"completion_tokens":2706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":2595}},"tokens_in":566,"tokens_out":2706,"duration_ms":18106,"temperature":1.0,"reasoning_tokens":2595,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:51:56.383981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact SRPO data pipeline and training recipe, then measure on a freshly released math and code benchmark that cannot have appeared in training data; if the 50.0 AIME24 / 41.6 LiveCodeBench scores fall to or below the published baseline, the reported lead is explained by training-data overlap rather than by two-stage training and history resampling.","supporting_citations":[{"cited_title":"Livecodebench: Holistic and con- tamination free evaluation of large language models for code","cited_arxiv_id":null,"evidence_quote":"Supplies the LiveCodeBench benchmark and version used for the code pass@1 evaluation."},{"cited_title":"Curriculum learning for reinforcement learning domains: A framework and survey","cited_arxiv_id":null,"evidence_quote":"Supplies the curriculum-learning rationale for keeping hard samples and dropping all-correct samples in history resampling."}],"review_version":1}