{"id":"326a04a1-33da-4a3e-9988-d9a85792de2b","arxiv_id":"2508.20722","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 14B model trained with agentic RL and a resample-on-correct rollout strategy scores 80.6% on AIME24 and 69.8% on AIME25, nearly matching DeepSeek-R1 (671B) in one week on 64 GPUs.","lead":"rStar2-Agent trains a 14B language model with a new reinforcement learning scheme, GRPO-RoC, to solve math problems by writing and running Python code. It reports AIME24/AIME25 scores comparable to the 671B DeepSeek-R1 after only 510 training steps on 64 GPUs, but the comparison has statistical and protocol caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed margin over DeepSeek-R1 is within sampling noise on a 30-problem benchmark and reverses on AIME25; 'surpassing' needs matched protocols and confidence intervals.","rationale":"The reader's weakest assumption is that published baseline scores are directly comparable under the same protocol. I agree this is a real risk, but I think an even more basic and load-bearing issue is statistical: the reported superiority over DeepSeek-R1 is a 0.8-point gap on a 30-question benchmark, and the model actually trails on AIME25. Without confidence intervals or per-problem data, the headline 'surpassing DeepSeek-R1' is not established even if protocols were perfectly matched. This is closely related to the reader's point about baseline comparability—both concern the validity of the cross-model comparison—but the statistical-power issue is more fundamental because it affects even the authors' own point estimates. The paper is otherwise a strong empirical report: it documents failed attempts, provides ablations for GRPO-RoC, and reports infrastructure details. The concern does not undermine the plausibility of the training recipe; it only undermines the specific superiority claim in the abstract. Therefore I keep the reader's CONDITIONAL verdict unchanged, with partial agreement on the weakest assumption.","tokens_in":23282,"tokens_out":4925,"duration_ms":49920,"concrete_test":"Compute bootstrap 95% confidence intervals for the AIME24 and AIME25 pass@1 difference between rStar2-Agent-14B and DeepSeek-R1, using the per-problem pass@1 scores from the paper's 16-sample evaluation protocol. If the AIME24 confidence interval includes 0, or if the AIME25 difference is negative outside the interval, the abstract's 'surpassing DeepSeek-R1' claim is not statistically supported. A second check: re-run DeepSeek-R1 under the exact rStar2-Agent evaluation protocol (same temperature, sample count, prompt, and with/without code tools held constant) and compare directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—'surpassing DeepSeek-R1 (671B)'—rests on Table 3, where rStar2-Agent-14B leads DeepSeek-R1 by 0.8 points on AIME24 (80.6 vs 79.8) but trails by 0.2 points on AIME25 (69.8 vs 70.0). AIME24 and AIME25 each contain only 30 problems, and pass@1 is averaged over 16 samples per problem. The standard error of a 30-problem mean at p≈0.8 is roughly sqrt(0.8·0.2/30) ≈ 7.3 percentage points, so a 0.8-point gap is far smaller than one standard error. Even using per-problem variance, the difference is not statistically distinguishable from chance. Additionally, the paper does not report confidence intervals, seeds, or per-problem scores, and the reported checkpoint is selected by evaluation scores, which inflates the expected peak. The comparison also mixes protocols: baseline numbers for DeepSeek-R1 and others are taken from external reports that may use different sampling temperatures, sample counts, prompt templates, or allow or disallow code tools. If any of these differ, the 'surpassing' statement is not supported by the data as presented. This is the most load-bearing weakness because the abstract's headline and the paper's contribution claim depend directly on these cross-model comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces rStar2-Agent, a 14B math reasoning model trained with agentic reinforcement learning in a Python code environment. The core contributions are: (1) an efficient multi-turn RL infrastructure on 64 MI300X GPUs; (2) GRPO-RoC, a GRPO variant that oversamples 2G rollouts and then resamples positive trajectories toward high-quality, low tool-error traces while downsampling failures; and (3) a three-stage training recipe that starts from non-reasoning SFT and progressively increases response length and data difficulty. The authors claim 510 RL steps produce state-of-the-art AIME24/AIME25 pass@1 scores (80.6/69.8), surpassing DeepSeek-R1 (671B) with shorter responses, plus strong generalization to science, alignment, and tool-use benchmarks. The paper also reports unsuccessful attempts and a previously unreported training-collapse failure mode at the model's reasoning ceiling.","tokens_in":23636,"tokens_out":7099,"duration_ms":71674,"significance":"If the central claims hold, the paper is significant: it demonstrates that small models can reach frontier-level math reasoning with substantially less compute than large reasoning models, and it provides a concrete infrastructure and algorithm recipe. The paper is unusually honest in reporting failed interventions and the observed collapse under continued RL, which is valuable for the community. The main algorithmic result—GRPO-RoC improves tool-use quality and training efficiency over vanilla agentic GRPO—is broadly credible from the training curves, though the specific ablation is confounded (see major comments). The headline cross-model comparison, however, is not currently supported by the evidence presented.","major_comments":[{"comment":"The abstract's central claim that rStar2-Agent-14B 'surpasses DeepSeek-R1 (671B)' is not supported by the evidence as presented. AIME24 and AIME25 each contain only 30 problems, and the pass@1 numbers are averaged over 16 samples per problem. The 0.8-point lead on AIME24 and the 0.2-point deficit on AIME25 are both within sampling noise. The paper reports no confidence intervals, per-problem scores, or multiple seeds. Moreover, baseline numbers for DeepSeek-R1, o3-mini, Claude-Opus-4.0, and QWQ-32B are taken from external sources whose sampling temperature, number of samples, prompt template, and tool-use policy are not shown to match the authors' protocol. The 'surpassing' statement should either be replaced by 'competitive with' or supported by a matched-protocol head-to-head evaluation with uncertainty quantification.","section":"§5.2, Table 3"},{"comment":"The total number of RL steps is internally inconsistent. Stage 1 is 300 steps and Stage 3 is 125 steps; Stage 2 is described as using the latest policy 'from the final 385 steps of Stage 2', which implies a Stage 2 length of at least 385 steps, for a total of at least 810. The abstract, Table 2, and §4.3 state 510 RL steps. If '385' is a typo for '85', this must be corrected; if Stage 2 actually ran 385 steps, the 510-step claim and the compute-efficiency comparison in Table 2 need to be revised.","section":"§4.3, Table 2, Abstract"},{"comment":"The ablation of Resample-on-Correct is confounded by the oversampling factor. GRPO-RoC generates 2G=32 rollouts per problem and then selects G=16, while the 'GRPO with Tool' baseline generates only G=16 rollouts and uses all of them. The observed gains could therefore be due to the larger candidate pool rather than the asymmetric quality filtering. An additional baseline that generates 32 rollouts and uses all of them—or otherwise controls for candidate count—is needed to attribute the improvement specifically to the RoC selection mechanism.","section":"§5.3, Fig. 9"},{"comment":"The final checkpoint is selected using the same AIME24/AIME25 benchmarks on which the headline numbers are reported. Fig. 8 shows evaluation scores being used to decide when to stop, and §4.3 states that 'performance begins to saturate and can even decline, so we stop at 125 steps.' This early stopping on the test set inflates the expected peak relative to a fixed training budget. Please report scores at a pre-specified step or use a held-out validation split for checkpoint selection, and acknowledge the selection effect in the reported pass@1 numbers.","section":"§4.3, §5.2"}],"minor_comments":[{"comment":"The text says the environment service achieves '45 calls per step' while also reporting 45K concurrent tool calls per step. This should be 45K; the inconsistency is confusing.","section":"Fig. 6 and §3.1"},{"comment":"'V APO' is misspelled; should be 'VAPO'.","section":"Table 6"},{"comment":"The normalization in the GRPO-RoC objective is unclear: after length-normalizing each trajectory inside the sum, the outer denominator appears to be the total selected token count. Please clarify the exact normalization and how it relates to the standard GRPO objective.","section":"Eq. (6)"},{"comment":"Several references are incomplete: [An et al.] lacks a year and venue, [Kimi] is a URL-only citation, and [Tulu3], [AIME], and [Huggingface] are datasets/websites without access dates or formal bibliographic information.","section":"References"},{"comment":"The figures are described as highlighting top 20% high-entropy tokens in green, but the text-only rendering loses this color cue. Consider adding a textual marker or a separate annotation for accessibility.","section":"Fig. 10 and Fig. 11"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline claim is too strong for the evidence in Table 3, and the step-count inconsistency directly affects the abstract's compute claim. I would be comfortable with acceptance after the authors add matched-protocol comparisons with confidence intervals, fix or clarify the training step count, and strengthen the RoC ablation to control for oversampling. The training-recipe and infrastructure contributions are credible and valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: this is a well-executed technical report that delivers what it promises—a concrete recipe for agentic RL with Python tools that reaches strong math performance on a 14B model in 510 steps. The GRPO-RoC idea is real: oversample 2G, downweight positive trajectories with tool errors or format issues, keep negative diversity. The ablation against vanilla GRPO with tools shows clear gains, and the infrastructure details (45K concurrent tool calls, 0.3s latency, load-balanced KV-cache scheduling) are practical contributions. The honest write-up of failed attempts (overlong filtering, n-gram repetition) is refreshing and useful.\n\nThe soft spots are the usual ones for a technical report that wants attention. The 'surpassing DeepSeek-R1' claim is load-bearing for the abstract, and it's not actually supported: the margins on AIME24 (0.8 points) and AIME25 (-0.2 points) are within noise for 30-problem benchmarks, and the baseline numbers are taken from other papers with no evidence the evaluation protocols match (temperature, samples, prompt, whether tools were allowed). The paper reports a checkpoint selected by evaluation scores, which inflates the expected peak. That said, these flaws don't sink the main message: the algorithm and recipe are clear, and the comparisons within the paper (GRPO vs GRPO-RoC, same base model, same protocol) are the right evidence.\n\nThere's also a selection issue: hyperparameters and stopping point were tuned on the same AIME benchmarks, and the code/data release isn't locatable yet from the preprint. But for a technical report, this is arguably acceptable if the recipe is reproducible—and it should be, given the level of detail.\n\nIn short, this paper is for people who want to train reasoning models with tools on a budget. It deserves serious refereeing, but the evaluation section needs revision before publication: matched baselines, confidence intervals or at least per-problem scores, and a headline that doesn't overclaim.","headline":"A solid, honest agentic RL recipe with a new resampling method, but the headline 'surpassing DeepSeek-R1' is noise on 30-problem benchmarks and should be tempered.","tokens_in":24189,"tokens_out":1974,"would_cite":true,"duration_ms":18819,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"rStar2-Agent claims that 510 steps of agentic reinforcement learning—a model writing and running Python to check its own reasoning—lift a 14B model to 80.6% pass@1 on AIME24, matching or beating the 671B DeepSeek-R1 with far shorter answers","keywords":["agentic reinforcement learning","GRPO-RoC","resample-on-correct","mathematical reasoning","code-interpreter tool use","14B reasoning model","AIME benchmarks","training efficiency"],"falsifier":"Re-run DeepSeek-R1 (and, if accessible, o3-mini and Claude Opus 4.0) on AIME24 and AIME25 using the paper's exact prompt template, 16 samples, temperature 0.6, and boxed-answer extraction: if matched-condition scores come out above 80.6% on AIME24 or above the paper's cross-benchmark average, the 'surpasses DeepSeek-R1' headline fails. A cheaper partial check is to reproduce Stage 1 from the released code and data and confirm AIME24 rises from the ~3.3% SFT baseline toward ~72% within 300 steps.","tokens_in":23188,"feed_emoji":"🧮","tokens_out":13600,"duration_ms":120916,"temperature":0.7,"pith_summary":"The paper tries to establish that agentic reinforcement learning—training a model to write and run Python code, then reason over the execution results—can by itself turn a 14B pre-trained model into a frontier-level math reasoner, without reasoning-heavy SFT and without huge model scale. The reported result is that rStar2-Agent-14B reaches 80.6% average pass@1 on AIME24 and 69.8% on AIME25 after only 510 RL steps on 64 GPUs within one week, which on average edges past the 671B DeepSeek-R1 (75.2% vs 74.9% across those two benchmarks) while answering in roughly 9–11K tokens versus the 14–17K the paper measures for other reasoning models. The mechanism is GRPO-RoC, a rollout-resampling strategy that filters noisy tool-use trajectories under pure answer-only rewards, plus a Python execution service that keeps thousands of concurrent tool calls fast and isolated, and a three-stage recipe that starts with a non-reasoning cold-start SFT. If correct, the work matters because it maps a concrete path to near-frontier reasoning on commodity hardware and shows that tool-augmented, shorter reasoning can beat longer chain-of-thought at the same accuracy.","feed_headline":"510 RL steps lift a 14B model past a 671B reasoning rival","feed_subtitle":"Python tools plus resample-on-correct RL reach frontier AIME24 scores in a week of GPU time.","key_machinery":"The load-bearing mechanism is GRPO-RoC, an RL algorithm pairing Group Relative Policy Optimization with a Resample-on-Correct rollout strategy. For each question it oversamples 2G multi-turn rollouts in the Python environment, then downsamples to G: negative trajectories are uniformly downsampled to keep diverse failure modes, while positive trajectories are preferentially kept when they have low tool-call error rates and clean formatting. Under an answer-only 0/1 reward, this asymmetric selection keeps the policy from treating buggy tool calls as acceptable; the paper shows it lowers tool-error rates in rewarded trajectories, raises accuracy, and shortens responses relative to vanilla GRPO","core_discovery":"The central discovery is that the bottleneck in agentic RL is the noise the tool environment injects into otherwise-correct trajectories: under naive GRPO, correctly answered rollouts still contain tool-call errors 10–15% of the time, and the model learns to tolerate them. GRPO-RoC oversamples 2G rollouts per question, then resamples down to G—failures kept uniformly to preserve error diversity, successes kept preferentially when clean and error-free. The authors report that this asymmetric selection cuts tool errors, raises AIME accuracy, and shortens responses, in only 510 RL steps, because the cold-start SFT teaches formatting and function-calling rather than reasoning. They also record t","pith_inferences":["The RoC principle is reward-design-agnostic: oversampling rollouts, then preferring clean successes while retaining diverse failures, should reduce 'correct-but-sloppy' policy drift in any tool-augmented RL setting with sparse rewards—code generation, formal proof, or web-search agents—not just Python math.","A matched-protocol replay is the decisive test of the headline: re-running DeepSeek-R1 and other baselines under the paper's exact prompt, 16 samples, and temperature 0.6 would confirm or overturn the 'surpasses DeepSeek-R1' claim, which currently rests on comparing against published numbers.","The reported post-peak collapse suggests the efficient frontier for this recipe is a step budget near saturation; teams with limited GPU time should stop at the accuracy knee rather than continue training, since extra steps measurably hurt.","Because the recipe only needs verifiable integer answers, the same pipeline should transfer to any domain with cheap correctness checks where a model can run tools to verify its own work, such as code correctness or data-cleaning tasks with checksums."],"forward_implications":["Frontier AIME-class math scores become reachable at 14B scale with one week of training on 64 GPUs, cutting the cost of a top-tier math reasoner by orders of magnitude versus 671B-scale training.","Tool-checked reasoning can be shorter than long CoT: at similar or better accuracy the model uses roughly 9–11K tokens per AIME response versus the 14–17K the paper measures for other reasoning models, lowering per-query inference cost.","Reasoning-heavy SFT is not needed: a cold-start SFT that only teaches formatting, function calling, and instruction following, followed by agentic RL, takes the model from 3.3%/0% to 80.6%/69.8% on AIME24/AIME25.","Math-only agentic RL transfers: GPQA-Diamond science reasoning rises from 42.1% to 60.9%, above DeepSeek-V3, implying the induced behaviors are general reasoning skills rather than memorized math templates.","There is a per-model ceiling: training past the 510-step peak collapses reward and accuracy despite multiple attempted fixes, so efficient recipes should aim to reach a base model's ceiling quickly rather than train long."],"supporting_citations":[{"why":"DeepSeek-R1: the 671B reasoning model the paper claims to surpass, and the outcome-reward RLVR paradigm the method extends.","marker":"[Guo et al., 2025]"},{"why":"Supplies GRPO, the group-relative optimization algorithm that GRPO-RoC modifies with resampling.","marker":"[Shao et al., 2024]"},{"why":"DAPO: source of the Clip-Higher exploration trick, the overlong-filtering baseline the paper tests and rejects, and part of the RL training data.","marker":"[Yu et al., 2025]"},{"why":"ReTool: the closest prior tool-augmented RL system, the main agentic baseline to beat at the 32B scale.","marker":"[Feng et al., 2025]"},{"why":"Supplies the high-entropy forking-token analysis used to interpret reflection behaviors, and the DAPO-Qwen-14B CoT baseline.","marker":"[Wang et al., 2025]"},{"why":"Qwen3 technical report: provides the Qwen3-14B-base model that the recipe boosts to frontier scores.","marker":"[Team, 2025]"},{"why":"ZTRL: an agentic RL baseline showing code-tool rollouts beat CoT-only RL at 32B scale, compared in Table 6.","marker":"[Mai et al., 2025]"},{"why":"The RL training framework the infrastructure extends; its static rollout allocation is the inefficiency the load-balanced scheduler fixes.","marker":"[Sheng et al., 2024]"},{"why":"The serving engine used for generation; its KV-cache limits motivate the dynamic rollout scheduler.","marker":"[Zheng et al., 2024]"}],"fun_headline_variants":["14B model beats 671B rival in 510 RL steps","Tool-noise fix: 14B model tops 671B on AIME in a week","510 steps to frontier: small model outshines 671B","Clean rollouts, not size: 14B beats 671B at math"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the published AIME scores for DeepSeek-R1, o3-mini, and Claude Opus 4.0 were produced under the same evaluation protocol the paper uses—16 samples at temperature 0.6 with the same prompt and answer extraction; if those baselines were scored differently, the claim of surpassing them does not follow.","fun_headline_variants_meta":{"raw":{"variants":["14B model beats 671B rival in 510 RL steps","Tool-noise fix: 14B model tops 671B on AIME in a week","510 steps to frontier: small model outshines 671B","Clean rollouts, not size: 14B beats 671B at math"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001024,"raw_usage":{"total_tokens":4195,"prompt_tokens":828,"completion_tokens":3367,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":3284}},"tokens_in":572,"tokens_out":3367,"duration_ms":25076,"temperature":1.0,"reasoning_tokens":3284,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:52:04.546960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run DeepSeek-R1 (and, if accessible, o3-mini and Claude Opus 4.0) on AIME24 and AIME25 using the paper's exact prompt template, 16 samples, temperature 0.6, and boxed-answer extraction: if matched-condition scores come out above 80.6% on AIME24 or above the paper's cross-benchmark average, the 'surpasses DeepSeek-R1' headline fails. A cheaper partial check is to reproduce Stage 1 from the released code and data and confirm AIME24 rises from the ~3.3% SFT baseline toward ~72% within 300 steps.","supporting_citations":[],"review_version":1}