{"id":"b7415aa2-a1a3-4a5d-a2e8-38dec2b4dadb","arxiv_id":"2507.07451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Replaying correct trajectories from an earlier RL run during GRPO training improves convergence speed and final accuracy on math reasoning benchmarks.","lead":"This paper proposes RLEP, a way to train math reasoning models by replaying previously verified correct answers during reinforcement learning. The authors report faster convergence and higher accuracy on AIME and AMC benchmarks with a Qwen 7B model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RLEP's 'faster convergence' and 'stronger final performance' claims omit the 400-step experience-collection run, so RLEP uses at least 535 steps to reach a 400-step baseline peak; final gains are confounded by total compute.","rationale":"The reader's weakest_assumption focuses on the off-policy bias in Eq. (4), which is a real theoretical flaw but is explicitly separated from the accuracy claim. I agree that flaw matters, but the more load-bearing issue for the paper's headline is the missing compute-matched control: the experience pool is generated by a full 400-step baseline run, and the replay phase restarts from scratch, so RLEP's total training budget is about double that of the reported baseline. The 'faster convergence' figure of 135 vs. 380 steps ignores the 400-step seed run, and the final accuracy comparison is confounded by total compute. A simple control — train the baseline for the same total number of steps or continue it for another 400 steps — would settle whether replay provides any benefit beyond additional training. The paper does provide code, checkpoints, and a detailed recipe, which is creditable, but the missing control undermines the causal claim. The reader's CONDITIONAL verdict remains appropriate; I would keep it based on this compute-confound concern, hence UNCHANGED.","tokens_in":7052,"tokens_out":11208,"duration_ms":115327,"concrete_test":"Train DAPO-nodyn-bs64 for 800 total PPO steps (or equal wall-clock to RLEP's full pipeline) and compare its peak and final AIME-2024/AIME-2025/AMC-2023 accuracy and convergence step against RLEP; also report RLEP's total PPO steps and wall-clock including the 400-step experience-collection run. If the extended baseline matches RLEP's final accuracy and reaches the same peak in similar total steps, the reported gains are a compute artifact rather than a replay effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that replaying high-quality examples yields faster convergence and higher final performance. The reported comparison, however, is not compute-matched. In Sec. 4.2, the experience pool is built by running DAPO-nodyn-bs64 for 400 PPO steps; the replay phase then restarts from the base model and requires at least 135 steps to match the baseline's peak (and a full replay run of about 400 steps for the final numbers). Thus RLEP's total training cost is at least 400 + 135 = 535 steps to match a 400-step baseline peak, and roughly 800 steps for the final reported accuracy. The abstract's 'substantially fewer updates' counts only replay-phase updates and omits the mandatory seed run. Moreover, because the baseline is only trained for 400 steps while RLEP enjoys the seed run plus a full replay run, the higher final accuracy on AIME/AMC could simply reflect roughly twice the total training updates, not a benefit of replay. No control is reported — e.g., a baseline trained for the same total steps, or a baseline continued for another 400 steps. Without such a control, neither 'faster convergence' nor 'stronger final performance' is established as an effect of experience replay rather than of additional compute.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RLEP, a two-phase reinforcement-learning-with-verifiable-rewards method for LLM reasoning. Phase 1 runs a standard GRPO-style baseline (DAPO-nodyn-bs64) for 400 PPO steps and uses the resulting seed policy to decode trajectories, retaining only rule-verified correct solutions in an experience pool. Phase 2 restarts from the base model and, at each update, mixes G fresh on-policy rollouts with M replayed success trajectories, computes token-level clipped GRPO advantages over the combined group (Eqs. (4)-(5)), and updates the policy. Experiments on Qwen2.5-Math-7B report faster convergence and higher final accuracy on AIME-2024, AIME-2025, and AMC-2023 relative to the tuned baseline. Code, datasets, and checkpoints are publicly released.","tokens_in":7364,"tokens_out":5571,"duration_ms":59251,"significance":"If the empirical claims held under proper controls, RLEP would be a simple, low-cost addition to existing GRPO pipelines: it requires no changes to the reward model or verifier, and it proposes a clean two-phase 'collect then restart with replay' design that differs from hard-prompt-rescue variants in the related work. The release of code, datasets, and checkpoints is a genuine strength for reproducibility. The significance is currently tempered by two load-bearing weaknesses: Eq. (4) applies an on-policy GRPO estimator to a batch containing off-policy replayed trajectories with no correction, and the reported comparisons are not compute-matched because the 400-step experience-collection run is omitted from the convergence and final-performance accounting. The contribution is plausible but not yet established; at present it is best viewed as a promising technical report rather than a validated method.","major_comments":[{"comment":"Equation (4) writes the objective as an expectation over trajectories drawn from π_θ_old, but the replayed trajectories in the mixed batch were generated by the seed policy used in the experience-collection phase, not by the current behavior policy. The ratio π_θ(o_i,t|q) / π_θ_old(o_i,t|q) is therefore not a valid importance-sampling correction for those trajectories, and the policy-gradient estimate is biased. Because replayed trajectories are all correct and typically receive high rewards, the bias is likely to inflate the replay signal rather than vanish. The paper should either incorporate a proper off-policy correction (for example, importance weights with clipping, or a separate conservative objective for replayed data) or provide an empirical diagnostic showing that the bias is negligible, such as comparing Eq. (4) with a corrected estimator on the same batch, or ablating how replayed advantages are normalized. Without this, the mechanism claimed for RLEP is not supported by the paper's own equations.","section":"§3.2, Eq. (4)"},{"comment":"The compute accounting is incomplete. The experience pool is constructed by running DAPO-nodyn-bs64 for 400 PPO steps, and the replay run then restarts from the base model. Thus reaching the baseline peak on AIME-2024 at replay step 135 costs at least 400 + 135 = 535 total PPO steps, and the final reported accuracies correspond to about 400 replay steps on top of the 400-step seed run. The baseline is trained for only 400 steps, and no control is reported, such as a baseline continued for another 400 steps or a baseline trained for the same total number of updates. With the current design, the higher final accuracy of RLEP is confounded with additional total compute, and the 'substantially fewer updates' claim in the abstract refers only to replay-phase steps. A compute-matched comparison is needed before 'faster convergence' or 'stronger final performance' can be attributed to experience replay rather than to additional training.","section":"§4.2"},{"comment":"The main results are based on a single run with one base model (Qwen2.5-Math-7B) and no error bars or seed-to-seed variance. The text explicitly reports the best points on AIME-2024 and AIME-2025 as 'peak value of the line' (second bullet). Given that the reported differences are small on AIME-2024 (+1.7pp) and AIME-2025 (+2.5pp), and that the benchmarks contain a limited number of problems, it is not possible to assess whether these differences are stable or within run-to-run noise. At minimum, the authors should report the mean and standard deviation over at least three seeds, and ideally full training curves with confidence bands, for the central RLEP-versus-baseline comparison.","section":"§4.2, Figure 4"},{"comment":"Equation (5) standardizes advantages over the mixed group of G' trajectories that contains both fresh rollouts and replayed successes. Because the replayed trajectories are all correct, their presence raises the group mean and changes the baseline used for the fresh rollouts. This is a design choice rather than an outright error, but its interaction with the off-policy issue in Eq. (4) is not analyzed. If replay is intended to provide a common baseline, the paper should at least compare with a variant in which replayed trajectories are excluded from the mean and standard deviation but still included in the policy-update loss, or otherwise show that the mixed-batch standardization does not distort the advantages assigned to fresh rollouts.","section":"§3.2, Eq. (5)"}],"minor_comments":[{"comment":"There are several typographical errors: 'wile' should be 'while', 'dynmic sampling' should be 'dynamic sampling', and 'we therefore use the configure' should be 'we therefore use the configuration'.","section":"§4.1"},{"comment":"The text first says 'For every question we maintain an experience pool' and later says 'randomly sample M successful trajectories from the experience pool'; please clarify whether each prompt has a separate replay pool and how the pool size varies across prompts when the minimum-correct-paths threshold is applied.","section":"§4.2"},{"comment":"The wall-clock cost of the experience-collection phase is not reported, even though the paper says 'per-step runtime increases by under 5s' for the replay phase; reporting total wall-clock time for both phases would help readers evaluate the compute-matched concern.","section":"§4.2"},{"comment":"Since LUFFY is cited as addressing off-policy guidance with a redesigned advantage estimator, a brief comparison with that approach, or an explanation of why the same correction is not needed here, would help position RLEP's claim of novelty.","section":"§2"},{"comment":"Reference [11] is listed as 'HybridFlow: A flexible and efficient rlhf framework,' but the text refers to 'Verl (Sheng et al., 2024)'; please make the framework name and citation title consistent.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a technical-report-style submission with a simple, easily described method. The central off-policy issue in Eq. (4) and the missing compute-matched control are fixable in principle but require additional experiments and analysis. I would not recommend rejection on novelty grounds: experience replay for LLM RL is an active area, and the specific 'collect with a seed policy, restart, and replay successes' design is a reasonable variant to evaluate. The release of code and checkpoints is a positive signal for reproducibility, but the current evidence does not meet the bar for acceptance without the requested corrections."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: RLEP does something slightly but genuinely different from EFRAME, LUFFY, and Dou et al. It collects verified successes from a converged seed policy, then restarts training from scratch and replays those successes uniformly across all prompts rather than only rescuing hard cases. That is a legitimate, easy-to-implement twist on the existing replay-for-LLM line, and the writeup is refreshingly concrete: code, data, and checkpoints are released, and there is an honest negative result on replaying failures. Good.\n\nThe problem is that the main claims do not follow from the reported experiments. The baseline runs 400 PPO steps. RLEP's experience pool is built by running a 400-step seed policy, and then the replay phase restarts from the base model and runs another ~400 steps for the final numbers. So \"reaches baseline peak by step 135\" is really 535 total steps against a 380-step baseline peak, and the final +1.7 to +5.2pp comparisons compare about 800 total steps to 400. There is no control: no baseline continued for another 400 steps, no compute-matched run. The stress-test note lands. The 'substantially fewer updates' language in the abstract counts only the replay phase, and 'stronger final performance' is confounded by roughly double the training compute. This is not a minor accounting issue; it is the load-bearing claim of the paper.\n\nThere is also a theory-level soft spot: Eq. (4) presents the GRPO objective with pi_theta_old as the behavior policy, but the replayed trajectories come from the fixed seed policy, not from pi_theta_old. The paper gives no off-policy correction and no argument that the bias is negligible. The method may still work empirically; the estimator as written is not what is being computed.\n\nOther weaknesses: one model, one seed, peak-value reporting (acknowledged in the text), no error bars. AMC-2023 is a held-out offline eval, which helps. The citation pattern is fine; the related work is honestly positioned.\n\nWho this is for: practitioners who want a concrete replay recipe for math reasoning RL will get value from the clean description and released artifacts. Researchers evaluating convergence-speed claims should treat the numbers skeptically until a compute-matched control appears.\n\nRecommendation: send it to peer review, but with a mandatory request for a compute-matched baseline—e.g., continue DAPO-nodyn-bs64 for another 400 steps—and error bars over at least a few seeds. The idea is timely and the release is clean, so referee time is warranted; the evidence as it stands is conditional.","headline":"RLEP's replay recipe is plausible and cleanly described, but the headline 'faster and stronger' claims do not survive once you count the omitted 400-step seed run.","tokens_in":7889,"tokens_out":2520,"would_cite":false,"duration_ms":26548,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By mixing a small number of previously verified correct reasoning paths into every GRPO update, RLEP makes math-reasoning RL converge faster and finish higher, gaining 1.7, 2.5, and 5.2 points on AIME-2024, AIME-2025, and AMC-2023.","keywords":["reinforcement learning","experience replay","LLM reasoning","GRPO","mathematical reasoning","policy stability","training convergence","verified rewards"],"falsifier":"Recompute the RLEP update with an off-policy correction that reweights replayed tokens by the ratio of the current policy's probability to the old policy's probability; if the corrected estimator removes the reported accuracy gains on AIME-2024, AIME-2025, or AMC-2023, then the mixed-batch GRPO update in Eq. (4) is the source of the improvement rather than the replayed content.","tokens_in":6853,"feed_emoji":"📈","tokens_out":15108,"duration_ms":136467,"temperature":0.7,"pith_summary":"This paper argues that replaying verified successful reasoning trajectories during reinforcement learning makes LLM math training both faster and stronger. After an ordinary RL run produces a seed policy, RLEP keeps every answer that passed the verifier and, at each update, mixes a handful of these old successes into the batch of freshly generated rollouts. The result, on a 7-billion-parameter math model, is that the policy reaches the tuned baseline's peak accuracy far sooner and then exceeds it: +1.7 points on AIME-2024, +2.5 points on AIME-2025, and +5.2 points on AMC-2023. If correct, RLEP is a near-free addition to existing GRPO-style training pipelines that attacks instability and weight drift without extra rollouts or new rewards.","feed_headline":"Replaying correct reasoning paths lifts math scores by 5.2 points","feed_subtitle":"Blending replayed successes with fresh rollouts reaches peak accuracy faster and beats the tuned baseline.","key_machinery":"The load-bearing object is the experience pool plus the replay-blended GRPO update. At every update step the policy generates $G$ fresh trajectories, the method appends $M$ verified successes sampled from the pool (group size $G'=G+M$), and the token-mean, clip-higher GRPO objective is applied to the mixed group with advantages standardized over all $G'$ trajectories. This makes each update simultaneously reinforce proven chains and explore new ones, and that joint signal is what the paper claims produces faster convergence and a higher final performance ceiling.","core_discovery":"RLEP's central claim is that the plateauing of reinforcement learning with verifiable rewards is not an information ceiling but a reuse problem: trajectories the policy already knows how to solve are abandoned, so each run re-explores from scratch. RLEP collects a pool of verifiably correct trajectories from a converged seed policy, then resumes training with mini-batches that blend fresh rollouts with randomly sampled high-quality replays. Advantages are computed over the enlarged mixed group, so replayed successes and new rollouts share one baseline. On three math benchmarks the replay run reaches the baseline's best accuracy in a fraction of the updates and ends higher, and the authors attribute both effects to replay shielding the policy from unproductive exploration and concentrating learning on promising paths.","pith_inferences":["Inference: because the advantage is normalized over the enlarged mixed group, replayed successes mechanically raise the group mean and shrink the advantage assigned to fresh correct rollouts; an ablation that replays the same pool but computes advantages over fresh rollouts only would reveal how much of the gain is due to this re-baselining rather than to the replayed content.","Inference: the recipe needs only a rule-based verifier, so it should transfer to other verifiable-reward domains such as code generation or theorem proving; a direct test would run RLEP unchanged on a code benchmark and look for the same early-convergence signature.","Inference: the paper's null result for replaying failures leaves open a plausible extension: failure trajectories filtered by near-miss structure, such as a sound derivation with a wrong final step, might still teach the policy even if unfiltered errors are too heterogeneous to help.","Inference: the claimed reduction of weight drift is testable without new training by measuring parameter distance from the pretrained checkpoint across matched baseline and RLEP runs; the paper reports accuracy curves but no such distance diagnostic."],"forward_implications":["RLEP reaches the baseline's peak AIME-2024 accuracy by step 135 while the baseline takes 380 steps, and it surpasses the baseline's best AIME-2025 score after only 50 steps.","Final accuracy improves from 38.2% to 39.9% on AIME-2024, from 19.8% to 22.3% on AIME-2025, and from 77.0% to 82.2% on AMC-2023.","Because replay is applied uniformly across all prompts rather than only to difficult cases, the benefit extends to the whole training distribution instead of patching hard examples.","Per-step runtime increases by under five seconds relative to the baseline, so the reported gains come without materially longer training."],"supporting_citations":[{"why":"This work supplies the tuned baseline's clip-higher, token-mean, and overlong-reward-shaping design, plus the configuration that RLEP modifies.","marker":"Yu et al. (2025)"},{"why":"This work introduces GRPO, the group-relative advantage estimator whose mixed-batch form Eq. (4) extends.","marker":"Shao et al. (2024)"},{"why":"This work provides the token-mean objective that Eq. (4) applies token by token.","marker":"Liu et al. (2025)"},{"why":"This work is the training framework whose rollout and update defaults the RLEP runs inherit.","marker":"Sheng et al. (2024)"},{"why":"This work provides the math-specific 7-billion-parameter base model used for all reported experiments.","marker":"Yang et al. (2024)"},{"why":"This work is the prior replay-for-LLM method RLEP contrasts with by replaying uniformly across all prompts rather than only hard cases.","marker":"Wang et al. (2025)"}],"fun_headline_variants":["Replaying correct reasoning paths lifts math scores by 5.2 points","RLEP replays proven trajectories to speed up RL and boost math","Experience replay helps LLMs avoid re-exploring solved paths","Blending replayed successes with fresh rollouts yields faster, stronger RL","LLM RL plateaus are a reuse problem; experience replay fixes them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that old successful answers generated by the earlier seed policy can be fed into the GRPO estimator as if the current policy produced them, with no correction for the mismatch between the two policies.","fun_headline_variants_meta":{"raw":{"variants":["Replaying correct reasoning paths lifts math scores by 5.2 points","RLEP replays proven trajectories to speed up RL and boost math","Experience replay helps LLMs avoid re-exploring solved paths","Blending replayed successes with fresh rollouts yields faster, stronger RL","LLM RL plateaus are a reuse problem; experience replay fixes them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001021,"raw_usage":{"total_tokens":4296,"prompt_tokens":921,"completion_tokens":3375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":3282}},"tokens_in":537,"tokens_out":3375,"duration_ms":29025,"temperature":1.0,"reasoning_tokens":3282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:40:24.873948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the RLEP update with an off-policy correction that reweights replayed tokens by the ratio of the current policy's probability to the old policy's probability; if the corrected estimator removes the reported accuracy gains on AIME-2024, AIME-2025, or AMC-2023, then the mixed-batch GRPO update in Eq. (4) is the source of the improvement rather than the replayed content.","supporting_citations":[],"review_version":1}