{"id":"18017e4d-a1d7-4848-9868-f02663e2c1c4","arxiv_id":"2506.09340","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RePO adds a replay buffer to GRPO so each prompt is optimized on both fresh and stored samples, yielding higher math benchmark scores than GRPO across several small language models.","lead":"The paper proposes RePO, which extends GRPO, a popular method for training AI reasoning models with reinforcement learning, by reusing previously generated answers from a replay buffer. On seven math benchmarks, the authors report average accuracy gains of up to 18.4 points for a 1.5-billion-parameter model, at about 15 percent extra compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RePO's headline gains may come from doubling the per-step sample count in the final epoch rather than from replay; the missing GRPO-with-64-samples-per-step control leaves the central attribution untested.","rationale":"The reader's weakest-assumption analysis identifies exactly the same concern I regard as most load-bearing: RePO's final epoch uses more samples per step than the GRPO baseline, so the observed gains may reflect additional data rather than the replay mechanism. This is not a question of internal inconsistency: the method is clearly described, the tables are internally consistent, and the code link is a point in favor of reproducibility. The issue is an uncontrolled comparison. The missing control is easy to run and would settle the attribution. Because the paper is otherwise plausible and the confound is fixable, the appropriate verdict remains CONDITIONAL rather than REJECT or ACCEPT. I therefore recommend no change to the reader's verdict. The secondary issues, such as post-hoc replay-strategy selection and the effective-step metric, are real but would matter less if the equal-sample control were provided.","tokens_in":12733,"tokens_out":5364,"duration_ms":60360,"concrete_test":"Run the two largest-gain configurations (Qwen2.5-Math-1.5B and Qwen3-1.7B) with a matched GRPO control that does not use replay: in the final epoch, give GRPO 64 samples per step (for example, 16 on-policy samples per prompt, or two 8-sample advantage groups per prompt with an equivalent loss update), keeping the number of optimizer updates, total training examples, learning rate, and all other hyperparameters identical to RePO. If this control reproduces RePO's average gain over the standard 32-per-step GRPO baseline, the headline attribution to replay is unsupported; if RePO still beats the equal-sample GRPO control by a meaningful margin on the same seven math benchmarks, the replay mechanism is corroborated. Report results over at least three seeds to also address the absence of error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RePO's central claim is that replaying stored off-policy samples is what improves GRPO. The experiments do not isolate that mechanism. Appendix A.1 states that GRPO uses 32 training examples per step in all epochs, whereas RePO increases to 64 examples per step in the final epoch (8 on-policy plus 8 off-policy per prompt), and the off-policy loss is active only in that final epoch. RePO therefore differs from GRPO in the final epoch along at least two coupled dimensions: it trains on twice as many examples per optimizer step, and it also has the replay-buffer loss. No GRPO control with 64 examples per step, and no GRPO control with an additional epoch, is reported. The 'No' baseline in Table 5 does not close this gap, because it is not a same-sample-count control. The 'effective-step' argument in Table 7 is also not a substitute: it defines a step as a single prompt and assumes advantage collapse when all rewards in that step are equal, but training actually uses batches of 32 prompts, so the collapse condition is not evaluated at the relevant unit. The post-hoc selection of replay strategy from Table 5 before reporting Table 1 compounds the issue, but the load-bearing gap is the missing equal-sample GRPO control. The method is clearly specified and the code link is useful evidence, but the reported gains cannot yet be firmly attributed to replay rather than to additional final-epoch data and optimization signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Replay-Enhanced Policy Optimization (RePO), an extension of GRPO that adds an off-policy replay-buffer loss to the standard on-policy GRPO objective. The off-policy term retrieves previously stored samples according to one of several replay strategies (full-scope, recency-based, reward-oriented, variance-driven) and computes separately normalized advantages for on- and off-policy groups. Experiments on five Qwen models across seven math benchmarks report average gains of up to 18.4 points over GRPO, with about 15% additional compute, plus smaller gains on general reasoning benchmarks. The paper also reports comparisons with Dr. GRPO, ablations of replay strategy and advantage-estimation variant, and an analysis of 'effective optimization steps.'","tokens_in":12964,"tokens_out":2380,"duration_ms":26917,"significance":"If the reported gains are genuinely caused by replaying off-policy samples, RePO would be a simple and practically useful modification of GRPO that improves data efficiency at modest cost. The manuscript is clearly written, the method is precisely specified with pseudocode and a release of code, and the gradient-ratio interpretation in Section 3.2 is a useful conceptual contribution. The replay-strategy ablations (Table 5) and the separate-advantage analysis (Table 4) provide informative design evidence. However, the central experimental claim currently rests on a comparison that confounds replay with an increase in per-step data and optimization signal, and on single-run results without variance estimates, so the magnitude and attribution of the headline gains are not yet established.","major_comments":[{"comment":"The main claim that replay improves GRPO is not supported by the current experimental design because RePO and GRPO differ in the final epoch in two coupled ways. Appendix A.1 states that GRPO uses 32 training examples per step in all epochs, while RePO increases to 64 examples per step in the last epoch (8 on-policy and 8 off-policy per prompt), with the off-policy loss active only in that epoch. No GRPO control with 64 examples per step or with an additional epoch is reported, so the gains in Table 1 may be due to the larger number of final-epoch samples and optimization signal rather than to the replay mechanism itself. Please add an equal-sample-count GRPO baseline, or a RePO ablation that uses 64 on-policy samples per step in the final epoch without the replay loss, to isolate the effect of replay.","section":"Appendix A.1 / Section 4.2"},{"comment":"All reported results are single point estimates with no seeds, confidence intervals, or significance tests. Several of the claimed improvements are very small (for example, Qwen2.5-Math-1.5B-Instruct average 45.8 vs. 46.1 in Table 1, and gains of 0.2–2.4 points in Table 3), and the paper makes strong claims of consistency ('consistently outperforms') on the strength of these differences. Please report results over at least three seeds with standard deviations for the main comparisons, or explicitly state if single runs were used and soften the claims accordingly.","section":"Section 4.2 / Tables 1 and 3"},{"comment":"The 'effective-step' analysis used to explain RePO's effectiveness is computed at a unit that does not match the training procedure. Footnote 1 assumes 'each step consists of a single prompt,' but Algorithm 1 samples a batch of 32 prompts per step and takes optimizer steps over that batch; the condition that all rewards in a group are equal is evaluated per prompt, not per batch, and the relevant quantity for whether a gradient step is 'effective' is the batch-level gradient, not the per-prompt advantage. The 47.8% relative increase in effective steps therefore does not directly support the stated mechanism. Please either redefine the metric at the batch level used in training or present direct evidence (e.g., measured fraction of near-zero batch gradients) that RePO's advantage survives the batch averaging.","section":"Section 4.4, Table 7, and footnote 1"},{"comment":"The replay strategy used in the headline results of Table 1 is selected per model class (Recency-based for base models, Reward-oriented for instruct models) based on the same seven math benchmarks reported in Table 5. Because the selection and the evaluation share the same test sets, the reported average gains in Table 1 carry an optimism bias from model selection over the replay-strategy choices. This does not invalidate the method, but the paper should either report results for a strategy chosen before seeing the test benchmarks, or present the full Table 5 as the primary comparison and treat Table 1 as a selected-configuration summary, with an explicit caveat about selection on the evaluation set.","section":"Section 4.3 and Table 5"}],"minor_comments":[{"comment":"The metric 'avg@32' for AIME24, AIME25, and AMC is not defined in the main text or appendix; please clarify whether it is a pass@32 estimate or a different aggregation, and state how the 32 samples are generated.","section":"Section 4.1"},{"comment":"The 'Rel. Time' column reports values normalized to GRPO with 4 on-policy samples, but the caption does not explain why GRPO with 8 on-policy samples is shown as ×2.02; please state the absolute training time or the precise recipe for the normalization.","section":"Table 6"},{"comment":"The notation in Table 4 labels rows as 'MIXED' and 'SPLIT,' while Section 3.2 always uses 'Split' in prose; please harmonize the capitalization. Additionally, the mixed-strategy objective is not written down; including its equation would help reproducibility.","section":"Section 3.2 and Table 4"},{"comment":"The x-axis 'Number of Replay Samples' is introduced without clarifying whether it is the number of off-policy samples per prompt (G_off) or the total replay-buffer size; the text and Algorithm 1 suggest the former, but the figure should state this explicitly.","section":"Section 4.4, Figure 2"},{"comment":"The statement that 'the off-policy update is applied only in the final epoch' means that E_off in Algorithm 1 is set to N; this should be named explicitly in the algorithm description rather than left implicit.","section":"Appendix A.1"},{"comment":"The paper references prior off-policy RL work in the LLM literature (e.g., LUFFY) but does not discuss off-policy correction beyond the importance ratio; a brief mention of how the clipping and separate advantage normalization relate to standard off-policy policy-gradient corrections would improve the positioning.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the method is simple, but the central comparison confounds replay with doubled final-epoch sample count, and all results are single-run estimates. A revision that adds an equal-sample GRPO control, reports variance, and either fixes or reframes the effective-step analysis would make the contribution publishable. I would not reject on the current evidence because the method is well specified and the issue is addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing about RePO is that it takes a standard idea—replay buffers and off-policy importance weighting—and applies it to GRPO cleanly. The paper is well written, the method is clearly specified, the gradient identity in Section 3.2 is a correct and honest statement that the off-policy loss is just the GRPO loss scaled by an importance ratio, and they ship code. That is real work. The empirical pattern is also consistent: RePO beats GRPO on average across five models.\n\nBut the headline numbers can't yet be trusted as evidence for replay. The training details in Appendix A.1 say that in the final epoch RePO trains on 64 examples per step while GRPO stays at 32. There is no GRPO control with 64 samples, and no GRPO run with an extra epoch. So the 18.4-point average gain on Qwen2.5-Math-1.5B could be mostly an artifact of more data in the final epoch rather than the replay mechanism. The stress-test note is right; this is the load-bearing gap. Table 5's 'No' baseline does not close it because it is not a same-sample-count control.\n\nTwo more soft spots, both minor in comparison. No seed-averaged numbers anywhere, so we have no sense of variance; some of these benchmark deltas are small. And the replay strategy for each model is chosen after looking at Table 5, which introduces a selection effect for Table 1. The effective-step argument in Table 7 depends on defining a step as a single prompt, which conflicts with the batched training.\n\nIs the method worth engaging with? Yes. The idea is sensible, the cost analysis is transparent (15% extra compute), and the confound is directly testable—run GRPO with 64 samples per step in the last epoch. That is a few A100-days and would settle the central attribution. I would send this to peer review with a request for that control and for error bars. It is not desk-reject material.\n\nIn short: a solid method paper with an honest derivation, but the evidence for the central claim is currently under-specified. A fixable problem, not a fatal one.","headline":"RePO is a clean method paper with a real confound: the final epoch doubles RePO's per-step samples relative to GRPO, so the headline gains aren't cleanly attributable to replay until a same-sample-count control is run.","tokens_in":13557,"tokens_out":1753,"would_cite":false,"duration_ms":18169,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RePO, a replay-enhanced variant of GRPO, improves math-reasoning accuracy by up to 18.4 average points at roughly 15% extra compute.","keywords":["reinforcement learning","GRPO","replay buffer","off-policy optimization","mathematical reasoning","large language models","data efficiency","reasoning RL"],"falsifier":"Run GRPO on Qwen2.5-Math-1.5B for three epochs with 64 training examples per step in the final epoch, keeping all other hyperparameters identical to the paper's setup, and evaluate on the same seven math benchmarks. If the GRPO baseline reaches the same ~35.8 average accuracy as RePO, the reported gains can be explained by the larger per-step data size rather than the replay mechanism.","tokens_in":12460,"feed_emoji":"♻️","tokens_out":7778,"duration_ms":71500,"temperature":0.7,"pith_summary":"RePO (Replay-Enhanced Policy Optimization) tries to establish that a reinforcement learning algorithm for language models can be made both cheaper and more data-efficient by reusing previously sampled outputs instead of generating everything fresh. The authors combine the standard on-policy GRPO loss with an off-policy term that pulls past generations from a replay buffer, and report average math-reasoning gains of 18.4, 2.0, and 4.1 points over GRPO on Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-1.7B for roughly 15% extra compute. A sympathetic reader would care because GRPO-style RL is the workhorse behind modern reasoning LLMs, and the paper offers a way to extract more learning signal from each rollout rather than just sampling more.","feed_headline":"Reusing past training samples lifts math scores by up to 18 points","feed_subtitle":"RePO reuses past rollouts to get more learning from each prompt, at 15% extra compute.","key_machinery":"The load-bearing object is the replay buffer plus a replay strategy. The buffer stores, for each prompt, previously sampled completions together with their generation probabilities under the behavior policy; the off-policy update term $J_{\\mathrm{off}}(\\theta)$ applies the same clipped GRPO-style loss to a replayed group, with the importance ratio $r^{\\mathrm{off}}_{i,t} = \\pi_\\theta(o^{\\mathrm{off}}_{i,t} \\mid q, o^{\\mathrm{off}}_{i,<t}) / \\pi_{\\theta_{\\mathrm{off}}}(o^{\\mathrm{off}}_{i,t} \\mid q, o^{\\mathrm{off}}_{i,<t})$ downweighting samples the current policy no longer assigns high probability. Four strategies are proposed: Full-scope (all past samples), Recency-based (most recent $K$), Reward-oriented (highest reward), and Variance-driven (highest reward variance, aimed at vanishing-gradient decay). The paper also separates advantage normalization between on- and off-policy groups (the 'Split' strategy), which is shown to beat mixing the two groups.","core_discovery":"The paper's central claim is that an off-policy replay term, added to the GRPO objective, improves policy optimization for mathematical reasoning without a large compute penalty. For each prompt, RePO stores sampled outputs with their generation probabilities in a replay buffer; at update time it retrieves a group of these old samples using a chosen replay strategy and optimizes the current policy on both the fresh on-policy group and the replayed off-policy group, estimating advantages separately for the two groups. Across five Qwen models and seven math benchmarks, this consistently beats plain GRPO, with the largest gains on the smallest base model (18.4 average points) and with the number of effective optimization steps rising by 48% for Qwen3-1.7B when both on- and off-policy samples are set to 8. The authors attribute the gains to increased sample diversity and to the rescue of gradient signal when a GRPO step would otherwise see identical rewards and thus zero advantages.","pith_inferences":["Editorial inference: RePO's replay mechanism could be applied to non-mathematical RL settings—code generation, tool use, instruction following—where reward signals are similarly sparse and fresh rollouts are expensive.","Editorial inference: The variance-driven strategy is designed for reward collapse, so a testable extension is to use it with learned reward models whose scores are noisy, where a larger group variance signals useful gradient information.","Editorial inference: The separate (Split) advantage estimation could be combined with an adaptive replay ratio that gradually shifts from recency-based to reward-oriented selection as training progresses, which the model-dependent results in Table 5 suggest might help."],"forward_implications":["If RePO is correct, GRPO-style RL for reasoning models can reuse past rollouts to make each prompt's samples contribute more signal, reducing the data and compute needed for the same accuracy.","The 48% increase in effective optimization steps suggests replay directly counters the degenerate all-reward-equal steps that give GRPO zero gradient, a known failure mode during training.","The method transfers to at least one GRPO variant besides the base algorithm: RePO (Dr. GRPO) beats Dr. GRPO by 1.3 and 4.2 average points on two models, so replay appears to be a plug-in ingredient.","Because the best replay strategy differs between base and instruct models, practitioners can tune replay selection per model; the paper's fixed choices (Recency-based for base, Reward-oriented for instruct) are a simple default.","The 15% extra compute buys more than 15% in effective optimization, implying replay is a compute-efficient way to spend an RL budget."],"supporting_citations":[{"why":"Defines GRPO, the on-policy baseline that RePO extends and the primary comparison across all experiments.","marker":"Shao et al., 2024"},{"why":"Provides the DeepMath training subset (1024 problems with answers) used for all RL runs.","marker":"He et al., 2025"},{"why":"Introduces Dr. GRPO, the variant used to show that RePO's replay idea transfers beyond plain GRPO.","marker":"Liu et al., 2025"},{"why":"Supports dropping the KL penalty in the on-policy update, a design choice RePO inherits.","marker":"Yu et al., 2025"},{"why":"Motivates the variance-driven replay strategy by identifying vanishing gradients in RL finetuning of language models.","marker":"Razin et al., 2024"}],"fun_headline_variants":["RePO: replay buffer lifts math scores by up to 18 points","Reusing old rollouts boosts math by 18, compute up 15%","Off-policy replay gives more learning per prompt, 18-point gain","Past samples power LLM math improvement, 18 points over GRPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison attributes the gains to replay itself, yet RePO's final training epoch uses 64 examples per optimization step while GRPO keeps 32, and no GRPO baseline with 64 examples per step (or an equivalent extra epoch) is reported.","fun_headline_variants_meta":{"raw":{"variants":["RePO: replay buffer lifts math scores by up to 18 points","Reusing old rollouts boosts math by 18, compute up 15%","Off-policy replay gives more learning per prompt, 18-point gain","Past samples power LLM math improvement, 18 points over GRPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1531,"prompt_tokens":940,"completion_tokens":591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":556,"tokens_out":591,"duration_ms":6201,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:50:29.578952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GRPO on Qwen2.5-Math-1.5B for three epochs with 64 training examples per step in the final epoch, keeping all other hyperparameters identical to the paper's setup, and evaluate on the same seven math benchmarks. If the GRPO baseline reaches the same ~35.8 average accuracy as RePO, the reported gains can be explained by the larger per-step data size rather than the replay mechanism.","supporting_citations":[],"review_version":1}