{"id":"4a2c0db3-61ab-4b35-843b-d3c0e1d28d43","arxiv_id":"2507.06892","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"ReMix cuts reinforcement finetuning rollout volume by 30x to 450x on math reasoning by mixing historical and on-policy data with a convex KL constraint and a mid-training switch to on-policy updates.","lead":"The authors propose ReMix, a reinforcement finetuning method that reuses old model responses alongside fresh ones, then switches to standard on-policy training halfway. On five math benchmarks, they report matching the accuracy of top 1.5B and 7B models while using 30 to 450 times fewer training rollouts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rollout-volume metric is load-bearing but likely conservative; token-level normalization is needed to verify the 30x-450x training-cost claim.","rationale":"The reader's weakest assumption is that the cross-model efficiency claims rely on rollout data volume as the foundational cost metric, which may not faithfully represent true compute because of response-length, context-length, and hardware differences. I agree this is the most load-bearing assumption in the paper, and I locate it precisely in the internal inconsistency between Section 4.1, which asserts average response lengths are roughly at the same scale, and Section 4.2, which reports that ReMix produces shorter responses. However, my reading also suggests the metric bias likely runs in ReMix's favor: if ReMix uses fewer responses and shorter responses, then a token-normalized comparison would make its efficiency advantage larger, not smaller. The baseline cost estimates are rough but the order-of-magnitude gaps are large enough that the qualitative 30x-450x reduction probably survives plausible corrections. The concern is therefore about the verification and precision of the headline numbers rather than about a likely false conclusion. The paper already has a CONDITIONAL verdict, and my stress-test does not move that verdict: the rollout-volume metric should be supplemented with token-level or FLOP-level measurements, and code/checkpoints should be released, before the exact cost-reduction factors are treated as established. The core methodological contribution, off-policy PPO with replay and policy reincarnation, is coherent and the controlled ReMix-versus-PPO comparisons support the efficiency narrative, so no rejection is warranted. I set agreement_with_reader to partial because I endorse the identified weakness but differ on its likely impact: the reader suggests the 30x-450x reductions could be substantially smaller, whereas the available evidence points toward them being conservative.","tokens_in":36904,"tokens_out":11650,"duration_ms":133249,"concrete_test":"Recompute the headline efficiency comparisons using total generated tokens instead of response counts, and where possible total training FLOPs including gradient updates. For each baseline in Tables 7 and 8, obtain the average response length from training logs, original papers, or official configs; for ReMix use the response-length curves in Figure 5. Compute rollout_count × average_response_length for ReMix-PPO versus DeepScaleR and AceReason-Nemotron, and compare the resulting reduction factors. If the 30x and 450x ratios persist under token and FLOP normalization, the central claim is validated; if the ratios drop below 30x, the abstract overstates the efficiency gain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ReMix achieves SOTA-level performance with an over 30x to 450x reduction in training cost in terms of rollout data volume. For this claim to hold as a statement about training cost, rollout data volume must faithfully proxy total compute. The paper states in Section 4.1 that a stricter metric should also account for rollout response length, and it justifies using response counts by asserting that average response lengths are roughly at the same scale. But Section 4.2 immediately undercuts this justification: the authors report that ReMix's average rollout response length is lower than the baselines, with evidence in Figure 5. Since LLM inference cost scales with generated tokens rather than with the number of responses, response count is not a length-invariant cost measure. The compared baselines also differ in maximum response length (8k to 32k contexts), generation count per prompt, and training phases, so per-response inference cost varies substantially. The quantitative 30x-450x factors are therefore an unverified proxy for the advertised 'training cost' reduction. Importantly, the likely direction of the bias favors ReMix: shorter ReMix responses and fewer total gradient updates imply the true token-level or FLOP-level reduction may be even larger, so the concern is about verification of the exact magnitude rather than invalidation of the qualitative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReMix, a method that adapts on-policy proximal policy gradient methods (PPO, GRPO) for LLM reinforcement finetuning to reuse off-policy data from historical policies. ReMix combines a mixed-policy clipped objective with an increased Update-To-Data ratio, a KL-convex constraint anchored to both the base model and the previous policy, and a policy-reincarnation transition to on-policy training after a fixed number of steps. Experiments on 1.5B and 7B DeepSeek-R1-Distill-Qwen models over five math benchmarks report ReMix-PPO reaching 52.10% average Pass@1 at 1.5B with 0.079M rollouts and 63.27%/64.39% at 7B with 0.007M/0.011M rollouts, which the authors compare to DeepScaleR (52.14%, 2.519M) and AceReason-Nemotron (63.24%, estimated 3.584M). The paper also presents ablations and analyses of off-policy training dynamics, including a claimed 'Whipping Effect' that explains shorter responses under off-policy training.","tokens_in":37173,"tokens_out":3524,"duration_ms":41852,"significance":"If the efficiency claims hold, ReMix is a practically valuable contribution: it shows that historical rollout data can be reused effectively in LLM reasoning RL, with controlled comparisons against PPO and GRPO that support the qualitative efficiency improvement. The method is presented with enough implementation detail to reproduce, and the ablations in Table 3 give a clear picture of which components matter. The strongest headline result, however, is an efficiency claim measured in rollout count, and the current manuscript does not fully verify that this metric is faithful for cross-model cost comparisons. The analysis of reasoning dynamics is also more suggestive than conclusive. These issues are fixable with additional measurements and framing, so the work is likely to be a solid contribution after revision.","major_comments":[{"comment":"The central 30x–450x training-cost reduction is expressed purely in rollout data volume. Section 4.1 acknowledges that a stricter metric should account for rollout response length and justifies rollout counts by asserting that average response lengths are roughly at the same scale, but Section 4.2 immediately notes that ReMix's responses are shorter (Figure 5), and the compared baselines use very different maximum response lengths (8k to 32k in Tables 7–8). Since LLM inference cost scales with generated tokens, rollout count is not a faithful cross-model cost measure, and the exact magnitude of the claimed reduction is unverified. I note the likely direction of the bias favors ReMix, so the qualitative claim may survive, but the paper should report token-level or FLOP-level normalization, or at minimum the average response lengths for each model, before advertising the specific 30x–450x factors.","section":"§4.1, §4.2, Figure 1, Tables 1–2"},{"comment":"All reported results are single-run point estimates without error bars or seed variance. This matters for the headline comparison: ReMix-PPO (1.5B, 350 steps) scores 52.10 versus DeepScaleR's 52.14, a difference well within typical run-to-run variation for RL training, yet the paper describes this as 'comparable' and uses it to support the 30x reduction claim. Similarly, the 7B results compare point estimates from a single checkpoint. At least a small number of independent seeds, or bootstrap intervals over checkpoints, should be reported for the main efficiency–performance comparisons.","section":"§4.2, Tables 1–2"},{"comment":"The formal analysis of the 'Whipping Effect' is circular as written. The argument assumes importance ratios are close to 1, infers negative advantages from the positive policy loss in Figure 9, and then claims that longer responses produce larger distribution shift on later states, which is exactly the quantity that needs direct measurement. No per-token importance-ratio or length-conditioned evidence is provided to connect the sign of the loss to response shortening. This part should be reframed as a hypothesis, or supplemented with a direct measurement of how importance ratios and gradients vary with response length.","section":"§4.4.2, Eq. (7), Figure 9"},{"comment":"The 450x reduction against AceReason-Nemotron depends on a rollout volume that is estimated from the original paper rather than measured from training logs (Section 4.2 footnote 7). Given that this is one of the two anchor numbers for the headline claim, the sensitivity of the reduction factor to this estimate should be reported, for example by giving a plausible range over the estimated rollout count.","section":"§4.2, footnote 7, Table 2"}],"minor_comments":[{"comment":"Section 3.3 is titled 'Policy Reincatenation' and the text contains several typos and grammatical errors; the paper would benefit from a careful copyedit.","section":"§1, §3.3"},{"comment":"The halved maximum response length is inconsistently reported as 4,096, 4,192, and 4,196 tokens in different places; unify these values.","section":"§4.4.3, Table 4"},{"comment":"Figure 3's caption says ReMix is 'around 3x to 6x faster' on Olympiad, while the AIME curve in the appendix reports 'around 1.2x to 1.6x faster'; check that the reported speedup factors match the actual curves.","section":"§4.2, Figure 3 and Appendix D"},{"comment":"The Pass@1 metric in Table 5 counts correct answers even when the format is wrong, while Format Cor is reported separately; state this explicitly in the main text so readers do not conflate the two columns.","section":"§4.4.4, Table 5"},{"comment":"The notation 'Underlined values denote the fresh on-policy rollout in addition to off-policy data reuse in ReMix' is helpful, but the rollout counts in Table 1 do not appear to make the off-policy reuse portion explicit; clarify how the stated rollout volumes are derived from the batch sizes and steps in Tables 7–8.","section":"§4.1, Tables 7–8"}],"recommendation":"major_revision","confidential_remarks":"The paper's core algorithmic idea is interesting and the controlled comparisons against PPO and GRPO are a real strength. The main risk is that the headline efficiency claim rests on a cost metric that is convenient but not fully validated, and the absence of error bars makes the 'SOTA-level parity' claims fragile. I would not reject on these grounds because the likely direction of the metric bias favors ReMix, but the authors should verify the magnitude and be more careful with the framing. I would also encourage the editor to ensure the authors do not overstate the Whipping Effect analysis, which is currently more narrative than derivation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid engineering paper with a plausible efficiency claim, and the headline factors are likely in the right direction but not yet certified. The core idea—reusing historical rollouts in PPO/GRPO-style RFT via a mix of off-policy and on-policy data, higher UTD, dual-anchor KL, and a reincarnation switch—is not conceptually new in RL, but the integration for LLM finetuning is clean and the controlled 1.5B/7B comparisons against PPO and GRPO are the right kind of evidence. The ablations are informative and the observation that off-policyness shortens responses and suppresses self-reflection is a genuine, useful insight.\n\nThe soft spots are real but not fatal. The 30x-450x reduction is measured in rollout count, not tokens, and the paper itself admits response length matters. Since ReMix generates shorter responses, token-level or FLOP-level accounting could make the savings even larger, but the claimed factors are unverified for baselines with different response lengths, context windows, and estimated costs (AceReason's 3.584M rollouts is an estimate from another paper). That weakens the abstract's \"training cost\" framing. More seriously, there are no seeds or error bars; RFT runs at these settings are noisy, and single-run gaps of a couple of points between methods need variance reporting to be convincing. No code or data is released. The Whipping Effect analysis is more suggestive than formal—read the policy-loss sign as evidence of the proposed mechanism, but the causal story is partly circular (shorter responses reduce future importance weights, which then looks like confirmation).\n\nBottom line: the qualitative claim—reusing off-policy data cuts compute substantially—is well supported by the controlled comparisons. The exact multipliers need standardization. This paper deserves a serious referee, but the revision should require token-level cost curves, multiple seeds, and ideally an artifact release. I would bring it to a reading group only if we care about efficient RFT; otherwise it is a useful citable reference for off-policy RFT techniques.","headline":"ReMix shows real efficiency gains by reusing off-policy data in RFT, but the 30x-450x headline is on rollout counts rather than tokens and deserves variance-aware verification before the exact magnitude is trusted.","tokens_in":37729,"tokens_out":1708,"would_cite":true,"duration_ms":21437,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReMix reuses historical RL rollouts to cut reasoning-model training data volume by more than 30x while matching top accuracy.","keywords":["off-policy reinforcement learning","reinforcement finetuning","large language models","math reasoning","proximal policy optimization","sample efficiency","policy reincarnation","GRPO"],"falsifier":"An audit that records total generated tokens, GPU-hours, and hardware configuration for ReMix and for DeepScaleR and AceReason-Nemotron would settle it: if the token-normalized or hardware-normalized cost gap disappears or reverses, the central efficiency claim fails; if it survives, the claim's magnitude should be re-expressed in those units.","tokens_in":1817,"feed_emoji":"🧠","tokens_out":2165,"duration_ms":67282,"temperature":0.7,"pith_summary":"This paper claims that the sample-inefficiency bottleneck in reinforcement finetuning (RFT) of large reasoning models can be broken by resurrecting off-policy RL: instead of discarding rollouts after each PPO or GRPO update, a method called ReMix reuses them in mixed training batches, applies more gradient updates per rollout, and then resets to on-policy training partway through. On five math benchmarks, ReMix-PPO reports 52.10% average Pass@1 accuracy at 1.5B scale with 0.079M rollouts versus DeepScaleR's 52.14% with 2.519M rollouts, and 64.39% at 7B scale with 0.011M rollouts versus AceReason-Nemotron's 63.24% with an estimated 3.584M rollouts. The paper further argues that off-policyness has a measurable behavioral signature: heavier reuse of historical data shortens responses and suppresses self-reflection tokens, which speeds early learning but can collapse later performance, so the method balances the mix and then reincarnates. If right, RFT becomes far cheaper to scale, and the behavioral analysis offers a concrete explanation for why off-policy training of LLMs often degrades.","feed_headline":"Reusing old RL data cuts LLM reasoning training costs up to 450x","feed_subtitle":"ReMix matches top math reasoning with 0.011M-0.079M rollouts instead of millions of fresh samples.","key_machinery":"The load-bearing object is the Mix-PPG objective: an importance-sampled clipped surrogate over a mixture of current-policy and historical-policy trajectories, combined with an increased update-to-data ratio that repeats gradient updates, a KL-convex constraint anchoring to both the base model and the previous policy, and a policy-reincarnation switch at a scheduled step that replaces the KL reference with the current policy and reverts to on-policy PPO or GRPO. The objective is grounded in a generalized off-policy policy improvement bound and uses a V-trace-corrected advantage estimator with truncated importance-sampling ratios. These components work together: the mixed objective and high update-to-data ratio give the early efficiency gain, the two-anchor KL keeps updates stable, and the reincarnation stage supplies the asymptotic convergence that pure off-policy training lacks.","core_discovery":"ReMix establishes that proximal policy-gradient RFT can learn from its own history without losing final accuracy. It mixes a proportion p of rollouts sampled from recent historical policies with on-policy rollouts, applies an increased update-to-data ratio, regularizes the policy with a convex combination of KL constraints to the base model and to the previous policy, and at a scheduled step resets the KL reference to the current policy and switches back to ordinary PPO or GRPO. The central empirical discovery is that this staged scheme converts the early-stage efficiency of off-policy reuse into state-of-the-art-level math reasoning: with far fewer rollouts and training steps, ReMix-PPO outperforms or matches both its on-policy counterparts and recent RFT baselines at 1.5B and 7B scales. A secondary discovery is the \"Whipping Effect\": because negative advantages dominate, an off-policy objective pushes the policy toward lower importance ratios, which is systematically easier for shorter responses, so higher off-policy proportions shorten generations; the paper links this to a collapse in self-reflection behavior and shows that ReMix degrades less than baselines when the maximum response length is halved.","pith_inferences":["If rollout-volume accounting were replaced by token-level or FLOP-level accounting, the headline 30x to 450x factors would likely shrink, since ReMix's own responses are shorter; the qualitative efficiency advantage may survive, but the stated magnitude depends on the chosen cost metric.","The Whipping Effect suggests a tunable control knob: an adaptive off-policy proportion that shrinks as the importance ratio drifts could extend the efficient early stage and postpone the need for reincarnation.","The self-reflection collapse result implies that reflection-token rate could serve as a cheap early-warning monitor for off-policy instability in other reasoning behaviors, such as tool use or verification loops.","The reincarnation schedule is a discrete switch; continuously interpolating the off-policy fraction during the transition would test whether the staging itself, rather than the stage identities, preserves asymptotic performance."],"forward_implications":["Standard PPO and GRPO can be upgraded to reuse historical rollouts through ReMix, yielding higher accuracy than their on-policy counterparts at the same or lower rollout volume.","State-of-the-art-level math reasoning at 1.5B and 7B scales is achievable with well under 0.1M rollouts, implying roughly 30x to 450x less rollout data than leading baselines.","Off-policy RFT has a characteristic failure mode: too much off-policyness shortens responses and suppresses self-reflection, so maintaining a sufficient on-policy fraction is critical for stable training.","ReMix-trained models are comparatively robust to halved maximum response lengths because they have learned more concise reasoning.","Removing explicit guide tokens from the prompt template during ReMix training preserves accuracy on MATH500 while improving format robustness under prompt variation."],"supporting_citations":[{"why":"Supplies the generalized off-policy policy improvement lower bound that legitimizes using historical trajectories in the Mix-PPG objective.","marker":"(Queeney et al., 2021)"},{"why":"Provides the PPO clipped surrogate objective that ReMix extends into its mixed-policy version.","marker":"(Schulman et al., 2017)"},{"why":"Provides GRPO, the group-relative on-policy method that ReMix-GRPO wraps.","marker":"(Shao et al., 2024)"},{"why":"Provides the DeepSeek-R1 distilled base models that ReMix is trained from.","marker":"(Guo et al., 2025)"},{"why":"Provides the DeepScaleR training dataset and the strongest 1.5B baseline used in the efficiency comparison.","marker":"(Luo et al., 2025)"},{"why":"Introduces the high update-to-data ratio mechanism that ReMix adopts for sample efficiency.","marker":"(Chen et al., 2021)"},{"why":"Provides the reincarnating RL concept behind the policy-reincarnation stage.","marker":"(Agarwal et al., 2022)"},{"why":"Inspires the dual-anchored KL-convex constraint that stabilizes off-policy updates.","marker":"(Ma et al., 2024)"},{"why":"Supplies the loss-transformation argument used to derive the Whipping Effect's preference for shorter responses.","marker":"(Fatemi et al., 2025)"},{"why":"Provides V-trace, the truncated-importance-ratio advantage estimator used for off-policy stability.","marker":"(Espeholt et al., 2018)"}],"fun_headline_variants":["ReMix recycles RL data for 450x cheaper LLM reasoning","Off-policy RL finetuning cuts LLM training cost 450x","ReMix: match top math reasoning with 0.011M rollouts","Old RL data slashes LLM reasoning training to 50 steps","ReMix efficient off-policy RL: 30x-450x fewer rollouts"],"cache_read_input_tokens":39808,"weakest_assumption_plain":"The load-bearing premise is that counting rollouts (generated responses) faithfully measures training cost across models that differ in response length, context length, and hardware; if it does not, the reported 30x to 450x reductions are not actual compute reductions.","fun_headline_variants_meta":{"raw":{"variants":["ReMix recycles RL data for 450x cheaper LLM reasoning","Off-policy RL finetuning cuts LLM training cost 450x","ReMix: match top math reasoning with 0.011M rollouts","Old RL data slashes LLM reasoning training to 50 steps","ReMix efficient off-policy RL: 30x-450x fewer rollouts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000601,"raw_usage":{"total_tokens":2939,"prompt_tokens":1206,"completion_tokens":1733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":822,"completion_tokens_details":{"reasoning_tokens":1647}},"tokens_in":822,"tokens_out":1733,"duration_ms":11506,"temperature":1.0,"reasoning_tokens":1647,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:52:10.749810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An audit that records total generated tokens, GPU-hours, and hardware configuration for ReMix and for DeepScaleR and AceReason-Nemotron would settle it: if the token-normalized or hardware-normalized cost gap disappears or reverses, the central efficiency claim fails; if it survives, the claim's magnitude should be re-expressed in those units.","supporting_citations":[{"cited_title":"Cassandras","cited_arxiv_id":null,"evidence_quote":"Supplies the generalized off-policy policy improvement lower bound that legitimizes using historical trajectories in the Mix-PPG objective."},{"cited_title":"Randomized ensembled double q-learning: Learning fast without a model","cited_arxiv_id":null,"evidence_quote":"Introduces the high update-to-data ratio mechanism that ReMix adopts for sample efficiency."},{"cited_title":"Courville, and Marc G","cited_arxiv_id":null,"evidence_quote":"Provides the reincarnating RL concept behind the policy-reincarnation stage."},{"cited_title":"IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures","cited_arxiv_id":null,"evidence_quote":"Provides V-trace, the truncated-importance-ratio advantage estimator used for off-policy stability."}],"review_version":1}