Pith. sign in

REVIEW 18 cited by

RePO: Replay-Enhanced Policy Optimization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.09340 v1 pith:FDDXSKF6 submitted 2025-06-11 cs.CL cs.AIcs.LG

RePO: Replay-Enhanced Policy Optimization

Siheng Li , Zhanhui Zhou , Wai Lam , Chao Yang , Chaochao Lu This is my paper
classification cs.CL cs.AIcs.LG
keywords optimizationrepopolicycomputationaldiversegrpollmsoff-policy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Reinforcement learning (RL) is vital for optimizing large language models (LLMs). Recent Group Relative Policy Optimization (GRPO) estimates advantages using multiple on-policy outputs per prompt, leading to high computational costs and low data efficiency. To address this, we introduce Replay-Enhanced Policy Optimization (RePO), which leverages diverse replay strategies to retrieve off-policy samples from a replay buffer, allowing policy optimization based on a broader and more diverse set of samples for each prompt. Experiments on five LLMs across seven mathematical reasoning benchmarks demonstrate that RePO achieves absolute average performance gains of $18.4$ and $4.1$ points for Qwen2.5-Math-1.5B and Qwen3-1.7B, respectively, compared to GRPO. Further analysis indicates that RePO increases computational cost by $15\%$ while raising the number of effective optimization steps by $48\%$ for Qwen3-1.7B, with both on-policy and off-policy sample numbers set to $8$. The repository can be accessed at https://github.com/SihengLi99/RePO.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models

    cs.CV 2026-04 unverdicted novelty 8.0

    OP-GRPO is the first off-policy GRPO method for flow-matching models that reuses trajectories via replay buffer and importance sampling corrections, matching on-policy performance with 34.2% of the training steps.

  2. Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

    cs.CL 2026-06 unverdicted novelty 7.0

    ZPPO improves distillation to small vision-language models by using binary and negative candidate prompts plus a replay buffer for hard questions, outperforming standard distillation and GRPO on a 31-benchmark suite w...

  3. Near-Future Policy Optimization

    cs.LG 2026-04 unverdicted novelty 7.0

    NPO uses a policy's own near-future checkpoint as auxiliary trajectories to maximize effective learning signal S = Q/V, improving performance from 57.88 to 63.15 on Qwen3-VL-8B-Instruct with GRPO while accelerating co...

  4. Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

    cs.LG 2026-04 unverdicted novelty 7.0

    This survey introduces the Generate-Filter-Control-Replay (GFCR) taxonomy to structure rollout pipelines for RL-based post-training of reasoning LLMs.

  5. Experience Augmented Policy Optimization for LLM Reasoning

    cs.LG 2026-06 conditional novelty 6.0

    EAPO injects a prior RL policy's token choices at critical decision points during rollout and beats standard RLVR baselines on math and science reasoning benchmarks.

  6. Rollout-Level Advantage-Prioritized Experience Replay for GRPO

    cs.LG 2026-06 conditional novelty 6.0

    Rollout-level advantage-prioritized experience replay for GRPO recycles high-advantage individual rollouts with age eviction and fresh-anchored batches to outperform standard GRPO on math benchmarks, with gains increa...

  7. Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

    cs.LG 2026-06 unverdicted novelty 6.0

    RLVR exhibits correct-set turnover where solved problems regress during training, and a periodic review mechanism exploiting a repair-window principle improves retention and performance over baselines.

  8. Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning

    cs.LG 2026-04 unverdicted novelty 6.0

    Kernel smoothing enables accurate low-variance value and gradient estimates for policy optimization in LLM reasoning under tight sampling constraints per prompt.

  9. Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning

    cs.LG 2026-04 unverdicted novelty 6.0

    Kernel smoothing yields accurate value and gradient estimates for low-variance policy learning in LLM reasoning under tight per-prompt sampling budgets.

  10. Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVR

    cs.CL 2026-02 conditional novelty 6.0

    Using the GFlowNet partition function as an online accuracy estimate for difficulty-based prompt selection and replay improves sample efficiency of RLVR for LLM reasoning.

  11. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.5

    Injecting correct reference-policy samples and optimizing a mixed importance-sampling ratio prevents validation collapse and raises asymptotic math-reasoning scores beyond reverse-KL baselines.

  12. RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

    cs.LG 2026-07 conditional novelty 5.5

    Reward-Swap Policy Optimization uses dense-reward exploration to generate diverse trajectories that then train the final policy strictly under true outcome rewards, lifting multi-turn agent performance.

  13. ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

    cs.LG 2026-07 conditional novelty 5.0

    ARMOR adds correct reference-policy anchor samples to each RL batch and optimizes a reference-mixture importance ratio, preventing validation collapse and extending performance gains.

  14. Experience Augmented Policy Optimization for LLM Reasoning

    cs.LG 2026-06 unverdicted novelty 5.0

    EAPO reuses prior RL policy experience adaptively at decision points in LLM rollouts with adapted importance sampling and reports gains over prior RLVR methods on math benchmarks.

  15. Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

    cs.LG 2026-06 unverdicted novelty 5.0

    Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.

  16. RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning

    cs.LG 2026-05 unverdicted novelty 5.0

    POPO uses recency-based prioritized group replay and decoupled off-policy optimization to avoid zero-variance ineffective samples in RLVR, accelerating LLM reasoning finetuning with fewer rollouts.

  17. Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

    cs.CL 2026-04 accept novelty 5.0

    LLM post-training is unified as off-policy or on-policy interventions that expand support for useful behaviors, reshape policies within reachable states, or consolidate behavior across training stages.

  18. AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

    cs.CL 2025-12 reject novelty 5.0

    AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.