REVIEW 3 major objections 4 minor 14 references
Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Prefix-normalized importance weights outperform response-level weighting in off-policy LLM RL, gaining 3.00 points over GSPO under four-epoch reuse and matching one-epoch performance with a quarter of the rollouts.
desk verdict A transparent and mathematically careful study of a plausible prefix-normalized importance weight; the reported four-epoch advantage is real but rests on one run, so treat it as strong preliminary evidence rather than a settled result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the prefix-normalized policy weight $w^{\mathrm{PN}}_{i,t}=C_{i,t}^{1/t}=\exp\!\big(\tfrac{1}{t}\sum_{k=1}^t\log\rho_{i,k}\big)$, the geometric mean of token likelihood ratios over the causal prefix up to $t$. It carries the argument because it is prefix-local: early positions never see the sampled suffix; scale-controlled: the log weight grows like an average rather than a sum; and it interpolates between the local ratio at $t=1$ and GSPO's full-response geometric mean at $t=L_i$. The reported objective wraps this weight in a position-dependent acceptance gate with tolerance widened by $\sqrt{L_i/t}$, response-level averaging, and GRPO-style group-relative outcome adva
What would settle it
Run the four-epoch comparison with identical evaluation frequency (every 10 steps) for GSPO and PNPO, plus at least five seeds; if PNPO's best observed macro Avg@32 no longer exceeds GSPO by roughly 3 points on all three benchmarks, the claimed off-policy advantage is not established.
Extended reading notes
Core claim
Starting from the exact autoregressive state--action density ratio $C_t=\prod_{k=1}^t \rho_k$ at position $t$, PNPO replaces it with the prefix-normalized policy weight $w^{\mathrm{PN}}_t=C_t^{1/t}=\exp\!\big(\tfrac{1}{t}\sum_{k=1}^t \log\rho_k\big)$, the geometric mean of the per-token likelihood ratios along the causal prefix. For $t>1$, this is not a true target-to-behavior density ratio; it is a deliberately biased transform that keeps prefix dependence, excludes future suffix shifts, and compresses the dynamic range of the cumulative log-ratio. The full objective multiplies this weight by a position-dependent acceptance gate $M_{i,t}$ and a group-relative outcome advantage, with gradien
Load-bearing premise
The headline advantage rests on comparing peak scores from runs evaluated at different frequencies (every 50 vs every 10 steps) with one seed per configuration, so the gap could reflect the denser evaluation of the four-epoch runs rather than the prefix-normalized weights.
Editorial extensions
If this is right
- At four epochs per rollout batch, PNPO attains the best observed Avg@32 on AMC 2023, AIME 2024, and AIME 2025; its unweighted average of 50.24 beats GSPO by 3.00 points.
- Under a matched 2,400-update budget, four-epoch PNPO reaches a final macro Avg@32 of 49.66 after 150 rollout batches, comparable to one-epoch performance of 49.56 after 600 batches, using one quarter as many newly generated responses.
- In the more off-policy four-epoch regime, PNPO leads GSPO at 14 of 15 evaluation checkpoints and finishes 2.66 points ahead, so the gap is visible in the trajectory and not only in independently selected peaks.
- At one epoch, PNPO does not consistently outperform GSPO; the benefit is specific to the larger learner--behavior mismatch, matching the paper's claim that prefix-normalized weighting helps as training moves further off-policy.
Reading between the lines
- Beyond the reported settings, the same prefix-local weighting should apply wherever batches lag the learner: replay buffers, asynchronous actor--learner systems, and offline data, all named by the paper as future work; a direct test would reuse a stale buffer with PNPO versus GSPO under identical evaluation.
- The paper's exact-gradient derivation suggests a different unbiased pairing: behavior advantages with cumulative prefix scores. One could combine PNPO-style scale control with that pairing to get an estimator that is less biased than the reported objective; if PNPO's gain comes from scale control rather than gating, this variant should preserve it.
- The denser evaluation schedule in the four-epoch runs is an untested rival explanation for the 3.00-point gap; a matched-frequency rerun with multiple seeds would distinguish method benefit from checkpoint-count benefit.
- Since $w^{\mathrm{PN}}_t$ approaches 1 for early tokens in long responses, the method may behave differently on very long reasoning traces; length-stratified evaluation would show whether the benefit concentrates in long or short responses.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Prefix-Normalized Policy Optimization (PNPO), which replaces the cumulative importance ratio C_t with its geometric mean over the causal prefix, w^PN_{i,t} = C_{i,t}^{1/t}, and combines it with a position-dependent acceptance gate, group-relative outcome advantages, and response-level aggregation. The authors derive the exact autoregressive state–action change-of-measure ratio and compare it with the local-ratio surrogate used by PPO/GRPO and the response-level statistic used by GSPO. They then evaluate PNPO against GRPO and GSPO in two off-policy regimes induced by one or four PPO epochs per rollout batch on long-context mathematical reasoning. The main empirical claims are that with four epochs PNPO attains the best observed Avg@32 on each of three benchmarks (Table 1), with an unweighted mean 3.00 percentage points above GSPO, and that under a matched 2,400-update budget four-epoch PNPO reaches a final macro Avg@32 of 49.66 after 150 rollout batches, comparable to 49.56 for one-epoch training after 600 batches. The paper is explicit that these are preliminary results with one run per configuration.
Significance. If the empirical claims hold, PNPO would be a practically useful modification for off-policy LLM RL: it preserves causal-prefix dependence while compressing the dynamic range of cumulative importance weights, and the four-epoch results suggest better amortization of rollout generation. The theoretical sections are a genuine strength: Section 2 and Appendix B carefully distinguish the exact cumulative-ratio gradient representations from the biased proximal surrogate, and the paper is transparent that Equation (11) is not an unbiased estimator of the target-policy gradient. The paper also honestly states in Section 6 that the experiments are limited to one model, three benchmarks, one run per configuration, and no ablation of the acceptance gate. However, the central empirical contribution rests on a single stochastic draw per configuration and on per-benchmark maxima of noisy evaluation checkpoints, so the quantitative claims currently have no uncertainty quantification. The significance would be substantially increased by repeated seeds, evaluation-seed variation, and an ablation isolating the prefix-normalized weight from the gate and aggregation choices.
major comments (3)
- [Section 4.2, Table 1, Section C] The headline 3.00 percentage-point advantage is based on one run per configuration (seed 42, Table 2) with no variance estimate. Avg@32 itself uses only 32 sampled responses at temperature 0.7, so every checkpoint has evaluation noise, and no repeated evaluation seeds are reported. Table 1 further selects the best Avg@32 per benchmark across 15 noisy checkpoints in the four-epoch setting, which inflates each entry and can differentially favor one method if its checkpoint-level noise is larger. Figure 2 shows a trajectory-level separation (14 of 15 evaluations and a 2.66 pp final gap), which lessens the peak-selection concern, but it is still a single stochastic run. The abstract and conclusion present these numbers as the main evidence; without standard errors, confidence intervals, or additional seeds, the central quantitative claim is not yet supported.
- [Section 3.2, Eq. (9), Table 2; Section 6] The complete PNPO objective includes a position-dependent acceptance gate with base tolerances (7e-4, 9.5e-4) and scale h(t, L_i)=sqrt(L_i/t), while GSPO uses different clip tolerances (3e-4, 4e-4). Section 6 acknowledges that the gate and response-level aggregation are not ablated, so the evidence applies to the complete configuration and does not isolate the prefix-normalized weight. This is load-bearing for the paper's central claim: the four-epoch advantage could be caused by the gate's tolerance choices or by response-level averaging rather than by the prefix geometric mean. A sensitivity analysis over the base tolerances and an ablation that replaces w^PN_{i,t} with a constant or with the local ratio under the same gate and aggregation are needed to attribute the gain to the proposed weighting.
- [Section 4.1, Table 1] The comparison of 'best observed Avg@32 within each training horizon' uses different evaluation schedules and horizons: one-epoch runs evaluate every 50 steps over 600 steps (12 checkpoints), while four-epoch runs evaluate every 10 steps over 150 steps (15 checkpoints). Within the four-epoch rows this does not affect the PNPO-vs-GSPO comparison, but it makes the abstract's matched-budget final-step comparison (49.66 vs 49.56) the only clean cross-regime comparison, and that comparison has no variance estimate. The paper should either evaluate one-epoch runs at the same frequency or explicitly state that the best-observed numbers are not comparable across the 1-epoch and 4-epoch regimes.
minor comments (4)
- [Section 4.1] The phrase 'evaluation horizon' is used for both the 600-step one-epoch runs and the 150-step four-epoch runs; since the step counts and evaluation cadences differ, it would be clearer to say 'within the corresponding training horizon' and restate the checkpoint counts.
- [Equation (9)] The acceptance gate's scale h(t, L_i)=sqrt(L_i/t) is introduced as a heuristic; a brief explanation of why the sqrt form is chosen, and how sensitive the results are to it, would help the reader interpret the method.
- [Equation (11)] The stop-gradient operator is defined in the text, but the notation could be clarified in a table or symbol list; likewise, the relationship between the response-level normalization 1/L_i after gating ('no renormalization by accepted count') is stated but deserves one sentence of motivation.
- [Figure 1] The bottom traces are described as schematic, but the caption does not specify what covariance model generated them; adding this detail (or marking it as illustrative only) would avoid confusion.
Circularity Check
No significant circularity: PNPO's weight is independently defined and the empirical comparison is not structurally forced.
full rationale
The paper's derivation chain is self-contained. PNPO's weight wPN_{i,t} is defined in Eq. (7) as the geometric mean of per-token likelihood ratios over the causal prefix, and the exact cumulative change-of-measure identity in Eqs. (3)-(4) is derived from the autoregressive factorization rather than assumed. Section B proves the equivalence between the two exact gradient representations (Eqs. (28) and (29)) by direct algebra and the score-function identity; this proof does not import the target conclusion. The practical PNPO surrogate in Eq. (11)/(46) is explicitly labeled a biased proximal approximation, with the acceptance gate and h(t,Li) introduced as heuristics, so no 'prediction' is a fitted parameter renamed. The headline empirical claim is a controlled comparison with matched prompts, update counts, and rollout budgets; Table 1 and Figure 2 report observed Avg@32 values. The stated limitations (single seed, no ablation, one model, Section 6) are robustness concerns, not circular construction. No load-bearing self-citation is used: citations to prior work are contextual and the central derivations are proved in the paper. Therefore the central claims do not reduce to their inputs by definition.
Assumptions & free parameters
free parameters (2)
- PNPO base acceptance-gate tolerances =
lower 7e-4, upper 9.5e-4
- Position-dependent gate scale h(t, Li) =
sqrt(Li / t)
assumptions (4)
- domain assumption Learner state-action distribution is absolutely continuous with respect to the behavior distribution (support condition).
- domain assumption Autoregressive state is the full prefix and transitions deterministically append the sampled token.
- ad hoc to paper The group-relative outcome advantage bA_{i,t} (Eq. 10) is an adequate proxy for the true advantage, and the approximation Aθ_t ≈ Aβ_t (Eq. 44) is acceptable.
- ad hoc to paper The dynamic range of cumulative log-ratios is harmful enough to justify the geometric-mean normalization.
Cite this review
Pith. "Pith review of Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning." pith.science (2026). https://pith.science/paper/MFSZ5HAV
@misc{pith2026260801418,
author = {Pith},
title = {Pith review of: Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/MFSZ5HAV}},
note = {Machine review of arXiv:2608.01418}
}
read the original abstract
Autoregressive rollout generation is a major computational cost in reinforcement learning for large language models. Reusing each rollout batch for additional learner updates amortizes this cost, but later updates become increasingly off-policy as the learner departs from the behavior policy. At a token position, exact off-policy correction must account for both the current action and the probability of reaching its prefix. The cumulative importance ratio provides this correction, but its product form can produce an unwieldy dynamic range. We study Prefix-Normalized Policy Optimization (PNPO), which replaces the cumulative ratio with the geometric mean of likelihood ratios along each causal prefix, preserving causal-prefix dependence at each position while compressing the log-weight scale. In controlled long-context mathematical reasoning experiments, we induce two off-policy regimes by using one or four policy-update epochs per rollout batch. PNPO does not consistently outperform GSPO with one epoch. With four epochs, it attains the highest observed Avg@32 on each benchmark; the unweighted mean of the three independently selected benchmark peaks is 50.24, 3.00 percentage points above GSPO. Under a matched 2,400-update budget, four-epoch PNPO reaches a final macro Avg@32 of 49.66 after 150 rollout batches, comparable to the 49.56 reached after 600 batches with one epoch. These results provide preliminary evidence that PNPO can be advantageous as training moves further off-policy.
Figures
Reference graph
Works this paper leans on
-
[1]
Efficient RL training for LLMs with experience replay.arXiv preprint arXiv:2604.08706,
Charles Arnal, Vivien Cabannes, Taco Cohen, Julia Kempe, and Remi Munos. Efficient RL training for LLMs with experience replay.arXiv preprint arXiv:2604.08706,
-
[4]
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
Zhenyu Hou, Yujiang Li, Jie Tang, and Yuxiao Dong. Single-rollout asynchronous optimization for agentic reinforcement learning.arXiv preprint arXiv:2607.07508,
-
[5]
Tianhao Hu, Xiangcheng Liu, Yuchun Miao, Youshao Xiao, Hongyu Zang, Yang Zheng, Xuan Huang, Jinrui Ding, Yufei Zhang, et al. DORA: A scalable asynchronous reinforcement learning system for language model training.arXiv preprint arXiv:2604.26256,
-
[6]
A step back: Prefix importance ratio stabilizes policy optimization.arXiv preprint arXiv:2601.22718,
Shiye Lei, Zhihao Cheng, and Dacheng Tao. A step back: Prefix importance ratio stabilizes policy optimization.arXiv preprint arXiv:2601.22718,
-
[7]
Xingyu Lin, Yilin Wen, Du Su, Jinchang Hou, En Wang, Wenbin Liu, Chenfu Bao, and Zhonghou Lv. Token-level policy optimization: Linking group-level rewards to token-level aggregation via sequence-level likelihood.arXiv preprint arXiv:2604.12736,
-
[11]
Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
Yuheng Zhang, Chenlu Ye, Shuowei Jin, Changlong Yu, Wei Xiong, Saurabh Sahu, and Nan Jiang. Rethinking importance sampling in LLM policy optimization: A cumulative token perspective. arXiv preprint arXiv:2605.07331,
-
[12]
Group sequence policy optimization
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. arXiv preprint arXiv:2507.18071,
-
[13]
LX t=1 CtAβ t tX k=1 zk # = LX k=1 Eπβ
10 A FINITE-HORIZONLOCALSURROGATES ANDEXACTPREFIXCHANGE OF MEASURE Fix a prompt x and consider an undiscounted autoregressive process with maximum horizon L. At positiont, lets t = (x, y<t)anda t =y t. For a policyπ, define dπ t (s|x) = Pr(st =s|x, π),(12) and let Jx(π) and Aπ t denote its expected return and finite-horizon advantage, respectively. For re...
work page 2017
Show all 14 references
-
[14]
LX t=k Aβ t (st, at) sk, ak # =E πθ
Under the determin- istic token-append transition, Aβ t (st, at) =rt(st, at) +V β t+1(st+1)−V β t (st).(36) 14 Substitution into the last expectation in Equation (35) gives Eπθ " LX t=k Aβ t (st, at) sk, ak # =E πθ " LX t=k rt +V β t+1 −V β t sk, ak # =E πθ " LX t=k rt sk, ak ...
2023
-
[2000]
LLMs can learn to reason via off-policy RL.arXiv preprint arXiv:2602.19362,
Daniel Ritter, Owen Oertell, Bradley Guo, Jonathan Chang, Kianté Brantley, and Wen Sun. LLMs can learn to reason via off-policy RL.arXiv preprint arXiv:2602.19362,
-
[2015]
Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,
-
[2017]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[2025]
9 Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, and Glen Berseth
doi: 10.1038/s41586-025-09422-z. 9 Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, and Glen Berseth. Align and filter: Improving performance in asynchronous on-policy RL. arXiv preprint arXiv:2603.01365,
-
[2026]
DeepSeek-V3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556,
DeepSeek-AI et al. DeepSeek-V3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.