REVIEW 3 major objections 5 minor 5 references
A token-level mix-policy method that reuses the model's own historical checkpoints as reasoning prefixes, and lets the current policy finish the sequence, beats standard on-policy RL and prior off-policy extensions on math reasoning benchma
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 06:55 UTC pith:VCFS2ZGN
load-bearing objection A genuinely new token-level mix-policy idea, but the importance-weighting math is not actually unbiased and the empirical reporting is too selective to support the 'consistent' claim. the 3 major comments →
Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the question for off-policy RL in LLMs is not whether to use stale data but where in the sequence it should appear. SOUP confines off-policy influence to a prefix sampled from an earlier policy checkpoint, then lets the reference policy (a snapshot of the current policy) generate the rest of the response. The importance ratio for each token is computed against the policy that actually generated it: πθ/πθbehavior for prefix tokens, πθ/πθold for continuation tokens. This token-level decomposition is argued to avoid the sample-level clipping saturation and distribution shift that plague sequence-level mixing, preserving gradient quality while restoring sampling diversi
What carries the argument
The key mechanism is the single-sample mix-policy construction with token-level importance weighting. A response is split into an off-policy prefix sampled by a stale behavior policy and an on-policy suffix sampled by the reference policy. The importance ratio for each token is defined against the generating policy of that token, so the gradient signal from the prefix is corrected by πθ/πθbehavior, while the suffix's ratio is πθ/πθold as in standard on-policy DAPO. The truncation point is flexible — either a fixed length-ratio or entropy-selected positions — and the method is shown to be robust to both choices.
Load-bearing premise
The method's validity depends on the token-level importance-ratio estimator in Equation 5 being a stable and low-bias surrogate for the policy gradient when the sequence is generated by two different policies and then optimized with DAPO's per-token clipping and group-relative advantages — a property the paper does not prove.
What would settle it
Measure the gradient estimator's bias and variance on a controlled task: construct a sequence with a known prefix distribution shift, compute the SOUP token-level importance-ratio gradient, and compare it with the true REINFORCE gradient over many samples; if the estimate diverges significantly from the true gradient even at small shift, the reported gains would be artifacts. Alternatively, ablate by replacing the off-policy prefix with a random prefix from the same checkpoint with the same token distribution — if performance drops, the mixing position, not the historical data, is doing the wo
If this is right
- On-policy training's performance ceiling is not intrinsic; replaying the model's own historical reasoning prefixes increases achievable average scores on hard math benchmarks.
- Entropy collapse can be mitigated without external off-policy data: diversity comes from the learner's own earlier checkpoints, so the method needs no stronger external model.
- Token-level importance ratios provide a principled way to compute gradients for mixed-source sequences; this extends the applicability of clipped surrogate objectives beyond on-policy data.
- The pass@k inference result suggests that mixing checkpoints at inference time (relay sampling) expands the explorable solution space, which could be exploited for test-time scaling.
- The method's robustness to different truncation strategies means the core benefit is the position-aware mix, not the specific truncation rule.
Where Pith is reading between the lines
- If the mechanism is as advertised, a natural extension is to adaptively choose the truncation point or the stale checkpoint based on prompt difficulty or token entropy, rather than fixed ratios; the paper's own limitation section notes this.
- The paper leaves open the question of whether the token-level importance-ratio estimator is biased; a variance/unbiasedness analysis on a controlled task would determine whether the reported gains are a true mixing effect or partly an artifact of hyperparameter selection.
- The relay-sampling pass@k result hints at a decentralized inference scheme where multiple stale checkpoints serve as prefix generators, effectively turning model training into a self-distilling exploration process; this could be tested at larger scale.
- The ~27% per-step overhead might shrink substantially with the proposed prefix-only generation; if so, the method would be a cheap alternative to replay-buffer approaches for off-policy LLM RL.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SOUP (Single-sample Mix-policy Unified Paradigm), a modification of on-policy DAPO/GRPO for LLM reinforcement learning. Each training sample consists of a prefix generated by a historical policy checkpoint—obtained by sampling a full response from πθbehavior and truncating it—and a continuation generated by the reference policy πθold. The DAPO objective is applied with token-level importance ratios: πθ/πθbehavior for prefix tokens and πθ/πθold for continuation tokens. The authors evaluate SOUP on six mathematical reasoning benchmarks with Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, comparing to on-policy DAPO and the off-policy methods LUFFY and M2PO, and report average improvements for the length-ratio-based variant. They also present ablations of the truncation ratio and sampling batch number T, an analysis of entropy and clipping across token positions, and a pass@k inference-time relay-sampling analysis.
Significance. The central idea—restricting off-policy influence to a prefix so that the critical continuation remains on-policy—is a novel and reasonable approach to reusing stale data in LLM RL, and it may stimulate further work on position-aware off-policy mixing. The paper provides detailed hyperparameter settings, ablations, and analyses of entropy/clipping, which are useful for reproducibility. However, the theoretical justification based on token-level importance ratios is flawed (the ratio omits the truncation selection factor), and the empirical evidence is weakened by post-hoc hyperparameter selection, absence of error bars, and the poor performance of the entropy-based variant. The strong claim that SOUP consistently outperforms standard on-policy training is not supported by the data as presented. The paper's conceptual contribution is promising, but the current form needs substantial revision.
major comments (3)
- [Sec. 3.2, Eq. (5)] The prefix importance ratio is biased. Since sθbehavior∼truncate(s) with s∼πθbehavior(·|q), the marginal distribution of the observed prefix is q(p)=∑_{s:truncate(s)=p}πθbehavior(s|q), not πθbehavior(p). For length-ratio and entropy truncation, the truncation point depends on the full response, so the ratio πθ(p)/πθbehavior(p) omits the selection factor πθbehavior(p)/q(p), which is data-dependent and generally ≠1. The token-level weight is therefore not a valid likelihood-ratio importance weight; the theoretical justification in Sec. 3.2 is not established. Since the abstract attributes the method's stability and gains to these ratios, this is load-bearing. Fixing requires changing the sampling procedure (e.g., direct prefix generation) or computing the correct q(p).
- [Table 1] The claim of 'consistently outperforms' (Abstract, Sec. 1, Conclusion) is contradicted by SOUPentropy. On DeepSeek-R1-Distill-Qwen-1.5B, SOUPentropy averages 49.46 vs 53.77 for on-policy DAPO and loses on every benchmark; on Qwen2.5-Math-7B it gains only 0.95. The qualification in Sec. 4.2 that methods 'outperform baselines under appropriate settings' is not a consistency claim. The abstract and conclusion should be revised to state the conditions under which each truncation variant helps.
- [Sec. 4.1] The statement 'The best performance across all tested settings is reported in the main results' and the lack of error bars / multiple seeds make the reported gains (e.g., +2.86 on Qwen, +0.68 on DeepSeek) hard to separate from hyperparameter tuning. Tables 2-3 show metric shifts of several points with ratio and T; selecting the best setting from the same runs inflates the apparent advantage. Report multiple seeds or standard errors and pre-specify the settings, or show that gains persist across all settings without selection.
minor comments (5)
- [Title/Abstract] The arXiv title 'Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs' differs from the full-text title 'SOUP: Token-level Single-sample Mix-policy Reinforcement Learning for Large Language Models'. Please unify the title used in the submission.
- [Tables 2 and 3] Table 2 reports an average of 50.81 at ratio=50%, while Table 1 reports 51.16 for the same setting (per Appendix A). The caption of Table 2 does not specify the T value, making the ablation non-reproducible. Please clarify the exact configuration used in each ablation.
- [Sec. 4.2] The statement that MATH-500 performance is 'comparably' to on-policy should be quantified; Table 1 shows both slight improvements and slight declines depending on model and variant.
- [Sec. 5.1] The pass@k relay-sampling experiment is an inference-time heuristic, not the training procedure used by SOUP. The link to training benefits is speculative and should be framed as an exploratory result rather than direct evidence for the training method.
- [Appendix A] The notation Tratio and Tentropy is not defined; clarify that these are the sampling-batch-number T for the length-ratio and entropy truncation variants, respectively.
Circularity Check
No significant circularity: SOUP's empirical gains are evaluated against fixed baselines; the questionable importance-ratio estimator in Eq. 5 is a correctness concern, not a circular derivation.
full rationale
The derivation in Sec. 3.2 constructs mixed-policy samples by truncating responses from a historical checkpoint (Eq. 2) and continuing on-policy (Eq. 3), then defines a token-level importance ratio (Eq. 5). The paper's central claim—that this improves exploration while preserving stability—is not obtained by inverting a fitted constant, by defining the target in terms of the method, or by relying on the authors' own prior results. The supporting evidence is empirical: comparisons against on-policy DAPO, LUFFY, and M2PO (Table 1), ablations over ratio and T (Tables 2-3), and behavioral analyses of entropy and clipping (Figure 5). No load-bearing step reduces by construction to its input: the performance metric is external, and the importance ratio is not derived from that metric. There is no self-citation chain used as the basis of the method; GRPO, DAPO, LUFFY, and M2PO are external prior work, and the novelty claim is an architectural modification rather than a renaming of a known result. The skeptical concern about Eq. 5—that the prefix distribution is the pushforward of truncate(s) applied to full responses, so πθbehavior(prefix) omits the truncation selection factor—is a potential bias or missing justification, not circularity, because the estimator is not defined in terms of the conclusion it is used to support. Similarly, reporting the best configuration across tested hyperparameters and selecting checkpoints on AIME24/AIME25 is benchmark selection, not a fitted parameter being renamed as a prediction. For these reasons, the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (3)
- truncation ratio ratio =
50% (Qwen7B), 30% (DeepSeek1.5B)
- sampling batch count T =
8 (Qwen ratio), 16 (Qwen entropy), 4 (DeepSeek)
- entropy truncation k =
32
axioms (5)
- domain assumption Token-level importance ratios in Eq.5 give low-bias, stable policy gradient estimates under the DAPO clipped objective with group-relative advantages.
- domain assumption Group-relative advantage normalization remains valid when samples in a group start from different stale prefixes.
- domain assumption Truncation preserves the reasoning context needed for the on-policy continuation to solve the problem.
- domain assumption The model's own historical checkpoints provide useful exploratory diversity.
- standard math DAPO/GRPO is an appropriate on-policy baseline and objective.
read the original abstract
On-policy reinforcement learning (RL) for language model post-training suffers from a fundamental tension: as training progresses, policy entropy collapses and sampling diversity diminishes, causing the model to ``forget'' its own earlier exploratory capacity. While off-policy data can restore diversity, existing methods mix entire trajectories at the sequence level, introducing severe policy mismatch and training instability. We argue that the core question is not \emph{whether} to use off-policy data, but \emph{where} in the sequence it should appear. Based on this insight, we propose \textbf{Thinking Seeds}, a token-level mix-policy framework that uses the model's own historical checkpoints as off-policy prefixes, providing diverse starting points for reasoning, while the critical continuation is generated on-policy to preserve gradient quality. Through token-level importance ratios, Thinking Seeds effectively leverages historical diversity without compromising training stability. Extensive experiments across models and mathematical reasoning benchmarks demonstrate that Thinking Seeds consistently outperforms standard on-policy training and existing off-policy extensions. Our analysis reveals that the method maintains higher effective entropy, reduces gradient loss from clipping, and expands the explorable solution space, clarifying how position-aware mix-policy modeling improves both exploration and final performance in LLM RL.
Figures
Reference graph
Works this paper leans on
-
[2]
Michael Noukhovitch, Shengyi Huang, Sophie Xhon- neux, Arian Hosseini, Rishabh Agarwal, and Aaron C
Revisiting group relative policy optimization: Insights into on-policy and off-policy training.CoRR, abs/2505.22257. Michael Noukhovitch, Shengyi Huang, Sophie Xhon- neux, Arian Hosseini, Rishabh Agarwal, and Aaron C. Courville. 2025. Asynchronous RLHF: faster and more efficient off-policy RL for language models. InThe Thirteenth International Conference ...
Pith/arXiv arXiv 2025
-
[5]
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang
DAPO: an open-source LLM reinforcement learning system at scale.CoRR, abs/2503.14476. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize rea- soning capacity in llms beyond the base model? CoRR, abs/2504.13837. Feng Zhang, Zezhong Tan, Xinhong Ma, Ziqiang Dong,...
Pith/arXiv arXiv 2025
-
[1063]
The MIT Press. Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. 2024. Understand- ing the performance gap between online and offline alignment algorithms.CoRR, abs/2405.08448. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shix- uan Liu...
Pith/arXiv arXiv 2024
-
[2022]
Training language models to follow instruc- tions with human feedback. InAdvances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. 2025. Demystifying reas...
Pith/arXiv arXiv 2022
-
[2025]
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D
Areal: A large-scale asynchronous reinforce- ment learning system for language reasoning.CoRR, abs/2505.24298. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. 2025. Cog- nitive behaviors that enable self-improving reason- ers, or, four habits of highly effective stars.CoRR, abs/2503.01307. Yaru Hao, Li Dong, Xun Wu, Sh...
Pith/arXiv arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.