Pith. sign in

REVIEW 3 major objections 5 minor 5 references

A token-level mix-policy method that reuses the model's own historical checkpoints as reasoning prefixes, and lets the current policy finish the sequence, beats standard on-policy RL and prior off-policy extensions on math reasoning benchma

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 06:55 UTC pith:VCFS2ZGN

load-bearing objection A genuinely new token-level mix-policy idea, but the importance-weighting math is not actually unbiased and the empirical reporting is too selective to support the 'consistent' claim. the 3 major comments →

arxiv 2601.21476 v2 pith:VCFS2ZGN submitted 2026-01-29 cs.CL

Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs

classification cs.CL
keywords reinforcement learninglarge language modelsoff-policy RLtoken-level importance ratioGRPODAPOentropy collapsemathematical reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard reinforcement-learning fine-tuning of LLMs suffers from entropy collapse: the policy gets overconfident and stops exploring. Prior attempts to reuse old data assumed whole trajectories should be mixed in, which introduces policy mismatch and instability. This paper argues that the off-policy data should appear only in the prefix of a sequence, with the continuation regenerated by the current policy. It proposes SOUP, which samples a reasoning prefix from a historical checkpoint, regenerates the suffix on-policy, and computes token-level importance ratios separately for the two parts. On math reasoning benchmarks across two model backbones, SOUP reports consistent gains over on-policy training and previous off-policy methods, with smoother reward curves and higher effective entropy.

Core claim

The central claim is that the question for off-policy RL in LLMs is not whether to use stale data but where in the sequence it should appear. SOUP confines off-policy influence to a prefix sampled from an earlier policy checkpoint, then lets the reference policy (a snapshot of the current policy) generate the rest of the response. The importance ratio for each token is computed against the policy that actually generated it: πθ/πθbehavior for prefix tokens, πθ/πθold for continuation tokens. This token-level decomposition is argued to avoid the sample-level clipping saturation and distribution shift that plague sequence-level mixing, preserving gradient quality while restoring sampling diversi

What carries the argument

The key mechanism is the single-sample mix-policy construction with token-level importance weighting. A response is split into an off-policy prefix sampled by a stale behavior policy and an on-policy suffix sampled by the reference policy. The importance ratio for each token is defined against the generating policy of that token, so the gradient signal from the prefix is corrected by πθ/πθbehavior, while the suffix's ratio is πθ/πθold as in standard on-policy DAPO. The truncation point is flexible — either a fixed length-ratio or entropy-selected positions — and the method is shown to be robust to both choices.

Load-bearing premise

The method's validity depends on the token-level importance-ratio estimator in Equation 5 being a stable and low-bias surrogate for the policy gradient when the sequence is generated by two different policies and then optimized with DAPO's per-token clipping and group-relative advantages — a property the paper does not prove.

What would settle it

Measure the gradient estimator's bias and variance on a controlled task: construct a sequence with a known prefix distribution shift, compute the SOUP token-level importance-ratio gradient, and compare it with the true REINFORCE gradient over many samples; if the estimate diverges significantly from the true gradient even at small shift, the reported gains would be artifacts. Alternatively, ablate by replacing the off-policy prefix with a random prefix from the same checkpoint with the same token distribution — if performance drops, the mixing position, not the historical data, is doing the wo

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On-policy training's performance ceiling is not intrinsic; replaying the model's own historical reasoning prefixes increases achievable average scores on hard math benchmarks.
  • Entropy collapse can be mitigated without external off-policy data: diversity comes from the learner's own earlier checkpoints, so the method needs no stronger external model.
  • Token-level importance ratios provide a principled way to compute gradients for mixed-source sequences; this extends the applicability of clipped surrogate objectives beyond on-policy data.
  • The pass@k inference result suggests that mixing checkpoints at inference time (relay sampling) expands the explorable solution space, which could be exploited for test-time scaling.
  • The method's robustness to different truncation strategies means the core benefit is the position-aware mix, not the specific truncation rule.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism is as advertised, a natural extension is to adaptively choose the truncation point or the stale checkpoint based on prompt difficulty or token entropy, rather than fixed ratios; the paper's own limitation section notes this.
  • The paper leaves open the question of whether the token-level importance-ratio estimator is biased; a variance/unbiasedness analysis on a controlled task would determine whether the reported gains are a true mixing effect or partly an artifact of hyperparameter selection.
  • The relay-sampling pass@k result hints at a decentralized inference scheme where multiple stale checkpoints serve as prefix generators, effectively turning model training into a self-distilling exploration process; this could be tested at larger scale.
  • The ~27% per-step overhead might shrink substantially with the proposed prefix-only generation; if so, the method would be a cheap alternative to replay-buffer approaches for off-policy LLM RL.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SOUP (Single-sample Mix-policy Unified Paradigm), a modification of on-policy DAPO/GRPO for LLM reinforcement learning. Each training sample consists of a prefix generated by a historical policy checkpoint—obtained by sampling a full response from πθbehavior and truncating it—and a continuation generated by the reference policy πθold. The DAPO objective is applied with token-level importance ratios: πθ/πθbehavior for prefix tokens and πθ/πθold for continuation tokens. The authors evaluate SOUP on six mathematical reasoning benchmarks with Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, comparing to on-policy DAPO and the off-policy methods LUFFY and M2PO, and report average improvements for the length-ratio-based variant. They also present ablations of the truncation ratio and sampling batch number T, an analysis of entropy and clipping across token positions, and a pass@k inference-time relay-sampling analysis.

Significance. The central idea—restricting off-policy influence to a prefix so that the critical continuation remains on-policy—is a novel and reasonable approach to reusing stale data in LLM RL, and it may stimulate further work on position-aware off-policy mixing. The paper provides detailed hyperparameter settings, ablations, and analyses of entropy/clipping, which are useful for reproducibility. However, the theoretical justification based on token-level importance ratios is flawed (the ratio omits the truncation selection factor), and the empirical evidence is weakened by post-hoc hyperparameter selection, absence of error bars, and the poor performance of the entropy-based variant. The strong claim that SOUP consistently outperforms standard on-policy training is not supported by the data as presented. The paper's conceptual contribution is promising, but the current form needs substantial revision.

major comments (3)
  1. [Sec. 3.2, Eq. (5)] The prefix importance ratio is biased. Since sθbehavior∼truncate(s) with s∼πθbehavior(·|q), the marginal distribution of the observed prefix is q(p)=∑_{s:truncate(s)=p}πθbehavior(s|q), not πθbehavior(p). For length-ratio and entropy truncation, the truncation point depends on the full response, so the ratio πθ(p)/πθbehavior(p) omits the selection factor πθbehavior(p)/q(p), which is data-dependent and generally ≠1. The token-level weight is therefore not a valid likelihood-ratio importance weight; the theoretical justification in Sec. 3.2 is not established. Since the abstract attributes the method's stability and gains to these ratios, this is load-bearing. Fixing requires changing the sampling procedure (e.g., direct prefix generation) or computing the correct q(p).
  2. [Table 1] The claim of 'consistently outperforms' (Abstract, Sec. 1, Conclusion) is contradicted by SOUPentropy. On DeepSeek-R1-Distill-Qwen-1.5B, SOUPentropy averages 49.46 vs 53.77 for on-policy DAPO and loses on every benchmark; on Qwen2.5-Math-7B it gains only 0.95. The qualification in Sec. 4.2 that methods 'outperform baselines under appropriate settings' is not a consistency claim. The abstract and conclusion should be revised to state the conditions under which each truncation variant helps.
  3. [Sec. 4.1] The statement 'The best performance across all tested settings is reported in the main results' and the lack of error bars / multiple seeds make the reported gains (e.g., +2.86 on Qwen, +0.68 on DeepSeek) hard to separate from hyperparameter tuning. Tables 2-3 show metric shifts of several points with ratio and T; selecting the best setting from the same runs inflates the apparent advantage. Report multiple seeds or standard errors and pre-specify the settings, or show that gains persist across all settings without selection.
minor comments (5)
  1. [Title/Abstract] The arXiv title 'Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMs' differs from the full-text title 'SOUP: Token-level Single-sample Mix-policy Reinforcement Learning for Large Language Models'. Please unify the title used in the submission.
  2. [Tables 2 and 3] Table 2 reports an average of 50.81 at ratio=50%, while Table 1 reports 51.16 for the same setting (per Appendix A). The caption of Table 2 does not specify the T value, making the ablation non-reproducible. Please clarify the exact configuration used in each ablation.
  3. [Sec. 4.2] The statement that MATH-500 performance is 'comparably' to on-policy should be quantified; Table 1 shows both slight improvements and slight declines depending on model and variant.
  4. [Sec. 5.1] The pass@k relay-sampling experiment is an inference-time heuristic, not the training procedure used by SOUP. The link to training benefits is speculative and should be framed as an exploratory result rather than direct evidence for the training method.
  5. [Appendix A] The notation Tratio and Tentropy is not defined; clarify that these are the sampling-batch-number T for the length-ratio and entropy truncation variants, respectively.

Circularity Check

0 steps flagged

No significant circularity: SOUP's empirical gains are evaluated against fixed baselines; the questionable importance-ratio estimator in Eq. 5 is a correctness concern, not a circular derivation.

full rationale

The derivation in Sec. 3.2 constructs mixed-policy samples by truncating responses from a historical checkpoint (Eq. 2) and continuing on-policy (Eq. 3), then defines a token-level importance ratio (Eq. 5). The paper's central claim—that this improves exploration while preserving stability—is not obtained by inverting a fitted constant, by defining the target in terms of the method, or by relying on the authors' own prior results. The supporting evidence is empirical: comparisons against on-policy DAPO, LUFFY, and M2PO (Table 1), ablations over ratio and T (Tables 2-3), and behavioral analyses of entropy and clipping (Figure 5). No load-bearing step reduces by construction to its input: the performance metric is external, and the importance ratio is not derived from that metric. There is no self-citation chain used as the basis of the method; GRPO, DAPO, LUFFY, and M2PO are external prior work, and the novelty claim is an architectural modification rather than a renaming of a known result. The skeptical concern about Eq. 5—that the prefix distribution is the pushforward of truncate(s) applied to full responses, so πθbehavior(prefix) omits the truncation selection factor—is a potential bias or missing justification, not circularity, because the estimator is not defined in terms of the conclusion it is used to support. Similarly, reporting the best configuration across tested hyperparameters and selecting checkpoints on AIME24/AIME25 is benchmark selection, not a fitted parameter being renamed as a prediction. For these reasons, the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

SOUP introduces no new physical or model-level entities. Its load-bearing content is a set of modeling choices: the truncation ratio, the checkpoint refresh interval, the entropy truncation window, and the unproven presumption that token-level importance ratios stabilize the mixed-policy gradient. The free parameters are selected on the evaluation benchmarks, which is the main empirical weakness.

free parameters (3)
  • truncation ratio ratio = 50% (Qwen7B), 30% (DeepSeek1.5B)
    Central hyperparameter controlling the off-policy prefix length; the best value is selected from ablation on the evaluation benchmarks, so it is fit to the target tasks.
  • sampling batch count T = 8 (Qwen ratio), 16 (Qwen entropy), 4 (DeepSeek)
    Controls how often the historical policy is refreshed; chosen per model and truncation variant from the ablation in Table 3, again on evaluation benchmarks.
  • entropy truncation k = 32
    Number of high-entropy candidate positions from which the truncation point is randomly sampled; fixed across models but a free design choice of the method.
axioms (5)
  • domain assumption Token-level importance ratios in Eq.5 give low-bias, stable policy gradient estimates under the DAPO clipped objective with group-relative advantages.
    The paper asserts that confining off-policy influence to prefixes and using token-level ratios preserves training stability, but provides no proof of unbiasedness or variance control when prefix and suffix come from different policies.
  • domain assumption Group-relative advantage normalization remains valid when samples in a group start from different stale prefixes.
    GRPO normalizes rewards by the mean/std of the group; SOUP mixes samples with different historical prefixes, so the relative ranking may be distorted. The paper does not analyze this.
  • domain assumption Truncation preserves the reasoning context needed for the on-policy continuation to solve the problem.
    Both length-ratio and entropy truncation must stop at a point where the remaining suffix can still contain the critical reasoning steps; if truncation removes the solution-bearing tokens, performance would drop.
  • domain assumption The model's own historical checkpoints provide useful exploratory diversity.
    SOUP's entire premise is that earlier versions of the learner are a good source of diverse prefixes; this is plausible but not independently established.
  • standard math DAPO/GRPO is an appropriate on-policy baseline and objective.
    The paper builds on the published DAPO formulation (Eq.1); this is prior art, not contributed by the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 13734 in / 10881 out tokens · 120093 ms · 2026-08-03T06:55:01.735883+00:00 · methodology

0 comments
read the original abstract

On-policy reinforcement learning (RL) for language model post-training suffers from a fundamental tension: as training progresses, policy entropy collapses and sampling diversity diminishes, causing the model to ``forget'' its own earlier exploratory capacity. While off-policy data can restore diversity, existing methods mix entire trajectories at the sequence level, introducing severe policy mismatch and training instability. We argue that the core question is not \emph{whether} to use off-policy data, but \emph{where} in the sequence it should appear. Based on this insight, we propose \textbf{Thinking Seeds}, a token-level mix-policy framework that uses the model's own historical checkpoints as off-policy prefixes, providing diverse starting points for reasoning, while the critical continuation is generated on-policy to preserve gradient quality. Through token-level importance ratios, Thinking Seeds effectively leverages historical diversity without compromising training stability. Extensive experiments across models and mathematical reasoning benchmarks demonstrate that Thinking Seeds consistently outperforms standard on-policy training and existing off-policy extensions. Our analysis reveals that the method maintains higher effective entropy, reduces gradient loss from clipping, and expands the explorable solution space, clarifying how position-aware mix-policy modeling improves both exploration and final performance in LLM RL.

Figures

Figures reproduced from arXiv: 2601.21476 by Chenxi Sun, Deyi Xiong, Lei Yang, Renren Jin, Wei Bi.

Figure 1
Figure 1. Figure 1: Overview of SOUP. SOUP unifies off-policy and on-policy data at the token level within a single sample. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training rewards of different methods. 8192 (Yan et al., 2025), where the response length is set to 8k. Unless otherwise noted, we used a batch size of 32, a clipping-higher of 0.28, and sampled 8 responses per prompt. Additionally, we set the entropy-based truncation parameter k at 32. For the length ratio ratio and the sampling batch T, we perform detailed ablation studies to test the performance robustn… view at source ↗
Figure 3
Figure 3. Figure 3: Word cloud diagrams of tokens at the trunca [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The pass@k difference between SOUP infer￾ence and single-model inference at the inference phase. The greater the value above the baseline of 0, the better SOUP performs. previous thinking, while others may continue it. 4.3 Ablation Study Varying the Length Ratio. We studied the effect of the off/on-policy token ratio within single sam￾ples by ablating the truncation parameter ratio, with all other hyperpar… view at source ↗
Figure 5
Figure 5. Figure 5: The relationship between entropy, clipping ratio, and relative position ratio (tokens are binned into 10% [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison between single-model inference and SOUP inference using the pass@ [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 5 linked inside Pith

  1. [2]

    Michael Noukhovitch, Shengyi Huang, Sophie Xhon- neux, Arian Hosseini, Rishabh Agarwal, and Aaron C

    Revisiting group relative policy optimization: Insights into on-policy and off-policy training.CoRR, abs/2505.22257. Michael Noukhovitch, Shengyi Huang, Sophie Xhon- neux, Arian Hosseini, Rishabh Agarwal, and Aaron C. Courville. 2025. Asynchronous RLHF: faster and more efficient off-policy RL for language models. InThe Thirteenth International Conference ...

  2. [5]

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang

    DAPO: an open-source LLM reinforcement learning system at scale.CoRR, abs/2503.14476. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize rea- soning capacity in llms beyond the base model? CoRR, abs/2504.13837. Feng Zhang, Zezhong Tan, Xinhong Ma, Ziqiang Dong,...

  3. [1063]

    Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney

    The MIT Press. Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. 2024. Understand- ing the performance gap between online and offline alignment algorithms.CoRR, abs/2405.08448. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shix- uan Liu...

  4. [2022]

    Training language models to follow instruc- tions with human feedback. InAdvances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. 2025. Demystifying reas...

  5. [2025]

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D

    Areal: A large-scale asynchronous reinforce- ment learning system for language reasoning.CoRR, abs/2505.24298. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. 2025. Cog- nitive behaviors that enable self-improving reason- ers, or, four habits of highly effective stars.CoRR, abs/2503.01307. Yaru Hao, Li Dong, Xun Wu, Sh...