Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Selective Importance Sampling turns off-policy tokens on-policy so LLM RL can reuse stale rollouts without exploding importance ratios.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 14:28 UTC pith:MEVPW3SF

load-bearing objection Practical plug-in that often works; theory is elementary and the top-K certificate is only approximate, but the empirics and stress tests are the real reason to care. the 3 major comments →

arxiv 2607.04728 v1 pith:MEVPW3SF submitted 2026-07-06 cs.CL cs.AIcs.LG

Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

Yu Li , Xiuyu Li , Mingyang Yi , Jiaxing Wang , zhangliangxu , Zhaolong Xing , Zhen Chen This is my paper
classification cs.CL cs.AIcs.LG
keywords selective importance samplingoff-policy RLLLM alignmentrejection samplingtoken-level importance ratiosGRPOpolicy gradientmixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Post-training large language models with reinforcement learning almost always updates on data rolled out from an older policy, so the training distribution is off-policy. Standard importance sampling corrects the mismatch, but token-level ratios multiply across long sequences and the variance explodes. This paper proposes Selective Importance Sampling (SIS): treat the behavior model as a proposal, run a cheap token-level accept/reject test, give accepted tokens unit weight as if they were on-policy, and leave rejected tokens with the usual ratio. The authors prove that this strictly shrinks the log-importance gap that bounds how far the practical token-level gradient sits from the true sequence-level gradient, and they show the method plugs into common algorithms with almost no extra compute. Across dense and mixture-of-experts models on math and agent tasks, SIS raises accuracy and keeps training stable even when rollouts are reused for many updates or when train and inference routing disagree.

Core claim

The paper claims that a token-level rejection test, with accepted tokens reweighted to 1 and rejected tokens kept under ordinary importance sampling, reduces the cumulative log-importance deviation that controls the error between sequence-level and token-level off-policy policy-gradient estimators, and that this single change is enough to improve multiple RL post-training objectives on real LLM alignment benchmarks while remaining a drop-in ratio swap.

What carries the argument

Selective Importance Sampling (SIS): at each token, the behavior policy is treated as a proposal; an acceptance probability equal to the token importance ratio divided by a top-K envelope of that ratio certifies the token as on-policy (weight 1) or leaves it off-policy (standard ratio). The resulting modified ratios are substituted into any existing policy loss.

Load-bearing premise

The practical top-K envelope is close enough to exact rejection sampling that accepted tokens can safely be treated as fully on-policy, which requires that almost all of the target policy’s probability mass sits inside those few top tokens.

What would settle it

On the same math or agent runs, measure residual mass outside the top-K set and retrain with a deliberately large residual (or with exact full-vocabulary Mt when feasible); if accuracy and the claimed reduction in log-importance deviation both vanish once residual mass is no longer tiny, the unit-weight certificate fails for the implemented estimator.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any off-policy token-level RL objective (GRPO, DAPO, GSPO and related variants) can replace its importance ratio with the SIS ratio and expect higher average accuracy with no change to the rest of the loss.
  • Reusing the same rollouts for many mini-batch updates becomes less harmful because a large share of tokens continue to receive unit weight even as the policy drifts.
  • Mixture-of-experts train–inference routing mismatch can be mitigated by SIS alone or further by stacking SIS with routing-replay methods.
  • Clipping can be removed or relaxed while still keeping training stable, because SIS itself suppresses the most severe off-policy ratios.
  • The theoretical bound on sequence-versus-token gradient error tightens whenever the accept rate is positive, giving a direct diagnostic (log-importance deviation) for off-policy health.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If accept rates stay high under longer tool-use trajectories, SIS could become a default stabilizer for multi-turn agent RL without redesigning the reward or the rollout system.
  • The same accept/reject idea might be applied to other multiplicative importance corrections (e.g., preference or distillation losses) wherever long products of ratios appear.
  • Token-level accept/reject clouds suggest SIS may act as an automatic filter that prefers reasoning tokens over formatting noise; that filtering effect could be measured as an auxiliary curriculum signal.
  • Because overhead is only about one percent of step time, SIS is a candidate default in production RL stacks that already cache old and new logits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Selective Importance Sampling (SIS), a plug-in for off-policy RL post-training of LLMs. Viewing the behavior policy as a proposal, SIS runs a token-level acceptance–rejection test: accepted tokens are treated as on-policy and receive unit importance weight, while rejected tokens keep the usual IS ratio. An efficient top-K envelope approximates the rejection constant. The authors prove that the pathwise total log-importance deviation D that bounds the sequence-level vs. token-level gradient gap is reduced (Thm. 1, Prop. 3), and show consistent accuracy and stability gains when SIS is plugged into GRPO, DAPO, and GSPO on dense and MoE Qwen3 models (and Llama-3.2) across math and agentic-search benchmarks, including under rollout staleness and MoE train–inference mismatch.

Significance. If the claims hold, SIS is a useful, low-overhead, algorithm-agnostic stabilization tool for the dominant “rollout then update” LLM RL stack. Strengths include: (i) a clean classical derivation from rejection sampling and IS factorization rather than ad-hoc clipping; (ii) a plug-in interface that only rewrites the token ratio and composes with existing objectives; (iii) elementary but relevant theory linking reduced D to a tighter sequence–token gap; (iv) broad empirical coverage (dense/MoE, math/agent, multiple base algorithms, staleness and MoE routing stress tests) with reported ~1% wall-clock overhead. The work is of practical interest to the RLHF/post-training community even if the exact on-policy certificate is only approximate under the implemented top-K estimator.

major comments (3)
  1. [§3.1–3.2, Prop. 1–2, Alg. 1] §3.1–3.2, Prop. 1 vs. Prop. 2 / Alg. 1: The central interpretive claim that accepted tokens “are already on-policy” and that unit weight “introduces zero bias” is exact only under Prop. 1 (full-vocabulary Mt). The implemented estimator uses the top-K envelope of Prop. 2, under which accepted tokens follow the conditional πθ(· | v ∈ VK) with D_TV = ξ_K exactly. Unit weight is therefore unbiased only for that conditional, not for πθ. The paper shows ξ_K is small in a few snapshots (Fig. 2B) but does not propagate residual mass into a bias bound for the token-level gradient (6) or for long-horizon / high-drift regimes that motivate SIS. Please either (a) extend Thm. 1 / Prop. 3 with an explicit O(ξ_K) (or similar) remainder for Alg. 1, or (b) restate all “zero bias / on-policy certificate” claims as holding only for the exact procedure and quantify approximation error for the implemented me
  2. [§3.1, Eqs. (8)–(10); §3.3, Prop. 3] §3.1, Eq. (8)–(10) and the claim after Prop. 1 that plugging ewt into (6) “results in identical gradient estimation”: even under exact rejection, E[ewt | yt] = (wt/Mt)·1 + (1−wt/Mt)·wt ≠ wt in general. The estimator is therefore not the same as standard token-level IS in expectation; it is a stochastic mixture of unit-weight and IS terms. The manuscript should state precisely what quantity is unbiased (or approximately unbiased): the true policy gradient ∇J, the sequence-level IS gradient, or only the token-level surrogate under a conditional law. Without this, the link from “D_SIS ≤ D” (Prop. 3) to improved approximation of ∇J remains incomplete, because Prop. 3 only tightens the sequence–token gap for a fixed realization of ratios, not the bias of the SIS mixture relative to on-policy sampling.
  3. [§4.2–4.3, Tables 1–2, Fig. 3–4] §4.2–4.3 and Tables 1–2 / Fig. 3–4: Empirical gains are consistent and the staleness/MoE stress tests are valuable, but several load-bearing comparisons lack uncertainty quantification (no multi-seed means/std, no statistical tests). For claims of “consistent improvements” and “substantially stronger robustness,” especially the large MoE math lifts (e.g., GRPO +6.37 Avg) and the collapse-vs-recovery narrative in Fig. 4, please report at least 2–3 seeds or error bars, and clarify whether evaluation (Avg@1 / Avg@32) uses a fixed decoding seed. Without this, it is hard to separate method effect from run-to-run variance in LLM RL.
minor comments (6)
  1. [Abstract, §1] Abstract and §1: several grammatical issues (“a efficient paradigm”, “can be combine with a vast vary of”, “while the token-level ratios compound… variance exploded”). A careful language pass would help.
  2. [Fig. 2] Fig. 2 caption and panels: “Accpet Rate” typo; panel labels A/B/C are useful but the residual-mass plot should state the model and step at which ξ_K is measured.
  3. [Table 4, §4.4] Table 4: SIS top-K differs by algorithm (10/30/50); §4.4 claims robustness to K but main results use different K per baseline. State the selection rule or unify K for fairer plug-in comparison.
  4. [Eq. (15), App. E.2] Eq. (15) and Appendix E.2: GSPO+SIS sequence ratio (40) uses a stop-gradient on πθ for accepted tokens; a one-sentence justification would clarify why that form preserves the intended sequence-level semantics.
  5. [§5] Related work §5: briefly contrast SIS with per-token masking / IcePop-style filters, since those also zero or downweight off-policy tokens rather than converting them.
  6. [§3, App. A.3] Notation: w(θ), wt(θ), ewt(θ), and wi,t(θ) are clear, but Mt vs. cMt and z_t vs. A in the appendix proof of Prop. 2 should be unified.

Circularity Check

0 steps flagged

No significant circularity: SIS derivation is classical rejection sampling plus an elementary bound reduction; empirical claims are external-benchmark measurements.

full rationale

The load-bearing theoretical chain is self-contained and non-circular. Proposition 1 is the standard acceptance-rejection identity (Robert & Casella) applied token-wise: P(y_t = v | z_t = 1) = π_θ(v | ·) follows directly from Bayes and the definition of M_t, without any fitted quantity or self-referential definition. Theorem 1 bounds the sequence-versus-token gap by the total log-importance deviation D = Σ |log w_t|; the short proof (Appendix A.2) uses only the elementary inequality |e^x − 1| ≤ e^|x| − 1 and the product factorization of the sequence ratio. Proposition 3 then observes that the SIS ratios e_w_t replace some of those |log w_t| by zero, so D_SIS ≤ D pathwise and the same bound tightens; this is an immediate algebraic consequence of the definition (10), not a prediction that re-uses a fitted constant. The top-K envelope (Proposition 2) introduces an explicit total-variation residual ξ_K that the paper acknowledges and measures (Fig. 2B); it does not redefine success in terms of that residual. All empirical claims (Tables 1–3, Figs. 3–4) are accuracy numbers on external held-out suites (MATH500, AIME, NQ, HotpotQA, \ldots) under fixed hyper-parameters shared with the baselines; nothing is fitted on the evaluation metrics and then reported as a “prediction.” Self-citations are ordinary background references to GRPO/DAPO/GSPO and related RLHF work; none of them supplies a uniqueness theorem or ansatz that is load-bearing for the SIS construction. Consequently the derivation does not reduce to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The load-bearing math is classical rejection sampling plus a product bound on importance ratios; the practical claim further depends on a hand-chosen top-K envelope and on the usual LLM-RL modeling choices (token-level surrogate gradient, group advantages, clipping from base algorithms). No new physical entities are postulated; free parameters are algorithmic knobs (especially K) rather than fitted physical constants.

free parameters (3)
  • top-K envelope size K
    Defines the rejection envelope VK; chosen by hand (often 10–50) and varied by algorithm/setting in Table 4. Ablations claim robustness, but the implemented “on-policy” certificate depends on this choice via residual mass ξK.
  • clipping bounds ε / ε_low / ε_high (from base methods)
    When SIS is composed with GRPO/DAPO/GSPO, rejected tokens still use clipped ratios; those thresholds are inherited hyperparameters that affect the reported gains of SIS+baseline combinations.
  • staleness N (mini-batch updates per rollout)
    Controls off-policy stress in stability experiments; not fitted to invent the method, but the robustness claim is demonstrated at chosen N∈{4,8,16}.
axioms (4)
  • standard math Standard acceptance-rejection sampling: with M ≥ max π/q, accepting y~q with probability (π(y)/q(y))/M yields samples from π.
    Used for Proposition 1 and the unit-weight rule for accepted tokens (§3.1, Appendix A.1).
  • domain assumption Token-level importance-weighted sum is a usable surrogate for the sequence-level off-policy policy gradient when behavior and target policies are not too far apart.
    Background §2 cites Kakade & Langford / recent LLM-RL practice; SIS theory then bounds the token–sequence gap rather than proving unbiasedness of the true ∇J.
  • domain assumption LLM next-token mass is concentrated on a small top-K set under the behavior policy, so ξK is small enough for the top-K envelope to be a faithful practical certificate.
    Justifies Algorithm 1 and Proposition 2; supported empirically in Fig. 2B but not proved for general policies.
  • domain assumption Outcome-only rule rewards (boxed exact/symbolic match for math; EM for agent) are adequate training signals for the reported benchmarks.
    Reward design in Appendix C.2; standard for these tasks but shapes what “improvement” means.
invented entities (1)
  • Selective Importance Sampling (SIS) modified ratio ewt independent evidence
    purpose: Binary accept/reject rewrite of the token IS coefficient so accepted tokens get weight 1 and rejected tokens keep wt.
    Core construction of the paper (§3.1, Eq. 10). It is an algorithmic object, not an unobserved physical mediator; independent evidence is the classical rejection-sampling certificate plus empirical plug-in gains.

pith-pipeline@v1.1.0-grok45 · 28421 in / 3311 out tokens · 34717 ms · 2026-07-11T14:28:44.667287+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is "transferring" these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.

Figures

Figures reproduced from arXiv: 2607.04728 by Jiaxing Wang, Mingyang Yi, Xiuyu Li, Yu Li, zhangliangxu, Zhaolong Xing, Zhen Chen.

Figure 1
Figure 1. Figure 1: Overview of SIS. At each token position, SIS applies a rejection sampling to determine [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Empirical validation of the three components of SIS on Qwen3-8B-Base. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics under increasing policy staleness ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics on Qwen3-30B-A3B-Base (MoE) with DAPO. Columns: AIME24/25 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Token-level SIS decisions. Word clouds of distinctive tokens in accepted(left, green) vs. rejected(right, red) samples. SIS fa￾vors mathematical reasoning tokens and rejects formatting/web artifacts. 5 RELATED WORK Importance Sampling Ratios for LLM RL. Importance-sampling correction for off-policy train￾ing dates back to Schulman et al. (2015); Sutton et al. (1998), but the per-token ratio introduces larg… view at source ↗
Figure 6
Figure 6. Figure 6: Accept rate of SIS on Qwen3-30B-A3B-Base under math reasoning (left) and Qwen3- [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-step wall-clock decomposition of GRPO+SIS on Qwen3-8B-Base across [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Training dynamics of SIS over the first 400 gradient steps for all entries in [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Token-level accept rate of SIS during the first [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 18 linked inside Pith

  1. [1]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. Preprint arXiv:2506.13585,

    Pith/arXiv arXiv

  2. [2]

    The entropy mechanism of reinforcement learning for reasoning language models

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. Preprint arXiv:2505.22617,

    Pith/arXiv arXiv

  3. [3]

    Soft adaptive policy optimization

    Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimization. Preprint arXiv:2511.20347, 2025a. Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scal...

    Pith/arXiv arXiv

  4. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  5. [5]

    Xue Jiang, Yihong Dong, Mengyang Liu, Hongyi Deng, Tian Wang, Yongding Tao, Rongyu Cao, Binhua Li, Zhi Jin, Wenpin Jiao, et al

    10 Preprint. Xue Jiang, Yihong Dong, Mengyang Liu, Hongyi Deng, Tian Wang, Yongding Tao, Rongyu Cao, Binhua Li, Zhi Jin, Wenpin Jiao, et al. Coderl+: Improving code generation via reinforcement with execution semantics alignment. Preprint arXiv:2510.18471,

    Pith/arXiv arXiv

  6. [6]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. Preprint arXiv:2503.09516,

    Pith/arXiv arXiv

  7. [7]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. InPro- ceedings of the 2020 conference on empirical methods in natural language processing (EMNLP),

    2020

  8. [8]

    A step back: Prefix importance ratio stabilizes policy optimization

    Shiye Lei, Zhihao Cheng, and Dacheng Tao. A step back: Prefix importance ratio stabilizes policy optimization. Preprint arXiv:2601.22718,

    arXiv

  9. [9]

    Ets: Energy- guided test-time scaling for training-free rl alignment

    Xiuyu Li, Jinkai Zhang, Mingyang Yi, Yu Li, Longqiang Wang, Yue Wang, and Ju Fan. Ets: Energy- guided test-time scaling for training-free rl alignment. InForty-third International Conference on Machine Learning, 2026a. Yingru Li, Jiacai Liu, Jiawei Xu, Yuxuan Tong, Ziniu Li, Qian Liu, and Baoxiang Wang. Trust region masking for long-horizon llm reinforcem...

    Pith/arXiv arXiv

  10. [10]

    Reasoning and tool-use compete in agentic rl: From quantifying interference to disentan- gled tuning

    Yu Li, Mingyang Yi, Xiuyu Li, Ju Fan, Fuxin Jiang, Binbin Chen, Peng Li, Jie Song, and Tieying Zhang. Reasoning and tool-use compete in agentic rl: From quantifying interference to disentan- gled tuning. Preprint arXiv:arXiv:2602.00994, 2026b. Zhaochun Li, Mingyang Yi, Yue Wang, Shisheng Cui, and Yong Liu. Towards a theoretical under- standing to the gene...

    Pith/arXiv arXiv

  11. [11]

    Sparse-rl: Breaking the memory wall in llm reinforcement learning via stable sparse rollouts

    Sijia Luo, Xiaokang Zhang, Yuxuan Hu, Bohan Zhang, Ke Wang, Jinbo Su, Mengshu Sun, Lei Liang, and Jing Zhang. Sparse-rl: Breaking the memory wall in llm reinforcement learning via stable sparse rollouts. Preprint arXiv:2601.10079,

    arXiv

  12. [12]

    Stabilizing moe reinforcement learning by aligning training and inference routers

    Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. Stabilizing moe reinforcement learning by aligning training and inference routers. Preprint arXiv:2510.11370,

    arXiv

  13. [13]

    When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. Preprint arXiv:2212.10511,

    Pith/arXiv arXiv

  14. [14]

    Measuring and narrowing the compositionality gap in language models

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023,

    2023

  15. [15]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  16. [16]

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. Preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  17. [17]

    Maximum likelihood reinforcement learning

    Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette. Maximum likelihood reinforcement learning. Preprint arXiv:2602.02710,

    arXiv

  18. [18]

    Every step evolves: Scaling reinforcement learning for trillion-scale thinking model

    Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, et al. Every step evolves: Scaling reinforcement learning for trillion-scale thinking model. Preprint arXiv:arXiv:2510.18855,

    arXiv

  19. [19]

    Ucob: Learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation.arXiv:2606.29502,

    Songjun Tu, Chengdong Xu, Qichao Zhang, Yiwen Ma, Yaocheng Zhang, Linjing Li, Dong Li, Xiangyuan Lan, and Dongbin Zhao. Ucob: Learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation.arXiv:2606.29502,

    Pith/arXiv arXiv

  20. [20]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. Preprint arXiv:2212.03533,

    Pith/arXiv arXiv

  21. [21]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. (arXiv:2409.12122),

    Pith/arXiv arXiv

  22. [22]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. Preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  23. [23]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Preprint arXiv:2503.14476,

    Pith/arXiv arXiv

  24. [24]

    Htam: Hierarchical transition-attended memory for operator optimization.arXiv preprint arXiv:2605.29734, 2026a

    Yining Zhang, Mingyang Yi, Chen Wang, Xuwen Xiang, Tianhe Jia, Zedong Dan, Chengqing Zong, and Yue Wang. Htam: Hierarchical transition-attended memory for operator optimization.arXiv preprint arXiv:2605.29734, 2026a. Yuxin Zhang, Meihao Fan, Ju Fan, Mingyang Yi, Yuyu Luo, Guoliang Li, Bin Wu, and Wenchao Zhou. Reward-sql: Boosting text-to-sql via stepwise...

    Pith/arXiv arXiv

  25. [25]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al

    URLhttps://ringtech.notion.site/icepop. Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. Preprint arXiv:2304.11277,

    Pith/arXiv arXiv

  26. [26]

    Stabilizing reinforcement learning with llms: For- mulation and practices

    Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, An Yang, Jingren Zhou, and Junyang Lin. Stabilizing reinforcement learning with llms: For- mulation and practices. Preprint arXiv:2512.01374, 2025a. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al...

    arXiv

  27. [27]

    For each token position, the algorithm first identifies the top-Kmost probable tokens under the behavior policy and restricts the rejection envelope to this set. Tokens falling insideV K undergo an acceptance test: accepted tokens are certified as on-policy and receive unit importance weight, while rejected tokens and those outsideV K retain their origina...

    2025

  28. [28]

    and the 2018 Wikipedia dump (Karpukhin et al.,

    2018

  29. [29]

    Table 4: Hyperparameters used in math experi- ments

    as the corpus. Table 4: Hyperparameters used in math experi- ments. Hyperparameter GRPO DAPO GSPO Learning rate 1×10−6 Mini-batch size (mbs) 256 Responses per prompt 8 Max prompt length 1,024 Max response length 3,072 Gradient steps 1,600 Train temperature 1.0 Eval temperature / top-p 1.0 / 0.7 Entropy loss coefficient 0 Training engine FSDP Inference eng...

    2025

  30. [30]

    D ADDITIONALEXPERIMENTALRESULTS D.1 SIS ENABLESSTABLETRAININGWITHOUTCLIPPING To test whether SIS alone suffices for training stability, we disable clipping entirely in GRPO+SIS on Qwen3-8B-Base. Table 7 shows thatSIS without clipping still outperforms the clip-based GRPO on both math and agent tasks, confirming that SIS is a stabilization mechanism in its...

    arXiv

  31. [31]

    Table 8: Performance of SIS under varying policy staleness on Qwen3-8B-Base, where stalenessN is the number of mini-batch updates per rollout

    20 Preprint. Table 8: Performance of SIS under varying policy staleness on Qwen3-8B-Base, where stalenessN is the number of mini-batch updates per rollout. Results are grouped by algorithm so that the effect of increasingNis directly comparable within each method. Shaded rows denote the application of SIS on top of each baseline. Within each algorithm–sta...

    arXiv 2024