Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

Haitao Mi; Junyao Yang; Kishan Panaganti; Leowei Liang; Ruhan Wang; Xiangxin Zhou; Yucheng Shi; Zhongzhi Li; Zongxia Li

REVIEW 3 major objections 6 minor 21 references

Adaptive shrinking of PPO's clip radius on stale tokens stabilizes asynchronous RL.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:32 UTC pith:A7VYGNGV

load-bearing objection A useful adaptive-clipping variant with a correct local-geometry proof, but the headline performance claim rests on one seed per cell and the margins are likely inside run-to-run noise. the 3 major comments →

arxiv 2607.18722 v3 pith:A7VYGNGV submitted 2026-07-21 cs.LG cs.CL

Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

Junyao Yang , Yucheng Shi , Zongxia Li , Zhongzhi Li , Ruhan Wang , Xiangxin Zhou , Kishan Panaganti , Haitao Mi

show 1 more author

Leowei Liang

This is my paper

classification cs.LG cs.CL

keywords asynchronous reinforcement learningPPO clippingstalenesstrust regionlog-ratio proxyLLM reasoningAIME24training stability

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that instability in asynchronous reinforcement learning comes from a heterogeneous, long-tailed mismatch between the rollout policy and the training policy, and that PPO-style clipping is too blunt to control it: a fixed clip radius either starves ordinary tokens or gives dangerously stale tokens too much freedom. The authors propose a Staleness-Adaptive Trust Region (SAT), which uses each token's detached log-ratio as a staleness signal, finds the high-mismatch tail of each batch, and shrinks only the outward side of the clip interval on those tokens. The paper proves that the adaptive interval stays inside PPO's interval, the surrogate is pointwise no larger than PPO's, and the two objectives differ only on a sign-selected outward band per gated token. Empirically, the best SAT configuration reaches AIME24 avg@8 of 35.83 at lag 1 and 34.79 at lag 8, beating fixed-clip baselines and avoiding the late training collapse seen in several lag-8 runs. A sympathetic reader would care because the result suggests that the clip radius itself—not just the data pipeline—can be made responsible for async stability.

Core claim

On the paper's own terms, the central claim is that PPO's sampled clip acts as a per-token gate that leaves high-staleness updates weakly controlled, and that the right fix is to make the gate's radius a function of the observed mismatch tail. SAT does this by replacing the fixed interval [1−ε_low, 1+ε_high] with [1−ε_low·c_minus, 1+ε_high·c_plus], where the contraction factors are 1 except on tokens whose detached |log r| exceeds the batch's 90th percentile, and where only the endpoint that would move the ratio farther from 1 is tightened. The paper proves the adaptive interval is contained in PPO's, the SAT surrogate is pointwise pessimistic relative to PPO, and the derivative difference i

What carries the argument

The central object is the adaptive clip interval, obtained by contracting one endpoint of PPO's nominal interval with a positive contraction factor. The factor is ψ(u;q)=1/(1+(u/q)^2) for u above the detached batch quantile q of |log r|, and 1 otherwise; the sign of log r selects which endpoint shrinks. This object carries the argument: it preserves baseline behavior on ordinary tokens and pull-back updates while cutting off outward updates exactly on the high-staleness tail, and Proposition 4.1 shows its effect is exactly one sign-selected outward band per gated token.

Load-bearing premise

The argument rests on the detached per-token log-ratio |log r| being a trustworthy sign of which tokens are dangerously stale—a proxy the paper itself says should not be identified with true divergence or version age, so if it points at the wrong tokens the whole contraction scheme loses its rationale.

What would settle it

Run the same grid with the gate inverted—contracting the radius on the bottom decile of |log r| instead of the top—and compare AIME24 and collapse timing. If the inverted version matches SAT's gains, the staleness signal is not what drives the improvement; if it collapses like fixed-clip baselines, the tail-selection mechanism is doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

If SAT is right, async RL can be stabilized at the optimizer level without changing the rollout/training decoupling that provides throughput.
Fixed clip radii are a wrong default under heterogeneous staleness: any constant choice either starves the bulk or over-permits the tail.
The best configuration's lower observed mismatch suggests adaptive contraction and routing replay complement each other: replay lowers the mismatch level, SAT changes where divergence gradients stop.
The containment and pointwise-pessimism results imply SAT can be dropped into any PPO-style objective and is bit-identical to the base when the gate is off.
Because the gate is batch-relative, the method self-calibrates to rescaling of the mismatch profile without a hand-tuned absolute threshold.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

One testable extension is to replace the detached log-ratio with a direct divergence estimate or with version age and compare gate decisions: the paper's own caveat in Section 6 implies the proxy is the least-grounded link.
The batch-relative quantile design suggests SAT should transfer to fully asynchronous queues where lag is random per token, an experiment the paper motivates but does not run.
The same sign-selected contraction idea could apply to other off-policy objectives, such as importance-weighted supervised fine-tuning, where staleness heterogeneity also appears.
A cheaper diagnostic of the mechanism than full benchmark runs: measure the realized gate rate and effective radii during training; SAT predicts tail suppression, which can be checked on existing logs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful adaptive-clipping variant with a correct local-geometry proof, but the headline performance claim rests on one seed per cell and the margins are likely inside run-to-run noise.

read the letter

Colleague, here's my read on arXiv:2607.18722. The paper proposes SAT, a drop-in replacement for PPO's clip that contracts one endpoint on the high-staleness tail of the batch, using the detached log-ratio as a proxy and a batch quantile to identify the tail. That's a small but real idea within the asymmetric-clipping family. What's new: the specific rule—sign-selected contraction with Hill-kernel scaling—isn't in the cited PPO/DPPO/GSPO work, and the formalization of realized mismatch (Eq. 2, separating policy lag from implementation effects) is genuinely useful. Prop 4.1 is correct: the adaptive interval is contained in PPO's, the objective is pointwise pessimistic, and the difference is confined to a sign-selected outward band. The Kakade-Langford framing restates known bounds, but the presentation is honest.

The soft spots are where the stress-test note lands, and the paper itself is transparent about them. Appendix F says explicitly: "Every configuration is a single training seed, so each cell is one run: the numbers are a snapshot rather than confidence-interval estimates." The headline numbers in Table 1—SAT-GSPO w/ R3 at 35.83 (lag 1) and 34.79 (lag 8)—come from one training run per cell. The margins over DPPO (1.87 and 2.08 points) and over GSPO (3.58 and 3.33) are small relative to typical LLM RL seed variance; Figure 4 shows jagged curves that could plausibly swap ranks with another seed. So the central performance claim is not yet supported. The proxy |log r| is also not DTV, and the paper acknowledges this in Section 6; I agree it's a legitimate design choice, but it means the method's success hinges on the proxy identifying the right tokens, which is untested. Missing ablations of alpha and the kernel exponent are minor by comparison.

On the credit side: the experimental setup controls data, reward, and model across all configurations, and the lag-8 collapse in the vanilla baselines is a real phenomenon worth explaining. The paper ships no code or data, which makes the single-seed issue harder to resolve externally.

Net: this is a serious, readable paper that deserves referee time, but the empirical contribution requires multi-seed runs with confidence intervals (or at minimum shipped code) before the performance claim should be believed. I'd send it to peer review, and I'd bring it to a reading group on async RL, but I wouldn't cite the empirical result until it's reproduced. The theoretical contribution, modest as it is, stands on its own.

Referee Report

3 major / 6 minor

Summary. The paper formalizes a batch-wise asynchronous RL setting in which the configured pipeline lag n is only a coarse label for the realized per-token mismatch d_{b,t}=log r_{b,t}, which mixes policy-update lag with implementation mismatch from kernels, numerics, and MoE routing. It argues that PPO clipping is a sampled surrogate rather than a full-policy trust region, and proposes SAT: using the detached |log r| as a staleness proxy, identifying a per-batch high-mismatch tail through a quantile q, and contracting only the sign-selected endpoint of PPO's clip interval on that tail via a Hill kernel. Proposition 4.1 proves that the adaptive interval is contained in PPO's interval and the surrogate is pointwise pessimistic. The experiments use Qwen3-30B-A3B-Base with SGLang/Megatron at configured lags 1 and 8, comparing GRPO/GSPO with R3, TIS, DPPO, and SAT variants; the headline result is SAT-GSPO w/ R3 achieving 35.83 at lag 1 and 34.79 at lag 8 on AIME24, with lower reported d_pi than plain baselines.

Significance. If the empirical results are reproducible, the paper would make a useful and honest contribution: it identifies a cheap, drop-in modification to PPO-style clipping whose local geometry is characterized exactly, and it is careful to separate the configured lag from the realized sampled mismatch. The controlled setup (shared optimizer, data, reward, model), the full-vocabulary sampling required by Lemma 3.1, and the explicit statement of limitations are strengths. The theoretical result, however, is local and descriptive: Proposition 4.1 establishes interval containment and pointwise pessimism, not an improvement or stability guarantee. The central claim that SAT stabilizes asynchronous RL therefore rests almost entirely on the empirical comparison, which currently consists of a single training run per configuration. The paper's own Appendix F concedes this. Until multi-seed evidence and sensitivity analysis are provided, the significance of the performance claim remains conditional.

major comments (3)

[Appendix F; Table 1] The central performance claim is supported by exactly one training run per configuration. Appendix F states: 'Every configuration is a single training seed, so each cell is one run: the numbers are a snapshot rather than confidence-interval estimates.' Table 1 reports best logged checkpoints, and the headline margins over DPPO are 1.87 points at lag 1 and 2.08 points at lag 8. For LLM RL, these margins are within typical run-to-run variance, and Figure 4 shows jagged trajectories with collapses at specific steps. Since Proposition 4.1 guarantees only local surrogate geometry and not stability or improvement, the empirical comparison carries the entire load. I would need repeated independent seeds with confidence intervals or seed-averaged curves before the claim 'SAT achieves better performance' is supported.
[Section 5.1; Eqs. (14)-(16)] SAT's behavior depends on two free constants: the reference quantile alpha=0.90 and the Hill kernel exponent 2. No sensitivity or ablation study is reported for either. Given that the method is advertised as adaptive, the absence of any analysis of these choices is a load-bearing gap: with a single seed, one cannot tell whether the reported advantage reflects the mechanism or a favorable hyperparameter/seed configuration. At minimum, a small grid over alpha and the kernel exponent, even at one lag, would help establish that the adaptive contraction itself is what matters.
[Prop. 3.2 vs. Eq. (18); Section 6] The theoretical motivation (Prop. 3.2) controls D_TV under a hard all-action ratio envelope, but SAT contracts only one endpoint of a sampled interval based on |log r|. The paper explicitly acknowledges the gap in Section 6: the proxy 'should not be identified with either D_TV or version age' and 'unobserved vocabulary actions remain unconstrained.' This is not an internal inconsistency, but it means the theory is motivational rather than directly load-bearing for the stability claim. To strengthen the paper, the authors should provide at least a diagnostic or ablation showing that the gated tokens are those that actually contribute to the divergence penalty, rather than merely those with large |log r|.

minor comments (6)

[General] There are several typos and inconsistencies: 'scenerio' in Section 1; the duplicated 'Propositions 4.1 and 4.1' in Section 5.4; and the title contains an extra space in 'T rust Regions.'
[Abstract / Table 1 / Appendix F] The metric is called 'AIME24 avg@8' in the abstract and Table 1, but Appendix F and Appendix E tables refer to 'AIME24 pass@1'. Please use one consistent name and define precisely what is being reported.
[Section 5.1 vs. Appendix F] Section 5.1 says sampling uses temperature 0.95, while Appendix F says temperature 1.0; both state top-p 1.0 and top-k -1. Please resolve the discrepancy.
[Appendix F] The statement that 'each logged score averages four evaluation runs' should clarify that this averages evaluation sampling noise only, not training-seed variation. The following sentence already says each cell is one run, but the distinction should be explicit in the main text.
[Section 2.2] The text says 'two instability symptoms recur' but then lists three bullet observations. Please renumber or rephrase.
[Figure 2 and 3] The figures refer to 'a variance method' without naming it; since later experiments use GRPO/GSPO, please identify the method in the figure captions or text.

Circularity Check

0 steps flagged

No load-bearing circularity; SAT is defined from log-ratios, not from the benchmark, and its propositions are analytic.

full rationale

The derivation chain is self-contained. SAT's contraction factors (Eq. 16) are computed from the detached per-token log-ratio via a batch quantile (Eq. 14) and a monotone kernel (Eq. 15); no AIME score or baseline performance enters the method definition. Proposition 4.1 is an analytic statement about the interval geometry of the constructed surrogate (containment, pointwise pessimism, locus of derivative changes), and its proof follows directly from Eqs. (16)-(19); it is not an empirical prediction. The paper explicitly disclaims that |log r| is a DTV or version-age estimate (Section 6), so the leap from the all-action DTV bound (Prop. 3.2) to the sampled proxy is acknowledged as a design choice rather than smuggled as a theorem. The AIME24 comparison is against external baselines on an external benchmark, not against values fitted from the method's own inputs. The only self-referential flavor is that the same |log r| signal both triggers clipping (Eq. 13) and is reported as the 'observed staleness' diagnostic (Eq. 52); this is a validation weakness, not a definitional equivalence, because SAT does not directly optimize d_bar_pi and the AIME headline is independent. Minor self-citations (DPPO [8], DRPO [14] include co-author Xiangxin Zhou) are used as baselines/related work and are not load-bearing. The single-seed limitation in Appendix F is a statistical-evidence concern, not a circularity. Consequently no circular step meets the evidentiary bar; score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

SAT introduces no new physical or model-level entities. Its load-bearing free parameters are the quantile threshold alpha, the kernel exponent, and the base PPO radii. The most fragile axioms are the ad hoc identification of |log r| as a staleness proxy and the top-decile tail, both acknowledged by the authors in Section 6.

free parameters (3)

Reference quantile alpha = 0.90
Controls the fraction of tokens treated as the high-staleness tail in Eq. (14); set without sensitivity analysis in Section 5.1 and Appendix F.
Hill kernel exponent = 2
Sets the contraction shape in psi(u;q)=1/(1+(u/q)^2); chosen without ablation, central to how strongly the radius shrinks.
Base clip radii epsilon_low/epsilon_high = 0.2 / 0.2
Inherited from the PPO baseline; SAT contracts these radii, so the method's effect depends on the base interval.

axioms (5)

standard math Finite-horizon improvement bound with |R(y)| <= xi and support compatibility pi << mu
Invoked in Section 3.1 (Eq. 4) and proved in Appendix D; used to argue that D_TV controls the approximation error.
domain assumption Support compatibility pi(·|s) << mu(·|s)
Needed for Lemma 3.1; experiments set top-p=1.0 and top-k=-1 to retain full-vocabulary support (Section 5.1).
domain assumption Decomposition d = Delta_pi + Delta_impl
Eq. (2) assumes a trainer-side reference pi^(l) exists and that implementation mismatch can be separated; Appendix C notes this is not a unique additive decomposition.
ad hoc to paper Sampled |log r| is a practical staleness proxy
Adopted in Eq. (13) as the gate signal; Section 6 explicitly says it is an empirical design choice, not a D_TV estimate.
ad hoc to paper Top-decile of |log r| identifies the high-mismatch tail
The quantile gate in Eqs. (14)-(16) assumes the 90th percentile is the right threshold; no independent validation or sensitivity analysis is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 21932 in / 12888 out tokens · 118327 ms · 2026-08-01T14:32:20.751321+00:00 · methodology

0 comments

read the original abstract

Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but the resulting staleness is an inevitable byproduct, compounded jointly by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch is critical: in the finite-horizon improvement bound, training-inference divergence governs the approximation error, whereas PPO clipping only gates sampled outward updates and therefore acts as a sampled surrogate rather than a full-policy constraint. As a result, the high-staleness update can remain weakly controlled in exactly the asynchronous regime where stale rollouts matter most. We introduce the Staleness-Adaptive Trust Region (SAT), which uses the detached sampled log-ratio as a practical staleness proxy, identifies the high-mismatch tail within each batch through Staleness-based kernel function scaling, and contracts only the sign-selected endpoint of the nominal PPO interval using Effective contraction factors. This design preserves the baseline behavior on ordinary tokens, while making the update more conservative exactly on newly intercepted outward bands. We evaluate SAT in a fully decoupled asynchronous reinforcement learning setup built on Qwen3-30B-A3B-Base, leveraging SGLang as the inference engine and Megatron as the training pipeline. In this setting, SAT-GSPO w/ R3 attains the best observed AIME24 avg@8, reaching 35.83 at lag 1 and 34.79 at lag 8, while SAT-GSPO reaches 34.17 at lag 1. More broadly, the results indicate that aligning the clip interval with observed staleness heterogeneity is an effective way to stabilize the reported asynchronous regime.

discussion (0)

Reference graph

Works this paper leans on

21 extracted references · 10 linked inside Pith

[1]

Staleness in fully asynchronous RL.https://appliedcompute

Chengyu Dong, Agastya Goel, and Dylan Yu. Staleness in fully asynchronous RL.https://appliedcompute. com/research/staleness-in-fully-async-rl, 2026. Applied Compute Research

2026
[2]

Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Sh...

Pith/arXiv arXiv 2026
[3]

Kpop: Taming training–inference mismatch in reinforcement learning with adaptive masking regions, May 2026

Jia Guo, Yan Sun, Zhenyu Huang, Zihao Wang, Zujie Wen, Zhiqiang Zhang, Jun Zhou, and Stanley Kok. Kpop: Taming training–inference mismatch in reinforcement learning with adaptive masking regions, May 2026. URL https://ringtech.notion.site/kpop

2026
[4]

Single-rollout asynchronous optimization for agentic reinforcement learning, 2026

Zhenyu Hou, Yujiang Li, Jie Tang, and Yuxiao Dong. Single-rollout asynchronous optimization for agentic reinforcement learning, 2026. URLhttps://arxiv.org/abs/2607.07508

Pith/arXiv arXiv 2026
[5]

Stabilizing rlvr via token-level gradient diagnosis and layerwise clipping, 2026

Guanhua Huang, Tingqiang Xu, Jinbo Wang, Guangming Sheng, Siheng Li, Evander Yang, Kejiao Li, Yunxiang Li, Zenan Xu, Qi Yi, Xue Gong, Ziyuan Nan, Yuhao Jiang, Chenchen Zhang, Taiqiang Wu, Feiyuan Zhang, Junhao Wang, Bo Zhou, Alex Chen, Di Wang, and Shunyu Yao. Stabilizing rlvr via token-level gradient diagnosis and layerwise clipping, 2026. URLhttps://hy....

2026
[6]

Approximately optimal approximate reinforcement learning

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. InProceedings of the 19th International Conference on Machine Learning (ICML), 2002

2002
[7]

Stabilizing MoE reinforcement learning by aligning training and inference routers.arXiv preprint arXiv:2510.11370, 2025

Wenhan Ma, Hailin Zhang, Liang Zhao, Yujiao Song, Yiyuan Wang, Zhifang Sui, and Fuli Luo. Stabilizing MoE reinforcement learning by aligning training and inference routers.arXiv preprint arXiv:2510.11370, 2025

arXiv 2025
[8]

Rethinking the trust region in LLM reinforcement learning

Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in LLM reinforcement learning. InForty-third International Conference on Machine Learning, 2026. URLhttps://openreview.net/forum?id=f4g8zblbwp

2026
[9]

Jordan, and Pieter Abbeel

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimiza- tion. InProceedings of the 32nd International Conference on Machine Learning (ICML), 2015. arXiv:1502.05477

Pith/arXiv arXiv 2015
[10]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

Pith/arXiv arXiv 2017
[11]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URLhttps://arxiv.org/abs/2402.03300

Pith/arXiv arXiv 2024
[12]

Megatron-LM: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019. 12

Pith/arXiv arXiv 1909
[13]

Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

Pith/arXiv arXiv 2025
[14]

Rethinking the divergence regularization in llm rl.arXiv preprint arXiv:2606.09821, 2026

Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, and Tianyu Pang. Rethinking the divergence regularization in llm rl.arXiv preprint arXiv:2606.09821, 2026

Pith/arXiv arXiv 2026
[15]

DAPO: An open-source LLM reinforcement learning system at scale

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao ...

2025
[16]

Small leak can sink a great ship–boost rl training on moe with icepop!, Sep 2025

Xin Zhao, Yongkang Liu, Kuan Xu, Jia Guo, Zihao Wang, Yan Sun, Xinyu Kong, Qianggang Cao, Liang Jiang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. Small leak can sink a great ship–boost rl training on moe with icepop!, Sep 2025. URLhttps://ringtech.notion.site/icepop

2025
[17]

Stabilizing reinforcement learning with llms: Formulation and practices, 2025

Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, and Junyang Lin. Stabilizing reinforcement learning with llms: Formulation and practices, 2025. URLhttps://arxiv.org/abs/2512.01374

arXiv 2025
[18]

Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

Pith/arXiv arXiv 2025
[19]

Gonzalez, Clark Barrett, and Ying Sheng

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs.arXiv preprint arXiv:2312.07104, 2023

Pith/arXiv arXiv 2023
[20]

X yt µ(yt |s t) π(yt |s t) µ(yt |s t) −1 # = 2ξ TX t=1 Est∼µ h 2DTV(µ(· |st)∥π(· |st)) i = 4ξE y∼µ

Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv. 13 A Notation Table 2Notation used throughout the paper. Symbol Meaning x, yb, sb,t Prompt; responsey b = (yb,1, . . . , yb,Tb ); states b,t = (x, yb,1, . . ....

2025
[21]

This is consistent with a sampledDTV gate improving stability on the reported stack, but it does not explain the remaining gap toSAT-GSPO w/ R3 or even toSAT-GRPO w/ R3

It stays above the GRPO baseline at both lags and outside the runs marked as later collapse. This is consistent with a sampledDTV gate improving stability on the reported stack, but it does not explain the remaining gap toSAT-GSPO w/ R3 or even toSAT-GRPO w/ R3. Table 7Reported best AIME24 pass@1 for the DPPO comparison. Method lag1lag8 GRPO31.25 30.17 ‡ ...

[1] [1]

Staleness in fully asynchronous RL.https://appliedcompute

Chengyu Dong, Agastya Goel, and Dylan Yu. Staleness in fully asynchronous RL.https://appliedcompute. com/research/staleness-in-fully-async-rl, 2026. Applied Compute Research

2026

[2] [2]

Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Sh...

Pith/arXiv arXiv 2026

[3] [3]

Kpop: Taming training–inference mismatch in reinforcement learning with adaptive masking regions, May 2026

Jia Guo, Yan Sun, Zhenyu Huang, Zihao Wang, Zujie Wen, Zhiqiang Zhang, Jun Zhou, and Stanley Kok. Kpop: Taming training–inference mismatch in reinforcement learning with adaptive masking regions, May 2026. URL https://ringtech.notion.site/kpop

2026

[4] [4]

Single-rollout asynchronous optimization for agentic reinforcement learning, 2026

Zhenyu Hou, Yujiang Li, Jie Tang, and Yuxiao Dong. Single-rollout asynchronous optimization for agentic reinforcement learning, 2026. URLhttps://arxiv.org/abs/2607.07508

Pith/arXiv arXiv 2026

[5] [5]

Stabilizing rlvr via token-level gradient diagnosis and layerwise clipping, 2026

Guanhua Huang, Tingqiang Xu, Jinbo Wang, Guangming Sheng, Siheng Li, Evander Yang, Kejiao Li, Yunxiang Li, Zenan Xu, Qi Yi, Xue Gong, Ziyuan Nan, Yuhao Jiang, Chenchen Zhang, Taiqiang Wu, Feiyuan Zhang, Junhao Wang, Bo Zhou, Alex Chen, Di Wang, and Shunyu Yao. Stabilizing rlvr via token-level gradient diagnosis and layerwise clipping, 2026. URLhttps://hy....

2026

[6] [6]

Approximately optimal approximate reinforcement learning

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. InProceedings of the 19th International Conference on Machine Learning (ICML), 2002

2002

[7] [7]

Stabilizing MoE reinforcement learning by aligning training and inference routers.arXiv preprint arXiv:2510.11370, 2025

Wenhan Ma, Hailin Zhang, Liang Zhao, Yujiao Song, Yiyuan Wang, Zhifang Sui, and Fuli Luo. Stabilizing MoE reinforcement learning by aligning training and inference routers.arXiv preprint arXiv:2510.11370, 2025

arXiv 2025

[8] [8]

Rethinking the trust region in LLM reinforcement learning

Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in LLM reinforcement learning. InForty-third International Conference on Machine Learning, 2026. URLhttps://openreview.net/forum?id=f4g8zblbwp

2026

[9] [9]

Jordan, and Pieter Abbeel

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimiza- tion. InProceedings of the 32nd International Conference on Machine Learning (ICML), 2015. arXiv:1502.05477

Pith/arXiv arXiv 2015

[10] [10]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

Pith/arXiv arXiv 2017

[11] [11]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URLhttps://arxiv.org/abs/2402.03300

Pith/arXiv arXiv 2024

[12] [12]

Megatron-LM: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019. 12

Pith/arXiv arXiv 1909

[13] [13]

Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

Pith/arXiv arXiv 2025

[14] [14]

Rethinking the divergence regularization in llm rl.arXiv preprint arXiv:2606.09821, 2026

Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee, Liefeng Bo, and Tianyu Pang. Rethinking the divergence regularization in llm rl.arXiv preprint arXiv:2606.09821, 2026

Pith/arXiv arXiv 2026

[15] [15]

DAPO: An open-source LLM reinforcement learning system at scale

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, YuYue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao ...

2025

[16] [16]

Small leak can sink a great ship–boost rl training on moe with icepop!, Sep 2025

Xin Zhao, Yongkang Liu, Kuan Xu, Jia Guo, Zihao Wang, Yan Sun, Xinyu Kong, Qianggang Cao, Liang Jiang, Zujie Wen, Zhiqiang Zhang, and Jun Zhou. Small leak can sink a great ship–boost rl training on moe with icepop!, Sep 2025. URLhttps://ringtech.notion.site/icepop

2025

[17] [17]

Stabilizing reinforcement learning with llms: Formulation and practices, 2025

Chujie Zheng, Kai Dang, Bowen Yu, Mingze Li, Huiqiang Jiang, Junrong Lin, Yuqiong Liu, Hao Lin, Chencan Wu, Feng Hu, An Yang, Jingren Zhou, and Junyang Lin. Stabilizing reinforcement learning with llms: Formulation and practices, 2025. URLhttps://arxiv.org/abs/2512.01374

arXiv 2025

[18] [18]

Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization.arXiv preprint arXiv:2507.18071, 2025

Pith/arXiv arXiv 2025

[19] [19]

Gonzalez, Clark Barrett, and Ying Sheng

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs.arXiv preprint arXiv:2312.07104, 2023

Pith/arXiv arXiv 2023

[20] [20]

X yt µ(yt |s t) π(yt |s t) µ(yt |s t) −1 # = 2ξ TX t=1 Est∼µ h 2DTV(µ(· |st)∥π(· |st)) i = 4ξE y∼µ

Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv. 13 A Notation Table 2Notation used throughout the paper. Symbol Meaning x, yb, sb,t Prompt; responsey b = (yb,1, . . . , yb,Tb ); states b,t = (x, yb,1, . . ....

2025

[21] [21]

This is consistent with a sampledDTV gate improving stability on the reported stack, but it does not explain the remaining gap toSAT-GSPO w/ R3 or even toSAT-GRPO w/ R3

It stays above the GRPO baseline at both lags and outside the runs marked as later collapse. This is consistent with a sampledDTV gate improving stability on the reported stack, but it does not explain the remaining gap toSAT-GSPO w/ R3 or even toSAT-GRPO w/ R3. Table 7Reported best AIME24 pass@1 for the DPPO comparison. Method lag1lag8 GRPO31.25 30.17 ‡ ...