Pith. sign in

REVIEW 3 major objections 6 minor 21 references

One rollout per prompt stabilizes async RL for LLMs

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 08:31 UTC pith:OFXK4YJZ

load-bearing objection SAO: single-rollout async RL with value-based advantages — solid empirical work with a real confound in the main comparison the 3 major comments →

arxiv 2607.07508 v1 pith:OFXK4YJZ submitted 2026-07-08 cs.LG cs.AI

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Zhenyu Hou , Yujiang Li , Jie Tang , Yuxiao Dong This is my paper
classification cs.LG cs.AI
keywords asynchronousagenticsingle-rollouttrainingimprovelearningmodeloptimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Single-rollout Asynchronous Optimization (SAO), a method for training large language models with reinforcement learning in asynchronous settings where rollouts of varying length arrive continuously. The core claim is that replacing group-wise sampling (as used in GRPO, which waits for multiple responses per prompt) with single-rollout sampling (one response per prompt, trained immediately upon completion) reduces off-policy drift and is structurally better suited to agentic and online tasks. To make single-rollout training stable despite its inherent high variance, the paper introduces three supporting mechanisms: direct double-sided token-level importance sampling that bypasses tracking historical policy checkpoints by using rollout log-probabilities directly with strict clipping; a value model trained with frozen attention layers and twice-per-step updates to track the rapidly changing policy; and a skip-observation GAE estimator that computes advantages across action-to-action boundaries in multi-turn agent trajectories, avoiding noise from environment-generated tokens. The paper claims SAO trains stably for approximately one thousand steps and consistently outperforms GRPO and its variants on agentic coding (SWE-Bench Verified) and math reasoning benchmarks (AIME2025, BeyondAIME, HMMT, IMOAnswerBench).

Core claim

The central finding is that the structural mismatch between group-wise sampling and asynchronous training can be resolved by switching to single-rollout sampling, provided that the resulting variance is controlled through a combination of aggressive token-level clipping (using rollout probabilities directly rather than maintaining old-policy checkpoints), a value model whose attention layers are frozen during RL training to reduce gradient instability, and a GAE formulation that skips environment-observation tokens in multi-turn trajectories. The paper demonstrates that this combination enables stable training over ~1000 steps and outperforms GRPO variants, with the single-rollout approach n

What carries the argument

SAO (Single-rollout Asynchronous Optimization) combines: (1) Direct Double-Sided Importance Sampling (DIS) — computes the probability ratio directly between the current policy and the rollout policy using saved log-probabilities, then masks tokens outside a trust region [1−ε_l, 1+ε_h] from gradient computation entirely; (2) Faster Value Update — the value model is updated K=2 times per policy gradient step; (3) Frozen-Attention Value Training — only MoE projection layers are optimized in the value model while attention weights remain fixed; (4) Skip-Observation Token-level GAE — modifies the Bellman target to bridge across environment feedback tokens, linking action-end values directly to下一个

Load-bearing premise

The method depends on the value model being accurate enough to compensate for the high variance of single-sample gradient estimates. The paper acknowledges this dependency but does not quantify how good the value model needs to be or provide sensitivity analysis on value model quality beyond the ablations shown.

What would settle it

Train SAO with value models of varying pretraining corpus sizes and measure whether single-rollout training stability degrades below a threshold of value model quality, and whether frozen-attention training remains stabilizing across different backbone architectures.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Asynchronous RL for LLMs can move beyond throughput-focused systems to achieve genuine training stability and task effectiveness, potentially making long-horizon agentic training more practical at scale.
  • The finding that single-rollout RL is naturally suited to online learning with non-stationary rewards suggests a path toward models that adapt to shifting user preferences or evolving environments in real time, without requiring group-based feedback.
  • The skip-observation GAE formulation could be applied more broadly to any RL setting where agent actions are interleaved with external observations, such as tool use, web browsing, or embodied agents.
  • The demonstration that frozen-attention value training stabilizes optimization while full-parameter training destabilizes it raises questions about which components of pretrained models are best left fixed during RL post-training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not provide sensitivity analysis on value model pretraining corpus scale or quality, despite identifying it as a critical bottleneck. A natural testable extension would be to measure SAO's stability and final performance as a function of value pretraining data quantity, which would reveal whether there is a threshold below which single-rollout training becomes impractical.
  • The experiments use a single backbone (Qwen3-30B-A3B). Whether the frozen-attention finding generalizes to dense (non-MoE) architectures is unclear, since the stability observation is specifically attributed to MoE layers remaining stable while attention layers do not.
  • The online learning simulation uses binary rewards from an LLM judge with discrete stylistic shifts. Real-world non-stationary environments typically involve continuous, gradual preference drift with noisy rewards, which may stress the value model differently than the clean phase transitions tested here.
  • The claim that single-rollout sampling reduces off-policy effects is somewhat counterintuitive — a single sample per prompt provides less information about the reward landscape, not more. The reduction in off-policy effects likely comes from reduced latency (faster training updates) rather than from the single-rollout design itself, and disentangling these two factors would clarify the mechanism.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Single-rollout Asynchronous Optimization (SAO), a method for stabilizing asynchronous RL training of LLMs on agentic tasks. SAO replaces GRPO's group-wise sampling with one rollout per prompt, introduces Direct Double-Sided Importance Sampling (DIS) that bypasses tracking old-policy checkpoints, and proposes several value-model training strategies (faster critic updates, frozen attention, skip-observation GAE, scaled pretraining). Experiments on Qwen3-30B-A3B across five benchmarks (AIME2025, BeyondAIME, HMMT Nov 2025, IMOAnswerBench, SWE-Bench Verified) show SAO outperforming GRPO variants, and an online learning simulation demonstrates adaptability to non-stationary rewards. The method is also reported as deployed in GLM-5.2 training.

Significance. The paper addresses a practically important problem—training stability in asynchronous RL for long-horizon agentic LLM tasks—and proposes a coherent set of mechanisms. The DIS strategy that directly uses rollout log-probabilities (dropping old-policy tracking) is a clean engineering contribution. The skip-observation GAE derivation for multi-turn agentic trajectories is well-motivated. Ablations in Tables 3–4 and training dynamics in Figure 4 provide useful evidence for each design choice. The online learning simulation (§4.5) is a nice addition showing the value of single-rollout in non-stationary settings. However, the central comparison between SAO and GRPO has a confound (see major comments) that weakens the causal attribution of gains to the proposed mechanisms.

major comments (3)
  1. §4.1, Tables 1–2: The SAO vs. GRPO comparison confounds single-rollout sampling with prompt diversity. SAO uses 128 prompts × 1 rollout = 128 unique prompts per batch, while GRPO uses 16 prompts × 8 rollouts = 16 unique prompts per batch (§4.1). Both have 128 total rollouts, but SAO sees 8× more unique prompts per step. The performance gap between SAO (97.3 AIME2025) and GRPO w/ DIS (93.5) could be partially or entirely attributable to greater data diversity rather than reduced off-policy effects or value-based advantage estimation. The paper attributes improvement to reduced off-policy effects (§3.2) but does not discuss this alternative explanation. A controlled comparison—e.g., GRPO with 128 prompts × 1 rollout using a group-free baseline, or SAO with 16 prompts × 8 rollouts—would isolate the single-rollout mechanism's contribution. Without this, the central claim that single-rollout
  2. §3.2 ('Scaling Value Pretraining'): The value pretraining corpus is described only as 'significantly increasing the scale' with no quantification (number of samples, tokens, or training steps). Since the paper explicitly states that 'the cold start problem in value estimation is a major bottleneck' and that scaling is 'essential,' this parameter is load-bearing for the method's reproducibility. Please provide concrete numbers and, if feasible, a sensitivity analysis showing how value pretraining scale affects final performance.
  3. Abstract and §4.5: The GLM-5.2 deployment is cited as evidence of SAO's effectiveness ('SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model'), but no metrics are provided for this deployment. If the deployment is meant to support the method's claims, quantitative results (even limited) should be included; otherwise, the statement should be softened to acknowledge that GLM-5.2 used SAO without presenting it as validating evidence.
minor comments (6)
  1. Tables 3 and 4 appear to contain overlapping information (both report ablation results for value model training strategy and critic update frequency on AIME2025 and BeyondAIME). Consider merging into a single table.
  2. §4.1: The clipping parameters differ between math (ε_low=0.3, ε_high=5.0) and coding (ε_low=0.8, ε_high=3.0) but no justification is provided for these choices. A brief note on how these were selected would strengthen the presentation.
  3. §4.5: The online learning simulation uses only 3 stylistic phases. While illustrative, a more comprehensive evaluation with additional shifts or quantitative adaptation-speed metrics would strengthen the claim of effectiveness in non-stationary environments.
  4. Appendix A.2 mentions SPO (Xu and Ding, 2025) as a single-rollout baseline but does not include it in the main results tables. Including SPO in the ablation would provide a more complete comparison to existing single-rollout methods.
  5. §3.1: The relationship between DIS and the IcePop mechanism (Team et al., 2025) is mentioned but not fully elaborated. A more detailed comparison of what is shared and what differs would help readers position the contribution.
  6. Figure 1: The y-axis label and tick marks are small and difficult to read. Consider enlarging or splitting into separate panels for reasoning and coding benchmarks.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points: (1) a confound between single-rollout sampling and prompt diversity in the SAO vs. GRPO comparison, (2) missing quantification of the value pretraining corpus, and (3) unsupported GLM-5.2 deployment claims. We address each below and commit to revisions for all three.

read point-by-point responses

  1. Referee: §4.1, Tables 1–2: The SAO vs. GRPO comparison confounds single-rollout sampling with prompt diversity. SAO uses 128 prompts × 1 rollout = 128 unique prompts per batch, while GRPO uses 16 prompts × 8 rollouts = 16 unique prompts per batch. The performance gap could be attributable to greater data diversity rather than reduced off-policy effects or value-based advantage estimation. A controlled comparison would isolate the single-rollout mechanism's contribution.

    Authors: The referee correctly identifies a genuine confound in our experimental design. SAO sees 8× more unique prompts per batch than GRPO (128 vs. 16), and the performance gap between SAO and GRPO w/ DIS could indeed be partially attributable to prompt diversity rather than the single-rollout mechanism itself. We agree this weakens causal attribution and must be addressed. In the revision, we will add a controlled experiment: GRPO with 128 prompts × 1 rollout (using a group-free baseline such as running-mean for advantage estimation) to isolate the effect of prompt diversity from the single-rollout mechanism. This will show whether the gains persist when both methods see the same number of unique prompts. We will also add an explicit discussion of this confound in §4.1 and acknowledge prompt diversity as a contributing factor. We cannot fully resolve this without the additional experiment, so we are committed to running it. revision: yes

  2. Referee: §3.2 ('Scaling Value Pretraining'): The value pretraining corpus is described only as 'significantly increasing the scale' with no quantification (number of samples, tokens, or training steps). Since the paper states that the cold start problem is a major bottleneck and scaling is essential, this parameter is load-bearing for reproducibility. Please provide concrete numbers and a sensitivity analysis.

    Authors: The referee is correct that this is a reproducibility gap. The manuscript describes the value pretraining scaling as essential but provides no concrete numbers, which is insufficient for reproducibility. We will add specific details in the revision: the number of pretraining samples, total token count, and training steps for the value pretraining corpus. Regarding the sensitivity analysis, we agree this would strengthen the paper's claims about the importance of scaling. We will include a sensitivity analysis with at least two pretraining scales (e.g., 1× and the full scale used) showing the effect on final performance, subject to compute availability. If a full sensitivity sweep is not feasible within the revision timeline, we will at minimum provide the concrete numbers and a comparison between the pretraining scale used and a smaller-scale baseline. revision: partial

  3. Referee: Abstract and §4.5: The GLM-5.2 deployment is cited as evidence of SAO's effectiveness but no metrics are provided. If the deployment is meant to support the method's claims, quantitative results should be included; otherwise the statement should be softened.

    Authors: The referee is right that the current phrasing presents GLM-5.2 deployment as validating evidence without providing supporting metrics. We have two options, and we will pursue both where feasible. First, if we can release even limited quantitative results from the GLM-5.2 training (e.g., training stability metrics or benchmark comparisons with and without SAO), we will include them. Second, regardless of whether metrics can be released, we will soften the statement in both the abstract and §4.5 to clarify that GLM-5.2 used SAO in its training pipeline, without presenting this as quantitative validation of the method. The revised wording will distinguish between 'SAO was deployed in production' (a statement about engineering applicability) and 'SAO improved GLM-5.2 performance by X%' (a claim requiring metrics, which we will either support or omit). revision: yes

Circularity Check

0 steps flagged

No significant circularity: SAO's method is derived from standard RL principles (PPO, GAE, importance sampling) with modifications justified by independent experiments and ablations.

full rationale

The paper's core technical contributions—direct double-sided importance sampling (DIS), single-rollout sampling, faster value updates, frozen-attention value training, and skip-observation GAE—are all derived from standard RL foundations (PPO, GAE, Bellman equations) with modifications motivated by pilot experiments and ablation studies. The DIS strategy (Eqs. 1-3) is a straightforward modification of PPO clipping that drops the old-policy tracking and uses rollout log-probabilities directly; this is a design choice validated empirically, not a circular definition. The skip-observation GAE (Eqs. 4-5) is a legitimate derivation from the Bellman equation adapted for agentic trajectories with observation tokens. The value-model training strategies (faster updates, frozen attention) are justified by ablation results in Tables 3-4 and training dynamics in Figure 4, showing independent evidence for each design choice. The GLM-5.2 deployment is mentioned as an application of SAO, not as evidence for its correctness. The self-citation to GLM Team [2025] refers to the deployed model, not to a theorem or result that the paper's argument depends on. The online learning experiment uses GLM-4.7 as a judge, which is a separate model from the authors' own prior work. No step in the derivation chain reduces to its inputs by construction, and no 'prediction' is a renamed fit. The skeptic's concern about confounded prompt diversity (128 unique prompts vs 16) is a valid experimental design critique but is not a circularity issue—it concerns confounding, not self-definitional reasoning.

Axiom & Free-Parameter Ledger

6 free parameters · 3 axioms · 0 invented entities

The method introduces no new entities (particles, forces, dimensions). It recombines existing RL components (PPO clipping, GAE, actor-critic) with engineering choices (frozen attention, skip-observation GAE). Free parameters are hyperparameters tuned per task type. The axioms are domain assumptions about LLM RL training dynamics, not mathematical postulates.

free parameters (6)
  • epsilon_low (math) = 0.3
    Token clipping lower bound, chosen for math reasoning tasks
  • epsilon_high (math) = 5.0
    Token clipping upper bound, chosen for math reasoning tasks
  • epsilon_low (coding) = 0.8
    Different clipping lower bound for SWE-Bench coding tasks
  • epsilon_high (coding) = 3.0
    Different clipping upper bound for SWE-Bench coding tasks
  • K (value update frequency) = 2
    Number of value model updates per policy update
  • alpha (length-adaptive GAE) = 1.5
    Controls GAE lambda scaling based on sequence length
axioms (3)
  • domain assumption Pre-trained attention weights possess sufficient semantic capability for value estimation
    §3.2: Justifies freezing attention layers in the value model; supported by gradient norm analysis but not independently verified beyond the specific Qwen3 backbone.
  • domain assumption Rollout log-probabilities from multiple policy versions can serve as a unified behavior proxy
    §3.1: The DIS strategy assumes using pi_rollout directly for importance sampling is sufficient without tracking exact historical policies.
  • domain assumption Value model pretraining at sufficient scale resolves the cold-start problem
    §3.2: States scaling value pretraining is 'essential' but does not quantify the scale or provide ablation on pretraining data volume.

pith-pipeline@v1.1.0-glm · 16657 in / 2300 out tokens · 515763 ms · 2026-07-09T08:31:41.003222+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

Figures

Figures reproduced from arXiv: 2607.07508 by Jie Tang, Yujiang Li, Yuxiao Dong, Zhenyu Hou.

Figure 1
Figure 1. Figure 1: The performance of SAO on reasoning and coding benchmarks. The four reasoning benchmarks are evaluated in a reasoning-with-Python-tool setting, where the baseline is the Qwen3- 30B-A3B SFT model; SWE-Bench Verified evaluates coding with the Qwen3-30B-A3B baseline. SAO outperforms the corresponding baseline and GRPO across all five benchmarks. ∗Equal Contribution. Work done while ZH and YL interned at Z.AI.… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of SAO with single rollout design. The numbers denote the generation order of [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison between SAO and GRPO (w/ DIS) during training. It can be [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of asynchronous single-rollout RL. (a) Explained Variance for SAO and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Online learning simulation under changing writing-style preferences. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training reward for token-level SAO training and step-level variants, where token-level [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 21 canonical work pages · 16 internal anchors

  1. [1]

    Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

    Arash Ahmadian et al. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.arXiv preprint arXiv:2402.14740,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    MathArena: Evaluating LLMs on Uncontaminated Math Competitions

    Mislav Balunovi´c, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovi´c, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions.arXiv preprint arXiv:2505.23281,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning.arXiv preprint arXiv:2505.24298,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

    Team GLM, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models.arXiv preprint arXiv:2508.06471,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Acme: A Research Framework for Distributed Reinforcement Learning

    10 Matt Hoffman et al. Acme: A research framework for distributed reinforcement learning.arXiv preprint arXiv:2006.00979,

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  7. [7]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Let's Verify Step by Step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Part ii: Roll flash–accelerating rlvr and agentic training with asynchrony.arXiv preprint arXiv:2510.11345,

    Han Lu, Zichen Liu, Shaopan Xiong, Yancheng He, Wei Gao, Yanan Wu, Weixun Wang, Jiashun Liu, Yang Li, Haizhou Zhao, et al. Part ii: Roll flash–accelerating rlvr and agentic training with asynchrony.arXiv preprint arXiv:2510.11345,

    work page arXiv

  10. [10]

    Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V . Le, and Junehyuk Jung. Towards robust mathematical reasoning. InProceedings of the 2025 C...

    work page 2025

  11. [11]

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu

    URL https:// aclanthology.org/2025.emnlp-main.1794/. V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. InInternational conference on machine learning, pages 1928–1937. PmLR,

    work page 2025

  12. [12]

    WebGPT: Browser-assisted question-answering with human feedback

    Reiichiro Nakano et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. arXiv preprint arXiv:2410.18252,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Zhihong Shao et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Every step evolves: Scaling reinforcement learning for trillion-scale thinking model.arXiv preprint arXiv:2510.18855,

    Ling Team, Anqi Shen, Baihui Li, Bin Hu, Bin Jing, Cai Chen, Chao Huang, Chao Zhang, Chaokun Yang, Cheng Lin, et al. Every step evolves: Scaling reinforcement learning for trillion-scale thinking model.arXiv preprint arXiv:2510.18855,

    work page arXiv

  17. [17]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    11 Xuezhi Wang et al. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [19]

    Zhongwen Xu and Zihan Ding

    URLhttps://arxiv.org/abs/2509.18119. Zhongwen Xu and Zihan Ding. Single-stream policy optimization.arXiv preprint arXiv:2509.13232,

    work page arXiv

  19. [20]

    Qwen3 Technical Report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Shihui Yang, Chengfeng Dou, Peidong Guo, Kai Lu, Qiang Ju, Fei Deng, and Rihui Xin. Dcpo: Dynamic clipping policy optimization.arXiv preprint arXiv:2509.02333, 2025b. S...

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [21]

    VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.arXiv preprint arXiv:2504.05118,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [22]

    Group Sequence Policy Optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization.arXiv preprint arXiv:2507.18071,

    work page internal anchor Pith review Pith/arXiv arXiv