Pith. sign in

REVIEW 3 major objections 4 minor 57 references

A readable, single-loop training framework for agentic reinforcement learning can match the throughput of a heavy production stack while guaranteeing that every trained token is one the framework generated.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:46 UTC pith:CIKPZKQ5

load-bearing objection Honest, well-built RL systems paper whose headline parity claim is throughput-only because the benchmark gate rejected every batch — fixable, but the abstract overstates. the 3 major comments →

arxiv 2607.21653 v1 pith:CIKPZKQ5 submitted 2026-07-22 cs.LG cs.CLcs.DC

Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning

Jian Hu , Huiying Li , Hao Zhang , Binfeng Xu , Yifan Zhang , Shaokun Zhang , Hemil Desai , Michael Demoret
show 3 more authors
Pavlo Molchanov Jan Kautz Yi Dong
This is my paper
classification cs.LG cs.CLcs.DC
keywords agentic reinforcement learningasynchronous training loopmixture-of-expertstoken-exactnesstraining-inference consistencyreadability-first designthroughput paritydistributed training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Molt is a training framework for agentic reinforcement learning built on the bet that a small, readable codebase can support frontier-scale training without hyperscale complexity. The paper claims that the entire algorithm flow—agent, rollout, reward, loss—fits in one asynchronous loop with a token-first contract: the framework never trains on a token it did not generate, keeping token ids and log-probabilities aligned from sampling to loss. If the bet holds, RL researchers would stop paying an infrastructure tax on every algorithmic change, since a new estimator or filtering stage becomes a single pure-function edit in a visible location. The paper's evaluation is designed to show that this leanness costs no throughput: under a matched, fully asynchronous protocol, its step time is statistically comparable to a state-of-the-art production stack, and the same loop scales by configuration from a dense 4B model to a 700B mixture-of-experts model.

Core claim

The central claim is that complexity in RL training infrastructure is inherited from hyperscale, not required for capable agentic research, and that a compact single-loop design can match the step time of a state-of-the-art production stack. Under a matched fully-asynchronous protocol with all shared settings pinned, the reported step time is 119.4 ± 2.3 seconds for Molt versus 109.5 ± 10.3 seconds for the comparison stack, a difference the paper treats as within cross-run variability. The paper is explicit that on the routing-sensitive 128-expert checkpoint, a distributed mixture-of-experts forward mismatch caused its sequence-level gate to reject every batch, so the reported times are thro

What carries the argument

The central object is a token-exact trajectory managed by one asynchronous loop. A 'token-first contract' keeps sampled token ids, per-token log-probabilities, action ranges, rewards, and multimodal tensors aligned from the serving engine's sampler to the loss; no component re-derives tokens from text. The loop sustains a pool of prompt groups in flight, pauses the engines on a weight update, broadcasts new weights directly to them, and resumes retained requests, using per-token off-policy correction behind a sequence-level gate when policy versions mix. For mixture-of-experts policies, the engine's per-token expert choices are replayed by the actor during training, closing the routing gap b

Load-bearing premise

The parity result rests on the assumption that the distributed mixture-of-experts forward mismatch that made every batch fail the safety gate is an upstream issue affecting both stacks equally, so the reported step times—throughput without an effective policy update—still measure a fair comparison.

What would settle it

Re-run the matched head-to-head after correcting the distributed mixture-of-experts forward mismatch; if Molt's actor log-probabilities still differ from a reference forward by about one nat (causing gate rejection) while the comparison stack's do not, the parity claim is unsupported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A researcher can author an agent in one Python file, launch it with one command, and see per-step reward, length, and timing statistics in the same log, so an algorithmic edit touches exactly one visible component.
  • Serving optimizations arrive as configuration flags: the paper measures a 5× generation speedup from speculative decoding and an 18.3 GB GPU-memory saving from optimizer offload.
  • The same launch path and visible loop train a dense 4B model and a 700B mixture-of-experts model at expert-parallelism 256 without a backend migration.
  • Under the matched fully-asynchronous protocol, step-time means are within cross-run spread of the production stack, so the paper claims no superiority in either direction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the token-first invariant is the real source of the design's reliability, a direct comparison of convergence quality against a retokenization-based pipeline on the same objective could isolate its value; the paper does not run such a test.
  • The fairness note implies the parity claim is, at least on the 128-expert checkpoint, a throughput-only claim; a natural next experiment is to re-run after the upstream fix and verify that the gate admits batches and both stacks improve the policy at comparable rates.
  • The readability-for-assistants principle yields a testable hypothesis the paper leaves unmeasured: a coding assistant should implement a new estimator in Molt with fewer context tokens and fewer errors than in a layered stack.
  • The 700B parameter run is asserted but no logs are provided; publishing per-step gate statistics and throughput for that run would substantiate the claim that scale is configuration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. Molt is a PyTorch-native framework for agentic RL with a deliberately small codebase (approximately 8.6K RL-path LOC), one asynchronous Ray queue, vLLM serving, and an FSDP2/AutoModel actor. The paper contributes five design principles, a token-first agent boundary (Env and ChatAgent with loopback capture), a transport with no engine forks and routing replay for MoE consistency, and an open implementation with recipes. The main empirical claims are: single-framework measurements on a 35B multimodal MoE workload (prefix caching, speculative decoding, optimizer offload); a head-to-head throughput comparison against a Megatron-based stack (slime) under a matched protocol, reported as statistically comparable (119.4±2.3 vs 109.5±10.3 s/step); and an asserted end-to-end run on a 700B MoE at EP256. The paper's fairness notes disclose that the head-to-head checkpoint causes the sequence gate to reject every batch, making the comparison throughput-only.

Significance. If the claims hold, Molt is a useful systems contribution: a readable codebase that preserves token-exact training invariants and avoids forking vLLM or AutoModel, with reproducible containers and recipes. It proposes no new RL objective, which is appropriate for a systems paper, and its estimators are imported from prior work. The honesty of the fairness notes is a strength, but it does not repair the gap between the abstract's 'statistically comparable' and the throughput-only nature of the headline parity run. The 700B EP256 claim is currently unverifiable. I therefore regard the core design as plausible and the empirical support as incomplete.

major comments (3)
  1. [Sec. 4.3 / Table 3 / fairness notes] The headline parity experiment is throughput-only. The fairness notes state that on the 128-expert checkpoint actor log-probabilities differ from a reference forward by approximately 1 nat and the [0.99,1.01] sequence gate rejects the batch, so no effective policy update occurs. The Table 3 figures 119.4±2.3 vs 109.5±10.3 s/step therefore measure generation/transport with the update path effectively disabled; they do not show that Molt's FSDP2/EP training path imposes no cost when gradients actually flow. The abstract and conclusion present 'statistically comparable' without this caveat. Please either rerun with a checkpoint that passes the gate, or explicitly reframe the claim as transport-throughput parity and add the caveat to the abstract and conclusion. In addition, because Molt introduces routing replay and route capture (Sec. 3.5), the ~1 nat mismatch could involve that machinery;
  2. [Sec. 3.5 / Sec. 6 / contribution 4] The 700B MoE at EP256 end-to-end run asserted in Sec. 3.5 and Sec. 6 is load-bearing for the 'scale is configuration' contribution, but no measurement, log, configuration artifact, or reproduction step is provided. Please supply at least a configuration and minimal evidence (resource count, step-time or memory numbers, logs), or downgrade the claim to 'the configuration path is supported'.
  3. [Sec. 4.3 / Sec. 3.6 / Table 3] The fairness note says the gate rejects every batch; please clarify what 'optimizer step' includes under gate rejection and whether the policy_train timing in Sec. 4.1/4.3 includes a backward pass and optimizer update. If not, the Tok/GPU/s column in Table 3 does not represent trained tokens and the comparison is not about training throughput.
minor comments (4)
  1. [Sec. 4.2] The 0.05s re-prefill measurement has no cache-miss baseline; the text acknowledges this, but the section heading could mislead. Please state explicitly that this is a path demonstration, not an end-to-end speedup.
  2. [Sec. 4.2] The 5x speculative-decoding speedup (329s to 64s) appears to be a single point; report whether this is mean ± s.d. over repeated runs to match the rigor of Sec. 4.3.
  3. [Sec. 4.3] The statement that longer trajectories will shrink the backend difference is a conjecture; no experiment at 32K–128K supports it. Please label it as such or add a longer-context measurement.
  4. [Sec. A.2] The 'geo3k' dataset is used without citation; please provide a reference. Also, Table 1's LOC counts are self-reported via import-graph tracing; including the exact counting script/commit would make the comparison reproducible.

Circularity Check

0 steps flagged

No derivation reduces to its inputs; the parity claim rests on an external benchmark, with only minor non-load-bearing self-citations.

full rationale

The paper's load-bearing claims are an implementation invariant and an empirical parity result, not derived predictions. 'Molt never trains on a token it did not generate' is enforced by token-in/token-out capture, not derived from a fitted or self-referential quantity. The central parity claim is benchmarked against an external system, slime/Megatron, under a pinned matched protocol (Sec. 4.3, Table 2), so it is not defined in terms of Molt's own outputs. The fairness note explicitly discloses that the checkpoint's distributed-MoE forward mismatch makes every row throughput-only: 'actor log-probabilities differ from an independent reference forward by approximately one nat, and the [0.99,1.01] sequence gate (Liu et al., 2025a) rejects the batch, so the reported step times measure throughput without an effective policy update.' This is a validity limitation on what the parity experiment establishes, not circularity: the comparison is external and the caveat is disclosed. Self-citations exist (REINFORCE++ by Hu et al. as the default estimator; OpenRLHF as a fork ancestor; Polar as related trajectory-capture work), but none is used to prove throughput or the token-exact invariant, and the paper explicitly states it proposes no new objective. No fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. Score 2 reflects only the presence of minor, non-load-bearing self-citations; the central derivation is independent.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No free parameters (no fitted constants); three load-bearing domain assumptions listed; no invented entities. The system is a composition of existing open-source components.

axioms (3)
  • domain assumption The upstream distributed-MoE forward mismatch in the 30B benchmark checkpoint is not caused by Molt's token-first or routing-replay path; both stacks are equally affected.
    Sec. 4.3 fairness notes attribute the ~1 nat divergence to the checkpoint and treat both rows as throughput-only; if the mismatch were Molt-specific, the parity result would be invalid.
  • domain assumption FSDP2, vLLM, and NeMo AutoModel compose correctly at large scale (EP256) without hidden interactions; the 700B run is asserted without measured convergence or throughput.
    Sec. 3.5 and Sec. 6 state the 700B end-to-end run but provide no quantitative evidence.
  • domain assumption The matched protocol's residual asymmetries (training layout, rollout engine vLLM vs SGLang, asynchrony style) do not dominate the step-time comparison.
    Sec. 4.3 fairness notes disclose but cannot control these asymmetries; the parity conclusion assumes they are second-order.

pith-pipeline@v1.3.0-alltime-deepseek · 13134 in / 12516 out tokens · 108582 ms · 2026-08-01T09:46:05.993197+00:00 · methodology

0 comments
read the original abstract

Agentic reinforcement learning research is constant algorithm modification, new estimators, new pipeline stages, new rollout schemes, and in mainstream frameworks each change threads through layers of trainer, distributed backend, and rollout glue: the cost lands on the researcher at every iteration. Molt is a PyTorch-native training framework built to keep that cost small: a codebase compact and clean enough for a researcher to hold in their head, and for an AI coding assistant to read and reason about in its entirety, so the algorithm flow can be traced and changed end to end. The agent is an ordinary program, and one asynchronous loop trains multimodal and mixture-of-experts policies while never training on a token it did not generate, consistent in tokens, policy versions, and model semantics. Leanness does not cost performance: under a matched, fully asynchronous protocol, Molt is statistically comparable to a state-of-the-art Megatron-based stack. Molt is open source and provides recipes and containers at https://github.com/NVIDIA-NeMo/labs-molt.

Figures

Figures reproduced from arXiv: 2607.21653 by Binfeng Xu, Hao Zhang, Hemil Desai, Huiying Li, Jan Kautz, Jian Hu, Michael Demoret, Pavlo Molchanov, Shaokun Zhang, Yi Dong, Yifan Zhang.

Figure 1
Figure 1. Figure 1: The whole system: three components and one loop. Molt composes user agents in plain Python, vLLM rollout engines behind a request router, and a single FSDP2 policy actor on NeMo AutoModel around one Ray asynchronous queue that implements the streaming pool and partial rollout. Trajectories flow token-exactly from the engines through the queue into training, and weight refit returns over NCCL directly to th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 20 linked inside Pith

  1. [2]

    Chen, Rongjian and Hu, Jianmin and Ye, Kejiang and Xu, Minxian , journal =

  2. [3]

    Hu, Tianhao and Liu, Xiangcheng and Xiao, Youshao and Zheng, Yang and Huang, Xuan and Ding, Jinrui and others , journal =

  3. [4]

    Zhang, Liujie and Ning, Benzhe and Yang, Rui and Yu, Xiaoyan and Li, Jiaxing and Wu, Lumeng and others , journal =

  4. [5]

    Gao, Wei and Zhao, Yuheng and Wu, Tianyuan and Xiong, Shaopan and Wang, Weixun and An, Dakai and others , journal =

  5. [6]

    Laminar: A Scalable Asynchronous

    Sheng, Guangming and Tong, Yuxuan and Wan, Borui and Zhang, Wang and Jia, Chaobo and Wu, Xibin and others , journal =. Laminar: A Scalable Asynchronous

  6. [7]

    Stabilizing

    Ma, Wenhan and Zhang, Hailin and Zhao, Liang and Song, Yifan and Wang, Yudong and Sui, Zhifang and Luo, Fuli , journal =. Stabilizing

  7. [8]

    Zhong, Yinmin and Zhang, Zili and Song, Xiaoniu and Hu, Hanpeng and Jin, Chao and Wu, Bingyang and others , journal =

  8. [9]

    Polar: Agentic

    Xu, Binfeng and Zhang, Hao and Zhang, Shaokun and Han, Songyang and Liu, Mingjie and Hu, Jian and Diao, Shizhe and Jin, Zhenghui and Zou, Yunheng and Demoret, Michael and Kautz, Jan and Dong, Yi , journal =. Polar: Agentic

  9. [10]

    Zhang, Hao and Liu, Mingjie and Zhang, Shaokun and Han, Songyang and Hu, Jian and Jin, Zhenghui and Zhang, Yuchi and Diao, Shizhe and Lu, Ximing and Xu, Binfeng and Yu, Zhiding and Kautz, Jan and Dong, Yi , journal =

  10. [11]

    Sheng, Guangming and Zhang, Chi and Ye, Zilingfeng and Wu, Xibin and Zhang, Wang and Zhang, Ru and Peng, Yanghua and Lin, Haibin and Wu, Chuan , booktitle =

  11. [12]

    Hu, Jian and Wu, Xibin and Shen, Wei and Liu, Jason Klein and Zhu, Zilin and Wang, Weixun and Jiang, Songlin and Wang, Haoran and Chen, Hao and Chen, Bin and Fang, Weikai and Xianyu and Cao, Yu and Xu, Haotian and Liu, Yiming , journal =

  12. [13]

    Hu, Jian and Liu, Jason Klein and Xu, Haotian and Shen, Wei , journal =

  13. [14]

    and Barrett, Clark and Sheng, Ying , journal =

    Zheng, Lianmin and Yin, Liangsheng and Xie, Zhiqiang and Sun, Chuyue and Huang, Jeff and Yu, Cody Hao and Cao, Shiyi and Kozyrakis, Christos and Stoica, Ion and Gonzalez, Joseph E. and Barrett, Clark and Sheng, Ying , journal =

  14. [15]

    Fu, Wei and Gao, Jiaxuan and Shen, Xujie and Zhu, Chen and Mei, Zhiyu and He, Chuyi and Xu, Shusheng and Wei, Guo and Mei, Jun and Wang, Jiashu and others , journal =

  15. [16]

    Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and Fan, Tiantian and Liu, Gaohong and Liu, Lingjun and Liu, Xin and others , journal =

  16. [17]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, YK and Wu, Yang and others , journal =

  17. [19]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Zhang, Ruoyu and Xu, Runxin and Zhu, Qihao and Ma, Shirong and Wang, Peiyi and Bi, Xiao and others , journal =

  18. [21]

    and Zhang, Hao and Stoica, Ion , booktitle =

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle =. Efficient Memory Management for Large Language Model Serving with

  19. [22]

    and Stoica, Ion , booktitle =

    Moritz, Philipp and Nishihara, Robert and Wang, Stephanie and Tumanov, Alexey and Liaw, Richard and Liang, Eric and Elibol, Melih and Yang, Zongheng and Paul, William and Jordan, Michael I. and Stoica, Ion , booktitle =. Ray: A Distributed Framework for Emerging

  20. [23]

    Zhao, Yanli and Gu, Andrew and Varma, Rohan and Luo, Liang and Huang, Chien-Chin and Xu, Min and Wright, Less and Shojanazeri, Hamid and Ott, Myle and Shleifer, Sam and others , journal =

  21. [24]

    Brockman, Greg and Cheung, Vicki and Pettersson, Ludwig and Schneider, Jonas and Schulman, John and Tang, Jie and Zaremba, Wojciech , journal =

  22. [25]

    Xie, Tianbao and Zhang, Danyang and Chen, Jixuan and Li, Xiaochuan and Zhao, Siheng and Cao, Ruisheng and Toh, Jing Hua and Cheng, Zhoujun and Shin, Dongchan and Lei, Fangyu and others , journal =

  23. [26]

    and Yang, Yuqing , journal =

    Luo, Xufang and Zhang, Yuge and He, Zhiyuan and Wang, Zilong and Zhao, Siyun and Li, Dongsheng and Qiu, Luna K. and Yang, Yuqing , journal =. Agent Lightning: Train

  24. [27]

    When Speed Kills Stability: Demystifying

    Liu, Jiacai and Li, Yingru and Fu, Yuqian and Wang, Jiawei and Liu, Qian and Shen, Yu , howpublished =. When Speed Kills Stability: Demystifying

  25. [29]

    Understanding

    Liu, Zichen and Chen, Changyu and Li, Wenjun and Qi, Penghui and Pang, Tianyu and Du, Chao and Lee, Wee Sun and Lin, Min , journal =. Understanding

  26. [30]

    International Conference on Learning Representations (ICLR) , year =

    High-Dimensional Continuous Control Using Generalized Advantage Estimation , author =. International Conference on Learning Representations (ICLR) , year =

  27. [32]

    Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLMs

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet \"U st \"u n, and Sara Hooker. Back to basics: Revisiting REINFORCE -style optimization for learning from human feedback in LLMs . arXiv preprint arXiv:2402.14740, 2024

    Pith/arXiv arXiv 2024

  28. [33]

    OpenAI Gym

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym . arXiv preprint arXiv:1606.01540, 2016

    Pith/arXiv arXiv 2016

  29. [34]

    RolloutPipe : Overlapping pipelined rollout and training in disaggregated on-policy LLM reinforcement learning

    Rongjian Chen, Jianmin Hu, Kejiang Ye, and Minxian Xu. RolloutPipe : Overlapping pipelined rollout and training in disaggregated on-policy LLM reinforcement learning. arXiv preprint arXiv:2606.26997, 2026

    Pith/arXiv arXiv 2026

  30. [35]

    DeepSeek-V3 technical report

    DeepSeek-AI . DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  31. [36]

    AReaL : A large-scale asynchronous reinforcement learning system for language reasoning

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. AReaL : A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24298, 2025

    Pith/arXiv arXiv 2025

  32. [37]

    RollArt : Disaggregated multi-task agentic RL training at scale

    Wei Gao, Yuheng Zhao, Tianyuan Wu, Shaopan Xiong, Weixun Wang, Dakai An, et al. RollArt : Disaggregated multi-task agentic RL training at scale. arXiv preprint arXiv:2512.22560, 2025

    arXiv 2025

  33. [38]

    DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  34. [39]

    OpenRLHF : An easy-to-use, scalable and high-performance RLHF framework

    Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, and Yiming Liu. OpenRLHF : An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405.11143, 2024

    Pith/arXiv arXiv 2024

  35. [40]

    REINFORCE++ : Stabilizing critic-free policy optimization with global advantage normalization

    Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE++ : Stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262, 2025

    Pith/arXiv arXiv 2025

  36. [41]

    DORA : A scalable asynchronous reinforcement learning system for language model training

    Tianhao Hu, Xiangcheng Liu, Youshao Xiao, Yang Zheng, Xuan Huang, Jinrui Ding, et al. DORA : A scalable asynchronous reinforcement learning system for language model training. arXiv preprint arXiv:2604.26256, 2026

    Pith/arXiv arXiv 2026

  37. [42]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), 2023

    2023

  38. [43]

    When speed kills stability: Demystifying RL collapse from the training-inference mismatch

    Jiacai Liu, Yingru Li, Yuqian Fu, Jiawei Wang, Qian Liu, and Yu Shen. When speed kills stability: Demystifying RL collapse from the training-inference mismatch. Online article, https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda, 2025 a

    2025

  39. [44]

    Understanding R1-Zero -like training: A critical perspective

    Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero -like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025 b

    Pith/arXiv arXiv 2025

  40. [45]

    Qiu, and Yuqing Yang

    Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent lightning: Train ANY AI agents with reinforcement learning. arXiv preprint arXiv:2508.03680, 2025

    Pith/arXiv arXiv 2025

  41. [46]

    Stabilizing MoE reinforcement learning by aligning training and inference routers

    Wenhan Ma, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. Stabilizing MoE reinforcement learning by aligning training and inference routers. arXiv preprint arXiv:2510.11370, 2025

    arXiv 2025

  42. [47]

    Jordan, and Ion Stoica

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. Ray: A distributed framework for emerging AI applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2018

    2018

  43. [48]

    High-dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR), 2016

    2016

  44. [49]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  45. [50]

    DeepSeekMath : Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepSeekMath : Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  46. [51]

    Laminar: A scalable asynchronous RL post-training framework

    Guangming Sheng, Yuxuan Tong, Borui Wan, Wang Zhang, Chaobo Jia, Xibin Wu, et al. Laminar: A scalable asynchronous RL post-training framework. arXiv preprint arXiv:2510.12633, 2025 a

    arXiv 2025

  47. [52]

    HybridFlow : A flexible and efficient RLHF framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow : A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys), 2025 b

    2025

  48. [53]

    OSWorld : Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. OSWorld : Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37: 0 52040--52094, 2024

    2024

  49. [54]

    Polar: Agentic RL on any harness at scale

    Binfeng Xu, Hao Zhang, Shaokun Zhang, Songyang Han, Mingjie Liu, Jian Hu, Shizhe Diao, Zhenghui Jin, Yunheng Zou, Michael Demoret, Jan Kautz, and Yi Dong. Polar: Agentic RL on any harness at scale. arXiv preprint arXiv:2605.24220, 2026

    Pith/arXiv arXiv 2026

  50. [55]

    Next-generation agentic reinforcement learning systems enable self-evolving agents

    Ran Yan, Wei Fu, Jiale Li, Shusheng Xu, Zhiyu Mei, Jiaxuan Gao, et al. Next-generation agentic reinforcement learning systems enable self-evolving agents. arXiv preprint arXiv:2607.01120, 2026

    Pith/arXiv arXiv 2026

  51. [56]

    DAPO : An open-source LLM reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO : An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    Pith/arXiv arXiv 2025

  52. [57]

    ProRL agent: Rollout-as-a-service for RL training of multi-turn LLM agents

    Hao Zhang, Mingjie Liu, Shaokun Zhang, Songyang Han, Jian Hu, Zhenghui Jin, Yuchi Zhang, Shizhe Diao, Ximing Lu, Binfeng Xu, Zhiding Yu, Jan Kautz, and Yi Dong. ProRL agent: Rollout-as-a-service for RL training of multi-turn LLM agents. arXiv preprint arXiv:2603.18815, 2026 a

    arXiv 2026

  53. [58]

    Relax : An asynchronous reinforcement learning engine for omni-modal post-training at scale

    Liujie Zhang, Benzhe Ning, Rui Yang, Xiaoyan Yu, Jiaxing Li, Lumeng Wu, et al. Relax : An asynchronous reinforcement learning engine for omni-modal post-training at scale. arXiv preprint arXiv:2604.11554, 2026 b

    Pith/arXiv arXiv 2026

  54. [59]

    PyTorch FSDP : Experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. PyTorch FSDP : Experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 16 0 (12): 0 3848--3860, 2023

    2023

  55. [60]

    Group sequence policy optimization

    Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025

    Pith/arXiv arXiv 2025

  56. [61]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang : Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2023

    Pith/arXiv arXiv 2023

  57. [62]

    StreamRL : Scalable, heterogeneous, and elastic RL for LLMs with disaggregated stream generation

    Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, et al. StreamRL : Scalable, heterogeneous, and elastic RL for LLMs with disaggregated stream generation. arXiv preprint arXiv:2504.15930, 2025

    Pith/arXiv arXiv 2025