Pith. sign in

REVIEW 3 major objections 4 minor 61 references

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TideRL claims that agentic RL training goodput—the rollout tokens that actually reach the trainer and advance an update—can rise by up to 5.6× over synchronous baselines and by more than 33% over asynchronous ones when rollout…

desk verdict A credible, well-engineered systems integration whose end-to-end numbers I'd trust only after code and error bars, and whose zero-overhead migration claim needs explicit scoping for evaluation tasks. read the letter →

arxiv 2608.10402 v1 pith:ZLSQVPEC submitted 2026-08-11 cs.LG cs.DC

Yanyu Ren , Xizheng Wang , Xiao Liu , Bowen Lv , Hanchen Zhang , Shudan Zhang , Hanyu Lai , Shuai Wang
show 3 more authors
Li Chen Dan Li Jie Tang
This is my paper · ORCID
classification cs.LGcs.DC
keywords agenticreinforcementlearningRLgoodputcontinuoustaskbatchingKVcachemanagementelasticresourcescalingasynchronoustrainingreference-actorpipelining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TideRL sets out to fix reinforcement learning for multi-turn agentic large language models, where tasks pause for external environments, resume with growing contexts, and finish at wildly different times. The paper's central claim is that the right scheduling unit is task readiness rather than raw GPU occupancy: keep useful rollout state resident, choose between two reference-actor execution modes from the backlog already at hand and the pace at which new micro-batches arrive, and move ranks between rollout and training as the bottleneck shifts. On that basis TideRL reports RL training goodput—the rollout tokens that actually reach the trainer and advance an update—improved by up to 5.6× over synchronous baselines and by more than 33% over asynchronous baselines on text-only and multi-modal agentic workloads, while staying within a 0.01 best-of-N reward deficit and a 0.5% pass-rate deficit of the best synchronous result after 100 steps. If true, that matters because it attacks the real cost of agentic RL: GPU waiting, repeated prefill recomputation, and model-swap thrashing, which grow with trajectory variance and environment latency rather than with model size.

What carries the argument

The central object is the readiness-signal pair (RAS, TPRM): RAS is the number of micro-batches already waiting in the global buffer when a training step begins, and TPRM is the interval at which additional micro-batches become ready during the step. Everything in the system is coupled to this pair: Continuous Task Batching shapes when trajectories become ready, RA2P selects decoupled streaming versus colocated aggregation from the pair, and Elastic Resource Scaling reallocates GPU ranks to shrink whichever side is waiting. The load-bearing identity is the natural weight-sync boundary of on-policy RL: once new weights arrive, all prior KV caches are mathematically stale and must be flushed, so a migrating rank can discard its cached context for free, join the weight broadcast, and resume with the new policy without a pipeline suspension or an NCCL group rebuild.

What would settle it

Run one training step that contains live rollout groups and an in-flight evaluation batch pinned to an anchored frozen model, and trigger a rank migration while that batch is active. If the migration forces the pipeline to suspend, rebuilds a communication group, or reloads any KV cache for the evaluation tasks, the zero-overhead elasticity claim is falsified; the wall-clock cost of the migration itself is the observable.

Watch

Extended reading notes

Core claim

TideRL's central claim is that the three inefficiencies of agentic RL training—KV-cache preemption while tasks wait on environments, trainer stalls or reference-actor swap thrashing while micro-batches arrive unevenly, and static GPU partitions that cannot follow a shifting producer-consumer bottleneck—share one root cause: nobody schedules from task-level readiness. The proposed mechanism is a single feedback loop. Continuous Task Batching keeps useful rollout state resident by admitting, pausing, and resuming tasks with token-aware priorities that favor evaluation boundaries, near-complete GRPO groups, active trajectories, and long contexts. Resource-Aware Ref-Actor Pipelining consumes ready micro-batches through two modes selected by readiness signals: decoupled streaming, with loss computation deferred into the backward pass so reference forward, actor forward, and actor backward overlap with zero model swaps, and colocated aggregation with zero-copy shared-memory transfer when data is sparse. Elastic Resource Scaling moves individual ranks between rollout and training using those same signals, and because on-policy RL already invalidates every KV cache at each weight-sync boundary, the migration can discard caches for free and hide weight movement under the synchronization broadcast. With this loop, the paper reports training-throughput gains up to 5.6× over the best synchronous baseline and more than 1.33× over the asynchronous baselines on text and multi-modal agentic workloads, with task performance within 0.01 best-of-N reward and 0.5% pass rate of the synchronous reference.

Load-bearing premise

The whole scheme assumes that every rollout task's cached context becomes worthless the instant new model weights arrive, so a rank can be moved without saving anything; the paper's own limitations concede that evaluation tasks need an anchored, frozen model version, and keeping that version in sync can disturb the asynchronous pipeline—the one place where that free-discard assumption does not obviously hold.

Editorial extensions

If this is right

  • Synchronous phase barriers for agentic RL sacrifice most of the available goodput; adopting readiness-aware scheduling recovers up to 5.6× training throughput on text workloads and over 33% on multimodal workloads at near-identical task performance.
  • Elevating rollout scheduling from requests to tasks raises KV cache hit rate by 1.58× and generation throughput by 1.15×, because paused multi-turn tasks keep their context instead of being evicted while waiting for environment feedback.
  • Reference-actor execution does not have to choose between computation stalls and model-swap thrashing: decoupled streaming with deferred loss suits abundant ready data, and ready-batch aggregation with zero-copy transfer suits sparse data, cutting per-step training time by up to 44.3%.
  • Rank migration between rollout and training can be made effectively free by aligning it with the post-update cache flush, so a fixed GPU pool can absorb shifting bottlenecks and cut total waiting time by up to 77.6%.
  • Asynchronous RL systems can go faster without adding GPUs; they need to react to readiness signals, since elastic reallocation within a fixed allocation beats the best static rollout fraction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same RAS/TPRM mode-selection logic could address the evaluation-phase gap the authors leave for future work: a relay weight-synchronization path that keeps a frozen evaluation model separate would let the zero-overhead migration premise extend to steps containing evaluation tasks.
  • Beyond the paper, readiness-driven scheduling of this kind could transfer to interactive agent serving, where cache pinning and prefill decisions are made at task granularity; the difference is that serving lacks the weight-sync boundary that makes TideRL's migration free, so elasticity there would need a different cost argument.
  • Beyond the paper, the free-migration argument depends on the algorithm being on-policy with periodic full weight synchronization; for any future RL variant that allows stale KV caches to remain useful across weight updates, the zero-overhead elasticity claim would need to be re-derived.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents TideRL, an asynchronous reinforcement learning (RL) system for multi-turn agentic workloads. Three co-designed mechanisms are introduced: Continuous Task Batching (CTB) for task-level KV-cache-aware rollout scheduling, Resource-Aware Ref-Actor Pipelining (RA2P) that switches between decoupled streaming and colocated aggregation based on data readiness, and Elastic Resource Scaling (ERS) that migrates ranks between rollout and training using the same readiness signals. The paper claims up to 5.6x training-goodput improvement over the synchronous VeRL baseline, over 33% over asynchronous baselines (AReaL, StreamRL), comparable task performance, a 1.58x KV cache hit-rate improvement, 44.3% per-step training time reduction, and up to 77.6% waiting-time reduction. The evaluation covers text-only (WebShop, AlfWorld) and multimodal (OSWorld, ScienceBoard) tasks with Qwen models on 32 H100 GPUs.

Significance. If the claims are validated, TideRL is a substantive systems contribution. The paper correctly identifies real bottlenecks in agentic RL (KV-cache preemption, ref-actor thrashing, static resource allocation) and proposes a coherent, well-motivated co-design. The goodput definition (training throughput excluding stale tokens) is appropriate for on-policy RL. The component-wise motivation is supported by measurements (Figures 2-4), and the implementation is modular. However, the empirical validation has gaps: no repeated runs or error bars, a quantitative inconsistency in the ablation table, and an unresolved tension between the zero-overhead elasticity claim and the paper's own limitations about evaluation tasks. These issues make the central quantitative claims not yet fully supported.

major comments (3)
  1. [Section 6.4, Table 2] The improvement percentages in Table 2 do not match the listed throughput values. The baseline is 20.2 k token/s; adding RA2P gives 23.1, which is an increase of 14.4%, not 10.4%. Adding ERS to that gives 31.8, an increase of 37.7% over 23.1, not 35.7%. The 4.8% for CTB is close (33.3/31.8 = 1.047). This internal inconsistency affects the paper's component-wise attribution of gains and must be corrected, ideally with confidence intervals across repeated runs.
  2. [Section 5.2 and Section 7] The zero-overhead cache-free migration claim is over-generalized for evaluation tasks. Section 5.2 states that after each weight sync, all KV caches are invalid and can be discarded for free, so migrated tasks resume with a fresh prefill under the new weights. However, Table 1 gives evaluation tasks the highest CTB priority because they 'require anchored weights,' and Section 7's Limitations admits that enforcing a frozen model version for evaluation 'can temporarily disrupt the asynchronous pipeline momentum' and leaves a relay synchronization mechanism to future work. Since the end-to-end measurements include evaluation phases (Table 3 profiles 200 eval groups vs. 32 train groups, and Section 6.4 explicitly includes the initial evaluation step), the paper must either (a) demonstrate that ERS never closes a rank hosting an active evaluation task and maintain zero overhead during evaluation phases, or (b) restrict the zero-overhead claim to training rollout and quantify the overheads incurred during evaluation.
  3. [Section 6] All reported throughput, training-time, and waiting-time numbers are single-point measurements with no error bars, no repeated runs, and no code/data release. Given the high variance in agentic workloads described in the paper (e.g., Figure 4 shows RWT/TWT crossing patterns even under a fixed ratio, and Section 6.3 mentions 'high variance in training workloads'), the central claims (5.6x, 33%, 77.6%, and the ablation percentages) require at least a small number of repeated runs with confidence intervals, or a clear statement of experimental variance, to be convincing.
minor comments (4)
  1. [Title/Abstract vs. Body] The system is called 'TideRL' in the title and abstract but 'TIDERL' in the body; similarly, Section 5 uses 'RA²P' in the heading while the rest of the paper uses 'RA2P'. Please standardize the notation.
  2. [Section 4.2, Eq. (1)] The priority-score weights ω1–ω4 are introduced but no default values or sensitivity analysis are provided; since the CTB preemption hierarchy depends on this score, a specification of the default weights and a brief ablation would improve reproducibility.
  3. [Figure 12] The caption 'CTB beats F10 and F9 on both metrics' is vague; it should state which metrics (hit rate and throughput) and under what configuration the comparison is made.
  4. [Appendix A] The formula for Cdec defines δi in a compressed manner; clarifying how N0 enters the definition (e.g., the range of i for which δ_i=0) would make the cost model easier to follow.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TideRL's goodput and component gains are measured against external baselines and workloads, not derived from the paper's own definitions or self-citations.

full rationale

The paper's central claims are empirical. Section 6 measures training throughput, BoN reward, pass rate, KV-cache hit rate, and RWT/TWT against independent baselines (VeRL, AReaL, StreamRL) on external benchmarks (WebShop, AlfWorld, OSWorld, ScienceBoard), with fixed GPU budgets and swept static ratios for the asynchronous baselines. Goodput is defined operationally as "the number of generated tokens that are eventually consumed by the Trainer per second" (Section 6.1), but this is an explicit measurement choice rather than a hidden equation whose output equals its input; the reported speedups are observed quantities on a testbed, not consequences of the definition. The main self-citation, AgentRL [44], appears only as architectural inspiration ("Inspired by AgentRL [44], TIDERL adopts a disaggregated architecture"), and no load-bearing mathematical claim is justified solely by that citation. The Limitations section does concede that enforcing an anchored, frozen model version for evaluation "can temporarily disrupt the asynchronous pipeline momentum" and defers relay weight synchronization to future work; this is a genuine scope caveat about the zero-overhead migration claim for evaluation-phase tasks, but it is a correctness/coverage limitation rather than a circular reduction of the cited goodput numbers. No derivation in the paper reduces, by construction or by fitted-parameter renaming, to its own inputs, so no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical systems result, so the ledger is mostly about undisclosed tunables and domain assumptions. The key load-bearing assumptions are the stale-cache invalidation at weight-sync boundaries (which the paper's own limitation note partially contradicts) and the benign effect of policy version drift. No new physical entities are introduced.

free parameters (3)
  • CTB priority weights omega1-omega4 = not reported
    Section 4.2 defines P(t)=omega1*I_eval+omega2*G_completion+omega3*I_active+omega4*L_context. These weights control which tasks are paused and resumed; different choices change rollout throughput and cache-hit rate. No values are given, so the scheduler is under-specified.
  • ERS HoL threshold multiplier tau = not reported
    Section 5.2 triggers decoupled RA2P when Head-of-Line latency exceeds tau*T_step. This threshold sets the sensitivity of elastic scaling, and the paper does not report its value or a sensitivity analysis.
  • elastic batching bounds B_min and B_ideal = not reported
    Section 5.2 says the trainer consumes any batch size in [B_min, B_ideal]. These bounds determine how often the trainer updates and hence the training throughput, but their values are not disclosed.
assumptions (4)
  • domain assumption GRPO requires full trajectory groups before advantages can be computed.
    Section 2.1 defines advantage as standardization of rewards within a group, and CTB's priority hierarchy (Table 1) prioritizes near-complete groups. The scheduler's group-completion logic depends on this grouping constraint being load-bearing for reward correctness.
  • domain assumption On-policy weight updates make all existing rollout KV caches stale at the sync boundary.
    Section 5.2's cache-free migration relies on this to discard caches without transfer. The paper's own Limitations section notes that evaluation tasks require an anchored old model version, which this invalidation premise does not cover.
  • domain assumption Asynchronous policy version drift does not degrade convergence beyond measured small deficits.
    Section 7 'Algorithmic Effectiveness' cites APRIL and AReaL rather than providing a proof; the system's throughput gain depends on this benign-drift assumption being true for GRPO.
  • domain assumption The analytical mode-selection cost model in Appendix A accurately ranks decoupled vs colocated latency under RA2P.
    RA2P and ERS choose execution modes based on Cdec vs Ccol; if the profiled constants or the pipeline-fill formula mis-estimate real latencies, the mode choice will be wrong, though this would degrade performance rather than invalidate the whole system.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling." pith.science (2026). https://pith.science/paper/ZLSQVPEC

@misc{pith2026260810402,
  author       = {Pith},
  title        = {Pith review of: TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLSQVPEC}},
  note         = {Machine review of arXiv:2608.10402}
}
abstract

Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, $\textrm{RA}^2\textrm{P}$ selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6$\times$ over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58$\times$, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.

Figures

Figures reproduced from arXiv: 2608.10402 by the authors.

Figure 1
Figure 1. The dataflow and workflow of different RL strate￾gies on agentic tasks. length, and wall-clock duration, producing workloads with ex￾treme variance across both spatial and temporal dimensions. To enhance these autonomous capabilities, researchers turn to reinforcement learning (RL). While RL has shown im￾mense success in single-turn reasoning tasks [11, 27, 28], applying it to multi-turn agentic scenarios requires o… view at source ↗
Figure 2
Figure 2. Multi-turn agentic tasks experience severe cache misses in one WebShop training step. The data is collected on one NVIDIA H100 GPU running Qwen2.5-7B [24]. multi-turn trajectories is sampled. Each trajectory may comprise different turns and lengths of messages. • Reward Calculation: Once the trajectories in a group are completed, the environment evaluates them. GRPO determines the advantage Ai by standardizing the r… view at source ↗
Figure 3
Figure 3. Trainer experiences significant stalls and thrashing due to long-tail tasks. Even without CPU offloading, model alternation still incurs non-negligible GPU context-switching overhead, as shown in §6.5 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Rollout Wait Time (RWT) and Training Wait Time (TWT) during the first 100 steps under varying frac￾tion of GPUs assigned to Rollout. These waits expose the two readiness signals that drive RA2P and ERS. The experiment is carried out on one node with 8 NVIDIA H100 GPUs,…
Figure 5
Figure 5. Figure 5: The overall architecture and workflow of TIDERL. Environments and reward computations are disaggregated to an external CPU cluster. The system features a continuous data flow orchestrated by CTB on Rollout ranks, RA2P on Trainer nodes, and an overarching ERS scheduler.…
Figure 6
Figure 6. Figure 6: The RA2P pipeline for decoupled streaming mode. The loss computation for Actor ranks is moved from the forward pass into the backward pass. TIDERL uses RAS (startup ready backlog) and TPRM (later ready interval) in two layers. First, RA2P provides two Ref-Actor executi…
Figure 7
Figure 7. Figure 7: The ERS architecture. The ERS coordinator dynam￾ically reallocates GPU resources between functional roles. while Bideal represents the full utilization of GPU HBM and computing resources. ERS generates scaling plans immedi￾ately after the Trainer finishes an Actor upda…
Figure 9
Figure 9. Figure 9: The BoN reward and pass rate for different models on text-only tasks [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: TIDERL achieves the highest training throughput on multi-modal tasks across four RL frameworks. The other asynchronous baselines more often fall back to one-step-stale data, which explains why TIDERL is closer to the synchronous learning curve while retaining much hig…
Figure 12
Figure 12. Figure 12: CTB beats F10 and F9 on both metrics [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 14
Figure 14. Figure 14: RA2P (CM, DM) has higher throughput under ready micro-batch execution [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 34 canonical work pages

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    arXiv 2025

  2. [2]

    Concur: High-throughput agentic batch inference of llm via congestion-based concurrency control.arXiv preprint arXiv:2601.22705, 2026

    Qiaoling Chen, Zhisheng Ye, Tian Tang, Peng Sun, Boyu Tian, Guoteng Wang, Shenggui Li, Yonggang Wen, Zhenhua Han, and Tianwei Zhang. Concur: High-throughput agentic batch inference of llm via congestion-based concurrency control.arXiv preprint arXiv:2601.22705, 2026

    arXiv 2026

  3. [3]

    Fast llm post-training via decoupled and fastest-of-n speculation.arXiv preprint arXiv:2511.16193, 2025

    Rongxin Cheng, Kai Zhou, Xingda Wei, Siyuan Liu, Mingcong Han, Mingjing Ai, Yeju Zhou, Baoquan Zhong, Wencong Xiao, Rong Chen, and Haibo Chen. Fast llm post-training via decoupled and fastest-of-n speculation.arXiv preprint arXiv:2511.16193, 2025

    arXiv 2025

  4. [4]

    Mindspeed rl: Distributed dataflow for scalable and efficient rl training on ascend npu cluster.arXiv preprint arXiv:2507.19017, 2025

    Laingjun Feng, Chenyi Pan, Xinjie Guo, Fei Mei, Ben- zhe Ning, Jianxiang Zhang, Xinyang Liu, Beirong Zhou, Zeng Shu, Chang Liu, Guang Yang, Zhenyu Han, Jiang- ben Wang, and Bo Wang. Mindspeed rl: Distributed dataflow for scalable and efficient rl training on ascend npu cluster.arXiv preprint arXiv:2507.19017, 2025

    arXiv 2025

  5. [5]

    AREAL: A large-scale asynchronous reinforcement learning system for language reasoning

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, WANG JIASHU, Tongkai Yang, Binhang Yuan, and Yi Wu. AREAL: A large-scale asynchronous reinforcement learning system for language reasoning. InThe Thirty- ninth Annual Conference on Neural Information Pro- cessing Systems, 2025

    work page 2025

  6. [6]

    Cost-Efficient large language model serving for multi-turn conversations with CachedAtten- tion

    Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. Cost-Efficient large language model serving for multi-turn conversations with CachedAtten- tion. In2024 USENIX Annual Technical Conference (USENIX ATC 24), pages 111–126, Santa Clara, CA, July 2024. USENIX Association

    work page 2024

  7. [7]

    An empirical study on low gpu utilization of deep learn- ing jobs

    Yanjie Gao, Yichen He, Xinze Li, Bo Zhao, Haoxiang Lin, Yoyo Liang, Jing Zhong, Hongyu Zhang, Jingzhou Wang, Yonghua Zeng, Keli Gui, Jie Tong, and Mao Yang. An empirical study on low gpu utilization of deep learn- ing jobs. InProceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering, ICSE ’24, New York, NY , USA, 2024. Associatio...

    work page 2024

  8. [8]

    Better & faster large language models via multi-token prediction.arXiv preprint arXiv:2404.19737, 2024

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction.arXiv preprint arXiv:2404.19737, 2024

    arXiv 2024

Show all 61 references
  1. [9]

    Elasticflow: An elastic server- less training platform for distributed deep learning

    Diandian Gu, Yihao Zhao, Yinmin Zhong, Yifan Xiong, Zhenhua Han, Peng Cheng, Fan Yang, Gang Huang, Xin Jin, and Xuanzhe Liu. Elasticflow: An elastic server- less training platform for distributed deep learning. In Proceedings of the 28th ACM International Conference on Archite...

    2023

  2. [10]

    Asyncflow: An asynchronous streaming rl framework for efficient llm post-training.arXiv preprint arXiv:2507.01663, 2025

    Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, and Jianping Wu. Asyncflow: An asynchronous streaming ...

    2025 arXiv

  3. [11]

    OpenRLHF: A ray-based easy-to-use, scal- able and high-performance RLHF framework

    Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Wenkai Fang, Xianyu, Yu Cao, Haotian Xu, and Yiming Liu. OpenRLHF: A ray-based easy-to-use, scal- able and high-performance RLHF framework. In Ivan Habernal, Peter Schula...

    2025

  4. [12]

    Gpipe: Efficient training of giant neural networks us- ing pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. Gpipe: Efficient training of giant neural networks us- ing pipeline parallelism. In H. Wallach, H. Larochelle, A. Beygelzimer, F...

    2019

  5. [13]

    Elastic resource sharing for distributed deep learning

    Changho Hwang, Taehyun Kim, Sunghyun Kim, Jinwoo Shin, and KyoungSoo Park. Elastic resource sharing for distributed deep learning. In18th USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 21), pages 721–739. USENIX Association, April 2021

    2021

  6. [14]

    Efficient memory man- agement for large language model serving with page- dattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory man- agement for large language model serving with page- dattention. InProceedings of the 29th Symposium on Operating Systems Principles...

    2023

  7. [15]

    Con- tinuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live.arXiv preprint arXiv:2511.02230, 2026

    Hanchen Li, Qiuyang Mang, Runyuan He, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Hangrui Zhou, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. Con- tinuum: Efficient and robust multi-turn llm agent scheduling with kv cache time-to-live.arXiv preprint arXiv:2511.02230, 2026

    2026 arXiv

  8. [16]

    Chimera: efficiently training large-scale neural networks with bidirectional pipelines

    Shigang Li and Torsten Hoefler. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. InProceedings of the International Con- ference for High Performance Computing, Networking, Storage and Analysis, SC ’21, New York, NY , USA,

  9. [17]

    Agentbench: Evaluating LLMs as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:...

    2024

  10. [18]

    Visualagent- bench: Towards large multimodal models as visual foun- dation agents

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Song XiXuan, Yifan Xu, Shudan Zhang, Hanyu Lai, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhe...

    2025

  11. [19]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. Pipedream: gen- eralized pipeline parallelism for dnn training. InPro- ceedings of the 27th ACM Symposium on Operating Systems Princip...

    2019

  12. [20]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters u...

    2021

  13. [21]

    Suffixdecoding: Extreme speculative decod- ing for emerging AI applications

    Gabriele Oliaro, Zhihao Jia, Daniel F Campos, and Au- rick Qiao. Suffixdecoding: Extreme speculative decod- ing for emerging AI applications. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  14. [22]

    Ganger, and Eric P

    Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subra- manya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gre- gory R. Ganger, and Eric P. Xing. Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learn- ing. In15th USENIX Symposium on Operating Sys- tems Design and Impl...

    2021

  15. [23]

    Seer: Online con- text learning for fast synchronous llm reinforcement learning.arXiv preprint arXiv:2511.14617, 2025

    Ruoyu Qin, Weiran He, Weixiao Huang, Yangkun Zhang, Yikai Zhao, Bo Pang, Xinran Xu, Yingdi Shan, Yongwei Wu, and Mingxing Zhang. Seer: Online con- text learning for fast synchronous llm reinforcement learning.arXiv preprint arXiv:2511.14617, 2025

    2025 arXiv

  16. [24]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Day- iheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, 14 Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Ke- qin Bao, Kexin...

    2025 arXiv

  17. [25]

    Qwen3.5: Towards native multimodal agents

    Qwen Team. Qwen3.5: Towards native multimodal agents. https://qwen.ai/blog?id=qwen3.5, Febru- ary 2026

    2026

  18. [26]

    Transcending cost-quality tradeoff in agent serving via session-awareness

    Yanyu Ren, Li Chen, Dan Li, Xizheng Wang, Zhiyuan Wu, Yukai Miao, and Yu Bai. Transcending cost-quality tradeoff in agent serving via session-awareness. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  19. [27]

    Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  20. [28]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  21. [29]

    Laminar: A scalable asynchronous rl post-training framework.arXiv preprint arXiv:2510.12633, 2025

    Guangming Sheng, Yuxuan Tong, Borui Wan, Wang Zhang, Chaobo Jia, Xibin Wu, Yuqi Wu, Xiang Li, Chi Zhang, Yanghua Peng, Haibin Lin, Xin Liu, and Chuan Wu. Laminar: A scalable asynchronous rl post-training framework.arXiv preprint arXiv:2510.12633, 2025

  22. [30]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. InProceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297, New York...

    2025

  23. [31]

    ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. InProceed- ings of the International Conference on Learning Representations (ICLR), 2021

    2021

  24. [32]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Inform...

    2020

  25. [33]

    Scienceboard: Evaluating multimodal au- tonomous agents in realistic scientific workflows.arXiv preprint arXiv:2505.19897, 2026

    Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Xiangru Tang, Tianbao Xie, Xiachong Feng, Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi, Lingpeng Kong, and Zhiyong Wu. Scien...

    2026 arXiv

  26. [34]

    dist_checkpointing package

    Megatron Team. dist_checkpointing package. https://docs.nvidia.com/megatron-core/ developer-guide/latest/api-guide/core/ dist_checkpointing.html. [Accessed 13-04-2026]

    2026

  27. [35]

    Efficient llm serving for agentic workflows: A data systems per- spective.arXiv preprint arXiv:2603.16104, 2026

    Noppanat Wadlom, Junyi Shen, and Yao Lu. Efficient llm serving for agentic workflows: A data systems per- spective.arXiv preprint arXiv:2603.16104, 2026

    2026

  28. [36]

    ByteCheckpoint: A unified checkpointing system for large foundation model development

    Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, Xin Liu, and Chuan Wu. ByteCheckpoint: A unified checkpointing system for large foundation model development. In22nd USENIX Symposium on Networked Syst...

    2025

  29. [37]

    Infinite sampling: Ef- ficient and stable grouped rl training for large language models.arXiv preprint arXiv:2506.22950, 2025

    Liangyu Wang, Huanyi Xie, Xinhai Wang, Tianjin Huang, Mengdi Li, and Di Wang. Infinite sampling: Ef- ficient and stable grouped rl training for large language models.arXiv preprint arXiv:2506.22950, 2025

    2025 arXiv

  30. [38]

    AntMan: Dynamic scaling on GPU clus- ters for deep learning

    Wencong Xiao, Shiru Ren, Yong Li, Yang Zhang, Pengyang Hou, Zhi Li, Yihui Feng, Wei Lin, and Yangqing Jia. AntMan: Dynamic scaling on GPU clus- ters for deep learning. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 533–548. USENIX Assoc...

    2020

  31. [39]

    OSWorld: Benchmarking multimodal agents for open-ended tasks in real com- puter environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhou- jun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yi- heng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents fo...

    2024

  32. [40]

    SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends - LMSYS Blog — lmsys.org

    Zhiqiang Xie. SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends - LMSYS Blog — lmsys.org. https://www.lmsys.org/blog/ 2025-09-10-sglang-hicache/ , 2025. [Accessed 13- 04-2026]. 15

    2025

  33. [41]

    AndroidLab: Training and systematic benchmarking of android autonomous agents

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. AndroidLab: Training and systematic benchmarking of android autonomous agents. In Wanx- iang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar...

    2025

  34. [42]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Infor- mation Processing Systems, vo...

    2022

  35. [43]

    Orca: A distributed serving system for Transformer-Based generative mod- els

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soo- jeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative mod- els. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July 20...

    2022

  36. [44]

    Agentrl: Scaling agentic reinforcement learning with a multi-turn, multi-task framework.arXiv preprint arXiv:2510.04206, 2025

    Hanchen Zhang, Xiao Liu, Bowen Lv, Xueqiao Sun, Bo- hao Jing, Iat Long Iong, Zhenyu Hou, Zehan Qi, Hanyu Lai, Yifan Xu, Rui Lu, Hongning Wang, Jie Tang, and Yuxiao Dong. Agentrl: Scaling agentic reinforcement learning with a multi-turn, multi-task framework.arXiv preprint arXi...

  37. [45]

    Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large lan- guage models

    Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang, and Xin Jin. Disttrain: Addressing model and data heterogeneity with disaggregated training for multimodal large lan- guage models. InProceedings of the ACM SIGCOMM 2025 Conferenc...

    2025

  38. [46]

    Py- torch fsdp: Experiences on scaling fully sharded data parallel.arXiv preprint arXiv:2304.11277, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Sho- janazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Py- torch fsdp: Experiences o...

    2023 arXiv

  39. [47]

    Gonzalez, Clark Bar- rett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Bar- rett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In A. Globerson, L. Mackey, D. B...

    2024

  40. [48]

    Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Dist- Serve: Disaggregating prefill and decoding for goodput- optimized large language model serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2...

    2024

  41. [49]

    Streamrl: Scalable, het- erogeneous, and elastic rl for llms with disaggregated stream generation.arXiv preprint arXiv:2504.15930, 2025

    Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, and Daxin Jiang. Streamrl: Scalable, het- erogeneous, and elastic rl for llms with disaggregated stream generation.arXiv p...

    2025 arXiv

  42. [50]

    Optimizing RLHF training for large language models with stage fusion

    Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, and Xin Jin. Optimizing RLHF training for large language models with stage fusion. In22nd USENIX Symposium on Networked Systems Design and Implementation (...

    2025

  43. [51]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neu- big. Webarena: A realistic web environment for building autonomous agents. InNeurIPS 2023 Foundation Mod- els for Decision Ma...

    2023

  44. [52]

    April: Active partial rollouts in rein- forcement learning to tame long-tail generation.arXiv preprint arXiv:2509.18521, 2025

    Yuzhen Zhou, Jiajun Li, Yusheng Su, Gowtham Ramesh, Zilin Zhu, Xiang Long, Chenyang Zhao, Jin Pan, Xi- aodong Yu, Ze Wang, Kangrui Du, Jialian Wu, Ximeng Sun, Jiang Liu, Qiaolin Yu, Hao Chen, Zicheng Liu, and Emad Barsoum. April: Active partial rollouts in rein- forcement lear...

    2025

  45. [53]

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. Megascale- infer: Efficie...

    2025

  46. [54]

    slime: An llm post-training framework for rl 16 scaling

    Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contrib- utors. slime: An llm post-training framework for rl 16 scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv. Appendix A Analytical Model for RA 2P Selection To decide the optimal exe...

    2025

  47. [57]

    Wait for Data:If the current time T<a j, the system idles until the next micro-batch arrives:T←a j

  48. [58]

    Let k be the number of batches such that their arrival time a j+k−1≤T, capped by the maximum memory capacity

    Determine Aggregation (k):The scheduler counts all ready micro-batches that have arrived by time T . Let k be the number of batches such that their arrival time a j+k−1≤T, capped by the maximum memory capacity. 17

  49. [59]

    The clock updates: T←T+t swap +k·tre f

    Execute Ref Model:The system incurs a swap overhead, then sequentially processes k reference passes. The clock updates: T←T+t swap +k·tre f

  50. [60]

    The clock updates: T←T+t swap +k·(tact_f +tact_b)

    Execute Actor Model:The system incurs another swap, followed by the Actor’s forward and backward passes. The clock updates: T←T+t swap +k·(tact_f +tact_b)

  51. [61]

    Ccol evaluates to the final clock timeT

    Advance Index:The pointer advances ( j←j+k ) and the cycle repeats until j>N. Ccol evaluates to the final clock timeT. Qualitative Mode Analysis.Based on the formulated cost models, the selection between the two execution modes hinges on both N0 and the post-start ready gaps. ...

  52. [2021]

    Association for Computing Machinery

  53. [2025]

    Association for Computational Linguistics

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.