Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Dynamic GPU reallocation between rollout and training cuts LLM post-training time by a third.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:09 UTC pith:OOWRA376

load-bearing objection A promising systems idea for dynamic GPU rebalancing in disaggregated RL post-training, but the headline numbers rest on a single synthetic 8-GPU workload and the 'preserves RL semantics' claim is unsupported. the 4 major comments →

arxiv 2607.22614 v1 pith:OOWRA376 submitted 2026-06-15 cs.AI

DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training

Hanlin Du , Zhiyuan Yan , Haiquan Chen , Jiarui Fang , Yungang Bao , Sa wang This is my paper
classification cs.AI
keywords dynamic resource allocationLLM post-trainingreinforcement learningpipeline bubblesGPU disaggregationelastic traininghysteresis control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that the pipeline bubbles in RL-based LLM post-training—where variable generation lengths leave training GPUs idle—are a physical resource mismatch, not an algorithmic flaw. It argues that GPUs can be dynamically switched between Rollout and Training at runtime without altering reinforcement learning semantics, by decomposing resizing into fine-grained, overlapped operations rather than a monolithic stop-and-rebuild. If true, this provides a system-level remedy that avoids the compromises of truncating rollouts or tolerating stale policies, and it composes with those algorithmic methods. The central evidence is a 66.5% improvement in end-to-end throughput and a 33% reduction in total execution time relative to the optimal static configuration, with 27% of switching overhead hidden.

Core claim

DynaResize demonstrates that resource resizing in disaggregated RL post-training can be made a lightweight runtime operation. It decomposes role switching into fine-grained steps, then overlaps, prewarms, or defers the expensive ones: communication metadata is cached and communicators prewarmed off the critical path; model weights and optimizer states stream through bounded host-memory buffers; and optimizer-state loading is deferred until after the first training step on the new topology. A hysteresis-based controller triggers resizing only under sustained imbalance. In experiments on an 8-GPU cluster with a high-variance prompt mix, this yields 66.5% higher end-to-end throughput and 33% lo

What carries the argument

The load-bearing mechanism is the fully-overlapping re-initialization module, which governs three data-plane strategies: communicator reuse (caching topology-scoped communication metadata and prewarming inter-role communicators), staged state reloading (chunked streaming of weights and optimizer states through host memory with deferred optimizer materialization), and hysteresis-guided reconfiguration (dwell-time, cooldown, and consecutive-signal filters). Together these convert a role switch from a global barrier into a sequence of operations whose non-critical components run in the background, reducing the critical-path time from 255 seconds to 187 seconds in the reported resize.

Load-bearing premise

The claim that DynaResize preserves RL semantics rests on the assumption that deferred optimizer-state loading and the mixed resize step produce exactly the same PPO updates that a static schedule would produce; if the optimizer state is incomplete during the first post-resize step, or the mixed step alters gradients, final model quality could diverge even if throughput improves.

What would settle it

Run a fixed-seed RL post-training job under DynaResize and under a static optimal split, and compare the model's reward curves and the exact optimizer moments (e.g., Adam state) after a resize; if the first trained step after a resize uses a partially loaded optimizer state, gradients will differ and the final model weights will diverge. A concrete check: log the optimizer state tensors immediately before and after the resize-triggered mixed step and verify bitwise equality with a no-resize baseline.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Runtime GPU reallocation can replace static partitioning in disaggregated RL post-training, eliminating most long-tail rollout bubbles without changing the RL algorithm.
  • Because the approach preserves RL semantics, it can be combined with algorithmic mitigations like truncation or asynchronous updates to handle extreme long-tail workloads.
  • With a break-even point of about 11 sustained steps, resizing pays for itself quickly in typical 500–5000 step post-training runs.
  • The hysteresis controller's >10-step cooldown ensures that resizing decisions are only made when imbalances persist, preventing thrashing that would negate gains.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'semantics-preserving' claim hinges on deferred optimizer loading and the mixed resize step producing exactly the PPO updates a static schedule would; a direct comparison of final model quality or optimizer-state equality would be a decisive test that the paper does not report.
  • The ~11-step break-even suggests a practical production rule: only resize when a directional imbalance has persisted longer than that threshold, which the hysteresis controller already enforces.
  • The host-memory staging pattern could generalize to other stateful elastic services (e.g., mixed training/inference clusters) that cannot afford full checkpoint round-trips.
  • One could sweep the hysteresis parameters (dwell time, cooldown, consecutive signals) on real long-tail traces to tune the tradeoff between responsiveness and thrashing in settings with different variance profiles.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DynaResize, a runtime system for dynamically reallocating GPUs between Rollout and Training in disaggregated LLM post-training. It decomposes the resize operation into overlappable fine-grained steps, using communicator reuse, bounded host-memory state staging with deferred optimizer-state loading, and hysteresis-based triggering to avoid thrashing. The authors report a 66.5% end-to-end throughput improvement and 33% execution-time reduction over the optimal static configuration on an 8×H20 cluster with Qwen3-8B, while hiding 27% of role-switching overhead. The central claim is that these gains are obtained without changing RL semantics, unlike truncation or async-RL alternatives.

Significance. If the semantics-preservation claim and throughput numbers hold, DynaResize would be a valuable system-level complement to algorithmic fixes for long-tail rollout imbalance, with a clear design: separating control and data planes, prewarming communicators, and staging parameters through host memory. The paper deserves credit for direct step-time measurements, a concrete overhead breakdown, and an amortization analysis. However, the central claim that the RL updates are unchanged is not substantiated, and the evaluation is too narrow—one synthetic workload, one 8-GPU configuration, no error bars—to support the headline quantitative conclusions.

major comments (4)
  1. [§3.3 and §5 (Exp#1, Fig. 4(b))] The paper's central differentiator is that DynaResize improves utilization 'without changing RL semantics' (Abstract, §1, Conclusion), but the manuscript provides no evidence for this. Fig. 4(b) describes a 'mixed step' executing '2 rollouts & 1 train'; if the static baseline consumes one rollout batch per PPO update, consuming two in a resize step changes the gradient and surrogate objective. §3.3 defers optimizer-state loading until after the first training step, but gives no timing guarantee that the Adam moments are materialized before the update is computed; otherwise the first update uses stale or partial moments. There is no comparison of final model quality, reward curves, or per-update state equality between DynaResize and a static schedule. This is load-bearing: without it, the gains may come from altering the RL algorithm, which is exactly the compromise the paper claims to av
  2. [§5 Experimental Setup / Exp#1] The headline numbers (66.5% throughput, 33% time reduction) rest on a single synthetic workload, one 8×H20 configuration, and a single normalized 100-step run with no error bars or repeated trials. Step times in Fig. 4(b) show one resize event; the 'optimal static' baseline is not defined (how was the 6:2 configuration selected, and was the space of static partitions searched?). Without multiple workload traces, seeds, and static-config selection details, the end-to-end claim cannot be separated from workload-specific tuning.
  3. [§3.4 and §5 (Exp#3)] The hysteresis cooldown interval ('>10 steps') is chosen directly from the measured ~11-step break-even in Exp#3, so the claim that the controller 'guarantee[s] strictly positive gains' is circular: the threshold is tuned on the same workload used to evaluate it. Please provide sensitivity analysis varying the cooldown and imbalance thresholds, and separate tuning from test workloads.
  4. [§5 (Exp#2, Fig. 6)] DynaResize reduces critical-path reconfiguration from 255 s to 187 s (27%), but 187 s is still about 7.8 post-resize step times (23.93 s). The paper does not explain how multiple resizes in a longer job interact with the amortization bound, nor how a 100-step normalized run containing one 187 s blocking event can be consistent with a 33% total-time reduction unless the pre-resize bubble dominates. Please report the full timeline (number of resizes, placement of events, total measured time) for the end-to-end experiment.
minor comments (5)
  1. [Abstract / §5] 'normalized to a 100-step baseline' (Abstract) and 'normalized 100-step run' (Exp#1) are ambiguous; specify whether total time is measured directly or extrapolated.
  2. [Fig. 4(a)] The label 'Imbalance (23.7s)' is unclear; define what the 23.7 s interval represents and how it is computed.
  3. [§3.3] The statement that optimizer states 'are only required at the end of a training step' needs a precise ordering argument: in Adam, moments are read at update time, so clarify the scheduling relative to the deferred load.
  4. [References] References [2] and [34] both appear to describe RLinf; if they are distinct works, please disambiguate. Also standardize the spelling of veRL/verl.
  5. [Fig. 3 and Fig. 5] The color/hatching schemes are hard to distinguish in the PDF; adding textual callouts or a clearer legend would improve readability.

Circularity Check

1 steps flagged

Central throughput claim is empirically grounded; the only built-in feedback loop is the hysteresis cooldown, which is set to exceed the measured break-even.

specific steps
  1. fitted input called prediction [§3.4 (Hysteresis-Guided Reconfiguration) and §5 Exp#3 (Cost amortization and Thrashing boundaries)]
    "Our evaluation identifies a break-even point of approximately 11 sustained steps. This brief amortization window ensures positive throughput gains in following steady steps whenever train-rollout imbalances persist beyond a short period. ... To prevent this, our Hysteresis-Guided controller enforces a >10-step cooldown interval, safely exceeding the amortization threshold."

    The >10-step cooldown is chosen to 'safely exceed' the ~11-step break-even measured in the same experiment. The subsequent claim that resizing is triggered only during 'sustained directional imbalances' and can 'guarantee strictly positive gains' therefore follows from the parameter choice rather than from an independent predictive test of the hysteresis policy. This is a localized tuning loop, not the central derivation: the 66.5%/33% throughput and 27% overhead-hiding results are direct measurements and do not reduce to this fitted threshold.

full rationale

No equation-level circularity is present: the paper's main claims are empirically measured (per-step timings, resize latency breakdown, end-to-end execution time) rather than derived from fitted parameters or self-citations. The central throughput claim does not reduce to any input by construction. The one self-confirming element is the hysteresis cooldown, taken from the measured ~11-step amortization point and then used to claim guaranteed positive gains; this is localized and non-central. The more serious concern, that the mixed step (2 rollouts + 1 train) and deferred optimizer-state restore may alter PPO updates despite the paper's 'without changing RL semantics' assertion, is an unverified correctness/evidence gap rather than circularity: no equation equates the claimed semantics preservation to the design, and no update-equality or convergence experiment is reported. That risk should be weighed separately from circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

No code or data is provided; central claims rest on a single synthetic 8-GPU workload and on unverified assumptions about RL-semantics preservation and framework behavior.

free parameters (4)
  • Hysteresis cooldown interval = >10 steps
    Set after measuring a ~11-step break-even in Exp#3; controls how often resizing is allowed and supports the 'strictly positive gains' claim.
  • Hysteresis imbalance thresholds / dwell-time / consecutive-signal requirements = unspecified
    Controller filters resize triggers but exact values are not reported in §3.4 or §5; they are tuned to prevent thrashing on the test workload.
  • State-staging chunk size = unspecified
    Chunked host-memory streaming bounds migration memory (§3.3); chunk size affects transfer latency and overlap but is not reported.
  • Pre-specified candidate topology set (e.g., 6:2 to 4:4) = pre-specified elastic configurations
    The controller only chooses among pre-specified partitions; the set and granularity are hand-chosen and not justified.
axioms (4)
  • domain assumption Pipeline imbalance in disaggregated RL post-training is primarily a physical resource mismatch rather than an algorithmic flaw (Section 2, final paragraph).
    Load-bearing: if imbalance is instead dominated by algorithmic factors such as scheduling or batching, GPU reallocation cannot produce the claimed gains.
  • domain assumption Deferred optimizer loading and mixed steps during resize yield the same PPO updates as a static schedule (§3.3, §5 Exp#1).
    No convergence or update-equality evidence is provided; the 'without changing RL semantics' claim relies on this unverified equivalence.
  • domain assumption Ray placement groups can be retained while actor roles change and communication topology is rebuilt without framework-level corruption (§4).
    The implementation depends on this property of Ray/veRL; the paper asserts it but ships no code to verify it.
  • domain assumption Standard NCCL/PyTorch collective semantics and host-memory staging preserve distributed state consistency across GPU role switches (§3.2-3.3).
    Background library behavior is assumed correct; no independent validation is provided.
invented entities (3)
  • DynaResize engine / Fully-overlapping Re-init Module no independent evidence
    purpose: Orchestrates overlapping of state export, worker re-init, communicator pre-warm, and deferred optimizer reload during role switch.
    Described in §3.1 and Figure 3; no released implementation or separate benchmark certifies it.
  • Communicator Manager topology-scoped cache no independent evidence
    purpose: Caches rank mappings and group membership so inter-role communicators can be pre-warmed off the critical path.
    Described in §3.2; caching behavior is not independently tested outside the paper.
  • Param Staging Manager / bounded host-memory streaming no independent evidence
    purpose: Transfers model weights and optimizer states chunk-by-chunk through host memory to avoid OOM and hide migration latency.
    Described in §3.3; exact buffer bounds and transfer protocol are not specified.

pith-pipeline@v1.3.0-alltime-deepseek · 10088 in / 11400 out tokens · 117490 ms · 2026-08-02T11:09:48.112338+00:00 · methodology

0 comments
read the original abstract

RL-based LLM post-training increasingly disaggregates Rollout and Training across separate GPU resources, but static GPU partitioning suffers from severe pipeline bubbles under long-tail rollout latency. We present DynaResize, a runtime GPU reallocation system that dynamically switches GPUs between Rollout and Training to balance stage execution times without changing RL semantics. DynaResize decomposes resizing into fine-grained operations and removes non-startup-critical work from the critical path through communicator reuse, bounded state staging, and hysteresis-based resizing. Experimental results show that DynaResize can improve end-to-end throughput by 66.5% and reduce total execution time by 33% over the optimal static configuration, while hiding 27% of role-switching overhead.

Figures

Figures reproduced from arXiv: 2607.22614 by Haiquan Chen, Hanlin Du, Jiarui Fang, Sa Wang, Yungang Bao, Zhiyuan Yan.

Figure 1
Figure 1. Figure 1: Comparison of approaches for mitigating asynchronous training-rollout imbalance [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of DynaResize Save Old Checkpoint Release Old Worker Init Actor Init Rollout Load Weight Weight Broadcast Transfer Param to Host Mem Release Old Worker Init Actor Comm Pre-warm Weight Sync New Step Init Rollout Load Weight Lazy Load Optimizer New Step Training Continue Load Optimizer Create Comm Release GPU for new topology Parallel init Overlapped transfer On-demand lazy loading (1) Hard-resize, … view at source ↗
Figure 3
Figure 3. Figure 3: Fully-overlapping workflow of DynaResize compared to hard stop-and-rebuild resizing existing GPUs between the two stages on demand, reducing hardware idling during pipeline stalls while preserving the original algorithmic semantics. 3 DynaResize’s Design 3.1 Overview As illustrated in Fig.2, DynaResize adopts a decoupled control-data plane architecture centered around an abstraction layer that isolates phy… view at source ↗
Figure 4
Figure 4. Figure 4: (Exp#1) Time per step saved by dynamic resizing. −50 0 50 100 150 200 250 −50 0 50 100 150 200 250 Time relative to blocking export start (s) Rollout step cycle Train step cycle Blocking model export Actor re-init Rollout re-init Actor model restore Comm / post process Deferred optimizer restore step before resize step after resize export states to host actor re-init rollout re-init staged weight restore c… view at source ↗
Figure 5
Figure 5. Figure 5: (Exp#2) Time breakdown for DynaResize comparing to hard resize without on-demand reloading. 0.0 50.0 100.0 150.0 200.0 250.0 Timeline since critical path start (s) LiveResize ( on￾demand ) Hard resize ( stop & rebuild ) 60.7s 32.6s 45.8s 49.9s 187.2s 59.7s 60.3s 104.1s 29.6s 254.8s State export Worker re-init Weight restore & Comm pre-warm Resize postprocess Weight sync Rollout manager [PITH_FULL_IMAGE:fi… view at source ↗
Figure 6
Figure 6. Figure 6: (Exp#2) Time flow of DynaResize caching and overlapping. a variance prompt dataset by mixing short-form QA tasks with long-form reasoning and coding tasks, deliberately inducing severe generation long-tail stragglers. (Exp#1) End-to-End Performance. We first evaluate end-to-end performance under different degrees of workload variance, measured by the execution time per PPO step. Under high prompt variance,… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 15 linked inside Pith

  1. [1]

    2025.Training-Free Group Relative Policy Optimization

    Yuzheng Cai, Siqi Cai, et al. 2025.Training-Free Group Relative Policy Optimization. arXiv:2510.08191

    arXiv 2025

  2. [2]

    RLInf Community. 2026. RLinf: Reinforcement Learning Infrastructure for Embodied and Agentic AI. https://github.com/RLinf/RLinf

    2026

  3. [3]

    DeepSeek-AI. 2026. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. https://huggingface.co/deepseek-ai/DeepSeek-V4- Pro

    2026

  4. [4]

    2024.The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024.The Llama 3 Herd of Models. arXiv:2407.21783

    Pith/arXiv arXiv 2024

  5. [5]

    2025.Truncated Proximal Policy Optimization

    Tiantian Fan, Lingjun Liu, Yu Yue, Jiaze Chen, Chengyi Wang, Qiying Yu, Chi Zhang, Zhiqi Lin, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Bole Ma, Mofan Zhang, Gaohong Liu, Ru Zhang, Haotian Zhou, Cong Xie, Ruidong Zhu, Zhi Zhang, Xin Liu, Mingxuan Wang, Lin Yan, and Yonghui Wu. 2025.Truncated Proximal Policy Optimization. arXiv:2506.15050

    Pith/arXiv arXiv 2025

  6. [6]

    Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. 2025. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. arXiv:2505.24298 [cs.LG] https://arxiv.org/abs/2505.24298

    Pith/arXiv arXiv 2025

  7. [7]

    Wei Gao, Yuheng Zhao, Dakai An, Tianyuan Wu, Lunxi Cao, Shaopan Xiong, Ju Huang, Weixun Wang, Siran Yang, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng, and Wei Wang. 2026. RollPacker: Taming Long-Tail Rollouts for RL Post-Training with Tail Batching. In23rd USENIX Symposium on Networked Systems Design and Implementation (NSDI 26). USENIX Association, Renton, ...

    2026

  8. [8]

    2026.ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL

    Wei Gao, Yuheng Zhao, Dilxat Muhtar, Dakai An, Xuchun Shang, Tianyuan Wu, Lunxi Cao, Shaopan Xiong, Weixun Wang, Ju Huang, Teng Ma, Siran Yang, Jiamang Wang, Lin Qu, Bo Zheng, and Wei Wang. 2026.ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL. arXiv:2605.06534

    Pith/arXiv arXiv 2026

  9. [9]

    2024.Multi-Bin Batching for Increasing LLM Inference Throughput

    Ozgur Guldogan, Jackson Kunde, Kangwook Lee, and Ramtin Pedarsani. 2024.Multi-Bin Batching for Increasing LLM Inference Throughput. arXiv:2412.04504

    Pith/arXiv arXiv 2024

  10. [10]

    2025.AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

    Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, Huazhong Ji, Wenjie Liu, Yu Huang, Yixiang Zhang, Chenyi Pan, Jing Wang, Xin Huang, Chunsheng Li, and Jianping Wu. 2025.AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training. arXiv:2507.01663

    Pith/arXiv arXiv 2025

  11. [11]

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury. 2023. Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates. InProceedings of the 29th Symposium on Operating Systems Principles(Koblenz, Germany)(SOSP ’23). Association for Computing Machinery, New York, NY, USA, 382–395. doi:10.1145/3600006.3613152

    arXiv 2023

  12. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180

    Pith/arXiv arXiv 2023

  13. [13]

    2024.ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

    Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. 2024.ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation. arXiv:2406.14088

    Pith/arXiv arXiv 2024

  14. [14]

    Jordan, and Ion Stoica

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 561–57...

    2018

  15. [15]

    NVIDIA Corporation. 2026. NVIDIA Collective Communication Library (NCCL). https://github.com/NVIDIA/nccl. Optimized primitives for collective multi-GPU communication

    2026

  16. [16]

    Or et al

    A. Or et al. 2020. TorchElastic: Elastic and fault-tolerant distributed training. PyTorch Blog

    2020

  17. [17]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human f...

    2022

  18. [18]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High...

    2019

  19. [19]

    Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. 2018. Optimus: an efficient dynamic resource scheduler for deep learning clusters. InProceedings of the Thirteenth EuroSys Conference(Porto, Portugal)(EuroSys ’18). Association for Computing Machinery, New York, NY, USA, Article 3, 14 pages. doi:10.1145/3190508.3190517

    arXiv 2018

  20. [20]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. InAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 53728...

    2023

  21. [21]

    Rajbhandari et al

    S. Rajbhandari et al. 2020. ZeRO: Memory optimizations toward training trillion parameter models. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC20)

    2020

  22. [22]

    Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. 2021. ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learning. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis(St. Louis, Missouri)(SC ’21). Association for Computing Machinery, New York, NY, US...

    arXiv 2021

  23. [23]

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. ZeRO- Offload: Democratizing Billion-Scale Model Training. In2021 USENIX Annual Technical Conference (USENIX ATC 21). USENIX Association, 551–564. https://www.usenix.org/conference/atc21/presentation/ren-jie

    2021

  24. [24]

    2017.Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.Proximal policy optimization algorithms. arXiv:1707.06347

    Pith/arXiv arXiv 2017

  25. [25]

    Guangming Sheng, Yuxuan Tong, Borui Wan, Wang Zhang, Chaobo Jia, Xibin Wu, Yuqi Wu, Xiang Li, Chi Zhang, Yanghua Peng, Haibin Lin, Xin Liu, and Chuan Wu. 2026. Laminar: A Scalable Asynchronous RL Post-Training Framework. InProceedings of the 21st European Conference on Computer Systems(McEwan Hall/The University of Edinburgh, Edinburgh, Scotland UK)(EUROS...

    arXiv 2026

  26. [26]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. InProceedings of the Twentieth European Conference on Computer Systems(Rotterdam, Netherlands) (EuroSys ’25). Association for Computing Machinery, New York, NY, USA, 1279–1297. doi:10...

    arXiv 2025

  27. [27]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: high-throughput generative inference of large language models with a single GPU. InProceedings of the 40th International Conference on Machine Learning(Honolulu, Hawaii, USA)(ICML’23). JMLR.org, Article 1288,...

    2023

  28. [28]

    2019.Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019.Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053 https://github.com/NVIDIA/Megatron-LM

    Pith/arXiv arXiv 2019

  29. [29]

    2025.Kimi k1.5: Scaling Reinforcement Learning with LLMs

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, et al. 2025.Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599

    Pith/arXiv arXiv 2025

  30. [30]

    2025.Qwen3 Technical Report

    Qwen Team. 2025.Qwen3 Technical Report. arXiv:2505.09388

    Pith/arXiv arXiv 2025

  31. [31]

    John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing Harry Xu. 2023. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 497–513. https://www.usenix....

    2023

  32. [32]

    verl community. 2026. verl: Volcano Engine Reinforcement Learning for LLMs. https://github.com/verl-project/verl

    2026

  33. [33]

    2026.ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning

    Jie Xiao, Meng Chen, Qingnan Ren, Jingwei Song, Jiaqi Huang, Yangshen Deng, Chris Tong, Wanyi Chen, Suli Wang, Ziqian Bi, Shuo Lu, Yiqun Duan, Xu Wang, Rymon Yu, Ween Yang, Lynn Ai, Eric Yang, and Bill Shi. 2026.ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning. arXiv:2602.02192

    Pith/arXiv arXiv 2026

  34. [34]

    2025.RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation

    Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, Chunyang Zhu, Junhao Hu, et al. 2025.RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation. arXiv:2509.15965

    arXiv 2025

  35. [35]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for Transformer- Based Generative Models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 521–538. https://www.usenix.org/conference/osdi22/presentation/yu

    2022

  36. [36]

    2025.DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Qiying Yu et al. 2025.DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476

    Pith/arXiv arXiv 2025

  37. [37]

    2026.ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents

    Hao Zhang et al. 2026.ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents. arXiv:2603.18815

    arXiv 2026

  38. [38]

    Yiqi Zhang, Huiqiang Jiang, Xufang Luo, Zhihe Yang, Chengruidong Zhang, Yifei Shen, Yuqing Yang Dongsheng Li, Lili Qiu, and Yang You. 2026. SortedRL: Accelerating RL Training for LLMs through Online Length-A ware Scheduling. arXiv:2603.23414

    arXiv 2026

  39. [39]

    2023.PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    Yanli Zhao, Andrew Gu, Rohan Vasanth, Olatunji Ruwase, Jason Johnson, Ashkan Nayak, Ben Fineran, et al. 2023.PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. arXiv:2304.11277 https://pytorch.org/docs/stable/fsdp.html

    Pith/arXiv arXiv 2023

  40. [40]

    2025.APRIL: Active Partial Rollouts in Reinforcement Learning to tame long-tail generation

    Yuzhen Zhou, Jiajun Li, et al. 2025.APRIL: Active Partial Rollouts in Reinforcement Learning to tame long-tail generation. arXiv:2509.18521

    arXiv 2025

  41. [41]

    Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. 2025. slime: An LLM post-training framework for RL Scaling. https://github.com/ THUDM/slime. GitHub repository. Corresponding author: Xin Lv

    2025