Pith. sign in

REVIEW 3 major objections 5 minor 12 cited by

StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Disaggregated, stream-based RL training can reach 2.66x the throughput of colocated systems.

desk verdict A credible systems contribution that rehabilitates disaggregated RL with stream generation; the async convergence evidence is the main soft spot, but the sync variant keeps the central claim standing. read the letter →

arxiv 2504.15930 v1 pith:B27XQPW7 submitted 2025-04-22 cs.LG cs.DC

Yinmin Zhong , Zili Zhang , Xiaoniu Song , Hanpeng Hu , Chao Jin , Bingyang Wu , Nuo Chen , Yukun Chen
show 6 more authors
Yu Zhou Changyi Wan Hongyu Zhou Yimin Jiang Yibo Zhu Daxin Jiang
This is my paper · ORCID
classification cs.LGcs.DC
keywords reinforcementlearningLLMpost-trainingdisaggregatedarchitecturestreamgenerationasynchronousRLlong-tailschedulingoutputlengthrankercross-datacentertraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RL post-training for LLMs normally runs generation and training on the same GPUs, switching between them. The paper argues that this colocated arrangement couples the two stages to identical resources even though generation is limited by memory bandwidth and training is limited by compute, which hurts scaling and cost. Its proposed system, StreamRL, instead keeps generation in a separate stream generation service that sends each completed sample to the trainer immediately, and it adds an output-length ranker that isolates the long-tail samples that dominate generation time. The reported result is up to 2.66x throughput over state-of-the-art colocated systems, and up to 1.33x better throughput per dollar in a heterogeneous cross-datacenter setup. If correct, the paper shows disaggregation, not colocation, is the right default for large-scale RL training.

What carries the argument

The load-bearing pieces are the stream generation service (SGS), which accepts generation requests and pushes each completed sample to the trainer immediately; dynamic-batch pipelining, which lets the trainer start sample-level work as soon as enough samples arrive instead of waiting for the whole batch; fully asynchronous pipelining, which overlaps weight transmission with the next iteration by accepting one-step-stale weights; the output-length ranker, a small LLM fine-tuned to rank prompts by expected output length; and skewness-aware dispatching with longest-processing-time-first scheduling, which assigns the predicted long tail to dedicated, smaller-batch generation instances. A profiler-based resource allocator chooses GPU counts and parallel strategies so the two stages finish near-simultaneously, and an elastic adjustment mechanism adds data-parallel generation units when generation begins to lag training as output lengths grow.

What would settle it

Run the same RL task, dataset, and compute budget with synchronous pipelining and with one-step asynchronous pipelining on additional model scales (for instance 7B and 72B) and at least one non-math task, then compare final task accuracy and reward curves; if the asynchronous run finishes measurably below the synchronous one, the justification for the fully overlapped mode and the headline speedup collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that the conventional disaggregated architecture was abandoned too quickly: with streaming, a dedicated generation service and a dedicated trainer can overlap almost completely, making disaggregation faster than colocation instead of slower. The generation service returns each finished sample the moment it completes, so the trainer can begin reference-model inference, KL computation, and reward scoring on a rolling basis; in the synchronous setting this dynamic-batch pipelining eliminates most pipeline bubbles, and in the one-step asynchronous setting weight transmission drops off the critical path entirely. On top of this, an output-length ranker model predicts which prompts will generate the longest outputs, and skewness-aware dispatching puts those long-tail prompts on dedicated, small-batch instances while regular prompts fill large batches, shortening the generation tail that otherwise idles GPUs. The authors report that the synchronous design improves throughput by 1.06–1.41x over a matched colocated baseline and the asynchronous design by 1.30–2.66x; they also demonstrate that one-step asynchronous training produces reward curves close to synchronous training on a 32B model, which is the empirical basis for treating the fully overlapped mode as safe.

Load-bearing premise

The load-bearing premise is that training on samples generated with slightly stale policy weights is safe: the fully asynchronous variant, which contributes about 25% of the reported throughput gain, assumes one-step staleness does not hurt convergence or final model quality, and the paper validates this on a single 32B reward-curve study while noting that generality and theoretical guarantees are outside its scope.

Editorial extensions

If this is right

  • If the central claim is correct, colocation should no longer be treated as the default for RL post-training; disaggregation with streaming overtakes it once generation and training can overlap.
  • Generation and training can run on different hardware types, so operators can choose memory-bandwidth-optimized, cheaper GPUs for generation and compute-optimized GPUs for training, improving throughput per dollar.
  • Cross-datacenter RL training becomes practical: the inter-stage traffic is point-to-point and small, and the paper measures weight-transfer overhead below 2% of iteration time over a dedicated 80 Gbps link.
  • One-step asynchronous training, where samples are generated with slightly stale weights, can be treated as a system design lever rather than a threat to convergence; the paper's reward-curve comparison supports this for at least one 32B model.
  • Long-tail generation, the dominant cost in reasoning-model RL, can be attacked by predicting prompt difficulty and scheduling around it, rather than by replay buffers that change the output distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ranker-plus-split scheduling idea is portable: it does not depend on RL, so it should also reduce makespan in long-context LLM serving or any workload where a small fraction of requests produce very long outputs; the paper only evaluates it inside RL training.
  • If one-step-async safety generalizes, RL post-training becomes a pure streaming dataflow, and generation speed becomes the main optimization target; inference-side techniques like better prefill/decode separation would then directly raise RL throughput.
  • The elastic adjustment only adds generation instances because trainer parallelism is rigid; a cluster scheduler that can reallocate trainer-side resources without restart would extend the same balancing idea to whole-job resource sharing.
  • The 1.33x cost-effectiveness advantage is tied to the specific GPU price ratio and link bandwidth tested; because generation time grows faster than training time as outputs lengthen, the disaggregated advantage should widen as reasoning models generate longer chains of thought.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper revisits the disaggregated architecture for LLM reinforcement learning, arguing that the prevailing colocated design suffers from resource coupling. It presents StreamRL, which splits the RL pipeline into a stream generation service and a trainer, uses streaming to enable dynamic-batch and fully asynchronous pipelining, and adds an output-length ranker plus skewness-aware dispatching to mitigate long-tail generation bubbles. The paper also contributes profiler-based static resource allocation and a dynamic adjustment mechanism, and evaluates the system on Qwen2.5 models from 7B to 72B under single-datacenter and cross-datacenter settings, reporting up to 2.66x throughput over verl and up to 1.33x cost-effectiveness improvement in heterogeneous deployment.

Significance. If the results hold, the paper makes a valuable, contrarian argument: disaggregation, with careful streaming and scheduling, can beat colocation for RL training. The main strengths are the controlled ColocationRL baseline that shares the in-house inference and training engine, the staged ablation of Table 3, the held-out test split for the ranker, and the direct measurement of throughput rather than derivation from fitted models. The main risks are that the headline 2.66x number conflates engine-level optimizations with the architecture-level contribution, and that the fully asynchronous mode relies on convergence evidence from a single self-disclaimed 32B experiment. The core architecture idea is plausible and the paper is clearly written, but the load-bearing async and ranker-evaluation evidence needs strengthening.

major comments (3)
  1. [§7.1, Table 3, Abstract] The 2.66x throughput claim in the Abstract and §7.1 is measured against verl, a colocated framework using vLLM and Megatron, while StreamRL uses a different in-house inference and training engine; the text itself notes the speedup is 'partially attributed to optimizations in the underlying inference and training framework.' The architecture-specific gain, isolated by the ColocationRL baseline that shares StreamRL's engine, is 1.23x for StreamRL-Sync and 1.48x for StreamRL-Async in Table 3, while §7.1 reports 1.06x-1.41x for Sync versus ColocationRL. The paper should present the 2.66x as an end-to-end system comparison and separately state the architecture-only gain, rather than leaving the impression that 2.66x is the benefit of the disaggregated design.
  2. [§7.4, Table 3] The fully asynchronous variant contributes 25% of the normalized throughput gain (1.23 to 1.48 in Table 3) and is the basis of the 'fully overlapping' claim, but its convergence evidence is a single PPO reward-curve comparison on Qwen2.5-32B, and §7.4 explicitly states that 'its generality and theoretical guarantees are beyond the scope of this paper.' Moreover, the §7 protocol forces all frameworks to generate outputs at predetermined ground-truth lengths, so the reward curves are obtained under a stationary length distribution rather than the emergent, growing distribution that motivates StreamRL's elastic adjustment and skewness mechanisms. This is a load-bearing evidential gap: either add convergence experiments across model families/tasks or multiple seeds with natural length evolution, or downgrade the main contribution to the synchronous variant and present async as preliminary.
  3. [§5.2 and §7] Section 5.2 states that the ranker is trained on (prompt, output-length) pairs from the target LLM, but §7 describes the evaluation dataset as an internal CodeMath prompt set with responses collected from DeepSeek-R1 as ground truth, and says all frameworks are modified to generate outputs with the same lengths following that ground truth. The ranker's recall of 87% for the top 20% long-tail samples (Table 4) and the generation-time savings in §7.2 are therefore for predicting an external model's output lengths, not the target Qwen2.5 model's own generation distribution. The authors should either train and evaluate the ranker on the actual target model's generations, or explicitly justify DeepSeek-R1 lengths as a valid proxy and quantify the sensitivity of the results to this choice.
minor comments (5)
  1. [§1, §2.1] There are typos: 'colocation becames the prevailing choice' and 'In the constrast, other algorithms' should read 'became' and 'In contrast'.
  2. [§2.3] The sentence 'As training scales out, As training scales out, cross-datacenter training...' contains a duplicated phrase.
  3. [§6] 'desinated SGS DP instance' should read 'designated SGS DP instance.'
  4. [§7.1] The sentence 'StreamRL-Async further ... achieving 1.30x-2.66x throughput improvement' should explicitly state whether the baseline is verl or ColocationRL, since the preceding comparison is to ColocationRL while the 2.66x value is the headline result versus verl.
  5. [§7] Throughput in Figure 8 is reported as single numbers over 20 iterations after warm-up; reporting variance or multiple runs would help assess stability, especially for the async variant that depends on stage speed matching.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the throughput and cost-effectiveness claims are end-to-end measurements, and the learned/profiled components are validated on held-out or directly measured data rather than being substituted into the headline result.

full rationale

The paper's central claims—up to 2.66x throughput and 1.33x cost-effectiveness—are presented as measured end-to-end results from the experimental sections (§7.1 and §7.3), not as outputs of a fitted equation or self-referential derivation. The two model-based components are the output-length ranker (§5.2) and the profiler-based latency model (§4.2). The ranker is trained on (prompt, output-length) pairs and its quality is reported as recall on a held-out test split (§7.2, Table 4), and the scheduling benefit is separately compared against a random-dispatch baseline and an oracle upper bound (Figure 9). The profiler is used to choose resource allocations, and its effectiveness is validated by measured stage-latency balance (Figure 10), not by feeding its own predictions back into the headline throughput number. Equations (1) and (2) are latency models used for dispatch decisions, while the reported throughput is a direct measurement over 20 training iterations. Self-citations to DistServe [57], RLHFuse [58, 59], and MegaScale [22] are contextual related-work references; the paper does not invoke a self-authored uniqueness theorem or otherwise rely on prior work to rule out architectural alternatives. The explicit disclaimer in §7.4 that the generality and theoretical guarantees of asynchronous training are beyond the scope of this paper is a limitation on external validity, not evidence of circular reasoning. Overall, the derivation chain is self-contained with respect to the measured performance claims.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

The central efficiency claims rest on four fitted quantities (alpha threshold, P50/P90 estimates, dynamic threshold, profiler parameters), on the ranker model's accuracy, and on the assumption that one-step async RL does not hurt convergence. None of these are independently released with code or data, so the ledger is substantial but not circular.

free parameters (4)
  • Long-tail threshold alpha = 0.2
    Chosen empirically; Section 5.3 states setting alpha to 20 yields good results. It determines which prompts are classified as long-tail.
  • P50/P90 output length estimates = derived from distribution D, values not reported
    Used in Algorithm 2 to estimate average output lengths for regular and long-tail samples; Section 5.3.
  • Dynamic adjustment threshold = not specified
    Section 4.2 triggers elastic scale-out when delta exceeds a threshold; the threshold value is not given.
  • Profiler latency model parameters = not reported
    The profiler P estimates stage latencies as a function of GPUs and workload; parameters fitted to measurements on H800/H20 (Section 4.2, Section 5.1).
assumptions (6)
  • standard math LPT scheduling is a 4/3-approximation for makespan minimization
    Used in Section 5.3 to justify the scheduling order algorithm.
  • domain assumption Sample latency equals per-token latency times output length, with per-token latency a monotonic function of batch size
    Equation (1) and (2) in Section 5.1; abstracts away memory, scheduling, and communication effects.
  • domain assumption Generation is memory-bandwidth-bound and training is compute-bound, with distinct scaling behaviors
    Core motivation for disaggregation in Section 2.2; widely accepted in the field.
  • domain assumption Relative output length ranks can be predicted by a fine-tuned ranker
    Section 5.2 relies on the hypothesis that prompt difficulty is stable; supported by recall experiments on one internal dataset.
  • domain assumption One-step asynchronous RL preserves convergence and final performance
    Section 4.1 and 7.4 rely on this to claim full overlapping; the paper cites prior work and shows one reward-curve comparison, admitting generality is beyond scope.
  • domain assumption DNN execution time is deterministic enough for profiler-based prediction
    Section 4.2 cites Clockwork [17] to justify accurate stage latency modeling.
invented entities (1)
  • Output-length ranker model independent evidence
    purpose: Predicts which prompts will produce long outputs so that SGS can dispatch them to dedicated instances with smaller batch sizes
    Trained via SFT on prompt-length pairs from the internal dataset; recall is measured on a held-out test split (Table 4), providing a falsifiable handle within the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation." pith.science (2026). https://pith.science/paper/B27XQPW7

@misc{pith2026250415930,
  author       = {Pith},
  title        = {Pith review of: StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B27XQPW7}},
  note         = {Machine review of arXiv:2504.15930}
}
read the original abstract

Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs). RL for LLMs involves two stages: generation and training. The LLM first generates samples online, which are then used to derive rewards for training. The conventional view holds that the colocated architecture, where the two stages share resources via temporal multiplexing, outperforms the disaggregated architecture, in which dedicated resources are assigned to each stage. However, in real-world deployments, we observe that the colocated architecture suffers from resource coupling, where the two stages are constrained to use the same resources. This coupling compromises the scalability and cost-efficiency of colocated RL in large-scale training. In contrast, the disaggregated architecture allows for flexible resource allocation, supports heterogeneous training setups, and facilitates cross-datacenter deployment. StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching and scheduling. Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems, and improves cost-effectiveness by up to 1.33x in a heterogeneous, cross-datacenter setting.

Figures

Figures reproduced from arXiv: 2504.15930 by the authors.

Figure 1
Figure 1. Two representative RL framework architectures. In contrast to traditional next-token prediction [10, 11, 32, 45, 46] in pre-training, RL enables the LLMs to learn by trial and error from reward signals. While numerous RL algorithms exist, such as PPO [36] and GRPO [37], the typical RL workflow for LLMs involves two main stages in serial: generation and training. In generation stage, the LLM produces samples on a bat… view at source ↗
Figure 2
Figure 2. The performance sensitivity difference of the gen￾eration and training stage under resource quantities (left) and sequence length (right). replicas. Tensor Parallelism (TP) divides individual opera￾tions across multiple GPUs, with each GPU responsible for a portion of the computation. Due to its high communication overhead, TP is typically confined to intra-node deployment where high-speed interconnects like NVLINK … view at source ↗
Figure 4
Figure 4. StreamRL system architecture. We first present a high-level overview of the overall work￾flow of StreamRL. Next, we describe in detail our techniques and designs for addressing pipeline bubbles (§4) and skew￾ness bubbles (§5), as well as the implementation details of the communication between SGS and Trainer (§6). Workflow. Given the clusters, models, and algorithm config￾urations, StreamRL first determines how to a… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: How streaming powers existing solutions to better mitigate pipeline bubbles. 𝑊𝑖 denotes the parameter version. Strawman solution 1: Mini-batch pipelining [27]. In synchronous RL, weights update happens after all samples have been processed. As shown in [PITH_FULL_IMAG…
Figure 6
Figure 6. Figure 6: Left: The advantage of skewness-aware dispatch￾ing over random dispatching. Right: The trend of per-token decoding latency for a 7B LLM profiled on NVIDIA H800 with vLLM [23] as the batch size increases. the added DP unit, which is negligible relative to the overall RL…
Figure 7
Figure 7. Figure 7: The prompt and output length distribution of the evaluation dataset. 6 Implementation RL Training Framework. SGS employs an in-house infer￾ence engine implemented in C++ with optimized CUDA ker￾nels, supporting continuous batching [52] to release shorter samples early …
Figure 8
Figure 8. Figure 8: End-to-end throughput of RL training systems under different sequence length and model size settings. not yet particularly long. This leads to three datasets, which we denote as 5K, 10K, and 20K for clarity. In each iteration, we use a global batch size of 1024 followi…
Figure 10
Figure 10. Figure 10: The iteration time breakdown compared between even resource split and our resource allocation algorithm when training 32B and 72B model on the 20K dataset. 0 2 4 6 8 10 12 14 16 18 20 22 24 Iterations -10 0 10 20 30 Delta Time (s) 0 16 32 48 64 #GPUs [PITH_FULL_IMAGE…
Figure 12
Figure 12. Figure 12: The throughput normalized by the hardware cost between cross- and single-datacenter deployment. match that of the synchronous version. This demonstrates that it is possible to maximize training efficiency through algorithm-system co-design without compromising model p…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training

    cs.DC 2026-07 accept novelty 7.0 of 10

    BiDiRL raises disaggregated asynchronous LLM RL throughput up to 1.94× by hot-switching idle GPUs between rollout and training under a model-guided bidirectional scheduler.

  2. History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL

    cs.LG 2025-08 conditional novelty 7.0 of 10

    RhymeRL accelerates LLM RL training by up to 2.6x by decoding speculatively from a prompt's previous-epoch responses and by alternating long and short rollouts across steps to remove GPU bubbles.

  3. TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

    cs.LG 2026-08 conditional novelty 6.0 of 10

    TideRL, a readiness-aware elastic RL system, raises agentic RL training goodput by up to 5.6x over synchronous and over 33% over asynchronous baselines by preserving rollout KV caches, pipelining reference and actor m...

  4. StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

    cs.DC 2026-01 conditional novelty 6.0 of 10

    StaleFlow enforces per-trajectory staleness bounds and coordinates rollout data routing, achieving 1.42–2.68x higher RL post-training throughput with comparable convergence.

  5. MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster

    cs.LG 2025-07 conditional novelty 6.0 of 10

    MindSpeed RL uses a distributed transfer dock and an allgather-swap method to raise RL training throughput on Ascend NPUs by 1.42 to 3.97 times over two open baselines.

  6. QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

    cs.AI 2026-07 conditional novelty 5.0 of 10

    QLPO resamples GRPO training groups to favor short correct and long incorrect responses, cutting reasoning length substantially while keeping accuracy roughly unchanged.

  7. Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Molt is a lean, readable PyTorch-native RL training framework that reports statistically comparable step-time throughput to a Megatron-based stack, with the caveat that the head-to-head comparison measured throughput only.

  8. ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning

    cs.LG 2026-02 conditional novelty 5.0 of 10

    Distributed rollout workers under a bounded-staleness budget can keep a centralized learner saturated and cut LLM post-training cost by roughly a third at matched reward.

  9. Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library

    cs.LG 2025-06 conditional novelty 5.0 of 10

    ROLL is a distributed RL training library for LLMs with sample-level rollout scheduling and flexible device mapping, demonstrated on RLVR and agentic tasks but without baseline or scaling measurements.

  10. DeepCEE: Efficient Cross-Region Model Distributed Training System under Heterogeneous GPUs and Networks

    eess.SY 2025-05 conditional novelty 5.0 of 10

    DeepCEE groups heterogeneous GPUs by network and compute speed, schedules a compact zero-bubble pipeline across regions, and adapts micro-batch sizes to network fluctuations, reporting 1.3-2.8x higher training through...

  11. AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training

    cs.LG 2025-07 conditional novelty 4.0 of 10

    AsyncFlow combines a distributed streaming data queue with delayed parameter updates to improve RL post-training throughput by 1.59x on average over verl on Ascend clusters.

  12. Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.

Reference graph

Works this paper leans on

64 extracted references · 39 canonical work pages · cited by 12 Pith papers

  1. [1]

    Distribution of AI training is needed

    2024. Distribution of AI training is needed. https://www. tomshardware.com/tech-industry/artificial-intelligence/microsoft- azure-cto-claims-distribution-of-ai-training-is-needed-as-ai- datacenters-approach-power-grid-limits . (2024)

    work page 2024

  2. [2]

    Introducing OpenAI o1

    2024. Introducing OpenAI o1. https://openai.com/index/openai-o3- mini/. (2024)

    work page 2024

  3. [3]

    Multi-Datacenter Training

    2024. Multi-Datacenter Training. https://semianalysis.com/2024/09/ 04/multi-datacenter-training-openais/ . (2024)

    work page 2024

  4. [4]

    OpenAI o3-mini: Pushing the frontier of cost-effective reasoning

    2024. OpenAI o3-mini: Pushing the frontier of cost-effective reasoning. https://openai.com/o1/. (2024)

    work page 2024

  5. [5]

    Qwen2.5: A Party of Foundation Models! https://qwenlm.github

    2024. Qwen2.5: A Party of Foundation Models! https://qwenlm.github. io/blog/qwen2.5/. (2024)

    work page 2024

  6. [6]

    Claude 3.7 Sonnet and Claude Code

    2025. Claude 3.7 Sonnet and Claude Code. https://www.anthropic. com/news/claude-3-7-sonnet . (2025)

    work page 2025

  7. [7]

    Seed-Thinking-v1.5: Advancing Superb Reasoning Models with Reinforcement Learning

    2025. Seed-Thinking-v1.5: Advancing Superb Reasoning Models with Reinforcement Learning. https://github.com/ByteDance-Seed/Seed- Thinking-v1.5. (2025)

    work page 2025

  8. [8]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Am- mar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, et al. 2022. Deepspeed inference: Enabling 13 efficient inference of transformer models at unprecedented scale.arXiv (2022)

    work page 2022

Show all 64 references
  1. [9]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan H...

    2022 arXiv

  2. [10]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al . 2020. Language models are few-shot learners. Advances in neural information processing systems (2020)

    2020

  3. [11]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling lan- guage modeling with pathways.arXiv preprint arXiv:2204.02311 (2022)

    2022 arXiv

  4. [12]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. (2019)

    2019

  5. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

  6. [14]

    Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, and Hao Zhang. 2024. Efficient LLM Scheduling by Learning to Rank. (2024). arXiv:cs.LG/2408.15792 https://arxiv.org/abs/2408.15792

    2024 arXiv

  7. [15]

    Ronald L. Graham. 1969. Bounds on multiprocessing timing anomalies. SIAM journal on Applied Mathematics (1969)

    1969

  8. [16]

    Diandian Gu, Yihao Zhao, Yinmin Zhong, Yifan Xiong, Zhenhua Han, Peng Cheng, Fan Yang, Gang Huang, Xin Jin, and Xuanzhe Liu. 2023. Elasticflow: An elastic serverless training platform for distributed deep learning. In ACM ASPLOS

    2023

  9. [17]

    Arpan Gujarati, Reza Karimi, Safya Alzayat, Wei Hao, Antoine Kauf- mann, Ymir Vigfusson, and Jonathan Mace. 2020. Serving DNNs like Clockwork: Performance Predictability from the Bottom Up. InUSENIX OSDI

    2020

  10. [18]

    Eric Harper, Somshubra Majumdar, Oleksii Kuchaiev, Li Jason, Yang Zhang, Evelina Bakhturina, Vahid Noroozi, Sandeep Subramanian, Koluguri Nithin, Huang Jocelyn, Fei Jia, Jagadeesh Balam, Xuesong Yang, Micha Livne, Yi Dong, Sean Naren, and Boris Ginsburg. 2025. NeMo: a toolkit ...

    2025

  11. [19]

    Jian Hu, Xibin Wu, Weixun Wang, Dehao Zhang, Yu Cao, et al. 2024. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv preprint arXiv:2405.11143 (2024)

    2024 arXiv

  12. [20]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism . In NeurIPS

    2019

  13. [21]

    Changho Hwang, Taehyun Kim, Sunghyun Kim, Jinwoo Shin, and KyoungSoo Park. 2021. Elastic Resource Sharing for Distributed Deep Learning. In USENIX NSDI

    2021

  14. [22]

    2024.{MegaScale}: Scaling large language model training to more than 10,000{GPUs}

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al . 2024.{MegaScale}: Scaling large language model training to more than 10,000{GPUs}. In USENIX NSDI

    2024

  15. [24]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  16. [25]

    Kinman Lei, Yuyang Jin, Mingshu Zhai, Kezhao Huang, Haoxing Ye, and Jidong Zhai. 2024. {PUZZLE}: Efficiently Aligning Large Language Models through{Light-Weight} Context Switch. In 2024 USENIX Annual Technical Conference (USENIX ATC 24) . 127–140

    2024

  17. [26]

    Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. 2023. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving. arXiv (2023)

    2023

  18. [27]

    InProceedings of the 29th Symposium on Operating Systems Principles

    Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles. 611–626

  19. [28]

    Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. 2024. ReaLHF: Optimized RLHF Training for Large Language Mod- els through Parameter Reallocation. arXiv preprint arXiv:2406.14088 (2024)

    2024 arXiv

  20. [29]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient transformer training over multiple gpus using automatic parallelism. InProceedings of the VLDB Endowment . 14

    2022

  21. [30]

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level. https: //pretty-radio-b75.no...

    2025

  22. [31]

    Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hos- seini, Rishabh Agarwal, and Aaron Courville. 2025. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. International Conference on Learning Representations (ICLR) (2025)

    2025

  23. [32]

    OpenAI. 2023. GPT-4 Technical Report. (2023)

    2023

  24. [33]

    Jordan, and Ion Stoica

    Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In13th USENIX Symposium on Operating Syste...

    2018

  25. [34]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 118– 132

    2024

  26. [35]

    Ganger, and Eric P

    Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021. Pollux: Co-adaptive Cluster Scheduling for Goodput- Optimized Deep Learning. In USENIX OSDI

    2021

  27. [36]

    Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv (2019)

    2019

  28. [37]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    2024 arXiv

  29. [38]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hy- bridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256 (2024)

    2024 arXiv

  30. [39]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

  31. [40]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi- billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  32. [41]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. (2020)

    2020

  33. [42]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. verl: Volcano Engine Reinforcement Learning for LLM. https://github.com/ volcengine/verl. (2024)

    2024

  34. [43]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al

  35. [44]

    Pytorch Team. 2025. Torch Distributed RPC Framework. (2025). https: //pytorch.org/docs/stable/rpc.html

    2025

  36. [45]

    Chenchen Shou, Guyue Liu, Hao Nie, Huaiyu Meng, Yu Zhou, Yimin Jiang, Wenqing Lv, Yelong Xu, Yuanwei Lu, Zhang Chen, et al. 2025. InfinitePOD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers. arXiv preprint arXiv:2502.03885 (2025)

    2025 arXiv

  37. [46]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is all you need. Neural Information Processing Systems (2017)

    2017

  38. [47]

    5: Scaling reinforcement learning with llms

    Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 (2025)

    2025 arXiv

  39. [48]

    Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. 2025. DistRL: An Asynchronous Distributed Reinforce- ment Learning Framework for On-Device Control Agents. (2025). arXiv:cs.LG/2410.14803 https://arxiv.org/abs/2410.14803

    2025 arXiv

  40. [49]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  41. [50]

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. 2023. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920 (2023)

    2023 arXiv

  42. [51]

    Minjie Wang, Chien-chin Huang, and Jinyang Li. 2019. Supporting very large models using automatic dataflow graph partitioning. In EuroSys

    2019

  43. [52]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A Distributed Serving System for {Transformer-Based} Generative Models. In USENIX OSDI

    2022

  44. [53]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism. arXiv preprint arXiv:2404.09526 (2024)

    2024 arXiv

  45. [54]

    Zili Zhang, Yinmin Zhong, Ranchen Ming, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, and Xin Jin. 2024. DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models. arXiv preprint arXiv:2408.04275 (2024)

    2024

  46. [55]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  47. [56]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. (2024)

    2024

  48. [57]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. 2025. DAPO: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 (2025)

    2025 arXiv

  49. [58]

    Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, et al

  50. [59]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Au- tomating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In USENIX OSDI

    2022

  51. [60]

    Ruidong Zhu, Ziheng Jiang, Chao Jin, Peng Wu, Cesar A. Stuardo, Dongyang Wang, Xinlei Zhang, Huaping Zhou, Haoran Wei, Yang Cheng, Jianzhe Xiao, Xinyi Zhang, Lingjun Liu, Haibin Lin, Li-Wen Chang, Jianxi Ye, Xiao Yu, Xuanzhe Liu, Xin Jin, and Xin Liu. 2025. MegaScale-Infer: Se...

    2025 arXiv

  52. [61]

    2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serv- ing

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xu- anzhe Liu, Xin Jin, and Hao Zhang. 2024.{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serv- ing. In USENIX OSDI

    2024

  53. [63]

    USENIX NSDI (2025)

    Rlhfuse: Efficient rlhf training for large language models with inter-and intra-stage fusion. USENIX NSDI (2025)

    2025

  54. [64]

    Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, and Xin Jin. 2024. RLHFuse: Efficient RLHF Training for Large Language Mod- els with Inter- and Intra-Stage Fusion. (2024). arXiv:cs.LG/2409.13221 https://...

    2024 arXiv

  55. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles

    Efficient Memory Management for Large Language Model Serv- ing with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles. 611–626

  56. [2025]

    arXiv preprint arXiv:2501.12948 (2025)

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.