Pith. sign in

REVIEW 3 major objections 8 minor 1 cited by

MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster

T0 review · 3 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MindSpeed RL claims that reorganizing RL training dataflow with distributed transfer docks and an allgather-swap mechanism raises end-to-end throughput by 1.42–3.97x over existing open-source systems on NPU clusters.

desk verdict Real system work with two sensible techniques and open-source code, but the 1.42–3.97x claim is established only at 16 NPUs, and the 671B scale has no baseline comparison. read the letter →

arxiv 2507.19017 v1 pith:JBBVU5HS submitted 2025-07-25 cs.LG cs.AI

Laingjun Feng , Chenyi Pan , Xinjie Guo , Fei Mei , Benzhe Ning , Jianxiang Zhang , Xinyang Liu , Beirong Zhou
show 6 more authors
Zeng Shu Chang Liu Guang Yang Zhenyu Han Jiangben Wang Bo Wang
This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningLLMalignmentdistributeddataflowtransferdockallgather-swapmixture-of-expertsNPUclusterGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the bottleneck in large-scale reinforcement learning for language models is not compute alone but the dataflow between workers—moving samples and moving model weights. It claims that replacing the centralized replay buffer with distributed transfer docks, and swapping update weights to host memory during generation, removes those bottlenecks. If right, RL alignment training on NPU clusters becomes 1.42–3.97x faster than current open-source systems, while freeing device memory for longer generation contexts. The experiments cover dense models up to 32B parameters, a 30B mixture-of-experts model, and a 671B mixture-of-experts model on 384 NPUs.

What carries the argument

The argument turns on two named mechanisms. The distributed transfer dock (TD) decomposes the centralized replay buffer into S warehouses spread across nodes, each holding real samples, while every worker state gets a TD controller carrying only scalar metadata such as sample index and warehouse index; workers get metadata from their controller and then fetch real data directly from a warehouse, cutting per-warehouse communication volume roughly by the number of warehouses. The allgather-swap technique handles resharding: it allgathers weights into a temporary buffer, copies the slices needed for generation, swaps the original update weights from device memory to host memory, and releases the temporary buffer; before the next update, the weights are swapped back and overlapped with inference. Together these mechanisms eliminate centralized dispatch congestion and redundant device memory, which is what the speedup and memory-freeing claims rest on.

What would settle it

Re-run the same GRPO workloads on the same 384-NPU cluster with the two baseline frameworks configured by their own teams and with the exact settings published; a throughput gap near 1x would show that the reported 1.42–3.97x advantage reflects baseline configuration rather than the new dataflow design.

Watch

Extended reading notes

Core claim

The central claim is that RL training systems should treat dataflow as a first-class distributed resource rather than a side effect of scheduling. The paper identifies two flows—sample flow (prompts, responses, logits, rewards between workers) and resharding flow (weights converted between training and generation parallelisms)—and argues each has a solvable structural defect. For sample flow, a central replay buffer becomes a communication bottleneck because every worker fetches large tensors from one place; the transfer dock splits storage into per-node warehouses and keeps only scalar metadata on per-worker controllers. For resharding flow, allgather-swap removes redundant TP/EP weight buffers by moving update weights to host memory during generation. The end-to-end claim follows from these two mechanisms plus integrated parallelization and fused kernels: throughput rises by 1.42–3.97x against two open-source baselines, and the 671B model trains at 200–250 TPS across 384 NPUs with stable rewards.

Load-bearing premise

The central claim assumes the two baseline systems were run near their best possible configuration on the same hardware, since the paper says it tuned their hyperparameters and kernels but does not list the exact settings, kernel support, or code changes used.

Editorial extensions

If this is right

  • RL practitioners on NPU clusters can train models up to 671B parameters at 200–250 TPS with stable reward growth, which the paper says few open-source frameworks support.
  • Decentralizing the replay buffer reduces sample-flow communication roughly in proportion to the number of nodes, predicting better scaling as cluster size grows.
  • Offloading update weights to host memory during generation frees device capacity for the KV cache, and the 8GB-per-device released memory in the 32B case directly lengthens feasible generation contexts.
  • The measured linearity of 81.1% at 192 NPUs, versus 40.4% and 61.9% for two baselines, indicates that the dataflow redesign, not just kernel fusion, preserves efficiency as workers scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The transfer-dock decomposition should transfer to GPU clusters and to asynchronous RL variants: any pipeline with a central sample buffer and distinct worker states has the same metadata-versus-data split, so the mechanism is not tied to this hardware.
  • The allgather-swap benefit is hardware-dependent: on nodes with host-device bandwidth well below the 50 GB/s assumed here, the swap cost could offset the freed memory, so the technique needs a bandwidth threshold before adoption.
  • A natural next experiment is to scale beyond 384 NPUs and increase global batch size; the paper's communication formula predicts the transfer dock's advantage should widen with batch and cluster size, which the current linearity data only begins to show.
  • A direct test of the 671B-model claim would be to run the same workload on the two baseline frameworks and report their throughput or memory failures, rather than comparing only smaller models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper introduces MindSpeed RL, a distributed reinforcement-learning training system for Ascend NPU clusters. It identifies two dataflow bottlenecks in RL training—dispatch overhead in sample flow and redundant memory in resharding flow—and proposes two techniques: a distributed transfer-dock strategy that splits the replay buffer into multiple warehouses and controllers, and an allgather–swap technique that offloads update weights to host memory during resharding. The system integrates a wide range of parallelization strategies and fused kernels. The experimental section reports end-to-end throughput comparisons against OpenRLHF and VeRL on Qwen2.5-7B/32B and Qwen3-MoE-30B at 16 NPUs, claiming a 1.42–3.97x speedup, plus a 384-NPU run for DeepSeek-R1-MoE-671B with no baseline comparison. The paper also includes ablations separating the two proposed techniques (MSRLP) and a conventional replay buffer (MSRLB).

Significance. If the stated speedups hold, this is a valuable systems contribution: two well-motivated dataflow optimizations, an open-source implementation, and the largest RL training run reported on Ascend NPUs. The communication-volume equations and memory estimates are internally consistent, and the ablations (MSRLP and MSRLB) provide evidence that the two techniques, rather than only the bundled kernel optimizations, contribute to the gains. However, the headline claim of 1.42–3.97x speedup is not fully supported by the evidence presented: the only direct comparison against external baselines is at 16 NPUs, and the baseline configurations are not disclosed. For the DeepSeek-R1-MoE-671B experiment, which is the most impressive scale result, no baseline throughput is measured. These gaps limit the strength of the central claim as written.

major comments (3)
  1. [Abstract and Evaluation: Figure 7 vs. Figure 11] The abstract and introduction state that comprehensive experiments on Qwen2.5-Dense-7B/32B, Qwen3-MoE-30B, and DeepSeek-R1-MoE-671B show a 1.42–3.97x throughput increase over existing systems. However, the end-to-end comparison in Figure 7 is explicitly at 16 NPUs and covers only the three smaller models. The section 'Results of Large-scale MoE Models' for DeepSeek-R1-MoE-671B at 384 NPUs reports only MindSpeed RL's own throughput (200–250 TPS) and reward curve; no OpenRLHF or VeRL comparison is presented. Consequently, the stated speedup range is not established for the 671B model, and the text should either present baseline measurements at that scale or qualify the claim in the abstract to reflect the actual comparison range.
  2. [Evaluation: 'fine-tune the hyperparameters'] The paragraph before Figure 7 states that for each framework the authors 'fine-tune the hyperparameters, e.g., parallelization strategies and fused kernels, to present the best performance,' but it does not disclose the exact configurations used for OpenRLHF and VeRL: parallelism degrees (TP/PP/DP/EP), whether vLLM-Ascend was used, which fused kernels were enabled, or what code modifications were needed to run these GPU-oriented frameworks on Ascend NPUs. Table 2 indicates that both baselines have partial support for several features (marked with '◦'), so the speedup may partly reflect missing optimizations in the baselines rather than the effect of the proposed dataflow techniques. The authors should provide a per-framework configuration table or appendix; without it, the fairness of the 1.42–3.97x comparison cannot be assessed.
  3. [Evaluation: Ablation Study, Figure 9] The linearity experiment in Figure 9 shows that at 192 NPUs, MindSpeed RL reaches 81.1% linearity, whereas VeRL and MSRLB reach 40.4% and 61.9%, respectively. This is a scalability claim, but it is not connected back to the end-to-end speedup numbers: Figure 7's comparisons are all at 16 NPUs, where the scalability advantage has little effect. To support the title's 'Scalable' claim, the authors should report end-to-end throughput at an intermediate scale (e.g., 64 or 128 NPUs) for at least one model, either for MSRL and the baselines, or explain why the linearity experiment alone demonstrates scalability of the speedup.
minor comments (8)
  1. [Results of Large-scale MoE Models] The text says 'The end-to-end throughput and rewards in 100 iterations are shown in Figure 10,' but Figure 10 already shows memory profiling for the allgather–swap technique; the large-scale MoE results appear in Figure 11. The reference should be corrected.
  2. [Introduction] There is a typo: 'allgthaer–swap' should be 'allgather–swap'.
  3. [Throughout the manuscript] The spelling of the proposed technique alternates between 'allgather–swap' and 'allgather-swap'; please use one consistent form.
  4. [Table 2] The symbol '◦' in the table is not explained in the caption. The text defines it later ('in development or partially supported'), but the caption should include this definition.
  5. [Equations (1), (2), and (4)] The term '10243' appears to mean 1024^3 (bytes per GB) but is rendered ambiguously. Please write it as 1024^3 or 1024^3 bytes/GB.
  6. [Problem Statement] The claim that the Qwen3-MoE-30B resharding flow has 'more than 60GB' of redundant memory is stated without a derivation; providing the parameters substituted into Eq. (3) would make this reproducible.
  7. [Evaluation: End-to-End Performance] Figure 7 reports a speedup range but does not show the underlying throughput values or error bars. Adding numeric labels or a table of throughputs would let readers verify the claimed range.
  8. [Introduction] The sentence 'To the best of the author's knowledge, most open-source frameworks cannot train all of these models' is an unsupported generality; it would be more precise to say that the authors are not aware of an existing open-source framework that does so.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the throughput claims rest on end-to-end measurements against external baselines, and the analytical formulas are input-driven accounting identities rather than self-referential derivations.

full rationale

The paper's central quantitative claims are empirical: MindSpeed RL is compared end-to-end against OpenRLHF and VeRL on an Ascend cluster, with throughput defined by Eq. (5) as a measured quantity over five iterations, and the reported 1.42–3.97x speedup comes from Figure 7. The transfer-dock communication-volume estimates in Eqs. (1), (2), and (4) are algebraic accounting identities that take stated inputs (G, N, B, PL, SL, n, M, C, S) and are used only for motivation, not as a substitute for measured performance. The allgather-swap technique is validated by memory profiling in Figure 10 and by the MSRLP ablation in Figure 7, which isolates the two proposed mechanisms from the full system. The linearity comparison in Figure 9 and the reward/score comparisons in Figure 8 and Table 3 are also direct measurements against an external framework. The paper does not define any claimed result in terms of its own fitted outputs, does not rename a known result as a new prediction, and does not rest its load-bearing claims on an unverified self-citation chain. Concerns about baseline configuration fairness or the absence of a large-scale baseline comparison are validity and scope issues, not circularity, and therefore do not affect the circularity score.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No fitted parameters drive the central result; the one hand-chosen parameter is the TD controller count C. The axioms are mostly domain assumptions about where bottlenecks lie, how communication cost scales, and how the comparison baselines were configured. No new physical or model entities are introduced.

free parameters (1)
  • TD controller count C = 5 to 10
    The number of TD controllers in Eq. (4) is a hand-chosen design parameter said to be 'usually [5,10]' and decided by the RL algorithm. It affects the metadata overhead in the sample-flow formula but is not fitted to the reported throughput.
assumptions (5)
  • domain assumption RL training workloads can be decomposed into sample flow and resharding flow.
    The entire design targets these two flows, following Sheng et al. 2025. If other data dependencies dominate, such as checkpointing or logging, the optimizations may miss the real bottleneck.
  • domain assumption Dispatch overhead scales with communication volume as modeled by Eq. (2), with bytes per token B=4 and per-warehouse traffic dividing by S.
    The transfer dock benefit follows from this linear volume model. Serialization and deserialization costs are mentioned but not included in the equation; if they do not scale with S, the gains differ.
  • domain assumption Host-device bandwidth of about 50 GB/s is sufficient for the swap overhead to be negligible or overlap with inference.
    The allgather-swap technique relies on fast D2H/H2D transfers on Ascend. On systems with slower host-device links, the tradeoff could reverse.
  • domain assumption Throughput as defined in Eq. (5), tokens per device per second, is a meaningful comparison metric for RL training systems.
    The 1.42 to 3.97 times claim is measured in this metric; it does not capture convergence quality or wall-clock to target reward, although the paper separately reports reward curves and benchmark scores.
  • domain assumption The baseline frameworks were configured close to their best achievable performance on Ascend NPUs.
    The paper states hyperparameters were tuned for each framework but gives no per-baseline configuration details; the fairness of the SOTA comparison rests on this premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster." pith.science (2026). https://pith.science/paper/JBBVU5HS

@misc{pith2026250719017,
  author       = {Pith},
  title        = {Pith review of: MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JBBVU5HS}},
  note         = {Machine review of arXiv:2507.19017}
}
read the original abstract

Reinforcement learning (RL) is a paradigm increasingly used to align large language models. Popular RL algorithms utilize multiple workers and can be modeled as a graph, where each node is the status of a worker and each edge represents dataflow between nodes. Owing to the heavy cross-node dependencies, the RL training system usually suffers from poor cluster scalability and low memory utilization. In this article, we introduce MindSpeed RL, an effective and efficient system for large-scale RL training. Unlike existing centralized methods, MindSpeed RL organizes the essential data dependencies in RL training, i.e., sample flow and resharding flow, from a distributed view. On the one hand, a distributed transfer dock strategy, which sets controllers and warehouses on the basis of the conventional replay buffer, is designed to release the dispatch overhead in the sample flow. A practical allgather--swap strategy is presented to eliminate redundant memory usage in resharding flow. In addition, MindSpeed RL further integrates numerous parallelization strategies and acceleration techniques for systematic optimization. Compared with existing state-of-the-art systems, comprehensive experiments on the RL training of popular Qwen2.5-Dense-7B/32B, Qwen3-MoE-30B, and DeepSeek-R1-MoE-671B show that MindSpeed RL increases the throughput by 1.42 ~ 3.97 times. Finally, we open--source MindSpeed RL and perform all the experiments on a super pod of Ascend with 384 neural processing units (NPUs) to demonstrate the powerful performance and reliability of Ascend.

Figures

Figures reproduced from arXiv: 2507.19017 by the authors.

Figure 1
Figure 1. Data dependencies in RL, including sample flow [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Centralized replay buffer mechanism in RL sys [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Distributed transfer dock strategy. Each worker [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (7 more)
Figure 3
Figure 3. Figure 3: Redundant memory issues in the distributed re [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 6
Figure 6. Figure 6: Architecture of MindSpeed RL, which consists [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 5
Figure 5. Figure 5: Allgather–swap technique in the distributed re [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 8
Figure 8. Figure 8: Reward comparison for MindSpeed RL and VeRL [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Linearity comparison for VeRL, MSRLB, and [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 11
Figure 11. Figure 11: Throughput and rewards of DeepSeek-R1-MoE [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]
Figure 10
Figure 10. Figure 10: Memory profiling for the resharding flow of [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

    cs.LG 2026-08 conditional novelty 6.0 of 10

    TideRL, a readiness-aware elastic RL system, raises agentic RL training goodput by up to 5.6x over synchronous and over 33% over asynchronous baselines by preserving rollout KV caches, pipelining reference and actor m...

Reference graph

Works this paper leans on

55 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ascend . 2024. MindSpeed. https://gitee.com/ascend/MindSpeed. Accessed: 2024-3-30

    work page 2024

  2. [2]

    Ascend . 2025. vLLM-Ascend. https://github.com/vllm-project/vllm-ascend/tree/main. Accessed: 2025-3-30

    work page 2025

  3. [3]

    Dao, T. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv:2307.08691

    arXiv 2023

  4. [4]

    DeepSeek-AI; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; Bi, X.; Zhang, X.; and et al., X. Y. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948

    arXiv 2025

  5. [5]

    Fu, W.; Gao, J.; Shen, X.; Zhu, C.; Mei, Z.; He, C.; Xu, S.; Wei, G.; Mei, J.; Wang, J.; Yang, T.; Yuan, B.; and Wu, Y. 2025. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning. arXiv:2505.24298

    arXiv 2025

  6. [6]

    Graef, N.; Wasielewski, A.; and Clapp, M. 2025. FlashNorm: Fast Normalization for LLMs. arXiv:2407.09577

    work page Pith review arXiv 2025

  7. [7]

    Havrilla, A.; Zhuravinskyi, M.; Phung, D.; Tiwari, A.; Tow, J.; Biderman, S.; Anthony, Q.; and Castricato, L. 2023. TrlX: A Framework for Large Scale Reinforcement Learning from Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 8578--8595

    work page 2023

  8. [8]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; and Eric. 2021. Measuring Mathematical Problem Solving With the MATH Dataset. In Advances in Neural Information Processing Systems (NeurlPS)

    work page 2021

Show all 55 references
  1. [9]

    K.; Zhu, Z.; Wang, W.; Jiang, S.; Wang, H.; Chen, H.; Chen, B.; Fang, W.; Xianyu; Cao, Y.; Xu, H.; and Liu, Y

    Hu, J.; Wu, X.; Shen, W.; Liu, J. K.; Zhu, Z.; Wang, W.; Jiang, S.; Wang, H.; Chen, H.; Chen, B.; Fang, W.; Xianyu; Cao, Y.; Xu, H.; and Liu, Y. 2025. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework. arXiv:2405.11143

    2025 arXiv

  2. [10]

    X.; Chen, D.; Lee, H

    Huang, Y.; Cheng, Y.; Bapna, A.; Firat, O.; Chen, M. X.; Chen, D.; Lee, H. J.; Ngiam, J.; Le, Q. V.; and Wu, Y. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Advances in Neural Information Processing Systems, 8--14. Vancouver(CA)

    2019

  3. [11]

    A.; Tanaka, M.; Zhang, C.; Zhang, M.; Aminabadi, R

    Jacobs, S. A.; Tanaka, M.; Zhang, C.; Zhang, M.; Aminabadi, R. Y.; Song, S. L.; Rajbhandari, S.; and He, Y. 2024. System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. In 2024 IEEE International Parallel and Distributed Processing Symposium Wo...

    2024

  4. [12]

    Li, S.; Fang, J.; Bian, Z.; Liu, H.; Liu, Y.; Huang, H.; Wang, B.; and You, Y. 2023. Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training. In Proceedings of the 52nd International Conference on Parallel Processing, 766--775

    2023

  5. [13]

    E.; and Stoica, I

    Liang, E.; Wu, Z.; Luo, M.; Mika, S.; Gonzalez, J. E.; and Stoica, I. 2021. RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem. arXiv:2011.12719

    2021 arXiv

  6. [14]

    Liu, H.; Zaharia, M.; and Abbeel, P. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889

    2023 arXiv

  7. [15]

    Y.; Roongta, M.; Cai, C.; Luo, J.; Li, L

    Luo, M.; Tan, S.; Wong, J.; Shi, X.; Tang, W. Y.; Roongta, M.; Cai, C.; Luo, J.; Li, L. E.; Popa, R. A.; and Stoica, I. 2025. DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL. https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Mo...

    2025

  8. [16]

    MATH-AI . 2024. AIME24. https://huggingface.co/datasets/math-ai/aime24

    2024

  9. [17]

    Mei, Z.; Fu, W.; Li, K.; Wang, G.; Zhang, H.; and Wu, Y. 2025. ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation. arXiv:2406.14088

    2025 arXiv

  10. [18]

    I.; and Stoica, I

    Moritz, P.; Nishihara, R.; Wang, S.; Tumanov, A.; Liaw, R.; Liang, E.; Elibol, M.; Yang, Z.; Paul, W.; Jordan, M. I.; and Stoica, I. 2018. Ray: A Distributed Framework for Emerging AI Applications. arXiv:1712.05889

    2018 arXiv

  11. [19]

    Narayanan, D.; Shoeybi, M.; Casper, J.; Legresley, P.; and Zaharia, M. 2021. Efficient Large-Scale Language Model Training on GPU Clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

    2021

  12. [20]

    Noukhovitch, M.; Huang, S.; Xhonneux, S.; Hosseini, A.; Agarwal, R.; and Courville, A. 2025. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models. In Proceedings of the 2025 International Conference on Learning Representations (ICLR)

    2025

  13. [21]

    OpenAI . 2024. OpenAI o1. https://openai.com/zh-Hans-CN/o1/. Accessed: 2024-9-12

    2024

  14. [22]

    OpenAI; Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; Avila, R.; Babuschkin, I.; Balaji, S.; Balcom, V.; Baltescu, P.; Bao, H.; Bavarian, M.; Belgum, J.; Bello, I.; Berdine, J.; Bernadett-Shapi...

    2024 arXiv

  15. [23]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language model...

    2022 arXiv

  16. [24]

    Qi, P.; Wan, X.; Huang, G.; and Lin, M. 2023. Zero Bubble Pipeline Parallelism. arXiv:2401.10241

    2023 arXiv

  17. [25]

    Y.; Awan, A

    Rajbhandari, S.; Li, C.; Yao, Z.; Zhang, M.; Aminabadi, R. Y.; Awan, A. A.; Rasley, J.; and He, Y. 2022. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale. arXiv:2201.05596

    2022 arXiv

  18. [26]

    Rajbhandari, S.; Rasley, J.; Ruwase, O.; and He, Y. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis

    2020

  19. [27]

    Rasley, J.; Rajbhandari, S.; Ruwase, O.; and He, Y. 2020. DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In KDD 20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

    2020

  20. [28]

    L.; Stickland, A

    Rein, D.; Hou, B. L.; Stickland, A. C.; Petty, J.; Pang, R. Y.; Dirani, J.; Michael, J.; and Bowman, S. R. 2023. GPQA: A Graduate-Level Google-Proof QA Benchmark. arXiv:2311.12022

    2023 arXiv

  21. [29]

    Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms

    2017

  22. [30]

    Shao, Z.; Wang, P.; Zhu, Q.; and et al. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300

    2024 arXiv

  23. [31]

    Shazeer, N. 2020. GLU Variants Improve Transformer. arXiv:2002.05202

    2020 arXiv

  24. [32]

    S.; Aithal, A.; and Kuchaiev, O

    Shen, G.; Wang, Z.; Delalleau, O.; Zeng, J.; Dong, Y.; Egert, D.; Sun, S.; Zhang, J.; Jain, S.; Taghibakhshi, A.; Ausin, M. S.; Aithal, A.; and Kuchaiev, O. 2024. NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment. arXiv:2405.01481

    2024 arXiv

  25. [33]

    Shen, W.; Liu, G.; Wu, Z.; Zhu, R.; Yang, Q.; Xin, C.; Yue, Y.; and Yan, L. 2025. Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback. arXiv:2503.22230

    2025 arXiv

  26. [34]

    Sheng, G.; Zhang, C.; Ye, Z.; Wu, X.; Zhang, W.; Zhang, R.; Peng, Y.; Lin, H.; and Wu, C. 2025. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, 1279–1297. ACM

    2025

  27. [35]

    Shoeybi, M.; Patwary, M.; Puri, R.; Legresley, P.; Casper, J.; and Catanzaro, B. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using GPU Model Parallelism. In Proceedings of CoRR

    2019

  28. [36]

    Singh, V.; Khalid, Z.; Oyallon, E.; and Belilovsky, E. 2025. Model Parallelism With Subnetwork Data Parallelism. arXiv:2507.09029

    2025 arXiv

  29. [37]

    Su, J.; Lu, Y.; Pan, S.; Murtadha, A.; Wen, B.; and Liu, Y. 2023. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv:2104.09864

    2023 arXiv

  30. [38]

    Team, K.; Du, A.; Gao, B.; Xing, B.; Jiang, C.; Chen, C.; Li, C.; Xiao, C.; Du, C.; and Liao, C. 2025. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599

    2025 arXiv

  31. [39]

    Wang, S.; Zhang, S.; Zhang, J.; Hu, R.; Li, X.; Zhang, T.; Li, J.; Wu, F.; Wang, G.; and Hovy, E. 2025. Reinforcement Learning Enhanced LLMs: A Survey. arXiv:2412.10400

    2025 arXiv

  32. [40]

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, 35:24824-24837, 2022

    2022

  33. [41]

    L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A

    Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020. HuggingFac...

    2020 arXiv

  34. [42]

    Yang, A.; Li, A.; Yang, B.; and et al. 2025. Qwen3 Technical Report. arXiv:2505.09388

    2025 arXiv

  35. [43]

    Y.; Ruwase, O.; Rajbhandari, S.; Wu, X.; Awan, A

    Yao, Z.; Aminabadi, R. Y.; Ruwase, O.; Rajbhandari, S.; Wu, X.; Awan, A. A.; Rasley, J.; Zhang, M.; Li, C.; and Holmes, C. 2023. DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales. arXiv:2308.01320

    2023 arXiv

  36. [44]

    Yu, Q.; Zhang, Z.; Zhu, R.; Yuan, Y.; and et al. 2025. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476

    2025 arXiv

  37. [45]

    Zhang, C.; Du, K.; Liu, S.; Kwon, W.; Mo, X.; Wang, Y.; Liu, X.; You, K.; Li, Z.; and Long, M. 2025. Jenga: Effective Memory Management for Serving LLM with Heterogeneity. arXiv:2503.18292

    2025 arXiv

  38. [46]

    Zhang, C.; Sheng, G.; Liu, S.; Li, J.; Feng, Z.; Liu, Z.; Liu, X.; Jia, X.; Peng, Y.; Lin, H.; and Wu, C. 2024. A Framework for Training Large Language Models for Code Generation via Proximal Policy Optimization. In Proceedings of NL2Code Workshop of ACM KDD 2024

    2024

  39. [47]

    Zhang, S.; Diao, L.; Wu, C.; Wang, S.; and Lin, W. 2022. Accelerating Large-Scale Distributed Neural Network Training with SPMD Parallelism. In Proceedings of the 13th Symposium on Cloud Computing

    2022

  40. [48]

    C.; Xu, M.; Wright, L.; Shojanazeri, H.; Ott, M.; and Shleifer, S

    Zhao, Y.; Gu, A.; Varma, R.; Luo, L.; Huang, C. C.; Xu, M.; Wright, L.; Shojanazeri, H.; Ott, M.; and Shleifer, S. 2024. FSDP2. https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_shard.html. Accessed: 2024-9-1

    2024

  41. [49]

    Zhao, Y.; Gu, A.; Varma, R.; Luo, L.; Huang, C.-C.; Xu, M.; Wright, L.; Shojanazeri, H.; Ott, M.; Shleifer, S.; Desmaison, A.; Balioglu, C.; Damania, P.; Nguyen, B.; Chauhan, G.; Hao, Y.; Mathews, A.; and Li, S. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Par...

    2023 arXiv

  42. [50]

    H.; Cao, S.; Kozyrakis, C.; Stoica, I.; Gonzalez, J

    Zheng, L.; Yin, L.; Xie, Z.; Sun, C.; Huang, J.; Yu, C. H.; Cao, S.; Kozyrakis, C.; Stoica, I.; Gonzalez, J. E.; Barrett, C.; and Sheng, Y. 2024. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104

    2024 arXiv

  43. [51]

    Zhong, Y.; Zhang, Z.; Song, X.; Hu, H.; Jin, C.; Wu, B.; Chen, N.; Chen, Y.; Zhou, Y.; Wan, C.; Zhou, H.; Jiang, Y.; Zhu, Y.; and Jiang, D. 2025 a . StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation. arXiv:2504.15930

    2025 arXiv

  44. [52]

    Zhong, Y.; Zhang, Z.; Wu, B.; Liu, S.; Chen, Y.; Wan, C.; Hu, H.; Xia, L.; Ming, R.; Zhu, Y.; and Jin, X. 2025 b . Optimizing RLHF Training for Large Language Models with Stage Fusion. arXiv:2409.13221

    2025 arXiv

  45. [53]

    Zhou, Y.; and Yang, K. 2024. Exploring TensorRT to Improve Real-Time Inference for Deep Learning. In 2022 IEEE 24th Int Conf on High Performance Computing (HPCC)

    2024

  46. [54]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.