Pith. sign in

REVIEW 4 major objections 6 minor 42 references

MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MegatronApp claims that four optional modules—MegaScan, MegaFBD, MegaDPP, and MegaScope—give distributed LLM training double-digit throughput gains, near-zero-overhead tracing, and live model inspection with minimal code changes.

desk verdict Design proposal with four useful-sounding modules and zero measurements; the key module contradicts itself, so the efficiency claims are unsupported. read the letter →

arxiv 2507.19845 v1 pith:WPOFTWJD submitted 2025-07-26 cs.DC

Bohan Zhao , Guang Yang , Shuo Chen , Ruitao Liu , Tingrui Zhang , Yongchao He , Wei Xu This is my paper · ORCID
classification cs.DC
keywords distributedtraininglargelanguagemodelsMegatronAppMegatron-LMstragglerdetectionpipelineparallelismheterogeneouscomputingmodelinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MegatronApp claims that distributed training of trillion-parameter language models can be made substantially easier to run, debug, and understand through four small, optional add-on modules that sit on top of the standard Megatron-LM training stack. The four modules—MegaScan, MegaFBD, MegaDPP, and MegaScope—target the pain points that emerge at cluster scale: stragglers and cascading slowdowns, rigid use of homogeneous GPUs, fixed pipeline schedules, and opaque model internals. If the claims hold, practitioners would get root-cause diagnosis from operator-level traces, the ability to run forward and backward passes on different hardware, schedules that adapt to changing network and compute conditions, and live visualization plus perturbation of intermediate tensors—all enabled by runtime flags with near-zero tracing overhead. The paper's central assertion is that these capabilities collectively produce double-digit gains in throughput and cluster utilization while preserving compatibility with the upstream framework.

What carries the argument

The load-bearing machinery is the design of each module. MegaScan's mechanism is CUDA-event tracing plus cross-rank dependency reconstruction and clock alignment anchored on collective operations, with a multi-stage heuristic that separates fault sources from victims via peer-comparison and bandwidth analysis. MegaFBD's mechanism is a virtual-rank/physical-rank split: forward and backward instances keep Megatron's original allocation rules while being placed on different resources, coordinated by a bit-vector readiness protocol that lets control threads launch a collective only when all participants have posted the same request. MegaDPP's mechanism is a two-dimensional task matrix indexed by model chunk and microbatch, traversed depth-first or breadth-first, combined with a lightweight asynchronous P2P communication library. MegaScope's mechanism is registration-based sampling of intermediate tensors, hierarchical on-host compression, and pluggable perturbation injection before tensors are written back to memory.

What would settle it

Run an unmodified Megatron-LM training job and the same job with MegaFBD enabled on a heterogeneous CPU/GPU cluster, matching hyperparameters and data order; if the loss curves diverge, gradients differ, or the bit-vector coordinator deadlocks under group-order execution, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a set of four composable modules—MegaScan, MegaFBD, MegaDPP, and MegaScope—collectively give Megatron-LM training an operational layer of reliability, efficiency, and transparency that the base framework lacks. MegaScan records GPU-side timestamps using CUDA Events, merges per-rank traces into the Chrome Tracing format, aligns timelines using synchronous collectives as anchors, and identifies genuine slow nodes by checking which rank is consistently the slowest member of every collective it joins and which shows degraded point-to-point bandwidth. MegaFBD decouples forward and backward computation into separate logical ranks that can live on different devices with different parallel degrees, using a communication coordinator with bit-vector readiness tables to prevent deadlocks. MegaDPP treats pipeline scheduling as a traversal order over a two-dimensional matrix of model chunks and microbatches, switching between depth-first and breadth-first orders and overlapping communication through an asynchronous P2P library. MegaScope provides registration-based sampling, on-host compression, and an interactive interface with perturbation injection for inspecting and manipulating intermediate tensors during training.

Load-bearing premise

MegaFBD's efficiency gains rest on the assumption that splitting forward and backward phases onto different ranks and different parallel configurations preserves exact training semantics—correct gradients, optimizer state, and no deadlocks—yet the paper provides no proof or convergence experiment for this.

Editorial extensions

If this is right

  • Users can enable any combination of the four modules through runtime flags, without rewriting training code.
  • Slow-node diagnosis becomes operator-granular: a throttled GPU is identified by comparing kernels across data-parallel peers and checking collective start delays and P2P bandwidth, not just utilization counters.
  • Forward and backward passes can be assigned to different devices with different parallel configurations, which the paper argues relieves memory contention and lets heterogeneous clusters contribute compute.
  • Pipeline schedules can switch between depth-first and breadth-first traversal and overlap P2P communication, flattening the effect of link jitter and down-clocking.
  • Researchers can capture, compress, and interactively view activations, attention maps, and token trajectories, and inject noise or bit flips into chosen layers, without dumping terabyte-scale logs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same forward-backward decoupling idea could be applied to inference workloads, splitting prefill and decode phases across heterogeneous devices; the paper only names inference support as future work.
  • The bit-vector coordination protocol is a general deadlock-avoidance scheme for multi-threaded collectives, but the paper does not report measured coordination overhead or scalability beyond the O(G) argument.
  • The claimed double-digit gains are stated without benchmark tables in this manuscript, so the decisive test is a direct A/B comparison of unmodified training versus the toolchain on identical hardware.
  • MegaScan's aligned traces and MegaDPP's adaptive scheduler are described as separate modules; closing the loop by feeding detected anomalies into schedule changes would be a natural next step that the current design leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents MegatronApp, a proposed toolchain of four optional modules for Megatron-LM: MegaScan (CUDA-event-based tracing and straggler detection), MegaFBD (forward/backward decoupling across heterogeneous resources), MegaDPP (dynamic pipeline scheduling), and MegaScope (interactive visualization and perturbation injection). The manuscript describes the motivation, architecture, and design of each module, and asserts in the abstract and conclusion that the toolchain is production-ready, delivers double-digit throughput gains, reduces peak memory, and imposes near-zero tracing overhead. The paper contains no experimental evaluation, no baseline comparisons, no correctness proof, and no quantitative evidence for any of these performance or reliability claims.

Significance. The problems targeted by the four modules are real and timely: straggler diagnosis, heterogeneous-resource utilization, pipeline-schedule flexibility, and interpretability are all active concerns in large-model training. The paper also provides a useful synthesis of related work in these areas and points to an open-source repository, which is commendable. If the design claims were substantiated, MegatronApp could be a valuable systems contribution. However, the current manuscript is a design proposal only: none of the central claims about efficiency, correctness, or production readiness is supported by measurements or formal arguments, so the significance cannot be assessed from the submitted text.

major comments (4)
  1. [Abstract and §7] The manuscript asserts double-digit gains in throughput and cluster utilization, near-zero tracing overhead, and production readiness, but it contains no experimental section, no benchmarks, no hardware description, no workload traces, and no comparisons against vanilla Megatron-LM or existing tools. These quantitative claims appear in the Abstract and in the Conclusion (§7) without any supporting measurement. Because the central contribution is a systems toolchain, the absence of evaluation is load-bearing and prevents verification of the paper's main claims.
  2. [§4.2 and Figure 2] MegaFBD's correctness is not established. Section 4.1 states that the forward pass may use "a smaller parallel degree" than the backward pass, yet Section 4.2 states that forward and backward instances have the same number of virtual ranks and that model partitioning follows Megatron's original allocation rules. Under tensor or pipeline parallelism, different parallel degrees imply different sharding of weights and activations, so the backward phase cannot consume forward-phase tensors without an explicit layout transformation. The communication coordinator's bit-vector readiness protocol only verifies that participants post the same collective request; it does not address tensor redistribution, optimizer-state handoff, or how gradient AllReduce groups are formed across differing physical rank sets. No proof, simulation, or experiment shows that gradients, optimizer state, and pipeline semantics match vanilla Megatron-LM. If this equivalence fails, MegaFBD would silently corrupt training or hang, invalidating the claimed throughput and memory benefits.
  3. [§3.2] The claim that MegaScan imposes "near-zero overhead" and "negligible performance impact" is asserted without measurement. CUDA-event injection, metadata attachment, per-rank trace generation, rank-0 gathering, and persistence to disk all consume host and I/O resources even if asynchronous. Furthermore, the timeline-alignment procedure uses collective-operation finish times as anchor points, but a late-finishing rank can shift the anchor for all participants, so the reconstruction may systematically misattribute delays. The paper provides no experiment quantifying tracing overhead or validating alignment accuracy, yet these points underlie the root-cause-diagnosis claims.
  4. [§5.1 and §5.2] MegaDPP's dynamic traversal switching is described only at the level of a schedule matrix. The text says users can "flexibly alternate" between depth-first and breadth-first strategies, but it does not specify a protocol for safe runtime switching between schedules within an iteration, nor does it explain how the number of in-flight microbatches, activation lifetimes, or gradient-synchronization ordering are preserved when the traversal changes. Without a correctness argument or experimental demonstration, the claimed "decisive breakthrough" over static schedules is unsupported.
minor comments (6)
  1. [Throughout] The paper repeatedly uses terms like "near-zero overhead," "double-digit gains," and "production-ready" without defining the metrics or the evaluation methodology; these should either be removed or made precise.
  2. [§2.2] Reference [16], the parameter-server paper, is cited as the basis of data parallelism; a more standard reference such as Megatron-LM or a data-parallel training survey would be more appropriate.
  3. [§5.1] The text mentions "PipeDream-2BW," "ZB1P," and "BitPipe" without providing references or full names; these should be cited or removed.
  4. [§6.1] BertViz is mentioned but not cited; please add the reference or omit the name.
  5. [References] Several references are incomplete: [25], [26], [30], and [32] are listed as "arXiv preprint" without arXiv identifiers, and [33] lacks a venue description. The repository URL in the Abstract is broken across a line break and should be typeset as a single link.
  6. [Figures 2–6] The figures are not referenced precisely in the text at the points where the corresponding modules are described, and Figure 2's distinction between baseline and MegaFBD deployments would benefit from explicit labels on both sides of the diagram.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a systems design description with no fitted parameters, no derived equations, and no load-bearing self-citations.

full rationale

MegatronApp does not derive any numerical result from a fitted input, so the standard circularity patterns do not apply. The paper is a design narrative for four optional modules, and its efficiency claims (e.g., 'double-digit gains in throughput and cluster utilization') are asserted without experiments rather than derived from the modules' own definitions. There is no equation-level reduction, no parameter fitted to a subset of data and then renamed as a prediction, and no uniqueness theorem imported from the authors' prior work. The background citations, including Megatron-LM, DeepSpeed, and pipeline-scheduling literature, are independent prior work and are not used to justify the central contribution in a load-bearing way. The closest issue is that MegaFBD's semantic preservation under different forward/backward parallel configurations is argued by description rather than proven or benchmarked; however, that is a validation or correctness gap, not circularity, because the paper does not define the claimed result in terms of the assumption. Similarly, MegaScan's slow-node heuristic defines a detection criterion and then applies it, which is a design choice rather than a self-referential derivation of an external prediction. Overall, the paper is self-contained as a proposal, and its weaknesses are lack of empirical support, not circular reasoning.

Assumptions & free parameters 0 free parameters · 5 assumptions · 4 invented entities

The central claims rest on five unbenchmarked domain assumptions, primarily about CUDA-event timing fidelity, DP-peer homogeneity, the slowest-member root-cause heuristic, semantic preservation under forward-backward splitting, and the benefits of dynamic traversal. There are no fitted numerical parameters because there is no data fitting anywhere in the paper. The four modules are the paper's invented entities and have no independent evidence beyond the design text; the claimed GitHub repository is not verifiable from the preprint.

assumptions (5)
  • domain assumption CUDA Event start/end markers provide accurate operator-level execution times with negligible overhead.
    MegaScan's tracing and anomaly detection in Section 3.2 rely on this premise; no measurements or error analysis are given.
  • domain assumption Data-parallel peers with identical PP/TP indices execute identical compute kernel sequences.
    The cross-DP slow-kernel comparison in Section 3.2 assumes this homogeneity; load imbalance or kernel-level nondeterminism would break it.
  • domain assumption The true fault source appears as the slowest member in every synchronous collective it joins.
    This heuristic in Section 3.2 is the basis for root-cause identification; no formal or empirical validation is provided.
  • domain assumption Forward and backward passes can be split onto distinct ranks with different parallel degrees without changing training semantics.
    MegaFBD's design in Section 4 relies on this premise; the paper provides no correctness proof or convergence test.
  • domain assumption Dynamic BFC/DFC traversal and async P2P transfers improve throughput without causing out-of-memory failures.
    MegaDPP's claimed gains in Section 5 depend on memory and bandwidth models that are described qualitatively, not evaluated.
invented entities (4)
  • MegaScan
    purpose: CUDA-event-driven tracing and slow-node detection at operator granularity.
    Proposed as a new module; no traces, overhead measurements, or external validation are included.
  • MegaFBD
    purpose: Decouple forward and backward passes across virtual and physical ranks for heterogeneous resource use.
    Proposed as a new module; no experiments or correctness checks validate the described decoupling.
  • MegaDPP
    purpose: Dynamic pipeline scheduling with BFC/DFC traversal switching and async P2P communication.
    Proposed as a new module; no benchmarks or reproduced schedules support the claimed throughput gains.
  • MegaScope
    purpose: Low-overhead capture, compression, and interactive visualization of intermediate tensors with perturbation injection.
    Proposed as a new module; no I/O overhead measurements or case studies are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training." pith.science (2026). https://pith.science/paper/WPOFTWJD

@misc{pith2026250719845,
  author       = {Pith},
  title        = {Pith review of: MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPOFTWJD}},
  note         = {Machine review of arXiv:2507.19845}
}
read the original abstract

The rapid escalation in the parameter count of large language models (LLMs) has transformed model training from a single-node endeavor into a highly intricate, cross-node activity. While frameworks such as Megatron-LM successfully integrate tensor (TP), pipeline (PP), and data (DP) parallelism to enable trillion-parameter training, they simultaneously expose practitioners to unprecedented systems-level challenges in performance optimization, diagnosis, and interpretability. MegatronApp is an open-source toolchain expressly designed to meet these challenges. It introduces four orthogonal, yet seamlessly composable modules--MegaScan, MegaFBD, MegaDPP, and MegaScope--that collectively elevate the reliability, efficiency, and transparency of production-scale training. This paper presents the motivation, architecture, and distinctive contributions of each module, and elucidates how their synergistic integration augments the Megatron-LM ecosystem.

Figures

Figures reproduced from arXiv: 2507.19845 by the authors.

Figure 1
Figure 1. MegaScan visualizes the trace file using Chrome Tracing (or Perfetto UI). Workload tracing. MegaScan adopts a timing mechanism based on CUDA Events. A CUDA event is a special marker, injected by the host into a CUDA stream, that is essentially an empty kernel. When the GPU’s execution flow reaches this marker, it records the current timestamp. By inserting a start event and an end event in the same CUDA stream immed… view at source ↗
Figure 2
Figure 2. Forward/backward instance deployment of MegaFBD compared with that of existing training frameworks. Thread-level workers and virtual ranks. To minimise changes to the original framework, MegaFBD keeps the fact that “the forward and backward phases use different parallel configurations” transparent to the training framework itself. Consequently, MegaFBD maintains two parallel naming systems as shown in [PITH_FULL_IM… view at source ↗
Figure 3
Figure 3. Training models with 8 chunks on 8 microbatches of data require 64 computation tasks per iteration. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: An example of visualizing text generation. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: An example of visualizing attention scores. Users can select the layer and attention head they wish to inspect. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: An example of visualizing the PCA dimensionality reduction feature.Users can visually inspect the clustering [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 30 canonical work pages

  1. [1]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Route sparse autoencoder to interpret large language models, 2025

    Wei Shi, Sihang Li, Tao Liang, Mingyang Wan, Guojun Ma, Xiang Wang, and Xiangnan He. Route sparse autoencoder to interpret large language models, 2025

    work page 2025

  3. [3]

    Interpreting learned feedback patterns in large language models, 2024

    Luke Marks, Amir Abdullah, Clement Neo, Rauno Arike, David Krueger, Philip Torr, and Fazl Barez. Interpreting learned feedback patterns in large language models, 2024

    work page 2024

  4. [4]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023

    2023

  5. [5]

    Tracr: Compiled transformers as a laboratory for interpretability, 2023

    David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability, 2023

    work page 2023

  6. [6]

    Zimmermann, David Klindt, and Wieland Brendel

    Roland S. Zimmermann, David Klindt, and Wieland Brendel. Measuring per-unit interpretability at scale without humans. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. Paper ID 56ed2bd15b66f709cd81cb1aaa0496b9, official BibTeX pending

    work page 2024

  7. [7]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  8. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 42 references
  1. [9]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  2. [10]

    Visualizing attention in transformer-based language representation models

    Jesse Vig. Visualizing attention in transformer-based language representation models. CoRR, abs/1904.02679, 2019

    1904 arXiv

  3. [11]

    Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunat...

  4. [12]

    Developments in MLflow: A system to accelerate the machine learning lifecycle

    Andrew Chen, Andy Chow, Aaron Davidson, Arjun D’Cunha, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Clemens Mewald, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, Avesh Singh, Fen Xie, Matei Zaharia, Richard Zang, Juntai Zheng, and Corey Zumar. Developments in MLfl...

    2020

  5. [13]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedin...

    2021

  6. [14]

    Reducing activation recomputation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023

    2023

  7. [15]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  8. [16]

    Scaling distributed machine learning with the parameter server

    Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ...

    2014

  9. [17]

    Bandwidth optimal all-reduce algorithms for clusters of workstations

    Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. Journal of Parallel and Distributed Computing, 69(2):117–124, 2009

    2009

  10. [18]

    Gpipe: Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019

    2019

  11. [19]

    Pipedream: Generalized pipeline parallelism for dnn training

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, p...

    2019

  12. [20]

    Memory-efficient pipeline- parallel dnn training

    Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory-efficient pipeline- parallel dnn training. In International Conference on Machine Learning, pages 7937–7947. PMLR, 2021

    2021

  13. [21]

    Deepspeed, 2024

    Microsoft. Deepspeed, 2024. https://github.com/microsoft/DeepSpeed

    2024

  14. [22]

    Zero-offload : Democratizing billion-scale model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload : Democratizing billion-scale model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564, 2021

    2021

  15. [23]

    Pytorch fsdp: experiences on scaling fully sharded data parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023

    2023 arXiv

  16. [24]

    Explainability for large language models: A survey

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. arXiv preprint, 2023

    2023

  17. [25]

    From understanding to utilization: A survey on explainability for large language models

    Haoyan Luo and Lucia Specia. From understanding to utilization: A survey on explainability for large language models. arXiv preprint, 2024

    2024

  18. [26]

    A gentle introduction to mechanistic interpretability of neural networks

    Neel Nanda, Collin Burns, Lawrence Chan, et al. A gentle introduction to mechanistic interpretability of neural networks. arXiv preprint, 2023

    2023

  19. [27]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Proceedings of the International Conference on Learning Representations (ICLR), 2023

    2023

  20. [28]

    Mass editing memory in a transformer

    Kevin Meng, Max Nadeau, Alex Andonian, and David Bau. Mass editing memory in a transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  21. [29]

    Advances of pipeline model parallelism for deep learning training: An overview

    Lei Guan, Dong-Sheng Li, Ji-Ye Liang, Wen-Jian Wang, Ke-Shi Ge, and Xi-Cheng Lu. Advances of pipeline model parallelism for deep learning training: An overview. Journal of Computer Science and Technology , 39(3):567–584, May 2024

    2024

  22. [30]

    Dawnpiper: A memory-scalable pipeline parallel training framework

    Xuan Peng, Xuanhua Shi, Haolin Zhang, Yunfei Zhao, and Xuehai Qian. Dawnpiper: A memory-scalable pipeline parallel training framework. arXiv preprint, 2025

    2025

  23. [31]

    Zero bubble (almost) pipeline parallelism

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. Zero bubble (almost) pipeline parallelism. InProceedings of the International Conference on Learning Representations (ICLR), 2024. Poster

    2024

  24. [32]

    Understanding stragglers in large model training using what-if analysis

    Jinkun Lin, Ziheng Jiang, Zuquan Song, Sida Zhao, Menghan Yu, Zhanghan Wang, Chenyuan Wang, Zuocheng Shi, Xiang Shi, Wei Jia, Zherui Liu, Shuguang Wang, Haibin Lin, Xin Liu, Aurojit Panda, and Jinyang Li. Understanding stragglers in large model training using what-if analysis....

    2025

  25. [33]

    DPro-SM: A distributed framework for proactive straggler mitigation using LSTM

    Aswathy Ravikumar and Harini Sriraman. DPro-SM: A distributed framework for proactive straggler mitigation using LSTM. Heliyon, 10(1):e23567, 2024

    2024

  26. [34]

    Greyhound: Hunting fail-slows in hybrid-parallel training at scale

    Tianyuan Wu, Wei Wang, Yinghao Yu, Siran Yang, Wenchao Wu, Qinkai Duan, Guodong Yang, Jiamang Wang, Lin Qu, and Liping Zhang. Greyhound: Hunting fail-slows in hybrid-parallel training at scale. In Proceedings of the 2025 USENIX Annual Technical Conference (USENIX ATC ’25), pag...

    2025

  27. [35]

    FlashFlex: Accommodating large language model training over heterogeneous environment

    Ran Yan, Youhe Jiang, Wangcheng Tao, Xiaonan Nie, Bin Cui, and Binhang Yuan. FlashFlex: Accommodating large language model training over heterogeneous environment. arXiv preprint, 2024

    2024

  28. [36]

    NVIDIA CUDA gets RISC-V support

    Chris Mellor. NVIDIA CUDA gets RISC-V support. https://www.theregister.com/2025/07/21 /nvidia_cuda_riscv/, 2025. Accessed: 2025-07-23

    2025

  29. [37]

    Varuna: scalable, low-cost training of massive deep learning models

    Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: scalable, low-cost training of massive deep learning models. In Proceedings of the Seventeenth European Conference on Computer Systems, pages 472–487, 2022

    2022

  30. [38]

    NVIDIA Data Center GPU Manager (DCGM)

    NVIDIA Corporation. NVIDIA Data Center GPU Manager (DCGM). https://developer.nvidia.com /dcgm, 2024. Version 3.x, Accessed: 2025-07-25. 14 MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training

    2024

  31. [39]

    CUDA Toolkit Documentation

    NVIDIA Corporation. CUDA Toolkit Documentation. NVIDIA, 2024. Version 12.x, Accessed: 2025-07-25

    2024

  32. [40]

    Chrome Tracing

    Google Inc. Chrome Tracing. https://www.chromium.org/developers/how-tos/trace-eve nt-profiling-tool/, 2013. Accessed: 2025-07-25

    2013

  33. [41]

    Perfetto Open-Source Tracing Project

    Google Inc. Perfetto Open-Source Tracing Project. https://perfetto.dev, 2024. Accessed: 2025-07-25. 15

    2024

  34. [2015]

    Software available from tensorflow.org

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.