Pith. sign in

REVIEW 5 major objections 6 minor 48 references

H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims its H2 system trains a 100-billion-parameter LLM on a cluster of 1,024 chips spanning four architectures faster than the same chips used as separate homogeneous jobs, with up to 16.37 percent surplus throughput.

desk verdict Useful heterogeneous-training engineering, but the superlinear claim is undercut by the paper's own same-GBS runs. read the letter →

arxiv 2505.17548 v1 pith:6LJO423G submitted 2025-05-23 cs.DC

Ding Tang , Jiecheng Zhou , Jiakai Hu , Shengwei Li , Huihuang Zheng , Zhilin Pei , Hui Wang , Xingcheng Zhang This is my paper · ORCID
classification cs.DC
keywords hyper-heterogeneousclusterlargelanguagemodeltrainingpipelineparallelismdevice-directRDMAautomaticstrategysearchactivationreshardingheterogeneoustokensperchipsecond
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to establish that a single heterogeneous training system can outperform the homogeneous baseline that treats each chip type as a separate cluster. It proposes H2, which combines a unified tensor-programming interface, a device-direct RDMA communication path, and an automatic search over pipeline stage sizes, layer shards, and recomputation settings. In evaluations on a 100-billion-parameter model, the mixed cluster reaches a HeteroSpeedupRatio above 100 percent in several configurations, up to 116.37 percent, meaning it produces more tokens per second than the sum of its parts run separately. If true, organizations could stop waiting for homogeneous allocations and train large models on whatever diverse chips they already own.

What carries the argument

The central object is the HeteroSpeedupRatio, defined as $N\cdot TGS \,/\, \sum_{i=1}^{C} N_i \cdot TGS_i$, where $TGS$ is training throughput in tokens per chip per second for the heterogeneous run, $N$ is the total chip count, $N_i$ is the number of chips of type $i$, and $TGS_i$ is that chip type's throughput measured on a 256-chip homogeneous run. A ratio above 100 percent is the claimed superlinear speedup. The mechanism that makes this ratio exceed 100 percent is HeteroAuto's cost model, which minimizes an estimated iteration time over pipeline stage counts per chip type, tensor-parallel degrees, layer assignments, and recomputation flags, subject to per-chip memory limits; combined with the memory-descending pipeline ordering, this lets memory-rich chips absorb early-stage memory pressure while compute-rich but memory-poor chips avoid expensive tensor-parallel widening and CPU offload.

What would settle it

Reproduce one reported configuration, such as Exp-C-2, and run the same 100B model for the same number of tokens twice: once as the H2 mixed cluster and once as the constituent homogeneous clusters with each cluster's optimized 3D parallel configuration, matching total global batch size and data. If the summed token throughput of the homogeneous runs equals or exceeds the mixed run's throughput, the superlinear claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that hyper-heterogeneous training at scale is not only feasible but can be superlinearly faster than homogeneous training on the same chips. H2 achieves this with three coordinated pieces: DiTorch supplies a single tensor-programming interface that accepts an existing training program with a one-line change and aligns numerical precision across chips; DiComm implements device-direct RDMA between chips of different vendors, reducing measured point-to-point latency by an average of 9.94 times relative to CPU-mediated TCP; and HeteroPP is a pipeline-parallel scheme in which each stage uses one chip type, chips are ordered by memory from large to small along the pipeline, layers are distributed unevenly, and stage-level tensor-parallel, data-parallel, and recomputation settings are free. HeteroAuto searches that space with a profiled cost model and depth-first enumeration, and a topology-aware activation resharding plus fine-grained P2P overlap cuts cross-node transfer costs. The quantitative payoff is a HeteroSpeedupRatio above 100 percent, up to 116.37 percent, on 100B-parameter training runs with up to 1,024 chips.

Load-bearing premise

The superlinear claim assumes that the tokens-per-second each chip type delivers on its own 256-chip run is a fair baseline for what the same chips contribute inside the 768-to-2,432-chip mixed run; if throughput per chip shifts with cluster size or global batch size, the measured ratio is not a real speedup.

Editorial extensions

If this is right

  • A mixed 1,024-chip run can deliver more tokens per second than the same chips partitioned into four homogeneous jobs, reaching 89.56 percent to 116.37 percent of the homogeneous sum depending on chip mix and global batch size.
  • Putting larger-memory chips in early pipeline stages lets memory-limited fast chips drop costly CPU offload or high tensor-parallelism, matching each stage's work to the chip's actual bottleneck.
  • Replacing CPU-mediated TCP cross-chip transfers with device-direct RDMA reduces measured point-to-point latency by an average of 9.94 times, with the largest gains at large message sizes.
  • Automatic strategy search completes in tens of seconds (0.62 to 12.29 seconds in the reported configurations), while comparable heterogeneous search tools are cited at minutes to hours for far smaller clusters.
  • A one-line modification of an existing training program is enough to execute it across all four chip types, because operator-level precision alignment holds each chip's loss within 1.5 percent mean relative error of the reference over 300 training iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a real superlinear ratio would change procurement logic: a fleet built from slower, cheaper chips plus a few large-memory chips could match or beat a homogeneous fleet at the same token throughput, but the paper does not measure price or power, so the economic claim is not established.
  • The ordering principle of memory-rich stages first and memory-poor stages later is a general scheduling heuristic that could apply to any asymmetric memory hierarchy, including single-vendor clusters with different GPU memory sizes, not only the multi-vendor case.
  • The precision check covers 300 iterations of a smaller model; an extension would be to track mean relative error over a full 100B pretraining run to see whether cross-vendor numerical drift accumulates beyond the 1.5 percent threshold.
  • A scale study holding global batch size fixed while varying only cluster size would separate the superlinear effect from the effect of using a larger global batch in the mixed run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents H2, a framework for training a 100B-parameter LLM on a hyper-heterogeneous cluster of more than 1,000 chips spanning four accelerator types. The system has three main components: DiTorch, a PyTorch-compatible unified interface with precision alignment; DiComm, an RDMA-based communication library with device-direct transfer; and HeteroPP with HeteroAuto, a heterogeneous pipeline-parallelism strategy with an automatic search for layer sharding, tensor-parallel sizes, and recomputation settings. The central evaluation claim, stated in the abstract and Section 1, is that H2 ``consistently achieves a superlinear speedup'' over homogeneous baseline training, by up to 16.37%, measured by the HeteroSpeedupRatio defined in Section 6.2.

Significance. If the headline claim were supported, the paper would make a strong practical contribution: it demonstrates a working path to integrating chips from multiple vendors with incompatible software stacks into one large-scale LLM training run, and it does so at a scale (over 1,000 chips) that is uncommon in the heterogeneous-training literature. The engineering strengths are real and should be credited: open-source components DiTorch and DIOPI are released, the search overhead is very low (Table 8), the ablation in Table 9 isolates the contribution of device-direct RDMA, topology-aware activation resharding, and fine-grained overlap, and the precision-alignment results in Section 3.1.2 provide a concrete, falsifiable acceptance criterion. However, the central performance claim is not supported by the paper's own data: the same-batch-size experiments in Figure 11 fall below 100% of the baseline, and all configurations that exceed 100% use a 3-4x larger global batch size than the baseline. The contribution is therefore defensible as a systems-building and evaluation paper only if the claims are substantially weakened and the baseline is re-measured or re-defined.

major comments (5)
  1. [Abstract and Section 6.2 (Figure 11)] The claim that H2 ``consistently achieves a superlinear speedup'' is directly contradicted by the paper's own measurements. Figure 11 reports HeteroSpeedupRatio values of 89.56% for Exp-A-1 and 77.45% for Exp-B-1, both below 100%, and the text in Section 6.2.1 explicitly acknowledges these values. The headline statement must be weakened to apply only to the configurations with enlarged global batch sizes, or the claim of consistency must be removed.
  2. [Section 6.2, Eq. (HeteroSpeedupRatio), Table 6 and Table 7] The HeteroSpeedupRatio baseline is not comparable to the heterogeneous runs because the global batch size differs. Table 6 measures per-chip TGS on 256-chip homogeneous runs with a global batch size of 2M tokens, while every configuration that exceeds 100% in Figure 11 uses GBS=6M or 8M (Exp-A-2, Exp-B-2, Exp-C-2, Exp-D) or GBS=4M (Exp-C-1). Since the paper uses a 1F1B pipeline schedule (Section 4.3.2, alpha=1), a larger GBS increases the number of micro-batches and reduces the pipeline bubble fraction even without any heterogeneity benefit. The two controlled same-GBS experiments, Exp-A-1 and Exp-B-1, reach only 89.56% and 77.45%, respectively. Thus the ratio as defined does not isolate the effect of H2; the apparent superlinearity is confounded with the batch-size effect. The authors should either re-measure homogeneous baselines at the same GBS as the heterogeneous runs, or report matched-GBS HeteroSpeedupRatio as the primary result.
  3. [Section 6.2.1 (paragraph on superlinear performance)] The explanation offered for the superlinear results is descriptive rather than quantitative. The text argues that Chip B's memory limits it to TP>=8 or recomputation, while Chip A has more memory, so H2 places Chip A in early pipeline stages and Chip B in later stages. However, no decomposition is provided to show how much of the measured gain comes from this reallocation versus from the larger global batch size, the different per-stage parallel configurations, or the improved communication stack. To support the central claim, the authors should provide a controlled comparison that holds GBS fixed and, if possible, isolates the contribution of the HeteroAuto-chosen strategy from the contribution of DiComm and the resharding optimizations.
  4. [Section 6.2 and Table 6 in relation to Section 6.1.2] The HeteroSpeedupRatio definition assumes that the per-chip TGS values measured on 256-chip runs are representative and linearly composable when the same chips are placed in 768- to 2,432-chip heterogeneous clusters. This assumption is not justified. Training throughput per chip can change with cluster size, communication topology, memory sharing, and pipeline depth, and the paper provides no evidence that the Table 6 values remain valid at the larger scales. The authors should either validate this linear-composability assumption or define the baseline at the same cluster configuration and GBS as the heterogeneous experiments.
  5. [Section 4.3.3 (two-stage search)] The second search stage treats every set of 128 homogeneous chips as a single heterogeneous group, which is a modeling assumption that prunes the search space. This is reasonable as a heuristic, but the paper does not report the sensitivity of the final strategy or the resulting throughput to this group size. Since the group size is a free parameter of the search, a brief sensitivity study would strengthen the claim that HeteroAuto's output is robust rather than tuned to this particular choice.
minor comments (6)
  1. [Section 6.2.1] The text contains the typo ``DiTrain achieves'' where DiTrain is not introduced elsewhere; this should read ``H2'' or ``HeteroPP.''
  2. [Section 6.1.2, text above Table 6] The sentence ``Figure 6 presents the training throughput...'' is incorrect; the throughput data appear in Table 6 and Figure 11, not in Figure 6, which shows communication latency comparisons. Figure and table cross-references should be corrected throughout the paper.
  3. [Table 8] The third row of Table 8 contains a corrupted string of private-use Unicode characters with no readable content. This must be fixed before publication.
  4. [Section 2.2 and Section 3.1.1] The paper cites reference [16] and [17] with identical content, and the reference list contains several duplicated entries (e.g., the Ang Li et al. GPU interconnect paper appears twice). The bibliography should be deduplicated.
  5. [Section 4.3.2, Eq. (cost model)] The bubble coefficient alpha is described as configurable and set to 1 for 1F1B, but the value of alpha depends on the pipeline schedule. Since the cost model is used to compare strategies across schedules, the paper should state explicitly whether all experiments use the same schedule and same alpha, and how alpha is estimated in practice.
  6. [Section 6.2.2] The comparison of search overhead with Metis and Alpa is presented without elaboration on hardware, model size, or search-space details, so the reported values (600 seconds for Metis, 240 minutes for Alpa) are not directly comparable to the H2 search times in Table 8. A one-sentence caveat about the comparability of these numbers would be appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported speedup is an empirical measurement against independent homogeneous baselines, and self-citations are peripheral rather than load-bearing.

full rationale

The central claim is an empirically measured throughput ratio, not a quantity derived from its own definition. HeteroSpeedupRatio in Section 6.2 compares measured heterogeneous-cluster TGS to the sum of per-type baselines independently measured on 256-chip homogeneous runs reported in Table 6, so the numerator and denominator come from separate experiments; the ratio is not forced to any particular value by construction. HeteroAuto's cost model in Section 4.3.2 is populated with profiled layer-wise times and used only to select a parallelism configuration, while the reported TGS values in Figure 11 and Table 9 are end-to-end measurements rather than read-backs of the cost model's estimate. Self-citations do appear: ZeroPP [33] is mentioned only as a compatible scheduling option, and DiTorch/DIOPI repositories are implementation components. None is load-bearing for the speedup claim: the large-scale evaluation uses the 1F1B schedule rather than ZeroPP, and the repositories are software artifacts rather than evidence establishing the measured ratio. The global-batch-size mismatch between the 2M-token homogeneous baselines and the 6-8M-token heterogeneous runs is a legitimate benchmarking-validity concern that could weaken the superlinear claim, but it is a confound rather than circularity because the baseline values remain independent of the heterogeneous result. Therefore, no circular step can be identified, and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's contribution is empirical, so the ledger captures the modeling choices in HeteroAuto's cost model, the batch-size settings used in the evaluation, and the assumption that 256-chip homogeneous TGS baselines scale to larger clusters. No new physical entities are introduced; the software components are engineering artifacts, not postulated entities.

free parameters (4)
  • alpha (bubble coefficient) = 1
    Set to 1 because the paper uses the 1F1B pipeline schedule; the authors state the resulting cost formula is equivalent to Metis. This modeling choice defines the optimum but is not fitted to data.
  • Global batch size per experiment = 2M, 6M, or 8M tokens
    GBS differs per experiment and partly determines whether HeteroSpeedupRatio is below or above 100%; the favorable 'sum of batch sizes' configurations are highlighted in the abstract.
  • Two-stage search group size = 128 homogeneous chips
    Section 6.2.2 states that every set of 128 homogeneous chips is treated as a single group in the second search stage; this hand-chosen grouping changes search overhead and the resulting strategy.
  • MRE threshold for precision alignment = 1.5%
    Section 3.1.2 defines precision alignment as mean relative error of training loss over 300 iterations below 1.5%; this is an author-chosen acceptance criterion, not derived.
assumptions (5)
  • domain assumption Chips of the same type are homogeneous and each pipeline stage is composed of chips of a single type.
    Observation #1 and Section 4.2 assign each pipeline stage to one chip type; if nodes of the same chip type differ in topology or software, the design may not hold.
  • domain assumption Layer-wise compute time is additive and independent across pipeline stages in the cost model.
    Section 4.3.2 computes T_comp and T_update as layer count times profiled per-layer times, with no interaction or contention terms between stages.
  • domain assumption Pipeline bubble time is proportional to single-microbatch computation time with a constant coefficient alpha.
    Section 4.3.2 states that bubble time can be considered proportional to computation time of a single microbatch, which is a simplification of the actual schedule.
  • domain assumption Per-chip TGS measured on 256-chip homogeneous clusters remains representative when the same chips are combined at 768-to-2,432-chip scale.
    Section 6.2 defines HeteroSpeedupRatio using Table 6 baseline TGS values; if throughput does not scale linearly with chip count and batch size, the ratio is not a valid speedup.
  • ad hoc to paper Homogeneous chips can be regrouped into arbitrary 128-chip groups for the second stage of HeteroAuto.
    Section 6.2.2 introduces the 128-chip grouping to manage search cost; the grouping size is not derived from hardware or model properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips." pith.science (2026). https://pith.science/paper/6LJO423G

@misc{pith2026250517548,
  author       = {Pith},
  title        = {Pith review of: H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6LJO423G}},
  note         = {Machine review of arXiv:2505.17548}
}
read the original abstract

Recent advancements in large language models (LLMs) necessitate extensive computational resources, prompting the use of diverse hardware accelerators from multiple vendors. However, traditional distributed training frameworks struggle to efficiently utilize hyper-heterogeneous clusters comprising thousands of chips due to significant disparities in software stacks, operator implementations, communication libraries, and hardware capabilities. To address these challenges, we propose H2, which stands for HyperHetero and is a systematic framework enabling efficient training of LLMs on clusters with over 1,000 heterogeneous chips. H2 incorporates DiTorch, a unified PyTorch-compatible interface ensuring program consistency across chips, and DiComm, a device-direct RDMA communication library optimized for heterogeneous environments. Furthermore, we introduce HeteroPP with HeteroAuto, an adaptive pipeline parallelism strategy that dynamically balances computational load, memory limitations, and communication overhead. Evaluations on a 100-billion-parameter LLM demonstrate that our approach consistently achieves a superlinear speedup, outperforming baseline homogeneous training solutions by up to 16.37% in our experiments. These findings validate the feasibility and efficiency of hyper-heterogeneous training at unprecedented scales.

Figures

Figures reproduced from arXiv: 2505.17548 by the authors.

Figure 1
Figure 1. Comparison of chip specifications between capability-incremental and our hyper-heterogeneous sce￾nario.In traditional heterogeneous scenarios, as indicated by the black dashed circles in the figure, chips show a trend of increasing capabilities in computation, communication, and memory. In contrast, in hyper-heterogeneous scenarios, as indicated by the red dashed circles in the figure, the capabili￾ties of chips in … view at source ↗
Figure 2
Figure 2. first, users provide the training data and model to the pre-training framework, which efficiently implements data processing and parallelization strategies through its dis￾tributed design. Then, these distributed pre-training frame￾works employ PyTorch’s interfaces to realize the parallel strategies. Finally, PyTorch leverages the high-performance computing and communication libraries specific to each chip to execut… view at source ↗
Figure 3
Figure 3. Intra-Node Bandwidth Performance in Different GPU Servers model training and experimentation. Compared to homoge￾neous clusters and standard heterogeneous environments, Hyper-Heterogeneity has the following characteristics: Distinct Software Stacks for Each Vendor’s Chip: Due to the "isolation" between hardware design and the corre￾sponding software stacks, the same operator implemented on different chips can produc… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: System Overview of DiTorch and DiComm. 3 Breaking Software and Communication Barrier To address the software and communication isolation chal￾lenges outlined previously, we introduce DiTorch and Di￾Comm in this section, whose system overview are shown in [PITH_FULL_IM…
Figure 5
Figure 5. Figure 5: DiTorch’s precision alignment across Chips A, B, C, and D compared to the NVIDIA A100 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Comparison of cross Chip-Communication latency with different strategies. 3.2.1 DiComm Performance. DiComm successfully en￾ables P2P communication between different chips using the device-direct strategy. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: A simple example of HeteroPP. In this setup, two types of chips are used, and the model consisting of 18 layers is partitioned into three pipeline stages. At each pipeline stage, different configurations for data parallelism, tensor parallel dimensions, layer sharding …
Figure 9
Figure 9. Figure 9: Overview of HeteroAuto. Moreover, to achieve balanced pipeline workloads due to the Observation #3 and fully exploit the computing, commu￾nication, and memory capabilities of the chips, our system supports non-uniform task distribution in three key aspects. First, we e…
Figure 10
Figure 10. Figure 10: An example of topology-aware send/recv + all￾gather activation resharding with TP size 4 on Chip-A server and TP size 2 on Chip-B server. less promising candidates. This strategy allows for a more nu￾anced exploration of heterogeneous parallelism, ultimately leading t…
Figure 11
Figure 11. Figure 11: Training throughput for individual homogeneous and heterogeneous training setups, and HeteroSpeedupRatio for heterogeneous training setups. 6.2 Evaluations on HeteroPP and HeteroAuto To validate the performance and efficiency of the proposed heterogeneous parallelism …
Figure 12
Figure 12. Figure 12: For end-to-end training of a small-scale 8- decoder-layer model, experiments were conducted with and without DDR. Uniform 1F1B schedule was employed, and the parallelism configuration was set to TP=4, PP=2, and DP=2. Two heterogeneous servers, with eight chips on each…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 23 canonical work pages

  1. [1]

    2016.{TensorFlow}: a system for{Large-Scale} machine learning

    Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016.{TensorFlow}: a system for{Large-Scale} machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16) . 265–283

    work page 2016

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al . 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Wenjun Gao, Kang Guan, Jianzhong Guo, Yongqiang Guo, Zhe Fu, Ying He, Panpan Huang, Jiashi Li, Wenfeng Liang, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yux- uan Liu, Shanghao Lu, Xuan Lu, Xiaotao Nie, Tian Pei, Junjie Qiu, Hui Qu, Zehui Ren, Zhangli Sha, Xuecheng Su,...

    arXiv 2024

  4. [4]

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalam- barkar, Laurent Kirsch, Michael...

    arXiv 2024

  5. [5]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, S...

    arXiv 2024

  6. [6]

    Li-Wen Chang, Wenlei Bao, Qi Hou, Chengquan Jiang, Ningxin Zheng, Yinmin Zhong, Xuanrun Zhang, Zuquan Song, Ziheng Jiang, Haibin Lin, Xin Jin, and Xin Liu. 2024. FLUX: Fast Software- based Communication Overlap On GPUs Through Kernel Fusion. arXiv:2406.06858 [cs.LG]

    arXiv 2024

  7. [7]

    Ping Chen, Wenjie Zhang, Shuibing He, Weijian Chen, Siling Yang, Kexin Huang, Yanlong Yin, Xuan Zhan, Yingjie Gu, Zhuwei Peng, et al

    work page

  8. [8]

    Jack Choquette, Edward Lee, Ronny Krashinsky, Vishnu Balan, and Brucek Khailany. 2021. 3.2 the a100 datacenter gpu and ampere ar- chitecture. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), Vol. 64. IEEE, 48–50

    work page 2021

Show all 48 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

  2. [10]

    Anne C Elster and Tor A Haugdahl. 2022. Nvidia hopper gpu and grace cpu highlights. Computing in Science & Engineering 24, 2 (2022), 95–100

    2022

  3. [11]

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al . 2021. 13 Arxiv, 2025, Ding Tang, Jiecheng Zhou, Jiakai Hu, Shengwei Li, Huihuang Zheng, Zhilin Pei, Hui Wang, and Xingcheng Zhang DAPPLE: A pipelined data ...

    2021

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning. arXiv preprint arXiv:2501.12948 (2025)

    2025 arXiv

  6. [14]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems 32 (2019)

    2019

  7. [15]

    Xianyan Jia, Le Jiang, Ang Wang, Wencong Xiao, Ziji Shi, Jie Zhang, Xinyuan Li, Langshi Chen, Yong Li, Zhen Zheng, et al. 2022. Whale: Efficient giant model training over heterogeneous{GPUs}. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) . 673–688

    2022

  8. [17]

    Ang Li, Shuaiwen Leon Song, Jieyang Chen, Jiajia Li, Xu Liu, Nathan R Tallent, and Kevin J Barker. 2019. Evaluating modern gpu interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect. IEEE Transactions on Parallel and Distributed Systems 31, 1 (2019), 94–110

    2019

  9. [18]

    Dacheng Li, Hongyi Wang, Eric Xing, and Hao Zhang. 2022. Amp: Automatically finding model parallel strategies with heterogeneity awareness. Advances in Neural Information Processing Systems 35 (2022), 6630–6639

    2022

  10. [19]

    Shigang Li and Torsten Hoefler. 2021. Chimera: efficiently training large-scale neural networks with bidirectional pipelines. InProceedings of the International Conference for High Performance Computing, Net- working, Storage and Analysis (St. Louis, Missouri) (SC ’21). Associ...

    2021

  11. [20]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. 2020. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704 (2020)

    2020 arXiv

  12. [21]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al

  13. [22]

    Dejan Milojicic, Paolo Faraboschi, Nicolas Dube, and Duncan Roweth

  14. [23]

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. 2019. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM symposium on operating sys- tems princ...

    2019

  15. [24]

    arXiv preprint arXiv:2412.19437 (2024)

    Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    2024 arXiv

  16. [25]

    NVIDIA. 2025. CUTLASS: CUDA Templates for Linear Algebra Subrou- tines. https://github.com/NVIDIA/cutlass Accessed: 2025-05-11

    2025

  17. [26]

    NVIDIA. 2025. NCCL: Optimized primitives for collective multi-GPU communication. https://github.com/NVIDIA/nccl Accessed: 2025-05- 11

    2025

  18. [27]

    Nathan Otterness and J Anderson. 2020. AMD GPUs as an alternative to NVIDIA for supporting real-time workloads. In Proceedings of the 32nd Euromicro Conference on Real-Time Systems

    2020

  19. [28]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGres- ley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron- lm. In ...

    2021

  20. [29]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al

  21. [30]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

  22. [31]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He

  23. [32]

    Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2024. Zero Bubble (Almost) Pipeline Parallelism. In The Twelfth International Con- ference on Learning Representations . https://openreview.net/forum? id=tuzTN0eIO5

    2024

  24. [33]

    Ding Tang, Lijuan Jiang, Jiecheng Zhou, Minxi Jin, Hengjie Li, Xingcheng Zhang, Zhilin Pei, and Jidong Zhai. 2024. ZeroPP: Unleash- ing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology. arXiv:2402.03791 [cs.DC] https://arxiv.org/abs/2402. 03791

    2024 arXiv

  25. [34]

    InternLM Team. 2023. InternLM: A Multilingual Language Model with Progressively Enhanced Capabilities. https://github.com/InternLM/ InternLM

    2023

  26. [35]

    Ajay Tirumala and Raymond Wong. 2024. Nvidia blackwell platform: Advancing generative ai and accelerated computing. In 2024 IEEE Hot Chips 36 Symposium (HCS) . IEEE Computer Society, 1–33

    2024

  27. [36]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  28. [37]

    Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeong- jae Jeon. 2024. Metis: Fast Automatic Distributed Training on Het- erogeneous{GPUs}. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). 563–578

    2024

  29. [38]

    In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505–3506

  30. [39]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi- billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  31. [40]

    Si Xu, Zixiao Huang, Yan Zeng, Shengen Yan, Xuefei Ning, Quanlu Zhang, Haolin Ye, Sipei Gu, Chunsheng Shui, Zhezheng Lin, et al. 2024. HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models. arXiv preprint arXiv:2405.16256 (2024)

    2024 arXiv

  32. [41]

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating inter-and{Intra-Operator} 14 H2: Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over ...

    2022

  33. [42]

    Yonghao Zhuang, Lianmin Zheng, Zhuohan Li, Eric Xing, Qirong Ho, Joseph Gonzalez, Ion Stoica, Hao Zhang, and Hexu Zhao. 2023. On optimizing the communication of model parallelism. Proceedings of Machine Learning and Systems 5 (2023), 526–540. 15

    2023

  34. [45]

    Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Het- erogeneous GPU Clusters. In 19th USENIX Symposium on Networked Systems Design and Im...

    2022

  35. [46]

    Peng Wu. 2023. Pytorch 2.0: The journey to bringing compiler tech- nologies to the core of pytorch (keynote). In Proceedings of the 21st ACM/IEEE International Symposium on Code Generation and Optimiza- tion. 1–1

    2023

  36. [2018]

    Improving language understanding by generative pre-training. (2018)

    2018

  37. [2019]

    In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)

    Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) . 4171–4186

    2019

  38. [2020]

    In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis

    Zero: Memory optimizations toward training trillion param- eter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 1–16

  39. [2021]

    In 2021 Design, Au- tomation & Test in Europe Conference & Exhibition (DATE)

    Future of HPC: Diversifying heterogeneity. In 2021 Design, Au- tomation & Test in Europe Conference & Exhibition (DATE) . IEEE, 276– 281

    2021

  40. [2024]

    arXiv preprint arXiv:2406.08756 (2024)

    Optimizing large model training through overlapped activation recomputation. arXiv preprint arXiv:2406.08756 (2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.