Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

A Survey of End-to-End Modeling for Distributed DNN Training: Workloads, Simulators, and TCO

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This survey claims that the scattered field of distributed DNN training simulators can be mapped by a fidelity-based taxonomy, and that the map exposes consistent gaps in validation, energy modeling, and cost integration.

desk verdict Useful survey with a solid taxonomy and TCO coverage, but Table VI's incomparable error metrics and a few internal contradictions keep it from being a reliable reference until revised. read the letter →

arxiv 2506.09275 v1 pith:W353UFFM submitted 2025-06-10 cs.DC

Jonas Svedas , Hannah Watson , Nathan Laubeuf , Diksha Moolchandani , Abubakr Nada , Arjun Singh , Dwaipayan Biswas , James Myers
show 1 more author
Debjyoti Bhattacharjee
This is my paper
classification cs.DC
keywords distributedDNNtrainingsimulatorsworkloadrepresentationperformancemodelingtotalcostofownershipcarbonemissionstaxonomyintermediate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey tries to establish that distributed DNN training simulators can be usefully organized along two axes: simulation fidelity (analytical, profiling-based, or execution-based) and workload representation (configuration-based versus operator- or layer-level intermediate representations). It argues that this organization supports informed tool selection and makes visible where the field is underdeveloped, especially validation at scale, operating-energy modeling, and integration of total cost of ownership (TCO) and carbon-emissions models. The practical stakes are real: full-scale training systems are too expensive to prototype, so simulation is the main lever for early design exploration, and a reliable map of simulator capabilities changes which tools designers trust and where they invest. The paper's contribution is the structured comparison itself, not a new simulator.

What carries the argument

The central object is the survey's taxonomy in Figure 5 and its companion tables (Table VI for simulators, Table VII for TCO/emissions models). The taxonomy classifies simulators on a fidelity axis as analytical, profiling-based, or execution-based, and crosses that with a workload-granularity axis (configuration-based, operator-level IR, layer-level IR); the tables then carry the argument by listing each tool's input format, error, target hardware, network model, and scalability, and each cost model's coverage of fabrication, technology, network, storage, node architecture, and workload.

What would settle it

A reader can settle the central comparison claim by returning to the papers behind Table VI's Error entries: if the values are indeed a mix of average error, maximum error, and one minus Pearson correlation, then the column cannot rank simulators, and the survey's stated goal of informed tool selection would need a standardized error metric instead.

Watch

Extended reading notes

Core claim

On the survey's own terms, the discovery is that the field has converged on a common architecture—a workload graph plus separate compute, network, and scheduler models—while diverging sharply in fidelity and abstraction, and that no surveyed simulator covers the whole stack. The comparison tables show profiling-based simulation emerging as the balance point between speed and accuracy, operator-level IRs displacing configuration-based inputs, NVIDIA GPUs dominating target hardware, network modeling remaining mostly analytical with only a few congestion-aware exceptions, and energy and TCO modeling almost entirely absent from simulators while TCO/emissions models stay decoupled in a separate literature. Synthesizing those patterns into one taxonomy is what the survey claims to add.

Load-bearing premise

The load-bearing premise is that the Error column in the main simulator comparison is a single comparable quantity, but its footnotes show the column mixes average errors, maximum errors, and a correlation-based value, so the tables can only guide tool selection if those discrepancies are ignored.

Editorial extensions

If this is right

  • Tool selection can be structured by fidelity class: analytical simulators for fast trend exploration, profiling-based for balanced studies, and execution-based for detailed bottleneck analysis.
  • The shift to operator-level IRs means new simulators should expect to consume or produce graphs like Chakra rather than configuration files.
  • Profiling-based simulators are the natural place to add operating-energy estimation, because their traces already capture compute and communication activity.
  • Validation at scale is the field's main bottleneck, so any simulator claiming large-scale accuracy should be tested against published cluster results.
  • TCO and emissions models should be integrated with simulators; none currently model environmental impact, so a coupled tool would open a new design dimension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the authors leave implicit is a shared benchmark suite with one workload, one hardware target, and one error definition; without it, the Error column cannot support quantitative ranking.
  • Because the survey shows profiling-based simulators already collect execution traces, pairing those traces with component power estimators (which the survey cites as available building blocks) is the shortest path to closing the operating-energy gap.
  • The finding that simulators are NVIDIA-centric implies a high-value extension would be porting one profiling-based simulator to a non-NVIDIA backend; the taxonomy predicts the trace-collection layer, not the core scheduler, is where most of the work sits.
  • If the taxonomy is correct, the field may converge on hybrid simulators that combine analytical compute models with congestion-aware network simulators, since that combination is what the profiling-based category already points toward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This survey reviews distributed DNN training simulators and TCO/emissions models. It proposes a taxonomy of analytical, profiling-based, and execution-driven simulation, cross-cut by workload representation granularity (configuration-based, operator/layer-level IR, machine-level). Section II reviews workload representations, Section III surveys simulators and presents a detailed comparison in Table VI, and Section IV reviews TCO/emissions models with a comparison in Table VII. The paper distills the results into a series of takeaways and claims that the structured comparison supports informed tool selection and identifies research gaps.

Significance. If the comparison tables and taxonomy are accurate, the survey would be a useful reference for researchers in ML systems, computer architecture, and sustainable computing. Its strengths are broad coverage of very recent 2023–2025 systems, a clearly presented taxonomy with a workload-fidelity dimension, explicit side-by-side comparison of simulator attributes, and the attempt to connect distributed training simulation with TCO/emissions modeling. The paper does not ship machine-checked proofs or reproducible code, so its value rests on the correctness and interpretability of its comparative tables. The inconsistencies identified below affect exactly that basis, so the contribution is currently only partially reliable.

major comments (3)
  1. [Table VI and its footnotes] The Error column of Table VI mixes validation statistics that are not commensurable. The footnotes disclose that DistIR's 0.071 is computed as (1 − Pearson correlation) and is not a direct error metric, while entries marked with the double-dagger footnote (FlexFlow, AMPeD, ATLAHS, Multiverse) are maximum errors and the remaining entries appear to be average errors. A maximum error is always at least as large as an average error, and a correlation-derived score is in different units, so entries such as Calculon's 0.0365 and DistSim's 0.04 cannot be meaningfully compared across rows. Additionally, the table does not record the validation workload or hardware scale per row, so even rows using the same metric are not readily comparable. Since the abstract promises 'comprehensive comparison tables' that 'support informed decision-making,' this column undercuts the table's central purpose. Please split the Error column by metric type, add validation conditions, or remove the column and discuss accuracy narratively.
  2. [Section III-C vs. Table VI] Section III-C, first bullet, groups Calculon with DistIR and Deepflow as simulators that 'employ IR-based inputs,' but Table VI lists Calculon's Input Format as 'cfg-based,' and Figure 5 places Calculon under configuration-based workload fidelity. This is a direct text-table contradiction on a taxonomy assignment, which is one of the paper's main contributions. The authors should determine which classification is correct and align Section III-C, Figure 5, and Table VI.
  3. [Takeaway III.4 vs. Table VI] Takeaway III.4 states that SimAI and MultiVerse 'have validated their results on 1024-node A100 GPU clusters,' but Table VI lists MultiVerse's Validation as '1024xH100' (not A100) and SimAI's Validation as '1024xA100, 1024xH100'; the notation appears to denote GPU counts rather than node counts. This takeaway is used to single out the strongest large-scale validation in the field, so the discrepancy and the GPU/node distinction should be corrected.
minor comments (6)
  1. [Table II and Figure 3] Table II lists 'XLA HLO' with reference [21], but Figure 3 attributes XLA HLO to reference [93], and reference [21] is StableHLO; the reference should be corrected.
  2. [Section II.B.1] The phrase 'profiling-based simulators 5' contains a stray superscript '5' with no corresponding footnote.
  3. [Figure 3 and Section II.B.1] The text 'Y AML' appears twice in Figure 3 and once in Section II.B.1; it should read 'YAML.'
  4. [Table VI and text] Tool names are used inconsistently: 'LLM Compass' in Table VI versus 'LLMCompass' in the text, and 'vtrain' in Table VI versus 'vTrain' in the text.
  5. [Section III-C] The first bullet of Section III-C reads 'Profiling-based simulators, a balance between fidelity and speed,' which is missing a verb; it should read 'strike a balance' or similar.
  6. [Table V] Table V's Workload row for Execution-driven lists Machine-level IR, but neither execution-driven simulator in Table VI (LLMCompass, LLMServingSim) uses a machine-level workload format; if Table V is an idealization rather than a property of the surveyed tools, the caption should say so.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; minor self-citations (AMPeD, STCO, imec.netzero) are not load-bearing.

full rationale

This survey performs no quantitative derivation: it makes no predictions, fits no parameters, and its conclusions are qualitative syntheses of the reviewed tools. Consequently there is no derivation chain that could reduce to its own inputs. The only self-referential element is that several reviewed items are authored by the survey team: AMPeD appears in Section III and Table VI, the STCO paper is cited in the Introduction to name system–technology co-optimisation, and imec.netzero appears in Table VII and Takeaway IV.1. None of these citations is load-bearing: the taxonomy (analytical vs profiling-based vs execution-based) and the comparison tables stand on the cited tools' own published descriptions and validation numbers, not on any claim imported from the authors' prior work. The Table VI footnote exposing that the Error column mixes average error, maximum error, and 1−Pearson correlation is a genuine comparability limitation for tool selection, but it is a correctness and validity concern, not circularity. Score 2 reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities are present because this is a literature survey. The central content rests on four assumptions: corpus completeness, comparability of error metrics, validity of the proposed taxonomy, and accuracy of transcribing cited results.

assumptions (4)
  • domain assumption The selected corpus of simulators and TCO models is complete enough to support the conclusions.
    The survey does not specify a systematic search or inclusion protocol before presenting Tables VI and VII, so completeness is assumed.
  • domain assumption Reported error values in Table VI are comparable despite mixed error definitions.
    Table VI's footnotes disclose average errors, maximum errors, and 1 minus Pearson correlation in one Error column, which assumes cross-simulator comparability.
  • ad hoc to paper The analytical, profiling-based, and execution-driven taxonomy cleanly partitions the field.
    The taxonomy is introduced in Section III-C and Figure 5 without validation; ASTRA-sim is listed as both analytical and profiling-based, so the categories are not exclusive.
  • domain assumption The table contents accurately transcribe the capabilities of the cited tools.
    The authors did not run the simulators or reproduce the cited numbers, and internal contradictions such as Calculon, MultiVerse, and vTrain show this assumption is imperfect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of End-to-End Modeling for Distributed DNN Training: Workloads, Simulators, and TCO." pith.science (2026). https://pith.science/paper/W353UFFM

@misc{pith2026250609275,
  author       = {Pith},
  title        = {Pith review of: A Survey of End-to-End Modeling for Distributed DNN Training: Workloads, Simulators, and TCO},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W353UFFM}},
  note         = {Machine review of arXiv:2506.09275}
}
read the original abstract

Distributed deep neural networks (DNNs) have become a cornerstone for scaling machine learning to meet the demands of increasingly complex applications. However, the rapid growth in model complexity far outpaces CMOS technology scaling, making sustainable and efficient system design a critical challenge. Addressing this requires coordinated co-design across software, hardware, and technology layers. Due to the prohibitive cost and complexity of deploying full-scale training systems, simulators play a pivotal role in enabling this design exploration. This survey reviews the landscape of distributed DNN training simulators, focusing on three major dimensions: workload representation, simulation infrastructure, and models for total cost of ownership (TCO) including carbon emissions. It covers how workloads are abstracted and used in simulation, outlines common workload representation methods, and includes comprehensive comparison tables covering both simulation frameworks and TCO/emissions models, detailing their capabilities, assumptions, and areas of focus. In addition to synthesizing existing tools, the survey highlights emerging trends, common limitations, and open research challenges across the stack. By providing a structured overview, this work supports informed decision-making in the design and evaluation of distributed training systems.

Figures

Figures reproduced from arXiv: 2506.09275 by the authors.

Figure 1
Figure 1. Overview of the DNN training and inference pipeline, showing forward propagation, loss aggregation, backward propagation, and parameters θ update. In practice, training modern LLMs requires not one but hundreds of thousands of training steps [80] to converge to an acceptable solution, each involving passes over batches of data. As model sizes and datasets grow, a single training step can become prohibitively slow an… view at source ↗
Figure 2
Figure 2. Parallelism strategies in distributed DNN training. Single Device: Training runs on a single GPU using the full batch. Data Parallelism (DP): Each GPU holds a model replica and processes a unique input shard; gradients are synchronized via All-Reduce. Fully Sharded Data Parallelism (FSDP): The model is sharded across GPUs; inputs are split; parameters, activations, optimizer states, and gradients are materialized as… view at source ↗
Figure 3
Figure 3. Representative taxonomy of distributed DNN workload representa￾tions, organized into configuration-based, operator / layer-level IRs. †denote MLIR [76] dialects. Table II: A summary of intermediate representations with potential for use in DNN simulations. † Refers to DistIR simulator that shares the same name with the IR. Intermediate Representation (IR) Portability Parallelism Support Simulator Integration Chakra … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Overview of a typical distributed DNN training simulator architecture. Increasing Simulation Fidelity Distributed DNN Training Simulators Analytical Profiling-based Execution-Driven AMPeD [2023] [87] Calculon [2023] [58] Optimus [2024] [73] Paleo [2017] [97] Deepflow […
Figure 5
Figure 5. Figure 5: Taxonomy of distributed DNN training simulators categorized into analytical, profiling-based, and execution-driven simulation methodologies. programming language. The next three groups—Workload, Compute Node, and Network—reflect the fundamental pil￾lars around which si…
Figure 6
Figure 6. Figure 6: Summary of a typical TCO modeling flow. The Inputs box covers the compute stack from workload to fabrication. The TCO and Environmental Cost box lists common CapEx/OpEx components, with outputs in currency or CO2 equivalent emissions (CO2e). However, limited data quali…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TileSight: A First-Principles Tile-Centric Analytical GPU Performance Model from Cores to Clusters

    cs.DC 2026-07 conditional novelty 7.0 of 10

    A tile-centric analytical model predicts GPU kernel latency and cache behavior purely from microbenchmark-calibrated hardware rates, reaching about 12% GEMM error and 13% end-to-end LLM serving error across five GPU lines.

Reference graph

Works this paper leans on

140 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [26]

    Total cost of ownership model for data center technology evaluation

    Yan Cui, Charles Ingalz, Tianyi Gao, and Ali Heydari. Total cost of ownership model for data center technology evaluation. In2017 16th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pages 936–942. IEEE, 2017

    2017

  2. [27]

    Total cost of ownership model for data center technology evaluation

    Yan Cui, Charles Ingalz, Tianyi Gao, and Ali Heydari. Total cost of ownership model for data center technology evaluation. In2017 16th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pages 936–942, 2017

    2017

  3. [1]

    Carbon explorer: A holistic framework for designing carbon aware datacenters

    Bilge Acun, Benjamin Lee, Fiodar Kazhamiaka, Kiwan Maeng, Udit Gupta, Manoj Chakkaravarthy, David Brooks, and Carole-Jean Wu. Carbon explorer: A holistic framework for designing carbon aware datacenters. InProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 118–132, 2023

    2023

  4. [2]

    Gar- net: A detailed on-chip network model inside a full-system simulator

    Niket Agarwal, Tushar Krishna, Li-Shiuan Peh, and Niraj K Jha. Gar- net: A detailed on-chip network model inside a full-system simulator. In2009 IEEE international symposium on performance analysis of systems and software, pages 33–42. IEEE, 2009

    2009

  5. [3]

    A survey of computer architecture simulation techniques and tools.Ieee Access, 7:78120–78145, 2019

    Ayaz Akram and Lina Sawalha. A survey of computer architecture simulation techniques and tools.Ieee Access, 7:78120–78145, 2019

    2019

  6. [4]

    Partir: Composing spmd partitioning strategies for machine learning

    Sami Alabed, Daniel Belov, Bart Chrzaszcz, Juliana Franco, Dominik Grewe, Dougal Maclaurin, James Molloy, Tom Natan, Tamara Norman, Xiaoyue Pan, et al. Partir: Composing spmd partitioning strategies for machine learning. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume ...

    2025

  7. [5]

    Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael V oznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Lang...

    2024

  8. [6]

    Deepflow: A cross-stack pathfinding framework for distributed ai systems.ACM Transactions on Design Automation of Electronic Systems, 29(2):1–20, 2024

    Newsha Ardalani, Saptadeep Pal, and Puneet Gupta. Deepflow: A cross-stack pathfinding framework for distributed ai systems.ACM Transactions on Design Automation of Electronic Systems, 29(2):1–20, 2024

    2024

Show all 140 references
  1. [7]

    Efficient large scale language modeling with mixtures of experts.arXiv preprint arXiv:2112.10684, 2021

    Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ra- makanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts.arXiv preprint arXiv:2112.10684, 2021

    2021 arXiv

  2. [8]

    Understanding the future of energy efficiency in multi-module gpus

    Akhil Arunkumar, Evgeny Bolotin, David Nellans, and Carole-Jean Wu. Understanding the future of energy efficiency in multi-module gpus. In2019 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 519–532. IEEE, 2019

    2019

  3. [9]

    vtrain: A simulation framework for evaluating cost- effective and compute-optimal large language model training

    Jehyeon Bang, Yujeong Choi, Myeongwoo Kim, Yongdeok Kim, and Minsoo Rhu. vtrain: A simulation framework for evaluating cost- effective and compute-optimal large language model training. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 153–167. ...

    2024

  4. [10]

    Dtco including sustainability: Power-performance-area-cost-environmental score (ppace) analysis for logic technologies

    M Garcia Bardon, P Wuytens, L-Å Ragnarsson, G Mirabelli, D Jang, G Willems, A Mallik, A Spessot, J Ryckaert, and B Parvais. Dtco including sustainability: Power-performance-area-cost-environmental score (ppace) analysis for logic technologies. In2020 IEEE Inter- national Elect...

    2020

  5. [11]

    The datacenter as a computer: Designing warehouse-scale machines

    Luiz André Barroso, Urs Hölzle, and Parthasarathy Ranganathan. The datacenter as a computer: Designing warehouse-scale machines. Springer Nature, 2019

    2019

  6. [12]

    Stco: driving the more than moore era

    Dwaipayan Biswas, James Myers, Srikanth B Samavedam, and Julien Ryckaert. Stco: driving the more than moore era. In2024 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pages 7–8. IEEE, 2024

    2024

  7. [13]

    A survey of cache simulators.ACM Computing Surveys (CSUR), 53(1):1–32, 2020

    Hadi Brais, Rajshekar Kalayappan, and Preeti Ranjan Panda. A survey of cache simulators.ACM Computing Surveys (CSUR), 53(1):1–32, 2020

    2020

  8. [14]

    Understanding gpu power: A survey of profiling, modeling, and simulation methods

    Robert A Bridges, Neena Imam, and Tiffany M Mintz. Understanding gpu power: A survey of profiling, modeling, and simulation methods. ACM Computing Surveys (CSUR), 49(3):1–27, 2016

    2016

  9. [15]

    Wattch: A framework for architectural-level power analysis and optimizations

    David Brooks, Vivek Tiwari, and Margaret Martonosi. Wattch: A framework for architectural-level power analysis and optimizations. ACM SIGARCH Computer Architecture News, 28(2):83–94, 2000

    2000

  10. [16]

    TVM: An automated End-to-End optimizing compiler for deep learning

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. TVM: An automated End-to-End optimizing compiler for deep learning. In13th USENIX Symposium on Operating Systems Design and Implementation ...

    2018

  11. [17]

    cudnn: Efficient primitives for deep learning.arXiv preprint arXiv:1410.0759, 2014

    Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Co- hen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning.arXiv preprint arXiv:1410.0759, 2014

    2014 arXiv

  12. [18]

    Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale.arXiv preprint arXiv:2408.05499, 2024

    Jaehong Cho, Minsu Kim, Hyunmin Choi, Guseul Heo, and Jongse Park. Llmservingsim: A hw/sw co-simulation infrastructure for llm inference serving at scale.arXiv preprint arXiv:2408.05499, 2024

    2024 arXiv

  13. [19]

    CodeCarbon: v2.4.1

    CodeCarbon contributors. CodeCarbon: v2.4.1. Zenodo, May 2024. https://doi.org/10.5281/zenodo.11171501

    2024 doi

  14. [20]

    Gloo100k: Collective communications library with various primitives for multi-machine training

    Gloo Contributors. Gloo100k: Collective communications library with various primitives for multi-machine training. https://github.com/ pytorch/gloo, 2019. GitHub repository

    2019

  15. [21]

    Stablehlo: A portable high-level operations set for machine learning models

    OpenXLA Contributors. Stablehlo: A portable high-level operations set for machine learning models. https://github.com/openxla/stablehlo,

  16. [22]

    Torchscript documentation

    PyTorch Contributors. Torchscript documentation. https://pytorch.org/ docs/stable/jit.html, 2025. Accessed: 2025-05-14

    2025

  17. [23]

    Available at https://docs.nvidia.com/cuda/ parallel-thread-execution/index.html

    NVIDIA Corporation.Parallel Thread Execution ISA Version 7.8, 2023. Available at https://docs.nvidia.com/cuda/ parallel-thread-execution/index.html

    2023

  18. [24]

    Nvidia dgx h200, 2024

    NVIDIA Corporation. Nvidia dgx h200, 2024. [Accessed: 7-Jan-2025]

    2024

  19. [25]

    The rising costs of training frontier ai models.arXiv preprint arXiv:2405.21015, 2024

    Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, and David Owen. The rising costs of training frontier ai models.arXiv preprint arXiv:2405.21015, 2024

    2024 arXiv

  20. [28]

    Logp: Towards a realistic model of parallel computation

    David Culler, Richard Karp, David Patterson, Abhijit Sahay, Klaus Erik Schauser, Eunice Santos, Ramesh Subramonian, and Thorsten V on Eicken. Logp: Towards a realistic model of parallel computation. InProceedings of the fourth ACM SIGPLAN symposium on Principles and practice o...

    1993

  21. [29]

    Large scale distributed deep networks.Advances in neural information processing systems, 25, 2012

    Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks.Advances in neural information processing systems, 25, 2012

    2012

  22. [30]

    Uptime institute global data center survey 2024

    Douglas Donnellan, Andy Lawrence, Daniel Bizo, Peter Judge, John O’Brien, Jacqueline Davis, Max Smolaks, Jabari Williams-George, and Rose Weinschenk. Uptime institute global data center survey 2024. Technical report, Uptime Institute, 07 2024

    2024

  23. [31]

    Proteus: Simulating the performance of distributed dnn training.IEEE Transactions on Parallel and Distributed Systems, 2024

    Jiangfei Duan, Xiuhong Li, Ping Xu, Xingcheng Zhang, Shengen Yan, Yun Liang, and Dahua Lin. Proteus: Simulating the performance of distributed dnn training.IEEE Transactions on Parallel and Distributed Systems, 2024

    2024

  24. [32]

    Cordoba: Carbon-efficient optimization framework for computing systems

    Mariam Elgamal, Doug Carmean, Elnaz Ansari, Okay Zed, Ramesh Peri, Srilatha Manne, Udit Gupta, Gu-Yeon Wei, David Brooks, Gage Hills, et al. Cordoba: Carbon-efficient optimization framework for computing systems. In2025 IEEE International Symposium on High Performance Computer...

    2025

  25. [33]

    Amant, Karthikeyan Sankaralingam, and Doug Burger

    Hadi Esmaeilzadeh, Emily Blem, Renee St. Amant, Karthikeyan Sankaralingam, and Doug Burger. Dark silicon and the end of multicore scaling. InProceedings of the 38th annual international symposium on Computer architecture, pages 365–376, 2011

    2011

  26. [34]

    Llmcarbon: Modeling the end-to-end carbon footprint of large language models, 2024

    Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Prateek Sharma, Fan Chen, and Lei Jiang. Llmcarbon: Modeling the end-to-end carbon footprint of large language models, 2024

    2024

  27. [35]

    Echo: Simulating distributed training at scale.arXiv preprint arXiv:2412.12487, 2024

    Yicheng Feng, Yuetao Chen, Kaiwen Chen, Jingzong Li, Tianyuan Wu, Peng Cheng, Chuan Wu, Wei Wang, Tsung-Yi Ho, and Hong Xu. Echo: Simulating distributed training at scale.arXiv preprint arXiv:2412.12487, 2024

    2024 arXiv

  28. [36]

    Mpi: A message-passing interface standard, 1994

    Message P Forum. Mpi: A message-passing interface standard, 1994

    1994

  29. [37]

    Estimating gpu memory consumption of deep learning models

    Yanjie Gao, Yu Liu, Hongyu Zhang, Zhengxian Li, Yonghao Zhu, Haoxiang Lin, and Mao Yang. Estimating gpu memory consumption of deep learning models. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw...

    2020

  30. [38]

    MIT Press, 2016

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org

    2016

  31. [39]

    Google data centers efficiency, 2024

    Google. Google data centers efficiency, 2024. [Accessed: 05-Dec- 2024]

    2024

  32. [40]

    Catch: a cost analysis tool for co- optimization of chiplet-based heterogeneous systems, 2025

    Alexander Graening, Jonti Talukdar, Saptadeep Pal, Krishnendu Chakrabarty, and Puneet Gupta. Catch: a cost analysis tool for co- optimization of chiplet-based heterogeneous systems, 2025

    2025

  33. [42]

    Gpgpu power modeling for multi-domain voltage-frequency scaling

    Joao Guerreiro, Aleksandar Ilic, Nuno Roma, and Pedro Tomas. Gpgpu power modeling for multi-domain voltage-frequency scaling. In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 789–800. IEEE, 2018

    2018

  34. [43]

    Accelerating design space ex- ploration for LLM training systems with multi-experiment parallel simulation

    Fei Gui, Kaihui Gao, Li Chen, Dan Li, Vincent Liu, Ran Zhang, Hongbing Yang, and Dian Xiong. Accelerating design space ex- ploration for LLM training systems with multi-experiment parallel simulation. In22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI...

    2025

  35. [44]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce- ment learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  36. [45]

    A survey on performance modeling and prediction for distributed dnn training.IEEE Transac- tions on Parallel and Distributed Systems, 2024

    Zhenhua Guo, Yinan Tang, Jidong Zhai, Tongtong Yuan, Jian Jin, Li Wang, Yaqian Zhao, and Rengang Li. A survey on performance modeling and prediction for distributed dnn training.IEEE Transac- tions on Parallel and Distributed Systems, 2024

    2024

  37. [46]

    Act: Designing sustainable computer systems with an architectural carbon modeling tool

    Udit Gupta, Mariam Elgamal, Gage Hills, Gu-Yeon Wei, Hsien-Hsin S Lee, David Brooks, and Carole-Jean Wu. Act: Designing sustainable computer systems with an architectural carbon modeling tool. In Proceedings of the 49th Annual International Symposium on Computer Architecture, ...

    2022

  38. [47]

    Chasing car- bon: The elusive environmental footprint of computing

    Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing car- bon: The elusive environmental footprint of computing. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages ...

    2021

  39. [48]

    Onnxim: A fast, cycle-level multi-core npu simulator, 2024

    Hyungkyu Ham, Wonhyuk Yang, Yunseon Shin, Okkyun Woo, Guseul Heo, Sangyeop Lee, Jongse Park, and Gwangsun Kim. Onnxim: A fast, cycle-level multi-core npu simulator, 2024

    2024

  40. [49]

    An analytical framework for estimating tco and exploring data center design space

    Damien Hardy, Marios Kleanthous, Isidoros Sideris, Ali G Saidi, Emre Ozer, and Yiannakis Sazeides. An analytical framework for estimating tco and exploring data center design space. In2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), p...

    2013

  41. [50]

    Ryckaert

    Geert Hellings, Sheng Yang, Pieter Weckx, Marie Bardon, Maarten Van de Put, and J. Ryckaert. The evolution towards cmos 2.0, a heterogeneous logic platform. InProceedings of the 2023 International Conference on Solid State Devices and Materials (SSDM), 09 2023

    2023

  42. [51]

    Towards the systematic reporting of the energy and carbon footprints of machine learning.Journal of Machine Learning Research, 21(248):1–43, 2020

    Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning.Journal of Machine Learning Research, 21(248):1–43, 2020

    2020

  43. [52]

    Towards the systematic reporting of the energy and carbon footprints of machine learning, 2022

    Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning, 2022

    2022

  44. [53]

    Group operation assembly language-a flexible way to express collective com- munication

    Torsten Hoefler, Christian Siebert, and Andrew Lumsdaine. Group operation assembly language-a flexible way to express collective com- munication. In2009 International Conference on Parallel Processing, pages 574–581. IEEE, 2009

    2009

  45. [54]

    An integrated gpu power and performance model

    Sunpyo Hong and Hyesoon Kim. An integrated gpu power and performance model. InProceedings of the 37th annual international symposium on Computer architecture, pages 280–289, 2010

    2010

  46. [55]

    Inseong Hwang, Junghyeok Lee, Huibeom Kang, Gilhyeon Lee, and Hyun Kim. Survey of cpu and memory simulators in computer architecture: A comprehensive analysis including compiler integration and emerging technology applications.Simulation Modelling Practice and Theory, 138:103032, 2025

    2025

  47. [56]

    imec.netzero public, 2024

    imec. imec.netzero public, 2024. [Accessed: 21-Dec-2024]

    2024

  48. [57]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. InInterna- tional conference on machine learning, pages 448–456. pmlr, 2015

    2015

  49. [58]

    Calculon: a methodology and tool for high-level co-design of systems and large language models

    Mikhail Isaev, Nic McDonald, Larry Dennison, and Richard Vuduc. Calculon: a methodology and tool for high-level co-design of systems and large language models. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages...

    2023

  50. [59]

    Deep- speed ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509, 2023

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deep- speed ulysses: System optimizations for enabling training of extreme long sequence transformer models.arXiv preprint arXiv:2309.14509, 2023

    2023 arXiv

  51. [60]

    A quantitative evaluation of contemporary gpu simulation methodology.Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2(2):1–28, 2018

    Akshay Jain, Mahmoud Khairy, and Timothy G Rogers. A quantitative evaluation of contemporary gpu simulation methodology.Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2(2):1–28, 2018

    2018

  52. [61]

    Scarif: Towards carbon modeling of cloud servers with accelerators

    Shixin Ji, Zhuoping Yang, Xingzhen Chen, Stephen Cahoon, Jingtong Hu, Yiyu Shi, Alex K Jones, and Peipei Zhou. Scarif: Towards carbon modeling of cloud servers with accelerators. In2024 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pages 496–501. IEEE, 2024

    2024

  53. [62]

    Beyond data and model parallelism for deep neural networks.Proceedings of Machine Learning and Systems, 1:1–13, 2019

    Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks.Proceedings of Machine Learning and Systems, 1:1–13, 2019

    2019

  54. [63]

    Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In Proceedings of the ...

    2023

  55. [64]

    In-datacenter performance analysis of a tensor processing unit

    Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. InProceedings of the 44th annual international symposium on ...

    2017

  56. [65]

    Accelwattch: A power modeling framework for modern gpus

    Vijay Kandiah, Scott Peverelle, Mahmoud Khairy, Junrui Pan, Amogh Manjunath, Timothy G Rogers, Tor M Aamodt, and Nikos Hardav- ellas. Accelwattch: A power modeling framework for modern gpus. InMICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, pages 7...

    2021

  57. [66]

    Miopen: An open source library for deep learning primitives.arXiv preprint arXiv:1910.00078, 2019

    Jehandad Khan, Paul Fultz, Artem Tamazov, Daniel Lowell, Chao Liu, Michael Melesse, Murali Nandhimandalam, Kamil Nasyrov, Ilya Perminov, Tejash Shah, et al. Miopen: An open source library for deep learning primitives.arXiv preprint arXiv:1910.00078, 2019

    1910 arXiv

  58. [67]

    Moonwalk: Nre optimization in asic clouds.SIGARCH Comput

    Moein Khazraee, Lu Zhang, Luis Vega, and Michael Bedford Taylor. Moonwalk: Nre optimization in asic clouds.SIGARCH Comput. Archit. News, 45(1):511–526, April 2017

    2017

  59. [68]

    Llmem: Estimating gpu memory usage for fine-tuning pre-trained llms.arXiv preprint arXiv:2404.10933, 2024

    Taeho Kim, Yanming Wang, Vatshank Chaturvedi, Lokesh Gupta, Seyeon Kim, Yongin Kwon, and Sangtae Ha. Llmem: Estimating gpu memory usage for fine-tuning pre-trained llms.arXiv preprint arXiv:2404.10933, 2024

    2024 arXiv

  60. [69]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  61. [70]

    Greenchip: A tool for evaluating holistic sustainability of modern computing systems

    Donald Kline Jr, Nikolas Parshook, Xiaoyu Ge, Erik Brunvand, Rami Melhem, Panos K Chrysanthis, and Alex K Jones. Greenchip: A tool for evaluating holistic sustainability of modern computing systems. Sustainable Computing: Informatics and Systems, 22:322–332, 2019

    2019

  62. [71]

    A simple model for determining true total cost of ownership for data centers

    Jonathan Koomey. A simple model for determining true total cost of ownership for data centers. Technical report, Uptime Institute, 03 2008

    2008

  63. [72]

    Reducing activation recomputation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catan- zaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341–353, 2023

    2023

  64. [73]

    Performance modeling and workload analysis of distributed large language model training and inference.arXiv preprint arXiv:2407.14645, 2024

    Joyjit Kundu, Wenzhe Guo, Ali BanaGozar, Udari De Alwis, Sourav Sengupta, Puneet Gupta, and Arindam Mallik. Performance modeling and workload analysis of distributed large language model training and inference.arXiv preprint arXiv:2407.14645, 2024

    2024 arXiv

  65. [74]

    Quantifying the carbon emissions of machine learning.arXiv preprint arXiv:1910.09700, 2019

    Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning.arXiv preprint arXiv:1910.09700, 2019

    1910 arXiv

  66. [75]

    Llvm: A compilation framework for lifelong program analysis & transformation

    Chris Lattner and Vikram Adve. Llvm: A compilation framework for lifelong program analysis & transformation. InProceedings of the 2004 International Symposium on Code Generation and Optimization (CGO), pages 75–86. IEEE, 2004

    2004

  67. [76]

    Mlir: Scaling compiler infrastructure for domain specific computation

    Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasi- lache, and Oleksandr Zinenko. Mlir: Scaling compiler infrastructure for domain specific computation. In2021 IEEE/ACM International Symposium o...

  68. [77]

    Data-driven forecasting of deep learning performance on gpus.arXiv preprint arXiv:2407.13853, 2024

    Seonho Lee, Amar Phanishayee, and Divya Mahajan. Data-driven forecasting of deep learning performance on gpus.arXiv preprint arXiv:2407.13853, 2024

    2024 arXiv

  69. [78]

    Mcpat: An integrated power, area, and timing modeling framework for multicore and manycore architectures

    Sheng Li, Jung Ho Ahn, Richard D Strong, Jay B Brockman, Dean M Tullsen, and Norman P Jouppi. Mcpat: An integrated power, area, and timing modeling framework for multicore and manycore architectures. InProceedings of the 42nd annual ieee/acm international symposium on microarc...

    2009

  70. [79]

    Towards universal performance modeling for machine learning training on multi-gpu platforms.IEEE Transactions on Parallel and Distributed Systems, 2024

    Zhongyi Lin, Ning Sun, Pallab Bhattacharya, Xizhou Feng, Louis Feng, and John D Owens. Towards universal performance modeling for machine learning training on multi-gpu platforms.IEEE Transactions on Parallel and Distributed Systems, 2024

    2024

  71. [80]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, pages 4–14, 2024

    AI @ Meta Llama Team. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, pages 4–14, 2024

    2024 arXiv

  72. [81]

    A survey of performance modeling and simulation techniques for accelerator-based computing.IEEE Transactions on Parallel and Distributed Systems, 26(1):272–281, 2014

    Unai Lopez-Novoa, Alexander Mendiburu, and Jose Miguel-Alonso. A survey of performance modeling and simulation techniques for accelerator-based computing.IEEE Transactions on Parallel and Distributed Systems, 26(1):272–281, 2014

    2014

  73. [82]

    Distsim: A performance model of large-scale hybrid distributed dnn training

    Guandong Lu, Runzhe Chen, Yakai Wang, Yangjie Zhou, Rui Zhang, Zheng Hu, Yanming Miao, Zhifang Cai, Li Li, Jingwen Leng, and Minyi Guo. Distsim: A performance model of large-scale hybrid distributed dnn training. InProceedings of the 20th ACM International Conference on Comput...

    2023

  74. [83]

    Estimating the carbon footprint of bloom, a 176b parameter language model.Journal of Machine Learning Research, 24(253):1–15, 2023

    Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, a 176b parameter language model.Journal of Machine Learning Research, 24(253):1–15, 2023

    2023

  75. [84]

    Review of asic accelerators for deep neural network.Microprocessors and Microsystems, 89:104441, 2022

    Raju Machupalli, Masum Hossain, and Mrinal Mandal. Review of asic accelerators for deep neural network.Microprocessors and Microsystems, 89:104441, 2022

    2022

  76. [85]

    Azure Total Cost of Ownership (TCO) Calcula- tor

    Microsoft Azure. Azure Total Cost of Ownership (TCO) Calcula- tor. https://azure.microsoft.com/en-us/pricing/tco/calculator/, n.d. Ac- cessed: 2025-05-20

    2025

  77. [86]

    Tensor operator set architecture (tosa) dialect

    MLIR Contributors. Tensor operator set architecture (tosa) dialect. https://mlir.llvm.org/docs/Dialects/TOSA/, 2023. Accessed: 2025-05- 14

    2023

  78. [87]

    Amped: An analytical model for performance in distributed training of transformers

    Diksha Moolchandani, Joyjit Kundu, Frederik Ruelens, Peter Vrancx, Timon Evenblij, and Manu Perumkunnil. Amped: An analytical model for performance in distributed training of transformers. In2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISP...

    2023

  79. [88]

    Efficient large-scale language model training on gpu clusters using megatron-lm

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. InProceeding...

    2021

  80. [89]

    Supply chain aware computer architecture

    August Ning, Georgios Tziantzioulis, and David Wentzlaff. Supply chain aware computer architecture. InProceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–15, 2023

    2023

  81. [90]

    Architectural simulators considered harmful.IEEE Micro, 35(6):4–12, 2015

    Tony Nowatzki, Jaikrishnan Menon, Chen-Han Ho, and Karthikeyan Sankaralingam. Architectural simulators considered harmful.IEEE Micro, 35(6):4–12, 2015

    2015

  82. [91]

    Available at https://developer.nvidia.com/nccl

    NVIDIA Corporation.NVIDIA Collective Communication Library (NCCL), 2025. Available at https://developer.nvidia.com/nccl

    2025

  83. [92]

    Open neural network exchange (onnx)

    ONNX. Open neural network exchange (onnx). https://onnx.ai, 2019

    2019

  84. [93]

    Xla architecture and high-level optimizer (hlo)

    OpenXLA Contributors. Xla architecture and high-level optimizer (hlo). https://openxla.org/xla/architecture, 2024. Accessed: 2025-05- 14

    2024

  85. [94]

    Survey on network simulators.International Journal of Computer Applications, 182(21):23–30, 2018

    Ronit L Patel, Maharshi J Pathak, and Amit J Nayak. Survey on network simulators.International Journal of Computer Applications, 182(21):23–30, 2018

    2018

  86. [95]

    Carbon emissions and large neural network training.arXiv preprint arXiv:2104.10350, 2021

    David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training.arXiv preprint arXiv:2104.10350, 2021

    2021 arXiv

  87. [96]

    Chiplet cloud: Building ai supercomputers for serving large generative language models, 2024

    Huwan Peng, Scott Davidson, Richard Shi, Shuaiwen Leon Song, and Michael Taylor. Chiplet cloud: Building ai supercomputers for serving large generative language models, 2024

    2024

  88. [97]

    Paleo: A performance model for deep neural networks

    Hang Qi, Evan R Sparks, and Ameet Talwalkar. Paleo: A performance model for deep neural networks. InInternational Conference on Learning Representations, 2017

    2017

  89. [98]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. InInternational conference on machine learnin...

    2022

  90. [99]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  91. [100]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 35...

    2020

  92. [101]

    James Reed, Zachary DeVito, Horace He, Ansley Ussery, and Jason Ansel. torch. fx: Practical program capture and transformation for deep learning in python.Proceedings of Machine Learning and Systems, 4:638–651, 2022

    2022

  93. [102]

    The ns-3 network simulator

    George F Riley and Thomas R Henderson. The ns-3 network simulator. InModeling and tools for network simulation, pages 15–34. Springer, 2010

    2010

  94. [103]

    Glow: Graph lowering compiler techniques for neural networks.arXiv preprint arXiv:1805.00907, 2018

    Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein, et al. Glow: Graph lowering compiler techniques for neural networks.arXiv preprint arXiv:1805.00907, 2018

    2018 arXiv

  95. [104]

    An overview of gradient descent optimization algorithms.arXiv preprint arXiv:1609.04747, 2016

    Sebastian Ruder. An overview of gradient descent optimization algorithms.arXiv preprint arXiv:1609.04747, 2016

    2016 arXiv

  96. [105]

    Learn- ing representations by back-propagating errors.nature, 323(6088):533– 536, 1986

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learn- ing representations by back-propagating errors.nature, 323(6088):533– 536, 1986

    1986

  97. [106]

    Distir: An intermediate representation for optimizing distributed neural networks

    Keshav Santhanam, Siddharth Krishna, Ryota Tomioka, Andrew Fitzgibbon, and Tim Harris. Distir: An intermediate representation for optimizing distributed neural networks. InProceedings of the 1st Workshop on Machine Learning and Systems, pages 15–23, 2021

    2021

  98. [107]

    Life-cycle emissions of ai hardware: A cradle-to-grave approach and generational trends.arXiv preprint arXiv:2502.01671, 2025

    Ian Schneider, Hui Xu, Stephan Benecke, David Patterson, Keguo Huang, Parthasarathy Ranganathan, and Cooper Elsworth. Life-cycle emissions of ai hardware: A cradle-to-grave approach and generational trends.arXiv preprint arXiv:2502.01671, 2025

    2025 arXiv

  99. [108]

    Training compute of frontier ai models grows by 4-5x per year, 2024

    Jaime Sevilla and Edu Roldán. Training compute of frontier ai models grows by 4-5x per year, 2024. Available at: https://epoch.ai/blog/ training-compute-of-frontier-ai-models-grows-by-4-5x-per-year, Ac- cessed: 2024-12-03

    2024

  100. [109]

    Atlahs: An application-centric network simulator toolchain for ai, hpc, and distributed storage.arXiv preprint arXiv:2505.08936, 2025

    Siyuan Shen, Tommaso Bonato, Zhiyi Hu, Pasquale Jordan, Tiancheng Chen, and Torsten Hoefler. Atlahs: An application-centric network simulator toolchain for ai, hpc, and distributed storage.arXiv preprint arXiv:2505.08936, 2025

    2025 arXiv

  101. [110]

    Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi- billion parameter language models using model parallelism.arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  102. [111]

    A survey on deep learning hardware accelerators for heterogeneous hpc platforms.arXiv preprint arXiv:2306.15552, 2023

    Cristina Silvano, Daniele Ielmini, Fabrizio Ferrandi, Leandro Fiorin, Serena Curzel, Luca Benini, Francesco Conti, Angelo Garofalo, Cris- tian Zambelli, Enrico Calore, et al. A survey on deep learning hardware accelerators for heterogeneous hpc platforms.arXiv preprint arXiv:2...

    2023 arXiv

  103. [112]

    A hybrid tensor-expert- data parallelism approach to optimize mixture-of-experts training

    Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Ra- jbhandari, Yuxiong He, and Abhinav Bhatele. A hybrid tensor-expert- data parallelism approach to optimize mixture-of-experts training. In Proceedings of the 37th International Conference on Supercomputing, pages 20...

    2023

  104. [113]

    Shift-left testing

    Larry Smith. Shift-left testing. https://www.drdobbs.com/ shift-left-testing/184404768, 2001

    2001

  105. [114]

    Greenscale: Carbon optimization for edge computing.IEEE Internet of Things Journal, pages 1–1, 2025

    Yonglak Son, Udit Gupta, Andrew McCrabb, Young Geun Kim, Valeria Bertacco, David Brooks, and Carole-Jean Wu. Greenscale: Carbon optimization for edge computing.IEEE Internet of Things Journal, pages 1–1, 2025

    2025

  106. [115]

    Chakra: Advancing performance benchmarking and co-design using standardized execution traces.arXiv preprint arXiv:2305.14516, 2023

    Srinivas Sridharan, Taekyung Heo, Louis Feng, Zhaodong Wang, Matt Bergeron, Wenyin Fu, Shengbao Zheng, Brian Coutinho, Saeed Rashidi, Changhai Man, et al. Chakra: Advancing performance benchmarking and co-design using standardized execution traces.arXiv preprint arXiv:2305.14516, 2023

    2023 arXiv

  107. [116]

    Energy and policy considerations for deep learning in nlp, 2019

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp, 2019

    2019

  108. [117]

    On the importance of initialization and momentum in deep learning

    Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. InInternational conference on machine learning, pages 1139–1147. PMLR, 2013

    2013

  109. [118]

    Mhlo: Mlir hlo dialect

    TensorFlow Authors. Mhlo: Mlir hlo dialect. https://github.com/ tensorflow/mlir-hlo, 2023. Accessed: 2025-05-14

    2023

  110. [119]

    tf.compat.v1.graphdef

    TensorFlow Contributors. tf.compat.v1.graphdef. https://www. tensorflow.org/api_docs/python/tf/compat/v1/GraphDef, 2025. Ac- cessed: 2025-05-14

    2025

  111. [120]

    Spir-v: The standard portable intermediate representation

    The Khronos Group Inc. Spir-v: The standard portable intermediate representation. https://www.khronos.org/spir/, 2024. Accessed: 2025- 05-14

    2024

  112. [121]

    Triton: an inter- mediate language and compiler for tiled neural network computations

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an inter- mediate language and compiler for tiled neural network computations. InProceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10–19, 2019

    2019

  113. [122]

    Carbon aware transformers through joint model-hardware optimization

    Irene Wang, Newsha Ardalani, Mostafa Elhoushi, Daniel Jiang, Samuel Hsia, Ekin Sumbul, Divya Mahajan, Carole-Jean Wu, and Bilge Acun. Carbon aware transformers through joint model-hardware optimization. arXiv preprint arXiv:2505.01386, 2025

    2025

  114. [123]

    Designing cloud servers for lower carbon

    Jaylen Wang, Daniel S Berger, Fiodar Kazhamiaka, Celine Irvene, Chaojie Zhang, Esha Choukse, Kali Frost, Rodrigo Fonseca, Brijesh Warrier, Chetan Bansal, et al. Designing cloud servers for lower carbon. In2024 ACM/IEEE 51st Annual International Symposium on Computer Architectu...

    2024

  115. [124]

    SimAI: Unifying architecture design and performance tuning for Large-Scale large language model training with scalability and precision

    Xizheng Wang, Qingxu Li, Yichi Xu, Gang Lu, Dan Li, Li Chen, Heyang Zhou, Linkang Zheng, Sen Zhang, Yikai Zhu, et al. SimAI: Unifying architecture design and performance tuning for Large-Scale large language model training with scalability and precision. In22nd USENIX Symposiu...

    2025

  116. [125]

    Enabling efficient large-scale deep learning training with cache coherent disag- gregated memory systems

    Zixuan Wang, Joonseop Sim, Euicheol Lim, and Jishen Zhao. Enabling efficient large-scale deep learning training with cache coherent disag- gregated memory systems. In2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 126–140. IEEE, 2022

    2022

  117. [126]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. InProceedings of the 2020 conference on empir- ical methods in n...

    2020

  118. [127]

    Memory is all you need: An overview of compute-in- memory architectures for accelerating large language model inference

    Christopher Wolters, Xiaoxuan Yang, Ulf Schlichtmann, and Toyotaro Suzumura. Memory is all you need: An overview of compute-in- memory architectures for accelerating large language model inference. arXiv preprint arXiv:2406.08413, 2024

    2024 arXiv

  119. [128]

    Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model train- ing at scale

    William Won, Taekyung Heo, Saeed Rashidi, Srinivas Sridharan, Sudarshan Srinivasan, and Tushar Krishna. Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model train- ing at scale. In2023 IEEE International Symposium on Performance Analysis of Sy...

    2023

  120. [129]

    Machine learning fleet efficiency: Analyzing and optimizing large-scale google tpu systems with ml productivity goodput.arXiv preprint arXiv:2502.06982, 2025

    Arissa Wongpanich, Tayo Oguntebi, Jose Baiocchi Paredes, Yu Emma Wang, Phitchaya Mangpo Phothilimthana, Ritwika Mitra, Zongwei Zhou, Naveen Kumar, and Vijay Janapa Reddi. Machine learning fleet efficiency: Analyzing and optimizing large-scale google tpu systems with ml product...

    2025 arXiv

  121. [130]

    Beyond efficiency: Scaling ai sustainably.IEEE Micro, 2024

    Carole-Jean Wu, Bilge Acun, Ramya Raghavendra, and Kim Hazel- wood. Beyond efficiency: Scaling ai sustainably.IEEE Micro, 2024

    2024

  122. [131]

    Sustainable ai: Environmental implications, chal- lenges and opportunities.Proceedings of Machine Learning and Systems, 4:795–813, 2022

    Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, New- sha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. Sustainable ai: Environmental implications, chal- lenges and opportunities.Proceedings of Machine Learning and Systems, 4:795–813, 2022

    2022

  123. [132]

    Gspmd: general and scalable parallelization for ml computation graphs.arXiv preprint arXiv:2105.04663, 2021

    Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yan- ping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, et al. Gspmd: general and scalable parallelization for ml computation graphs.arXiv preprint arXiv:2105.04663, 2021

    2021 arXiv

  124. [133]

    Ai competition and firm value: Evidence from deepseek’s disruption.Finance Research Letters, page 107447, 2025

    Xing Yang. Ai competition and firm value: Evidence from deepseek’s disruption.Finance Research Letters, page 107447, 2025

    2025

  125. [134]

    The design and use of simplepower: a cycle-accurate energy estimation tool

    Wu Ye, Narayanan Vijaykrishnan, Mahmut Kandemir, and Mary Jane Irwin. The design and use of simplepower: a cycle-accurate energy estimation tool. InProceedings of the 37th Annual Design Automation Conference, pages 340–345, 2000

    2000

  126. [135]

    Opencarboneval: A unified carbon emission estimation framework in large-scale ai models.arXiv preprint arXiv:2405.12843, 2024

    Zhaojian Yu, Yinghao Wu, Zhuotao Deng, Yansong Tang, and Xiao- Ping Zhang. Opencarboneval: A unified carbon emission estimation framework in large-scale ai models.arXiv preprint arXiv:2405.12843, 2024

    2024 arXiv

  127. [136]

    A memory perspective: The effects of fine-tuning llms with high-bandwidth memory, 2024

    Felippe Vieira Zacarias, Kiran Palli, Sudharshan Vazhkudai, and Evelyn Grevelink. A memory perspective: The effects of fine-tuning llms with high-bandwidth memory, 2024. Avail- able at: https://www.micron.com/content/dam/micron/global/public/ documents/products/product-flyer/l...

    2024

  128. [137]

    Mase: An efficient representation for software-defined ml hardware system exploration

    Cheng Zhang, Jianyi Cheng, Zhewen Yu, and Yiren Zhao. Mase: An efficient representation for software-defined ml hardware system exploration. InNeurIPS 2023 Workshop on Machine Learning Systems (MLSys), 2023. Submission #19. https://openreview.net/forum?id= MX5XN2u8CV

    2023

  129. [138]

    Llmcompass: Enabling efficient hardware design for large language model inference

    Hengrui Zhang, August Ning, Rohan Baskar Prabhakar, and David Wentzlaff. Llmcompass: Enabling efficient hardware design for large language model inference. In2024 ACM/IEEE 51st Annual Interna- tional Symposium on Computer Architecture (ISCA), pages 1080–1096. IEEE, 2024

    2024

  130. [139]

    Alpa: Automating inter-and Intra-Operator par- allelism for distributed deep learning

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. Alpa: Automating inter-and Intra-Operator par- allelism for distributed deep learning. In16th USENIX Symposium on Operating Systems Des...

    2022

  131. [140]

    Day- dream: Accurately estimating the efficacy of optimizations for DNN training

    Hongyu Zhu, Amar Phanishayee, and Gennady Pekhimenko. Day- dream: Accurately estimating the efficacy of optimizations for DNN training. In2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 337–352, 2020

    2020

  132. [2023]

    Accessed: 2025-05-14

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.