Pith. sign in

REVIEW 3 major objections 5 minor 66 references

System-performance and cost modeling of Large Language Model training and inference

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A modeling framework that couples roofline-based performance prediction with a chiplet cost model can forecast distributed LLM training and inference behavior, including FlashAttention speedups within 8 percent of published measurements.

desk verdict A useful engineering extension of Optimus with honest validation, but the FlashAttention model has an under-specified HBM-traffic term that leaves long-context speedups unverified. read the letter →

arxiv 2507.02456 v1 pith:4EXAVEAT submitted 2025-07-03 cs.AR

Wenzhe Guo , Joyjit Kundu , Uras Tos , Weijiang Kong , Giuliano Sisto , Timon Evenblij , Manu Perumkunnil This is my paper · ORCID
classification cs.AR
keywords LLMperformancemodelingFlashAttentionmixture-of-experts5Dparallelismchipletcostmodelnetworktopologycost-performancetrade-offanalytical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single analytical framework, built by extending an existing performance model, can predict both the runtime and the manufacturing cost of distributed LLM training and inference across FlashAttention, mixture-of-experts, five-dimensional parallelism, and several network topologies. The authors validate the FlashAttention speedup predictions against published GPU measurements to within 8%, and they show that mixture-of-experts and network-communication predictions sit within the bounds of published strong-scaling and simulation data. If correct, this gives system architects a fast pre-silicon way to explore performance-cost trade-offs for next-generation AI accelerators before committing to hardware. The framework also supports hybrid parallelization and cost analysis in one automated pipeline.

What carries the argument

The carrying mechanism is a hierarchical roofline model that classifies each kernel as compute-bound or memory-bound at multiple cache levels and uses that classification to estimate execution time. Three additions do the new work: a tiled FlashAttention timing equation (HBM load plus GEMM plus pointwise-reduction plus HBM store), a capacity-based all-to-all communication model for expert routing, and a topology-aware collective communication library whose delay formula decomposes into serialization, link, and switching terms. The design-space expansion hinges on the identity that the tensor-parallel by expert-parallel by expert-data-parallel degree equals the tensor-parallel by attention-data-parallel degree, which lets expert parallelism be traded against data parallelism. Cost is computed from a chiplet stack equation that divides per-die cost by yield, adds assembly cost, and then divides by assembly yield.

What would settle it

Take a workload outside the validation set (for instance, a FlashAttention kernel on a next-generation accelerator or a chiplet-based design), measure the actual execution time and manufacturing cost, and compare with the framework's predictions; a FlashAttention error beyond 8 percent, or a MoE strong-scaling prediction falling outside the published baseline-to-optimized bounds, would refute the central claim.

Watch

Extended reading notes

Core claim

At the core of the work is the claim that an analytical, roofline-based performance model can be extended to cover the three dominant bottlenecks of large-model systems: memory-bound attention, sparsely activated experts, and topology-dependent collective communication. The model represents FlashAttention execution time as the sum of HBM load time, GEMM time, pointwise/reduction operations, and HBM store time for each tile, and represents mixture-of-experts communication with a capacity-based all-to-all expression. It expands the parallelism space to five dimensions by adding expert parallelism alongside data, tensor, sequence, and pipeline parallelism, and it couples the performance predictions to a hierarchical chiplet cost model that accounts for die yield, assembly yield, and packaging. Validation against published A100/V100 measurements and simulation data places FlashAttention speedup errors within 8%, MoE strong-scaling predictions between baseline and optimized published curves, and network communication times between a network simulator and real-system measurements.

Load-bearing premise

The load-bearing premise is that the inherited hierarchical roofline model accurately estimates kernel times on any accelerator; validation only covers current-generation GPUs, so extrapolating to next-generation chips is an unvalidated step.

Editorial extensions

If this is right

  • Larger batch sizes and newer GPU generations make FlashAttention increasingly beneficial, because the tiling converts memory-bound attention GEMMs into compute-bound ones.
  • In mixture-of-experts training, compute time stays nearly constant as the number of experts grows, but all-to-all communication can exceed 30 percent of batch time; raising inter-node bandwidth reduces that overhead.
  • Combining FlashAttention with expert parallelism compounds speedups: in the paper's case, a 6.1x mixture-of-experts speedup over a dense baseline rises to about 6.5x when FlashAttention is added.
  • For latency-sensitive inference, fully-connected and switch topologies beat ring and 2D mesh at equal off-GPU bandwidth because hop counts stay small.
  • Adding HBM stacks improves memory-bound inference more than compute-bound training, and the reference accelerator's memory configuration looks near-optimal when training and inference are weighted equally.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because validation is confined to existing GPU generations, predictions for hypothetical next-generation accelerators are extrapolations; the uncertainty grows with any change in memory hierarchy that the roofline model does not capture.
  • The cost model covers chiplet manufacturing but not data-center operating costs, so extending it with power, cooling, and floor-space costs would make the trade-off analysis closer to total cost of ownership.
  • The MoE validation intentionally excludes load imbalance and token-dropping optimizations; including those effects is a direct way to test whether the prediction shifts toward the optimized published curve.
  • The network model's communication times sit between a simulator and real-system measurements, suggesting that the switching-delay term is the main uncertainty; measuring switching latency on real hardware would tighten the model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper extends the Optimus analytical performance-cost modeling framework for distributed LLM training and inference to cover FlashAttention, mixture-of-experts models with expert parallelism (expanding the parallelism space to 5D), topology-specific collective communication algorithms, and an integrated chiplet-based cost model. The authors validate their FlashAttention model against published GPT-2 speedups on A100s, their MoE model against DeepSpeed-TED strong-scaling data on V100s, and their network model against ASTRA-sim and real-system all-reduce measurements. Case studies then use the framework to explore FlashAttention speedup versus batch size and GPU generation, MoE communication overhead versus network bandwidth, topology impact on Llama-70B inference, and performance-cost tradeoffs for A100-like chiplet configurations with varying HBM stacks.

Significance. If the modeling assumptions hold, the framework would give system architects a fast, pre-silicon tool for exploring performance-cost tradeoffs in large-scale LLM systems, filling a genuine gap left by the base Optimus framework. The paper's strengths are its use of externally published validation data rather than self-generated measurements, its reuse of a previously validated chiplet cost model, and the breadth of the case studies spanning compute, communication, and manufacturing cost. The paper also makes concrete, falsifiable predictions (e.g., FlashAttention speedup trends with batch size and GPU generation) that could be tested by later measurements. However, the FlashAttention model is under-specified at exactly the point that matters for long-context predictions, and the validation evidence is thinner than the claims require, especially for the next-generation-GPU scenarios emphasized in the case studies.

major comments (3)
  1. [Section III-B, Eq. (1)] The HBM terms t_HBM_ld and t_HBM_st are described only verbally as loading Q, K, V, and O 'along with the surrogate sequences' and writing O back, with no byte counts or tile-loop iteration counts. In FlashAttention-v1, each K/V tile is re-read from HBM for every query block, so the K/V HBM traffic is O(N^2 / B), where B is set by the SRAM constraint 4Bd ≈ M; it is not a single pass over each matrix. As written, Eq. (1) cannot be reproduced or checked, and if the implementation instead counts each matrix once, the model will undercount HBM traffic at long sequence lengths and overstate FlashAttention's benefit. Please provide the explicit byte accounting, including the number of query blocks and the K/V re-read factor, and verify it against the algorithm in [15].
  2. [Section IV-A and Section V-A] The FlashAttention validation in Fig. 2 reports only the ratio of FlashAttention speedup over the Megatron implementation, for GPT2-small/medium at 1k-4k context on A100. A multiplicative error in both numerator and denominator can cancel in such a ratio, and the short-context regime does not significantly exercise the O(N^2/B) K/V re-read traffic that dominates long-context attention. The case studies in Section V-A then extrapolate to H100/B200 and to batch size 2048 with no supporting measurements. Please add absolute kernel-time comparisons, or validation at longer contexts (e.g., 8k-32k), or explicitly state that the long-context and next-generation-GPU predictions are unvalidated extrapolations.
  3. [Section IV-B, Fig. 3, and references [12]/[53]] The text says the MoE validation reproduces DeepSpeed-TED hybrid parallelism and cites [53], but the Fig. 3 caption and Section V-B attribute the dotted comparison curves to [12]. These are different papers, so the provenance of the validation data is unclear and must be corrected. In addition, 'our prediction lies between the two bounds of [12]' is not a quantitative validation for a model that is intended to reproduce the baseline implementation: the baseline and optimized curves bracket the prediction, but a model without the duplicate-token-dropping and activation-checkpointing optimizations should lie on or close to the baseline curve. Please report the error relative to the baseline curve and explain the 256-GPU departure.
minor comments (5)
  1. [Abstract and Section I] The word 'parallellism' is misspelled in both the abstract and the introduction; it should be 'parallelism'.
  2. [Section III-B] The term 'surrogate sequences' is used without definition; please define it or refer explicitly to the online-softmax derivation in [52].
  3. [Fig. 5, top-left panel] The axis labels '2,52,62,72,82,93' are garbled and should be replaced with readable numeric labels.
  4. [Eq. (3)] The data volume formula K = ECD × precision should be written with explicit multiplication and dimensions, e.g., K = E · C · D · precision, and the units of E, C, D, and precision should be stated.
  5. [Section V-D and Table I] The text says inference is performed in one node using only tensor parallelism, yet Table I lists SP=8 in the inference row; please clarify how SP is counted relative to TP in this configuration.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: new FlashAttention, MoE, and network models are validated against external published measurements; the Optimus and IMEC self-citations are minor and non-load-bearing.

full rationale

The derivation chain is self-contained against external benchmarks. Each new module is validated with published measurements that are not inputs to the model: FlashAttention speedups against the external FlashAttention paper [15] (Fig. 2, within 8%), MoE strong-scaling against DeepSpeed-TED [53] (Fig. 3), and all-reduce times against ASTRA-sim2.0 and a real 16-GPU V100 system (Fig. 4). Eq. 1 is a kernel-level decomposition (t_HBM_ld + t_GEMM + t_pt-reduc-ops + t_HBM_st) whose tile size B is fixed by the SRAM constraint 4Bd approximately M, not fitted to measured speedups; the Megatron baseline is computed by the same base framework, so the speedup ratio is a genuine model output rather than a by-construction equality. The MoE model adopts the DeepSpeed-TED parallelization identity (Eq. 2) and the GShard all-to-all volume formula (Eq. 3), and the paper explicitly states that it does not model load imbalance and observes a departure from the measured trend at 256 GPUs, ruling out hidden fitting to the validation data. The cost model is adopted from external open-source work [29], and its A100 backend parameters are physical values (TechInsights die sizes; IMEC research-fab cost per mm2, defect density, NRE). Two self-citations exist but are not load-bearing in a circular sense: [28] (Optimus, sharing authors Kundu and Guo) supplies the framework structure whose base components (MLP, projections, DP communication) are exercised and externally validated inside this paper's integrated predictions; [58] (sharing Sisto and Perumkunnil) supplies cost parameters that are fab data, not values fitted to the case-study conclusion that the A100 configuration is cost-performance optimal. The skeptic's Eq. 1 concern (missing byte-count derivation for K/V HBM traffic) is a correctness risk for long-context and H100/B200 extrapolation, not circularity, because no mechanism in the paper forces the 8% validation error by construction. Score 2 reflects the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The model uses hardware specification values and published measurements as inputs (SRAM size, HBM bandwidth, NCCL all-to-all bandwidth, cost-per-mm2) rather than fitting parameters to the target case studies. The load-bearing assumptions are the accuracy of the Optimus roofline model, the GShard expert-capacity communication approximation, the NVIDIA NCCL bandwidth inputs, and the Graening chiplet cost model. No new physical entities or fitted constants are introduced.

assumptions (4)
  • domain assumption The hierarchical roofline model from DeepFlow/Optimus accurately estimates compute and memory-bound kernel times on modeled GPUs.
    Invoked in Section III-A as the prediction engine's core; the framework inherits this base model and its error behavior.
  • domain assumption The GShard expert-capacity token-buffer model captures all-to-all communication volume for MoE.
    Used in Eq. (3) in Section III-C to compute TA2A; the paper notes it ignores load imbalance.
  • domain assumption The Graening chiplet cost model and its yield formulas correctly compute manufacturing cost.
    Adopted in Section III-E and Eq. (5); the paper cites prior validation [58]-[62].
  • domain assumption NVIDIA NCCL bandwidth measurements represent achievable all-to-all bandwidth in the modeled networks.
    BWA2A is taken from NVIDIA measurements [54] in Section III-C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of System-performance and cost modeling of Large Language Model training and inference." pith.science (2026). https://pith.science/paper/4EXAVEAT

@misc{pith2026250702456,
  author       = {Pith},
  title        = {Pith review of: System-performance and cost modeling of Large Language Model training and inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EXAVEAT}},
  note         = {Machine review of arXiv:2507.02456}
}
read the original abstract

Large language models (LLMs), based on transformer architectures, have revolutionized numerous domains within artificial intelligence, science, and engineering due to their exceptional scalability and adaptability. However, the exponential growth in LLM size and complexity has outpaced advancements in compute capacity, memory bandwidth, network performance, and cost efficiency, posing significant challenges to their scalability on distributed systems. To address these limitations, alternative model architectures, optimization strategies, communication-aware network topologies, and novel system design approaches have been proposed in literature. This paper introduces a performance-cost modeling methodology for LLM training and inference that integrates state-of-the-art compute techniques with memory optimizations, and latest communication techniques. Building on an analytical performance model, our approach incorporates recent innovations such as the flash attention technique and mixture of experts models to address the memory bandwidth and compute bottlenecks. It also considers the impact of different network topologies and topology-specific communication algorithms with 5D parallellism. The framework also integrates a chiplet cost model. The proposed modeling methodology provides valuable insights to guide future compute system design and facilitates hardware-software co-development, in particular due to its ability to analyze performance-cost trade-offs for various system architectural configurations.

Figures

Figures reproduced from arXiv: 2507.02456 by the authors.

Figure 1
Figure 1. Overview of the proposed framework consisting of a system module, a performance perdiction module and a cost [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 4
Figure 4. Validation of the network model. Inset showing the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 3
Figure 3. Validation of MoE: strong scaling of the GPT3-6.7B [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The speed-up due to FlashAttention as a function of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Performance-cost tradeoffs for various designs based on Nvidia A100. The GPT3-175B model on 1024 GPUs is used [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 58 canonical work pages

  1. [42]

    Comprehensive performance modeling and system design insights for foundation models,

    S. Subramanian, E. Rrapaj, P. Harrington, S. Chheda, S. Farrell, B. Austin, S. Williams, N. Wright, and W. Bhimji, “Comprehensive performance modeling and system design insights for foundation models,” 2024

    work page 2024

  2. [15]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 16 344–16 359

    work page 2022

  3. [53]

    A hybrid tensor-expert-data parallelism approach to optimize mixture- of-experts training,

    S. Singh, O. Ruwase, A. A. Awan, S. Rajbhandari, Y . He, and A. Bhatele, “A hybrid tensor-expert-data parallelism approach to optimize mixture- of-experts training,” in Proceedings of the 37th International Conference on Supercomputing, ser. ICS ’23. ACM, Jun. 2023

    work page 2023

  4. [12]

    DeepSpeed-MoE: Advancing mixture-of- experts inference and training to power next-generation AI scale,

    S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y . Aminabadi, A. A. Awan, J. Rasley, and Y . He, “DeepSpeed-MoE: Advancing mixture-of- experts inference and training to power next-generation AI scale,” in Proceedings of the 39th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 17–23 Jul 2022, pp. 1...

    work page 2022

  5. [1]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , vol. 21, no. 140, pp. 1–67, 2020

    2020

  6. [2]

    GLM: General language model pretraining with autoregressive blank infilling,

    Z. Du, Y . Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “GLM: General language model pretraining with autoregressive blank infilling,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Dublin, Ireland, May 2022, pp. 320–335

    work page 2022

  7. [3]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers) , Minneapolis, Minnesota, Jun. 2019, pp....

    work page 2019

  8. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” CoRR, vol. abs/2010.11929, 2020

    arXiv 2010

Show all 66 references
  1. [5]

    Training data-efficient image transformers & amp; distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou, “Training data-efficient image transformers & amp; distillation through attention,” in Proceedings of the 38th International Conference on Ma- chine Learning, ser. Proceedings of Machine Learning Research,...

    2021

  2. [6]

    Multiple physics pretraining for physical surrogate models,

    M. McCabe, B. R.-S. Blancard, L. H. Parker, R. Ohana, M. Cranmer, A. Bietti, M. Eickenberg, S. Golkar, G. Krawezik, F. Lanusse, M. Pettee, T. Tesileanu, K. Cho, and S. Ho, “Multiple physics pretraining for physical surrogate models,” 2024

    2024

  3. [7]

    Highly accurate protein structure prediction with alphafold,

    J. Jumper et al., “Highly accurate protein structure prediction with alphafold,” Nature, vol. 596, pp. 583––589, 2021

    2021

  4. [8]

    Evolutionary-scale prediction of atomic-level protein structure with a language model,

    Z. Lin et al., “Evolutionary-scale prediction of atomic-level protein structure with a language model,” Science, vol. 379, no. 6637, pp. 1123– 1130, 2023

    2023

  5. [9]

    A foundation model for atomistic materials chemistry,

    I. Batatia et al., “A foundation model for atomistic materials chemistry,”

  6. [10]

    Scaling laws for neural language models,

    J. Kaplan et al., “Scaling laws for neural language models,” 2020. [Online]. Available: https://arxiv.org/abs/2001.08361

    2020 arXiv

  7. [11]

    Efficient large-scale language model training on GPU clusters using megatron-lm,

    D. Narayanan et al., “Efficient large-scale language model training on GPU clusters using megatron-lm,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021, pp. 1–15

    2021

  8. [13]

    DeepSpeed- inference: enabling efficient inference of transformer models at unprece- dented scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, and Y . He, “DeepSpeed- inference: enabling efficient inference of transformer models at unprece- dented scale,” ser. SC ’22, 2022

    2022

  9. [14]

    OpenAI’s massive GPT-3 model is impressive, but size isn’t everything,

    K. Wiggers, “OpenAI’s massive GPT-3 model is impressive, but size isn’t everything,” 2020. [Online]. Available: https://venturebeat.com/ 2020/06/01/ai-machine-learning-openai-gpt-3-size-isnt-everything/

    2020

  10. [16]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” arXiv preprint arXiv:2307.08691 , 2023

    2023 arXiv

  11. [17]

    Glam: Efficient scaling of language models with mixture-of-experts,

    N. Du, Y . Huang, A. M. Dai, S. Tong, D. Lepikhin, Y . Xu, M. Krikun, Y . Zhou, A. W. Yu, O. Firat et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in ICML, 2022

    2022

  12. [18]

    Hymba: A hybrid-head architecture for small language models,

    X. Dong, Y . Fu, S. Diao, W. Byeon, Z. Chen, A. S. Mahabaleshwarkar, S.-Y . Liu, M. V . Keirsbilck, M.-H. Chen, Y . Suhara, Y . Lin, J. Kautz, and P. Molchanov, “Hymba: A hybrid-head architecture for small language models,” 2024. [Online]. Available: https://arxiv.org/abs/2411.13676

    2024 arXiv

  13. [19]

    Flashattention-3: Fast and accurate attention with asynchrony and low- precision,

    J. Shah, G. Bikshandi, Y . Zhang, V . Thakkar, P. Ramani, and T. Dao, “Flashattention-3: Fast and accurate attention with asynchrony and low- precision,” 2024

    2024

  14. [20]

    From words to watts: Benchmarking the energy costs of large language model inference,

    S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Berg- eron, J. Kepner, D. Tiwari, and V . Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” in 2023 IEEE High Performance Extreme Computing Conference (HPEC) , 20...

    2023

  15. [21]

    Energy- efficiency limits on training AI systems using learning-in-memory,

    Z. Chen, J. Leugering, G. Cauwenberghs, and S. Chakrabartty, “Energy- efficiency limits on training AI systems using learning-in-memory,” 2024

    2024

  16. [22]

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding,

    Y . Huang, Y . Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, N. Arivazhagan, Y . Cao, R. Aharoni, M. Johnson, and Y . Wu, “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding,” in Proceedings of the International Conference on Learning Representa...

    2021

  17. [23]

    Astra-sim: Enabling sw/hw co-design exploration for distributed dl training plat- forms,

    S. Rashidi, S. Sridharan, S. Srinivasan, and T. Krishna, “Astra-sim: Enabling sw/hw co-design exploration for distributed dl training plat- forms,” in 2020 IEEE International Symposium on Performance Analysis of Systems and Software , 2020, pp. 81–92

    2020

  18. [24]

    AI and Memory Wall ,

    A. Gholami et al., “ AI and Memory Wall ,” IEEE Micro, vol. 44, no. 03, pp. 33–39, May 2024

    2024

  19. [25]

    (2024, Dec) Nvidia blackwell platform arrives to power a new era of computing

    NVIDIA Newsroom. (2024, Dec) Nvidia blackwell platform arrives to power a new era of computing. Accessed: 2024-12-13. [Online]. Available: https://nvidianews.nvidia.com/news/ nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing

    2024

  20. [26]

    (2023, Jun) Amd mi300: Taming the hype - ai perfor- mance

    SemiAnalysis. (2023, Jun) Amd mi300: Taming the hype - ai perfor- mance. Accessed: 2024-12-13. [Online]. Available: https://semianalysis. com/2023/06/12/amd-mi300-taming-the-hype-ai-performance/

    2023

  21. [27]

    (2023) Why chiplets are so critical in automotive

    SemiEngineering. (2023) Why chiplets are so critical in automotive. Accessed: 2024-12-13. [Online]. Available: https://semiengineering.com/ why-chiplets-are-so-critical-in-automotive/

    2023

  22. [28]

    Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference ,

    J. Kundu, W. Guo, A. BanaGozar, U. De Alwis, S. Sengupta, P. Gupta, and A. Mallik, “ Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference ,” in 2024 IEEE International Symposium on Workload Characterization (IISWC) , Los Alamito...

    2024

  23. [29]

    Chiplets: How small is too small?

    A. Graening, S. Pal, and P. Gupta, “Chiplets: How small is too small?” in 2023 60th ACM/IEEE Design Automation Conference , 2023, pp. 1–6

    2023

  24. [30]

    Analyzing CUDA workloads using a detailed GPU simulator,

    A. Bakhoda, G. L. Yuan, W. W. L. Fung, H. Wong, and T. M. Aamodt, “Analyzing CUDA workloads using a detailed GPU simulator,” in 2009 IEEE International Symposium on Performance Analysis of Systems and Software, 2009, pp. 163–174

    2009

  25. [31]

    Accel-Sim: An extensible simulation framework for validated GPU modeling,

    M. Khairy, Z. Shen, T. M. Aamodt, and T. G. Rogers, “Accel-Sim: An extensible simulation framework for validated GPU modeling,” in 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020, pp. 473–486

    2020

  26. [32]

    Cross- architecture performance prediction (XAPP) using CPU code to predict GPU performance,

    N. Ardalani, C. Lestourgeon, K. Sankaralingam, and X. Zhu, “Cross- architecture performance prediction (XAPP) using CPU code to predict GPU performance,” in 2015 48th Annual IEEE/ACM International Sym- posium on Microarchitecture (MICRO) , 2015, pp. 725–737

    2015

  27. [33]

    Principal kernel analysis: A tractable methodology to simulate scaled GPU workloads,

    C. Avalos Baddouh, M. Khairy, R. N. Green, M. Payer, and T. G. Rogers, “Principal kernel analysis: A tractable methodology to simulate scaled GPU workloads,” in MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’21, New York, NY , USA, 202...

    2021

  28. [34]

    Activation in network for NoC-based deep neural network accelerator,

    W. Zhu, Y . Chen, and Z. Lu, “Activation in network for NoC-based deep neural network accelerator,” in 2024 International VLSI Symposium on Technology, Systems and Applications (VLSI TSA) , 2024, pp. 1–4

    2024

  29. [35]

    Performance modeling and scalability optimization of distributed deep learning systems,

    F. Yan, O. Ruwase, Y . He, and T. Chilimbi, “Performance modeling and scalability optimization of distributed deep learning systems,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2015, p. 1355–1364

    2015

  30. [36]

    Paleo: A performance model for deep neural networks,

    Qi, E. R. Sparks, and A. Talwalkar, “Paleo: A performance model for deep neural networks,” in International Conference on Learning Representations, 2016

    2016

  31. [37]

    Performance prediction of GPU- based deep learning applications,

    E. Gianniti, L. Zhang, and D. Ardagna, “Performance prediction of GPU- based deep learning applications,” in 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC- PAD), 2018, pp. 167–170

    2018

  32. [38]

    Habitat: A runtime- based computational performance predictor for deep neural network training,

    G. X. Yu, Y . Gao, P. Golikov, and G. Pekhimenko, “Habitat: A runtime- based computational performance predictor for deep neural network training,” in USENIX Annual Technical Conference , 2021

    2021

  33. [39]

    AMPeD: An analytical model for performance in dis- tributed training of transformers,

    D. Moolchandani, J. Kundu, F. Ruelens, P. Vrancx, T. Evenblij, and M. Perumkunnil, “AMPeD: An analytical model for performance in dis- tributed training of transformers,” in 2023 IEEE International Symposium on Performance Analysis of Systems and Software , 2023, pp. 306–315

    2023

  34. [40]

    Calculon: a methodology and tool for high-level co-design of systems and large language models,

    M. Isaev, N. Mcdonald, L. Dennison, and R. Vuduc, “Calculon: a methodology and tool for high-level co-design of systems and large language models,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , ser. SC ’23, Ne...

    2023

  35. [41]

    DeepFlow: A cross-stack pathfinding framework for distributed AI systems,

    N. Ardalani, S. Pal, and P. Gupta, “DeepFlow: A cross-stack pathfinding framework for distributed AI systems,”ACM Trans. Des. Autom. Electron. Syst., vol. 29, no. 2, Feb. 2024

    2024

  36. [43]

    A simple model for portable and fast prediction of execution time and power consumption of gpu kernels,

    L. Braun, S. Nikas, C. Song, V . Heuveline, and H. Fr ¨oning, “A simple model for portable and fast prediction of execution time and power consumption of gpu kernels,” ACM Trans. Archit. Code Optim. , vol. 18, no. 1, Dec. 2021

    2021

  37. [44]

    GPGPU performance and power estimation using machine learning,

    G. Wu, J. L. Greathouse, A. Lyashevsky, N. Jayasena, and D. Chiou, “GPGPU performance and power estimation using machine learning,” in 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA), 2015, pp. 564–576

    2015

  38. [45]

    GPU static modeling using PTX and deep structured learning,

    J. Guerreiro, A. Ilic, N. Roma, and P. Tom ´as, “GPU static modeling using PTX and deep structured learning,” IEEE Access , vol. 7, pp. 159 150– 159 161, 2019

    2019

  39. [46]

    Program analysis and machine learning–based approach to predict power consumption of cuda kernel,

    G. Alavani, J. Desai, S. Saha, and S. Sarkar, “Program analysis and machine learning–based approach to predict power consumption of cuda kernel,” ACM Trans. Model. Perform. Eval. Comput. Syst. , vol. 8, no. 4, Jul. 2023

    2023

  40. [47]

    Forecasting gpu performance for deep learning training and inference,

    S. Lee, A. Phanishayee, and D. Mahajan, “Forecasting gpu performance for deep learning training and inference,” 2024

    2024

  41. [48]

    Cost analysis and cost-driven IP reuse methodology for SoC design based on 2.5D/3D integration,

    D. Stow, I. Akgun, R. Barnes, P. Gu, and Y . Xie, “Cost analysis and cost-driven IP reuse methodology for SoC design based on 2.5D/3D integration,” in 2016 IEEE/ACM International Conference on Computer- Aided Design (ICCAD) , 2016, pp. 1–6

    2016

  42. [49]

    Cost-effective design of scalable high-performance systems using active and passive interposers,

    D. Stow, Y . Xie, T. Siddiqua, and G. H. Loh, “Cost-effective design of scalable high-performance systems using active and passive interposers,” in 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2017, pp. 728–735

    2017

  43. [50]

    Chiplet actuary: a quantitative cost model and multi- chiplet architecture exploration,

    Y . Feng and K. Ma, “Chiplet actuary: a quantitative cost model and multi- chiplet architecture exploration,” in Proceedings of the 59th ACM/IEEE Design Automation Conference , ser. DAC ’22, New York, NY , USA, 2022, p. 121–126

    2022

  44. [51]

    Online normalizer calculation for softmax,

    M. Milakov and N. Gimelshein, “Online normalizer calculation for softmax,” 2018. [Online]. Available: https://arxiv.org/abs/1805.02867

    2018 arXiv

  45. [52]

    From online softmax to flashattention,

    Z. Ye, “From online softmax to flashattention,” 2023. [Online]. Available: https://courses.cs.washington.edu/courses/cse599m/23sp/notes/ flashattn.pdf

    2023

  46. [54]

    Training deep learning models at scale: How nccl enables best performance on ai data center networks,

    S. Jeaugey, “Training deep learning models at scale: How nccl enables best performance on ai data center networks,” in Proceedings of the NVIDIA GPU Technology Conference (GTC) , 2024, session S62129. [Online]. Available: https://www.nvidia.com/en-us/on-demand/ session/gtc24-s62129/

    2024

  47. [55]

    Optimization of collective communication operations in mpich,

    R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in mpich,” The International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66, 2005

    2005

  48. [56]

    Highly available data parallel ml training on mesh networks,

    S. Kumar and N. Jouppi, “Highly available data parallel ml training on mesh networks,” 2020

    2020

  49. [57]

    An overview of manufacturing yield and reliability modeling for semiconductor products,

    W. Kuo and T. Kim, “An overview of manufacturing yield and reliability modeling for semiconductor products,” Proceedings of the IEEE, vol. 87, no. 8, pp. 1329–1344, 1999

    1999

  50. [58]

    Cost-performance co- optimization for the chiplet era,

    A. Graening, D. Adil, G. Sisto, E. Lenormand, M. Perumkunnil, N. Pan- tano, V . B. Y . Kumar, P. Gupta, and A. Mallik, “Cost-performance co- optimization for the chiplet era,” in Proceedings of 2024 IEEE 26th Electronics Packaging Technology Conference (EPTC), 2024

    2024

  51. [59]

    Smoothing disruption across the stack: Tales of memory, heterogeneity, and compilers,

    M. Niemier, Z. Enciso, M. Sharifi, X. Hu, I. O’Connor, A. Graening, R. Sharma, P. Gupta, J. Castrillon, J. Lima, A. Khan, H. Farzaneh, N. Afroze, A. Khan, and J. Ryckaert, “Smoothing disruption across the stack: Tales of memory, heterogeneity, and compilers,” in 2024 Design, A...

    2024

  52. [60]

    Eco-chip: Estimation of carbon footprint of chiplet-based architectures for sustainable vlsi,

    C. C. Sudarshan, N. Matkar, S. Vrudhula, S. S. Sapatnekar, and V . A. Chhabria, “Eco-chip: Estimation of carbon footprint of chiplet-based architectures for sustainable vlsi,” in 2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA) , 2024, pp. 671– 685

    2024

  53. [61]

    Exploiting chiplet integration technol- ogy for fast high-capacity dram modules,

    Z. Xia, C. Song, R. Krishna, A. Victor, S. Penta, M. S. Bakir, E. Rosen- baum, N. S. Kim, and M. Kang, “Exploiting chiplet integration technol- ogy for fast high-capacity dram modules,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems , pp. 1–13, 2025

    2025

  54. [62]

    Reed: Chiplet-based accelerator for fully homomorphic encryption,

    A. Aikata, A. C. Mert, S. Kwon, M. Deryabin, and S. S. Roy, “Reed: Chiplet-based accelerator for fully homomorphic encryption,” 2025. [Online]. Available: https://arxiv.org/abs/2308.02885

    2025 arXiv

  55. [63]

    Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale,

    W. Won, T. Heo, S. Rashidi, S. Sridharan, S. Srinivasan, and T. Krishna, “Astra-sim2.0: Modeling hierarchical networks and disaggregated systems for large-model training at scale,” in 2023 IEEE International Symposium on Performance Analysis of Systems and Software , 2023, pp. 283–294

    2023

  56. [64]

    (2020) Nvidia dgx a100 system architecture

    Nvidia. (2020) Nvidia dgx a100 system architecture. [Online]. Avail- able: https://resources.nvidia.com/en-us-dgx-systems/dgxa100-system? xs=489761

    2020

  57. [65]

    TechInsights, https://www.techinsights.com, 2025

    2025

  58. [2024]

    Available: https://arxiv.org/abs/2401.00096

    [Online]. Available: https://arxiv.org/abs/2401.00096

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.