Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Scaling Intelligence: Designing Data Centers for Next-Gen Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A data center network that gives every GPU equal high-bandwidth access to every other—called FullFlat in the paper—could raise large-language-model training utilization to 70% or more, while a conventional two-tier fabric stalls below 50%.

desk verdict A useful co-design study with a real contribution——first systematic evaluation of FullFlat optical topologies for LLM training——but the 30-60% headline gain rests on optimistic efficiency assumptions and validation deferred to another paper. read the letter →

arxiv 2506.15006 v3 pith:HMMLE2M7 submitted 2025-06-17 cs.AR cs.AIcs.DCcs.ETcs.PF

Jesmin Jahan Tithi , Hanjiang Wu , Avishaii Abuhatzera , Fabrizio Petrini This is my paper · ORCID
classification cs.ARcs.AIcs.DCcs.ETcs.PF
keywords datacenterco-designFullFlatopticalnetworkco-packagedopticslargelanguagemodeltrainingmixtureofexpertsFLOPSUtilizationtopologysensitivitytensorparallelism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the conventional two-tier data center network—a fast scale-up domain inside a small group of GPUs and a much slower scale-out fabric between groups—is the main barrier to training trillion-parameter language models well. It proposes and evaluates a FullFlat architecture in which co-packaged optics give every pair of GPUs the same 1.6 TB/s class bandwidth, treating the whole cluster as one flat fabric. Using an analytical model extended for mixture-of-experts models and validated to within 10%, it reports that FullFlat improves throughput by 30–60% over near-future two-tier systems with 200 GB/s scale-out bandwidth and can push Model FLOPS Utilization to 70% or more across sparse and dense transformers. The broader point is that co-designing network topology, memory, and parallelism together exposes where added resources stop paying off, and that flatter networks reduce dependence on finicky software tuning.

What carries the argument

The argument is carried by two instruments: the FullFlat network architecture and the extended analytical performance model. FullFlat is defined as a high-radix, low-diameter, all-optical fabric with co-packaged optics that equalizes scale-up and scale-out bandwidth (1.6 TB/s per direction at 80% communication efficiency in the baseline), giving every GPU the same class of connection to any other without requiring physical $N^{2}$ links. The model, an extension of an open-source analytical simulator to mixture-of-experts layers, expert parallelism and sharding, dynamic routing, and all-to-all collectives, predicts training runtime within 10% and is used to exhaustively search optimization configurations over 65,536-GPU clusters. Together they turn the question of which topology supports trillion-parameter training into a quantifiable sensitivity analysis over FLOPS, HBM bandwidth and capacity, scale-up and scale-out bandwidth, and high-bandwidth-domain size.

What would settle it

Run the same models and configurations on a prototype or high-fidelity simulation of a FullFlat optical fabric at 1,024–4,096 GPUs with 1.6 TB/s flat bandwidth, and compare measured runtime and Model FLOPS Utilization against the paper's predictions; if the model error exceeds 10%, or if the FullFlat advantage over a 200 GB/s two-tier baseline is not in the 30–60% range and MFU stays below 70%, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that for next-generation, multi-trillion-parameter language models, network topology, not raw compute, is the binding constraint: once expert-parallel and tensor-parallel traffic crosses from a fast scale-up domain into a slow scale-out domain, throughput falls and utilization plateaus below 50%. The paper claims that a FullFlat network—built from high-radix, low-diameter optical topologies with equal bandwidth between any two nodes and no scale-up/scale-out distinction—removes this bottleneck, delivering 30–60% higher performance than a two-tier network with 200 GB/s scale-out bandwidth, cutting the spread among the top 5,000 configurations from about 80% to about 5%, and raising MFU and system utilization to 70% or more on GPT-4-scale mixture-of-experts models and on a dense 175-billion-parameter model. It further claims that the margin comes mainly from holding all-to-all expert communication inside the high-bandwidth fabric, and that this makes performance less sensitive to missing optimization steps such as compute-communication overlap and hardware-accelerated collectives.

Load-bearing premise

The load-bearing premise is that the analytical model, validated only on a 64-GPU cluster, keeps its 10% accuracy when extrapolated to 65,536 GPUs and to a FullFlat optical network that does not yet exist; if its error grows at scale or on the new topology, the reported 30–60% gains and 70% MFU values could shift substantially.

Editorial extensions

If this is right

  • On a FullFlat fabric with roughly 1.6 TB/s flat bandwidth, GPT-4-scale mixture-of-experts training could run 30–60% faster than on a two-tier fabric with 200 GB/s scale-out, reaching Model FLOPS Utilization at or above 70%.
  • The performance gap between the best and a suboptimal parallelization configuration shrinks from about 80% to about 5%, so software tuning burden and sensitivity to missing optimizations drop sharply.
  • For mixture-of-experts models, enlarging the high-bandwidth domain past the point where expert communication fits gives little extra return; scale-out bandwidth then handles a small residual share of traffic.
  • For dense models, bandwidth is less critical than for sparse models, but compute-communication overlap and hardware-accelerated collectives matter more: missing overlap costs up to 43% and missing hardware collectives up to 29% at 16K GPUs.
  • A concrete near-future design target emerges: high-radix, low-latency networks with at least 1.6 TB/s scale-up and 200 GB/s scale-out bandwidth, roughly 1.3 TB of HBM per GPU, 30 TB/s HBM bandwidth, and hardware-accelerated collectives.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 64-GPU validation of the model scales faithfully to 65,536 GPUs, then flat optical fabrics should be treated as a primary design option for the 2026/2027 generation of data centers, not just an exotic alternative; the paper itself stops short of building or simulating the FullFlat fabric at that scale.
  • The saturation result—that once expert communication fits in the high-bandwidth domain, larger high-bandwidth domains stop helping—suggests a simple sizing rule for MoE data centers: match the high-bandwidth domain to the expert count and expected top-k traffic, then spend remaining budget on scale-out bandwidth and HBM capacity.
  • If dense models are indeed more harmed by missing overlap and collectives than sparse ones, then a fleet optimized for MoE training should still deploy hardware collectives and overlap-aware scheduling, because the same data center will also run distilled dense models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a co-design methodology for future LLM data centers. It extends the open-source Calculon analytical performance model to support Mixture-of-Experts (MoE) models, introduces a hypothetical FullFlat optical network topology with equalized scale-up and scale-out bandwidth, and uses this tool to compare two-tier networks (HBD8, HBD64/128) against FullFlat for GPT-4-scale MoE models and GPT-3-175B. The central claim is that FullFlat with 1.6 TB/s bandwidth at 80% efficiency improves training performance by 30-60% over two-tier networks with 200 GB/s scale-out bandwidth, raising MFU to 70% or more. The paper also reports sensitivity sweeps over HBD size, scale-up/scale-out bandwidth, FLOPS, HBM bandwidth/capacity, compute-communication overlap, and hardware-accelerated collectives, and it derives qualitative recommendations for future AI data centers.

Significance. If the model extrapolations hold, the paper provides useful architectural insights: it quantifies diminishing returns from enlarging the HBD once expert communication fits inside it, shows that dense models are more sensitive to missing compute-communication overlap, and identifies HBM capacity as a key bottleneck. The paper is transparent about the fact that the extended tool's validation is deferred to a separate paper, and it builds on the open-source Calculon framework. The systematic sensitivity analyses and the explicit introduction of FullFlat as a design point are valuable contributions. However, the headline quantitative claims rest on an unvalidated extrapolation from a 64-GPU cluster to 65,536 GPUs and to a non-existing FullFlat topology, and on optimistic efficiency constants. The qualitative direction of the results is plausible, but the specific 30-60% range and the MFU values should currently be treated as projections rather than established measurements.

major comments (4)
  1. [§2.4] The central quantitative claim depends on a performance model whose validation does not cover the regimes where the headline results are produced. The extension is validated on a 64-GPU H100 cluster with an HBD of 8, while Section 3 reports results at up to 65,536 GPUs and on a FullFlat topology that the validation cannot exercise (equalized HBD/LBD bandwidth, high-radix all-optical routing, MoE all-to-all at scale). The manuscript explicitly defers the tool details and validation to a separate paper. The error bound of ±10% is therefore established only for small clusters of an existing topology; if the model error grows with scale or with the new topology, the reported 30-60% gain and MFU values would change. The paper should include intermediate-scale validation, an error-scaling analysis, or a stated uncertainty bound on the extrapolated numbers.
  2. [§3 (efficiency assumptions)] The assumed 99% flop efficiency for operations over size 128 and the 80% communication efficiency are load-bearing but not justified or validated in this manuscript. The FLOPS sensitivity study in Section 3.7 (Figure 5g) shows that increasing compute throughput increases the relative speedup of FullFlat over TwoTier-HBD64; conversely, lowering the flop efficiency below 99%—which is more realistic for tensor-core GEMMs and full training steps—would increase compute time and, because compute and communication are overlapped, shrink the exposed communication fraction that FullFlat's higher bandwidth removes. The authors should report the headline FullFlat advantage and MFU for a range of flop efficiencies (e.g., 0.70-0.95) and state at what efficiency the claimed 30-60% benefit is no longer obtained.
  3. [§3.11] The headline claim that FullFlat improves performance by 30-60% and achieves MFU of 70% or more is not backed by a directly verifiable table or figure. Section 3.1 states a 30% gap for both MoE models, Section 3.9 refers to a '30% - 12%' gap, and Figure 8 plots 'Compute Efficiency' without comparing systems or showing the claimed range. Since the exhaustive-search optimization can choose different configurations for different systems, the comparative claim should be supported by a summary table listing, for each system and model, the best configuration, throughput, and MFU at the cluster sizes used (e.g., 4K, 8K, 16K, 65K GPUs). Without this, the 30-60% range cannot be reproduced from the paper.
  4. [§3.3] The quantitative effect of hardware-accelerated collectives is based on unstated assumptions: a 13% GPU cycle saving, a 2× traffic multiplier for all-reduce, and a 1.5× multiplier for reduce-scatter/all-gather. These values drive the 16% slowdown claim for TwoTier-HBD64 and the 'no hardware collectives' impact factors in Table 7. The paper should either cite benchmarks for these parameters or provide a sensitivity analysis, because the relative advantage of FullFlat depends on how much collective traffic and cycle overhead the network actually removes.
minor comments (6)
  1. [Table 3] Specify whether FLOPS and memory bandwidth values are per-GPU or per-system; the unit 'PF/s' for FLOPS is ambiguous in the current caption.
  2. [Abstract and §2.4] The abstract's claim that the tool predicts runtime within 10% of real-world measurements should be qualified as applying only to the validated 64-GPU/NVLink configuration, not to the FullFlat extrapolation.
  3. [§3.11] Reconcile the 30-60% range with the 30% and the '30% - 12%' figures reported in Sections 3.1 and 3.9 so that the reader can trace the provenance of the headline number.
  4. [Figure 8] Clarify the relationship between 'Compute Efficiency' and the MFU metric defined in the abstract, and state explicitly whether recomputation FLOPs are included in the numerator.
  5. [Table 5] The SU and SO notation is introduced after the text has already used HBD and LBD for the same concepts; align the two naming conventions.
  6. [Table 3] For the FullFlat row, state the effective HBD size or explain that HBD and LBD are equalized, since the table currently lists HBD Size 64|128 in all columns.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the FullFlat speedup and MFU figures are simulator outputs driven by explicitly stated design parameters and efficiency constants, not by fitted targets or load-bearing self-citation.

full rationale

The derivation chain is self-contained in the sense relevant to circularity: the central quantitative claims (the 30-60% FullFlat advantage and MFU of 70% or more) are outputs of the extended Calculon simulator, not quantities fitted to those claims. The bandwidth values (1600 GB/s versus 200 GB/s) are design parameters of the scenarios being compared, and the model's performance equations relate communication time to those parameters; such a sensitivity result is a model prediction, not an identity. Calculon [17] is external prior work by Isaev et al., and the MoE extension is validated on a 64-GPU Mistral cluster; extrapolation to 65,536 GPUs and to a not-yet-existing FullFlat optical fabric is an uncertainty and validity risk, not circular reasoning. The only author-overlapping citations (Ridgeline [10] and PolarStar [20]) are background or recommendation references and are not load-bearing for the FullFlat speedup claim. The 99% flop-efficiency and 80% communication-efficiency constants are stated inputs with cited sources, not values fitted to the paper's own conclusions; changing them changes the magnitude of the results but does not make the derivation circular. The manuscript itself flags that full tool details, FullFlat technical details, and validations are deferred to future papers; those are omitted-support limitations, not circular steps. Therefore no step reduces the predicted outcome to its own inputs, and the paper receives a non-circular verdict.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central results depend on several assumed efficiencies (communication, FLOP, HBM transfer) and on model architecture assumptions (GPT-4 with 16 experts, GPT-29T hypothetical). These are inputs, not fitted to the output claims. The biggest assumption is the validity of the unshipped tool at 65K-GPU scale.

free parameters (5)
  • Communication efficiency (scale-up and scale-out) = 80%
    Assumed for both SU and SO domains in all experiments, based on internal simulation and NVIDIA SHARP results (Section 3).
  • FLOP efficiency = 99%
    Assumed for operations over size 128 (Section 3).
  • HBM transfer efficiency = 90%
    Assumed for data sizes exceeding 100 MB, benchmarked on Calculon (Section 3).
  • GPU cycle savings from hardware collectives = 13%
    Used to estimate benefit of hardware-accelerated collectives (Section 3.3).
  • Software collective traffic multipliers = 2x all-reduce, 1.5x reduce-scatter/allgather
    Used to model software collectives slowdown (Section 3.3).
assumptions (4)
  • domain assumption GPT-4 architecture has 1.8T parameters, 120 layers, 16 experts, topK=2.
    GPT-4 params not officially disclosed; sourced from blogs and SemiAnalysis (Section 2.2.1).
  • ad hoc to paper A FullFlat optical network with equalized HBD/LBD bandwidth will be feasible by 2026/2027 with co-packaged optics.
    The paper assumes this futuristic network exists; technical details deferred to future paper (Section 2.3).
  • ad hoc to paper The extended Calculon model predicts runtime within 10% at scales up to 65,536 GPUs.
    Validation only shown for 64 GPUs; details deferred to separate paper (Section 2.4).
  • domain assumption Model parameters for GPT-29T (120 layers, 128 experts, topK=2) represent a plausible future LLM.
    Hypothetical model introduced for scalability study (Section 3).
invented entities (1)
  • FullFlat network architecture
    purpose: Equal-bandwidth, all-optical scale-up/scale-out fabric for LLM training.
    Proposed and evaluated in the model, but not implemented; CPO advances cited, but no FullFlat deployment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Intelligence: Designing Data Centers for Next-Gen Language Models." pith.science (2026). https://pith.science/paper/HMMLE2M7

@misc{pith2026250615006,
  author       = {Pith},
  title        = {Pith review of: Scaling Intelligence: Designing Data Centers for Next-Gen Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HMMLE2M7}},
  note         = {Machine review of arXiv:2506.15006}
}
read the original abstract

The explosive growth of Large Language Models (LLMs), such as GPT-4 with 1.8 trillion parameters, demands a fundamental rethinking of data center architecture to ensure scalability, efficiency, and cost-effectiveness. Our work provides a comprehensive co-design framework that jointly explores FLOPS, HBM bandwidth and capacity, multiple network topologies (two-tier vs. FullFlat optical), the size of the scale-out domain, and popular parallelism/optimization strategies used in LLMs. We introduce and evaluate FullFlat network architectures, which provide uniform high-bandwidth, low-latency connectivity between all nodes, and demonstrate their transformative impact on performance and scalability. Through detailed sensitivity analyses, we quantify the benefits of overlapping compute and communication, leveraging hardware-accelerated collectives, widening the scale-out domain, and increasing memory capacity. Our study spans both sparse (mixture of experts) and dense transformer-based LLMs, revealing how system design choices affect Model FLOPS Utilization (MFU = Model FLOPS per token * Observed tokens per second / Peak FLOPS of the hardware) and overall throughput. For the co-design study, we utilized an analytical performance modeling tool capable of predicting LLM runtime within 10% of real-world measurements. Our findings offer actionable insights and a practical roadmap for designing AI data centers that can efficiently support trillion-parameter models, reduce optimization complexity, and sustain the rapid evolution of AI capabilities.

Figures

Figures reproduced from arXiv: 2506.15006 by the authors.

Figure 1
Figure 1. Running LLMs with suboptimal parameter configurations [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (Top) Typical LLM components. Adopted from [4]. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. illustrates MoE communication patterns. Expert par￾allelism requires extensive all-to-all communication, typically un￾necessary for dense LLMs. Common communication and collective operations such as all-reduce, all-to-all, reduce-scatter, all-gather, and point-to-point communication can often be interleaved with computation with additional complexity. Estimating optimal LLM performance is difficult due to the vast o… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Network topologies: (left) 2D HyperX, (center) Rails of 2D [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance trends for GPT4-1.8T (solid) and GPT4-29T (dashed) with sequence size 32,768 and batch size 1,024. Colors: Maroon=Two [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance Impact of HBM Capacity. In this experiment, [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Performance trends for smaller, nonMOE, dense model GPT3-175B. Seq size = 2048, Batch size = 1024. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Scaling of Compute Efficiency with Number of GPUs for [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distributed Training under Packet Loss

    cs.DC 2025-07 reject novelty 6.0 of 10

    The paper proposes unbiased gradient renormalization and stale-parameter broadcasts to keep distributed training convergent under random packet loss, with bounded inter-worker drift.

Reference graph

Works this paper leans on

40 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query trans- former models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)

    arXiv 2023

  2. [2]

    Paul Alcorn. 2024. Intel Details Gaudi 3 at Vision 2024: New AI Accelerator Sampling to Partners Now, Volume Production in Q3 . https://www.tomshardware.com/pc- components/cpus/intel-details-guadi-3-at-vision-2024-new-ai-accelerator- sampling-to-partners-now-volume-production-in-q3 Accessed: 2025-02-25

    work page 2024

  3. [3]

    AMD. 2024. AMD Instinct MI300X Platform Data Sheet. https: //www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/data- sheets/amd-instinct-mi300x-platform-data-sheet.pdf Accessed: 2025-02-25

    work page 2024

  4. [4]

    Abhimanyu Bambhaniya, Ritik Raj, Geonhwa Jeong, Souvik Kundu, Sudarshan Srinivasan, Midhilesh Elavazhagan, Madhu Kumar, and Tushar Krishna. 2024. Demystifying Platform Requirements for Diverse LLM Inference Use Cases.arXiv preprint arXiv:2406.01698 (2024)

    arXiv 2024

  5. [5]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954 (2024)

    arXiv 2024

  6. [6]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang

  7. [7]

    Yu-Han Chang

    Dr. Yu-Han Chang. 2024. Co-Packaged Optics (CPO) 2025-2035: Technolo- gies, Market, and Forecasts. https://www.idtechex.com/en/research-report/co- packaged-optics/1019 Comprehensive analysis of CPO technology, packaging trends, and market forecasts

    work page 2024

  8. [8]

    Yu-Han Chang

    Dr. Yu-Han Chang. 2024. Co-Packaged Optics (CPO): Evaluating Dif- ferent Packaging Technologies. IDTechEx Research Article (2024). https://www.idtechex.com/en/research-article/co-packaged-optics-cpo- evaluating-different-packaging-technologies/31608 In-depth insights into CPO packaging approaches and technology benchmarks

    work page 2024

Show all 40 references
  1. [9]

    Yu-Han Chang

    Dr. Yu-Han Chang. 2024. Co-Packaged Optics (CPO) Market Outlook & Packaging Technology Trends. IDTechEx Research Article (2024). https://www.idtechex.com/en/research-article/co-packaged-optics-cpo- market-outlook-and-packaging-technology-trends/31440 Market trajectory and pack...

    2024

  2. [10]

    Fabio Checconi, Jesmin Jahan Tithi, and Fabrizio Petrini. 2022. Ridgeline: A 2D Roofline Model for Distributed Systems. arXiv:2209.01368 [cs.DC] https: //arxiv.org/abs/2209.01368

    2022

  3. [11]

    Cisco. 2024. Cisco Massively Scalable Data Center Network Fabric Design and Operation White Paper. https://www.cisco.com/c/en/us/products/collateral/ switches/nexus-9000-series-switches/white-paper-c11-743245.html Accessed: 2025-02-14

    2024

  4. [12]

    Intel Corporation. 2020. Intel(R) Tofino(TM) Native Architecture – Public Ver- sion. Technical Report. Intel Corporation. https://raw.githubusercontent.com/ barefootnetworks/Open-Tofino/master/PUBLIC_Tofino-Native-Arch.pdf

    2020

  5. [13]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    2022

  6. [14]

    Jens Domke, Satoshi Matsuoka, Ivan R Ivanov, Yuki Tsushima, Tomoya Yuki, Akihiro Nomura, Shinichi Miura, Nie McDonald, Dennis L Floyd, and Nicolas Dube. 2019. HyperX topology: First at-scale implementation and comparison to the fat-tree. (2019), 1–23

    2019

  7. [15]

    Richard L Graham, Lion Levi, Devendar Burredy, Gil Bloch, Gilad Shainer, David Cho, George Elias, Daniel Klein, Joshua Ladd, Ophir Maor, et al. 2020. Scalable hi- erarchical aggregation and reduction protocol (sharp) tm streaming-aggregation hardware design and evaluation. In ...

    2020

  8. [16]

    Dallin Grimm. 2024. UALink Consortium Poised to Compete with Nvidia’s NVLink. Tom’s Hardware (2024). https://www.tomshardware.com/tech- industry/ualink-consortium-officially-incorporates-nvlink-competitor- headed-by-amd-and-intel-opens-doors-to-contributor-members Accessed: 2025-02-27

    2024

  9. [17]

    Mikhail Isaev, Nic McDonald, Larry Dennison, and Richard Vuduc. 2023. Cal- culon: a methodology and tool for high-level co-design of systems and large language models. In Proceedings of the International Conference for High Perfor- mance Computing, Networking, Storage and Anal...

    2023

  10. [18]

    Hong Jiang. 2022. Intel’s Ponte Vecchio GPU: Architecture, Systems & Software. In 2022 IEEE Hot Chips 34 Symposium (HCS) . IEEE Computer Society, 1–29

    2022

  11. [19]

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et al . 2024. {MegaScale}: Scaling large language model training to more than 10,000 {GPUs}. In 21st USENIX Symposium on Networked Systems Design and Impl...

    2024

  12. [20]

    Kartik Lakhotia, Kelly Isham, Laura Monroe, Maciej Besta, Torsten Hoefler, and Fabrizio Petrini. 2023. In-network Allreduce with Multiple Spanning Trees on PolarFly. In Proceedings of the 35th ACM Symposium on Parallelism in Algorithms and Architectures (Orlando, FL, USA) (SPA...

    2023

  13. [21]

    Kartik Lakhotia, Laura Monroe, Kelly Isham, Maciej Besta, Nils Blach, Torsten Hoefler, and Fabrizio Petrini. 2024. PolarStar: Expanding the Horizon of Diameter- 3 Networks. (2024), 345–357

    2024

  14. [22]

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. 2023. Accelerating distributed{MoE} training and inference with lina. In 2023 USENIX Annual Technical Conference (USENIX ATC 23). 945–959

    2023

  15. [23]

    Lightmatter. 2025. Lightmatter Announces Passage L200, the Fastest Co-Packaged Optics for AI. https://lightmatter.co/press-release/lightmatter-announces- passage-l200-the-fastest-co-packaged-optics-for-ai/. Accessed: 2025-04-02

    2025

  16. [24]

    Mistral AI. 2025. Models Overview. https://docs.mistral.ai/getting-started/ models/models_overview/ Accessed: 2025-04-14

    2025

  17. [25]

    MOE. 2023. GPT-4 Architecture and Infrastructure. https://152334h.github.io/ blog/knowing-enough-about-moe/ Accessed: 2025-02-14

    2023

  18. [26]

    Samuel K. Moore. 2025. A Crucial Optical Technology Has Finally Arrived. IEEE Spectrum (March 2025). https://spectrum.ieee.org/co-packaged-optics

    2025

  19. [27]

    NVIDIA. 2025. NeMo: Neural Modules Framework. https://github.com/NVIDIA/ NeMo. Accessed: 2025-02-27

    2025

  20. [28]

    NVIDIA Corporation. 2023. NVIDIA SHARP Collective Library. https://docs.nvidia. com/networking/display/SHARPv300/NVIDIA+SHARP+Collective+Library Ver- sion 3.0.0

    2023

  21. [29]

    Dylan Patel, Myron Xie, Daniel Nishball, Ivan Chiam, Patrick Zhou, Doug O’Laughlin, and Wega Chu. 2025. NVIDIA GTC 2025 – Built For Reasoning, Vera Rubin, Kyber, CPO, Dynamo Inference, Jensen Math, Feynman. SemiAnalysis (19 March 2025). https://semianalysis.com/2025/03/19/nvid...

    2025

  22. [30]

    Reddit. 2023. Rumoured GPT-4 architecture: simplified visualisation. https://www.reddit.com/r/LocalLLaMA/comments/1c1en6n/rumoured_gpt4_ architecture_simplified/?rdt=47320 Accessed: 2025-02-14

    2023

  23. [31]

    Omar Sanseviero, Lewis Tunstall, Philipp Schmid, Sourab Mangrulkar, Younes Belkada, and Pedro Cuenca. 2023. Mixture of Experts Explained. https: //huggingface.co/blog/moe

    2023

  24. [32]

    SemiAnalysis. 2023. GPT-4 Architecture and Infrastructure. https://semianalysis. com/2023/07/10/gpt-4-architecture-infrastructure/ Accessed: 2025-02-14

    2023

  25. [33]

    Nicole Sheynin. 2024. 9 Data Center Trends and Outlook for 2025. https: //www.alpha-sense.com/blog/trends/data-center-trends/ Accessed: 2025-02-14

    2024

  26. [34]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  27. [35]

    Cerebras Systems. 2025. DARPA Selects Cerebras to Deliver Next Generation, Real-Time Compute Platform for Advanced Military and Commercial Applica- tions. https://www.businesswire.com/news/home/20250401398990/en/DARPA- Selects-Cerebras-to-Deliver-Next-Generation-Real-Time-Comp...

    2025

  28. [36]

    Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mohamed Mekkouri, Leandro Werra, and Thomas Wolf. 2025. The Ultra-Scale Playbook: Training LLMs on GPU Clusters

    2025

  29. [37]

    Ultra Ethernet Consortium. 2025. Ultra Ethernet Consortium. https:// ultraethernet.org/. Accessed: 2025-02-27

    2025

  30. [38]

    Wikipedia contributors. 2025. NVLink. Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/wiki/NVLink Accessed: 2025-02-25

    2025

  31. [39]

    Nitta, Venkatesh Akella, and S

    Yawei Yin, Roberto Proietti, Xiaohui Ye, Christopher J. Nitta, Venkatesh Akella, and S. J. Ben Yoo. 2013. LIONS: An AWGR-Based Low-Latency Optical Switch for High-Performance Computing and Data Centers. IEEE Journal of Selected Topics in Quantum Electronics 19, 2 (2013), 36004...

    2013

  32. [2024]

    arXiv preprint arXiv:2407.06204 (2024)

    A survey on mixture of experts. arXiv preprint arXiv:2407.06204 (2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.