Pith. sign in

REVIEW 2 major objections 4 minor 127 references

RailX: A Flexible, Scalable, and Low-Cost Network Architecture for Hyper-Scale LLM Training Systems

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RailX claims that separating node edges into Hamiltonian rail rings, interconnected by optical circuit switches and an on-package mesh, yields a flat network that scales beyond 100,000 chips and cuts per-bandwidth cost to under 10% of a…

desk verdict The architecture is genuinely interesting, but the headline cost claims don't survive a close look at the actual external port counts. read the letter →

arxiv 2507.18889 v1 pith:IUHQEA6C submitted 2025-07-25 cs.AR cs.DCcs.NI

Yinxiao Feng , Tiancheng Chen , Yuchen Wei , Siyuan Shen , Shiju Wang , Wei Li , Kaisheng Ma , Torsten Hoefler This is my paper · ORCID
classification cs.ARcs.DCcs.NI
keywords RailXLLMtrainingnetworkopticalcircuitswitchingHamiltoniandecompositionall-to-alltopologyAll-Reducefat-treecostdimensionsplitting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RailX argues that hyper-scale LLM training networks do not need expensive multi-tier fat-trees. The paper proposes a flat, reconfigurable network built from two inexpensive ingredients: a high-bandwidth 2D-mesh inside each multi-chip node, and optical circuit switches that wire node edges into separate rails. By configuring those rails as Hamiltonian rings with different node orderings, a small number of rings becomes an all-to-all mesh: any two nodes meet directly on two different rings. On that base, RailX claims to interconnect more than 100,000 chips with a single flat switching layer, diameter 2–4 inter-node hops, at under 10% of fat-tree cost per injection/All-Reduce bandwidth and under 50% of fat-tree cost per bisection/All-to-All bandwidth; the paper prices a 200K-chip system with 1.8TB/s per chip at about $1.3B. A reader should care because networking is a fast-growing share of LLM training cost, and the same fabric must serve both ring collectives and all-to-all traffic such as mixture-of-experts communication.

What carries the argument

The load-bearing machinery is the rail-ring-based all-to-all interconnection built on Hamiltonian decomposition, the partition of a complete graph's edges into Hamiltonian cycles that each visit every vertex once. It creates direct links between every node pair from what would otherwise be separate rings. Around it, RailX uses the intra-node 2D-mesh as a high-bandwidth virtual switch, so long-distance optical links appear only at node edges, and a 2D-organized array of optical circuit switches replaces the centralized switching layer that limits earlier OCS-based designs. The dynamic-configuration counterpart is Dimension Splitting, which regroups rails into logical dimensions of chosen scale and bandwidth, letting one physical fabric emulate torus, HyperX, Dragonfly, or five-dimensional heterogeneous topologies.

What would settle it

Build a small RailX testbed with intra-node bandwidth equal to inter-node bandwidth ($k=1$) and measure all-to-all throughput; the paper's simulation predicts a sharp collapse at that ratio, so sustained near-theoretical throughput would falsify the mesh-switch premise. A bottom-up price quote for the 200K-chip, 1.8TB/s configuration that exceeds the claimed $1.3B would test the cost claim.

Watch

Extended reading notes

Core claim

The central discovery is a topological construction. In a complete directed graph on $k$ vertices, Hamiltonian decomposition partitions the edges into $k-1$ directed Hamiltonian cycles; physically, a node with $k-1$ rails can be wired on each rail as a ring with a different vertex order, so every pair of nodes is directly connected on two different rings, with small exceptions at $k=4$ and $k=6$. RailX asserts that this arrangement turns separate rings into an all-to-all topology, giving a diameter of only 2–4 inter-node hops and bisection bandwidth sufficient for all-to-all traffic. The paper further claims that placing an $m \times m$ 2D-mesh inside each node, using that mesh as a virtual switch, and organizing optical circuit switches in a 2D row/column layout removes the centralized switching bottleneck: with switch radix $R=128$ and $m=5$, 102,400 chips fit under one flat switching layer, and a 200K-chip system with 1.8TB/s per chip can be built for about $1.3B. On this base, ring-based All-Reduce and all-to-all communication are simultaneously optimized, and dimension splitting maps high-dimensional parallelism flexibly.

Load-bearing premise

The construction requires the node's internal 2D-mesh to move data at least twice as fast as its external optical links; if that ratio is not met, the mesh becomes the bottleneck and the claimed all-to-all throughput collapses.

Editorial extensions

If this is right

  • A single flat tier of 128-port optical switches can interconnect 102,400 chips, and the 200,704-chip configuration needs no second switching tier.
  • RailX's ring-collective and all-to-all traffic coexist: the same rails that give near-theoretical all-to-all throughput also feed hierarchical All-Reduce algorithms that beat 2D-ring on torus and HammingMesh.
  • Cost per injection/All-Reduce bandwidth falls to under 10% of a non-blocking fat-tree, and cost per bisection/All-to-All bandwidth falls to under 50%, with the 200K-chip system priced near $1.3B.
  • Dimension Splitting maps TP/CP/EP/DP/PP onto distinct dimensions with adjustable per-dimension bandwidth, so heterogeneous parallelism no longer forces a fixed-shape torus.
  • Optical reconfiguration routes around failed rows and columns; at a 0.1% failure rate, single-job availability stays above 90%, and MLaaS-style multi-job allocation can use essentially all remaining nodes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same ring-as-clique construction would work for any dense all-to-all workload, such as embedding lookups, recommender training, or scientific halo exchange, not just LLM training.
  • Beyond the paper, the 10% and 50% cost ratios rest on today's relative prices of OCS ports, passive copper, and active optical transceivers; a sensitivity analysis with future pricing could locate the crossover where fat-trees become cheaper.
  • Beyond the paper, because the paper's own figures show that $k=2$ internal bandwidth is nearly sufficient and $k=4$ adds little, the practical headroom is set by packaging and co-packaged-optics yields rather than by topology; a measured $k=2$ node demonstration would de-risk the claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. RailX is a proposed reconfigurable, flat optical-circuit-switched network for hyper-scale LLM training. Each node is an m×m 2D-mesh of chips with high-bandwidth on-package links; boundary ports are converted to optics and wired to a 2D-organized set of OCSs. Using Hamiltonian decomposition, the paper organizes rails into rings that give two direct links between every node pair, enabling Torus, HyperX, and Dragonfly configurations, plus a dimension-splitting mechanism for heterogeneous parallelism. The claimed results are scalability beyond 100K chips with a flat 128-port switching layer, diameter 2–4 inter-node hops, and network cost per injection/All-Reduce bandwidth below 10% of Fat-Tree, with a roughly $1.3B system for 200K chips. Evaluation combines an analytical model, a cycle-based simulator, and component-level cost tables.

Significance. If the cost and performance claims were correct, RailX would be an important architecture: it would combine flat OCS scalability, all-to-all connectivity, and low diameter at a small fraction of the cost of Fat-Tree-based fabrics. The Hamiltonian-ring construction and the 2D OCS organization are elegant, and the scaling formulas in Eqs. (1)–(4) are internally consistent. The paper also provides a transparent component-level cost model and a detailed simulation setup, which are strengths. However, the central cost-effectiveness claim is undermined by an external-port accounting inconsistency: the cost tables count only about 5 optical ports per chip for the flagship configuration while crediting 36 ports per chip in the cost-per-injection comparison. This is a load-bearing issue for the headline claims, not a presentation detail; the architecture remains interesting, but the quantitative contributions need substantial correction.

major comments (2)
  1. [§6.2, Table 6, Eq. (1)] The cost comparison is internally inconsistent. With R=128, m=7, n=9, Eq. (1) gives N_s = rR = 8064 OCSs, hence 8064×128 = 1,032,192 OCS (and AOT) ports. For N=200,704 chips this is 1,032,192/200,704 = 5.14 optical ports per chip, i.e., about 2.06 Tb/s at 400G, not the 36×400G = 14.4 Tb/s per chip assumed in §6.2 for a fair comparison. The AOT count in Table 6 (1032.2K) confirms this. Consequently the 'Cost/Inject' column, which sets 2-Tier FT to 1, credits RailX7Mesh with a per-chip injection bandwidth it does not have: using the actual 5.14 ports/chip, the corrected ratio is (1314.4/415.9)×(2048×36)/(200704×5.14) ≈ 0.226×, not the reported 0.03×. Against the 4-tier nonblocking Fat-Tree at the same scale (cost/Inject 2.10×), the ratio is ≈10.8%, so the abstract's '<10% of Fat-Tree' is not supported, and the claimed $1.3B system with 1.8TB/s per-chip bandwidth is not backed by the disclosed component counts. The same correction applies to RailX4Mesh (589,824 AOTs over 65,536 chips = 9 ports/chip). Because §6.2 states that cost per injection bandwidth approximates cost per All-Reduce bandwidth, the cost-per-All-Reduce claim inherits this error.
  2. [§3.2, §6.3, Fig. 14] The performance evaluation uses a different port-count convention than the physical architecture. The RailX-2D-HyperX simulation in Fig. 14 is stated as m=4, n=2, but the definitions in §3.2 and Eq. (1) imply only 4n/m = 2 optical ports per chip for that configuration; the simulator nevertheless gives every chip 8 flits/cycle/chip injection bandwidth ('each chip has 8 ports'). If Fig. 14 is an equal-port-count comparison, this needs to be stated explicitly, together with the implied n and OCS radix; if it is meant to model the physical RailX configuration, it overstates the per-chip external bandwidth by 4×. Either way, the simulation results cannot be directly combined with the cost tables to support the cost-per-bandwidth claims.
minor comments (4)
  1. [Abstract and §3.3] The abstract states that the diameter is only 2–4 inter-node hops, but the Torus configuration has diameter R (Table 2); the statement should be qualified to apply to the HyperX and Dragonfly configurations, not to RailX in general.
  2. [§3.2 and §6.2] The symbol n is used inconsistently: §3.2 defines n as the number of off-package ports per chip edge, while §6.2 says 'n=9 rails per chip.' These are different quantities, and the inconsistency is a direct source of the accounting confusion in Table 6.
  3. [Table 3] The column 'Glob. BW [% Inject]' mixes the two port-count conventions: for RailX7Mesh the 7.1% value is computed relative to a 36-port/chip injection bandwidth, whereas with the actual 5.14 ports/chip the same all-to-all throughput is 50% of the per-chip injection bandwidth. The column should be recomputed and explicitly defined.
  4. [Throughout] There are several typos and unclear statements that should be cleaned up, including 'Draonglfy' in the §3.3 heading, 'Tours' in §2.2.2, and the phrase 'default input buffer size ... 16 flits (i.e., maximum message size)' in §6.1.2, which conflicts with the stated packet length of 4 flits.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; RailX's topology rests on an external Hamiltonian-decomposition theorem, and cost ratios follow a disclosed cost model, though the external-port accounting inconsistency is a correctness risk rather than a circular step.

full rationale

The central topological claim (Lemma 3.1) is grounded in Tillson's external Hamiltonian Decomposition theorem [110], with an explicit constructive sketch in Appendix A.1; it is not imported from the authors' own prior work. The cost comparison in Section 6.2 is built from disclosed component prices, switch radices, and Eq. (1), so the '<10% of Fat-Tree' cost-per-bandwidth figures follow arithmetically from the stated model rather than from a fitted parameter or a self-citation chain. The k>2 intra-node bandwidth condition in Eq. (5) is checked by the paper's own cycle-based simulations (Figure 14b), not by circular appeal to an author-derived theorem. The self-citations present (CNSim [37], Switch-Less Dragonfly [36], Interface Grouping [38]) are tooling or motivation references and do not uniquely force the headline result; the Hamiltonian-ring construction and the disclosed cost arithmetic are independent content. The external-port accounting issue raised by the skeptic (Table 6's 8064 OCSes x 128 ports gives about 5.14 optical ports per chip for RailX7Mesh, while the cost-per-injection metric credits each chip with 36x400G off-package bandwidth) is a real correctness and consistency concern, but it is not circularity: the cost model is explicit, and correcting the denominator would change the quantitative claim without collapsing the derivation into its inputs.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central scaling and cost claims rest on a classical graph-decomposition theorem, on the availability of advanced packaging with k > 2 intra-node bandwidth, and on OCS pricing assumptions. The design introduces no new physical entities, but it does rely on several aggressive hardware-feasibility and cost assumptions that are not independently validated at the claimed scale.

free parameters (3)
  • intra-node to inter-node bandwidth multiplier k = k = 4 in simulations; k > 2 claimed sufficient
    The all-to-all and All-Reduce performance claims depend on the 2D-mesh having k times the inter-node bandwidth. The paper sweeps k = 1, 2, 4 and asserts k > 2 is needed (Eq. 5, Fig. 14b). This is a design parameter chosen by hand, not derived.
  • off-package rails per chip edge n = 9 in the cost model
    Section 6.2 sets n = 9 to match the 36x400G port assumption, but in the 7x7 node case this yields about 5.1 external optical ports per chip, not 36. The choice drives the cost-per-bandwidth ratios.
  • node mesh scale m = m = 7 for the cost case, m = 5 for the 100K-chip claim
    The node side length m determines how many chips share the node's external ports. Values are chosen to fit the OCS radix budget (r = m*n <= R/2) and to reach the claimed scales, rather than derived independently.
assumptions (6)
  • standard math A directed complete graph K_k* can be decomposed into k-1 directed Hamiltonian cycles for all k except 4 and 6 (Tillson's theorem).
    Used in Lemma 3.1 and Appendix A.1 to construct all-to-all interconnection from separate rail rings.
  • domain assumption Advanced packaging and co-packaged optics deliver intra-node bandwidth at least 2-4x the inter-node optical bandwidth per chip.
    Eq. (5) in Section 3.3.5 requires k > 2 for the 2D-mesh to serve as a non-blocking virtual switch. This is a physical feasibility assumption not demonstrated at the proposed scale.
  • domain assumption A 128-port optical circuit switch costs about the same as a 64-port electrical packet switch.
    This pricing equivalence, stated in Section 6.2 and Appendix A.7, is a load-bearing input for the claim that RailX costs less than 10% of fat-tree per All-Reduce bandwidth.
  • domain assumption Optical circuit switch reconfiguration time fits into the millisecond-scale gap between communication phases in training.
    Section 5.2 and Appendix A.4 assume the 6 ms gap between CP and EP communication is enough to reconfigure the OCS. This is measured on a small 64-GH200 testbed, not at hyper-scale.
  • domain assumption The analytical model calibrated on 64 GH200 chips extrapolates to 200K chips.
    Section 6.1.1 and Appendix A.4 validate on small-scale hardware and scale by FLOPS multiples and communication volumes. The extrapolation assumes no scale-dependent effects dominate.
  • domain assumption Co-packaged optics provide the port density needed for the node edge ports (e.g., 32 optical ports per chip edge).
    Section 3.3.2 cites Broadcom's CPO result for 32 ports per chip edge. The scalability formulas rely on enough off-package ports per chip edge to reach the claimed chip counts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RailX: A Flexible, Scalable, and Low-Cost Network Architecture for Hyper-Scale LLM Training Systems." pith.science (2026). https://pith.science/paper/IUHQEA6C

@misc{pith2026250718889,
  author       = {Pith},
  title        = {Pith review of: RailX: A Flexible, Scalable, and Low-Cost Network Architecture for Hyper-Scale LLM Training Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUHQEA6C}},
  note         = {Machine review of arXiv:2507.18889}
}
abstract

Increasingly large AI workloads are calling for hyper-scale infrastructure; however, traditional interconnection network architecture is neither scalable nor cost-effective enough. Tree-based topologies such as the \textit{Rail-optimized} network are extremely expensive, while direct topologies such as \textit{Torus} have insufficient bisection bandwidth and flexibility. In this paper, we propose \textit{RailX}, a reconfigurable network architecture based on intra-node direct connectivity and inter-node circuit switching. Nodes and optical switches are physically 2D-organized, achieving better scalability than existing centralized circuit switching networks. We propose a novel interconnection method based on \textit{Hamiltonian Decomposition} theory to organize separate rail-based rings into \textit{all-to-all} topology, simultaneously optimizing ring-collective and all-to-all communication. More than $100$K chips with hyper bandwidth can be interconnected with a flat switching layer, and the diameter is only $2\sim4$ inter-node hops. The network cost per injection/All-Reduce bandwidth of \textit{RailX} is less than $10\%$ of the Fat-Tree, and the cost per bisection/All-to-All bandwidth is less than $50\%$ of the Fat-Tree. Specifically, only $\sim$\$$1.3$B is required to interconnect 200K chips with 1.8TB bandwidth. \textit{RailX} can also be used in the ML-as-a-service (MLaaS) scenario, where single or multiple training workloads with various shapes, scales, and parallelism strategies can be flexibly mapped, and failures can be worked around.

Figures

Figures reproduced from arXiv: 2507.18889 by the authors.

Figure 1
Figure 1. Scalability comparison of different topologies. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Rail-optimized Fat-Tree. The 𝑖-th GPU in each node is connected to the 𝑖-th rail switch. packages into large system-level networks [109], the mis￾matched bandwidth can lead to low utilization. Besides, map￾ping highly parallelized AI training workloads on such net￾works is also challenging. 2.2 Challenges of Existing Networks for AI 2.2.1 Fat-Tree is expensive. In existing architectures, processors within a node are… view at source ↗
Figure 4
Figure 4. High-dimensional (TP, CP, EP, DP, PP) het [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (15 more)
Figure 5
Figure 5. Figure 5: Rail-ring-based interconnection. (a) Each [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: RailX physical architecture. (a) Chips within a node are connected into a 2D-mesh topology by high [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: 2D-HyperX configuration of 25 nodes. 4 X [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Dragonfly configuration. Five nodes are lo [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: High-dimensional heterogeneous topology (4 × 3 × 3 × 𝐷 × 𝑃) and the mapping of the training workload (𝑇 𝑃 × 𝐶𝑃 × 𝐸𝑃 × 𝐷𝑃 × 𝑃𝑃). A more detailed version is shown in § A.3 [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Minimal and non-minimal adaptive routing [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 12
Figure 12. Figure 12: Expert parallelism. algorithm on the 𝑚2 × 𝑝 × 𝑝 RailX network is estimated as 𝑇RailX ≈ 2 × 𝑉 2𝒌𝑛𝐵 +  4𝑝𝛼 + 𝑉 /𝒎2 2𝑛𝐵/𝒎  = 4𝑝𝛼 +  2 𝑘 + 1 𝑚  𝑉 2𝑛𝐵 , (8) where 𝑘 is the multiple of on-package bandwidth (in local 2D-mesh) over off-package bandwidth. We can see that t…
Figure 13
Figure 13. Figure 13: Circuit switching to allocate all bandwidth [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: All-to-all performance. (a) Different topolo [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: All-Reduce performance. 6.4 All-Reduce Performance We also evaluate the All-Reduce performance of RailX. 1D￾ring, 2D-Torus [48], and hierarchical-2D-Torus algorithm presented in § 4.2 are evaluated with different scales and sizes. We assume each chip has four ports (d…
Figure 17
Figure 17. Figure 17: Availability of single allocation with OCS [PITH_FULL_IMAGE:figures/full_fig_p012_17.png]
Figure 18
Figure 18. Figure 18: Hamiltonian decomposing of𝐾 ∗ 2𝑚+1 complete graph. It has been proven that a directed complete graph 𝐾𝑛 (𝑛 ≠ 4, 6) can always be decomposed into 𝑛 −1 directed Hamilton￾ian cycles [11, 110]. If 𝑛 = 2𝑚 + 1, there is a straightforward construction to decompose the graph …
Figure 19
Figure 19. Figure 19: High-dimensional heterogeneous topology ( [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Allocation of jobs with failure nodes. In the MLaaS scenario, we can use multiple small work￾loads to fully utilize all functional nodes in the faulted RailX. As shown in [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 21
Figure 21. Figure 21: Trace example. CP and EP are separated by 6 ms. [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

127 extracted references · 16 canonical work pages

  1. [1]

    Common Electrical I/O (CEI) - Electrical and Jitter Interoper- ability Agreements for 6G+ Bps, 11G+ Bps, 25G+ Bps, 56G+ Bps and 112G+ Bps I/O

    2022. Common Electrical I/O (CEI) - Electrical and Jitter Interoper- ability Agreements for 6G+ Bps, 11G+ Bps, 25G+ Bps, 56G+ Bps and 112G+ Bps I/O. (Dec. 2022). www.oiforum.com/wp-content/uploads/ OIF-CEI-5.1.pdf

    2022

  2. [2]

    Marvell Teralynx 51.2T Ethernet Switch Enters Volume Produc- tion for Global AI Cloud Deployments

    2024. Marvell Teralynx 51.2T Ethernet Switch Enters Volume Produc- tion for Global AI Cloud Deployments. https://www.marvell.com/ company/newsroom/marvell-teralynx-512t-ethernet-switch- enters-volume-production-for-global-ai-cloud-deployments .html. (2024). https://www .marvell.com/company/newsroom/marvell- teralynx-512t-ethernet-switch-enters-volume-prod...

    2024

  3. [3]

    Photonic Optical Circuit Switching | CALIENT Technologies

    2024. Photonic Optical Circuit Switching | CALIENT Technologies. https://www.calient.net/. (2024). https://www.calient.net/

    2024

  4. [4]

    Universal Chiplet Interconnect Express (UCIe) Specification Revision 2.0

    2024. Universal Chiplet Interconnect Express (UCIe) Specification Revision 2.0. (Aug. 2024). https://www.uciexpress.org/specifications

    2024

  5. [5]

    Mellanox Quantum-2 QM9700 64-Port Non-Blocking Managed NDR 400Gb/s InfiniBand Switch - Part ID: MQM9700- NS2F - Switches - Colfax Direct

    2025. Mellanox Quantum-2 QM9700 64-Port Non-Blocking Managed NDR 400Gb/s InfiniBand Switch - Part ID: MQM9700- NS2F - Switches - Colfax Direct. https://www .colfaxdirect.com/ store/pc/viewPrd.asp?idproduct=4162&idcategory=7. (2025). https://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct= 4162&idcategory=7

    2025

  6. [6]

    NVIDIA 800Gb/s, Twin-Port, OSFP Multimode SR8 Transceiver - Finned Top - Part ID: MMA4Z00-NS - Colfax Direct

    2025. NVIDIA 800Gb/s, Twin-Port, OSFP Multimode SR8 Transceiver - Finned Top - Part ID: MMA4Z00-NS - Colfax Direct. https://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct= 4272&idcategory=0. (2025). https://www.colfaxdirect.com/store/pc/ viewPrd.asp?idproduct=4272&idcategory=0

    2025

  7. [7]

    NVIDIA Passive Copper Cable, IB Twin Port NDR 800Gb/s, OSFP, 0.5 Meter - Part ID: MCP4Y10-N00A - Colfax Direct

    2025. NVIDIA Passive Copper Cable, IB Twin Port NDR 800Gb/s, OSFP, 0.5 Meter - Part ID: MCP4Y10-N00A - Colfax Direct. https://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct= 4166&idcategory=0. (2025). https://www.colfaxdirect.com/store/pc/ viewPrd.asp?idproduct=4166&idcategory=0

    2025

  8. [8]

    NVIDIA/Megatron-LM

    2025. NVIDIA/Megatron-LM. NVIDIA Corporation. (Jan. 2025). https://github.com/NVIDIA/Megatron-LM

    2025

Show all 127 references
  1. [9]

    Schreiber

    Jung Ho Ahn, Nathan Binkert, Al Davis, Moray McLaren, and Robert S. Schreiber. 2009. HyperX: Topology, Routing, and Packaging of Ef- ficient Large-Scale Networks. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis . ACM, Portland Or...

    2009

  2. [10]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyan- skiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. (Dec. 2023). https://doi.org/10.48550/arXiv.2305.13245 arXiv:cs/2305.13245

    2023 doi arXiv

  3. [11]

    Brian Alspach. 2008. The Wonderful Walecki Construction. 52 (2008), 7–20

    2008

  4. [12]

    Andrew Gibiansky. 2017. Bringing HPC Techniques to Deep Learning . Technical Report. Baidu Research. https://andrew .gibiansky.com/ blog/machine-learning/baidu-allreduce/

    2017

  5. [13]

    Luiz André Barroso, Urs Hölzle, and Ranganathan Parthasarathy. 2019. The Datacenter as a Computer: Designing Warehouse-Scale Machines (third ed.). Springer, Cham, Switzerland

    2019

  6. [14]

    Camara, Miquel Moreto, Enrique Vallejo, Ramon Beivide, Jose Miguel-Alonso, Carmen Martinez, and Javier Navaridas

    Jose M. Camara, Miquel Moreto, Enrique Vallejo, Ramon Beivide, Jose Miguel-Alonso, Carmen Martinez, and Javier Navaridas. 2010. Twisted Torus Topologies for Enhanced Interconnection Networks. IEEE Transactions on Parallel and Distributed Systems 21, 12 (Dec. 2010), 1765–1778. ...

    2010 doi

  7. [16]

    Shuangliang Chen, Saptadeep Pal, and Rakesh Kumar. 2024. Wafer- scale Network Switches. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) . IEEE, Buenos Aires, Argentina, 215–229. https://doi.org/10.1109/ISCA59077.2024.00025

    2024

  8. [17]

    Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. 2022. Towards Understanding the Mixture-of-Experts Layer in Deep Learning. In Advances in Neural Information Process- ing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. ...

    2022

  9. [18]

    M. Cho, U. Finkler, M. Serrano, D. Kung, and H. Hunter. 2019. BlueCon- nect: Decomposing All-Reduce for Deep Learning on Heterogeneous Network Hierarchy. IBM Journal of Research and Development 63, 6 (Nov. 2019), 1:1–1:11. https://doi.org/10.1147/JRD.2019.2947013

    2019

  10. [19]

    Jack Choquette. 2023. NVIDIA Hopper H100 GPU: Scaling Perfor- mance. IEEE Micro 43, 3 (May 2023), 9–17. https://doi.org/10.1109/ MM.2023.3256796 13

    2023

  11. [20]

    Shu-Rong Chun, Tin-Hao Kuo, Hao-Yi Tsai, Chung-Shi Liu, Chuei- Tang Wang, Jeng-Shien Hsieh, Tsung-Shu Lin, Terry Ku, and Dou- glas Yu. 2020. InFO_SoW (System-on-Wafer) for High Performance Computing. In 2020 IEEE 70th Electronic Components and Technology Conference (ECTC). IEE...

    2020

  12. [21]

    Kari A. Clark, Daniel Cletheroe, Thomas Gerard, Istvan Haller, Krzysztof Jozwik, Kai Shi, Benn Thomsen, Hugh Williams, Geor- gios Zervas, Hitesh Ballani, Polina Bayvel, Paolo Costa, and Zhixin Liu. 2020. Synchronous Subnanosecond Clock and Data Recovery for Optically Switched ...

    2020 doi

  13. [22]

    Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture...

    2024 doi arXiv

  14. [23]

    Dally and Brian Towles

    William J. Dally and Brian Towles. 2004. Principles and Practices of Interconnection Networks. Morgan Kaufmann Publishers, Amsterdam ; San Francisco

    2004

  15. [24]

    Davide Tonietto. 2023. Energy Efficiency in Serial Links. (Aug. 2023). https://old.hoti.org/2023/event/diamond-sponsor-talks/index.html

    2023

  16. [25]

    McMahon, Duncan Roweth, and Torsten Hoefler

    Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon, Duncan Roweth, and Torsten Hoefler. 2020. An In-Depth Analysis of the Slingshot Interconnect. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis. IEEE, Atlanta, GA, USA, ...

    2020 arXiv

  17. [27]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei...

    2025 doi arXiv

  18. [28]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei...

    2024 doi arXiv

  19. [29]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guant- ing Chen, Guowei...

    2024 doi arXiv

  20. [30]

    Douglas Yu. 2021. TSMC Packaging Technologies for Chiplets and 3D. In Proceedings of the 2021 IEEE Hot Chips (HCS)

    2021

  21. [31]

    Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, Xipeng Qiu, Dahua Lin, Yonggang Wen, Xin Jin, Tian- wei Zhang, and Peng Sun. 2024. Efficient Training of Large Lan- guage Models on Distributed Inf...

    2024 arXiv

  22. [32]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

  23. [33]

    Elster and Tor A

    Anne C. Elster and Tor A. Haugdahl. 2022. Nvidia Hopper GPU and Grace CPU Highlights. Computing in Science & Engineering 24, 2 (March 2022), 95–100. https://doi.org/10.1109/MCSE.2022.3163817

    2022

  24. [35]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Trans- formers: Scaling to Trillion Parameter Models with Simple and Effi- cient Sparsity. (June 2022). https://doi.org/10.48550/arXiv.2101.03961 arXiv:cs/2101.03961

    2022 doi arXiv

  25. [36]

    Yinxiao Feng and Kaisheng Ma. 2024. Switch-Less Dragonfly on Wafers: A Scalable Interconnection Architecture Based on Wafer- Scale Integration. In SC24: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis . IEEE, Atlanta, GA, USA, 1–17. ...

    2024 arXiv

  26. [37]

    Yinxiao Feng, Yuchen Wei, Dong Xiang, and Kaisheng Ma. 2024. Evaluating Chiplet-Based Large-Scale Interconnection Networks via Cycle-Accurate Packet-Parallel Simulation. In 2024 USENIX Annual Technical Conference (USENIX ATC 24). USENIX Association, Santa Clara, CA, USA, 731–7...

    2024

  27. [38]

    Yinxiao Feng, Dong Xiang, and Kaisheng Ma. 2023. A Scalable Methodology for Designing Efficient Interconnection Network of Chiplets. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, Montreal, QC, Canada, 1059–

    2023

  28. [39]

    Michael Y. Frankel. 2021. Prospects for Optical Transceivers Expand- ing to Access, Metro and Long-Haul. In Optical Fiber Communication Conference (OFC) 2021. Optica Publishing Group, Washington, DC, Tu5A.2. https://doi.org/10.1364/OFC.2021.Tu5A.2

    2021 doi

  29. [40]

    Mingye Fu, Guangyao Liu, Roberto Proietti, Yichi Zhang, and S. J. Ben Yoo. 2021. First Demonstration of Monolithic Silicon Photonic Integrated Circuit 32 ×32 Thin-CLOS AWGR for All-to-All Inter- connections. In 2021 European Conference on Optical Communica- tion (ECOC). IEEE, ...

    2021

  30. [41]

    Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Rif- tadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, Shuqiang Zhang, Mikel Jimenez Fernandez, Shashidhar Gandham, and Hongyi Zeng

  31. [42]

    Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu, Radu Mantu, and Costin Raiciu

    Alexandru M. Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu, Radu Mantu, and Costin Raiciu. 2024. I’ve Got 99 Problems But FLOPS Ain’t One. In Proceed- ings of the 23rd ACM Workshop on Hot Topics in Networks. ACM, Irvine CA USA, 195–2...

    2024

  32. [43]

    Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu, Radu Mantu, and Costin Raiciu

    Alexandru M. Gherghescu, Vlad-Andrei Bădoiu, Alexandru Agache, Mihai-Valentin Dumitru, Iuliu Vasilescu, Radu Mantu, and Costin Raiciu. 2024. A Look Into Training Large Language Models on Next Generation Datacenters. (July 2024). arXiv:cs/2407.12819 http:// arxiv.org/abs/2407.12819

    2024 arXiv

  33. [45]

    Google. 2024. TPU v4 Document. https://cloud.google.com/tpu/docs/ v4. (2024). https://cloud.google.com/tpu/docs/v4

    2024

  34. [46]

    Maltz, and Parveen Patel

    Albert Greenberg, James Hamilton, David A. Maltz, and Parveen Patel. 2008. The Cost of a Cloud: Research Problems in Data Center Networks. ACM SIGCOMM Computer Communication Review 39, 1 (Dec. 2008), 68–73. https://doi.org/10.1145/1496091.1496103

    2008

  35. [47]

    Glass and L.M

    C.J. Glass and L.M. Ni. 1992. The Turn Model for Adaptive Routing. In Proceedings the 19th Annual International Symposium on Computer Architecture. IEEE, Gold Coast, Australia, 278–287. https://doi.org/ 10.1109/ISCA.1992.753324

    1992

  36. [48]

    Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, Salvatore Di Girolamo, Shigang Li, Marco Heddes, Jon Belk, Deepak Goel, Miguel Castro, and Steve Scott. 2022. HammingMesh: A Network Topology for Large-Scale Deep Learning. In SC22: International Conference for High Performanc...

    2022 arXiv

  37. [49]

    Jason Howard. 2023. The First Direct Mesh-to-Mesh Photonic Fabric. In 2023 IEEE Hot Chips 35 Symposium (HCS) . IEEE, Palo Alto, CA, USA, 1–17. https://doi.org/10.1109/HCS59251.2023.10254719

    2023

  38. [50]

    Chuanxiong Guo, Guohan Lu, Dan Li, Haitao Wu, Xuan Zhang, Yunfeng Shi, Chen Tian, Yongguang Zhang, and Songwu Lu. 2009. BCube: A High Performance, Server-Centric Network Architecture for Modular Data Centers. In Proceedings of the ACM SIGCOMM 2009 Conference on Data Communicat...

    2009

  39. [51]

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, and zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural 16 Networks Using Pipeline Parallelism. In Advances in Neural Informa- tion Proc...

    2019

  40. [52]

    Patrick Iff, Maciej Besta, Matheus Cavalcante, Tim Fischer, Luca Benini, and Torsten Hoefler. 2023. HexaMesh: Scaling to Hundreds of Chiplets with an Optimized Chiplet Arrangement. In 2023 60th ACM/IEEE Design Automation Conference (DAC). IEEE, San Francisco, CA, USA, 1–6. htt...

    2023

  41. [53]

    Yang Hu, Xinhan Lin, Huizheng Wang, Zhen He, Xingmao Yu, Ji- ahao Zhang, Qize Yang, Zheng Xu, Sihan Guan, Jiahao Fang, Hao- ran Shang, Xinru Tang, Xu Dai, Shaojun Wei, and Shouyi Yin. 2024. Wafer-Scale Computing: Advancements, Challenges, and Future Per- spectives [Feature]. I...

    2024

  42. [54]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. (Oct. 2023). arXiv:cs/2309.14509 http://arxi...

    2023 arXiv

  43. [55]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Men- sch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-An...

    2024 doi arXiv

  44. [56]

    Alexander Ishii and Ryan Wells. 2022. The Nvlink-Network Switch: Nvidia’s Switch Chip for High Communication-Bandwidth Super- pods. In 2022 IEEE Hot Chips 34 Symposium (HCS) . IEEE, Cupertino, CA, USA, 1–23. https://doi.org/10.1109/HCS55958.2022.9895480

    2022

  45. [57]

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi...

    2024

  46. [58]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Clifford Young, Xiang Zhou, Zongwei Zhou, and David A Patterson. 2023. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learn...

    2023

  47. [59]

    Becker, Brian Towles, William J

    Nan Jiang, James Balfour, Daniel U. Becker, Brian Towles, William J. Dally, George Michelogiannakis, and John Kim. 2013. A Detailed and Flexible Cycle-Accurate Network-on-Chip Simulator. In 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPA...

    2013

  48. [60]

    M. R. Siavash Katebzadeh, Paolo Costa, and Boris Grot. 2020. Evalua- tion of an InfiniBand Switch: Choose Latency or Bandwidth, but Not Both. In 2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, Boston, MA, USA, 180–191. https://...

    2020

  49. [61]

    Kevin Zhang. 2024. An Option Besides CoWoS: System-on-Wafer (TSMC-SoW). (2024)

    2024

  50. [62]

    Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Ben- jamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. (Jan. 2020). arXiv:cs, stat/2001.08361 http://arxiv.org/abs/2001.08361

    2020 arXiv

  51. [63]

    Dally, Steve Scott, and Dennis Abts

    John Kim, Wiliam J. Dally, Steve Scott, and Dennis Abts. 2008. Technology-Driven, Highly-Scalable Dragonfly Topology. In 2008 International Symposium on Computer Architecture . IEEE, Beijing, China, 77–88. https://doi.org/10.1109/ISCA.2008.19

    2008 doi

  52. [64]

    Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. 2022. Reducing Activation Recomputation in Large Transformer Models. (May 2022). arXiv:cs/2205.05198 http://arxiv.org/abs/2205.05198

    2022 arXiv

  53. [65]

    Mehrdad Khani, Manya Ghobadi, Mohammad Alizadeh, Ziyi Zhu, Madeleine Glick, Keren Bergman, Amin Vahdat, Benjamin Klenk, and Eiman Ebrahimi. 2021. SiP-ML: High-Bandwidth Optical Network Interconnects for Machine Learning Training. In Proceedings of the 2021 ACM SIGCOMM 2021 Con...

    2021

  54. [66]

    Sameer Kumar and Norm Jouppi. 2020. Highly Available Data Parallel ML Training on Mesh Networks. (Nov. 2020). arXiv:cs/2011.03605 http://arxiv.org/abs/2011.03605

    2020 arXiv

  55. [67]

    Sabuj Laskar, Pranati Majhi, Sungkeun Kim, Farabi Mahmud, Ab- dullah Muzahid, and Eun Jung Kim. 2024. Enhancing Collective Communication in MCM Accelerators for Deep Learning Training. In 2024 IEEE International Symposium on High-Performance Com- puter Architecture (HPCA). IEE...

    2024

  56. [68]

    John H. Lau. 2019. Recent Advances and Trends in Fan-Out Wafer/Panel-Level Packaging. Journal of Electronic Packaging 141, 4 (Dec. 2019), 040801. https://doi.org/10.1115/1.4043341

    2019 doi

  57. [69]

    Tushar Krishna. 2017. Garnet2.0: A Detailed On-Chip Network Model Inside a Full-System Simulator. (2017)

    2017

  58. [70]

    Hong Liu, Ryohei Urata, Kevin Yasumura, Xiang Zhou, Roy Bannon, Jill Berger, Pedram Dashti, Norm Jouppi, Cedric Lam, Sheng Li, Erji Mao, Daniel Nelson, George Papen, Mukarram Tariq, and Amin Vah- dat. 2023. Lightwave Fabrics: At-Scale Optical Circuit Switching for Datacenter a...

    2023

  59. [71]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring Attention with Blockwise Transformers for Near-Infinite Context. (Nov. 2023). arXiv:cs/2310.01889 http://arxiv.org/abs/2310.01889

    2023 arXiv

  60. [72]

    Kefei Liu, Jiao Zhang, Zhuo Jiang, Xuan Zhang, Shixian Guo, Yangyang Bai, Yongbin Dong, Zhang Zhang, Xiang Shi, Lei Wang, Haoran Wei, Zicheng Wang, Yongchen Pan, Tian Pan, and Tao Huang

  61. [73]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GShard: Scaling Giant Models with Condi- tional Computation and Automatic Sharding. (June 2020). arXiv:cs, stat/2006.16668 http://arxiv.or...

    2020 arXiv

  62. [74]

    Lizhong Chen and T. M. Pinkston. 2013. Worm-Bubble Flow Con- trol. In 2013 IEEE 19th International Symposium on High Perfor- mance Computer Architecture (HPCA) . IEEE, Shenzhen, 366–377. https://doi.org/10.1109/HPCA.2013.6522333 17

    2013

  63. [75]

    Piotr Luczynski, Lukas Gianinazzi, Patrick Iff, Leighton Wilson, Daniele De Sensi, and Torsten Hoefler. 2024. Near-Optimal Wafer- Scale Reduce. In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing. ACM, Pisa Italy, 334–347. ...

    2024

  64. [76]

    Pavlos Maniotis and Daniel M. Kuchta. 2024. Exploring the Benefits of Using Co-Packaged Optics in Data Center and AI Supercomputer Networks: A Simulation-Based Analysis [Invited]. Journal of Optical Communications and Networking 16, 2 (Feb. 2024), A143. https:// doi.org/10.136...

    2024 doi

  65. [77]

    In Proceedings of the 8th Asia-Pacific Workshop on Net- working

    Hostmesh: Monitor and Diagnose Networks in Rail-Optimized RoCE Clusters. In Proceedings of the 8th Asia-Pacific Workshop on Net- working. ACM, Sydney Australia, 122–128. https://doi.org/10.1145/ 3663408.3663426

  66. [80]

    Cyriel Minkenberg, Rajagopal Krishnaswamy, Aaron Zilkie, and David Nelson. 2021. Co-packaged Datacenter Optics: Opportuni- ties and Challenges. IET Optoelectronics 15, 2 (April 2021), 77–91. https://doi.org/10.1049/ote2.12020

    2021 doi

  67. [81]

    NADDOD. 2024. 64-Port Ethernet Switch,800Gb Managed Switch,L3 4U Switch - NADDOD. https://www .naddod.com/products/ 102322.html. (2024). https://www.naddod.com/products/102322.html

    2024

  68. [82]

    Nie McDonald, Mikhail Isaev, Adriana Flores, Al Davis, and John Kim. 2019. Practical and Efficient Incremental Adaptive Routing for HyperX Networks. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, Denver Col...

    2019

  69. [84]

    NVIDIA. 2022. Doubling All2all Performance with NVIDIA Collective Communication Library 2.12. https: //developer.nvidia.com/blog/doubling-all2all-performance- with-nvidia-collective-communication-library-2-12/. (Feb. 2022). https://developer.nvidia.com/blog/doubling-all2all-pe...

    2022

  70. [85]

    NVIDIA. 2023. NVIDIA DGX SuperPOD: Next Generation Scalable Infrastructure for AI Leadership. https://docs .nvidia.com/https: /docs.nvidia.com/dgx-superpod-reference-architecture-dgx- h100.pdf. (2023). https://docs .nvidia.com/https:/docs.nvidia.com/ dgx-superpod-reference-arc...

    2023

  71. [86]

    NVIDIA. 2024. Context Parallelism Overview. https:// docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/ context_parallel.html. (2024). https://docs .nvidia.com/megatron- core/developer-guide/latest/api-guide/context_parallel.html

    2024

  72. [87]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phan- ishayee, and Matei Zaharia. 2021. Efficient Large-Scale Language Model Training on GPU Cl...

    2021

  73. [88]

    Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krish- namoorthi, Ansha Yu, V...

    2019 arXiv

  74. [89]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

    2024 arXiv

  75. [90]

    Iyer, Sudhakar Pamarti, Rakesh Ku- mar, and Puneet Gupta

    Saptadeep Pal, Jingyang Liu, Irina Alam, Nicholas Cebry, Haris Suhail, Shi Bu, Subramanian S. Iyer, Sudhakar Pamarti, Rakesh Ku- mar, and Puneet Gupta. 2021. Designing a 2048-Chiplet, 14336- Core Waferscale Processor. In 2021 58th ACM/IEEE Design Automa- tion Conference (DAC) ...

    2021

  76. [91]

    Giannis Patronas, Nikos Terzenidis, Prethvi Kashinkunti, Eitan Za- havi, Dimitris Syrivelis, Louis Capps, Zsolt-Alon Wertheimer, Nikos Argyris, Athanasios Fevgas, Craig Thompson, Avraham Ganor, Julie Bernauer, Elad Mentovich, and Paraskevas Bakopoulos. 2025. Opti- cal Switchin...

    2025 doi

  77. [92]

    NVIDIA. 2024. NVIDIA Grace Hopper Superchip Architecture Whitepaper. https://resources.nvidia.com/en-us-grace-cpu/nvidia- grace-hopper. (2024). https://resources.nvidia.com/en-us-grace-cpu/ nvidia-grace-hopper

    2024

  78. [93]

    NVIDIA. 2024. NVIDIA Nsight Systems. https: //developer.nvidia.com/nsight-systems. (2024). https: //developer.nvidia.com/nsight-systems

    2024

  79. [94]

    Puente, C

    V. Puente, C. Izu, R. Beivide, J.A. Gregorio, F. Vallejo, and J.M. Prellezo

  80. [95]

    Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, Chao Wang, Peng Wang, Pengcheng Zhang, Xianlong Zeng, Eddie Ruan, Zhiping Yao, Ennan Zhai, and Dennis Cai. 2024. Alibaba HPN: A Data Center Network for Large Languag...

    2024

  81. [96]

    Sudarsanan Rajasekaran, Manya Ghobadi, and Aditya Akella. 2024. CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24) . 1403–1420. https://www.usenix.org/ conference/nsdi24/presenta...

    2024

  82. [97]

    Olusogo Popoola and Bernardi Pranggono. 2018. On Energy Con- sumption of Switch-Centric Data Center Networks. The Journal of Supercomputing 74, 1 (Jan. 2018), 334–369. https://doi.org/10.1007/ s11227-017-2132-5

    2018

  83. [98]

    Leon Poutievski, Omid Mashayekhi, Joon Ong, Arjun Singh, Mukar- ram Tariq, Rui Wang, Jianan Zhang, Virginia Beauregard, Patrick Conner, Steve Gribble, Rishi Kapoor, Stephen Kratzer, Nanfang Li, Hong Liu, Karthik Nagaraj, Jason Ornstein, Samir Sawhney, Ry- ohei Urata, Lorenzo V...

    2022

  84. [99]

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. Scaling Vision with Sparse Mixture of Experts. (June 2021). arXiv:cs, stat/2106.05974 http://arxiv.org/abs/2106.05974

    2021 arXiv

  85. [100]

    Sella, Andrew W

    Omer S. Sella, Andrew W. Moore, and Noa Zilberman. 2018. FEC Killed The Cut-Through Switch. In Proceedings of the 2018 Work- shop on Networking for Emerging Applications and Technologies . ACM, Budapest Hungary, 15–20. https://doi.org/10.1145/3229574.3229577

    2018

  86. [101]

    Daniele De Sensi, Tommaso Bonato, David Saam, and Torsten Hoefler

  87. [102]

    Aashaka Shah, Vijay Chidambaram, Meghan Cowan, Saeed Maleki, Madan Musuvathi, Todd Mytkowicz, Jacob Nelson, Olli Saarikivi, and Rachee Singh. 2023. TACCL: Guiding Collective Algorithm Synthe- sis Using Communication Sketches. In 20th USENIX Symposium on Networked Systems Desig...

    2023

  88. [103]

    Noam Shazeer. 2020. GLU Variants Improve Transformer. (Feb. 2020). https://doi.org/10.48550/arXiv.2002.05202 arXiv:cs/2002.05202

    2020 doi arXiv

  89. [104]

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power next-Generation AI Scale. In Pro- ceedings of the 39th Internatio...

    2022

  90. [105]

    Saeed Rashidi, William Won, Sudarshan Srinivasan, Srinivas Srid- haran, and Tushar Krishna. 2022. Themis: A Network Bandwidth- Aware Collective Scheduling Policy for Distributed Training of DL Models. In Proceedings of the 49th Annual International Symposium on Computer Archit...

    2022

  91. [106]

    Priyank Shukla. 2022. Short Reach Interconnect for the Emerging Multi-Die System Era. (2022). https://www .ieeetoronto.ca/ wp-content/uploads/2022/12/Short-Reach-Interconnect-for-the- Emerging-Multi-Die-System-Era.pdf

    2022

  92. [107]

    Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. 2023. A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of- Experts Training. In Proceedings of the 37th International Conference on Supercomputing. ACM,...

    2023

  93. [108]

    Synopsys. 2024. UALink IP Solution | Synopsys. https: //www.synopsys.com/designware-ip/interface-ip/ualink.html. (2024). https://www .synopsys.com/designware-ip/interface-ip/ ualink.html

    2024

  94. [109]

    In 21st USENIX Symposium on Networked Systems Design and Implemen- tation (NSDI 24)

    Swing: Short-Cutting Rings for Higher Bandwidth Allreduce. In 21st USENIX Symposium on Networked Systems Design and Implemen- tation (NSDI 24) . 1445–1462. https://www .usenix.org/conference/ nsdi24/presentation/de-sensi

  95. [110]

    Timothy W Tillson. 1980. A Hamiltonian Decomposition of K2m∗, 2m≥ 8. Journal of Combinatorial Theory, Series B 29, 1 (Aug. 1980), 68–74. https://doi.org/10.1016/0095-8956(80)90044-1

    1980 doi

  96. [111]

    Ajay Tirumala and Raymond Wong. 2024. NVIDIA Blackwell Plat- form: Advancing Generative AI and Accelerated Computing. In 2024 19 IEEE Hot Chips 36 Symposium (HCS) . IEEE, Stanford, CA, USA, 1–33. https://doi.org/10.1109/HCS61935.2024.10665247

    2024

  97. [112]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. (Jan. 2017). https://doi.org/10.48550/arXiv.1701.06538 arXiv:cs/1701.06538

    2017 doi arXiv

  98. [113]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. (March 2020). arXiv:cs/1909.08053 http://arxiv.org/abs/1909.08053

    2020 arXiv

  99. [114]

    Torres Arango

    Victor Avelar, Patrick Donovan, Paul Lin, Wendy Torell, and Maria A. Torres Arango. 2023. The AI Disruption: Challenges and Guidance for Data Center Design . Technical Report. Energy Management Research Center. https://globalitresearch.com/wp-content/uploads/ 2024/10/68834-Sch...

    2023

  100. [115]

    Songtao Wang, Dan Li, Yang Cheng, Jinkun Geng, Yanshu Wang, Shuai Wang, Shutao Xia, and Jianping Wu. 2020. A Scalable, High-Performance, and Fault-Tolerant Network Architecture for Distributed Machine Learning. IEEE/ACM Transactions on Net- working 28, 4 (Aug. 2020), 1752–1764...

    2020

  101. [116]

    Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. 2024. Rail-Only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters. (July 2024). arXiv:cs/2307.12169 http://arxiv.org/abs/2307.12169

    2024 arXiv

  102. [117]

    Emil Talpes, Debjit Das Sarma, Doug Williams, Sahil Arora, Thomas Kunjan, Benjamin Floering, Ankit Jalote, Christopher Hsiong, Chan- drasekhar Poorna, Vaidehi Samant, John Sicilia, Anantha Kumar Nivarti, Raghuvir Ramachandran, Tim Fischer, Ben Herzberg, Bill McGee, Ganesh Venk...

    2023

  103. [118]

    Weihao Wang, Jie Li, Chuanzhi Wang, Rong Cao, Shunbin Li, Zhi- quan Wan, Guandong Liu, Qingwen Deng, and Ruyun Zhang. 2023. Demonstration of a Wafer-Level Integration for System-on-Wafer Architecture. In 2023 24th International Conference on Electronic Pack- aging Technology (...

    2023

  104. [119]

    Ying Wei, Yi Chieh Huang, Haiming Tang, Nithya Sankaran, Ish Chadha, Dai Dai, Olakanmi Oluwole, Vishnu Balan, and Edward Lee. 2023. 9.3 NVLink-C2C: A Coherent Off Package Chip-to- Chip Interconnect with 40Gbps/Pin Single-Ended Signaling. In 2023 IEEE International Solid- State...

    2023

  105. [120]

    Davide Tonietto. 2022. The Future of Short Reach Interconnect. In ESSCIRC 2022- IEEE 48th European Solid State Circuits Confer- ence (ESSCIRC) . IEEE, Milan, Italy, 1–8. https://doi .org/10.1109/ ESSCIRC55480.2022.9911398

    2022

  106. [121]

    Ryohei Urata, Hong Liu, Kevin Yasumura, Erji Mao, Jill Berger, Xiang Zhou, Cedric Lam, Roy Bannon, Darren Hutchinson, Daniel Nelson, Leon Poutievski, Arjun Singh, Joon Ong, and Amin Vahdat. 2022. Mis- sion Apollo: Landing Optical Circuit Switching at Datacenter Scale. (Aug. 20...

    2022 arXiv

  107. [122]

    Charlotte Trueman Have your say. 2024. Nvidia Increases Black- well Orders from TSMC by 25 Percent; $1.8m GB200 NVL36 Server Cabinet Expected to Account for Bulk of Deliveries. https://www.datacenterdynamics.com/en/news/nvidia-increases- blackwell-orders-from-tsmc-by-25-percen...

    2024

  108. [123]

    Dianhai Yu, Liang Shen, Hongxiang Hao, Weibao Gong, Huachao Wu, Jiang Bian, Lirong Dai, and Haoyi Xiong. 2024. MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services. (Aug. 2024). arXiv:cs/2205.10034 http: //arxiv.org/abs/2205.10034

    2024 arXiv

  109. [124]

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. 2024. LLaMA-MoE: Building Mixture- of-Experts from LLaMA with Continual Pre-Training. (June 2024). arXiv:cs/2406.16554 http://arxiv.org/abs/2406.16554

    2024 arXiv

  110. [125]

    Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch

  111. [130]

    William Won, Midhilesh Elavazhagan, Sudarshan Srinivasan, Swati Gupta, and Tushar Krishna. 2024. TACOS: Topology-Aware Col- lective Algorithm Synthesizer for Distributed Machine Learning. In 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, Austin,...

    2024

  112. [131]

    Yongji Wu, Yechen Xu, Jingrong Chen, Zhaodong Wang, Ying Zhang, Matthew Lentz, and Danyang Zhuo. 2024. MCCS: A Service-Based Approach to Collective Communication for Multi-Tenant Cloud. In Proceedings of the ACM SIGCOMM 2024 Conference . ACM, Sydney NSW Australia, 679–690. htt...

    2024

  113. [135]

    Yazhou Zu, Alireza Ghaffarkhah, Hoang-Vu Dang, Brian Towles, Steven Hand, Safeen Huda, Adekunle Bello, Alexander Kolbasov, Arash Rezaei, Dayou Du, Steve Lacy, Hang Wang, Aaron Wisner, Chris Lewis, and Henri Bahini. 2024. Resiliency at Scale: Man- aging Google’s TPUv4 Machine L...

    2024

  114. [675]

    https://doi.org/10.1145/3452296.3472900

  115. [767]

    https://www.usenix.org/conference/nsdi23/presentation/wang- weiyang

  116. [1071]

    https://doi.org/10.1109/HPCA56546.2023.10070981

    2023

  117. [2001]

    The Adaptive Bubble Router. J. Parallel and Distrib. Comput. 61, 9 (Sept. 2001), 1180–1208. https://doi.org/10.1006/jpdc.2001.1746

    2001

  118. [2023]

    In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)

    TopoOpt: Co-Optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23) . 739–

  119. [2024]

    (July 2024)

    The Llama 3 Herd of Models. (July 2024). arXiv:cs/2407.21783 http://arxiv.org/abs/2407.21783

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.