Pith. sign in

REVIEW 4 major objections 4 minor 91 references

InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A transceiver-centric HBD with embedded OCS can scale LLM training to datacenter size while keeping interconnect cost at 31% of NVL-72 and GPU waste near zero.

desk verdict Genuinely new transceiver-centric OCS design with real silicon behind it, but the headline scaling numbers come from an unreleased simulator and the TP-only assumption is the load-bearing hinge. read the letter →

arxiv 2502.03885 v6 pith:WW7BNC5M submitted 2025-02-06 cs.NI cs.DCcs.LG

Chenchen Shou , Guyue Liu , Hao Nie , Huaiyu Meng , Yu Zhou , Yimin Jiang , Wenqing Lv , Yelong Xu
show 6 more authors
Yuanwei Lu Zhang Chen Yanbo Yu Yichen Shen Yibo Zhu Daxin Jiang
This is my paper · ORCID
classification cs.NIcs.DCcs.LG
keywords High-BandwidthDomainOpticalCircuitSwitchingSiliconPhotonicsTensorParallelismLLMtrainingringtopologyfaultresiliencedatacenternetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to prove that the High-Bandwidth Domain (HBD) needed for Tensor Parallelism in LLM training can be built at datacenter scale by moving switching from central switches into the optical transceivers themselves. Its InfiniteHBD design puts a compact silicon-photonic optical circuit switch inside every 800G transceiver, so links can be reconfigured in 60 to 80 microseconds to form rings, bypass failed nodes, and adapt TP group sizes. The claimed payoff is that interconnect cost drops to roughly 31% of NVIDIA's NVL-72, a single node failure wastes only that node's GPUs, and healthy GPUs keep full bandwidth. A reader should care because this directly attacks the two barriers that keep HBDs small today: cost explosion and fault explosion radius.

What carries the argument

The load-bearing object is OCSTrx, a silicon-photonic OCS transceiver that embeds a compact MZI switch matrix inside a commercial QSFP-DD 800G module, providing reconfigurable point-to-multipoint paths with 60-80 microseconds reconfiguration, an average 3.3dB insertion loss, and under 3.2W for the OCS core. On top of it sits the reconfigurable K-Hop Ring topology, where each node connects to neighbors within K hops, intra-node loopbacks create variable-size GPU rings, and backup links bypass failed nodes. The third pillar is a binary-search orchestration algorithm that places TP groups to keep cross-ToR DCN traffic near zero. Together these turn fault isolation from a switch-level or cube-level property into a node-level property while keeping interconnect cost linear in node count.

What would settle it

Run a production MoE training workload with real expert parallelism (AllToAll) on a hardware OCSTrx testbed at a group size larger than eight; if measured AllToAll throughput follows the O($p^{2}$) ring bound and the binary-exchange extension is not realized on hardware, then the architecture's generality claim for MoE workloads would fail. Alternatively, show a production fault trace where correlated multi-node failures exceed the K-hop bypass capacity, causing bandwidth degradation beyond the node level.

Watch

Extended reading notes

Core claim

The central discovery is that a transceiver-centric architecture, with OCS embedded in each transceiver rather than in a centralized switch, can simultaneously achieve scalability, linear cost, and node-level fault isolation, which the paper argues previous HBD designs could not. OCSTrx uses a Mach-Zehnder interferometer based silicon-photonic switch matrix integrated in a QSFP-DD 800G transceiver, enabling reconfigurable point-to-multipoint paths with 60 to 80 microseconds of switching latency. The reconfigurable K-Hop Ring topology uses intra-node loopback paths to form arbitrary-sized GPU rings and backup links to bypass failed nodes, while an HBD-DCN orchestration algorithm places TP groups to minimize cross-ToR traffic. On a real 348-day fault trace from a 3K-GPU cluster, TP-32 jobs show 0.53% GPU waste versus 10.04% for NVL-72 and 7.56% for TPUv4, interconnect cost is 30.86% of NVL-72, and MFU improves 3.37x over NVIDIA DGX 8-GPU systems.

Load-bearing premise

The design rests on the claim that mainstream LLM training can be served entirely by Tensor-Parallel Ring-AllReduce communication, so HBDs do not need efficient AllToAll or Expert-Parallel traffic.

Editorial extensions

If this is right

  • Datacenter operators can run TP sizes such as 32 and 64 without the fragmentation that small HBDs cause, cutting GPU waste below 1% on production fault traces.
  • A failed node no longer degrades bandwidth for surrounding healthy GPUs; the ring reroutes through backup links, so training continues at full bandwidth for the remaining nodes.
  • Interconnect cost scales roughly linearly with GPU count because OCSTrx replaces centralized NVLink switches and MEMS OCS fabrics; the K=2 configuration costs about $2,627 per GPU versus $9,563 per GPU for NVL-36/72.
  • The orchestration algorithm keeps cross-ToR traffic near zero up to roughly 7% node fault ratio, so DCN congestion from data-parallel and pipeline-parallel traffic is sharply reduced.
  • Larger optimal TP sizes for models like Llama 3.1-405B become reachable, yielding a 3.37x MFU improvement over 8-GPU DGX systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 60-80 microsecond reconfiguration can be overlapped with computation, the same OCSTrx hardware could time-slice topologies for multi-dimensional parallelism; the paper describes this possibility but does not validate it on hardware.
  • The TP-only design premise is a bet that MoE expert traffic will keep being converted to TP; if AllToAll-heavy expert parallelism remains dominant, the binary-exchange extension, analyzed only theoretically in Appendix G, must be demonstrated on hardware for the architecture to retain its generality claim.
  • The cost argument assumes the $600 OCSTrx price holds at production volume; silicon-photonic pricing curves could move the 31% cost ratio in either direction.
  • The node-fault independence assumption justifies K=3; a cluster with correlated failures, such as rack-wide power or cooling events, would require a larger K or an additional protection strategy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes InfiniteHBD, a transceiver-centric high-bandwidth domain architecture that embeds a silicon-photonics optical circuit switch (OCSTrx) inside each QSFP-DD transceiver, enabling reconfigurable K-hop ring topologies with node-level fault isolation and dynamic TP-group sizing. The authors report concrete hardware measurements for OCSTrx (60–80 µs reconfiguration latency, average 3.3 dB insertion loss, BER behavior, and power consumption) and a small-scale 32-GPU ring experiment. The datacenter-scale claims about GPU waste ratio, MFU, cost, and orchestration are obtained from an in-house simulator driven by a 348-day fault trace from the authors' own cluster. The architecture is deliberately designed for TP Ring-AllReduce traffic; §2.3 argues that EP/AllToAll is unnecessary for mainstream LLM training, while §7 and Appendix G acknowledge that AllToAll support is theoretical and not applied in the deployed design.

Significance. The transceiver-centric OCS design is a genuinely interesting architectural idea, and the hardware measurements are a concrete strength: the OCSTrx reconfiguration latency, insertion loss, BER, and power figures are directly measured, and the fault trace is open-sourced. If the scaling, cost, and MFU results were established, this would be a significant contribution to HBD design for LLM training. However, the flagship datacenter-scale claims are not yet established: they rest on an unreleased in-house simulator that embeds the paper's own design principles, on a TP-only traffic assumption that the paper's own appendix shows is not general for MoE workloads, and on cost assumptions for an unproduced component. The central hardware/architecture contribution is solid, but the scope of the headline claims needs to be either validated or substantially narrowed.

major comments (4)
  1. [§2.3, §7, Appendix G] The load-bearing design principle that HBD only needs to support TP Ring-AllReduce is contradicted by the paper's own appendix. Section 2.3 concludes that "ensuring support for Ring-AllReduce is sufficient for mainstream LLM training scenarios" based on a simulated GPT-MoE with a synthetic 20% expert-imbalance coefficient, but Appendix G states that "large MoE models such as DeepSeek-v3, both training and inference heavily rely on expert parallelism (EP)" and that the ring topology's AllToAll is O(p^2). The proposed Binary Exchange fix is explicitly "not yet validated on real hardware," imposes TP×EP≤2048 on 8-GPU nodes (TP×EP≤64 on 4-GPU nodes), adds GPU forwarding overhead and scheduling complexity, and §7 says "Therefore, it is not applied." Consequently, the architecture's guaranteed full-bandwidth property covers only TP Ring-AllReduce; for EP/AllToAll workloads, the paper does not establish datacenter-scale full-bandwidth support. The authors should either validate the AllToAll extension on real hardware and quantify its constraints, or recast the claims as TP-specific.
  2. [§6.1, §6.3, Tables 2 and 4] The flagship MFU results, including the 3.37x MFU improvement and the optimal TP sizes in Table 2, rest entirely on an unreleased in-house simulator. The simulation sets the expert-imbalance coefficient to 20% (§6.3) without production data showing this value is representative, even though Table 4 shows MFU is sensitive to this coefficient. The TP efficiency degradation model is referenced only to [53] without specifying the assumed curve, and the simulator's communication and overlap model is not described in enough detail to reproduce the results. I ask the authors to release the simulator (or a detailed validated model), provide sensitivity analysis over the imbalance coefficient and TP-efficiency assumptions, and validate the simulated MFU against at least one real training configuration.
  3. [§6.5, Table 8] The central cost claim that InfiniteHBD costs 30.86% of NVL-72 depends on an assumed OCSTrx unit cost of $600 and a 60% wholesale discount on public component prices, with no sensitivity analysis. The OCSTrx is a not-yet-mass-produced component, so its cost is a critical free parameter, and the NVL-36/72 cost figures are assembled from semi-analysis posts and internal estimates. Since interconnect cost is a headline contribution, please provide a breakeven analysis for the OCSTrx unit cost and discount rate, and report how the cost ratios in Table 6 change as these parameters vary.
  4. [§5.2, §6.2, Appendix A] The cluster-level fault-resilience and full-bandwidth claims are not backed by hardware. The 32-GPU experiment in §5.2 uses fixed optical modules with manual reconfiguration, not dynamic OCSTrx switching; the near-zero GPU waste results in §6.2 come from a simulator that "linearly maps the fault trace onto different network architectures." The fault trace is a strength, but the normalization from 8-GPU to 4-GPU nodes in Appendix A assumes i.i.d. GPU faults, which may not hold for correlated failures such as rack-level or power-domain faults. Please either demonstrate dynamic failover on a prototype or clearly mark these results as simulation-only, and justify the i.i.d. normalization against the trace data.
minor comments (4)
  1. [§5.1, Figure 12] The statement that OCSTrx "introduces zero additional bit error rate in most cases" is imprecise because Figure 12 shows occasional errors at very low OMA at 50°C and 75°C; please state the exact conditions under which BER is zero versus nonzero.
  2. [Figures 13–16] The legends list "InfiniteHBD(K=2), InfiniteHBD(K=3) & Big-Switch" as a single entry; if the curves overlap, please state this explicitly in the captions, and if they do not, separate the entries.
  3. [Appendix C] The waste-ratio upper bound in Equation (1) assumes independent breakpoints and a line topology; the text should state these assumptions explicitly and note that correlated fault events could invalidate the independence assumption.
  4. [§2.3] Table 2 is presented in Section 2.3 but depends on the simulator described only in §6.3; please add a forward pointer so readers know the methodology is deferred.

Circularity Check

0 steps flagged · score 0.0 of 10

No by-construction circularity found: the headline waste, cost, and MFU numbers are computed from a real fault trace, an analytic bound, and component pricing, not fitted to the paper's own conclusions.

full rationale

The paper's headline quantities are not circular. The 0.53% GPU waste ratio for TP-32 is obtained by replaying a real 348-day fault trace from the authors' 3K-GPU cluster through a simulator and is cross-checked by the analytic upper bound in Appendix C (Equation 1), a derivation that does not assume the target result. The 30.86% cost figure is an arithmetic normalization of component prices listed in Appendix F (Table 8); the OCSTrx unit price is vendor-supplied by the authors' affiliated company and the fault trace is the authors' own, but neither is a fitted parameter renamed as a prediction. The MFU comparisons are computed by an in-house simulator rather than measured end-to-end, which makes the evaluation self-referential: the same simulator used in Section 2.3 to argue that TP Ring-AllReduce is sufficient is used in Section 6.3 to validate the TP-only architecture. That is a dependency and a generality risk, but not a reduction by construction, because the simulator's outputs are not fitted to produce InfiniteHBD's results and the dense-model MFU gain over a TP-8 DGX baseline is an analytic consequence of removing the TP-8 cap. The paper itself concedes in Appendix G that large MoE models such as DeepSeek-V3 rely on EP and that the proposed Binary Exchange AllToAll fix is 'not yet validated on real hardware,' imposing TP×EP≤2048 and additional GPU forwarding overhead; this undercuts the generality of the 'mainstream LLM training scenarios' premise but is a correctness, scope, and evidence concern, not circularity. No load-bearing self-citation or imported uniqueness theorem is invoked: references with overlapping authors (e.g., [36], [40]) support well-known claims about communication intensity and expert imbalance and are corroborated by independent citations. The quantitative predictions for fault resilience, cost, and cross-ToR traffic are therefore self-contained derivations from stated data and assumptions rather than by-construction restatements of the inputs.

Assumptions & free parameters 6 free parameters · 5 assumptions · 3 invented entities

The central claims rest on one real hardware entity (OCSTrx prototype), one unvalidated topology concept, and a set of cost and workload assumptions. The cost advantage is especially sensitive to the self-determined $600 OCSTrx price and the 60% wholesale discount. The TP-only workload assumption is the most consequential domain assumption because it defines the architecture's scope.

free parameters (6)
  • OCSTrx unit cost = $600 per 800Gbps module
    Table 8 sets OCSTrx cost at $600, from authors' internal data (Lightelligence). The headline '31% of NVL-72 cost' depends directly on this number; no public pricing or sensitivity analysis is provided.
  • Per-GPU HBD bandwidth = 6.4 Tbps (8 x 800Gbps)
    Assumed in simulations (§6.1) as the sum of 8 QSFP-DD OCSTrx; affects MFU and cost normalization.
  • Expert imbalance coefficient = 20%
    Set in §6.3 for GPT-MoE EP simulation; drives the conclusion that TP beats EP. The value is illustrative, not from a measured deployment.
  • NVL-36/72 redundant backup overhead = 11% (1/9 GPUs)
    Modeling choice in §6.2: counts NVIDIA's redundant NVLink backup GPUs as permanent waste, which inflates InfiniteHBD's relative advantage.
  • Fault repair duration = average from trace
    Fixed repair duration in §6.2 fault-waiting simulations; real repair times vary.
  • Wholesale discount for public component prices = 60%
    Applied to public component prices in §6.5; validated 'against internal data', not public.
assumptions (5)
  • domain assumption LLM training HBD traffic can be reduced to TP Ring-AllReduce with large messages; EP can be served by TP.
    Load-bearing design principle in §2.3 'Key findings'; justifies excluding any-to-any AllToAll from the architecture. Appendix G admits AllToAll support is not validated on real hardware.
  • domain assumption Fault non-locality: consecutive multi-node failures are exponentially unlikely, so a K-hop ring with K=3 can bypass all realistic failures.
    §4.2 'Fault non-locality' observation; Appendix C derives waste upper bound assuming independent node failures.
  • domain assumption GPU faults are i.i.d. across GPUs, so an 8-GPU node trace can be normalized to 4-GPU nodes by Bayes' rule.
    Appendix A converts the 348-day 8-GPU node trace to 4-GPU nodes; if failures are correlated within a node or across nodes, the derived waste ratios change.
  • domain assumption OCSTrx reconfiguration (60-80us) can be overlapped with computation so dynamic switching does not stall training.
    §5.1 measures transceiver reconfiguration latency; the end-to-end training benefit in §6 assumes overlap and no protocol-level reconnection delay.
  • ad hoc to paper Thermo-optic MZI OCS integrated in QSFP-DD adds zero new failure modes and can be mass-produced at the assumed cost.
    §4.1 and §6.5; hardware shown in prototype, but mass production yield and cost at $600 are internal estimates.
invented entities (3)
  • OCSTrx independent evidence
    purpose: Embedded OCS in each transceiver for reconfigurable point-to-multipoint links, enabling ring construction and fault bypass.
    A working prototype in QSFP-DD 800Gbps is described and measured (§5.1: BER, insertion loss, power), so it is a real entity rather than a hypothetical; scale-up claims remain assumed.
  • Reconfigurable K-hop ring topology
    purpose: Pruned full-mesh to linear cost with node-level fault isolation via backup links.
    Evaluated only in simulation and with fixed-module small-scale cluster; dynamic reconfiguration and fault bypass were not executed on the OCSTrx hardware in a cluster (§5.2).
  • Binary Exchange AllToAll rewire topology
    purpose: Theoretical support for AllToAll on InfiniteHBD by rewiring to powers-of-two distances.
    Proposed in Appendix G and explicitly stated as 'not yet been validated on real hardware'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers." pith.science (2026). https://pith.science/paper/WW7BNC5M

@misc{pith2026250203885,
  author       = {Pith},
  title        = {Pith review of: InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WW7BNC5M}},
  note         = {Machine review of arXiv:2502.03885}
}
read the original abstract

Scaling Large Language Model (LLM) training relies on multi-dimensional parallelism, where High-Bandwidth Domains (HBDs) are critical for communication-intensive parallelism like Tensor Parallelism. However, existing HBD architectures face fundamental limitations in scalability, cost, and fault resiliency: switch-centric HBDs (e.g., NVL-72) incur prohibitive scaling costs, while GPU-centric HBDs (e.g., TPUv3/Dojo) suffer from severe fault propagation. Switch-GPU hybrid HBDs (e.g., TPUv4) take a middle-ground approach, but the fault explosion radius remains large. We propose InfiniteHBD, a transceiver-centric HBD architecture that integrates connectivity and dynamic switching at the transceiver level by embedding Optical Circuit Switching (OCS) within each transceiver. It enables reconfigurable point-to-multipoint communication and scalable variable-size ring topologies. InfiniteHBD achieves datacenter-scale scalability without cost explosion, fault isolation at the node level, and full bandwidth utilization for healthy GPUs. Key innovations include a Silicon Photonic-based OCS transceiver (OCSTrx), a reconfigurable k-hop ring topology, and an HBD-DCN orchestration algorithm. The evaluation demonstrates that InfiniteHBD reduces cost to 31% of NVL-72, achieves a near-zero GPU waste ratio (over 10x lower than NVL-72 and TPUv4), maintains near-zero cross-ToR traffic under 7% node fault ratio, and improves Model FLOPs Utilization by 3.37x compared to NVIDIA DGX (8 GPUs/node).

Figures

Figures reproduced from arXiv: 2502.03885 by the authors.

Figure 1
Figure 1. Illustrative examples of HBD architectures. N represents Node, and S represents Switch. Red (with cross hatch) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. InfiniteHBD overview. Photonic Integrated Circuit Modulator Laser Linear Driver OCS Controller Linear TIA Tx Electric Signals OCS Control Signals Rx Electric Signals Fiber Connector Fiber Connector Internal Loopback Path External Path 1 Tx Optical Signal Rx Optical Signal External Path 2 OCS Input Output3 Output1 Output2 Photodetector (a) Components of OCS transceivers. OCS: MZI Switch Matrix Input U Output 1 U Outp… view at source ↗
Figure 3
Figure 3. Design of OCS Transceivers. The core component is OCS integrated in transceivers. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: OCSTrx connection within nodes. Each block con￾tains multiple OCSTrx as one bundle, e.g., 8×800𝐺𝑏𝑝𝑠 OCSTrx for a 6.4Tbps GPU. Inter-node Topology. We construct the inter-node topology by pruning the full-mesh design, based on two key observations: i) Traffic locality: …
Figure 5
Figure 5. Figure 5: 4-GPU node with OCSTrx. 4.3 HBD-DCN Orchestration Algorithm InfiniteHBD is designed to work with arbitrary DCNs, including Rail-Optimized [52, 65] and Fat-Tree [2]. This section co-optimizes communication performance for both HBD and DCN in Infinite￾HBD. Problem Statem…
Figure 7
Figure 7. Figure 7: Illustration of the deployment phase, showing only [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: Illustration of the node orchestration problem state [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: OCSTrx integrated in QSFP-DD 800Gbps transceiver. OCS Controller Board Linear Driver + Modulator Board TIA + Photodetector Board OCS Control Signal Optical Signal OCS Controller Chip Optical Chip (Modulator) Optical Chip (Photodetector) [PITH_FULL_IMAGE:figures/full_f…
Figure 9
Figure 9. Figure 9: , OCSTrx integrates an OCS Controller Chip and a Photonic Integrated Circuit (PIC) that includes an MZI switch matrix. The Controller Chip, measuring 4𝑚𝑚 × 4𝑚𝑚, is manufactured using a 28nm process, while the PIC, sized at 10.5𝑚𝑚 × 13𝑚𝑚, uses a 65nm CMOS process. OCSTr…
Figure 11
Figure 11. Figure 11: Insertion loss distribution of the core module in OCSTrx under different ambient temperatures. 0.5 1.0 OMA (mW) 0 1e-14 1e-12 1e-10 1e-8 1e-6 1e-4 1e-2 1e0 BER (a) -5°C. 0.5 1.0 OMA (mW) 0 1e-14 1e-12 1e-10 1e-8 1e-6 1e-4 1e-2 1e0 BER (b) 25°C. 0.5 1.0 OMA (mW) 0 1e-1…
Figure 12
Figure 12. Figure 12: Bit error rate of OCSTrx under varying OMA and ambient temperatures. architectures (§6.5). The simulations demonstrate that InfiniteHBD outperforms other architectures across all metrics. 6.1 Methodology and Setup An in-house simulator dedicated to LLM training is use…
Figure 13
Figure 13. Figure 13: CDF of GPU waste ratio over 4-GPU node based [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 15
Figure 15. Figure 15: Maximal job scale supported by 2,880 GPUs. [PITH_FULL_IMAGE:figures/full_fig_p010_15.png]
Figure 16
Figure 16. Figure 16: Job fault-waiting time over the 4-GPU node with [PITH_FULL_IMAGE:figures/full_fig_p010_16.png]
Figure 17
Figure 17. Figure 17: DCN traffic optimization analysis and aggregate normalized cost varies across different architectures under different [PITH_FULL_IMAGE:figures/full_fig_p012_17.png]
Figure 18
Figure 18. Figure 18: Fault node trace in the production AI DC. [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: Break point can cause node waste compared to [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 20
Figure 20. Figure 20: GPU waste ratio over production fault trace, 4-GPU node. [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]
Figure 21
Figure 21. Figure 21: CDF of GPU waste ratio over production fault trace, 4-GPU node. [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: GPU waste ratio with different GPU fault ratio, 4-GPU node. [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]
Figure 23
Figure 23. Figure 23: Job fault-waiting duration under different job scales, 4-GPU node. [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]
Figure 24
Figure 24. Figure 24: Illustration of Binary Exchange AllToAll algorithm under TP4+EP4 configuration and InfiniteHBD topology for AllToAll. This limits general applicability in some scenarios. The 8-GPU Node alleviates this issue, as it supports up to 8 OCSTrx bundles, enabling connections…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 62 canonical work pages

  1. [53]

    NVIDIA. 2023. Matrix Multiplication Background User’s Guide. (2023). https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix- multiplication/index.html

    work page 2023

  2. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Trans- former Models from Multi-Head Checkpoints. (2023). arXiv:cs.CL/2305.13245 https://arxiv.org/abs/2305.13245

    arXiv 2023

  3. [2]

    Mohammad Al-Fares, Alexander Loukissas, and Amin Vahdat. 2008. A scalable, commodity data center network architecture. InProceedings of the ACM SIG- COMM 2008 Conference on Data Communication (SIGCOMM ’08). Association for Computing Machinery, New York, NY, USA, 63–74. https://doi.org/10.1145/1402 958.1402967

    work page doi:10.1145/1402 2008

  4. [3]

    AMD. 2025. AMD Instinct™MI300 Series Microarchitecture. (2025). https: //rocm.docs.amd.com/en/latest/conceptual/gpu-arch/mi300.html

    2025

  5. [5]

    Hitesh Ballani, Paolo Costa, Raphael Behrendt, Daniel Cletheroe, Istvan Haller, Krzysztof Jozwik, Fotini Karinou, Sophie Lange, Kai Shi, Benn Thomsen, and Hugh Williams. 2020. Sirius: A Flat Datacenter Network with Nanosecond Optical Switching. InProceedings of the Annual Conference of the ACM Spe- cial Interest Group on Data Communication on the Applicat...

    work page arXiv 2020

  6. [6]

    Bruck, Ching-Tien Ho, S

    J. Bruck, Ching-Tien Ho, S. Kipnis, E. Upfal, and D. Weathersby. 1997. Efficient algorithms for all-to-all communications in multiport message-passing systems. IEEE Transactions on Parallel and Distributed Systems8, 11 (1997), 1143–1156. https://doi.org/10.1109/71.642949

    work page doi:10.1109/71.642949 1997

  7. [7]

    Calient.AI. 2024. (2024). https://www.calient.net/

    2024

  8. [8]

    Kai Chen, Ankit Singla, Atul Singh, Kishore Ramachandran, Lei Xu, Yueping Zhang, Xitao Wen, and Yan Chen. 2012. OSA: An Optical Switching Architec- ture for Data Center Networks with Unprecedented Flexibility. In9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12). USENIX Association

    2012

Show all 91 references
  1. [9]

    Li Chen, Kai Chen, Zhonghua Zhu, Minlan Yu, George Porter, Chunming Qiao, and Shan Zhong. 2017. Enabling Wide-Spread Communications on Optical Fabric with MegaSwitch. In14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, ...

    2017

  2. [10]

    Intel Corporation. 2023. Intel®Gaudi®3 AI Accelerator White Paper. (2023). https://www.intel.com/content/www/us/en/content-details/817486/intel- gaudi-3-ai-accelerator-white-paper.html

    2023

  3. [11]

    2018.Accelerated Computing and the Democratization of Supercomputing: Technical Overview

    NVIDIA Corporation. 2018.Accelerated Computing and the Democratization of Supercomputing: Technical Overview. Technical Report. NVIDIA Corporation

    2018

  4. [12]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2025 arXiv

  5. [13]

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al

  6. [14]

    Nathan Farrington, George Porter, Sivasankar Radhakrishnan, Hamid Hajabdolali Bazzaz, Vikram Subramanya, Yeshaiahu Fainman, George Papen, and Amin Vahdat. 2010. Helios: a hybrid electrical/optical switch architecture for modular data centers. InProceedings of the ACM SIGCOMM 2...

    2010

  7. [15]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39

    2022

  8. [16]

    fibermall. 2024. OSFP-400G-FR4 400G FR4 OSFP PAM4 CWDM4 2km LC SMF FEC Optical Transceiver Module. (2024). https://www.fibermall.com/sale-459190- osfp-400g-fr4-cwdm4-2km.htm

    2024

  9. [17]

    FIBERMALL.COM. 2024. (2024). https://www.fibermall.com/

    2024

  10. [18]

    FS. 2024. (2024). https://www.fs.com/

    2024

  11. [19]

    FS. 2024. 1.5m (5ft) Generic Compatible 400G OSFP Flat Top Passive Direct Attach Copper Twinax Cable. (2024). https://www.fs.com/products/219579.html

    2024

  12. [20]

    FS. 2024. 1.5m (5ft) NVIDIA/Mellanox MCP1650-V01AE30 Compatible 200G QSFP56 Ethernet Passive Direct Attach Copper Twinax Cable. (2024). https: //www.fs.com/products/155618.html

    2024

  13. [21]

    FS. 2024. 1m (3ft) Generic Compatible 1.6T OSFP Close Top Passive Direct Attach Copper Twinax Cable. (2024). https://www.fs.com/products/244361.html

    2024

  14. [22]

    FS. 2024. 50m (164ft) Fiber Patch Cable, LC UPC to LC UPC, Duplex, 2 Fibers, Single Mode (OS2), Riser (OFNR), 2.0mm, Tight-Buffered, Yellow. (2024). https: //www.fs.com/products/177394.html

    2024

  15. [23]

    Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. Megablocks: Efficient sparse training with mixture-of-experts.Proceedings of Machine Learning and Systems5 (2023), 288–304

    2023

  16. [24]

    Adithya Gangidi, Rui Miao, Shengbao Zheng, Sai Jayesh Bondu, Guilherme Goes, Hany Morsy, Rohit Puri, Mohammad Riftadi, Ashmitha Jeevaraj Shetty, Jingyi Yang, Shuqiang Zhang, Mikel Jimenez Fernandez, Shashidhar Gandham, and Hongyi Zeng. 2024. RDMA over Ethernet for Distributed ...

    2024

  17. [25]

    Google. 2024. TPUv5p. (2024). https://cloud.google.com/tpu/docs/v5p

    2024

  18. [26]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2024 arXiv

  19. [27]

    Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo

    Juncheng Gu, Mosharaf Chowdhury, Kang G. Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. 2019. Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19). USENI...

    2019

  20. [28]

    Navid Hamedazimi, Zafar Qazi, Himanshu Gupta, Vyas Sekar, Samir R Das, Jon P Longtin, Himanshu Shah, and Ashish Tanwer. 2014. Firefly: A reconfigurable wireless data center fabric using free-space optics. InProceedings of the 2014 ACM conference on SIGCOMM. 319–330

    2014

  21. [29]

    Ralf Hauffe and Klaus Petermann. 2006. Thermo-Optic Switching. InOptical Switching, Tarek S. El-Bawab (Ed.). Springer US, Boston, MA, 111–139. https: //doi.org/10.1007/0-387-29159-8_4

    2006 doi

  22. [30]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  23. [31]

    Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo

  24. [32]

    Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi...

    2024 arXiv

  25. [33]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Clifford Young, Xiang Zhou, Zongwei Zhou, and David A Patterson. 2023. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learn...

    2023

  26. [34]

    Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson

    Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. 2020. A domain-specific supercomputer for training deep neural networks.Commun. ACM(Jun 2020), 67–78. https: //doi.org/10.1145/3360307

    2020 doi

  27. [35]

    Mehrdad Khani, Manya Ghobadi, Mohammad Alizadeh, Ziyi Zhu, Madeleine Glick, Keren Bergman, Amin Vahdat, Benjamin Klenk, and Eiman Ebrahimi. 2021. SiP-ML: high-bandwidth optical network interconnects for machine learning training. InProceedings of the 2021 ACM SIGCOMM 2021 Conf...

    2021

  28. [36]

    Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. 2023. Accelerating distributed{MoE} training and inference with lina. In2023 USENIX Annual Technical Conference (USENIX ATC 23). 945–959

    2023

  29. [37]

    Jiamin Li, Hong Xu, Yibo Zhu, Zherui Liu, Chuanxiong Guo, and Cong Wang

  30. [38]

    Xudong Liao, Yijun Sun, Han Tian, Xinchen Wan, Yilun Jin, Zilong Wang, Zheng- hang Ren, Xinyang Huang, Wenxue Li, Kin Fai Tse, Zhizhen Zhong, Guyue Liu, Ying Zhang, Xiaofeng Ye, Yiming Zhang, and Kai Chen. 2025. mFab- ric: An Efficient and Scalable Fabric for Mixture-of-Expert...

    2025 arXiv

  31. [39]

    Hong Liu, Ryohei Urata, Kevin Yasumura, Xiang Zhou, Roy Bannon, Jill Berger, Pedram Dashti, Norm Jouppi, Cedric Lam, Sheng Li, Erji Mao, Daniel Nelson, George Papen, Mukarram Tariq, and Amin Vahdat. 2023. Lightwave Fabrics: At- Scale Optical Circuit Switching for Datacenter an...

    2023

  32. [40]

    Juncai Liu, Jessie Hui Wang, and Yimin Jiang. 2023. Janus: A unified distributed training framework for sparse mixture-of-experts models. InProceedings of the ACM SIGCOMM 2023 Conference. 486–498

    2023

  33. [41]

    Kshiteej Mahajan, Arjun Balasubramanian, Arjun Singhvi, Shivaram Venkatara- man, Aditya Akella, Amar Phanishayee, and Shuchi Chawla. 2020. Themis: Fair and Efficient GPU Cluster Scheduling. In17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). USENI...

    2020

  34. [42]

    William M Mellette, Rajdeep Das, Yibo Guo, Rob McGuinness, Alex C Snoeren, and George Porter. 2020. Expanding across time to deliver bandwidth efficiency and low latency. In17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). 1–18

    2020

  35. [43]

    William M Mellette, Rob McGuinness, Arjun Roy, Alex Forencich, George Papen, Alex C Snoeren, and George Porter. 2017. Rotornet: A scalable, low-complexity, optical datacenter network. InProceedings of the Conference of the ACM Special 14 InfiniteHBD SIGCOMM ’25, September 8–11...

    2017

  36. [44]

    Introducing Llama 3.1: Our most capable models to date. 2024. (2024). https: //ai.meta.com/blog/meta-llama-3-1/

    2024

  37. [45]

    OSFP MSA. 2022. The Next Generation of Pluggable Optical Module Solutions from the OSFP MSA. (2022). https://osfpmsa.org/assets/pdf/OSFP1600_and_OS FP-XD.pdf

    2022

  38. [46]

    QSFP-DD MSA. 2024. QSFP-DD/QSFP-DD800/QSFP-DD1600 Hardware Specifi- cation. (2024). http://www.qsfp-dd.com/wp-content/uploads/2024/07/QSFP-DD- Hardware-Rev7.1.pdf

    2024

  39. [47]

    NADDOD. 2024. N9500-128QC, 128x400G QSFP112 Ethernet L3 4U Managed Switch, 51.2Tbps, Broadcom Tomahawk 5, Support RoCEv2, for AI/ML/Cloud Data Center/HPC. (2024). https://www.naddod.com/products/102323.html

    2024

  40. [48]

    NADDOD.COM. 2024. (2024). https://www.naddod.com/

    2024

  41. [49]

    Mark Nowell, Cisco Attila Aranyosi, Vu Le, Jeffery J Maki, Juniper Networks Scott Sommers, Tom Palkert, and Weiming Chen. 2018. QSFP-DD: Enabling 15 Watt Cooling Solutions. (2018)

    2018

  42. [50]

    NVIDIA. 2018. NVIDIA NVSwitch Technical Overview. (2018). https://images.n vidia.com/content/pdf/nvswitch-technical-overview.pdf

    2018

  43. [51]

    NVIDIA. 2020. NCCL AllToAll. (2020). https://docs.nvidia.com/deeplearning/n ccl/user-guide/docs/usage/p2p.html#all-to-all

    2020

  44. [52]

    NVIDIA. 2021. Doubling all2all Performance with NVIDIA Collective Communi- cation Library 2.12. (2021). https://developer.nvidia.com/blog/doubling-all2all- performance-with-nvidia-collective-communication-library-2-12/

    2021

  45. [54]

    NVIDIA. 2024. NVIDIA DGX SuperPOD. (2024). https://www.nvidia.com/en- us/data-center/dgx-superpod

    2024

  46. [55]

    NVIDIA. 2024. NVIDIA GB200 NVL72. (2024). https://www.nvidia.com/en- us/data-center/gb200-nvl72/

    2024

  47. [56]

    NVIDIA. 2024. NVIDIA H100 Tensor Core GPU. (2024). https://www.nvidia.c om/en-us/data-center/h100/

    2024

  48. [57]

    NVIDIA. 2024. NVLink and NVLink Switch. (2024). https://www.nvidia.com/en- us/data-center/nvlink

    2024

  49. [58]

    Open Accelerator Infrastructure (OAI). 2023. Universal Baseboard (UBB) Base Specification r2.0 v1.0. (2023). https://www.opencompute.org/documents/oai- ubb-base-specification-r2-0-v1-0-20230919-pdf

    2023

  50. [59]

    Deepika Pandey. 2012. Comparative Analysis of Different Topologies Based On Network-on-Chip Architectures.International Journal of Engineering Research and Development1, 11 (2012), 71–76

    2012

  51. [60]

    Pitch Patarasuk and Xin Yuan. 2009. Bandwidth optimal all-reduce algorithms for clusters of workstations.J. Parallel and Distrib. Comput.69, 2 (2009), 117–124

    2009

  52. [61]

    Yanghua Peng, Yibo Zhu, Yangrui Chen, Yixin Bao, Bairen Yi, Chang Lan, Chuan Wu, and Chuanxiong Guo. 2019. A Generic Communication Scheduler for Dis- tributed DNN Training Acceleration. InProceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP ’19). Associa...

    2019

  53. [62]

    NVIDIA DGX Platform. 2024. (2024). https://www.nvidia.com/en-us/data- center/dgx-platform/

    2024

  54. [63]

    Polatis. 2025. Homepage. (2025). www.polatis.com

    2025

  55. [64]

    George Porter, Richard Strong, Nathan Farrington, Alex Forencich, Pang Chen- Sun, Tajana Rosing, Yeshaiahu Fainman, George Papen, and Amin Vahdat. 2013. Integrating microsecond circuit switching into the data center.ACM SIGCOMM Computer Communication Review43, 4 (2013), 447–458

    2013

  56. [65]

    Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, Chao Wang, Peng Wang, Pengcheng Zhang, Xianlong Zeng, Eddie Ruan, Zhiping Yao, Ennan Zhai, and Dennis Cai

  57. [66]

    Ganger, and Eric P

    Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing. 2021. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. In15th USENIX Symposium on Operating Systems Design and Implem...

    2021

  58. [67]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. (2020). arXiv:cs.LG/1910.02054 https://arxiv.org/abs/1910.02054

    2020 arXiv

  59. [68]

    SemiAnalysis. 2023. Google OCS Apollo: The >$3 Billion Game-Changer in Datacenter Networking. (2023). https://semianalysis.com/2023/03/17/google- apollo-the-3-billion-game/

    2023

  60. [69]

    SemiAnalysis. 2024. GB200 Hardware Architecture – Component Supply Chain & BOM. (2024). https://semianalysis.com/2024/07/17/gb200-hardware-architecture- and-component/

    2024

  61. [70]

    SemiAnalysis. 2024. NVIDIA’s Blackwell Reworked – Shipment Delays & GB200A Reworked Platforms. (2024). https://semianalysis.com/2024/08/04/nvidias- blackwell-reworked-shipment/

    2024

  62. [71]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. (2020). arXiv:cs.CL/1909.08053 https://arxiv.org/abs/1909.08053

    2020 arXiv

  63. [72]

    Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michae...

    2022 arXiv

  64. [73]

    Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, Jiahao Bu, Zhongzhi Chen, Xuemeng Huang, Fengzong Lian, Saiyong Yang, Jianfeng Yan, Yuyuan Zeng, Xiaoqin Ren, Chao Yu, Lulu Wu, Yue Mao, Jun Xia, Tao Yang...

    2024 arXiv

  65. [74]

    Telescent. 2025. Homepage. (2025). www.telescent.com/products

    2025

  66. [75]

    Tesla. 2024. Tesla Dojo. (2024). https://en.wikipedia.org/wiki/Tesla_Dojo

    2024

  67. [76]

    Rajeev Thakur and William D Gropp. 2003. Improving the performance of collective operations in MPICH. InEuropean Parallel Virtual Machine/Message Passing Interface Users’ Group Meeting. Springer, 257–267

    2003

  68. [77]

    AWS Trainium. 2024. (2024). https://aws.amazon.com/ai/machine-learning/train ium/

    2024

  69. [78]

    Ryohei Urata, Hong Liu, Kevin Yasumura, Erji Mao, Jill Berger, Xiang Zhou, Cedric Lam, Roy Bannon, Darren Hutchinson, Daniel Nelson, Leon Poutievski, Arjun Singh, Joon Ong, and Amin Vahdat. 2022. Mission Apollo: Landing Optical Circuit Switching at Datacenter Scale. (2022). ar...

    2022 arXiv

  70. [79]

    Guohui Wang, David G Andersen, Michael Kaminsky, Konstantina Papagiannaki, TS Eugene Ng, Michael Kozuch, and Michael Ryan. 2010. c-Through: Part-time optics in data centers. InProceedings of the ACM SIGCOMM 2010 Conference. 327–338

    2010

  71. [80]

    Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani. 2024. Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters. (2024). arXiv:cs.NI/2307.12169 https://arxiv.org/ abs/2307.12169

    2024 arXiv

  72. [81]

    Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Manya Ghobadi, Zhihao Jia, Dheevatsa Mudigere, Ying Zhang, and Anthony Kewitsch. 2023. TopoOpt: Co- optimizing Network Topology and Parallelization Strategy for Distributed Train- ing Jobs. In20th USENIX Symposium on Networked Syste...

    2023

  73. [82]

    Qizhen Weng, Wencong Xiao, Yinghao Yu, Wei Wang, Cheng Wang, Jian He, Yong Li, Liping Zhang, Wei Lin, and Yu Ding. 2022. MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters. In19th USENIX Symposium on Networked Systems Design and Imple...

    2022

  74. [83]

    Wikipedia. 2024. Mach–Zehnder interferometer. (2024). https://en.wikipedia.o rg/wiki/Mach-Zehnder_interferometer

    2024

  75. [84]

    Yiting Xia, Mike Schlansker, TS Eugene Ng, and Jean Tourrilhes. 2015. Enabling Topological Flexibility for Data Centers Using{OmniSwitch}. In7th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 15)

    2015

  76. [85]

    Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou. 2018. Gandiva: Introspective Cluster Sched- uling for Deep Learning. In13th USENIX Symposium on Oper...

    2018

  77. [86]

    Law, and A

    Tze-Wei Yeow, K.L.E. Law, and A. Goldenberg. 2001. MEMS optical switches. IEEE Communications Magazine39, 11 (2001), 158–163

    2001

  78. [87]

    Zili Zhang, Yinmin Zhong, Ranchen Ming, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, and Xin Jin. 2024. DistTrain: Addressing model and data heterogeneity with disaggregated training for multimodal large language models.arXiv preprint 15 SIGCOMM ’25, September 8–11, 2025, Coi...

    2024

  79. [88]

    Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, et al . 2024. RLHFuse: Efficient rlhf training for large language models with inter-and intra-stage fusion. arXiv preprint arXiv:2409.13221(2024). 16 Infin...

    2024 arXiv

  80. [2020]

    In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)

    A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters. In14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20). USENIX Association, 463–479. https://www.usenix.org/conference/osdi20/presentation/jiang

  81. [2022]

    In International Conference on Machine Learning

    Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning. PMLR, 5547–5569

  82. [2023]

    InProceedings of the Eighteenth European Conference on Computer Systems (EuroSys ’23)

    Lyra: Elastic Scheduling for Deep Learning Clusters. InProceedings of the Eighteenth European Conference on Computer Systems (EuroSys ’23). Association for Computing Machinery, New York, NY, USA, 835–850. https://doi.org/10.114 5/3552326.3587445

  83. [2024]

    InProceedings of the ACM SIGCOMM 2024 Conference (ACM SIGCOMM ’24)

    Alibaba HPN: A Data Center Network for Large Language Model Training. InProceedings of the ACM SIGCOMM 2024 Conference (ACM SIGCOMM ’24). 691–706

    2024

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.