Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure

Myoungsoo Jung

REVIEW 4 major objections 4 minor 296 references

Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the dominant cost in modern AI infrastructure is the communication tax, and that a cache-coherent CXL memory fabric plus a hybrid CXL-over-XLink interconnect can replace most RDMA-scale-out traffic, cutting RAG time…

desk verdict Real CXL pooling measurements are buried under an unsupported CXL-over-XLink pitch; the paper needs major revision to separate the two. read the letter →

arxiv 2507.07223 v2 pith:YSRSVFQT submitted 2025-07-09 cs.DC cs.AR

Myoungsoo Jung This is my paper

classification cs.DCcs.AR

keywords CXLNVLinkUALinkXLinkmemorydisaggregationcachecoherenceAIinfrastructuredatacenterarchitecture

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern AI workloads spend 35 to 70 percent of their training time moving data, not computing. The paper claims that this communication tax is the real scalability bottleneck, and that a disaggregated, cache-coherent memory architecture built on Compute Express Link (CXL) can remove most of it. The key move is to make remote memory look like local memory, so accelerators no longer pay RDMA's protocol and software overhead. On a prototype built from lightweight open-source CPU and GPU cores, the paper reports roughly 8x less end-to-end RAG execution time and 3.3x higher DLRM throughput compared with RDMA baselines, and it extends the idea with a hybrid CXL-over-XLink fabric for accelerator clusters. If these numbers carry to production hardware, AI data centers could scale memory and accelerators independently instead of buying tightly coupled GPU nodes.

What carries the argument

The load-bearing object is the CXL-over-XLink supercluster: multiple accelerator racks whose internal NVLink or UALink clusters are joined by a hierarchical CXL 3.0 switch fabric that exposes accelerator-local and pooled memory as one coherent address space. The design also includes tiered memory—XLink-attached accelerator-local HBM as tier 1 and capacity-oriented CXL memory trays as tier 2—so that latency-sensitive data stays close to the accelerator while large embedding tables and caches live in composable pools. This combination converts collective operations such as All-Reduce and scatter/gather from explicit RDMA messages into implicit coherent memory accesses, which is what removes the communication tax.

What would settle it

A concrete test: on a production GPU cluster with CXL 3.0 memory pooling, run the same RAG and DLRM workloads against an RDMA baseline and measure end-to-end execution time; if pooled-memory access does not reduce execution time, or per-access latency stays above the microsecond range, the claimed scalability gains do not hold.

Watch

Extended reading notes

Core claim

The central claim is that the communication tax—latency and bandwidth spent on inter-GPU and inter-node data movement—is the dominant cost in modern AI infrastructure, and that CXL 3.0 memory pooling plus a hybrid CXL-over-XLink interconnect can eliminate most of it while preserving cache coherence. The paper reports prototype results: CXL pooled memory shortens a complete RAG workflow by about 8.05x, Graph-RAG by up to 14.35x, and improves DLRM inference throughput by about 3.32x, with MPI communication overhead down by up to 5.02x. The mechanism is direct hardware-mediated load/store access to shared memory pools, replacing RDMA-based network transfers, with XLink (NVLink or UALink) kept for latency-critical accelerator-to-accelerator traffic.

Load-bearing premise

The load-bearing premise is that speedups measured on lightweight open-source CPU and GPU cores, modified with custom CXL controllers, will carry over to production-grade accelerators, because no commercial CXL 3.0 GPUs are available yet.

Editorial extensions

If this is right

Rack- and row-level accelerator clusters can be joined into a single scale-up domain, so inter-rack traffic no longer needs Ethernet or InfiniBand round trips for every collective operation.
Memory capacity per node can scale beyond local HBM limits, since embedding tables, KV caches, and RAG vector stores can sit in pooled CXL memory reachable by load/store.
NVIDIA and non-NVIDIA accelerators can coexist in one system: the CXL fabric bridges NVLink and UALink clusters despite their incompatible physical layers.
The same coherent-memory pattern applies to MPI scientific codes, cutting synchronization overhead in particle-in-cell and CFD simulations.
Resource disaggregation lets data centers right-size CPU, GPU, and memory trays per workload instead of provisioning fixed CPU-GPU nodes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the prototype speedups come from lightweight open-source CPU and GPU cores, not shipping GPUs, so the first production CXL 3.0 accelerators could show smaller gains; the architectural direction, however, would still be testable.
Beyond the paper: one could validate the CXL-over-XLink idea incrementally by placing a CXL switch fabric between two existing NVLink domains and measuring All-Reduce time against RDMA, without waiting for native CXL GPUs.
Beyond the paper: if pooled CXL memory reaches tens of terabytes per rack, RAG vector databases could be served directly from coherent memory, potentially removing the separate retrieval cluster from the inference path.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is really two documents. One is a tutorial on Transformers, data-center hierarchies, and CXL evolution, which is fine but mostly standard material. The other is a prototype evaluation of a CXL 3.0 composable memory system against an RDMA baseline for RAG, Graph-RAG, DLRM, and MPI workloads. That second part is genuinely new: the authors built real hardware modifications on open-source RISC-V and Vortex cores and report substantial speedups. Credit where due: getting a CXL 3.0 stack working, even on lightweight cores, is non-trivial, and the measured trends (e.g., 8–14x for retrieval workloads, 3.3x for DLRM) are noteworthy if they hold up.

The problem is the packaging. The title and abstract lead with CXL-over-XLink, a hybrid interconnect idea that is never evaluated. Section 6.2 is a design sketch with no XLink hardware, no bridge measurement, and no conversion-overhead quantification. The paper itself admits the protocol conversion adds latency and that the proposed HBM-buffered bridging is speculative. So the central claim is unsupported by the evidence in the manuscript. That is a load-bearing flaw, not a minor caveat.

The prototype measurements also have soft spots of their own. Section 5.2 lacks error bars, repetitions, dataset details, and code. The use of lightweight open-source cores is an asserted proxy for production GPUs, not a demonstrated one. The author is also the CEO of Panmnesia, which sells the CXL controller IP used in the prototype; the self-referential evidence is a legitimate reason to ask for independent validation, though not by itself grounds for rejection.

All that said, the paper is not incoherent. The tutorial sections are accurate, the CXL pooling measurements appear internally consistent, and the design discussion is honest about its own limitations. The citation pattern is heavy on the author's own company work, but the evaluated prototype is real, which distinguishes this from pure marketing.

My recommendation: as a research contribution on CXL-over-XLink, this is not ready for publication. But the underlying CXL pooling evaluation deserves referee attention if the authors refocus the paper accordingly. A serious referee could push for a revised manuscript that either (a) drops or clearly labels the hybrid architecture as speculation and presents the measurements with proper methodology, or (b) adds even a minimal XLink bridge experiment. I would not let this pass as is, but I would not desk-reject a paper with this much working silicon-level effort. Send it to peer review with the expectation of major revision.

Referee Report

4 major / 4 minor

Summary. The paper argues that modern AI infrastructure is bottlenecked by memory and communication rather than compute, and proposes a modular, CXL-based composable data center architecture. It introduces CXL-over-XLink, a hybrid design in which UALink/NVLink clusters are connected through a CXL fabric, plus a hierarchical memory tiering scheme. The experimental section reports large speedups for CXL pooled memory over RDMA/InfiniBand baselines for RAG, Graph-RAG, DLRM, and MPI workloads, using a prototype built from Vortex GPUs and RISC-V CPUs with Panmnesia CXL controller IP.

Significance. If fully supported, the CXL pooled-memory measurements would be a useful engineering data point, and a working CXL-over-XLink fabric would be architecturally significant. The paper deserves credit for a concrete end-to-end prototype, for covering four workload classes, and for explicitly discussing implementation obstacles such as protocol conversion. However, the XLink half of the architecture is never measured, and the prototype's representativeness for production accelerators is asserted rather than demonstrated; as submitted, the paper's headline claim is not established.

major comments (4)

[Abstract; §§5.2, 6.2] The central advertised contribution, CXL-over-XLink, has no experimental support. Every speedup reported in Section 5.2 (Figures 33–37) compares composable CXL memory pooling against an RDMA/InfiniBand baseline; no experiment includes UALink, NVLink, NVLink Fusion, a CXL-to-XLink bridge, or an XLink cluster. Section 6.2 is a design sketch that itself acknowledges protocol-conversion latency and labels the HBM-buffered bridge and data-placement strategies as speculative. The abstract's claim that evaluations demonstrate the proposed architecture cannot be read as support for the hybrid interconnect; the authors must either supply an implementation, simulation, or quantitative analysis of the XLink-CXL boundary, or reframe the paper so that the hybrid is explicitly a proposal rather than a demonstrated result.
[§5.2, Experimental infrastructure] The load-bearing representativeness assumption for the prototype is not established. The evaluation uses open-source Vortex GPU and RISC-V CPU cores modified with Panmnesia CXL controllers because commercial CXL 3.0 accelerators do not exist; the paper asserts that these IPs can be integrated with third-party hardware but gives no data comparing the prototype's memory hierarchy, cache behavior, or interconnect overhead to production GPU-class systems. Consequently, the reported values (e.g., 14.35x RAG, 3.32x DLRM) are single measurements with no repetitions, error bars, or detailed baseline configuration, and cannot be generalized to data-center-scale AI infrastructure.
[§5.2, Figures 31–37] The quantitative summary is not presented with a consistent methodology. Figure 31 reports a 14.35x RAG execution-time reduction, while Figure 33d reports component-level speedups of 14x and 2.78x, and the paper does not state how the end-to-end ratio is computed or whether data-movement time is included. The same ambiguity affects 'up to 21.1x decreases in data movement' and the 3.32x DLRM throughput figure. The authors should define the measurement boundary, the conversion used between latency and throughput ratios, and the exact workload parameters for each experiment.
[§6.1, Table 3] The taxonomy on which the hybrid design rests is inconsistent. Section 6 states that XLink technologies do not support protocol-level cache coherence or memory pooling, but the same section describes NVLink Fusion as a coherent C2C interface for external processors, which contradicts that dichotomy. Because the CXL-over-XLink proposal assigns coherence to CXL and non-coherence to XLink, this inconsistency affects the architectural justification and should be resolved.

minor comments (4)

[§2] Sections 2.1–2.3 are a lengthy textbook-style tutorial on RNNs, attention, and Transformers; for a journal audience this material should be condensed to a few paragraphs with pointers to references.
[§3.3, Table 2] Several quantitative claims in the text and Table 2 (e.g., '35%–70% of training time', '35–40% GPU utilization', '<1 µs' RDMA latency, '100–250 ns' CXL latency) should be tied to specific sources or explicitly labeled as estimates.
[§5.2, Figures 33–37] The demo video links appear as placeholder '[Link]' strings and are not usable in the manuscript.
[Full text] The submitted text contains numerous formatting artifacts from the arXiv conversion, including missing spaces between words (e.g., the opening of Section 2.1); these should be cleaned before any revised submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: measured speedups are empirical CXL-vs-RDMA comparisons, and the untested CXL-over-XLink proposal is a design sketch rather than a prediction forced by the measurements.

full rationale

The paper contains no formal derivation chain in which a fitted parameter or definition is renamed as a prediction. Section 5.2 reports measured speedups of a composable CXL prototype against an RDMA baseline, e.g., 'the composable CXL architecture reduced total workflow execution time by approximately 8.05x relative to the conventional RDMA-based baseline' and 'approximately 3.32x compared to the RDMA-based system.' These are empirical measurements of a constructed system, not quantities forced by construction or by the paper's definitions. The CXL-over-XLink supercluster of Section 6.2 is presented as a proposed architecture and is explicitly not evaluated; the text states that 'protocol conversion and data transitions between XLink-based intra-cluster domains and CXL-based inter-cluster domains introduce additional latency' and then describes HBM-buffered bridging and data-placement mitigations as optimizations to pursue. That is an evidentiary gap or an overclaim relative to the abstract, but it is not a circular reduction: the XLink half is never asserted to be validated by the CXL-only measurements. The prototype uses the author's own Panmnesia CXL controller IP with open-source Vortex GPU and RISC-V CPU cores, which raises external-validity and reproducibility concerns, but the reported results are not derived from the claims and the central argument does not rest on a uniqueness theorem or on a self-citation chain. Under the hard rules, lack of external validation without an equation-level equivalence or a fitted-input-renamed-as-prediction does not constitute circular reasoning. Therefore the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The report contributes a synthesis of existing CXL and accelerator-link technologies plus a set of prototype measurements; it rests on several unverified domain assumptions (CXL 3.0 device counts, XLink topology constraints, RDMA overhead) and two ad hoc assumptions specific to this paper (bridge mitigation, prototype representativeness). No free parameters are fitted to data.

assumptions (5)

domain assumption CXL 3.0 can connect up to 4,096 memory devices and 256 accelerators per root port via multi-level switch cascading and PBR routing.
Section 4.2 and Table 1 take these CXL 3.0 capabilities as given from the specification; the proposed architectures rely on this scale for their feasibility.
domain assumption UALink and NVLink both use single-hop Clos topologies and cannot provide protocol-level cache coherence or memory pooling.
Section 6.1 contrasts XLink with CXL based on this characterization; the motivation for CXL-over-XLink depends on XLink's limitations being accurate and persistent.
domain assumption RDMA-based networking introduces tens to hundreds of times more latency than CXL due to software overhead such as kernel and user mode switches and memory copies.
Section 4.2 uses this comparison to justify replacing RDMA with CXL; the empirical baseline comparisons in Section 5.2 rely on this overhead being as large as stated.
ad hoc to paper Protocol conversion overhead between XLink and CXL domains can be largely mitigated by specialized bridge SoCs with HBM caching.
Section 6.2 proposes this mitigation without measurements or simulation; the viability of CXL-over-XLink depends on this unverified assumption.
ad hoc to paper The modified open-source Vortex GPU and RISC-V CPU prototype is representative of production accelerators for evaluating AI infrastructure performance.
Section 5.2 builds the entire empirical case on this lightweight prototype because no commercial CXL 3.0 GPUs exist; representativeness is asserted, not demonstrated.

invented entities (1)

CXL-over-XLink hybrid fabric
purpose: Unify accelerator-centric intra-cluster links (UALink and NVLink) with coherent CXL inter-cluster memory sharing to create a scalable supercluster.
This architecture is proposed in Section 6.2 but is never built or measured; no independent falsifiable evidence is provided, so it is a postulated design rather than an established result.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure." pith.science (2026). https://pith.science/paper/YSRSVFQT

@misc{pith2026250707223,
  author       = {Pith},
  title        = {Pith review of: Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YSRSVFQT}},
  note         = {Machine review of arXiv:2507.07223}
}

read the original abstract

Modern AI workloads such as large language models (LLMs) and retrieval-augmented generation (RAG) impose severe demands on memory, communication bandwidth, and resource flexibility. Traditional GPU-centric architectures struggle to scale due to growing inter-GPU communication overheads. This report introduces key AI concepts and explains how Transformers revolutionized data representation in LLMs. We analyze large-scale AI hardware and data center designs, identifying scalability bottlenecks in hierarchical systems. To address these, we propose a modular data center architecture based on Compute Express Link (CXL) that enables disaggregated scaling of memory, compute, and accelerators. We further explore accelerator-optimized interconnects-collectively termed XLink (e.g., UALink, NVLink, NVLink Fusion)-and introduce a hybrid CXL-over-XLink design to reduce long-distance data transfers while preserving memory coherence. We also propose a hierarchical memory model that combines local and pooled memory, and evaluate lightweight CXL implementations, HBM, and silicon photonics for efficient scaling. Our evaluations demonstrate improved scalability, throughput, and flexibility in AI infrastructure.

Figures

Figures reproduced from arXiv: 2507.07223 by the authors.

**Figure 2.** Minimizing the loss function. advances in “parallel computing” and “hardware acceleration” to overcome scalability limitations of earlier approaches, enhancing AI’s applicability to real-world scenarios. Lastly, we outline how Transformers evolved into LLMs and emphasize their implications for hardware infrastructure and scalability, both critical factors in their widespread practical adoption. 2.1. Understanding T… view at source ↗

**Figure 3.** Sequence-to-sequence (Seq2Seq) framework. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** RNN-based Seq2Seq architecture and limitation. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Figures from the paper (40 more)

**Figure 5.** Figure 5: Attention-based RNNs and Transformer architecture. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Transformer layer operations: self-attention and FFNs. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Mixture of experts (MoE) architecture. key vectors (K) denote information each token provides, and value vectors (V) encapsulate the actual content shared by tokens. The self-attention process involves comparing each query vector with all key vectors using scaled dot-…

**Figure 8.** Figure 8: LLM pre-training through parallel processing. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: Auto-regressive model workflow. ensures coherent and contextually consistent outputs, it limits parallel computation, reducing inference speed relative to parallelized generation methods. Despite these computational constraints, auto-regression remains widely used due …

**Figure 10.** Figure 10: Inference Optimization Techniques. which to generate its final response. Although this can reduce hallucinations and improves factual correctness, it introduces additional computational steps, including query embedding generation, similarity-based vector retrieval, an…

**Figure 11.** Figure 11: Transformer-based model partitioning and synchronization overhead. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

**Figure 12.** Figure 12: Pipeline parallelism (PP) and tensor parallelism (TP) for transformer architecture. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

**Figure 13.** Figure 13: Expert parallelism (EP) for mixture of experts (MoE) architecture. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

**Figure 14.** Figure 14: Communication and memory overhead in KV caching and RAG inference. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

**Figure 15.** Figure 15: Scale-up vs. scale-out and networked GPUs. [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]

**Figure 16.** Figure 16: Hierarchical data center architecture. plete physical resource disaggregation has not been fully realized because CPUs should act as host processors, managing the bus interfaces and memory controllers that interface directly with GPUs or accelerators. Instead, recent …

**Figure 17.** Figure 17: Node-level configuration of GB200. acceleration for networking functions, support high-bandwidth communication (typically 400 to 800 Gb/s per node), and enable RDMA capabilities for high-throughput data transfers. The integration of such NICs within a node ensures tha…

**Figure 18.** Figure 18: Rack-level configuration of NVL72. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]

**Figure 19.** Figure 19: Row and floor-level configuration. internal interconnect switches (e.g., NVSwitches [262, 263]) carefully distributed across the rack. Such internal interconnects provide high-bandwidth, low-latency communication, ideal for workloads requiring intensive intrarack dat…

**Figure 20.** Figure 20: Overall data center topology and building configuration. [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]

**Figure 21.** Figure 21: Hyperscaler’s site area and data center count. [PITH_FULL_IMAGE:figures/full_fig_p022_21.png]

**Figure 22.** Figure 22: Relative importance of performance metrics across different operational scenarios. [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]

**Figure 23.** Figure 23: Competing constraints of performance dimensions [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]

**Figure 24.** Figure 24: Relocation of memory controller in CXL. as soon as possible. However, conventional infrastructures incur significant latency from frequent intermediate data movements across nodes and additional software overhead introduced by traditional communication stacks (e.g., E…

**Figure 25.** Figure 25: Evolution of CXL: from direct connections to multi-level switching. [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]

**Figure 26.** Figure 26: Tray-based disaggregation with CXL cache-coherent sharing. [PITH_FULL_IMAGE:figures/full_fig_p030_26.png]

**Figure 27.** Figure 27: Composable rack architecture with CXL. frequent reuse of intermediate data structures, including transformer inference scenarios and KV caching tasks. In parallel, [PITH_FULL_IMAGE:figures/full_fig_p031_27.png]

**Figure 28.** Figure 28: Tray-based disaggregation with CXL cache-coherent sharing. [PITH_FULL_IMAGE:figures/full_fig_p033_28.png]

**Figure 29.** Figure 29: Topology comparison: Clos, 3D Torus, and Dragonfly. [PITH_FULL_IMAGE:figures/full_fig_p035_29.png]

**Figure 30.** Figure 30: Switch placement options. caches closer to accelerators can reduce latency and enhance throughput in LLM inference scenarios utilizing tensor parallelism. Effective accelerator resource management also impacts overall system performance. As discussed previously, conv…

**Figure 31.** Figure 31: Summary of performance gains: RAG, Graph-RAG, DLRM, and MPI. [PITH_FULL_IMAGE:figures/full_fig_p037_31.png]

**Figure 32.** Figure 32: A experimental end-to-end infrastructure compliant with the CXL 3.0 specification. [PITH_FULL_IMAGE:figures/full_fig_p037_32.png]

**Figure 33.** Figure 33: RAG use case: recipe recommendation (Demo Video: [Link]). CXL infrastructures, we evaluated a user-friendly RAG scenario integrated with a contemporary LLM. As depicted in [PITH_FULL_IMAGE:figures/full_fig_p038_33.png]

**Figure 34.** Figure 34: Graph-RAG use case: knowledge graph and query retrieval. [PITH_FULL_IMAGE:figures/full_fig_p038_34.png]

**Figure 35.** Figure 35: DLRM use case (Demo Video: [Link]). two primary operational phases: knowledge graph construction, followed by query-driven retrieval and inference. Initially, raw textual data sources were processed using standard graph embedding techniques (e.g., RDF embeddings [363–…

**Figure 36.** Figure 36: MPI use cases: plasma simulation (Demo Video: [Link]). into improved user experiences for large-scale commercial platforms, including personalized content delivery in e-commerce and streaming services. Consequently, adopting composable, independently scalable CXL infr…

**Figure 37.** Figure 37: MPI use cases: fluid simulation. nificantly streamline data management, improving scalability and operational efficiency in distributed computing environments. Practically, these enhancements substantially increase scalability and efficiency of large-scale scientific…

**Figure 38.** Figure 38: A high-level viewpoint of hybrid link architectures (CXL-over-XLink). [PITH_FULL_IMAGE:figures/full_fig_p041_38.png]

**Figure 39.** Figure 39: Accelerator-centric interconnects. employs Ethernet-based communication optimized primarily for large-sized data transfers, whereas NVLink utilizes NVIDIA’s proprietary electrical signaling, tailored for small-to-medium-sized data exchanges, such as tensor transfers a…

**Figure 40.** Figure 40: Accelerator-centric intra-cluster design. [PITH_FULL_IMAGE:figures/full_fig_p045_40.png]

**Figure 41.** Figure 41: Exemplary CXL-over-XLink supercluster configurations. [PITH_FULL_IMAGE:figures/full_fig_p046_41.png]

**Figure 42.** Figure 42: Cache coherence and data movement management in CXL-over-XLink. [PITH_FULL_IMAGE:figures/full_fig_p047_42.png]

**Figure 43.** Figure 43: Optimizing hardware and software for integrated XLink and CXL architectures. [PITH_FULL_IMAGE:figures/full_fig_p048_43.png]

**Figure 44.** Figure 44: Tiered memory hierarchy configured with CXL memory pool. [PITH_FULL_IMAGE:figures/full_fig_p049_44.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

296 extracted references · 14 canonical work pages

[1]

Avoiding another AI winter,

J. Hendler, “Avoiding another AI winter,”IEEE Intelligent Systems, vol. 23, no. 02, pp. 2–4, 2008

2008
[2]

A Brief History of AI: How to Prevent Another Winter,

A. Toosi, A. G. Bottino, B. Saboury, E. Siegel, and A. Rahmim, “A Brief History of AI: How to Prevent Another Winter,”PET Clinics, vol. 16, no. 4, pp. 449–469, 2025

2025
[3]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

2017
[4]

Annotated History of Modern AI and Deep Learning,

J. Schmidhuber, “Annotated History of Modern AI and Deep Learning,” 2022. [Online]. Available: https://arxiv.org/abs/2212.11279

arXiv 2022
[5]

ImageNet Classification with Deep Convolutional Neural Networks,

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25. Curran Associates, Inc., 2012. [Online]. Available: https: //proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3...

2012
[6]

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , J. Burstein, C. Doran, and T. Solorio, Ed...

2019
[7]

The Llama 3 Herd of Models,

A. G. et al., “The Llama 3 Herd of Models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

arXiv 2024
[8]

Generative Adversarial Nets,

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurip...

2014

Show all 296 references

[9]

Learning to learn by gradient descent by gradient descent,

M. Andrychowicz, M. Denil, S. Gómez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. de Freitas, “Learning to learn by gradient descent by gradient descent,” in Advances in Neural Information Processing Systems , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar...

2016
[10]

Gradient Descent: The Ultimate Optimizer,

K. Chandra, A. Xie, J. Ragan-Kelley, and E. MEIJER, “Gradient Descent: The Ultimate Optimizer,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 8214–8225...

2022
[11]

An overview of gradient descent optimization algorithms,

S. Ruder, “An overview of gradient descent optimization algorithms,”arXiv preprint arXiv:1609.04747, 2016

2016 arXiv
[12]

Optimization methods for large-scale machine learning,

L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for large-scale machine learning,”SIAM review, vol. 60, no. 2, pp. 223–311, 2018

2018
[13]

Practical recommendations for gradient-based training of deep architectures,

Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” inNeural net- works: Tricks of the trade: Second edition. Springer, 2012, pp. 437–478

2012
[14]

On the Number of Linear Regions of Deep Neural Networks,

G. Montúfar, R. Pascanu, K. Cho, and Y. Bengio, “On the Number of Linear Regions of Deep Neural Networks,” inAdvances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [...

2014
[15]

Representation learning: A review and new perspectives,

Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,”IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1798–1828, 2013

2013
[16]

Intrinsic dimension of data representations in deep neural networks,

A. Ansuini, A. Laio, J. H. Macke, and D. Zoccolan, “Intrinsic dimension of data representations in deep neural networks,”Advances in Neural Information Processing Systems, vol. 32, 2019

2019
[17]

High dimensional, tabular deep learning with an auxiliary knowledge graph,

C. Ruiz, H. Ren, K. Huang, and J. Leskovec, “High dimensional, tabular deep learning with an auxiliary knowledge graph,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates...

2023
[18]

General-purpose, long-context autoregressive modeling with Perceiver AR,

C. Hawthorne, A. Jaegle, C. Cangea, S. Borgeaud, C. Nash, M. Malinowski, S. Dieleman, O. Vinyals, M. Botvinick, I. Simon, H. Sheahan, N. Zeghidour, J.-B. Alayrac, J. Carreira, and J. Engel, “General-purpose, long-context autoregressive modeling with Perceiver AR,” in Proceedin...

2022
[19]

A survey on hardware accelerators: Taxonomy, trends, challenges, and perspectives,

B. Peccerillo, M. Mannino, A. Mondelli, and S. Bartolini, “A survey on hardware accelerators: Taxonomy, trends, challenges, and perspectives,”Journal of Systems Architecture, vol. 129, p. 102561, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S138...

2022
[20]

The transformational role of GPU computing and deep learning in drug discovery,

P. M. et al., “The transformational role of GPU computing and deep learning in drug discovery,”Nature Machine Intelligence, vol. 4, no. 3, pp. 211–221, 2022

2022
[21]

ZeRO: Memory optimizations Toward Training Trillion Parameter Models,

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “ZeRO: Memory optimizations Toward Training Trillion Parameter Models,” in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 2020, pp. 1–16

2020
[22]

DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters,

J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, “DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters,” inProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , ser. KDD ’20. New York...

2020
[23]

Gpu graph processing on cxl-based microsecond-latency external memory,

S. Sano, Y. Bando, K. Hiwada, H. Kajihara, T. Suzuki, Y. Nakanishi, D. Taki, A. Kaneko, and T. Sh- iozawa, “Gpu graph processing on cxl-based microsecond-latency external memory,” inProceedings of the SC’23 Workshops of the International Conference on High Performance Computin...

2023
[24]

Emogi: Efficient memory-access for out-of-memory graph-traversal in gpus,

S. W. Min, V. S. Mailthody, Z. Qureshi, J. Xiong, E. Ebrahimi, and W.-m. Hwu, “Emogi: Efficient memory-access for out-of-memory graph-traversal in gpus,”arXiv preprint arXiv:2006.06890, 2020

2006 arXiv
[26]

Efficient memory management for large language model serving with pagedattention,

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY, ...

2023
[27]

H2O: Heavy-hitter oracle for efficient generative inference of large language models,

Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. A. Wang, and B. Chen, “H2O: Heavy-hitter oracle for efficient generative inference of large language models,” in Advances in Neural Information Processing Systems , A. Oh, T. Nauman...

2023
[28]

Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,

M. Adnan, A. Arunkumar, G. Jain, P. J. Nair, I. Soloveychik, and P. Kamath, “Keyformer: Kv cache reduction through key tokens selection for efficient generative inference,” in Proceedings of Machine Learning and Systems , P. Gibbons, G. Pekhimenko, and C. D. Sa, Eds., vol. 6, ...

2024
[29]

TCCL: Co-optimizing Collective Communication and Traffic Routing for GPU-centric Clusters,

B. Li, X. Wang, J. Wang, Y. Liu, Y. Gong, H. Lu, W. Dang, W. Zhang, X. Huang, M. Chen, J. Chen, C. He, Y. Liu, X. Hu, C. Liu, X. Ji, Y. Xia, X. Li, Z. He, Y. Wang, and X. Zou, “TCCL: Co-optimizing Collective Communication and Traffic Routing for GPU-centric Clusters,” inProcee...

2024
[30]

Sˆ3: Increasing GPU Utilization during Generative Inference for Higher Throughput,

Y. Jin, C.-F. Wu, D. Brooks, and G.-Y. Wei, “Sˆ3: Increasing GPU Utilization during Generative Inference for Higher Throughput,” in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Ass...

2023
[31]

NVIDIA Grace GPU

NVIDIA, “NVIDIA Grace GPU.” [Online]. Available: https://www.nvidia.com/en-us/data-center/ grace-cpu-superchip/ 53 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure May 26, 2025

2025
[32]

Characterization and prediction of deep learning workloads in large-scale GPU datacenters,

Q. Hu, P. Sun, S. Yan, Y. Wen, and T. Zhang, “Characterization and prediction of deep learning workloads in large-scale GPU datacenters,” inProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC ’21. New York, NY, ...

2021
[33]

Towards Memory Disaggregation via NVLink C2C: Benchmarking CPU-Requested GPU Memory Access,

F. Werner, M. Weisgut, and T. Rabl, “Towards Memory Disaggregation via NVLink C2C: Benchmarking CPU-Requested GPU Memory Access,” inProceedings of the 4th Workshop on Heterogeneous Composable and Disaggregated Systems, ser. HCDS ’25. New York, NY, USA: Association for Computin...

2025
[34]

Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters,

H. Zhang, Z. Zheng, S. Xu, W. Dai, Q. Ho, X. Liang, Z. Hu, J. Wei, P. Xie, and E. P. Xing, “Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters,” in 2017 USENIX Annual Technical Conference (USENIX ATC 17). Santa Clara, CA: USENIX Ass...

2017
[35]

A memory perspective: The effects of finetuning llms with highbandwidth memory,

F. V. Zacarias, K. Palli, S. Vazhkudai, and E. Grevelink, “A memory perspective: The effects of finetuning llms with highbandwidth memory,” inMicron, 2024

2024
[36]

Whitepaper: NVDIA GH200 Grace Hopper Superchip Architecture,

NVIDIA, “Whitepaper: NVDIA GH200 Grace Hopper Superchip Architecture,” 2023

2023
[37]

Streambox-hbm: Stream analytics on high bandwidth hybrid memory,

H. Miao, M. Jeon, G. Pekhimenko, K. S. McKinley, and F. X. Lin, “Streambox-hbm: Stream analytics on high bandwidth hybrid memory,” inProceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, 2019, pp. 167–181

2019
[38]

Performance evaluation and optimization of HBM- enabled GPU for data-intensive applications,

M. Zhu, Y. Zhuo, C. Wang, W. Chen, and Y. Xie, “Performance evaluation and optimization of HBM- enabled GPU for data-intensive applications,”IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 26, no. 5, pp. 831–840, 2018

2018
[39]

NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Train- ing and Real-Time Inference

NVIDIA, “NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Train- ing and Real-Time Inference.” [Online]. Available: https://developer.nvidia.com/blog/ nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/
[40]

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022. [Online]. Available: http://jmlr.org/papers/v23/21-0998.html

2022
[41]

Retrieval augmented language model pre-training,

K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, “Retrieval augmented language model pre-training,” in Proceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul...

2020
[42]

Improving language models by retrieving from trillions of tokens,

S. e. a. Borgeaud, “Improving language models by retrieving from trillions of tokens,” inProceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato...

2022
[43]

Megatron-lm: Training multi-billion parameter language models using model parallelism,

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” 2020. [Online]. Available: https://arxiv.org/abs/1909.08053

2020 arXiv
[44]

How much knowledge can you pack into the parameters of a language model?

A. Roberts, C. Raffel, and N. Shazeer, “How much knowledge can you pack into the parameters of a language model?” 2020. [Online]. Available: https://arxiv.org/abs/2002.08910

2020 arXiv
[45]

Pipedream: generalized pipeline parallelism for dnn training,

D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” inProceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, N...

2019
[46]

Number of parameters in gpt-4 (latest data),

J. Howarth, “Number of parameters in gpt-4 (latest data),” 2025. [Online]. Available: https: //explodingtopics.com/blog/gpt-parameters 54 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure May 26, 2025

2025
[47]

NVIDIA GB200 NVL72

NVIDIA, “NVIDIA GB200 NVL72.” [Online]. Available: https://www.nvidia.com/en-us/data-center/ gb200-nvl72
[48]

NVIDIA GB300 NVL72

NVIDIA, “NVIDIA GB300 NVL72.” [Online]. Available: https://www.nvidia.com/en-us/data-center/ gb300-nvl72/
[49]

NVIDIA DGX GB300

NVIDIA, “NVIDIA DGX GB300.” [Online]. Available: https://www.nvidia.com/en-us/data-center/ dgx-gb300/
[50]

Understanding communication characteristics of distributed training,

W. Li, X. Liu, Y. Li, Y. Jin, H. Tian, Z. Zhong, G. Liu, Y. Zhang, and K. Chen, “Understanding communication characteristics of distributed training,” inProceedings of the 8th Asia-Pacific Workshop on Networking, ser. APNet ’24. New York, NY, USA: Association for Computing Mac...

2024
[51]

MAD-Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems,

S. Hsia, A. Golden, B. Acun, N. Ardalani, Z. DeVito, G.-Y. Wei, D. Brooks, and C.-J. Wu, “MAD-Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA...

2024
[52]

Flux: fast software-based communication overlap on gpus through kernel fusion,

L.-W. Chang, W. Bao, Q. Hou, C. Jiang, N. Zheng, Y. Zhong, X. Zhang, Z. Song, C. Yao, Z. Jiang et al., “Flux: fast software-based communication overlap on gpus through kernel fusion,”arXiv preprint arXiv:2406.06858, 2024

2024 arXiv
[53]

Collective communication library NCCL,

NVIDIA, “Collective communication library NCCL,” 2021. [Online]. Available: https://developer.nvidia. com/nccl

2021
[54]

Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping,

C. Jiang, Y. Tian, Z. Jia, S. Zheng, C. Wu, and Y. Wang, “Lancet: Accelerating mixture-of-experts training via whole graph computation-communication overlapping,”Proceedings of Machine Learning and Systems, vol. 6, pp. 74–86, 2024

2024
[55]

Traci: Network acceleration of input-dynamic communication for large-scale deep learning recommendation model,

G. Huang, H. Li, L. Qin, J. Huang, Y. Kang, Y. Ding, and Y. Xie, “Traci: Network acceleration of input-dynamic communication for large-scale deep learning recommendation model,” inProceedings of the 52nd Annual International Symposium on Computer Architecture , ser. ISCA ’25. ...

2025
[56]

GPU graph processing on cxl-based microsecond-latency external memory,

S. Sano, Y. Bando, K. Hiwada, H. Kajihara, T. Suzuki, Y. Nakanishi, D. Taki, A. Kaneko, and T. Sh- iozawa, “GPU graph processing on cxl-based microsecond-latency external memory,” inProceedings of the SC’23 Workshops of the International Conference on High Performance Computin...

2023
[57]

Understanding routable{PCIe} performance for composable infrastructures,

W. Hou, J. Zhang, Z. Wang, and M. Liu, “Understanding routable{PCIe} performance for composable infrastructures,” in21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 2024, pp. 297–312

2024
[58]

Understanding PCIe performance for end host networking,

R. Neugebauer, G. Antichi, J. F. Zazo, Y. Audzevich, S. López-Buedo, and A. W. Moore, “Understanding PCIe performance for end host networking,” inProceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, 2018, pp. 327–341

2018
[59]

CXL 1.0 specification,

CXL Consortium, “CXL 1.0 specification,” 2019. [Online]. Available: https://computeexpresslink.org/ wp-content/uploads/2024/02/CXL-1.0-Specification.pdf

2019
[60]

CXL 2.0 specification,

CXL Consortium, “CXL 2.0 specification,” 2020. [Online]. Available: https://computeexpresslink.org/ wp-content/uploads/2024/02/CXL-2.0-Specification.pdf

2020
[61]

CXL 3.2 specification,

CXL Consortium, “CXL 3.2 specification,” 2024. [Online]. Available: https://computeexpresslink.org/ cxl-specification/

2024
[62]

Overcoming the memory wall with CXL-Enabled SSDs,

S.-P. Yang, M. Kim, S. Nam, J. Park, J. yong Choi, E. H. Nam, E. Lee, S. Lee, and B. S. Kim, “Overcoming the memory wall with CXL-Enabled SSDs,” in2023 USENIX Annual Technical Conference (USENIX ATC 23). Boston, MA: USENIX Association, Jul. 2023, pp. 601–617. [Online]. Availab...

2023
[63]

An introduction to the compute express link (cxl) interconnect,

D. Das Sharma, R. Blankenship, and D. Berger, “An introduction to the compute express link (cxl) interconnect,” ACM Comput. Surv. , vol. 56, no. 11, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3669900

2024 doi
[64]

Hello bytes, bye blocks: Pcie storage meets compute express link for memory expansion (cxl- ssd),

M. Jung, “Hello bytes, bye blocks: Pcie storage meets compute express link for memory expansion (cxl- ssd),” in Proceedings of the 14th ACM Workshop on Hot Topics in Storage and File Systems, 2022, pp. 45–51

2022
[65]

Pond: CXL-Based Memory Pooling Systems for Cloud Platforms,

H. Li, D. S. Berger, L. Hsu, D. Ernst, P. Zardoshti, S. Novakovic, M. Shah, S. Rajadnya, S. Lee, I. Agarwal, M. D. Hill, M. Fontoura, and R. Bianchini, “Pond: CXL-Based Memory Pooling Systems for Cloud Platforms,” in Proceedings of the 28th ACM International Conference on Arch...

2023
[66]

Breaking the Memory Wall with Compute Express Link (CXL),

A. Godbole, “Breaking the Memory Wall with Compute Express Link (CXL),”
[67]

Direct access,{High-Performance} memory disaggregation with {DirectCXL},

D. Gouk, S. Lee, M. Kwon, and M. Jung, “Direct access,{High-Performance} memory disaggregation with {DirectCXL},” in2022 USENIX Annual Technical Conference (USENIX ATC 22), 2022, pp. 287–294

2022
[68]

Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks,

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks,” in Advances in Neural Information Processing Systems , H. Larochell...

2020
[69]

Active retrieval augmented generation,

Z. Jiang, F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig, “Active retrieval augmented generation,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Assoc...

2023
[70]

Benchmarking large language models in retrieval-augmented generation,

J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking large language models in retrieval-augmented generation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 17754–17762

2024
[71]

From local to global: A graph rag approach to query-focused summarization,

D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson, “From local to global: A graph rag approach to query-focused summarization,”arXiv preprint arXiv:2404.16130, 2024

2024 arXiv
[72]

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization,

T. Zhang, J. Yi, Z. Xu, and A. Shrivastava, “KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization,” in Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zh...

2024
[73]

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management,

W. Lee, J. Lee, J. Seo, and J. Sim, “InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). Santa Clara, CA: USENIX Association, Jul. 2024, pp. 155–...

2024
[75]

Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava, “Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression 56 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects i...

2025
[76]

UALink 200G 1.0 Specification,

UALink Consortium, “UALink 200G 1.0 Specification,” 2025. [Online]. Available: https: //ualinkconsortium.org/wp-content/uploads/2025/04/UALink-1.0-White_Paper_FINAL.pdf

2025
[77]

[Online]

UALink Consortium. [Online]. Available: https://ualinkconsortium.org/
[78]

How Ultra Ethernet and UALink Enable High-Performance, Scalable AI Net- works,

J. Ames and R. Lowman, “How Ultra Ethernet and UALink Enable High-Performance, Scalable AI Net- works,” 2025. [Online]. Available: https://www.synopsys.com/articles/ultra-ethernet-ualink-ai-networks. html#5

2025
[79]

What is NVLink?

NVIDIA, “What is NVLink?” 2023. [Online]. Available: https://blogs.nvidia.com/blog/ what-is-nvidia-nvlink

2023
[80]

NVLink NVIDIA

NVIDIA, “NVLink NVIDIA.” [Online]. Available: https://www.nvidia.com/en-us/design-visualization/ nvlink-bridges
[81]

WNVIDIA NVLink and NVLink Switch,

NVIDIA, “WNVIDIA NVLink and NVLink Switch,” 2023. [Online]. Available: https://www.nvidia.com/ en-us/data-center/nvlink/

2023
[82]

NVIDIA Fusion,

NVIDIA, “NVIDIA Fusion,” 2025. [Online]. Available: https://www.nvidia.com/en-us/data-center/ nvlink-fusion/

2025
[83]

NVIDIA Unveils NVLink Fusion for Industry to Build Semi-Custom AI Infrastructure With NVIDIA Partner Ecosystem,

A. Sharpiro, “NVIDIA Unveils NVLink Fusion for Industry to Build Semi-Custom AI Infrastructure With NVIDIA Partner Ecosystem,” 2025. [Online]. Available: https://nvidianews.nvidia.com/news/ nvidia-nvlink-fusion-semi-custom-ai-infrastructure-partner-ecosystem

2025
[84]

Design guidelines for high performance RDMA systems,

A. K. anda Michael Kaminsky and D. G. Anderson, “Design guidelines for high performance RDMA systems,” in 2016 USENIX annual technical conference (USENIX ATC 16). Denver, CO: USENIX Association, 2016. [Online]. Available: https://www.usenix.org/system/files/conference/atc16/at...

2016
[85]

An optimized rdma qp communication mechanism for hyperscale ai infrastructure,

J. Wang, B. Lin, J. Zhang, M. Sun, and Y. Pan, “An optimized rdma qp communication mechanism for hyperscale ai infrastructure,”Cluster Computing, vol. 28, 2024

2024
[86]

Rdma over commodity ethernet at scale,

C. Guo, H. Wu, Z. Deng, G. Soni, J. Ye, J. Padhye, and M. Lipshteyn, “Rdma over commodity ethernet at scale,” in Proceedings of the 2016 ACM SIGCOMM Conference, ser. SIGCOMM ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 202–215. [Online]. Available: htt...

2016
[87]

Rdma over ethernet for distributed training at meta scale,

A. Gangidi, R. Miao, S. Zheng, S. J. Bondu, G. Goes, H. Morsy, R. Puri, M. Riftadi, A. J. Shetty, J. Yang, S. Zhang, M. J. Fernandez, S. Gandham, and H. Zeng, “Rdma over ethernet for distributed training at meta scale,” inProceedings of the ACM SIGCOMM 2024 Conference, ser. AC...

2024
[88]

Graph 500 benchmarks,

Graph500, “Graph 500 benchmarks,” https://graph500.org/
[89]

NASA Parallel Benchmarks,

NASA Advanced Supercomputing Division, “NASA Parallel Benchmarks,” https://www.nas.nasa.gov/ software/npb.html
[90]

Lattice Boltzmann Method Benchmarks,

Lattice Boltzmann Method, “Lattice Boltzmann Method Benchmarks,” https://www.spec.org/cpu2017/ Docs/benchmarks/619.lbm_s.html
[91]

Tealeaf: A mini-application to enable design-space explorations for iter- ative sparse linear solvers,

S. McIntosh-Smith, M. Martineau, T. Deakin, G. Pawelczak, W. Gaudin, P. Garrett, W. Liu, R. Smedley- Stevenson, and D. Beckingsale, “Tealeaf: A mini-application to enable design-space explorations for iter- ative sparse linear solvers,” in2017 IEEE International Conference on ...

2017
[92]

Advances in Numerical Modeling for Heat Transfer and Thermal Management: A Review of Computational Approaches and Environmental Impacts,

Łach and D. Svyetlichnyy, “Advances in Numerical Modeling for Heat Transfer and Thermal Management: A Review of Computational Approaches and Environmental Impacts,”Energies, vol. 18, no. 5, 2025. [Online]. Available: https://www.mdpi.com/1996-1073/18/5/1302

2025
[93]

Sequence to sequence learning with neural networks,

I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” inPro- ceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, p. 3104–3112

2014
[94]

State-of-the-Art Speech Recognition with Sequence-to-Sequence Models,

C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, “State-of-the-Art Speech Recognition with Sequence-to-Sequence Models,” in2018 IEEE International Conference on A...

2018
[95]

Multi-task Sequence to Sequence Learning,

M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser, “Multi-task Sequence to Sequence Learning,” 2016. [Online]. Available: https://arxiv.org/abs/1511.06114

2016 arXiv
[96]

Improving language understanding by generative pre-training,

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018

2018
[97]

New models and developer products announced at devday,

OpenAI, “New models and developer products announced at devday,” 2023. [Online]. Available: https://openai.com/index/new-models-and-developer-products-announced-at-devday/

2023
[98]

Models. Llama3.1, 3.2, 3.3, Llama4

Meta, “Models. Llama3.1, 3.2, 3.3, Llama4.” [Online]. Available: https://www.llama.com/
[99]

GPT-4 Turbo,

OpenAI, “GPT-4 Turbo,” 2025. [Online]. Available: https://platform.openai.com/docs/models/ gpt-4-turbo

2025
[100]

Gemini 2.5: Our most intelligent AI model,

K. Kavukcuoglu, “Gemini 2.5: Our most intelligent AI model,” 2025. [Online]. Available: https://blog. google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking

2025
[101]

Report: Google DeepMind Gemini,

A. Thompson, “Report: Google DeepMind Gemini,” 2024. [Online]. Available: https://lifearchitect.ai/ gemini-report/

2024
[103]

Learning phrase representations using rnn encoder-decoder for statistical machine translation,

K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” 2014. [Online]. Available: https://arxiv.org/abs/1406.1078

2014 arXiv
[104]

Generating sequences with recurrent neural networks,

A. Graves, “Generating sequences with recurrent neural networks,”arXiv preprint arXiv:1308.0850, 2013

2013 arXiv
[105]

Long short-term memory,

A. Graves, “Long short-term memory,”Supervised sequence labelling with recurrent neural networks, pp. 37–45, 2012

2012
[106]

Language Models are Few-Shot Learners,

B. e. a. Tom, “Language Models are Few-Shot Learners,” inAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901. [Online]. Available: https://proceedings.neuri...

2020
[107]

Scaling language models: Methods, analysis & insights from training gopher,

J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., “Scaling language models: Methods, analysis & insights from training gopher,”arXiv preprint arXiv:2112.11446, 2021

2021 arXiv
[108]

Palm: Scaling language modeling with pathways,

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmannet al., “Palm: Scaling language modeling with pathways,”Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

2023
[109]

Bloom: A 176b-parameter open-access multilingual language model,

T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Galléet al., “Bloom: A 176b-parameter open-access multilingual language model,” 2023

2023
[110]

Learningrepresentationsbyback-propagatingerrors,

D.E.Rumelhart,G.E.Hinton,andR.J.Williams,“Learningrepresentationsbyback-propagatingerrors,” nature, vol. 323, no. 6088, pp. 533–536, 1986. 58 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure May 26, 2025

1986
[111]

Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation,

T. Kim, J. Oh, N. Kim, S. Cho, and S.-Y. Yun, “Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation,”arXiv preprint arXiv:2105.08919, 2021

2021 arXiv
[112]

Mse loss with outlying label for imbalanced classification,

S. Kato and K. Hotta, “Mse loss with outlying label for imbalanced classification,” arXiv preprint arXiv:2107.02393, 2021

2021 arXiv
[113]

Balanced mse for imbalanced visual regression,

J. Ren, M. Zhang, C. Yu, and Z. Liu, “Balanced mse for imbalanced visual regression,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 7926–7935

2022
[114]

On information and sufficiency,

S. Kullback and R. A. Leibler, “On information and sufficiency,”The annals of mathematical statistics, vol. 22, no. 1, pp. 79–86, 1951

1951
[115]

Generalized cross entropy loss for training deep neural networks with noisy labels,

Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” Advances in neural information processing systems, vol. 31, 2018

2018
[116]

The real-world-weight cross-entropy loss function: Modeling the costs of mislabel- ing,

Y. Ho and S. Wookey, “The real-world-weight cross-entropy loss function: Modeling the costs of mislabel- ing,” IEEE access, vol. 8, pp. 4806–4813, 2019

2019
[117]

Uses and abuses of the cross-entropy loss: Case studies in modern deep learning,

E. Gordon-Rodriguez, G. Loaiza-Ganem, G. Pleiss, and J. P. Cunningham, “Uses and abuses of the cross-entropy loss: Case studies in modern deep learning,” 2020

2020
[118]

Cross-entropy loss functions: Theoretical analysis and applications,

A. Mao, M. Mohri, and Y. Zhong, “Cross-entropy loss functions: Theoretical analysis and applications,” in International conference on Machine learning. PMLR, 2023, pp. 23803–23828

2023
[119]

A rapidly convergent descent method for minimization,

R. Fletcher and M. J. Powell, “A rapidly convergent descent method for minimization,”The computer journal, vol. 6, no. 2, pp. 163–168, 1963

1963
[120]

Gradient-based learning applied to document recogni- tion,

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recogni- tion,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998

1998
[121]

Learning to learn by gradient descent by gradient descent,

M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Fre- itas, “Learning to learn by gradient descent by gradient descent,”Advances in neural information pro- cessing systems, vol. 29, 2016

2016
[122]

Learning characteristics of stochastic-gradient-descent algorithms: A general study, anal- ysis, and critique,

W. A. Gardner, “Learning characteristics of stochastic-gradient-descent algorithms: A general study, anal- ysis, and critique,”Signal processing, vol. 6, no. 2, pp. 113–133, 1984

1984
[123]

Backpropagation and stochastic gradient descent method,

S.-i. Amari, “Backpropagation and stochastic gradient descent method,”Neurocomputing, vol. 5, no. 4-5, pp. 185–196, 1993

1993
[124]

Online learning and stochastic approximations,

L. Eon Bottou, “Online learning and stochastic approximations,”Online learning in neural networks, vol. 17, no. 9, p. 142, 1998

1998
[125]

Imagenet classification with deep convolutional neural networks,

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012

2012
[126]

Train faster, generalize better: Stability of stochastic gradient de- scent,

M. Hardt, B. Recht, and Y. Singer, “Train faster, generalize better: Stability of stochastic gradient de- scent,” inInternational conference on machine learning. PMLR, 2016, pp. 1225–1234

2016
[127]

On the convergence of stochastic gradient descent with adaptive stepsizes,

X. Li and F. Orabona, “On the convergence of stochastic gradient descent with adaptive stepsizes,” in The 22nd international conference on artificial intelligence and statistics. PMLR, 2019, pp. 983–992

2019
[128]

Recent advances in stochastic gradient descent in deep learning,

Y. Tian, Y. Zhang, and H. Zhang, “Recent advances in stochastic gradient descent in deep learning,” Mathematics, vol. 11, no. 3, p. 682, 2023

2023
[129]

Adam: A method for stochastic optimization,

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[130]

Improved adam optimizer for deep neural networks,

Z. Zhang, “Improved adam optimizer for deep neural networks,” in2018 IEEE/ACM 26th international symposium on quality of service (IWQoS). Ieee, 2018, pp. 1–2

2018
[131]

An improvement of the convergence proof of the adam-optimizer,

S. Bock, J. Goppold, and M. Weiß, “An improvement of the convergence proof of the adam-optimizer,” arXiv preprint arXiv:1804.10587, 2018. 59 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure May 26, 2025

2018 arXiv
[132]

Cnn based traffic sign classification using adam optimizer,

S. Mehta, C. Paunwala, and B. Vaidya, “Cnn based traffic sign classification using adam optimizer,” in 2019 international conference on intelligent computing and control systems (ICCS). IEEE, 2019, pp. 1293–1298

2019
[133]

An effective optimization method for machine learning based on adam,

D. Yi, J. Ahn, and S. Ji, “An effective optimization method for machine learning based on adam,”Applied Sciences, vol. 10, no. 3, p. 1073, 2020

2020
[134]

Review of Adaptive Activation Function in Deep Neural Network,

M. M. Lau and K. Hann Lim, “Review of Adaptive Activation Function in Deep Neural Network,” in2018 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), 2018, pp. 686–690

2018
[135]

Rectified linear units improve restricted boltzmann machines,

V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” inProceedings of the 27th international conference on machine learning (ICML-10), 2010, pp. 807–814

2010
[136]

Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations,

B. Neyshabur, Y. Wu, R. R. Salakhutdinov, and N. Srebro, “Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations,” in Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Curran As...

2016
[137]

Dual Rectified Linear Units (DReLUs): A replacement for tanh activation functions in Quasi-Recurrent Neural Networks,

F. Godin, J. Degrave, J. Dambre, and W. De Neve, “Dual Rectified Linear Units (DReLUs): A replacement for tanh activation functions in Quasi-Recurrent Neural Networks,” Pattern Recognition Letters, vol. 116, pp. 8–14, 2018. [Online]. Available: https://www.sciencedirect.com/sc...

2018
[138]

Activation functions in deep learning: A comprehensive survey and benchmark,

S. R. Dubey, S. K. Singh, and B. B. Chaudhuri, “Activation functions in deep learning: A comprehensive survey and benchmark,”Neurocomputing, vol. 503, pp. 92–108, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231222008426

2022
[139]

On the Properties of Neural Machine Translation: Encoder-Decoder Approaches,

K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches,” 2014. [Online]. Available: https://arxiv.org/abs/1409.1259

2014 arXiv
[140]

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions,

S. Hochreiter, “The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions,” International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems , vol. 06, no. 02, pp. 107–116, 1998. [Online]. Available: https://doi.org/10.1142/S0218488598000094

1998 doi
[141]

Very Deep Convolutional Networks for Text Classification,

A. Conneau, H. Schwenk, L. Barrault, and Y. Lecun, “Very Deep Convolutional Networks for Text Classification,” 2017. [Online]. Available: https://arxiv.org/abs/1606.01781

2017 arXiv
[142]

Effective approaches to attention-based neural machine translation,

M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” 2015. [Online]. Available: https://arxiv.org/abs/1508.04025

2015 arXiv
[143]

Neural machine translation by jointly learning to align and translate,

D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014

2014 arXiv
[144]

A high-speed and low-complexity architecture for softmax function in deep learning,

M. Wang, S. Lu, D. Zhu, J. Lin, and Z. Wang, “A high-speed and low-complexity architecture for softmax function in deep learning,” in2018 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), 2018, pp. 223–226

2018
[145]

Large-margin softmax loss for convolutional neural networks,

W. Liu, Y. Wen, Z. Yu, and M. Yang, “Large-margin softmax loss for convolutional neural networks,”
[146]

Incrementally Learning the Hierarchical Softmax Function for Neural Language Models,

H. Peng, J. Li, Y. Song, and Y. Liu, “Incrementally Learning the Hierarchical Softmax Function for Neural Language Models,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, Feb. 2017. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article...

2017
[147]

A structured self-attentive sentence embedding,

Z. Lin, M. Feng, C. N. d. Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, “A structured self-attentive sentence embedding,”arXiv preprint arXiv:1703.03130, 2017

2017 arXiv
[148]

Is attention interpretable?

S. Serrano and N. A. Smith, “Is attention interpretable?”arXiv preprint arXiv:1906.03731, 2019

1906 arXiv
[149]

Assessing the use of attention weights to interpret bert-based stance classification,

C. A. Córdova Sáenz and K. Becker, “Assessing the use of attention weights to interpret bert-based stance classification,” inIEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 2021, pp. 194–201. 60 Compute Can’t Handle the Truth: Why Co...

2021
[150]

Attention is not not explanation,

S. Wiegreffe and Y. Pinter, “Attention is not not explanation,”arXiv preprint arXiv:1908.04626, 2019

1908 arXiv
[151]

Self-attention with relative position representations,

P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” 2018. [Online]. Available: https://arxiv.org/abs/1803.02155

2018 arXiv
[152]

Exploring self-attention for image recognition,

H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10076–10085

2020
[153]

Self-attention and behavior: A review and theoretical update,

F. X. Gibbons, “Self-attention and behavior: A review and theoretical update,”Advances in experimental social psychology, vol. 23, pp. 249–303, 1990

1990
[154]

Self-attention generative adversarial networks,

H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in International conference on machine learning. PMLR, 2019, pp. 7354–7363

2019
[155]

Stand-alone self-attention in vision models,

P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,”Advances in neural information processing systems, vol. 32, 2019

2019
[156]

Reformer: The efficient transformer,

N. Kitaev, Ł. Kaiser, and A. Levskaya, “Reformer: The efficient transformer,” arXiv preprint arXiv:2001.04451, 2020

2001 arXiv
[157]

Big bird: Transformers for longer sequences,

M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, and A. Ahmed, “Big bird: Transformers for longer sequences,” in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan,...

2020
[158]

On the relationship between self-attention and convolutional layers,

J.-B. Cordonnier, A. Loukas, and M. Jaggi, “On the relationship between self-attention and convolutional layers,” arXiv preprint arXiv:1911.03584, 2019

1911 arXiv
[159]

Relative positional encoding for transformers with linear complexity,

A. Liutkus, O. Cıfka, S.-L. Wu, U. Simsekli, Y.-H. Yang, and G. Richard, “Relative positional encoding for transformers with linear complexity,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 7067–7079

2021
[160]

Roformer: Enhanced transformer with rotary position embedding,

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,”Neurocomputing, vol. 568, p. 127063, 2024

2024
[161]

On the diversity of multi-head attention,

J. Li, X. Wang, Z. Tu, and M. R. Lyu, “On the diversity of multi-head attention,”Neurocomputing, vol. 454, pp. 14–24, 2021

2021
[162]

Multi-head attention with disagreement regularization,

J. Li, Z. Tu, B. Yang, M. R. Lyu, and T. Zhang, “Multi-head attention with disagreement regularization,” arXiv preprint arXiv:1810.10183, 2018

2018 arXiv
[163]

GQA: Training generalized multi-query transformer models from multi-head checkpoints,

J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebron, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pi...

2023
[164]

Beyond uniform query distribution: Key-driven grouped query attention,

Z. Khan, M. Khaquan, O. Tafveez, B. Samiwala, and A. A. Raza, “Beyond uniform query distribution: Key-driven grouped query attention,” 2024. [Online]. Available: https://arxiv.org/abs/2408.08454

2024 arXiv
[165]

Optimised grouped-query attention mechanism for transformers,

Y. Chen, C. Zhang, X. Gao, R. D. Mullins, G. A. Constantinides, and Y. Zhao, “Optimised grouped-query attention mechanism for transformers,” 2024. [Online]. Available: https://arxiv.org/abs/2406.14963

2024 arXiv
[166]

Gaussian Error Linear Units (GELUs),

D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs),” 2023. [Online]. Available: https://arxiv.org/abs/1606.08415

2023 arXiv
[167]

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 2017. [Online]. Available: https://arxiv.org/abs/1701.06538

2017 arXiv
[168]

Gshard: Scaling giant models with conditional computation and automatic sharding,

D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” 2020. [Online]. Available: https://arxiv.org/abs/2006.16668 61 Compute Can’t Handle the Truth: Why C...

2020 arXiv
[169]

Sparsely activated mixture-of-experts are robust multi-task learners,

S. Gupta, S. Mukherjee, K. Subudhi, E. Gonzalez, D. Jose, A. H. Awadallah, and J. Gao, “Sparsely activated mixture-of-experts are robust multi-task learners,”arXiv preprint arXiv:2204.07689, 2022

2022 arXiv
[170]

Designing effective sparse expert models,

B. Zoph, “Designing effective sparse expert models,” in2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 2022, pp. 1044–1044

2022
[171]

DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale,

S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Y. Aminabadi, A. A. Awan, J. Rasley, and Y. He, “DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation AI scale,” in Proceedings of the 39th International Conference on Machine Learning, ser. Pro...

2022
[172]

Available: https://platform.openai.com/docs/models

OpenAI, “Models.” [Online]. Available: https://platform.openai.com/docs/models
[173]

Introducing gemini 2.0: our new ai model for the agentic era,

S. Pichai, D. Hassabis, and K. Kavukcuoglu, “Introducing gemini 2.0: our new ai model for the agentic era,” 2024. [Online]. Available: https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024

2024
[174]

Gemini: A Family of Highly Capable Multimodal Models,

G. Team and R. e. a. Anil, “Gemini: A Family of Highly Capable Multimodal Models,” 2025. [Online]. Available: https://arxiv.org/abs/2312.11805

2025 arXiv
[175]

RoBERTa: A Robustly Optimized BERT Pretraining Approach,

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692

2019 arXiv
[176]

XLNet: Generalized Autoregressive Pretraining for Language Understanding,

Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “XLNet: Generalized Autoregressive Pretraining for Language Understanding,” in Advances in Neural Information Processing Systems , H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alché-Buc, E. Fox, and...

2019
[177]

Spanbert: Improving pre-training by representing and predicting spans,

M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, “Spanbert: Improving pre-training by representing and predicting spans,”Transactions of the Association for Computational Linguistics, vol. 8, pp. 64–77, 01 2020. [Online]. Available: https://doi.org/10.1162/t...

2020 doi
[178]

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,

Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,” 2020. [Online]. Available: https://arxiv.org/abs/1909.11942

2020 arXiv
[179]

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,

Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and z. Chen, “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism,” inAdvances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelz...

2019
[180]

The Longest Training Run

Epoch AI, “The Longest Training Run.” [Online]. Available: https://epoch.ai/blog/ the-longest-training-run
[181]

BigScience Model Training Launched

Bigscience, “BigScience Model Training Launched.” [Online]. Available: https://bigscience.huggingface. co/blog/model-training-launched
[182]

LLM Training: Mastering the Art of Language Model Development

Rao, R., “LLM Training: Mastering the Art of Language Model Development.” [Online]. Available: https://www.wevolver.com/article/llm-training-mastering-the-art-of-language-model-development
[183]

Opt: Open pre-trained transformer language models,

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Linet al., “Opt: Open pre-trained transformer language models,”arXiv preprint arXiv:2205.01068, 2022

2022 arXiv
[184]

Finetuned Language Models Are Zero-Shot Learners,

J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned Language Models Are Zero-Shot Learners,” 2022. [Online]. Available: https://arxiv.org/abs/2109.01652 62 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memor...

2022 arXiv
[185]

Large Language Models are Zero-Shot Reasoners,

T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large Language Models are Zero-Shot Reasoners,” inAdvances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 2...

2022
[186]

Language models are unsupervised multitask learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

2019
[187]

Kvquant: Towards 10 million context length llm inference with kv cache quantization,

C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, “Kvquant: Towards 10 million context length llm inference with kv cache quantization,”Advances in Neural Information Processing Systems, vol. 37, pp. 1270–1303, 2024

2024
[188]

Retrieval augmentation reduces hallucination in conversation,

K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” 2021. [Online]. Available: https://arxiv.org/abs/2104.07567

2021 arXiv
[189]

Reducing hallucination in structured outputs via retrieval-augmented generation,

O. Ayala and P. Bechard, “Reducing hallucination in structured outputs via retrieval-augmented generation,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Tra...

2024 doi
[190]

Hybridrag: Integrating knowledge graphs and vector retrieval augmented generation for efficient information extraction,

B. Sarmah, D. Mehta, B. Hall, R. Rao, S. Patel, and S. Pasquali, “Hybridrag: Integrating knowledge graphs and vector retrieval augmented generation for efficient information extraction,” inProceedings of the 5th ACM International Conference on AI in Finance, 2024, pp. 608–616

2024
[191]

Optimize Vector Databases, Enhance RAG-Driven Generative AI

Intel Tech, “Optimize Vector Databases, Enhance RAG-Driven Generative AI.” [Online]. Available: https: //medium.com/intel-tech/optimize-vector-databases-enhance-rag-driven-generative-ai-90c10416cb9c
[192]

Chat2data: An interactive data analysis system with rag, vector databases and llms,

X. Zhao, X. Zhou, and G. Li, “Chat2data: An interactive data analysis system with rag, vector databases and llms,”Proceedings of the VLDB Endowment, vol. 17, no. 12, pp. 4481–4484, 2024

2024
[193]

What Is Retrieval-Augmented Generation, aka RAG?

NVIDIA, “What Is Retrieval-Augmented Generation, aka RAG?” [Online]. Available: https: //blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/
[194]

Large language models for information retrieval: A survey,

Y. Zhu, H. Yuan, S. Wang, J. Liu, W. Liu, C. Deng, H. Chen, Z. Liu, Z. Dou, and J.-R. Wen, “Large language models for information retrieval: A survey,”arXiv preprint arXiv:2308.07107, 2023

2023
[195]

Exploring the integration strategies of retriever and large language models,

Y. Liu, S. Yavuz, R. Meng, M. Moorthy, S. Joty, C. Xiong, and Y. Zhou, “Exploring the integration strategies of retriever and large language models,”arXiv preprint arXiv:2308.12574, 2023

2023 arXiv
[196]

Deep neural networks for youtube recommendations,

P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube recommendations,” in Proceedings of the 10th ACM Conference on Recommender Systems , ser. RecSys ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 191–198. [Online]. Available: https:/...

2016
[197]

Wide & deep learning for recommender systems,

H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispir, R. Anil, Z. Haque, L. Hong, V. Jain, X. Liu, and H. Shah, “Wide & deep learning for recommender systems,” inProceedings of the 1st Workshop on Deep Learning for Reco...

2016
[198]

The youtube video recommendation system,

J. Davidson, B. Liebald, J. Liu, P. Nandy, T. Van Vleet, U. Gargi, S. Gupta, Y. He, M. Lambert, B. Livingston, and D. Sampath, “The youtube video recommendation system,” in Proceedings of the Fourth ACM Conference on Recommender Systems, ser. RecSys ’10. New York, NY, USA: Ass...
[199]

Learning to rank using gradient descent,

C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to rank using gradient descent,” inProceedings of the 22nd International Conference on Machine Learning, ser. ICML ’05. New York, NY, USA: Association for Computing Machinery, 2005,...

2005
[200]

Learning to rank: from pairwise approach to listwise approach,

Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li, “Learning to rank: from pairwise approach to listwise approach,” inProceedings of the 24th International Conference on Machine Learning, ser. ICML ’07. New York, NY, USA: Association for Computing Machinery, 2007, p. 129–136. [...

2007
[201]

Highly accurate protein structure prediction with alphafold,

J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenkoet al., “Highly accurate protein structure prediction with alphafold,”nature, vol. 596, no. 7873, pp. 583–589, 2021

2021
[202]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

2016
[203]

Scaling laws for autoregressive generative modeling,

T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, C. Hallacy, B. Mann, A. Radford, A. Ramesh, N. Ryder, D. M. Ziegler, J. Schulman, D. Amodei, and S. McCandlish, “Scaling laws for autoregressive generative modeling,” 202...

2020 arXiv
[204]

Flashdecoding++: Faster large language model inference on gpus,

K. Hong, G. Dai, J. Xu, Q. Mao, X. Li, J. Liu, K. Chen, Y. Dong, and Y. Wang, “Flashdecoding++: Faster large language model inference on gpus,”arXiv preprint arXiv:2311.01282, 2023

2023 arXiv
[205]

A study of single and multi-device synchronization methods in nvidia gpus,

L. Zhang, M. Wahib, H. Zhang, and S. Matsuoka, “A study of single and multi-device synchronization methods in nvidia gpus,” in 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2020, pp. 483–493

2020
[206]

Large scale distributed deep networks,

J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang et al., “Large scale distributed deep networks,”Advances in neural information processing systems, vol. 25, 2012

2012
[207]

Accurate, large minibatch sgd: Training imagenet in 1 hour,

P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,”arXiv preprint arXiv:1706.02677, 2017

2017 arXiv
[208]

Osp: Boosting distributed model training with 2-stage synchronization,

Z. Chen, L. Shi, X. Liu, J. Li, S. Liu, and Y. Xu, “Osp: Boosting distributed model training with 2-stage synchronization,” inProceedings of the 52nd International Conference on Parallel Processing, 2023, pp. 102–111

2023
[209]

Dreamddp: Accelerating data parallel distributed llm training with layer-wise scheduled partial synchronization,

Z. Tang, Z. Tang, J. Huang, X. Pan, R. Yan, Y. Wang, A. C. Zhou, S. Shi, X. Chu, and B. Li, “Dreamddp: Accelerating data parallel distributed llm training with layer-wise scheduled partial synchronization,” arXiv preprint arXiv:2502.11058, 2025

2025 arXiv
[210]

Acco: Accumulate while you communicate, hiding communications in distributed llm training,

A. Nabli, L. Fournier, P. Erbacher, L. Serrano, E. Belilovsky, and E. Oyallon, “Acco: Accumulate while you communicate, hiding communications in distributed llm training,”arXiv preprint arXiv:2406.02613, 2024

2024
[211]

A quadratic synchronization rule for distributed deep learning,

X. Gu, K. Lyu, S. Arora, J. Zhang, and L. Huang, “A quadratic synchronization rule for distributed deep learning,” arXiv preprint arXiv:2310.14423, 2023

2023 arXiv
[212]

Terngrad: Ternary gradients to reduce communication in distributed deep learning,

W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, “Terngrad: Ternary gradients to reduce communication in distributed deep learning,”Advances in neural information processing systems, vol. 30, 2017

2017
[213]

MG-WFBP:Merginggradientswiselyforefficientcommunicationindistributed deep learning,

S.Shi,X.Chu,andB.Li,“MG-WFBP:Merginggradientswiselyforefficientcommunicationindistributed deep learning,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 8, pp. 1903–1917, 2021

1903
[214]

Beyond data and model parallelism for deep neural networks

Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks.” in Proceedings of Machine Learning and Systems, A. Talwalkar, V. Smith, and M. Zaharia, Eds., vol. 1, 2019, pp. 1–13. [Online]. Available: https://proceedings.mlsys.org/paper_files/...

2019
[215]

Tesseract: Parallelize the tensor parallelism efficiently,

B. Wang, Q. Xu, Z. Bian, and Y. You, “Tesseract: Parallelize the tensor parallelism efficiently,” inPro- ceedings of the 51st International Conference on Parallel Processing, 2022, pp. 1–11. 64 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interc...

2022
[216]

PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices,

K. Osawa, S. Li, and T. Hoefler, “PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices,” inProceedings of Machine Learning and Systems, D. Song, M. Carbin, and T. Chen, Eds., vol. 5. Curan, 2023, pp. 708–727. [Online]. Availa...

2023
[217]

Pipelayer: A pipelined reram-based accelerator for deep learning,

L. Song, X. Qian, H. Li, and Y. Chen, “Pipelayer: A pipelined reram-based accelerator for deep learning,” in 2017 IEEE international symposium on high performance computer architecture (HPCA). IEEE, 2017, pp. 541–552

2017
[218]

Pipetransformer: Automated elastic pipelining for distributed training of transformers,

C. He, S. Li, M. Soltanolkotabi, and S. Avestimehr, “Pipetransformer: Automated elastic pipelining for distributed training of transformers,”arXiv preprint arXiv:2102.03161, 2021

2021 arXiv
[219]

Mpi: A message-passing interface standard,

M. P. Forum, “Mpi: A message-passing interface standard,” 1994

1994
[220]

Optimization of collective communication operations in mpich,

R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of collective communication operations in mpich,” The International Journal of High Performance Computing Applications, vol. 19, no. 1, pp. 49–66, 2005. [Online]. Available: https://doi.org/10.1177/1094342005051521

2005 doi
[221]

Efficient sparse collective communication and its application to accelerate distributed deep learning,

J. Fei, C.-Y. Ho, A. N. Sahu, M. Canini, and A. Sapio, “Efficient sparse collective communication and its application to accelerate distributed deep learning,” inProceedings of the 2021 ACM SIGCOMM 2021 Conference, ser. SIGCOMM ’21. New York, NY, USA: Association for Computing...

2021
[222]

AutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and Parallel DNN Training,

G. Xu, Z. Le, Y. Chen, Z. Lin, Z. Jin, Y. Miao, and C. Li, “AutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and Parallel DNN Training,” in 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25) . Philadelphia, PA: USENIX ...

2025
[223]

SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization,

M. Zhai, J. He, Z. Ma, Z. Zong, R. Zhang, and J. Zhai, “SmartMoE: Efficiently training Sparsely-Activated models through combining offline and online parallelization,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23). Boston, MA: USENIX Association, Jul. 2023, pp. 96...

2023
[224]

Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference,

H. Huang, N. Ardalani, A. Sun, L. Ke, H.-H. S. Lee, A. Sridhar, S. Bhosale, C.-J. Wu, and B. Lee, “Towards moe deployment: Mitigating inefficiencies in mixture-of-expert (moe) inference,”arXiv preprint arXiv:2303.06182, 2023

2023 arXiv
[225]

How NVIDIA GB200 NVL72 and NVIDIA Dynamo Boost Inference Per- formance for MoE Models

Mitra, T. and Borkar, R. and Elmeleegy, A. and Kapasi, U. and Darvish, B., “How NVIDIA GB200 NVL72 and NVIDIA Dynamo Boost Inference Per- formance for MoE Models.” [Online]. Available: https://developer.nvidia.com/blog/ how-nvidia-gb200-nvl72-and-nvidia-dynamo-boost-inference-...
[226]

Demystifying AI Inference Deployments for Trillion Parameter Large Language Models

Elmeleegy, A. and Raj, S. and Slechta, B. and Mehta, V., “Demystifying AI Inference Deployments for Trillion Parameter Large Language Models.” [Online]. Available: https://developer.nvidia.com/blog/ demystifying-ai-inference-deployments-for-trillion-parameter-large-language-models/
[227]

Fast Distributed InferenceServingforLargeLanguageModels,

B. Wu, Y. Zhong, Z. Zhang, S. Liu, F. Liu, Y. Sun, G. Huang, X. Liu, and X. Jin, “Fast Distributed InferenceServingforLargeLanguageModels,”2024.[Online].Available:https://arxiv.org/abs/2305.05920

2024 arXiv
[228]

Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline,

Z. Zheng, X. Ren, F. Xue, Y. Luo, X. Jiang, and Y. You, “Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds....

2023
[229]

Efficient LLMs Training and Inference: An Introduction,

R. Li, D. Fu, C. Shi, Z. Huang, and G. Lu, “Efficient LLMs Training and Inference: An Introduction,” IEEE Access, vol. 13, pp. 32944–32970, 2025

2025
[230]

Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,

R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasleyet al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” in SC22: International Conference for High Performance Computi...

2022
[231]

Inference-optimized ai and high performance computing for gravitational wave detection at scale,

P. Chaturvedi, A. Khan, M. Tian, E. Huerta, and H. Zheng, “Inference-optimized ai and high performance computing for gravitational wave detection at scale,”Frontiers in Artificial Intelligence, vol. 5, p. 828672, 2022

2022
[232]

Mastering LLM Techniques: Inference Optimization

Verma, S. and Vaidya, N., “Mastering LLM Techniques: Inference Optimization.” [Online]. Available: https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
[233]

Splitwise: Efficient Gener- ative LLM Inference Using Phase Splitting,

P. Patel, E. Choukse, C. Zhang, A. Shah, Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient Gener- ative LLM Inference Using Phase Splitting,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 118–132

2024
[234]

Dense passage retrieval for open-domain question answering

V. Karpukhin, B. Oguz, S. Min, P. S. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering.” inEMNLP (1), 2020, pp. 6769–6781

2020
[235]

QA-RAG: Exploring LLM reliance on external knowledge,

A. Mansurova, A. Mansurova, and A. Nugumanova, “QA-RAG: Exploring LLM reliance on external knowledge,” Big Data and Cognitive Computing, vol. 8, no. 9, p. 115, 2024

2024
[236]

Retrieval- augmented generation for large language models: A survey,

Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, and H. Wang, “Retrieval- augmented generation for large language models: A survey,”arXiv preprint arXiv:2312.10997, vol. 2, no. 1, 2023

2023 arXiv
[237]

Accelerating Retrieval-Augmented Generation,

D. Quinn, M. Nouri, N. Patel, J. Salihu, A. Salemi, S. Lee, H. Zamani, and M. Alian, “Accelerating Retrieval-Augmented Generation,” in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ser. ...

2025
[238]

Measured performance of an Ethernet local network,

J. F. Shoch and J. A. Hupp, “Measured performance of an Ethernet local network,”Commun. ACM, vol. 23, no. 12, p. 711–721, Dec. 1980. [Online]. Available: https://doi.org/10.1145/359038.359044

1980
[239]

Topology discovery for large ethernet networks,

B. Lowekamp, D. O’Hallaron, and T. Gross, “Topology discovery for large ethernet networks,” in Proceedings of the 2001 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications, ser. SIGCOMM ’01. New York, NY, USA: Association for Compu...

2001
[240]

PAST: scalable ethernet for data centers,

B. Stephens, A. Cox, W. Felter, C. Dixon, and J. Carter, “PAST: scalable ethernet for data centers,” in Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, ser. CoNEXT ’12. New York, NY, USA: Association for Computing Machinery,...

2012
[241]

Optimized Routing for Large-Scale InfiniBand Networks,

T. Hoefler, T. Schneider, and A. Lumsdaine, “Optimized Routing for Large-Scale InfiniBand Networks,” in 2009 17th IEEE Symposium on High Performance Interconnects, 2009, pp. 103–111

2009
[242]

InfiniBand Network Monitoring: Challenges and Possibilities,

K. Hintze, S. Graham, S. Dunlap, and P. Sweeney, “InfiniBand Network Monitoring: Challenges and Possibilities,” in Critical Infrastructure Protection XV, J. Staggs and S. Shenoi, Eds. Cham: Springer International Publishing, 2022, pp. 187–208

2022
[243]

Buyya, T

R. Buyya, T. Cortes, and H. Jin,An Introduction to the InfiniBand Architecture, 2002, pp. 616–632

2002
[244]

SRNIC: A scalable architecture for RDMA NICs,

Z. Wang, L. Luo, Q. Ning, C. Zeng, W. Li, X. Wan, P. Xie, T. Feng, K. Cheng, X. Geng, T. Wang, W. Ling, K. Huo, P. An, K. Ji, S. Zhang, B. Xu, R. Feng, T. Ding, K. Chen, and C. Guo, “SRNIC: A scalable architecture for RDMA NICs,” in20th USENIX Symposium on Networked Systems De...

2023
[245]

GPUrdma: GPU-side library for high performance networking from GPU kernels,

F. Daoud, A. Watad, and M. Silberstein, “GPUrdma: GPU-side library for high performance networking from GPU kernels,” inProceedings of the 6th International Workshop on Runtime and Operating Systems for Supercomputers, ser. ROSS ’16. New York, NY, USA: Association for Computin...

2016
[246]

GPUDirect Async: Exploring GPU synchronous communication techniques for InfiniBand clusters,

E. Agostini, D. Rossetti, and S. Potluri, “GPUDirect Async: Exploring GPU synchronous communication techniques for InfiniBand clusters,”Journal of Parallel and Distributed Computing, vol. 114, pp. 28–45,
[248]

A scalable, commodity data center network architecture,

M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commodity data center network architecture,” in Proceedings of the ACM SIGCOMM 2008 Conference on Data Communication, ser. SIGCOMM ’08. New York, NY, USA: Association for Computing Machinery, 2008, p. 63–74. [Online]. Avai...

2008
[249]

Jellyfish: Networking data centers randomly,

A. Singla, C.-Y. Hong, L. Popa, and P. B. Godfrey, “Jellyfish: Networking data centers randomly,” in9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12), 2012, pp. 225–238

2012
[250]

Meta’s Hyperscale Infrastructure: Overview and Insights,

C. Tang, “Meta’s Hyperscale Infrastructure: Overview and Insights,”Commun. ACM, vol. 68, no. 2, p. 52–63, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3701296

2025 doi
[251]

Reinventing Facebook’s data center network,

A. Andreyev, W. Xu, and A. Eckert, “Reinventing Facebook’s data center network,” 2019. [Online]. Available: https://engineering.fb.com/2019/03/14/data-center-engineering/f16-minipack/

2019
[252]

NVIDIA Blackwell Architecture

NVIDIA, “NVIDIA Blackwell Architecture.” [Online]. Available: https://www.nvidia.com/en-us/ data-center/technologies/blackwell-architecture/
[253]

Available: https://www.micron.com/products/memory/hbm/hbm3e

Micron, “HBM3E.” [Online]. Available: https://www.micron.com/products/memory/hbm/hbm3e
[254]

NVIDIA NVLink-C2C

NVIDIA, “NVIDIA NVLink-C2C.” [Online]. Available: https://www.nvidia.com/en-us/data-center/ nvlink-c2c/
[255]

NVIDIA Blackwell Architecture Technical Brief,

NVIDIA, “NVIDIA Blackwell Architecture Technical Brief,” 2024. [Online]. Available: https: //resources.nvidia.com/en-us-blackwell-architecture

2024
[256]

NVIDIA Grace

NVIDIA, “NVIDIA Grace.” [Online]. Available: https://www.nvidia.com/en-us/data-center/grace-cpu/
[257]

9.3 nvlink-c2c: A coherent off package chip-to-chip interconnect with 40gbps/pin single-ended signaling,

Y. Wei, Y. C. Huang, H. Tang, N. Sankaran, I. Chadha, D. Dai, O. Oluwole, V. Balan, and E. Lee, “9.3 nvlink-c2c: A coherent off package chip-to-chip interconnect with 40gbps/pin single-ended signaling,” in 2023 IEEE International Solid-State Circuits Conference (ISSCC). IEEE, ...

2023
[258]

Nvidia Data Center Processing Unit (DPU) Architecture,

I. Burstein, “Nvidia Data Center Processing Unit (DPU) Architecture,” in2021 IEEE Hot Chips 33 Symposium (HCS), 2021, pp. 1–20

2021
[259]

An introduction to nvidia connectx-5 network adapter,

F. Inc., “An introduction to nvidia connectx-5 network adapter,” 2024. [Online]. Available: https://www.fs.com/blog/an-introduction-to-nvidia-connectx5-network-adapter-2558.html

2024
[260]

Accelerating with xdp over mellanox connectx nics,

N. Shankarappa, “Accelerating with xdp over mellanox connectx nics,” 2020. [Online]. Available: https://developer.nvidia.com/blog/accelerating-with-xdp-over-mellanox-connectx-nics/

2020
[261]

NVIDIA Contributes NVIDIA GB200 NVL72 Designs to Open Compute Project,

A. Elmeleegy, “NVIDIA Contributes NVIDIA GB200 NVL72 Designs to Open Compute Project,” 2024. [Online]. Available: https://developer.nvidia.com/blog/ nvidia-contributes-nvidia-gb200-nvl72-designs-to-open-compute-project/

2024
[262]

NVIDIA NVLink and NVLink Switch

NVIDIA, “NVIDIA NVLink and NVLink Switch.” [Online]. Available: https://www.nvidia.com/en-us/ data-center/nvlink/
[263]

NVIDIA NVLink and NVIDIA NVSwitch Supercharge Large Lan- guage Model Inference,

Brian Slechta and Nick Comly and Ashraf Eassa and Joe DeLaere and Shivam Raj, “NVIDIA NVLink and NVIDIA NVSwitch Supercharge Large Lan- guage Model Inference,” 2024. [Online]. Available: https://developer.nvidia.com/blog/ nvidia-nvlink-and-nvidia-nvswitch-supercharge-large-lan...

2024
[264]

High throughput data center topology design,

A. Singla, P. B. Godfrey, and A. Kolla, “High throughput data center topology design,” in11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), 2014, pp. 29–41

2014
[265]

What is an aggregate switch?

F. Inc., “What is an aggregate switch?” 2023. [Online]. Available: https://www.fs.com/blog/ what-is-an-aggregate-switch-1340.html 67 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure May 26, 2025

2023
[266]

Dumbnet: a smart data center network fabric with dumb switches,

Y. Li, D. Wei, X. Chen, Z. Song, R. Wu, Y. Li, X. Jin, and W. Xu, “Dumbnet: a smart data center network fabric with dumb switches,” inProceedings of the Thirteenth EuroSys Conference, ser. EuroSys ’18. New York, NY, USA: Association for Computing Machinery, 2018. [Online]. Ava...

2018
[267]

Whitepaper: Cisco aci multi-tier architecture,

Cisco, “Whitepaper: Cisco aci multi-tier architecture,” 2024. [On- line]. Available: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/ application-centric-infrastructure/white-paper-c11-742214.pdf

2024
[268]

What is spine-leaf architecture?

HPE, “What is spine-leaf architecture?” 2024. [Online]. Available: https://www.hpe.com/emea_africa/ en/what-is/spine-leaf-architecture.html

2024
[269]

On the data path performance of leaf-spine datacenter fabrics,

M. Alizadeh and T. Edsall, “On the data path performance of leaf-spine datacenter fabrics,” in2013 IEEE 21st Annual Symposium on High-Performance Interconnects, 2013, pp. 71–74

2013
[270]

QM87xx 1U HDR 200Gb/s InfiniBand Switch Systems User Manual

NVIDIA, “QM87xx 1U HDR 200Gb/s InfiniBand Switch Systems User Manual.” [Online]. Available: https://docs.nvidia.com/qm87xx-1u-hdr-200gb-s-infiniband-switch-systems-user-manual.pdf
[271]

NVIDIA Quantum InfiniBand Switches

NVIDIA, “NVIDIA Quantum InfiniBand Switches.” [Online]. Available: https://www.nvidia.com/en-us/ networking/infiniband-switching/
[272]

NVIDIA Quantum-2 InfiniBand Platform Datasheet

NVIDIA, “NVIDIA Quantum-2 InfiniBand Platform Datasheet.” [Online]. Available: https://nvdam. widen.net/s/dps8txlsrf/infiniband-ndr-400g-architecture-datasheet-1620877-r4
[273]

NVIDIA Spectrum-X Datasheet

NVIDIA, “NVIDIA Spectrum-X Datasheet.” [Online]. Available: https://resources.nvidia.com/ en-us-networking-ai/networking-ethernet-1
[274]

Data center projects: establishing a floor plan,

N. Rasmussen and W. Torell, “Data center projects: establishing a floor plan,”White Paper, vol. 144, 2007

2007
[275]

Understanding Coolant Distribution Units (CDUs) for Liquid Cooling,

Vertiv, “Understanding Coolant Distribution Units (CDUs) for Liquid Cooling,” 2023. [On- line]. Available: https://www.vertiv.com/en-us/about/news-and-insights/articles/educational-articles/ understanding-coolant-distribution-units-cdus-for-liquid-cooling/

2023
[276]

Data Center Cooling Systems: Coolant Distribution Unit Liquid Cooling

BOYD, “Data Center Cooling Systems: Coolant Distribution Unit Liquid Cooling.” [Online]. Available: https://www.boydcorp.com/blog/data-center-cooling-systems-coolant-distribution-unit-liquid-cooling. html
[277]

DC Power Distribution Unit for V2 Open Rack,

S. Kala and S. Mills, “DC Power Distribution Unit for V2 Open Rack,” 2015. [Online]. Available: https://www.opencompute.org/wiki/Open_Rack/SpecsAndDesigns

2015
[278]

Open Rack V3 48V PSU Specification Rev 1.0,

H. Keyhani, “Open Rack V3 48V PSU Specification Rev 1.0,” 2022. [Online]. Available: https://www.opencompute.org/wiki/Open_Rack/SpecsAndDesigns

2022
[279]

Meta Open Rack V3 BBU Module,

Meta, “Meta Open Rack V3 BBU Module,” 2022. [Online]. Available: https://www.opencompute.org/ wiki/Open_Rack/SpecsAndDesigns

2022
[280]

A{High-Performance} design, implementation, deployment, and evaluation of the slim fly network,

N. Blach, M. Besta, D. De Sensi, J. Domke, H. Harake, S. Li, P. Iff, M. Konieczny, K. Lakhotia, A. Ku- bicek et al., “A{High-Performance} design, implementation, deployment, and evaluation of the slim fly network,” in 21st USENIX Symposium on Networked Systems Design and Imple...

2024
[281]

Design and implementation of mpich2 over infiniband with rdma support,

J. Liu, W. Jiang, P. Wyckoff, D. K. Panda, D. Ashton, D. Buntinas, W. Gropp, and B. Toonen, “Design and implementation of mpich2 over infiniband with rdma support,” in18th International Parallel and Distributed Processing Symposium, 2004. Proceedings.IEEE, 2004, p. 16

2004
[282]

Exploring gpu-to-gpu communication: Insights into supercomputer interconnects,

D. De Sensi, L. Pichetti, F. Vella, T. De Matteis, Z. Ren, L. Fusco, M. Turisini, D. Cesarini, K. Lust, A. Trivediet al., “Exploring gpu-to-gpu communication: Insights into supercomputer interconnects,” in SC24: International Conference for High Performance Computing, Networki...

2024
[283]

A survey and evaluation of data center network topologies,

B. Lebiednik, A. Mangal, and N. Tiwari, “A survey and evaluation of data center network topologies,” arXiv preprint arXiv:1605.01701, 2016. 68 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure May 26, 2025

2016 arXiv
[284]

Portland: a scalable fault-tolerant layer 2 data center network fabric,

R. Niranjan Mysore, A. Pamboris, N. Farrington, N. Huang, P. Miri, S. Radhakrishnan, V. Subramanya, and A. Vahdat, “Portland: a scalable fault-tolerant layer 2 data center network fabric,” SIGCOMM Comput. Commun. Rev., vol. 39, no. 4, p. 39–50, Aug. 2009. [Online]. Available: ...

2009
[285]

Temperature aware workload management in geo-distributed datacenters,

H. Xu, C. Feng, and B. Li, “Temperature aware workload management in geo-distributed datacenters,” SIGMETRICS Perform. Eval. Rev. , vol. 41, no. 1, p. 373–374, Jun. 2013. [Online]. Available: https://doi.org/10.1145/2494232.2465539

2013
[286]

Global data center map

Baxtel, “Global data center map.” [Online]. Available: https://baxtel.com/map
[287]

Global infrastructure regions & azs

AWS, “Global infrastructure regions & azs.” [Online]. Available: https://aws.amazon.com/about-aws/ global-infrastructure/regions_az/
[288]

Global infrastructure

Microsoft, “Global infrastructure.” [Online]. Available: https://azure.microsoft.com/en-us/explore/ global-infrastructure
[289]

What are azure availability zones?

Microsoft, “What are azure availability zones?” [Online]. Available: https://learn.microsoft.com/en-us/ azure/reliability/availability-zones-overview?tabs=azure-cli
[290]

Global (asia, europe, u.s.) - meta data centers

Meta, “Global (asia, europe, u.s.) - meta data centers.” [Online]. Available: https://datacenters.atmeta. com/all-locations/
[291]

Global locations - regions & zones

Google, “Global locations - regions & zones.” [Online]. Available: https://cloud.google.com/about/ locations
[292]

Accelerating collective communication in data parallel training across deep learning frameworks,

J. Romero, J. Yin, N. Laanait, B. Xie, M. T. Young, S. Treichler, V. Starchenko, A. Borisevich, A. Sergeev, and M. Matheson, “Accelerating collective communication in data parallel training across deep learning frameworks,” in19th USENIX Symposium on Networked Systems Design a...

2022
[293]

Exploiting simultaneous communications to accelerate data parallel distributed deep learning,

S. Shi, X. Chu, and B. Li, “Exploiting simultaneous communications to accelerate data parallel distributed deep learning,” inIEEE INFOCOM 2021-IEEE Conference on Computer Communications. IEEE, 2021, pp. 1–10

2021
[294]

Redsync: reducing synchronization bandwidth for distributed deep learning training system,

J. Fang, H. Fu, G. Yang, and C.-J. Hsieh, “Redsync: reducing synchronization bandwidth for distributed deep learning training system,”Journal of Parallel and Distributed Computing, vol. 133, pp. 30–39, 2019

2019
[295]

Synthesizing optimal parallelism placement and reduc- tion strategies on hierarchical systems for deep learning,

N. Xie, T. Norman, D. Grewe, and D. Vytiniotis, “Synthesizing optimal parallelism placement and reduc- tion strategies on hierarchical systems for deep learning,”Proceedings of Machine Learning and Systems, vol. 4, pp. 548–566, 2022

2022
[296]

1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns

F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns.” inInterspeech, vol. 2014. Singapore, 2014, pp. 1058–1062

2014
[297]

Accdp: Accelerated data-parallel dis- tributed dnn training for modern gpu-based hpc clusters,

N. Alnaasan, A. Jain, A. Shafi, H. Subramoni, and D. K. Panda, “Accdp: Accelerated data-parallel dis- tributed dnn training for modern gpu-based hpc clusters,” in2022 IEEE 29th International Conference on High Performance Computing, Data, and Analytics (HiPC), 2022, pp. 32–41

2022
[2017]

Available: https://arxiv.org/abs/1612.02295

[Online]. Available: https://arxiv.org/abs/1612.02295

arXiv
[2018]

[Online]. Available: https://www.sciencedirect.com/science/article/pii/S0743731517303386 66 Compute Can’t Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure May 26, 2025

2025
[2024]

Available: https://community.intel.com/t5/Blogs/Tech-Innovation/Data-Center/ Breaking-the-Memory-Wall-with-Compute-Express-Link-CXL/post/1594848

[Online]. Available: https://community.intel.com/t5/Blogs/Tech-Innovation/Data-Center/ Breaking-the-Memory-Wall-with-Compute-Express-Link-CXL/post/1594848

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Avoiding another AI winter,

J. Hendler, “Avoiding another AI winter,”IEEE Intelligent Systems, vol. 23, no. 02, pp. 2–4, 2008

2008

[2] [2]

A Brief History of AI: How to Prevent Another Winter,

A. Toosi, A. G. Bottino, B. Saboury, E. Siegel, and A. Rahmim, “A Brief History of AI: How to Prevent Another Winter,”PET Clinics, vol. 16, no. 4, pp. 449–469, 2025

2025

[3] [3]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

2017

[4] [4]

Annotated History of Modern AI and Deep Learning,

J. Schmidhuber, “Annotated History of Modern AI and Deep Learning,” 2022. [Online]. Available: https://arxiv.org/abs/2212.11279

arXiv 2022

[5] [5]

ImageNet Classification with Deep Convolutional Neural Networks,

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25. Curran Associates, Inc., 2012. [Online]. Available: https: //proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3...

2012

[6] [6]

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , J. Burstein, C. Doran, and T. Solorio, Ed...

2019

[7] [7]

The Llama 3 Herd of Models,

A. G. et al., “The Llama 3 Herd of Models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

arXiv 2024

[8] [8]

Generative Adversarial Nets,

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurip...

2014