Pith. sign in

REVIEW 3 major objections 6 minor 51 references

A passive optical fiber shuffle turns CXL memory into a 32 TB, switch-free pool that keeps LLM KV caches hot and cuts first-token latency by 6.6× under heavy multi-turn load.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 11:09 UTC pith:Y7KTRBCG

load-bearing objection Solid memory-wall characterization and a real appliance architecture; the 6.6× TTFT number is mostly a capacity story, not proof the optical shuffle wins under load. the 3 major comments →

arxiv 2607.27187 v1 pith:Y7KTRBCG submitted 2026-07-29 cs.PF

A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference

Jing Ding , Yash Nishant , Chandrish Ambati , Jyothsna Kamati , Trung Diep This is my paper
classification cs.PF
keywords photonic interconnectsCXLLLM inferenceKV cachememory disaggregationdatacenter architecturetime-to-first-token
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large-language-model inference hits a memory wall: the key-value (KV) cache needs tens of terabytes at hundreds of gigabytes per second, and no existing tier supplies both. Host DRAM is fast enough to beat recomputation by up to 100× but runs out after tens of long-context users; SSDs have capacity but often lose to recomputation on bandwidth. Electrical CXL pooling could close the gap, yet switch hops, short copper reach, and power scaling stop it at practical TB scale. This paper introduces a photonic-CXL memory appliance that replaces the electrical switch with a passive 16×16 fiber shuffle, giving 16 hosts a full-mesh, switch-free view of 32 TB of shared DDR5 at 128 GB/s each. Emulation shows more than 50 % lower access latency than switched CXL pools; serving simulations show the appliance erases eviction cliffs and holds time-to-first-token flat, delivering a 6.6× improvement at 300 concurrent multi-turn conversations.

Core claim

Replacing electrical CXL switches with a passive optical fiber shuffle yields a practical 32 TB shared-memory appliance that simultaneously supplies the capacity and the >100 GB/s bandwidth required for production-scale KV-cache management, cutting average 64-byte pool latency by more than half versus switched CXL and eliminating the eviction-driven TTFT cliffs that appear once host DRAM is exhausted.

What carries the argument

The Photonic Fabric Memory Appliance: 16 CXL 3.1 Type-3 modules joined by a passive 16×16 fiber shuffle that forms 256 dedicated optical paths (full mesh), so every host reaches every memory module without intermediate switching, arbitration, or retimers.

Load-bearing premise

The latency, bandwidth, and end-to-end serving gains measured on a two-chip emulator and a profile-driven simulator will still hold on a full 16-module physical appliance under real multi-host GPU DMA traffic.

What would settle it

Build or obtain the physical 16-module appliance, run the same multi-turn LLaMA-405B workload at 300 concurrent conversations with production vLLM/SGLang connectors, and check whether mean TTFT stays near the simulated 2.7 s floor or rises once real optical, thermal, and multi-host contention appear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single 8-GPU node plus the appliance can retain KV state for roughly 64× more concurrent long-context or multi-turn sessions than 2 TB host DRAM alone, without eviction.
  • Prefill-to-decode hand-off and cross-host prefix reuse become ordinary load/store operations instead of network RPCs, removing KV-locality-aware scheduling.
  • Rack-scale memory pools become feasible because photonic links remove copper reach limits and switch-hop latency.
  • The same tier can later hold shared model weights or checkpoints, not only KV blocks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the passive-shuffle idea generalizes, other byte-addressable fabrics (not only CXL) could adopt the same optical crossbar to escape electrical switch power walls.
  • Cost and laser-source reliability will decide whether operators prefer one large photonic appliance over simply buying more GPU servers with local DRAM.
  • Once physical hardware exists, the interesting measurement becomes tail latency under simultaneous GPU DMA from all 16 hosts, not just average idle latency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper characterizes KV-cache retrieval across GPU HBM, host DRAM, and SSD on A100/H100/H200 systems with LLaMA models (8B–405B, contexts to 4M tokens), showing host-memory retrieval yields up to 100× speedup over recompute but is capacity-limited, while SSD is bandwidth-limited. It then proposes the Marvell Photonic Fabric (PF) Memory Appliance: a CXL 3.1 Type-3 design that replaces electrical CXL switches with a passive 16×16 fiber shuffle, claiming 32 TB shared DDR5 across 16 hosts at 128 GB/s unidirectional per host with switch-free full-mesh connectivity. Siemens Veloce Strato-M emulation of a PF-NIC plus one PF Memory Module reports idle latencies of 224 ns (write) / 283 ns (read) and ~350 ns average 64-byte access—over 50% lower than Beluga’s published 800–900 ns electrical CXL pool. LLMServingSim end-to-end runs attribute a flat TTFT and 6.6× improvement at 300 multi-turn conversations to the 32 TB tier versus a 2 TB host-DRAM baseline.

Significance. If the architecture delivers as claimed, it would close a real capacity–bandwidth gap for production LLM serving and for rack-scale CXL memory pooling more generally. The empirical characterization (Section II, Table I, Fig. 1) is concrete, multi-platform, and independently useful to the systems community. The passive fiber-shuffle design is a clear architectural alternative to switched electrical CXL, and the paper is unusually candid about pending physical validation (Section VIII). Strengths include cycle-visible emulation with VirtuaLab traces, an explicit latency decomposition (Eq. 4), bandwidth saturation under linear/random 32B–1KB patterns, and a practical software path (DAX, offset addressing, rendezvous consistency, pluggable connectors for vLLM/SGLang/Dynamo/NIXL). These are genuine contributions even before full hardware exists.

major comments (3)
  1. [Section VI, Fig. 8] Section VI and Fig. 8: the headline 6.6× TTFT and “eliminates eviction cliffs” result compares 32 TB PF memory at 128 GB/s against a 2 TB host-DRAM baseline only. For multi-GB KV blocks, transfer time at 128 GB/s is tens of ms; the emulated 350 ns vs Beluga’s 800–900 ns difference is negligible on that path. The flattening is therefore almost pure capacity, not evidence that the passive fiber shuffle outperforms switched CXL under serving load. A same-capacity electrical-CXL (or idealized low-latency pool) arm—or an ablation that holds capacity fixed and varies only interconnect latency/bandwidth—is needed to substantiate the photonic claim in the serving evaluation.
  2. [Section IV; Abstract; Section VIII] Section IV: bandwidth and latency are measured on a two-SoC Veloce configuration (one PF-NIC + one PF Memory Module, QEMU mmap CPU traffic, VirtuaLab CXL transactor). The run does not exercise the 16×16 optical mesh, concurrent multi-host contention on HBM/DDR, GPU DMA through the PF-NIC, or the rendezvous coherence protocol of Section V. Section VIII correctly notes that physical end-to-end validation is pending. The abstract and strongest claims (“over 50% latency reduction,” “16 hosts,” full-mesh) should be scoped explicitly to what the partial emulation actually demonstrates, with multi-host/GPU-DMA results marked as projection until hardware exists.
  3. [Section II-C, III-C; related work on Beluga/TraCT] Section II-C and III-C motivate the PF design by electrical CXL switch latency (70–500 ns/hop), reach, and power. Beluga already reports 8 TB / 16 hosts on a CXL 2.0 switch. The paper never quantifies how much of the serving gap would remain if an electrical pool of comparable capacity (e.g., 32 TB via hierarchical switches) were available, nor does it give switch-hierarchy latency/power numbers at that scale. Without that comparison, the claim that electrical CXL “prevent[s] practical TB-scale deployments” remains qualitative relative to the proposed alternative.
minor comments (6)
  1. [Table I] Table I: disk results are sparse (only 8B on A100/H200 and 405B); several cells are “—”. Either complete the matrix or state clearly why disk was omitted for intermediate models.
  2. [Section II-A, Eqs. (1)–(3), Fig. 1] Eqs. (1)–(3): MFU, K, and α are not fitted or tabulated; the super-linear speedup discussion would be stronger with measured vs predicted S for the 405B curves in Fig. 1.
  3. [Fig. 6, Section IV-C] Fig. 6: loaded-latency histograms lack sample counts, exact credit limits, and whether the distribution is over one or many runs; add these for reproducibility.
  4. [Section V] Section V: the rendezvous consistency protocol is described at a high level but without failure/timeout, multi-producer, or partial-write behavior. A short pseudocode block or state machine would help implementers.
  5. Typos/clarity: “pr event” (Conclusion); “Pytoch” / “heterogenous” (Section V); “super linear” vs “super-linear”; abstract “full- crossbar” spacing; “CXL pod” (IV-C) is undefined.
  6. [References] References [1]–[4] cite 2025 model names (GPT-5.4, Gemini 3, LLaMA 4) that may not be stable archival sources; prefer technical reports with fixed identifiers where possible.

Circularity Check

0 steps flagged

No derivation circularity: empirical baselines, external Beluga comparison, and forward simulation from stated capacity/latency knobs.

full rationale

This is a systems architecture paper, not a first-principles derivation. The load-bearing quantitative claims do not reduce to their inputs by construction. (1) Host-memory and disk KV retrieval speedups (Table I, Fig. 1) are measured against recompute and SSD baselines on real A100/H100/H200 hardware; the speedup formulas (Eqs. 1–3) are accounting identities over measured T_compute and T_memory, not fitted parameters renamed as predictions. (2) The >50% latency claim compares emulated PF round-trip latency (idle ~224–283 ns; loaded ~350 ns average cited vs Beluga) to Yang et al.’s independently published electrical CXL pool numbers (800–900 ns), an external benchmark. (3) The 6.6× TTFT result (Section VI, Fig. 8) is a forward run of third-party LLMServingSim given explicit capacity (2 TB vs 32 TB) and bandwidth/latency knobs from Section IV; the simulator is not fit to the target TTFT and then used to ‘predict’ it. Self-citations [17, 18, 21] describe photonic component choices (EAMs, fiber shuffle) but do not supply a uniqueness theorem or force the serving metric. Gaps noted by the skeptic—partial two-SoC emulation, capacity confound vs same-size electrical CXL, pending physical validation (Section VIII)—are external-validity/correctness issues, not circular reduction of outputs to inputs. Score 0; steps empty.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 2 invented entities

The central performance claims rest on (1) standard CXL/PCIe and LLM serving assumptions, (2) the premise that a passive optical full-mesh plus HBM cache will deliver the emulated latency and full link bandwidth at 16-host scale, and (3) that LLMServingSim driven by those parameters predicts real multi-turn TTFT. No numerical free parameters are fitted to force the 6.6× result; the large invented entity is the PF Memory Appliance itself as a productized system.

free parameters (2)
  • PF address-interleave configuration bits = unspecified fixed set enabling 25.6 GB/s per port
    Chosen so traffic spreads uniformly across DDR/HBM channels to claim 100% link utilization; described as a fixed optimized set but not derived from first principles.
  • HBM direct-mapped cache sizing / use as write-through vs pooled memory = 36 GB × 2 stacks per module; policy configurable
    1.125 TB HBM3E is provisioned with ‘significantly higher bandwidth than the photonic link’; exact cache policy and hit-rate assumptions affect whether DDR5 becomes the bottleneck under real KV patterns.
axioms (5)
  • domain assumption CXL.mem load/store semantics plus GPU DMA (cudaHostRegister / cudaMemcpyAsync) suffice for correct multi-host KV sharing with only a rendezvous-based software consistency protocol on non-coherent memory.
    Section V; follows TraCT/Beluga-style designs but is required for the software integration story.
  • domain assumption Passive fiber shuffle provides contention-free, deterministic optical paths with distance-independent power and no switch arbitration delay at 16×16 full mesh.
    Section III; core architectural premise contrasting electrical CXL switches.
  • domain assumption LLMServingSim profile-driven operator model (1.9% average error cited) plus emulated tier bandwidth/latency accurately predicts production multi-turn TTFT and prefix hit rates.
    Section VI; load-bearing for the 6.6× claim.
  • standard math PCIe Gen6 / CXL 3.1 host interface caps a single host at 128 GB/s unidirectional into the appliance.
    Stated interface constraint used throughout bandwidth discussion.
  • domain assumption GeSi electro-absorption modulators plus 2.5D PIC/ASIC co-packaging deliver required thermal stability, bandwidth density, and energy without MRR tuning or long-reach DSP.
    Section III.B; component-level premise for power and reach advantages.
invented entities (2)
  • Marvell Photonic Fabric (PF) Memory Appliance / PF Memory Module / PF-NIC no independent evidence
    purpose: Concrete 32 TB, 16-host, switch-free photonic-CXL memory pool product that is the object of emulation and simulation claims.
    The paper’s primary artifact; independent physical evidence is not yet available (validation pending per Section VIII).
  • Rendezvous-based cross-host consistency protocol over DAX-mapped CXL region no independent evidence
    purpose: Provide ordering for KV metadata without per-cache-line flushes on every write.
    Described in Section V as the multi-host coherence mechanism; not evaluated for correctness or performance under contention.

pith-pipeline@v1.2.0-daily-grok45 · 21474 in / 3638 out tokens · 57919 ms · 2026-07-30T11:09:34.096838+00:00 · methodology

0 comments
read the original abstract

LLM inference at scale faces a memory wall. The KV cache demands tens of terabytes at hundreds of gigabytes per second, yet no current memory tier delivers both at once. Characterization across multi-generation GPU systems with various LLaMA models shows host memory retrieval achieves up to 100x speedup over re-computation but supports only tens of concurrent long-context users. Electrical CXL pooling theoretically bridges this gap, but switch latency, cable reach limits, and power-scaling issues prevent practical TB-scale deployments. We present the Marvell Photonic Fabric Memory Appliance, a photonic-CXL hybrid architecture replacing electrical switches with a passive fiber shuffle to deliver 32 TB shared memory across 16 hosts via a switch-free full- crossbar topology. Emulation results demonstrate over 50 percent latency reduction versus electrical CXL pools. Simulation results show that the PF Memory Appliance eliminates cache eviction cliffs by improving time-to-first-token by 6.6x for multi-turn conversations workloads.

Figures

Figures reproduced from arXiv: 2607.27187 by Chandrish Ambati, Jing Ding, Jyothsna Kamati, Trung Diep, Yash Nishant.

Figure 1
Figure 1. Figure 1: KV retrieval from host memory speedup compared to GPU re￾computation versus (a) Prompt Length and (b) Batch size for LLaMA 405B model on eight H200 systems. Disk storage performance. Disk-based caching diverges significantly from host memory patterns. For LLaMA-8B on the A100 system, disk retrieval provides speedups only for sequences exceeding 50K tokens, with shorter contexts actually showing performance… view at source ↗
Figure 2
Figure 2. Figure 2: Evolution of KV cache transfer data paths in disaggregated LLM inference. Solid lines represent the KV data plane; dashed lines represent CPU control operations. Each generation removes components from the critical path: (a)→(b) eliminates DRAM bounces; (b)→ (c,d) eliminates the network. CXL modes additionally enable write-once-read-many KV reuse and eliminate KV￾locality-aware scheduling. CXL shared memor… view at source ↗
Figure 4
Figure 4. Figure 4: PF Memory Appliance server connectivity via Photonic Fabric NIC (PF-NIC) over PCIe/CXL interface. B. PF Memory Module Design The advantage of the PF Memory Module compared to other photonic interconnect solutions lies in its thermal stability, bandwidth density, and energy efficiency, achieved through careful selection of photonic components and co-packaging strategies [21]. The PF Memory Module employs Ge… view at source ↗
Figure 3
Figure 3. Figure 3: PF Memory Appliance architecture: 256 optical paths creating full￾mesh connectivity connecting up to 16 host servers without intermediate switching. The passive fiber shuffle represents a fundamental departure from traditional switched architectures. Rather than routing memory transactions through active switch elements that introduce arbitration delays and potential congestion, the fiber shuffle provides … view at source ↗
Figure 6
Figure 6. Figure 6: Latency distribution under loaded conditions for (a) Write and (b) Read operations. Under the idle latency configuration, we measure the latency components defined in Eq. (2). The memory access component, Tmem, is measured at 153ns for write operations and 212 ns for read operations, reflecting the expected asymmetry due to data return overhead in reads. To ensure stability, we repeat the experiment multip… view at source ↗
Figure 5
Figure 5. Figure 5: Block diagram of the latency breakdown. To quantify the remaining components, we rely on VirtuaLab protocol analyzer traces, which record timestamps at the RC. These timestamps are correlated with waveform signals captured at the CXL protocol interface and at the point where the request exits PF-NIC to extract precise timing points along the request path. Using these measurements, the round-trip latency is… view at source ↗
Figure 7
Figure 7. Figure 7: Software integration approach of PF Memory Appliance with various ML Frameworks Inference framework integration. The PF Memory Appliance can integrate with vLLM, SGLang, and NVIDIA Dynamo through their existing pluggable interfaces, without modifications to engine internals. For vLLM, we will implement a CXL KV connector whose store() and load() methods issue GPU-to-PF Memory Appliance and PF Memory Applia… view at source ↗
Figure 8
Figure 8. Figure 8: (a) shows mean TTFT as the total token working set scales from 50 conversations to 300 conversations. With the 32 TB PF Memory Appliance, TTFT remains flat at 2,690 ms regardless of conversation count—all KV cache entries are retained in the pooled memory, and subsequent turns benefit from full prefix cache hits (82% hit rate). In contrast, the 2 TB baseline degrades rapidly: at 100 conversations, mean TTF… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 5 canonical work pages

  1. [1]

    Claude model family,

    Anthropic, "Claude model family," 2025. [Online]. Available: https://www.anthropic.com/claude

    2025

  2. [2]

    Gemma 3 Technical Report,

    Google DeepMind, " Gemma 3 Technical Report," 2025. [Online]. Available: https://arxiv.org/abs/2503.19786

    Pith/arXiv arXiv 2025

  3. [3]

    GPT-5.4,

    OpenAI, "GPT-5.4," 2025. [Online]. Available: https://chatgpt.com/overview?openaicom_referred=true

    2025

  4. [4]

    LLaMA 4 model family,

    Meta, "LLaMA 4 model family," 2025. [Online]. Available: https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  5. [5]

    How to reduce KV cache bottlenecks with NVIDIA Dynamo,

    NVIDIA, "How to reduce KV cache bottlenecks with NVIDIA Dynamo," NVIDIA Technical Blog, 2025. [Online]. Available: https://developer.nvidia.com/blog/how-to-reduce-kv-cache-bottlenecks- with-nvidia-dynamo/

    2025

  6. [6]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with PagedAttention," in Proc. ACM SOSP, 2023. https://doi.org/10.1145/3600006.3613165

    arXiv 2023

  7. [7]

    LMCache: An efficient KV cache layer for enterprise-scale LLM inference,

    Y. Liu, H. Li, Y. Du, S. Song, J. Zhu, J. Peng, S. Chen, K. Xiong, and J. Lu, "LMCache: An efficient KV cache layer for enterprise-scale LLM inference," arXiv:2510.09665, 2025. https://doi.org/10.48550/arXiv.2510.09665

    doi:10.48550/arxiv.2510.09665 2025

  8. [8]

    SGLang HiCache: Fast hierarchical KV caching with storage backends,

    LMSYS, "SGLang HiCache: Fast hierarchical KV caching with storage backends," 2025. [Online]. Available: https://lmsys.org/blog/2025-09- 10-sglang-hicache/

    2025

  9. [9]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, "DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving," in Proc. USENIX OSDI, 2024. https://doi.org/10.48550/arXiv.2401.09670

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.09670 2024

  10. [10]

    Splitwise: Efficient generative LLM inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini, "Splitwise: Efficient generative LLM inference using phase splitting," in Proc. IEEE/ACM ISCA, 2024. https://doi.org/10.1109/ISCA59077.2024.00019

    arXiv 2024

  11. [11]

    Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot,

    R. Qin et al., "Mooncake: Trading more storage for less computation—a KVCache-centric architecture for serving LLM chatbot," ACM Trans. Storage, 2025. https://doi.org/10.1145/3773772

    doi:10.1145/3773772 2025

  12. [12]

    GPUDirect RDMA documentation

    NVIDIA, "GPUDirect RDMA documentation." [Online]. Available: https://docs.nvidia.com/cuda/gpudirect-rdma/

  13. [13]

    NIXL: NVIDIA Inference Transfer Library,

    NVIDIA, "NIXL: NVIDIA Inference Transfer Library," 2025. [Online]. Available: https://github.com/ai-dynamo/nixl

    2025

  14. [14]

    Compute Express Link Specification 3.1,

    CXL Consortium, "Compute Express Link Specification 3.1," 2024. [Online]. Available: https://www.computeexpresslink.org/download-the- specification

    2024

  15. [15]

    TraCT: Disaggregated LLM serving with CXL shared memory KV cache at rack-scale,

    J. Yoon et al., "TraCT: Disaggregated LLM serving with CXL shared memory KV cache at rack-scale," arXiv:2512.18194, 2025. https://doi.org/10.48550/arXiv.2512.18194

    doi:10.48550/arxiv.2512.18194 2025

  16. [16]

    Beluga: A CXL-based memory architecture for scalable and efficient LLM KVCache management,

    Z. Yang et al., "Beluga: A CXL-based memory architecture for scalable and efficient LLM KVCache management," arXiv:2511.20172, 2025. https://doi.org/10.48550/arXiv.2511.20172

    doi:10.48550/arxiv.2511.20172 2025

  17. [17]

    A 56-Gb/s hybrid silicon photonic and 5-nm CMOS 3-D-integrated transceiver for optical compute I/O,

    G. Balamurugan et al., "A 56-Gb/s hybrid silicon photonic and 5-nm CMOS 3-D-integrated transceiver for optical compute I/O," IEEE J. Solid-State Circuits, 2026

    2026

  18. [18]

    Photonic Fabric for memory and compute disaggregation,

    S. Sahni et al., "Photonic Fabric for memory and compute disaggregation," in Proc. OFC, paper W3D.1, 2025. https://doi.org/10.1364/OFC.2025.W3D.1

    work page doi:10.1364/ofc.2025.w3d.1 2025

  19. [19]

    Benchmarking GPUDirect RDMA on modern server platforms,

    S. Potluri, K. Hamidouche, D. Bureddy, and D. K. Panda, "Benchmarking GPUDirect RDMA on modern server platforms," NVIDIA Technical Blog, 2014. [Online]. Available: https://developer.nvidia.com/blog/benchmarking-gpudirect-rdma-on- modern-server-platforms/

    2014

  20. [20]

    Distributed KV cache scheduling and offloading,

    llm-d Project, "Distributed KV cache scheduling and offloading," 2025. [Online]. Available: https://llm-d.ai

    2025

  21. [21]

    Photonic Fabric: Co-packaged photonic interconnect technology,

    P. Winterbottom et al., "Photonic Fabric: Co-packaged photonic interconnect technology," Celestial AI, 2025

    2025

  22. [22]

    CXL Type 3 Memory Device Software Guide,

    Intel, "CXL Type 3 Memory Device Software Guide," Document 643805, Rev 1.1. [Online]. Available: https://www.intel.com/content/www/us/en/content-details/643805/

  23. [23]

    KV Connector API

    vLLM Project, "KV Connector API." [Online]. Available: https://docs.vllm.ai

  24. [24]

    Inside vLLM's new KV offloading connector,

    vLLM Blog, "Inside vLLM's new KV offloading connector," Jan. 2026. [Online]. Available: https://blog.vllm.ai

    2026

  25. [25]

    CXL-attached memory allocation for long-context LLM fine-tuning,

    S. Seo et al., "CXL-attached memory allocation for long-context LLM fine-tuning," arXiv:2507.03305, 2025. https://doi.org/10.48550/arXiv.2507.03305

    work page doi:10.48550/arxiv.2507.03305 2025

  26. [26]

    Intra-device heterogeneous memory allocation support,

    PyTorch, "Intra-device heterogeneous memory allocation support," RFC #153745, 2025. [Online]. Available: https://github.com/pytorch/pytorch/issues/153745

    2025

  27. [27]

    LLMServingSim: A runtime-driven HW/SW co-simulation framework for LLM inference serving,

    S. Cho, H. Choi, G.-I. Kwon, L. Kim, and M. Rhu, "LLMServingSim: A runtime-driven HW/SW co-simulation framework for LLM inference serving," in Proc. IEEE IISWC, 2024; extended in IEEE Comput. Archit. Lett., 2025. https://doi.org/10.1109/IISWC63097.2024.10763697

    arXiv 2024

  28. [28]

    SGLang: Efficient execution of structured language model programs,

    L. Zheng et al., "SGLang: Efficient execution of structured language model programs," arXiv:2312.07104, 2024. https://doi.org/10.48550/arXiv.2312.07104

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.07104 2024

  29. [29]

    CacheBlend: Fast large language model serving for RAG with cached knowledge fusion,

    J. Yao, H. Li, Y. Liu, S. Song, K. Zhang, Y. Xiong, and J. Lu, "CacheBlend: Fast large language model serving for RAG with cached knowledge fusion," in Proc. ACM EuroSys, 2025. https://doi.org/10.1145/3689031.3696098

    arXiv 2025

  30. [30]

    Pond: CXL-based memory pooling systems for cloud platforms,

    H. Li et al., "Pond: CXL-based memory pooling systems for cloud platforms," in Proc. ACM ASPLOS, 2023. https://doi.org/10.1145/3575693.3578835

    arXiv 2023

  31. [31]

    InfiniGen: Efficient generative inference of large language models with dynamic KV cache management,

    J. Lee et al., "InfiniGen: Efficient generative inference of large language models with dynamic KV cache management," in Proc. USENIX OSDI,

  32. [32]

    KVQuant: Towards 10 million context length LLM inference with KV cache quantization,

    C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and A. Gholami, "KVQuant: Towards 10 million context length LLM inference with KV cache quantization," in Proc. NeurIPS,

  33. [33]

    FlashAttention: Fast and memory-efficient exact attention with IO-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, "FlashAttention: Fast and memory-efficient exact attention with IO-awareness," in Proc. NeurIPS, 2022. https://doi.org/10.48550/arXiv.2205.14135

    Pith/arXiv arXiv doi:10.48550/arxiv.2205.14135 2022

  34. [34]

    https://doi.org/10.48550/arXiv.2401.18079

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.18079

  35. [35]

    DirectCXL: Direct memory access for CXL-enabled memory disaggregation,

    D. Gouk, S. Lee, M. Kwon, and M. Jung, "DirectCXL: Direct memory access for CXL-enabled memory disaggregation," in Proc. USENIX ATC, 2022. [Online]. Available: https://www.usenix.org/conference/atc22/presentation/gouk

    2022

  36. [36]

    FlashAttention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, "FlashAttention-2: Faster attention with better parallelism and work partitioning," in Proc. ICLR, 2024. https://doi.org/10.48550/arXiv.2307.08691

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.08691 2024

  37. [37]

    Melody: Systematic CXL memory characterization and performance analysis at scale,

    J. Liu et al., "Melody: Systematic CXL memory characterization and performance analysis at scale," in Proc. ACM ASPLOS, 2025. https://doi.org/10.1145/3676641.3715987

    arXiv 2025

  38. [38]

    TPP: Transparent page placement for CXL-enabled tiered memory,

    H. Maruf, H. Wang, A. Dhakal, H. Li, Z. Cai, M. Mosharaf, and K. Akeley, "TPP: Transparent page placement for CXL-enabled tiered memory," in Proc. ACM ASPLOS, 2023. https://doi.org/10.1145/3582016.3582063

    arXiv 2023

  39. [39]

    RCMP: Reconstructing RDMA-based memory disaggregation via CXL,

    Z. Wang et al., "RCMP: Reconstructing RDMA-based memory disaggregation via CXL," ACM Trans. Archit. Code Optim., 2024. https://doi.org/10.1145/3634916

    work page doi:10.1145/3634916 2024

  40. [40]

    COAXIAL: A CXL-centric memory system for scalable servers,

    A. Cho et al., "COAXIAL: A CXL-centric memory system for scalable servers," in Proc. IEEE/ACM SC, 2024. https://doi.org/10.1109/SC41406.2024.00101

    Pith/arXiv arXiv 2024

  41. [41]

    Tomahawk 6 (Davisson) co-packaged optics switch,

    Broadcom, "Tomahawk 6 (Davisson) co-packaged optics switch," 2025. [Online]. Available: https://www.broadcom.com/products/ethernet- connectivity/switching/strataxgs/bcm78900

    2025

  42. [42]

    Scaling out data parallel applications with CXL-Ethernet hybrid interconnects,

    DFabric, "Scaling out data parallel applications with CXL-Ethernet hybrid interconnects," arXiv:2409.05404, 2024. https://doi.org/10.48550/arXiv.2409.05404

    work page doi:10.48550/arxiv.2409.05404 2024

  43. [43]

    Monolithic electro-optic platform on silicon with bandwidth of 100 GHz and beyond,

    L. Steckler et al., "Monolithic electro-optic platform on silicon with bandwidth of 100 GHz and beyond," Nature Communications, Nov

  44. [44]

    Optical Compute Interconnect (OCI),

    Intel, "Optical Compute Interconnect (OCI)," demonstrated at OFC,

  45. [45]

    Available: https://www.intel.com/content/www/us/en/research/optical-compute- interconnect.html

    [Online]. Available: https://www.intel.com/content/www/us/en/research/optical-compute- interconnect.html

  46. [46]

    TeraPHY: 3rd generation UCIe optical chiplet,

    Ayar Labs, "TeraPHY: 3rd generation UCIe optical chiplet," presented at OFC, 2025. [Online]. Available: https://ayarlabs.com/teraphy/

    2025

  47. [47]

    A new era in data center networking with NVIDIA silicon photonics-based network switching,

    NVIDIA, "A new era in data center networking with NVIDIA silicon photonics-based network switching," NVIDIA Technical Blog, 2025. [Online]. Available: https://developer.nvidia.com/blog/a-new-era-in- data-center-networking-with-nvidia-silicon-photonics-based-network- switching/

    2025

  48. [48]

    Beyond-110 GHz C-band GeSi electro-absorption modulator on 300 mm silicon photonics,

    imec, "Beyond-110 GHz C-band GeSi electro-absorption modulator on 300 mm silicon photonics," presented at ECOC, 2025

    2025

  49. [49]

    Passage M1000 photonic superchip,

    Lightmatter, "Passage M1000 photonic superchip," 2025. [Online]. Available: https://lightmatter.co/products/passage/

    2025

  50. [2024]

    Available: https://www.usenix.org/conference/osdi24/presentation/lee

    [Online]. Available: https://www.usenix.org/conference/osdi24/presentation/lee

  51. [2025]

    https://doi.org/10.1038/s41467-025-66566-2

    work page doi:10.1038/s41467-025-66566-2