Pith. sign in

REVIEW 5 major objections 5 minor 84 references

PLoRA: An NDP-Enhanced Pooled-Memory System for Cost-Efficient Multi-LoRA Serving

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-LoRA serving can be served cost-efficiently out of pooled memory with near-data compute, taking the link off the critical path.

desk verdict A genuine new design for NDP+pooled-memory multi-LoRA serving, worth reading, but the headline numbers are simulator extrapolations until the NDP core timing is validated on real hardware or released as an artifact. read the letter →

arxiv 2608.05483 v1 pith:63NAJU5J submitted 2026-08-06 cs.AR cs.DC

Zhongkai Yu , Ohm Rishabh Venkatachalam , Zheng Wang , Yikai Li , Yichen Lin , Zihao Yu , Yuke Wang , Liu Liu
show 4 more authors
Xulong Tang Shuyi Pei Yangwook Kang Yufei Ding
This is my paper · ORCID
classification cs.ARcs.DC
keywords multi-LoRAservingpooledmemorynear-dataprocessingCXLKVcacheoffloadLLMinferencestrategyselectioncostmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-LoRA serving turns one base model into thousands of specialized variants, and the adapters plus their KV caches overflow GPU memory; every published system stages them from CPU DRAM over PCIe, paying a kernel stop and a host copy on each access. PLoRA claims that putting those adapters and KV caches in memory-semantic pooled memory and adding near-data compute beside the pool removes the link from the critical path. The paper builds a device where the GPU drives read-compute requests with ordinary loads and stores, a cost-model-driven selector chooses one of four LoRA strategies and one of two attention strategies per adapter, and only reduced results cross the link. On one H100 serving 1000 adapters, it reports the lowest decode latency on every workload measured, averaging 6.6x below a real-machine S-LoRA at under 3.4% added device area. If correct, cost-efficient multi-LoRA serving needs far less interconnect and far less host machinery than current systems assume.

What carries the argument

The load-bearing mechanism is the read-compute request: a 204-bit descriptor that fits in one CXL 3.1 flit and extends the load/store interface with a command such as GEMM, GEMV, or softmax, telling the NDP core to combine GPU-supplied operands with device-resident data and return only the reduced result. Around it sit an AXI network-on-chip joining the CXL.mem and CXL.io interfaces, DRAM controllers, and the NDP core; four LoRA execution strategies (E1-E4) and three data placements trading GPU memory against bytes crossing the link; a distributed attention scheme that partitions each request's KV cache across devices and uses FlashAttention-style partial softmax so intermediate results stay off the link; and a link-parameterized cost model that ranks strategies under a GPU memory budget. The exchange rate that makes the whole design work is the operator's reduction ratio, set by the matrix dimensions, not by the fabric's speed.

What would settle it

Build a PLoRA device with the described NDP core (2 TFLOPS per device, 1.1 TB/s DRAM, 512 GB) behind a CXL 3.1 link on one H100 and measure decode TPOT on Llama2-7B Uniform at batch 32. If throughput keeps improving as the link is widened past 64 GB/s, or if the decode step time falls short of the 4.18 ms HBM-streaming floor, the plateau claim and the 6.6x average are falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the memory-hungry part of multi-LoRA inference - adapter matrices and KV cache, which take 97.8% of memory but only 3.0% of computation - is also the most reducible part, so computing it beside the data converts a link-bandwidth problem into a capacity problem. PLoRA keeps adapters and KV cache in pooled memory, lets the GPU address them with load/store semantics, and returns only reduced results (for example, the attention product $QK^T$ shrinks by a factor $D \ge 4096$ before crossing the link). A GPU memory manager chooses among four LoRA execution strategies and two attention strategies under a memory budget, using a cost model that touches the fabric only through its bandwidth and latency. The reported consequence is that decode throughput saturates at 32-64 GB/s, well below the assumed 128 GB/s CXL 3.1 link, and on short contexts a decode step takes exactly the time to stream the base-model weights from HBM - the floor a GPU with unlimited local memory would also hit.

Load-bearing premise

All headline numbers come from an event-driven simulator whose timing parameters are calibrated on a real CXL memory device without NDP and on GPU utilization measurements, not from a working PLoRA prototype; if that calibration does not accurately predict NDP-core timing, DRAM contention, and CXL-link behavior in the proposed device, the 6.6x average and the bandwidth plateau do not transfer to hardware.

Editorial extensions

If this is right

  • A single accelerator with a CXL-class pool plus near-data compute can serve 1000-plus adapters at lower decode latency than a real-machine host-staged system, averaging 6.6x and reaching 13.1x on the models measured.
  • Link bandwidth beyond roughly 32-64 GB/s stops improving decode throughput, because the step time becomes the time to stream the base model from HBM; a quarter of a CXL 3.1 link is already enough.
  • Replacing link bandwidth with near-data compute costs under 3.4% added device area and about 7.7% power, while a no-NDP system would need 95x to 145x a CXL 3.1 link to match it on long contexts.
  • In tensor-parallel deployments, sharding the adapter wave together with the model makes per-GPU pooled-memory bandwidth demand fall as the model grows, down to 8-16 GB/s per GPU at 1.2T parameters.
  • The same design transfers unchanged to faster memory-semantic fabrics, where the extra bandwidth does not add speed but doubles the usable pool size from 32 to 64 devices before aggregation traffic collapses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the read-compute pattern generalizes beyond LoRA serving: any high-reduction, memory-bound operator whose operands already sit in the pool, such as large embedding lookups or batched GEMV in recommender systems, could use the same interface and cost-model machinery.
  • If the plateau holds, the editorial consequence is that CXL-class bandwidth is already over-provisioned for this workload class, so memory-semantic fabric roadmaps should spend silicon on capacity and on-pool compute rather than raw link speed.
  • A testable extension would stress the strategy selector with rank and batch-size distributions concentrated at intermediate values, where E2 and E4 boundaries shift, to see whether the reported 10-59% advantage over fixed strategies persists outside the paper's uniform and skewed workloads.
  • The headline numbers should be read as conditional on simulator calibration: an end-to-end prototype on a real CXL+NDP device would be the natural confirmation, since the timing parameters come from an NDP-less hardware configuration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes PLoRA, an NDP-enhanced pooled-memory system for serving many concurrent LoRA adapters from a memory-semantic fabric such as CXL, with the goal of keeping adapters and KV cache in the pool and returning only reduced results to the GPU. The architecture adds an NDP core to a CXL Type-3 memory expander and exposes a read-compute interface driven by GPU load/store semantics. A GPU-side memory manager chooses among four LoRA execution strategies (E1-E4) and two attention strategies using a link-parameterized cost model and an eviction-based budget allocator (Algorithm 1). The evaluation uses an event-driven simulator calibrated on GPU utilization measurements and on a Samsung CMM-D NDP-less CXL device, plus a Verilog synthesis of the NDP core. The paper claims lowest decode latency on all tested workloads, averaging 6.6x below a real-machine S-LoRA, with under 3.4% added device area, link saturation at 32-64 GB/s, and favorable scaling to a modeled 1.2T-parameter tensor-parallel deployment.

Significance. If the performance results hold, PLoRA is a significant contribution: it articulates a concrete architecture and software stack for serving LoRA adapters from pooled memory, and its central finding---that near-data execution removes the link from the critical path for multi-LoRA decode---would be an important data point for CXL and NVLink-class pooled-memory design. The paper has real strengths: the cost model is deliberately parameterized only by link bandwidth and latency, making the design fabric-agnostic; the strategy space (E1-E4, distributed attention) is well motivated by the batch-size dynamics and the memory/compute inversion of multi-LoRA serving; the Verilog synthesis of the NDP core gives a credible area/power estimate; and the Section VIII scope caveats are unusually honest. These strengths do not, however, overcome the lack of validation of the NDP timing model, which is load-bearing for the headline quantitative claims.

major comments (5)
  1. [Section VII-A and Section VIII, Findings 1-2] The headline claims---6.6x average speedup over S-LoRA, saturation at 32-64 GB/s, and reaching the HBM floor---rest on an event-driven simulator whose NDP core timing is not validated. The paper states in Section VII-A: 'We calibrate on real hardware: GPU utilization across batch sizes gives compute time, and memory access patterns on Samsung CMM-D, an NDP-less CXL device, give timing parameters.' That calibration covers neither NDP core compute throughput, nor read-compute request handling, nor DRAM bank/refresh contention under NDP traffic, nor AXI NoC arbitration, nor SFU behavior. The Verilog synthesis in Section VII-H validates area and power but is not connected to the simulator's timing. The sensitivity study in Fig. 10(a) shows the plateau is robust to moderate changes in NDP throughput, but it does not validate the absolute level: if real GEMV/softmax throughput is several times lower due to pipeline stalls or contention, E3/E4 become slower, the strategy selector shifts work back to E1/E2, link traffic rises, and the 32 GB/s saturation point moves toward or beyond the CXL link. Please provide an end-to-end validation (RTL-in-the-loop, FPGA prototype, or at minimum a detailed microbenchmark-based validation of the NDP timing model on a CXL-PNM-class device), and release the simulator artifact so the community can reproduce the extrapolation.
  2. [Section VI-B, Table V] The attention device transfer term is written as '2Llink + 2D×B×S / Wlink', with no dependence on Ndev. In the distributed attention scheme of Section V-B, each of the Ndev devices computes a partial output O_i of size D×B×S and returns it to the GPU, so the total transfer should grow with Ndev (at least Ndev×D×B×S, plus any Q distribution). This omission directly contradicts Finding 3 in Section VIII, which explains the throughput collapse at 64 devices as caused by 'each device returns one partial result for the GPU to aggregate, so aggregation traffic grows with the pool while the link does not.' As written, the cost model underestimates attention transfer for Ndev > 2, which can mis-rank the strategies in Algorithm 1. Please correct the formula, state whether the simulator uses the corrected version, and re-run the strategy-selection and device-count results (including Fig. 13(c)) with the corrected cost model.
  3. [Section VI-B, Table IV] The E3 and E4 device-side terms divide by Ndev, e.g., '4Σ B_i3 D R_i / (Cdev×Ndev)' and the corresponding DRAM term. However, Section V-A and Fig. 6 describe each adapter's A and B as stored in the pool and computed by 'a device NDP core'; the text does not state that a single adapter's matrices are sharded across multiple PLoRA devices. If each adapter is placed on one device, then a hot adapter with a large batch is limited by that device's Cdev and W_DRAM, not by the aggregate Ndev times those values. The sum-over-adapters divided by Ndev is a load-balancing assumption that is not stated or justified. Please specify the placement and sharding of adapter matrices, and either use a max over per-device load in the cost model or demonstrate that the assumed perfect balance holds in the simulator.
  4. [Section VII-B and Fig. 8] The headline comparison of PLoRA against S-LoRA mixes a real-machine measurement (S-LoRA on two models) with simulated PLoRA and simulated baselines. The paper explicitly says 'S-LoRA is a real-machine measurement and appears only for the two models we could run,' so the 6.6x average is not an apples-to-apples comparison. The unvalidated components of the simulator---NDP timing, read-compute overhead, kernel-launch behavior---are exactly the components that could make PLoRA look artificially fast. Please run S-LoRA inside the same simulator framework (at least for a configuration where PLoRA reduces to S-LoRA-like GPU-side execution), or validate PLoRA on real hardware for a small end-to-end configuration, and report the simulator's error against real S-LoRA on that configuration.
  5. [Section VIII, Finding 5 and Fig. 12] The 1.2T-parameter scaling projection is not an independent validation. The text says the tensor-parallel model's 'link coefficients are fitted from the simulated ladders, reproducing them within 1% on the reference model.' Fitting coefficients to the simulated ladders and then reusing those coefficients to predict the 30B-1.2T points is circular in the sense that the extrapolation carries the simulator's assumptions forward without new evidence. The 'sharded TP' conclusion---per-GPU demand falls to 8-16 GB/s at 1.2T---also depends on the unvalidated assumption that adapter waves are sharded with the tensor parallelism (row-parallel A, R-wide all-reduce, column-parallel B). Please validate the tensor-parallel model against at least one real multi-GPU deployment, or provide analytical bounds that do not reuse simulator-fitted parameters.
minor comments (5)
  1. [Fig. 8 caption] The caption should state explicitly which bars are real-machine measurements and which are simulated; currently that information appears only in the body text.
  2. [Algorithm 1, line 8] The condition 'if Size(coldAdapters) > (M em − memLimit)' contains a typo ('M em') and the variable naming is hard to follow; please rename for clarity.
  3. [Section III-A, Table III] The memory percentages sum to 100.1% due to rounding; please adjust the displayed values.
  4. [Section VII-C, Table VIII] The row sharing between Llama2-7B and Llama3-8B is explained, but the caption should state that the prefill model ignores the FFN difference, since this is a nontrivial simplification.
  5. [Section IV-C] The phrase 'fits one 256-Byte CXL 3.1 flit' is confusing because a 204-bit descriptor is far smaller than a 256-byte flit; please rephrase to 'fits within one flit' or similar.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: PLoRA's headline results rest on an externally calibrated event-driven simulator and an independent Verilog synthesis, not on definitions or self-citations.

full rationale

Walking the paper's derivation chain, the central claims are not circular. The architectural position (Section III-A) follows from the reduction ratio of the operators, not from the conclusions. The cost model (Section VI-B) is used only to rank execution strategies; the evaluation itself uses an event-driven simulator that is calibrated on real hardware and on an NDP-less CXL device (Section VII-A: 'We calibrate on real hardware: GPU utilization across batch sizes gives compute time, and memory access patterns on Samsung CMM-D, an NDP-less CXL device, give timing parameters'). The headline '6.6x below S-LoRA' compares a simulated PLoRA against a real-machine S-LoRA measurement, so it is a simulated measurement against an external baseline rather than a fitted identity. The 32-64 GB/s saturation and the HBM-floor plateau are explicit arithmetic (14 GB / 3350 GB/s = 4.18 ms) and a physical bound, not a restatement of an input parameter. The area and power overhead is independently grounded in Verilog synthesis (Section VII-H). The one mildly self-referential element is Finding 5's tensor-parallel extrapolation, where link coefficients are fitted from the simulator ladders and reused to predict saturation bandwidths at 70B-1.2T scale; however, the paper validates the fitted model by reproducing the reference ladders within 1%, and the predicted saturation points are model outputs rather than the fitted parameters themselves. That is calibration-and-extrapolation, not a reduction by construction. The paper's own 'Scope of the claim' caveats openly state that no NVLink-attached memory part exists and that the device-side calibration uses an NDP-less device; these are correctness and external-validity risks, not circularity. Overall, the derivation is self-contained with respect to its inputs, so no circular step is present.

Assumptions & free parameters 8 free parameters · 5 assumptions · 2 invented entities

The paper's central performance claims rest on a simulator plus assumed fabric properties. Several hand-picked parameters (NDP throughput, device count, link latency, temperature heuristic) and fitted TP coefficients are load-bearing. The proposed NDP device and read-compute interface are new entities without independent evidence outside the paper.

free parameters (8)
  • NDP core compute throughput per device = 2 TFLOPS FP16
    Chosen as conservative relative to Samsung CXL-PNM's 4.9 TFLOPS; sensitivity sweep shows 1-2 TFLOPS saturates, so the choice is motivated but still a hand-picked parameter.
  • Number of PLoRA memory devices = 4
    Evaluation config; the paper notes 4 sits at the short-context knee and short of the long-context optimum (Section VIII Finding 3).
  • CXL link latency = 200 ns
    Assumed one-way latency for CXL 3.1; the sweep shows low sensitivity, but it is an input parameter.
  • CXL link bandwidth = 128 GB/s per direction
    Assumed CXL 3.1 x16 rate; used for all main results.
  • Temperature relaxation factor = 0.1
    Heuristic in Section VI-A for hot/cold adapter classification; no sensitivity study is reported.
  • Pre-cache threshold = 50 hottest non-serving adapters
    Chosen in Section VI-A; no sweep or justification beyond intuition.
  • Tensor-parallel link coefficients = not disclosed (fitted to within 1% of reference model)
    Section VIII Finding 5 fits coefficients from simulated ladders; used to predict 70B-1.2T saturation bandwidth.
  • Per-device cost estimate = $3,000 per device
    Section VII-G cost analysis assumes $3,000 for 512GB LPDDR5X plus FPGA; authors call it a conservative bound but it is an estimate.
assumptions (5)
  • domain assumption Memory-semantic fabric properties F1, F2, F3 are available: accelerator load/store to pool, reachable device compute, capacity independent of host DIMMs.
    PLoRA's entire design depends on these properties; Section II-C asserts CXL 3.1 and NVLink-class fabrics provide them, but no shipping NVLink-attached memory expander exists.
  • ad hoc to paper A CXL Type 3 memory expander can host an NDP core and still expose a CXL.mem read-compute interface without violating the CXL 3.1 specification.
    Section IV-C claims the interface uses standard CXL.mem/io and adds no opcode, but this is not demonstrated against a spec conformance test or a real device.
  • standard math Partial softmax with anchors, as in FlashAttention, produces the exact global attention output on distributed KV cache without device-to-device communication.
    Section V-B.2 invokes FlashAttention's known technique; this is mathematically established, but the implementation on NDP cores is unverified.
  • ad hoc to paper The event-driven simulator, calibrated on an NDP-less CXL device (Samsung CMM-D) and GPU utilization measurements, accurately models the proposed PLoRA device including NDP timing and DRAM contention.
    Section VII-A describes calibration, but no end-to-end validation against a real NDP-capable CXL device is provided.
  • ad hoc to paper Tensor-parallel scaling model, with link coefficients fitted from simulator ladders, extrapolates saturation bandwidth from 7B to 1.2T parameters.
    Section VIII Finding 5 fits coefficients to reproduce the simulator ladders and then uses them to predict larger deployments; the fit is not a physical derivation.
invented entities (2)
  • PLoRA memory device (CXL Type 3 expander with NDP core)
    purpose: Stores LoRA adapters and KV cache in pooled memory and executes reduction-heavy GEMM/GEMV/softmax near the data, returning only reduced results to the GPU.
    No such device exists commercially; the paper synthesizes the NDP core in Verilog for area/power but does not build or measure the full device.
  • Read-compute request descriptor and HDM submission queue
    purpose: Extends load/store semantics with a compute command so the GPU can direct NDP cores without host mediation.
    Described in Section IV-C as a 204-bit descriptor in one CXL flit; no implementation or conformance test is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PLoRA: An NDP-Enhanced Pooled-Memory System for Cost-Efficient Multi-LoRA Serving." pith.science (2026). https://pith.science/paper/63NAJU5J

@misc{pith2026260805483,
  author       = {Pith},
  title        = {Pith review of: PLoRA: An NDP-Enhanced Pooled-Memory System for Cost-Efficient Multi-LoRA Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/63NAJU5J}},
  note         = {Machine review of arXiv:2608.05483}
}
read the original abstract

Multi-LoRA serving is how one base model becomes thousands of specialized variants, one adapter per user, task, or agent, and the deployments can hold 1000-plus adapters. Serving them is hard because the workload inverts what GPUs provide: terabytes of memory against only tens of TFLOPS, and because every published system stages its adapters from CPU DRAM over PCIe, where each access pays a kernel stop and a host-run copy and capacity ends at the motherboard's DIMM slots. Meanwhile, memory-semantic fabrics such as CXL and NVLink are converging on pooled memory that an accelerator addresses with its own loads and stores, and near-data processing (NDP) can place compute beside the pooled data. How to serve multi-LoRA workloads on such hardware remains unexplored. This paper introduces PLoRA, an NDP-enhanced pooled-memory system for cost-efficient multi-LoRA serving. PLoRA keeps adapters and KV cache in the pool and returns only reduced results over the link, through a read-compute interface the GPU drives with its own loads and stores. Above this architecture, a GPU memory management system picks among four LoRA and two attention execution strategies for each adapter and caches the most performance-critical bytes in GPU memory, guided by a link-parameterized cost model. On one H100 serving 1000 adapters, PLoRA attains the lowest decode latency on every model and workload we measure, averaging 6.6x below a real-machine S-LoRA at under 3.4% added device area. The link itself stops mattering: throughput saturates at 32 GB/s on short contexts, a quarter of CXL 3.1, and the verdict survives scale: per-GPU demand falls from 7B to a modeled 1.2T deployment once adapter traffic shards with the tensor parallelism. The design runs unchanged from CXL-class to NVLink-class fabrics, and surplus bandwidth buys pooled capacity rather than speed.

Figures

Figures reproduced from arXiv: 2608.05483 by the authors.

Figure 1
Figure 1. Batch sizes differ across adapters, and each adapter’s batch varies [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Pooled memory with NDP against the PCIe+CPU offload path, in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) The best of the four strategies for one LoRA adapter changes with [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The read-compute request descriptor. C. Why is it Difficult to Serve Multi-LoRA Inference even with CXL + NDP? 1. The optimal execution strategy of each LoRA adapter varies. Compute and memory sit on both the GPU and the device, so every adapter admits several executio…
Figure 5
Figure 5. Figure 5: The PLoRA system and architecture [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Three data placements with varying GPU memory demand enable [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: A collaborative approach for attention computation utilizing GPU and [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Decode latency (TPOT, lower is better) on H100. S-LoRA is a real-machine measurement and appears only for the two models we could run. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Ablation on Llama2-7B, bars as a percentage of full PLoRA (NSD). [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Sensitivity of PLoRA decode throughput to the two device-side [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 12
Figure 12. Figure 12: Saturation link bandwidth against model scale. Filled markers are [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: PLoRA across the interconnect ladder: (a) throughput against link bandwidth (Llama2-7B, Uniform; rules mark per-direction fabric rates), (b) [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

84 extracted references · 69 canonical work pages

  1. [1]

    Exploiting cxl-based memory for distributed deep learning,

    M. Arif, K. Assogba, M. M. Rafique, and S. Vazhkudai, “Exploiting cxl-based memory for distributed deep learning,” inProceedings of the 51st International Conference on Parallel Processing, 2022

    2022

  2. [2]

    Arm neoverse and NVIDIA NVLink fusion: AMBA CHI C2C interoperability for custom silicon,

    Arm Ltd., “Arm neoverse and NVIDIA NVLink fusion: AMBA CHI C2C interoperability for custom silicon,” Arm Newsroom, 2025, 17 Nov. 2025. [Online]. Available: https://newsroom.arm.com/news/arm- neoverse-nvidia-nvlink

    2025

  3. [3]

    Chameleon: Versatile and practical near-dram acceleration architecture for large memory systems,

    H. Asghari-Moghaddam, Y . H. Son, J. H. Ahn, and N. S. Kim, “Chameleon: Versatile and practical near-dram acceleration architecture for large memory systems,” in2016 49th annual IEEE/ACM interna- tional symposium on Microarchitecture (MICRO). IEEE, 2016

    2016

  4. [4]

    Nvidia spent over $900 million on Enfabrica CEO, AI startup technology,

    J. Babu, “Nvidia spent over $900 million on Enfabrica CEO, AI startup technology,” Reuters, 2025, 18 Sep. 2025. NVIDIA licensed Enfabrica’s server-fabric technology and hired its chief executive. [Online]. Available: https://www.cnbc.com/2025/09/18/nvidia-spent- over-900-million-on-enfabrica-ceo-ai-startup-technology.html

    2025

  5. [5]

    Punica: Multi-tenant lora serving,

    L. Chen, Z. Ye, Y . Wu, D. Zhuo, L. Ceze, and A. Krishnamurthy, “Punica: Multi-tenant lora serving,”Proceedings of Machine Learning and Systems, 2024

    work page 2024

  6. [6]

    Compute express link (CXL) 3.1 specification,

    Compute Express Link Consortium, “Compute express link (CXL) 3.1 specification,” CXL Consortium webinar presentation, Feb. 2024. [Online]. Available: https://computeexpresslink.org/wp-content/uploads/ 2024/03/CXL 3.1-Webinar-Presentation Feb 2024.pdf

    work page 2024

  7. [7]

    A complete survey on LLM-based AI chatbots,

    S. K. Dam, C. S. Hong, Y . Qiao, and C. Zhang, “A complete survey on LLM-based AI chatbots,”arXiv preprint arXiv:2406.16937, 2024

    arXiv 2024

  8. [8]

    FlashAttention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “FlashAttention-2: Faster attention with better parallelism and work partitioning,” inInternational Conference on Learning Represen- tations (ICLR), 2024

    2024

Show all 84 references
  1. [9]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in neural information processing systems, 2022

    2022

  2. [10]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in neural information processing systems, 2023

    2023

  3. [11]

    The true processing in memory accelerator,

    F. Devaux, “The true processing in memory accelerator,” in2019 IEEE Hot Chips 31 Symposium (HCS). IEEE Computer Society, 2019

    2019

  4. [12]

    EMFASYS: Elastic memory fabric system,

    Enfabrica Corporation, “EMFASYS: Elastic memory fabric system,” Product page, 2025, launched 29 Jul. 2025. Up to 18 TB of CXL DDR5 per system across up to 144 CXL memory lanes, fronted by RDMA Ethernet. [Online]. Available: https://enfabrica.net/solution/emfasys

    2025

  5. [13]

    NVIDIA Grace Hopper superchip architecture in-depth,

    J. Evans, M. Andersch, V . Sethi, G. Brito, and V . Mehta, “NVIDIA Grace Hopper superchip architecture in-depth,” NVIDIA Technical Blog, 2022, 10 Nov. 2022. Reports NVLink-C2C at 900 GB/s total, 450 GB/s per direction, and states that fourth-generation NVLink accesses peer mem...

    2022

  6. [14]

    Techniques for an efficient fabric attached memory,

    J. Feehrer, D. Foley, M. Hummel, V . Venkataraman, R. Gummadi, S. H. Duncan, G. Dearth, and B. Kelleher, “Techniques for an efficient fabric attached memory,” U.S. Patent 11,182,309 B2, assigned to NVIDIA Corporation, 2021, filed 4 Nov. 2019, granted 23 Nov. 2021. [Online]. Av...

    2021

  7. [15]

    Techniques for an efficient fabric attached memory,

    J. Feehrer, D. Foley, M. Hummel, V . Venkataraman, R. Gummadi, S. H. Duncan, G. Dearth, and B. Kelleher, “Techniques for an efficient fabric attached memory,” U.S. Patent 11,822,491 B2, assigned to NVIDIA Corporation, 2023, filed 20 Oct. 2021, granted 21 Nov. 2023. [Online]. A...

    2023

  8. [16]

    Low-overhead general-purpose near- data processing in CXL memory expanders,

    H. Ham, J. Hong, G. Park, Y . Shin, O. Woo, W. Yang, J. Bae, E. Park, H. Sung, E. Lim, and G. Kim, “Low-overhead general-purpose near- data processing in CXL memory expanders,” inProceedings of the 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2024, ...

    2024

  9. [17]

    Newton: A dram-maker’s accelerator-in- memory (aim) architecture for machine learning,

    M. He, C. Song, I. Kim, C. Jeong, S. Kim, I. Park, M. Thot- tethodi, and T. N. Vijaykumar, “Newton: A dram-maker’s accelerator-in- memory (aim) architecture for machine learning,” in2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2020

    2020

  10. [18]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inInternational conference on machine learning, 2019

    2019

  11. [19]

    Latency and bandwidth impact on GPU-systems,

    R. J. Hovland, “Latency and bandwidth impact on GPU-systems,” Norwegian University of Science and Technology, Tech. Rep., 2008

    2008

  12. [20]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” inInternational Conference on Learning Representations (ICLR), 2022

    2022

  13. [21]

    Achieving page-mapping ftl performance at block-mapping ftl cost by hiding address translation,

    Y . Hu, H. Jiang, D. Feng, L. Tian, S. Zhang, J. Liu, W. Tong, Y . Qin, and L. Wang, “Achieving page-mapping ftl performance at block-mapping ftl cost by hiding address translation,” in2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), 2010

    2010

  14. [22]

    Beacon: Scalable near-data-processing accelerators for genome analysis near memory pool with the cxl support,

    W. Huangfu, K. T. Malladi, A. Chang, and Y . Xie, “Beacon: Scalable near-data-processing accelerators for genome analysis near memory pool with the cxl support,” in2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2022

    2022

  15. [23]

    Pifs-rec: Process- in-fabric-switch for large-scale recommendation system inferences,

    P. Huo, A. Devulapally, H. Al Maruf, M. Park, K. Nair, M. Arunachalam, G. G. Akbulut, M. T. Kandemir, and V . Narayanan, “Pifs-rec: Process- in-fabric-switch for large-scale recommendation system inferences,” in 2024 57th IEEE/ACM International Symposium on Microarchitecture (...

    2024

  16. [24]

    CXL- ANNS:Software-Hardware collaborative memory disaggregation and computation for Billion-Scale approximate nearest neighbor search,

    J. Jang, H. Choi, H. Bae, S. Lee, M. Kwon, and M. Jung, “CXL- ANNS:Software-Hardware collaborative memory disaggregation and computation for Billion-Scale approximate nearest neighbor search,” in 2023 USENIX Annual Technical Conference (USENIX ATC 23), 2023

    2023

  17. [25]

    Demystifying a CXL type-2 device: A heterogeneous cooperative computing perspective,

    H. Ji, S. Vanavasam, Y . Zhou, Q. Xia, J. Huang, Y . Yuan, R. Wang, P. Gupta, B. Chitlur, I. Jeong, and N. S. Kim, “Demystifying a CXL type-2 device: A heterogeneous cooperative computing perspective,” in 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO)....

    2024

  18. [26]

    Near-memory processing in action: Accelerating personalized recommendation with AxDIMM,

    L. Ke, X. Zhang, J. So, J.-G. Lee, S.-H. Kang, S. Lee, S. Han, Y . Cho, J. H. Kim, Y . Kwon, K. Kim, J. Jung, I. Yun, S. J. Park, H. Park, J.-H. Song, J. Cho, K. Sohn, N. S. Kim, and H.-H. S. Lee, “Near-memory processing in action: Accelerating personalized recommendation with...

    2022

  19. [27]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, 2023

    2023

  20. [28]

    25.4 a 20nm 6GB function-in-memory DRAM, based on HBM2 with a 1.2TFLOPS programmable computing unit using bank- level parallelism, for machine learning applications,

    Y .-C. Kwon, S. H. Lee, J. Lee, S.-H. Kwon, J. M. Ryu, J.-P. Son, S. O, H.-S. Yu, H. Lee, S. Y . Kim, Y . Cho, J. G. Kim, J. Choi, H.-S. Shin, J. Kim, B. Phuah, H. Kim, M. J. Song, A. Choi, D. Kim, S. Kim, E.-B. Kim, D. Wang, S. Kang, Y . Ro, S. Seo, J. Song, J. Youn, K. Sohn,...

    2021

  21. [29]

    TensorDIMM: A practical near-memory processing architecture for embeddings and tensor operations in deep learning,

    Y . Kwon, Y . Lee, and M. Rhu, “TensorDIMM: A practical near-memory processing architecture for embeddings and tensor operations in deep learning,” inProceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2019, pp. 740–753

    2019

  22. [30]

    Tensor casting: Co-designing algorithm- architecture for personalized recommendation training,

    Y . Kwon, Y . Lee, and M. Rhu, “Tensor casting: Co-designing algorithm- architecture for personalized recommendation training,” in2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2021

    2021

  23. [31]

    A 1ynm 1.25v 8gb, 16gb/s/pin gddr6-based accelerator- in-memory supporting 1tflops mac operation and various activation functions for deep-learning applications,

    S. Lee, K. Kim, S. Oh, J. Park, G. Hong, D. Ka, K. Hwang, J. Park, K. Kang, J. Kim, J. Jeon, N. Kim, Y . Kwon, K. Vladimir, W. Shin, J. Won, M. Lee, H. Joo, H. Choi, J. Lee, D. Ko, Y . Jun, K. Cho, I. Kim, C. Song, C. Jeong, D. Kwon, J. Jang, I. Park, J. Chun, and J. Cho, “A 1...

    2022

  24. [32]

    Hardware architecture and software stack for PIM based on commercial DRAM technology: Industrial product,

    S. Lee, S.-h. Kang, J. Lee, H. Kim, E. Lee, S. Seo, H. Yoon, S. Lee, K. Lim, H. Shin, J. Kim, S. O, A. Iyer, D. Wang, K. Sohn, and N. S. Kim, “Hardware architecture and software stack for PIM based on commercial DRAM technology: Industrial product,” in2021 ACM/IEEE 48th Annual...

    2021

  25. [33]

    Evaluating modern GPU interconnect: PCIe, NVLink, NV- SLI, NVSwitch and GPUDirect,

    A. Li, S. L. Song, J. Chen, J. Li, X. Liu, N. R. Tallent, and K. J. Barker, “Evaluating modern GPU interconnect: PCIe, NVLink, NV- SLI, NVSwitch and GPUDirect,”IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 1, pp. 94–110, 2020. 12

    2020

  26. [34]

    Pond: CXL-based memory pooling systems for cloud platforms,

    H. Li, D. S. Berger, L. Hsu, D. Ernst, P. Zardoshti, S. Novakovic, M. Shah, S. Rajadnya, S. Lee, I. Agarwal, M. D. Hill, M. Fontoura, and R. Bianchini, “Pond: CXL-based memory pooling systems for cloud platforms,” inProceedings of the 28th ACM International Conference on Archi...

    2023

  27. [35]

    CaraServe: CPU-assisted and rank-aware LoRA serving for generative LLM inference,

    S. Li, H. Lu, T. Wu, M. Yu, Q. Weng, X. Chen, Y . Shan, B. Yuan, and W. Wang, “CaraServe: CPU-assisted and rank-aware LoRA serving for generative LLM inference,”arXiv preprint arXiv:2401.11240, 2024

    2024 arXiv

  28. [36]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” inProceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2021, pp. ...

    2021

  29. [37]

    Disaggregated memory for expansion and sharing in blade servers,

    K. Lim, J. Chang, T. Mudge, P. Ranganathan, S. K. Reinhardt, and T. F. Wenisch, “Disaggregated memory for expansion and sharing in blade servers,”ACM SIGARCH computer architecture news, 2009

    2009

  30. [38]

    Enabling efficient large recommendation model training with near cxl memory processing,

    H. Liu, L. Zheng, Y . Huang, J. Zhou, C. Liu, R. Wang, X. Liao, H. Jin, and J. Xue, “Enabling efficient large recommendation model training with near cxl memory processing,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024

    2024

  31. [39]

    Make LLM inference affordable to everyone: Augmenting GPU memory with NDP-DIMM,

    L. Liu, S. Zhao, B. Li, H. Ren, Z. Xu, M. Wang, X. Li, Y . Han, and Y . Wang, “Make LLM inference affordable to everyone: Augmenting GPU memory with NDP-DIMM,” in2025 IEEE International Sympo- sium on High-Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1751–1765

    2025

  32. [40]

    The llama 3 herd of models,

    Llama Team, AI @ Meta, “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  33. [41]

    LLaMAX: Scaling linguistic horizons of LLM by enhancing translation capabilities beyond 100 languages,

    Y . Lu, W. Zhu, L. Li, Y . Qiao, and F. Yuan, “LLaMAX: Scaling linguistic horizons of LLM by enhancing translation capabilities beyond 100 languages,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 10 748–10 772

    2024

  34. [42]

    PEFT: State-of-the-art parameter-efficient fine-tuning methods,

    S. Mangrulkar, S. Gugger, L. Debut, Y . Belkada, S. Paul, and B. Bossan, “PEFT: State-of-the-art parameter-efficient fine-tuning methods,” https: //github.com/huggingface/peft, 2022

    2022

  35. [43]

    SiFive bakes NVIDIA’s NVLink fusion into its RISC-V CPU IP,

    T. Mann, “SiFive bakes NVIDIA’s NVLink fusion into its RISC-V CPU IP,” The Register, 2026, 15 Jan. 2026. [Online]. Available: https://www.theregister.com/2026/01/15/sifive nvidia nvlink/

    2026

  36. [44]

    Tpp: Transparent page placement for cxl-enabled tiered-memory,

    H. A. Maruf, H. Wang, A. Dhanotia, J. Weiner, N. Agarwal, P. Bhat- tacharya, C. Petersen, M. Chowdhury, S. Kanaujia, and P. Chauhan, “Tpp: Transparent page placement for cxl-enabled tiered-memory,” in Proceedings of the 28th ACM International Conference on Architectural Suppor...

    2023

  37. [45]

    Marvell introduces breakthrough Structera CXL product line to address server memory bandwidth and capacity challenges in cloud data centers,

    Marvell Technology, Inc., “Marvell introduces breakthrough Structera CXL product line to address server memory bandwidth and capacity challenges in cloud data centers,” Marvell press release, 2024, structera A places 16 Arm Neoverse V2 cores behind a CXL memory port, and Struc...

    2024

  38. [46]

    Specinfer: Accelerating large language model serv- ing with tree-based speculative inference and verification,

    X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y . Y . Wong, A. Zhu, L. Yang, X. Shi, C. Shi, Z. Chen, D. Arfeen, R. Ab- hyankar, and Z. Jia, “Specinfer: Accelerating large language model serv- ing with tree-based speculative inference and verification,” inProce...

    2024

  39. [47]

    Micron launches memory expansion module portfolio to accelerate CXL 2.0 adoption,

    Micron Technology, “Micron launches memory expansion module portfolio to accelerate CXL 2.0 adoption,” 2023, cZ120, CXL 2.0 Type-3, PCIe Gen5×8, up to 36 GB/s measured read/write bandwidth. [Online]. Available: https://www.globenewswire.com/news- release/2023/08/07/2719764/144...

    2023

  40. [48]

    Nvidia licenses NVLink memory ports to CPU and accelerator makers,

    T. P. Morgan, “Nvidia licenses NVLink memory ports to CPU and accelerator makers,” The Next Platform, 2025, 19 May 2025. [Online]. Available: https://www.nextplatform.com/2025/05/19/nvidia- licenses-nvlink-memory-ports-to-cpu-and-accelerator-makers/

    2025

  41. [49]

    Using an llm to help with code understanding,

    D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers, “Using an llm to help with code understanding,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24, 2024

    2024

  42. [50]

    Understanding pcie performance for end host networking,

    R. Neugebauer, G. Antichi, J. F. Zazo, Y . Audzevich, S. L ´opez-Buedo, and A. W. Moore, “Understanding pcie performance for end host networking,” inProceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, 2018

    2018

  43. [51]

    NVIDIA A100 Tensor Core GPU Archi- tecture,

    NVIDIA, “NVIDIA A100 Tensor Core GPU Archi- tecture,” NVIDIA Whitepaper, 2020. [Online]. Avail- able: https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/ nvidia-ampere-architecture-whitepaper.pdf

    2020

  44. [52]

    NVIDIA H100 Tensor Core GPU Architecture,

    NVIDIA, “NVIDIA H100 Tensor Core GPU Architecture,” NVIDIA Whitepaper, 2023. [Online]. Available: https://resources.nvidia.com/en- us-tensor-core

    2023

  45. [53]

    NVIDIA unveils NVLink fusion for industry to build semi-custom AI infrastructure with NVIDIA partner ecosystem,

    NVIDIA Corporation, “NVIDIA unveils NVLink fusion for industry to build semi-custom AI infrastructure with NVIDIA partner ecosystem,” NVIDIA Newsroom, COMPUTEX, Taipei, 2025, 18 May 2025. [Online]. Available: https://nvidianews.nvidia.com/news/nvidia-nvlink- fusion-semi-custom...

    2025

  46. [54]

    Extended GPU memory,

    NVIDIA Corporation, “Extended GPU memory,” CUDA C++ Programming Guide, v13.3, Sec. 4.17, 2026. [Online]. Avail- able: https://docs.nvidia.com/cuda/cuda-programming-guide/04-special- topics/extended-gpu-memory.html

    2026

  47. [55]

    NVLink-C2C: Chip interconnect technology,

    NVIDIA Corporation, “NVLink-C2C: Chip interconnect technology,” NVIDIA product page, 2026, states that NVLink-C2C “supports Arm’s AMBA CHI (Coherent Hub Interface) or Compute Express Link (CXL) industry standard protocols for interoperability between devices”. [Online]. Availa...

    2026

  48. [56]

    NVLink: The scale-up network for AI factories,

    NVIDIA Corporation, “NVLink: The scale-up network for AI factories,” NVIDIA Technical Blog, 2026, sixth-generation NVLink at 3.6 TB/s bidirectional per GPU and NVLink-C2C at 1.8 TB/s coherent CPU- GPU bandwidth. [Online]. Available: https://developer.nvidia.com/blog/ nvidia-nv...

    2026

  49. [57]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  50. [58]

    Attacc! unleashing the power of pim for batched transformer- based generative model inference,

    J. Park, J. Choi, K. Kyung, M. J. Kim, Y . Kwon, N. S. Kim, and J. H. Ahn, “Attacc! unleashing the power of pim for batched transformer- based generative model inference,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages...

    2024

  51. [59]

    Trim: Enhancing processor-memory interfaces with scalable tensor reduction in memory,

    J. Park, B. Kim, S. Yun, E. Lee, M. Rhu, and J. H. Ahn, “Trim: Enhancing processor-memory interfaces with scalable tensor reduction in memory,” inMICRO-54: 54th Annual IEEE/ACM International Sym- posium on Microarchitecture, ser. MICRO, 2021

    2021

  52. [60]

    An LPDDR-based CXL-PNM platform for TCO-efficient inference of transformer-based large language models,

    S.-S. Park, K. Kim, J. So, J. Jung, J. Lee, K. Woo, N. Kim, Y . Lee, H. Kim, Y . Kwon, J. Kim, J. Lee, Y . Cho, Y . Tai, J. Cho, H. Song, J. H. Ahn, and N. S. Kim, “An LPDDR-based CXL-PNM platform for TCO-efficient inference of transformer-based large language models,” in2024 ...

    2024

  53. [61]

    System optimization of data analytics platforms using compute express link (CXL) memory,

    S. Ryu, S. Kim, J. Jun, D. Moon, K. Lee, J. Choi, S. Kim, H. Kim, L. Kim, W. H. Choi, M. Nam, D. Hwang, H. Roh, and Y .-P. Joo, “System optimization of data analytics platforms using compute express link (CXL) memory,” in2023 IEEE International Conference on Big Data and Smart...

    2023

  54. [62]

    Samsung CXL solutions: CMM-H (CXL memory module – hybrid),

    Samsung Electronics, “Samsung CXL solutions: CMM-H (CXL memory module – hybrid),” Samsung Semiconductor Tech Blog,

  55. [63]

    Flashattention-3: Fast and accurate attention with asynchrony and low- precision,

    J. Shah, G. Bikshandi, Y . Zhang, V . Thakkar, P. Ramani, and T. Dao, “Flashattention-3: Fast and accurate attention with asynchrony and low- precision,”Advances in Neural Information Processing Systems, 2024

    2024

  56. [64]

    Pci express® 6.0 specification at 64.0 gt/s with pam- 4 signaling: a low latency, high bandwidth, high reliability and cost- effective interconnect,

    D. D. Sharma, “Pci express® 6.0 specification at 64.0 gt/s with pam- 4 signaling: a low latency, high bandwidth, high reliability and cost- effective interconnect,” in2020 IEEE Symposium on High-Performance Interconnects (HOTI), 2020

    2020

  57. [65]

    S-LoRA: Serving thousands of concurrent LoRA adapters,

    Y . Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer, J. E. Gonzalez, and I. Stoica, “S-LoRA: Serving thousands of concurrent LoRA adapters,” inProceedings of Machine Learning and Systems (MLSys), vol. 6, 2024

    2024

  58. [66]

    Flexgen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” inInternational Conference on Machine Learning. PMLR, 2023

    2023

  59. [67]

    NVIDIA announces NVLink fusion: Bringing NVLink to third-party CPUs and accelerators,

    R. Smith, “NVIDIA announces NVLink fusion: Bringing NVLink to third-party CPUs and accelerators,” ServeTheHome, 2025, 19 May 2025. 13 [Online]. Available: https://www.servethehome.com/nvidia-announces- nvlink-fusion-bringing-nvlink-to-third-party-cpus-and-accelerators/

    2025

  60. [68]

    Demysti- fying CXL memory with genuine CXL-ready systems and devices,

    Y . Sun, Y . Yuan, Z. Yu, R. Kuper, C. Song, J. Huang, H. Ji, S. Agarwal, J. Lou, I. Jeong, R. Wang, J. H. Ahn, T. Xu, and N. S. Kim, “Demysti- fying CXL memory with genuine CXL-ready systems and devices,” in Proceedings of the 56th Annual IEEE/ACM International Symposium on M...

    2023

  61. [69]

    NVIDIA’s NVLink fusion stays proprietary, third parties can only work around it,

    TechPowerUp, “NVIDIA’s NVLink fusion stays proprietary, third parties can only work around it,” TechPowerUp, 2025. [Online]. Available: https://www.techpowerup.com/338156/nvidias-nvlink-fusion- stays-proprietary-third-parties-can-only-work-around-it

    2025

  62. [70]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Ho...

    2023 arXiv

  63. [71]

    Introducing UALink 200g 1.0 specification,

    UALink Consortium, “Introducing UALink 200g 1.0 specification,” 2025, published Apr. 2025. An open accelerator interconnect scaling to 1024 accelerators, 200 GT/s per lane with 212.5 GT/s signaling, in×1, ×2, and×4 port widths. [Online]. Available: https://ualinkconsortium. or...

    2025

  64. [72]

    RAG-based LLM chatbot using Llama-2,

    S. Vakayil, D. S. Juliet, J. Anitha, and S. Vakayil, “RAG-based LLM chatbot using Llama-2,” in2024 7th International Conference on De- vices, Circuits and Systems (ICDCS). IEEE, 2024, pp. 1–5

    2024

  65. [73]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, 2017

    2017

  66. [74]

    CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. H. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” inProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021, pp. 8696– 8708

    2021

  67. [75]

    Towards memory disaggregation via NVLink C2C: Benchmarking CPU-requested GPU memory access,

    F. Werner, M. Weisgut, and T. Rabl, “Towards memory disaggregation via NVLink C2C: Benchmarking CPU-requested GPU memory access,” inProceedings of the 4th Workshop on Heterogeneous Composable and Disaggregated Systems (HCDS). Rotterdam, Netherlands: ACM, 2025, pp. 8–14

    2025

  68. [76]

    dLoRA: Dynami- cally orchestrating requests and adapters for LoRALLM serving,

    B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin, “dLoRA: Dynami- cally orchestrating requests and adapters for LoRALLM serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024

    2024

  69. [77]

    Overcoming the memory wall with CXL- EnabledSSDs,

    S.-P. Yang, M. Kim, S. Nam, J. Park, J.-y. Choi, E. H. Nam, E. Lee, S. Lee, and B. S. Kim, “Overcoming the memory wall with CXL- EnabledSSDs,” in2023 USENIX Annual Technical Conference (USENIX ATC 23), 2023

    2023

  70. [78]

    Orca: A distributed serving system for Transformer-Based generative models,

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, “Orca: A distributed serving system for Transformer-Based generative models,” in 16th USENIX Symposium on Operating Systems Design and Implemen- tation, 2022

    2022

  71. [80]

    SGLang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y . Sheng, “SGLang: Efficient execution of structured language model programs,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 37, 2024

    2024

  72. [81]

    DistServe: Disaggregating prefill and decoding for goodput- optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput- optimized large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024

    2024

  73. [82]

    Remap-SSD: Safely and efficiently exploiting SSD address remapping to eliminate duplicate writes,

    Y . Zhou, Q. Wu, F. Wu, H. Jiang, J. Zhou, and C. Xie, “Remap-SSD: Safely and efficiently exploiting SSD address remapping to eliminate duplicate writes,” in19th USENIX Conference on File and Storage Technologies (FAST 21), 2021, pp. 187–202

    2021

  74. [83]

    NeoMem: Hardware/software co- design for CXL-native memory tiering,

    Z. Zhou, Y . Chen, T. Zhang, Y . Wang, R. Shu, S. Xu, P. Cheng, L. Qu, Y . Xiong, J. Zhang, and G. Sun, “NeoMem: Hardware/software co- design for CXL-native memory tiering,” inProceedings of the 2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2024, pp....

    2024

  75. [2024]

    Available: https://semiconductor.samsung.com/news- events/tech-blog/samsung-cxl-solutions-cmm-h/

    [Online]. Available: https://semiconductor.samsung.com/news- events/tech-blog/samsung-cxl-solutions-cmm-h/

  76. [2025]

    Available: https://arxiv.org/abs/2503.22017

    [Online]. Available: https://arxiv.org/abs/2503.22017

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.