Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Hardware-based Heterogeneous Memory Management for Large Language Model Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read H2M2 claims that putting accelerators on both fast HBM and cheap LPDDR memory, plus a runtime kernel-placement rule, runs LLM inference 1.46–2.94x faster than an LPDDR-only system and within 5% of the optimal mapping.

desk verdict Solid asymmetric-memory proposal for LLM serving, but the near-optimal mapping claim needs the hidden arithmetic-intensity parameter disclosed before I fully trust it. read the letter →

arxiv 2504.14893 v1 pith:Z4I6ECC6 submitted 2025-04-21 cs.AR

Soojin Hwang , Jungwoo Kim , Sanghyeon Lee , Hongbeen Kim , Jaehyuk Huh This is my paper · ORCID
classification cs.AR
keywords largelanguagemodelsasymmetricmemoryHBMLPDDRKVcachekernel-memorymappingabstractionhardwareaccelerator
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the most cost-effective way to serve large language models is not to move everything into expensive high-bandwidth memory, and not to treat slow memory as a backing store, but to attach a small accelerator to each memory type and run both in parallel. For LLM generation, where attention kernels are bandwidth-bound and the KV cache grows every token, the central problem becomes deciding which kernels live on the HBM side and which on the LPDDR side at each moment. H2M2 solves this with a greedy runtime mapping algorithm and a hardware memory abstraction that lets the KV cache grow and the mapping change without rewriting kernels. On GPT3-175B, Chinchilla-70B, and Llama2-70B, the paper reports 1.46x, 1.55x, and 2.94x speedups over an LPDDR-only system, staying within 5 percent of an oracle mapping that always knows the best placement.

What carries the argument

The load-bearing object is the asymmetric memory system itself: one HBM3 module (96 GB, 3 TB/s) and one LPDDR5X module (512 GB, 544 GB/s), each with its own accelerator chip, linked by a high-bandwidth interconnect. Three mechanisms sit on top. First, head-aware mapping granularity splits qkv-linear and attention by attention heads and splits fc by column, so both sides compute in parallel with minimal synchronization. Second, Algorithm 1 is a greedy min-max solver that processes sublayers in the priority order attention, qkv-linear, fc and picks the number of heads to place in HBM so that the estimated peak execution times on the two sides are balanced while memory capacity is respected. Third, a hardware memory abstraction with per-side MMUs and flat page tables decouples logical contiguity from physical placement, making KV-cache allocation, head-fusion, and mapping migration cheap.

What would settle it

Use H2M2's mapping algorithm on a held-out LLM whose layer shape differs from the three evaluated models, for example a mixture-of-experts model or one with very different head and group sizes, without re-tuning the arithmetic-intensity hyperparameter, and compare against the exhaustive-search oracle; if the placement gap exceeds 5 percent on average, the near-optimal mapping claim fails.

Watch

Extended reading notes

Core claim

The central claim is that an asymmetric memory architecture with computation attached to both the bandwidth-centric and the capacity-centric memory can serve decoder-based LLM inference close to the speed of an oracle that always picks the best kernel-memory mapping, while using far less high-bandwidth memory. The key pattern the paper establishes is that attention is the sublayer that must be favored in HBM: its GEMV kernels have constant arithmetic intensity and are highly bandwidth-bound, and its KV-cache footprint grows with sequence length, whereas qkv-linear and fc become less critical as batch size grows. A second claim is that the best mapping can be found cheaply at runtime by a greedy algorithm that, for each sublayer, chooses how many attention heads go to HBM so as to balance the estimated peak execution times on both sides. A third claim is that hardware page-based memory abstraction keeps the cost of dynamic remapping and KV-cache growth small, with measured overhead under 1.4 percent.

Load-bearing premise

The runtime mapping algorithm's estimate of how long each sublayer takes is scaled by an undisclosed hyperparameter meant to capture arithmetic intensity, and if that parameter was tuned on the same three models used for the speedup measurements, the reported closeness to oracle placement is calibration rather than prediction.

Editorial extensions

If this is right

  • LLM serving can keep most weights in large, cheap LPDDR and reserve HBM for the bandwidth-bound attention kernels, reducing the number of HBM devices needed to serve a given model.
  • Dynamic KV-cache growth and variable sequence lengths can be handled by page-level allocation rather than contiguous pre-reservation, so HBM capacity is used close to fully across sequence lengths.
  • The greedy mapping algorithm tracks the oracle within 5 percentage points on all three evaluated models, including a 128-iteration scenario with randomly ending requests.
  • Memory abstraction overhead stays below 1.4 percent, so the runtime gains of asymmetric memory are not offset by translation and migration costs.
  • Compared with an eight-HBM system, H2M2 uses roughly 0.58x the memory energy per token, making the cost-performance tradeoff favorable for large models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The attention-first priority likely transfers to any memory-bound stage of an LLM, such as long-context attention or mixture-of-experts routing, but the paper evaluates only three dense decoder models, so this broader claim is our inference, not the paper's.
  • If the undisclosed arithmetic-intensity hyperparameter in Algorithm 1 was calibrated on the same three workloads used for evaluation, the reported near-oracle gap is a fitted result; the natural test is a held-out fourth model, which we give as the falsifier.
  • The hardware memory abstraction is not limited to KV caches: it could also support transparent offloading of weights or activations in other heterogeneous-memory workloads, though the paper does not explore that.
  • The head-aware split suggests a path to scaling asymmetric memory beyond one HBM plus one LPDDR pair, for example by adding more capacity-centric modules and extending Algorithm 1 to a multi-way min-max problem.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes H2M2, an asymmetric memory architecture for LLM generation-phase inference in which an HBM-attached accelerator and a large LPDDR-attached accelerator cooperate on a heterogeneous memory board. The main contributions are a head-aware kernel-memory mapping granularity, a runtime mapping algorithm (Algorithm 1) that greedily allocates sublayers to HBM in the order attention, qkv-linear, fc, and a hardware memory abstraction based on flat page tables and MMUs to handle KV-cache growth and mapping changes. Using a cycle-level simulator, the authors report speedups of 1.46x, 1.55x, and 2.94x over an LPDDR-only capacity-centric baseline for GPT3-175B, Chinchilla-70B, and Llama2-70B, with mapping overheads of 2.56%, 3.76%, and 0.60% relative to an oracle mapping.

Significance. If the results hold, this is a valuable systems contribution: it demonstrates a concrete way to combine bandwidth-centric HBM and capacity-centric LPDDR with accelerators on both sides, and it provides quantitative evidence for the importance of kernel-to-memory mapping granularity and dynamic mapping in LLM inference. The energy comparison and sensitivity study strengthen the practical case. The main speedup over the LPDDR-only baseline is fairly robust because even the suboptimal FlexGen-style mapping already achieves 1.30x in Figure 7. However, the paper's headline near-optimal mapping claim is not yet supported with sufficient rigor: it depends on an undisclosed arithmetic-intensity hyperparameter and on a priority order derived from the same three workloads used for evaluation, and no artifact is released. The central architectural idea is promising, but the mapping contribution needs disclosure and independent validation before the quantitative claims can be accepted.

major comments (4)
  1. [Section 4.3.2, Algorithm 1 and Table 3] The performance model for mapping decisions is not fully specified. The text states that the ideal execution time is the total arithmetic operations divided by peak throughput, multiplied by a hyperparameter that reflects the arithmetic intensity. No value, per-sublayer treatment, or calibration procedure is provided. Because arithmetic intensity differs between attention, qkv-linear, and fc and varies with batch size and sequence length, a single scalar cannot represent it. Since the same three workloads are used both to motivate the priority order and to report the mapping overheads, the reported closeness to Oracle (2.56%, 3.76%, and 0.60% in Table 3) may be a fitted outcome rather than an independent prediction. Please disclose the hyperparameter, specify how it is obtained for each sublayer, and validate the mapping algorithm on held-out workloads or hardware configurations.
  2. [Section 4.3.2, Algorithm 1] Algorithm 1 is too underspecified to reproduce. The three bullets under Find n express feasibility and a min-max objective verbally, but no equations define the capacity constraints, the peak execution time, or the role of the arithmetic-intensity hyperparameter. The paper calls the problem a simple linear problem but gives no linear program. This also makes the claimed 0.05ms solve time impossible to verify.
  3. [Section 5.1] The evaluation relies on a custom cycle-level simulator, but the manuscript only says it was cross-validated by profiling the open-source multi-core NPU simulator and DRAM simulator without reporting any validation data. The simulator and workload configurations are not released, so the absolute speedup numbers cannot be independently checked. Please provide a validation section or an artifact, or clearly state the limitations.
  4. [Section 3.2, Figure 7] The FlexGen comparison is not reproducible as presented. The authors say they modified Eq. (1) to suit asymmetric memory systems, but the modified optimization problem is never shown. Without the modified constraints and objective, the claim that FlexGen's model is structurally incapable of finding the best mapping cannot be verified.
minor comments (6)
  1. [Section 5.3] H2D2 should be H2M2 in the sentence reporting the 1.48x speedup.
  2. [Section 5.2.3] There are typos in this section: Lllama2 should be Llama2, and excedding should be exceeding.
  3. [Section 5.5] trasnlated should be translated.
  4. [Figure 7 caption] The caption says batch size 32, but the x-axis includes B16, B32, and B64; please fix the inconsistency.
  5. [Figure 9] Several labels in Figure 9 are garbled, including underu,liza,on and J underu,liza,on; these should be cleaned up.
  6. [Section 2.2.3] The statement that attention cannot be batched with multiple requests is imprecise; attention over multiple requests is routinely executed as batched GEMV with per-request KV caches (e.g., vLLM). Please clarify that the point is that attention does not become a single GEMM rather than that it cannot be batched at all.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline speedups are simulator-measured comparisons, and the undisclosed arithmetic-intensity hyperparameter is a reproducibility concern rather than a circular reduction.

full rationale

The paper's central performance claims are produced by a cycle-level simulator whose architecture parameters are taken from Grace Hopper, TPU, and published memory-latency data, and the baseline is a same-compute LPDDR-only system; these speedups do not reduce by construction to the paper's own inputs. The dynamic mapping algorithm in Algorithm 1 is a greedy min-max policy, and the evaluation compares its resulting mapping overhead against an exhaustive-search Oracle; nothing in the text states that the arithmetic-intensity hyperparameter mentioned in Section 4.3.2 was fitted to the three evaluation workloads, so the claimed closeness to Oracle is not demonstrably a renamed fit. The attention-first priority is motivated by the Section 3.3 empirical analysis on the same benchmarks, which is a generalizability or overfitting risk rather than a circular dependency. The self-citation to mNPUsim [16] is used only to cross-validate the simulator and does not supply the measured results. Thus no load-bearing step is equivalent to its inputs by definition, and the appropriate finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central quantitative results rest on a calibrated performance model with an undisclosed hyperparameter, a non-released simulator, and a set of hardware and workload assumptions drawn from prior work. The main free parameter is the arithmetic-intensity multiplier in Algorithm 1; the axioms are standard domain assumptions about LLM workload dominance and the simulator's fidelity.

free parameters (1)
  • Arithmetic-intensity hyperparameter in mapping performance model
    Algorithm 1 (Section 4.3.2) multiplies the ideal execution time (ops divided by peak throughput) by 'a hyperparameter that reflects the arithmetic intensity' to estimate kernel execution time. No value or calibration procedure is given; the mapping decisions and the reported near-optimality depend on this parameter.
assumptions (4)
  • domain assumption Decoder layers dominate decoder-based LLM inference time, so evaluating only the generation phase's decoder layers captures the relevant performance.
    Stated in Sections 2.1 and 5.1 of the paper; the evaluation measures single-iteration generation phase performance without prompt phase or TTFT.
  • domain assumption The cycle-accurate simulator faithfully reproduces the hardware behavior.
    Section 5.1 says the simulator is cross-validated by profiling mNPUsim and DRAMsim3, but no validation data or simulator code is provided.
  • domain assumption A Grace Hopper-like asymmetric memory configuration with equal compute on both HBM and LPDDR is a realistic and cost-effective substrate.
    Section 4.1 configures HBM3 96GB/3TB/s and LPDDR5X 512GB/544GB/s following NVIDIA Grace Hopper; the cost-effectiveness argument is qualitative.
  • domain assumption Head-aware mapping granularity applies to LLMs with independent heads (or experts in MoE), and fc layers can be split column-wise.
    Section 3.1 introduces this granularity; it is assumed to hold for all three evaluated models.
invented entities (1)
  • H2M2 hardware memory abstraction (MMU with flat page tables)
    purpose: Decouple logical and physical addresses so KV cache can grow, mapping can change, and kernels can access remote memory without copying whole tensors.
    Described in Section 4.2 and evaluated only in simulation; no prototype, benchmark, or formal model is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hardware-based Heterogeneous Memory Management for Large Language Model Inference." pith.science (2026). https://pith.science/paper/Z4I6ECC6

@misc{pith2026250414893,
  author       = {Pith},
  title        = {Pith review of: Hardware-based Heterogeneous Memory Management for Large Language Model Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4I6ECC6}},
  note         = {Machine review of arXiv:2504.14893}
}
read the original abstract

A large language model (LLM) is one of the most important emerging machine learning applications nowadays. However, due to its huge model size and runtime increase of the memory footprint, LLM inferences suffer from the lack of memory capacity in conventional systems consisting of multiple GPUs with a modest amount of high bandwidth memory. Moreover, since LLM contains many bandwidthintensive kernels, only focusing on the memory capacity without considering the bandwidth incurs a serious performance degradation. To handle such conflicting memory capacity and bandwidth demands in a cost-effective way, this study investigates the potential of heterogeneous memory systems, proposing H2M2. It uses an asymmetric memory architecture consisting of capacity-centric and bandwidthcentric memory with computation units attached to each memory device. With the asymmetric memory, we first analyze the effect of kernel-memory mapping for the asymmetric memory. Second, we propose a dynamic runtime algorithm that finds a mapping solution considering the characteristics of LLM operations and the change of footprint during LLM inference. Third, we advocate the need for memory abstraction for the efficient management of the asymmetric memory. H2M2 outperforms the conventional homogeneous memory system with LPDDR by 1.46x, 1.55x, and 2.94x speedup in GPT3-175B, Chinchilla-70B, and Llama2-70B, respectively.

Figures

Figures reproduced from arXiv: 2504.14893 by the authors.

Figure 1
Figure 1. Accelerator substrate used by H2M2. the goal, this paper investigates a heterogeneous memory ar￾chitecture with the capacity-centric memory module and the bandwidth-centric memory module. However, the strict hi￾erarchical organization of conventional heterogeneous mem￾ory which uses the high bandwidth memory as the first level memory backed by the low bandwidth memory is not suitable for LLMs, as the low locality of… view at source ↗
Figure 2
Figure 2. The topology of the decoder layer in GPT3. Orange boxes with W labels represent the weight parameter tensors, blue boxes represent the input activation tensors, and gray boxes represent the KV cache tensors. 256 512 768 1024 1280 1536 1792 2048 Sequence Length 0 20 40 60 80 100 Breakdown (%) qkv-linear attention fc Footprint 0 128 256 384 512 Footprint (GB) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Footprint breakdown for batch size 32, sequence length growing from 256 to 2048 in GPT3-175B. The portion of attention increases due to the increase of KV cache size. of this paper, we classify these decoder kernels into three groups: qkv-linear, attention, and fc layers. During the com￾putation of transformer decoder, the embedding vector T is first passed to qkv-linear sublayer to generate LLM contexts: Query (Q),… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Four possible configurations of memory systems applicable for LLM inference acceleration. devices [2, 4]. Moreover, although incorporating hundreds of gigabytes of HBM into a single module would satisfy both memory demands of capacity and bandwidth for LLMs, this appro…
Figure 6
Figure 6. Figure 6: Comparison of the performance of the asymmetric memory with two mapping granularity options in GPT3- 175B. A synchronization barrier ensures computational correct￾ness after each kernel. Furthermore, kernels from multiple heads mapped to the same memory module can be f…
Figure 5
Figure 5. Figure 5: Techniques for supporting optimal mapping gran￾ularity with asymmetric memory, for a single decoder layer. For clarity in explanation, a simplified view of a decoder layer is presented, visualizing GEMM/GEMV kernels mainly. cache parallelism, where decoder layers consi…
Figure 7
Figure 7. Figure 7: The relative speedup over the baseline with two mapping policies of asymmetric memory in GPT3-175B, batch size 32. FlexGen follows the mapping found by the model of Equation 1, and Best follows the mapping decision found by exhaustive search with 𝑁 3 times profiling […
Figure 9
Figure 9. Figure 9: The need for dynamic memory management in asymmetric memory systems: three motivations for memory abstraction to decouple the logical address space from the physical address space. proportional to the batch size and sequence length, as these values are directly related…
Figure 10
Figure 10. Figure 10: Overview of H2M2 with hardware substrates. the HBM and LPDDR sides may have different contents, as the same virtual page containing weights can be duplicated in both HBM and LPDDR. Dynamic mapping support: The memory mapping can change for two major reasons. First, th…
Figure 11
Figure 11. Figure 11: Architecture of the accelerator core: (a) Overview, (b) Matrix-Matrix multiplication unit, and (c) Matrix-Vector multiplication unit. An accelerator chip of each memory module includes 4 cores. not huge for the LLM accelerator, as the entire capacity is bounded by the…
Figure 12
Figure 12. Figure 12: The relative speedup over the baseline (LPDDR￾only) for hierarchical memory and asymmetric memory in GPT3-175B, batch size 32. batch size and sequence length [13]. In this section, we fix the batch size for each model: 32 for GPT3, 64 for Chinchilla, and 128 for Llama…
Figure 14
Figure 14. Figure 14: HBM footprint breakdown of H2M2 in GPT3- 175B, batch size 32. that H2M2 is a promising cost-efficient solution for LLM acceleration. Chinchilla-70B [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: illustrates the impact of GQA on utilizing the asymmetric memory architecture. We use a batch size of 128, and the x-axis and y-axis respectively represent the speedup over the baseline and the sequence length. Note that a heterogeneous memory architecture becomes mea…
Figure 16
Figure 16. Figure 16: The relative speedup over the baseline for asym￾metric memory, in dynamic sequence length scenario in GPT3-175B. Name Component Configuration Original No modification HBMcap-Less HBM capacity 48GB (0.5×) HBMcap-More HBM capacity 192GB (2×) HBMbw-Less HBM bandwidth 2.2…
Figure 18
Figure 18. Figure 18: The performance improvement of H2M2 and multi-HBM system (8-HBM) over the baseline (LPDDR-only) for GPT3-175B, batch size 32. LPDDR (LPDDRbw) and the computational power of LPDDR￾side accelerator (LPDDRChip) also shows some impacts on performance, as illustrated in […
Figure 19
Figure 19. Figure 19: The relative memory energy per token H2M2 and 8-HBM over the baseline (LPDDR-only) for GPT3-175B, batch size 32. Lower is better. speedup of 1.46× over the baseline, 8-HBM outperforms the baseline with an average speedup of 2.29×, which can be trasnlated into 1.57× ad…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SHIFT: Dynamic Compute Relocation Framework for Communication-Aware Chiplet-Based Systems

    cs.AR 2026-06 unverdicted novelty 6.0 of 10

    SHIFT is a topology-agnostic dynamic compute relocation framework for chiplet-based systems that reports 75.2-97.9% successful relocations, up to 76.8% latency reduction, 12.5x throughput gain, and 4.9x/5.9x/1.8x impr...

Reference graph

Works this paper leans on

56 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatam- ifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2023. Llm in a flash: Efficient large language model inference with limited memory. arXiv preprint arXiv:2312.11514 (2023)

    arXiv 2023

  2. [2]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis...

    2022

  3. [3]

    Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al

    Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. PaLM 2 Technical Report. arXiv preprint arXiv:2305.10403 (2023). arXiv:2305.10403 [cs.CL]

    arXiv 2023

  4. [4]

    Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. 2024. Distributed Inference and Fine-tuning of Large Language Models Over The Internet. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, 14 Hardware-based Heterogeneous Memory Management for Large Language Model Inference Grechen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, J...

    work page 2020

  6. [6]

    Yu-Hsin Chen, Joel Emer, and Vivienne Sze. 2016. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH computer architecture news 44, 3 (2016), 367–379

    work page 2016

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al . 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)

    arXiv 2022

  8. [8]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

Show all 56 references
  1. [9]

    Mingyu Gao, Jing Pu, Xuan Yang, Mark Horowitz, and Christos Kozyrakis. 2017. Tetris: Scalable and efficient neural network accelera- tion with 3d memory. InProceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operatin...

    2017

  2. [10]

    Saugata Ghose, Tianshi Li, Nastaran Hajinazar, Damla Senol Cali, and Onur Mutlu. 2019. Demystifying Complex Workload-DRAM Interactions: An Experimental Study. 3, 3 (2019)

    2019

  3. [11]

    Google. 2018. CloudTPU. https://cloud.google.com/tpu/docs/system- architecture-tpu-vm

    2018

  4. [12]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)

    2023 arXiv

  5. [13]

    Guseul Heo, Sangyeop Lee, Jaehong Cho, Hyunmin Choi, Sanghyeon Lee, Hyungkyu Ham, Gwangsun Kim, Divya Mahajan, and Jongse Park. 2024. NeuPIMs: A NPU-PIM Heterogeneous Acceleration for Batched Inference of Large Language Model. In Proceedings of the 29th ACM International Confe...

    2024

  6. [14]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  7. [15]

    Seongmin Hong, Seungjae Moon, Junsoo Kim, Sungjae Lee, Minsub Kim, Dongsoo Lee, and Joo-Young Kim. 2022. DFX: A Low-latency Multi-FPGA Applicance for Accelerating Transformer-based Text Gen- eration. In 2022 55th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)

    2022

  8. [16]

    Soojin Hwang, Sunho Lee, Jungwoo Kim, Hongbeen Kim, and Jaehyuk Huh. 2023. mNPUsim: Evaluating the Effect of Sharing Resources with Multi-core NPUs. In 2023 IEEE International Symposium on Workload Charcterization (IISWC)

    2023

  9. [17]

    Bongjoon Hyun, Youngeun Kwon, Yujeong Choi, John Kim, and Min- soo Rhu. 2020. NeuMMU: Architectural Support for Efficient Address Translations in Neural Processing Units. In Proceedings of the 25th ACM International Conference on Architectural Support for Program- ming Languag...

    2020

  10. [18]

    Hongsun Jang, Jaeyong Song, Jaewon Jung, Jaeyoung Park, Youngsok Kim, and Jinho Lee. 2024. Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System. In2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). I...

    2024

  11. [19]

    Jinwoo Jeong, Seungsu Baek, and Jeongseob Ahn. 2023. Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access. In Proceedings of the Eighteenth European Conference on Computer Systems (Rome, Italy) (EuroSys ’23). 249–265. https://doi.org/10.1145/ 3552326.3567508

    2023

  12. [20]

    Hongshin Jun, Jinhee Cho, Kangseol Lee, Ho-Young Son, Kwiwook Kim, Hanho Jin, and Keith Kim. 2017. HBM (High Bandwidth Mem- ory) DRAM Technology and Architecture. In 2017 IEEE International Memory Workshop

    2017

  13. [21]

    Sheng-Chun Kao, Suvinay Subramanian, Gaurav Agrawal, Amir Yaz- danbakhsh, and Tushar Krishna. 2023. FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and ...

    2023

  14. [22]

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catan- zaro. 2023. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems 5 (2023), 341– 353

    2023

  15. [23]

    Hyoukjun Kwon, Prasanth Chatarasi, Michael Pellauer, Angshuman Parashar, Vivek Sarkar, and Tushar Krishna. 2019. Understanding Reuse, Performance, and Hardware Cost of DNN Dataflow: A Data- Centric Approach. In Proceedings of the 52nd Annual IEEE/ACM Inter- national Symposium ...

    2019

  16. [24]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica

  17. [25]

    Sukhan Lee, Shin-haeng Kang, Jaehoon Lee, Hyeonsu Kim, Eojin Lee, Seungwoo Seo, Hosang Yoon, Seungwon Lee, Kyounghwan Lim, Hyun- sung Shin, Jinhyun Kim, O Seongil, Anand Iyer, David Wang, Kyomin Sohn, and Nam Sung Kim. 2021. Hardware Architecture and Software Stack for PIM Bas...

    2021

  18. [26]

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI)

    2024

  19. [27]

    Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing, Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. Lightseq: Sequence level parallelism for distributed training of long context transformers. arXiv preprint arXiv:2310.03294 (2023)

    2023 arXiv

  20. [28]

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. 2021. Sequence parallelism: Long sequence training from system perspective. arXiv preprint arXiv:2105.13120 (2021)

    2021 arXiv

  21. [29]

    S. Li, Z. Yang, D. Reddy, A. Srivastava, and B. Jacob. 2020. DRAMsim3: A Cycle-Accurate, Thermal-Capable DRAM Simulator. 19, 2 (2020), 106–109

    2020

  22. [30]

    NVIDIA. 2006. CUDA Toolkit Document. https://docs.nvidia.com/ cuda/cuda-runtime-api

    2006

  23. [31]

    NVIDIA. 2014. Unified Memory Programming. https://docs.nvidia.com/cuda/cuda-c-programming- guide/index.html#um-unified-memory-programminghd

    2014

  24. [32]

    NVIDIA. 2023. Grace Hopper Superchip. https://www.nvidia.com/en- us/data-center/grace-hopper-superchip/

    2023

  25. [33]

    NVIDIA. 2024. Blackwell Superchip. https://www.nvidia.com/en- us/data-center/gb200-nvl72/

    2024

  26. [34]

    OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303. 08774 (2023). arXiv:2303.08774 [cs.CL]

    2023 arXiv

  27. [35]

    Jaehyun Park, Jaewan Choi, Kwanhee Kyung, Michael Jaemin Kim, Yongsuk Kwon, Nam Sung Kim, and Jung Ho Ahn. 2024. AttAcc! 15 Soojin Hwang, Jungwoo Kim, Sanghyeon Lee, Hongbeen Kim, and Jaehyuk Huh Unleashing the Power of PIM for Batched Transformer-based Gener- ative Model Infe...

    2024

  28. [36]

    Sang-Soo Park, KyungSoo Kim, Jinin So, Jin Jung, Jonggeon Lee, Ky- oungwan Woo, Nayeon Kim, Younghyun Lee, Hyungyo Kim, Yongsuk Kwon, et al. 2024. An LPDDR-based CXL-PNM Platform for TCO- efficient Inference of Transformer-based Large Language Models. In 2024 IEEE Internationa...

    2024

  29. [37]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems 5 (2023)

    2023

  30. [38]

    Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ram- jee, and Ashish Panwar. 2024. vAttention: Dynamic Memory Man- agement for Serving LLMs without PagedAttention. arXiv preprint arXiv:2405.04437 (2024)

    2024 arXiv

  31. [39]

    Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. 2016. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MI- CRO). IEEE, 1–13

    2016

  32. [40]

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al . 2023. BLOOM: A 176B- Parameter Open-Access Multilingual Language Model. arXiv preprint arXiv:2211.05100 (2...

    2023 arXiv

  33. [41]

    Minseok Seo, Xuan Truong Nguyen, Seok Joong Hwang, Yongkee Kwon, Guhyun Kim, Chanwook Park, Ilkon Kim, Jaehan Park, Jeong- bin Kim, Woojae Shin, Jongsoon Won, Haerang Choi, Kyuyoung Kim, Daehan Kwon, Chunseok Jeong, Sangheon Lee, Yongseok Choi, Wooseok Byun, Seungcheol Baek, H...

    2024

  34. [42]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. 2023. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. In Proceedings of the 40th International Confer...

    2023

  35. [43]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  36. [44]

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. In Proceedings of the 30th Symposium on Operating Systems Principles (SOSP)

    2024

  37. [45]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  38. [46]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Alma- hairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 arXiv

  39. [47]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In 31st Conference on Neural Information Processing Systems (NIPS 2017)

    2017

  40. [48]

    Yannan Nellie Wu, Joel S Emer, and Vivienne Sze. 2019. Accelergy: An architecture-level energy estimation methodology for accelerator designs. In 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). 1–8

    2019

  41. [49]

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22) . 521–538

    2022

  42. [50]

    Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wenheng Ma, Hanbo Sun, Shiyao Li, Zixiao Huang, et al. 2024. FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGA. arXiv preprint arXiv:2401.03868 (2024)

    2024 arXiv

  43. [51]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Vic- toria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer....

    2022 arXiv

  44. [52]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2023 arXiv

  45. [53]

    Youpeng Zhao, Di Wu, and Jun Wang. 2024. ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)

    2024

  46. [54]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Au- tomating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In 16th USE...

    2022

  47. [2022]

    Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    2022

  48. [2023]

    In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP’23)

    Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (Koblenz, Germany) (SOSP’23). https: //doi.org/10.1145/3600006.3613165

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.