Pith. sign in

REVIEW 4 major objections 5 minor 46 references

This paper argues that stacking large M3D caches directly above the logic layer can cut LLM serving energy by up to 44 percent, with the gain limited to workloads that reuse data enough to amortize the cache's own access energy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:36 UTC pith:N3MSYRNT

load-bearing objection LLMET is a genuinely useful new simulator for a question that matters, but the headline energy numbers rest on co-authored M3D circuit values that are not externally validated—send it to referees, with sensitivity analysis and code required. the 4 major comments →

arxiv 2607.26491 v1 pith:N3MSYRNT submitted 2026-07-29 cs.AR cs.AIcs.ETcs.LG

LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving

Ming-Yen Lee , Hanchen Yang , Faaiq Waqar , Harsono Simka , Tushar Krishna , Muhammed Ahosan Ul Karim , Shimeng Yu This is my paper
classification cs.AR cs.AIcs.ETcs.LG
keywords LLM servingenergy efficiencymonolithic 3D memoryM3Dcache hierarchycross-layer simulationprefilldecode
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether radically enlarging on-chip cache using emerging monolithic 3D (M3D) memories can make LLM serving more energy-efficient. It claims that expanding the L2 cache from 40MB to 1GB with M3D memory reduces chip energy by 44% during the prefill phase for Llama 3.1 70B at 16K context, with smaller gains on larger platforms and in edge decode. The authors build a cross-layer simulator, LLMET, that propagates device-level memory energy and area numbers all the way to system-level serving energy, so the benefits and costs of GB-scale caches can be studied before the technology is fabricated. The overall answer is a qualified yes: large M3D caches help, but only within a workload- and technology-dependent regime.

Core claim

LLMET is a cross-layer simulation framework that carries device-calibrated models of emerging M3D two-transistor gain-cell (2T-GC) memories through the full stack—operator mapping, cache hierarchy, per-component energy, and chip area—making it possible to quantify the energy impact of GB-scale on-chip caches before the technology exists. Using LLMET, the paper shows that growing a 40MB L2 to 1GB with M3D memory cuts prefill energy by 44% for Llama 3.1 70B on a dual A100 system, that extending L2 from 128MB to 4GB on a B200-like platform saves up to 24% for Llama 405B, and that edge decode energy falls by about 30% at 256MB and by 75–80% when the quantized model fits fully on chip. The mechan

What carries the argument

The central object is LLMET, a cross-layer simulation framework that couples a front-end execution trace (operator types, tile sizes, capacity-aware mapping choices, byte counts per memory level) with a back-end power, performance, and area model calibrated to device-level data for compute units, SRAM, and M3D 2T-GC memories. The load-bearing mechanism is capacity-aware mapping combined with operator fusion: given an L2 size, the simulator pins one operand on chip, chooses among four tiling regimes for matrix multiplication, and fuses attention stages so intermediate logits stay on chip, with grouped-query attention KV heads pinned across a group when they fit. This makes off-chip traffic a

Load-bearing premise

The paper's energy numbers assume the per-bit access energy and area of M3D 2T-GC memories at 7nm and 3nm (0.77–1.08 pJ/bit) are accurate, even though such memories have not been fabricated at full capacity; if actual access energy, refresh overhead, or thermal penalties are higher, the claimed savings shrink or disappear.

What would settle it

Fabricate a fully integrated GB-scale M3D 2T-GC cache at 7nm and measure its per-bit access energy and refresh overhead; if the average access energy exceeds roughly 2 pJ/bit or requires frequent refresh, the claimed 44% prefill energy savings on the A100 platform do not hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • For server prefill, there is an optimal cache size per workload—roughly 256MB–1GB for 16K–64K contexts on Llama-70B and 2–4GB for Llama-405B beyond 16K—and larger caches yield diminishing or negative returns.
  • A memory technology's access energy sets the break-even point for ultra-large caches: M3D 2T-GC stays far below HBM energy, but a technology approaching HBM's per-bit energy would lose the benefit.
  • On edge devices, full model residency (about 512MB for a 1B INT4 model) can cut decode DRAM traffic by over 90% and decode energy by 75–80%, though this capacity is an upper-bound target under realistic area constraints.
  • Phase-disaggregated serving architectures can harvest more of the benefit, since prefill is where the energy savings concentrate, while decode shows smaller and more capacity-sensitive gains.
  • The per-component energy breakdown from LLMET lets designers see exactly where memory energy moves as cache grows, enabling cache sizing that optimizes end-to-end phase composition rather than either phase in isolation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The workload-dependent 'knee' in energy savings suggests future chips might expose a configurable M3D cache that software could power-gate or resize per inference phase, rather than fixing one capacity at design time.
  • The same cross-layer evaluation could be applied to other emerging on-chip memories (ferroelectric, MRAM, etc.) whenever device models exist; the paper's break-even criterion—access energy far below HBM—gives a quick screen for any candidate technology.
  • Because the edge prefill and decode respond oppositely to larger caches, a mixed-capacity design—small fast SRAM for latency-critical decode plus a large M3D region for weights—might beat a single uniform cache for edge deployments.
  • If thermal effects of dense M3D reads and writes in a real accelerator cause throttling, the energy numbers could shift; a thermal-aware extension of the framework would test this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents LLMET, a cross-layer simulation framework for evaluating the system-level energy impact of emerging monolithic-3D (M3D) on-chip memories on LLM serving. The front-end produces an operator trace with capacity-aware tiling and attention fusion; the back-end uses ASAP7, NS-Cache, and NeuroSim to estimate PPA. The framework is validated against NVIDIA A100 die area within 7%. Three platforms are explored: dual-A100 with Llama-3.1 70B, an 8x B200-like 3nm platform with Llama-3.1 405B, and a Jetson-class edge platform with Llama-3.2 1B INT4. The headline results are a 44% prefill energy reduction on A100 when L2 is expanded from 40MB SRAM to 1GB M3D 2T-GC, a 24% reduction on the B200-like platform when L2 is expanded from 128MB to 4GB, and up to a 30% decode-energy reduction on the edge platform when L2 is expanded from 8MB to 256MB. The paper concludes with a qualified yes: M3D memories can help, but only in workload- and technology-dependent regimes.

Significance. If the framework and its inputs are trustworthy, this is a useful contribution. LLMET is, to my knowledge, the first LLM serving simulator that integrates device-level M3D memory parameters into system-level energy accounting, and it includes genuinely useful components: capacity-aware mapping, attention fusion, per-component energy breakdowns, and a broad workload/platform sweep. The A100 area validation is a real, independent check, and the paper does not fit parameters to the target energy savings. The qualitative insights — that access-energy ratio sets the break-even point, that benefits saturate beyond a working-set knee, and that edge prefill and decode respond oppositely to cache scaling — are plausible and well supported by the trends. The main weakness is that the central energy results depend on M3D 2T-GC per-bit energy and area values drawn from co-authored prior circuit models for a technology that has not been fabricated at full capacity, with no sensitivity analysis or measured-power calibration. That makes the quantitative claims — especially the 44% headline — less robust than the paper's 'validated' framing suggests.

major comments (4)
  1. [§3.3, Table 2; Insight 2] The M3D 2T-GC per-bit access energies (0.768–1.08 pJ/bit) are load-bearing: the 44% saving results from trading HBM accesses at 5.74–6.6 pJ/bit for L2 accesses at up to 1.08 pJ/bit. These values come from NS-Cache [37]/CMOS+X [38], co-authored prior work on a technology not yet fabricated at full capacity. The A100 validation in §4.1 uses 2D SRAM only, so it does not constrain this input. If effective 2T-GC energy rises to 2–4 pJ/bit (refresh, thermal, 3D parasitics), Insight 2's break-even condition can invert and the claimed savings shrink or vanish. Please add a sensitivity sweep over M3D and SRAM per-bit energies (e.g., 0.5–4 pJ/bit) and report the resulting savings surfaces. Without this, the central quantitative claim is not robust to input uncertainty.
  2. [§4.1, Fig. 4] The validation is area-only: total die area and core area within 7% for an A100 whose caches are 2D SRAM. There is no comparison of energy components (DRAM, L2, compute) against measured A100 power or energy. The abstract and §3 call LLMET a 'validated cross-layer simulation framework,' which overstates what Fig. 4 demonstrates. The energy results should be labeled as simulation projections, not validated measurements, and ideally calibrated against at least one measured energy/power point before the framework is described as validated for energy.
  3. [§4.2, Fig. 6] The paper itself notes that HBM traffic fluctuates for batch sizes 2–8 because 'some FFN layers under 128MB and 256MB caches still rely on heuristic mapping, so the selected tiling may not always minimize traffic.' Since the mapping heuristic directly determines off-chip traffic and therefore energy, this is not a cosmetic artifact. Please quantify the sensitivity of the energy savings to the mapping search — for example, compare against an optimal or exhaustive tiling for a subset of layers, or provide upper/lower bounds on the savings. This matters for the capacity recommendations in Insight 1, which assume the mapping is representative.
  4. [§4.3, Table 2; Fig. 8] The B200-like 3nm platform uses projected PPA parameters from NeuroSim/IRDS with no error bars or sensitivity analysis. The 24% saving is smaller than the A100 case and depends on the 3nm 2T-GC values (0.768 pJ/bit at 1GB, 0.981 pJ/bit at 4GB) staying far below HBM3E (5.74 pJ/bit). A sensitivity sweep over the 3nm assumptions — e.g., ±30–50% on per-bit energy, area, and the Blackwell extrapolations — would establish whether the technology-scaling conclusion survives parameter uncertainty. Without it, the claim that 2–4GB is optimal for 16K–64K sequences is not supported beyond a single point estimate.
minor comments (5)
  1. [Figure 1, caption] The annotation '±P/±D' is confusing: the markers are described as left=prefill and right=decode, but the caption says 'annotated as ±P/±D, where the first value is the prefill and the second is the decode total-energy reduction (%).' Please clarify the sign convention and marker positions.
  2. [Figure 6, caption] The caption repeats 'Batch Size=1' under both figures when the right panel is batch-size sweep; the labeled conditions do not match the plot. Please correct.
  3. [§4.4, Figure 9] Prefill and decode panels use different y-axis ranges. Consider using a shared axis or adding explicit annotations to make cross-phase comparison easier.
  4. [Availability] No artifacts or code release is mentioned. Given the dependence on in-house models and the heuristic mapping, releasing the simulator and configuration files would greatly improve reproducibility. Please add an artifact availability statement.
  5. [Reference [22]] Reference [22] is cited as 'IEEE Micro (2026)' without a DOI or arXiv link. If it is not yet publicly available, provide a preprint identifier or state its status.

Circularity Check

0 steps flagged

No definitional circularity: energy savings are simulation outputs from supplied device parameters, not fitted inputs; co-authored M3D PPA values are a robustness concern, not a circular reduction.

full rationale

LLMET's headline reductions (44%/24%/30%) are produced by a forward simulation: cache-aware mapping and operator fusion change the HBM/L2 traffic mix, and the per-bit energies in Table 2 (SRAM, 2T-GC, HBM) are inputs, not parameters fitted to the target savings. The A100 die-area validation (Section 4.1, within 7%) is a genuinely external check of the architectural model, and it does not constrain the M3D energy path. The load-bearing M3D 2T-GC energy/area values are taken from co-authored NS-Cache [37] and CMOS+X [38] work, and the paper itself says this enables 'forward-looking evaluation of memory technologies that have not yet been fabricated at full capacity' (Section 3). That is an evidence-quality and robustness limitation: if actual BEOL 2T-GC energy is higher, the savings shrink or invert, and a sensitivity sweep or code release would be needed. But this is not circularity in the derivation-chain sense: no equation reduces to another by construction, no fitted parameter is renamed as a prediction, and the cited circuit models have device/circuit/stacking assumptions that are not defined in terms of the LLM energy results. Insight 2 restates the assumed energy margin, but the headline result also depends on an independently derived traffic and mapping model. Therefore no significant circularity is present; score 0 reflects that no circular step was identified.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

LLMET introduces no new physical entity; it is a simulation framework. The load-bearing inputs are technology PPA numbers taken from co-authored prior models and reference data, plus the assumption that those models extrapolate to un-fabricated GB-scale M3D caches. The free parameters listed are not fitted to the target energy savings, but they are hand-chosen inputs that the central quantitative claims depend on.

free parameters (4)
  • M3D 2T-GC per-bit access energy = 0.768–1.08 pJ/bit (7nm/3nm)
    Input from co-authored NS-Cache model [37]; load-bearing for all energy savings claims because the savings depend on on-chip access energy staying far below HBM.
  • 2D SRAM per-bit access energy at GB scale = 2.42–2.78 pJ/bit
    Used to argue planar SRAM scaling is energy- and area-expensive; from the same NS-Cache flow. Affects the comparison baseline and the 'M3D is better than SRAM' conclusion.
  • HBM/HBM3E/LPDDR5 per-bit access energy = 5.74/6.6 pJ/bit
    Taken from foundry/reference values [25,32]; the margin between DRAM and on-chip access energy determines the magnitude of savings. Treated as input, not measured in this paper.
  • 3nm B200-like platform PPA parameters = Not fully disclosed
    Extrapolated from Blackwell disclosures and NeuroSim/IRDS projections; directly controls the 24% B200-like prefill saving and the optimal 2–4 GB capacity recommendation.
axioms (5)
  • domain assumption M3D 2T-GC memory at 128 MB per layer can be integrated in BEOL without thermal, reliability, or yield degradation at GB scale.
    Section 2.2 and 3.3 assume foundry milestones [8,40] scale to full capacity; no fabricated GB-scale M3D cache exists yet to confirm energy and area at that size.
  • domain assumption The front-end built on LLMCompass [42], calibrated to A100, transfers correctly to B200-like and Jetson-like platforms with modified memory parameters.
    Section 4.1 validates area only on A100; Sections 4.2–4.4 extend the same front-end to very different hardware without an independent correctness check.
  • domain assumption The capacity-aware mapping and operator-fusion policy in LLMET represents real compiler/runtime scheduling choices.
    Section 3.2 introduces four mapping regimes and fusion heuristics; the paper admits in Section 4.2 that some FFN layers rely on heuristic mapping that can produce traffic fluctuations, so the policy is not guaranteed optimal or production-equivalent.
  • ad hoc to paper Area validation implies sufficient energy-model validity.
    Section 4.1 validates die area within 7%, but the central claims are about energy; the paper does not validate energy against any measured GPU power.
  • domain assumption Per-bit HBM/LPDDR energy values from [25,32] apply to the modeled A100, B200-like, and edge platforms.
    Table 2 uses fixed per-bit DRAM energies across platforms and does not model traffic pattern-dependent DRAM energy, refresh, or bandwidth contention.

pith-pipeline@v1.3.0-daily-deepseek · 11932 in / 12144 out tokens · 103980 ms · 2026-08-01T14:36:39.124060+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving." pith.science (2026). https://pith.science/paper/N3MSYRNT

@misc{pith2026260726491,
  author       = {Pith},
  title        = {Pith review of: LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3MSYRNT}},
  note         = {Machine review of arXiv:2607.26491}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs. A key contributor to chip energy dissipation is data movement between limited on-chip cache and off-chip High Bandwidth Memory (HBM). Meanwhile, emerging memory technologies such as monolithic 3D (M3D) integration of cache memories at the Back-End-Of-Line (BEOL) of logic chips enable larger and denser on-chip memories, creating new opportunities to reduce costly off-chip traffic. However, it remains unclear whether continuously scaling on-chip memory using emerging technologies can effectively improve the energy efficiency of LLM serving. To address this gap, we develop LLMET (LLM with Emerging Technology), a validated cross-layer simulation framework, and conduct a comprehensive study on the impact of large-capacity on-chip memory technologies across a broad range of models, applications and platforms. Utilizing M3D technology to expand the L2 cache from 40MB to 1GB yields a 44% reduction in chip energy during the Llama3.1-70B prefill phase with a 16K context window, based on LLMET simulation on a dual NVIDIA A100 GPU setup. On the 8x NVIDIA B200-like platform, extending the L2 cache from 128MB to 4GB saves the prefill energy by up to 24%. For the edge platform and workloads, the decode energy saving reaches 30% when increasing the 8MB cache size to 256MB. These results highlight the promise of ultra-large on-chip memories for energy-efficient LLM serving systems.

Figures

Figures reproduced from arXiv: 2607.26491 by Faaiq Waqar, Hanchen Yang, Harsono Simka, Ming-Yen Lee, Muhammed Ahosan Ul Karim, Shimeng Yu, Tushar Krishna.

Figure 2
Figure 2. Figure 2: AI accelerators with large Monolithic 3D (M3D) on [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Impact of large on-chip memory across LLM appli [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) LLMET overview. (b) Cache-aware mapping in LLMET. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A100 GPU die area breakdown of different cache [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 8
Figure 8. Figure 8: B200-like: relative HBM access and overall energy [PITH_FULL_IMAGE:figures/full_fig_p005_8.png] view at source ↗
Figure 6
Figure 6. Figure 6: A100: Relative HBM access and overall energy re [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Llama3.1 70B prefill energy breakdown (one layer) [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Edge accelerator: relative DRAM access and overall [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Edge accelerator: per-component energy break [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 4 linked inside Pith

  1. [1]

    Agrawal et al

    A. Agrawal et al. 2024. Vidur: A Large-Scale Simulation Framework for LLM Inference.Proceedings of Machine Learning and Systems6 (2024), 351–366

    2024

  2. [2]

    Agrawal, N

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and R. Ramjee. 2024. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 117–134

    2024

  3. [3]

    Ainslie, J

    J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai

  4. [4]

    Bambhaniya et al

    A. Bambhaniya et al. 2024. Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM Models.arXiv preprint arXiv:2406.01698 (2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Ravi Bhargava and Kai Troester. 2024. AMD next-generation “Zen 4” core and 4th gen AMD EPYC server CPUs.IEEE Micro44, 3 (2024), 8–17

    2024

  6. [6]

    T. B. Brown, B. Mann, N. Ryder, et al . 2020. Language Models Are Few-Shot Learners. InAdvances in Neural Information Processing Systems (NeurIPS). 1877– 1901

    2020

  7. [7]

    Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze. 2017. Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks.IEEE Journal of Solid-State Circuits52, 1 (2017), 127–138

    2017

  8. [8]

    K. H. Chiang et al. 2025. Integration of 0.75V VDD Oxide-Semiconductor 1T1C Memory with Advanced Logic for An Ultra-Low-Power Low-Latency Cache Solution. In2025 IEEE Symposium on VLSI Technology and Circuits. Kyoto, Japan

    2025

  9. [9]

    J. Cho, H. Choi, and J. Park. 2025. LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure.IEEE Computer Architecture Letters24, 2 (July–Dec. 2025), 361–364

    2025

  10. [10]

    L. T. Clark et al. 2016. ASAP7: A 7-nm FinFET Predictive Process Design Kit. Microelectronics Journal53 (2016), 105–115

    2016

  11. [11]

    T. Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.arXiv preprint arXiv:2307.08691(2023)

    Pith/arXiv arXiv 2023

  12. [12]

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems (NeurIPS). 16344–16359

    2022

  13. [13]

    Frantar, S

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. 2023. GPTQ: Accurate Post- Training Quantization for Generative Pre-Trained Transformers. InInternational Conference on Learning Representations (ICLR)

    2023

  14. [14]

    Gholami, Z

    A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer. 2024. AI and Memory Wall.IEEE Micro44, 3 (2024), 33–39

    2024

  15. [15]

    Grattafiori, A

    A. Grattafiori, A. Dubey, et al. 2024. The Llama 3 Herd of Models.arXiv preprint arXiv:2407.21783(2024)

    Pith/arXiv arXiv 2024

  16. [16]

    Horowitz

    M. Horowitz. 2014. 1.1 Computing’s Energy Problem (and What We Can Do About It). In2014 IEEE International Solid-State Circuits Conference (ISSCC). 10– 14

    2014

  17. [17]

    IEEE. 2024. International Roadmap for Devices and Systems: 2024 Edition. IEEE International Roadmap for Devices and Systems. https://irds.ieee.org/editions/ irds2024/

    2024

  18. [18]

    Isaev, N

    M. Isaev, N. McDonald, L. Dennison, and R. Vuduc. 2023. Calculon: A Methodol- ogy and Tool for High-Level Co-Design of Systems and Large Language Models. InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC). Denver, CO, USA, Article 71, 14 pages

    2023

  19. [19]

    N. P. Jouppi, C. Young, N. Patil, et al. 2017. In-Datacenter Performance Analysis of a Tensor Processing Unit. InProceedings of the 44th Annual International Symposium on Computer Architecture (ISCA). 1–12

    2017

  20. [20]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP). 611–626

    2023

  21. [21]

    Lee et al

    J. Lee et al . 2024. NeuroSim v1.4: Extending Technology Support for Digital Compute-in-Memory Toward 1nm Node.IEEE Transactions on Circuits and Systems I: Regular Papers71, 4 (2024), 1733–1744

    2024

  22. [22]

    Lee et al

    M.-Y. Lee et al. 2026. Architecting Long-Context LLM Acceleration with Packing- Prefetch Scheduler and Ultra-Large Capacity On-Chip Memories.IEEE Micro (2026)

    2026

  23. [23]

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han. 2024. AWQ: Activation-Aware Weight Quantization for On- Device LLM Compression and Acceleration. InProceedings of Machine Learning and Systems (MLSys)

    2024

  24. [24]

    G. H. Loh. 2008. 3D-Stacked Memory Architectures for Multi-Core Processors. In2008 International Symposium on Computer Architecture (ISCA). 453–464

    2008

  25. [25]

    Moon, H.-Y

    K.-I. Moon, H.-Y. Son, and K. Lee. 2023. Advanced Packaging Technologies in Memory Applications for Future Generative AI Era. In2023 International Electron Devices Meeting (IEDM). IEEE

    2023

  26. [26]

    NVIDIA Corp. 2024. NVIDIA DGX B200 Datasheet. NVIDIA. https://resources. nvidia.com/en-us-dgx-systems/dgx-b200-datasheet?ncid=no-ncid

    2024

  27. [27]

    2020.NVIDIA A100 Tensor Core GPU Architecture

    NVIDIA Corporation. 2020.NVIDIA A100 Tensor Core GPU Architecture. Technical Report. NVIDIA. https://images.nvidia.com/aem-dam/en-zz/Solutions/data- center/nvidia-ampere-architecture-whitepaper.pdf

    2020

  28. [28]

    NVIDIA Corporation. 2025. NVIDIA Blackwell Tuning Guide. NVIDIA Docu- mentation. https://docs.nvidia.com/cuda/blackwell-tuning-guide/index.html

    2025

  29. [29]

    NVIDIA Corporation. 2026. NVIDIA Jetson AGX Orin for Next-Gen Robotics. NVIDIA. https://www.nvidia.com/en-us/autonomous-machines/embedded- systems/jetson-orin/

    2026

  30. [30]

    Parashar, P

    A. Parashar, P. Raina, Y. S. Shao, Y.-H. Chen, V. A. Ying, A. Mukkara, R. Venkate- san, B. Khailany, S. W. Keckler, and J. Emer. 2019. Timeloop: A Systematic Approach to DNN Accelerator Evaluation. In2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 304–315

    2019

  31. [31]

    Patel, E

    P. Patel, E. Choukse, C. Zhang, A. Shah, Í. Goiri, S. Maleki, and R. Bianchini

  32. [32]

    Thomas Pawlowski

    J. Thomas Pawlowski. 2019. Prospects for Memory. MCHPC Keynote Pre- sentation. Micron Technology. https://passlab.github.io/mchpc/mchpc2019/ presentations/MCHPC_Pawlowski_keynote.pdf

    2019

  33. [33]

    R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean. 2023. Efficiently Scaling Transformer Inference. In Proceedings of Machine Learning and Systems (MLSys)

    2023

  34. [34]

    N. Shazeer. 2019. Fast Transformer Decoding: One Write-Head Is All You Need. arXiv preprint arXiv:1911.02150(2019)

    Pith/arXiv arXiv 2019

  35. [35]

    Stojkovic et al

    J. Stojkovic et al . 2025. Dynamollm: Designing LLM Inference Clusters for Performance and Energy Efficiency. In2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE

    2025

  36. [36]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention Is All You Need. InAdvances in Neural Information Processing Systems (NeurIPS). 5998–6008

    2017

  37. [37]

    Waqar et al

    F. Waqar et al. 2026. Optimization and Benchmarking of Monolithically Stackable Gain Cell Memory for Last-Level Cache.IEEE Trans. Comput.75, 3 (March 2026), 760–775

    2026

  38. [38]

    Waqar, M.-Y

    F. Waqar, M.-Y. Lee, S. Yoon, S. Lim, and S. Yu. 2025. CMOS+X: Stacking Persistent Embedded Memories Based on Oxide Transistors upon GPGPU Platforms. In Proceedings of the International Symposium on Memory Systems. 64–77

    2025

  39. [39]

    G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. InInternational Conference on Learning Representations (ICLR)

    2024

  40. [40]

    Yang et al

    J.-E. Yang et al. 2024. A-IGZO FETs with High Current and Remarkable Stability for Vertical Channel Transistor (VCT) / 3D DRAM Applications. In2024 IEEE Symposium on VLSI Technology and Circuits. Honolulu, HI, USA, 1–2

    2024

  41. [41]

    G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun. 2022. Orca: A Distributed Serving System for Transformer-Based Generative Models. In16th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI). 521–538

    2022

  42. [42]

    Zhang, A

    H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff. 2024. LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 1080–1096

    2024

  43. [43]

    Zhang, Y

    Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang, and B. Chen. 2023. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. InAdvances in Neural Information Processing Systems (NeurIPS). 34661–34710

    2023

  44. [44]

    Zhong, S

    Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang. 2024. DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving. In18th USENIX Symposium on Operating Systems Design and Implementation (OSDI). 193–210

    2024

  45. [2023]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    GQA: Training Generalized Multi-Query Transformer Models from Multi- Head Checkpoints. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 4895–4901

    2023

  46. [2024]

    In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA)

    Splitwise: Efficient Generative LLM Inference Using Phase Splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). 118–132

    2024