Pith. sign in

REVIEW 4 major objections 6 minor 68 references

Rethinking Unified Memory for NPU-PIM Systems: Dual-View Memory for Dynamic Inference of LLM

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PFM: a dual-view memory design that lets NPU and PIM share one physical layout with device-specific logical views, enabling dynamic LLM inference without duplication or relayout.

desk verdict A genuinely new mechanism for NPU-PIM memory, with a thorough evaluation, but Eq. (7) has a units error that biases the mapping search; numbers should not be trusted until that is corrected. read the letter →

arxiv 2608.06989 v1 pith:CLUQCELE submitted 2026-08-07 cs.AR

Shixin Zhao , Lian Liu , Tianhua Han , Mengdi Wang , Yinhe Han , Ying Wang This is my paper · ORCID
classification cs.AR
keywords processing-in-memoryNPU-PIMsystemsunifiedmemoryLLMinferencedual-viewaddressremappingschedulingMixture-of-Experts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing unified memory designs for NPU-PIM systems fail on modern LLM inference because they bind each tensor to one device at allocation time, while LLM workloads like MoE routing and prefill/decode shifts make the best device change at runtime. It proposes PFM, which stores each tensor once in a jointly optimized physical layout and gives NPU and PIM different logical views of the same data, so both can access it at high bandwidth without duplication or relayout. A memory-controller address remapping unit and flexible scheduler implement the dual views, and offline multi-objective optimization over 2 MB superpages chooses the physical mapping. The paper reports up to 2.32x end-to-end throughput over the best prior unified memory baseline and near-peak bandwidth utilization on both devices. If right, PFM removes the static-allocation assumption that limits NPU-PIM memory systems for dynamic LLM serving.

What carries the argument

The load-bearing object is the per-superpage address-bit assignment matrix $X \in \{0,1\}^{m\times n}$, which says which physical-address bits go to which DRAM level (channel, bank group, bank, row, column). PFM solves $\max_f \text{NPU}(f)+\alpha\,\text{PIM}(f)$ for each tensor's 2 MB superpage, where $\text{NPU}(f)$ estimates achievable bandwidth from row-activation and column counts and $\text{PIM}(f)$ estimates bank-level parallelism gain from the number of active banks and intra-bank access time. The resulting mapping is stored as a MapID descriptor, and at runtime the Address Remapping Unit applies it to translate physical addresses into hardware addresses in an accessor-specific way. The paper's case study shows the optimizer placing channel bits low and row bits high so a single Mixtral expert weight page is channel-parallel for the NPU and bank-local for PIM.

What would settle it

Measure, on real HBM-PIM hardware or a validated cycle-accurate simulator, the achieved DRAM bandwidth for the specific superpage mappings PFM's solver selects under NPU-only, PIM-only, and mixed NPU/PIM request streams, and compare those measurements with the model's predicted $B_{\rm NPU}$ and $B_{\rm PIM}$ values. If the balanced layouts systematically fall well short of the predicted 92.1% NPU and 94.85% PIM utilization, the mapping optimizer and the 2.32x end-to-end claim would not carry over to hardware.

Watch

Extended reading notes

Core claim

The central claim is that a fixed physical layout can serve two devices with conflicting address-mapping preferences if each device reads it through its own logical view. PFM stores LLM tensors in a physical layout chosen offline by solving a per-superpage multi-objective problem that balances the NPU's need for channel interleaving against the PIM's need for bank-local continuity and parallelism. A memory-controller Address Remapping Unit then translates the same physical address differently for NPU and PIM requests, and a Flexible Access Scheduler reorders NPU requests for channel concurrency and row hits while batching PIM requests for bank-parallel broadcasts. The paper reports that on Mixtral this yields 92.1% NPU and 94.85% PIM memory bandwidth utilization, and that across LLaMA3-8B, DeepSeekMoE-16B, Mixtral-8x7B, and GPT-OSS-120B it improves end-to-end throughput by up to 2.32x over the best prior unified-memory design (PUM), reaching 96.8% of an oracle's decode efficiency.

Load-bearing premise

The result depends on the assumption that the paper's equations for estimating memory bandwidth closely match what real HBM-PIM hardware delivers for the mixed NPU and PIM access patterns it chooses.

Editorial extensions

If this is right

  • If PFM is correct, LLM operators can be re-dispatched between NPU and PIM at decoding-step granularity without paying for data migration or layout conversion.
  • MoE expert weights and KV caches can occupy one shared physical address space and still deliver near-peak bandwidth to both NPU and PIM, which is what drives the 2.32x end-to-end throughput improvement.
  • The static device-biased mappings of prior unified memory designs leave most of the oracle speedup unrealized; FACIL captures only 29.5% of the oracle gain, while PFM's dual-view design reaches 96.8% of oracle efficiency in decode.
  • Because only the memory controller is modified, the same dual-view approach can be applied to HBM, GDDR, and LPDDR systems and is independent of the NPU architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the dual-view abstraction should transfer to any heterogeneous system where the same data is consumed by accessors with conflicting layout preferences, such as GPU-CPU or GPU-DPU pairs, not only NPU-PIM.
  • A testable extension is to close the loop online: instead of requiring offline re-profiling when expert routing shifts, the MapID table could be repopulated from measured access streams, turning the mapping search into a continuous optimizer.
  • The paper's offline solver takes up to 30 minutes per tensor, so serving many concurrently changing model variants would need cheaper heuristics or reuse of MapIDs across shape classes; the paper assumes limited map templates suffice for common deployments.
  • Because the runtime scheduler prioritizes NPU bursts and drains PIM during NPU compute windows, the design could also produce predictable QoS for latency-sensitive LLM serving, though the paper does not report tail-latency or multi-tenant contention results.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes PFM (PIM-as-Flexible-Memory), a dual-view memory system for NPU-PIM architectures that decouples physical data layout from accessor-visible logical views. PFM stores tensors in a single jointly optimized physical layout and exposes different address interpretations to the NPU and PIM, supported by an Address Remapping Unit (ARU) and a Flexible Access Scheduler (FAS). An offline mapping optimizer (Eq. 10) searches over address bit assignments, and a runtime roofline model (Eqs. 11-12) assigns each operator to the NPU or PIM. The paper evaluates PFM with a GPGPU-Sim/Ramulator simulator across LLaMA3, DeepSeekMoE, Mixtral, and GPT-OSS, reporting up to 2.32x throughput improvement over a PIM-friendly unified-memory baseline (PUM) and near-peak bandwidth utilization for both devices (92.1% NPU, 94.85% PIM).

Significance. The problem is timely and real: existing unified-memory designs assume static device-biased tensors, which mismatches the dynamic device preferences of MoE routing and prefill/decode phase changes in LLM inference. The dual-view concept is a plausible and potentially valuable alternative to data migration or duplication. The paper includes a detailed memory-controller design, an analytical model, a case study, and an area-overhead estimate, and it ships real MoE activation traces. However, the central quantitative claims rest on an analytical model that appears to contain a dimensional error, and on several offline-fitted parameters without sensitivity analysis. If these issues are corrected and the results remain robust, the contribution could be significant.

major comments (4)
  1. [IV-B, Eq. (7)] Eq. (7) defines T_ref with the first term divided by N_row (rows per bank), but the number of row activations per bank for a data volume of 2^{p_n+p_k}/#active_banks(f) elements should be that volume divided by N_col (elements per row). For the evaluated HBM configuration (2 MB page, 1024 banks, N_row ~16K, N_col ~32), the first term is undercounted by roughly N_row/N_col, i.e., about 512x. Since T_ref appears in the numerator of PIM(f) in Eq. (5), the PIM objective loses its intended dependence on per-bank data volume and becomes numerically negligible relative to NPU(f) in Eq. (10). The offline mapping search therefore is not solving the claimed joint dual-view optimization, and the selected layouts are not demonstrated to be jointly optimal. The Ramulator evaluation in Section VII-A cannot repair this because it only validates the timing of a given mapping. Please correct Eq. (7) (replace N_row with N_col) and re-run the mapping search; if the selected mappings and reported speedups are unchanged, state that explicitly with evidence.
  2. [V-B (Eqs. 11-12), IV-B4] The runtime device-selection model uses gamma and T_setup that are determined offline (Section V-B), and the mapping optimizer uses alpha chosen through profiling (Section IV-B4). The paper does not report the numerical values of alpha, gamma, and T_setup, nor any sensitivity analysis. Because the end-to-end speedups depend on the scheduler decisions made by this model, and because the simulator and the scheduling model share the same analytical model family, the reported gains could be partly an artifact of parameter fitting. Please report the fitted values and perform a sensitivity analysis (e.g., vary each parameter over a plausible range and report the resulting throughput). In addition, validate the predicted PIM execution times for the selected mappings directly against the detailed Ramulator timing model.
  3. [Abstract, VII-B] The abstract claims 'up to 2.32x end-to-end throughput over prior unified memory designs,' but Section VII-A states that in large-batch settings, the most flexible prior unified design (FACIL) is equivalent to PSM, and the speedup over PSM is up to 2.29x (Figure 13). The 2.32x figure is over PUM, an all-PIM-friendly layout that is not representative of the strongest prior unified design in large-batch settings. Please qualify the headline claim or present the comparison against the strongest baseline throughout the paper.
  4. [VII-A] The simulator is validated against real A100 measurements for the NPU side, but the HBM-PIM timing model is not validated against any real PIM hardware. Since the claimed 94.85% PIM bandwidth utilization (Table III) is a central result, please provide additional evidence that the modeled PIM timing constraints (bank-locking, SIMD width, command scheduling) match a published HBM-PIM implementation, or at least discuss how deviations from real hardware would affect the conclusions.
minor comments (6)
  1. [IV-B, Eq. (4)] The notation Ppn−1j=0 X_row,j would benefit from an explicit definition of X_row,j as the entry of the mapping matrix X corresponding to the row level and address bit j; currently the index convention is unclear.
  2. [VI-A, Figure 12] The bit ranges [11:7], [20:19], and similar in the text are not labeled directly on the mapping diagram; please add the bit ranges to the figure or caption for readability.
  3. [VII-A] The description of the PSM baseline as 'NeuPIMs-like' is vague; please state whether migration cost between the separated NPU and PIM regions is modeled, as this affects the baseline's absolute performance.
  4. [III-A] The 'Oracle' strategy is described only qualitatively; please specify whether it assumes zero address-translation and scheduling overhead and whether it can re-optimize the layout per operator without cost.
  5. [Abstract vs. IX] The abstract reports 'up to 2.32x' improvement while the conclusion reports 'up to 2.26x'; please reconcile these numbers.
  6. [VIII] The Limitations paragraph lists re-profiling and MDT size, but it does not mention the sensitivity of the results to the alpha/gamma/T_setup parameters; we recommend adding a sentence acknowledging this dependency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dual-view mapping and scheduling are validated by independent cycle-level simulation, not by the analytical objective used to select them.

full rationale

The paper's derivation chain is: (i) observe dynamic device-switching in LLM inference; (ii) formulate an offline mapping-search problem with analytical NPU/PIM objectives (Eqs. 1-10); (iii) implement the selected mapping in the ARU/FAS hardware; and (iv) evaluate through a Ramulator-based cycle-level simulation against external baselines. I find no step where a claimed prediction reduces to an input by construction. The mapping optimizer does maximize an objective NPU(f)+alpha*PIM(f) that resembles the bandwidth-utilization metrics reported in Table III, but the table's values are obtained from Ramulator replay with HBM-PIM timing, not from Eqs. (1)-(10), so the evaluation is an independent check of the model rather than a restatement of the objective. Similarly, the runtime scheduler's gamma and T_setup are offline-determined parameters; the paper does not describe fitting them to the reported end-to-end results, and the simulator includes detailed DRAM timing. The acknowledged limitation that PFM may require re-profiling when workloads change is a calibration and scoping caveat, not a circular step. No load-bearing self-citation or author-imported uniqueness theorem appears; prior-work citations (AttAcc, FACIL, IANUS, NeuPIMs) are external baselines. The apparent N_row/N_col dimension issue in Eq. (7) is a correctness or calibration risk in the analytical model, not a circularity: even if the model is miscalibrated, the mapping search and the final simulation are distinct computations. Under the hard rule requiring an explicit reduction, no circular step can be substantiated.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim depends on fitted parameters (alpha, gamma, T_setup), domain assumptions about DRAM layout preferences, and the accuracy of a custom analytical model. The only newly introduced hardware is assessed by synthesis, not fabricated silicon.

free parameters (3)
  • alpha (α) = not reported, chosen from operator profiling
    Weights NPU vs PIM objectives in Eq (10); tuned per operator based on measured arithmetic intensity.
  • gamma (γ) = not reported, determined offline
    Bank-level parallelism factor scaling PIM effective bandwidth in Eq (12); inferred from profiling.
  • T_setup = not reported, determined offline
    Fixed PIM command injection and bank-locking overhead in Eq (12); fit to simulation.
assumptions (4)
  • domain assumption NPUs prefer channel-interleaved layouts; PIM prefers bank-local continuity
    Section II-C states this as the basis for the conflict and the joint mapping targets.
  • domain assumption A 2 MB superpage spans all HBM channels and banks, enabling joint optimization
    Section III-B assumes HBM3 stack with 16 channels, 1024 banks, 2 KB per bank per superpage; not true for all configurations.
  • ad hoc to paper The analytical bandwidth model (Eqs. 1-9) accurately reflects real DRAM timing
    The model is used both to select mappings and to evaluate them; its accuracy is key but only validated indirectly.
  • ad hoc to paper Runtime roofline model (Eqs. 11-12) correctly predicts device execution times
    Used for operator-to-device scheduling; depends on fitted gamma and T_setup.
invented entities (2)
  • Address Remapping Unit (ARU)
    purpose: Translates physical addresses into device-specific hardware coordinates using per-page mapping descriptors
    Only RTL synthesis results shown; no silicon measurement.
  • Flexible Access Scheduler (FAS)
    purpose: Reorders NPU and PIM requests and arbitrates to maintain high bandwidth
    Only RTL synthesis results shown; no silicon measurement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Unified Memory for NPU-PIM Systems: Dual-View Memory for Dynamic Inference of LLM." pith.science (2026). https://pith.science/paper/CLUQCELE

@misc{pith2026260806989,
  author       = {Pith},
  title        = {Pith review of: Rethinking Unified Memory for NPU-PIM Systems: Dual-View Memory for Dynamic Inference of LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CLUQCELE}},
  note         = {Machine review of arXiv:2608.06989}
}
abstract

Heterogeneous architectures that combine neural processing unit (NPU) and processing-in-memory (PIM) are increasingly adopted to accelerate LLM inference. Prior work focuses on building a unified memory that allows NPUs and PIM to share data without duplication. However, these designs implicitly assume that each tensor is bound to a fixed execution device, and therefore rely on static, device-biased data mappings. We observe that this assumption does not hold in modern LLM workloads. Due to phase changes (e.g., prefill vs. decode) and dynamic behaviors such as MoE routing, the optimal execution device for the same tensor can change at runtime. Under such dynamic execution, device-biased mappings become mismatched to access patterns, leading to substantial bandwidth underutilization and performance loss. This paper presents PFM (PIM-as-Flexible-Memory), a dual-view memory system that decouples physical data layout from accessor-visible logical views. PFM stores data in a jointly optimized physical layout and exposes different logical interpretations to NPUs and PIM, enabling efficient access across devices without data duplication or relayout. We further design accessor-aware address translation and runtime scheduling mechanisms to support dynamic execution when LLM workloads fluctuate and the optimal execution device dynamically changes. Our evaluation across LLMs shows that PFM improves end-to-end throughput by up to 2.32$\times$, demonstrating its effectiveness and broad applicability as a unified memory management solution for NPU-PIM systems.

Figures

Figures reproduced from arXiv: 2608.06989 by the authors.

Figure 1
Figure 1. Motivation for PFM. (a) Step-varying expert intensity in MoE [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Heterogeneous NPU-PIM architecture. Row Bank Channel Column PU PU PU PU NPU-centric Row Column Bank Channel PIM-centric … … Predefined Address Mapping PA-to-HA PA Allocation: f(Shape, device)→mapping Tensor … … … … 0 PU PU PU PU … … … … … … … … 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 Bank 0 Bank 1 Bank 2 Bank 3 Bank 0 Bank 1 Bank 2 Bank 3 Channel 0 Channel N NPU Bank-level parallelism computing path of PIM … Channel-level cur… view at source ↗
Figure 2
Figure 2. LLM architecture and procedure. (a) Classic LLM architecture. (b) Dynamic expert activation behaviors during the decode step in the MoE model. (c) The architecture of MoE layers with the example of expert selection. 2) LLM Inference Procedure: LLM inference comprises two phases: the prefill phase for prompt processing and the autoregressive decode phase for token generation. Each phase consists of several core opera… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: LLM inference on NPU-PIM system. D. Unified Memory Management on NPU-PIM To address these inefficiencies, recent research has pursued unified memory management, enabling the NPU and PIM to share a physical memory space. The cornerstone of these unified systems is a lay…
Figure 7
Figure 7. Figure 7: The bandwidth under different mapping strategies when accessing an [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 9
Figure 9. Figure 9: Illustration of address mapping formulation. [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: The PFM controller path: dual-view hardware address translation and scheduling. [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: LLM inference workflow with PFM. enabling rapid dispatch for arbitration. Priority arbitration. The arbiter produces the joint sched￾ule. It first services mandatory DRAM maintenance (e.g., refresh) to preserve timing guarantees. When both queues are non-empty, it pri…
Figure 12
Figure 12. Figure 12: The case study of PFM design. (a) Illustration of the mapping strategy acquired by PFM’s dual-view mapping for one tensor in Mixtral-8×7B; Mapping strategies of (b) FACIL and (c) IANUS; (d) Illustration of weight tensor address with PFM’s mapping; (e) Tensor layout in…
Figure 13
Figure 13. Figure 13: Normalized end-to-end inference throughput (tokens/s) on LLaMA3, Mixtral, DSMoE and GPT-OSS. a means Lin = 1024, Lout = 128; b means Lin = 128, Lout = 1024. Norm. Latency 0 0.2 0.4 0.6 0.8 1 1 4 1 4 1 4 1 4 1 4 1 4 1 4 1 4 1,024 128 1,024 128 1,024 128 1,024 128 llama…
Figure 14
Figure 14. Figure 14: The normalized end-to-end inference latency with small batch sizes (BS=1 and 4). a means Lin = 1024, Lout = 128; b means Lin = 128, Lout = 1024. linear layers benefit from increased data reuse [41], [17], attention remains memory-bound due to batch-independent KV cach…
Figure 16
Figure 16. Figure 16: The normalized inference latency on MoE blocks. [PITH_FULL_IMAGE:figures/full_fig_p012_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 38 canonical work pages

  1. [1]

    cublas docs,

    “cublas docs,” 2024, https://docs.nvidia.com/cuda/cublas/index.html

    work page 2024

  2. [2]

    High bandwidth memory dram (hbm1, hbm2) jesd235d,

    J. S. S. T. Association, “High bandwidth memory dram (hbm1, hbm2) jesd235d,” 2020. [Online]. Available: https://www.jedec.org/ sites/default/files/docs/JESD235D.pdf

    work page 2020

  3. [3]

    Analyzing cuda workloads using a detailed gpu simulator,

    A. Bakhoda, G. L. Yuan, W. W. Fung, H. Wong, and T. M. Aamodt, “Analyzing cuda workloads using a detailed gpu simulator,” in2009 IEEE international symposium on performance analysis of systems and software. IEEE, 2009, pp. 163–174

    2009

  4. [4]

    Moe-lightning: High-throughput moe inference on memory-constrained gpus,

    S. Cao, S. Liu, T. Griggs, P. Schafhalter, X. Liu, Y . Sheng, J. E. Gon- zalez, M. Zaharia, and I. Stoica, “Moe-lightning: High-throughput moe inference on memory-constrained gpus,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, 2025, pp. 715–730

    2025

  5. [5]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalezet al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,”See https://vicuna. lmsys. org (accessed 14 April 2023), vol. 2, no. 3, p. 6, 2023

    2023

  6. [6]

    Palm: Scal- ing language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmannet al., “Palm: Scal- ing language modeling with pathways,”Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  7. [7]

    Asap7: A 7-nm finfet predictive process design kit,

    L. T. Clark, V . Vashishtha, L. Shifren, A. Gujja, S. Sinha, B. Cline, C. Ramamurthy, and G. Yeric, “Asap7: A 7-nm finfet predictive process design kit,”Microelectronics Journal, vol. 53, pp. 105–115, 2016

    2016

  8. [8]

    Projects – COIN-OR: Computational infrastruc- ture for operations research,

    COIN-OR Foundation, “Projects – COIN-OR: Computational infrastruc- ture for operations research,” https://www.coin-or.org/projects/, 2025

    work page 2025

Show all 68 references
  1. [9]

    Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experts language models,

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y . Wuet al., “Deepseekmoe: Towards ultimate expert special- ization in mixture-of-experts language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu...

    2024

  2. [10]

    The true processing in memory accelerator,

    F. Devaux, “The true processing in memory accelerator,” in2019 IEEE Hot Chips 31 Symposium (HCS). IEEE Computer Society, 2019, pp. 1–24

    2019

  3. [11]

    Accelerating llm inference throughput via asynchronous kv cache prefetching,

    Y . Dong, Y . Miao, W. Li, X. Zheng, C. Wang, and F. Lyu, “Accelerating llm inference throughput via asynchronous kv cache prefetching,”arXiv preprint arXiv:2504.06319, 2025, accessed: 2025-08-18

    2025

  4. [12]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    Klotski: Efficient mixture-of-expert inference via expert- aware multi-batch pipeline,

    Z. Fang, Y . Huang, Z. Hong, Y . Lyu, W. Chen, Y . Yu, F. Yu, and Z. Zheng, “Klotski: Efficient mixture-of-expert inference via expert- aware multi-batch pipeline,” inProceedings of the 30th ACM Interna- tional Conference on Architectural Support for Programming Languages and ...

    2025

  6. [14]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,”Journal of Machine Learning Research, vol. 23, no. 120, pp. 1–39, 2022

    2022

  7. [15]

    The future of low-latency memory: Why near memory requires a new interface,

    J. Handy and T. Coughlin, “The future of low-latency memory: Why near memory requires a new interface,” Objective Analysis and Coughlin Associates, Tech. Rep., 2021

    2021

  8. [16]

    Papi: Exploiting dynamic parallelism in large language model decoding with a processing-in-memory-enabled computing system,

    Y . He, H. Mao, C. Giannoula, M. Sadrosadati, J. G ´omez-Luna, H. Li, X. Li, Y . Wang, and O. Mutlu, “Papi: Exploiting dynamic parallelism in large language model decoding with a processing-in-memory-enabled computing system,” inProceedings of the 30th ACM International Confer...

    2025

  9. [17]

    Neupims: Npu-pim heterogeneous acceleration for batched llm inferencing,

    G. Heo, S. Lee, J. Cho, H. Choi, S. Lee, H. Ham, G. Kim, D. Ma- hajan, and J. Park, “Neupims: Npu-pim heterogeneous acceleration for batched llm inferencing,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operatin...

    2024

  10. [18]

    Lightllm: A versatile large language model for predictive light sensing,

    J. Hu, H. Jia, M. Hassan, L. Yao, B. Kusy, and W. Hu, “Lightllm: A versatile large language model for predictive light sensing,” in Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems, 2025, pp. 158–171

    2025

  11. [19]

    Decoding llm performance—prefill phase is compute bound on npu,

    Intel NPU Acceleration Library, “Decoding llm performance—prefill phase is compute bound on npu,” https://intel.github.io/ intel-npu-acceleration-library/llm performance.html, accessed: 2025- 08-18

    2025

  12. [20]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressandet al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  13. [21]

    The battle of chatbot giants: an ex- perimental comparison of chatgpt and bard,

    A. T. Kabakus ¸ and ˙I. Dogru, “The battle of chatbot giants: an ex- perimental comparison of chatgpt and bard,”International Journal of Engineering Research and Development, vol. 16, no. 2, pp. 679–691, 2024

    2024

  14. [22]

    Accel-sim: An extensible simulation framework for validated gpu modeling,

    M. Khairy, Z. Shen, T. M. Aamodt, and T. G. Rogers, “Accel-sim: An extensible simulation framework for validated gpu modeling,” in 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2020, pp. 473–486

    2020

  15. [23]

    Sk hynix ai-specific computing memory solution: From aim device to heterogeneous aimx-xpu system for comprehensive llm inference,

    G. Kim, J. Kim, N. Kim, W. Shin, J. Won, H. Joo, H. Choi, B. An, G. Shin, D. Yunet al., “Sk hynix ai-specific computing memory solution: From aim device to heterogeneous aimx-xpu system for comprehensive llm inference,” in2024 IEEE Hot Chips 36 Symposium (HCS). IEEE Computer S...

    2024

  16. [24]

    Monde: Mixture of near-data experts for large-scale sparse models,

    T. Kim, K. Choi, Y . Cho, J. Cho, H.-J. Lee, and J. Sim, “Monde: Mixture of near-data experts for large-scale sparse models,” inProceedings of the 61st ACM/IEEE Design Automation Conference, 2024, pp. 1–6

    2024

  17. [25]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626

    2023

  18. [26]

    vllm: Easy, fast, and cheap llm serving for everyone

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica., “vllm: Easy, fast, and cheap llm serving for everyone.” September 2023. [Online]. Available: https://github.com/vllm-project/vllm

    2023

  19. [27]

    A 1ynm 1.25 v 8gb, 16gb/s/pin gddr6-based accelerator-in-memory supporting 1tflops mac operation and various activation functions for deep-learning applications,

    S. Lee, K. Kim, S. Oh, J. Park, G. Hong, D. Ka, K. Hwang, J. Park, K. Kang, J. Kimet al., “A 1ynm 1.25 v 8gb, 16gb/s/pin gddr6-based accelerator-in-memory supporting 1tflops mac operation and various activation functions for deep-learning applications,” in2022 IEEE In- ternati...

    2022

  20. [28]

    Hardware architecture and software stack for pim based on commercial dram technology: Industrial product,

    S. Lee, S.-h. Kang, J. Lee, H. Kim, E. Lee, S. Seo, H. Yoon, S. Lee, K. Lim, H. Shinet al., “Hardware architecture and software stack for pim based on commercial dram technology: Industrial product,” in 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture...

    2021

  21. [29]

    H2-llm: Hardware-dataflow co-exploration for heterogeneous hybrid-bonding-based low-batch llm inference,

    C. Li, Y . Yin, X. Wu, J. Zhu, Z. Gao, D. Niu, Q. Wu, X. Si, Y . Xie, C. Zhanget al., “H2-llm: Hardware-dataflow co-exploration for heterogeneous hybrid-bonding-based low-batch llm inference,” in Proceedings of the 52nd Annual International Symposium on Computer Architecture, ...

    2025

  22. [30]

    Ascend: a scalable and unified architecture for ubiquitous deep neural network computing: Industry track paper,

    H. Liao, J. Tu, J. Xia, H. Liu, X. Zhou, H. Yuan, and Y . Hu, “Ascend: a scalable and unified architecture for ubiquitous deep neural network computing: Industry track paper,” in2021 IEEE International Sympo- sium on High-Performance Computer Architecture (HPCA). IEEE, 2021, p...

    2021

  23. [31]

    Davinci: A scalable architecture for neural network computing,

    H. Liao, J. Tu, J. Xia, and X. Zhou, “Davinci: A scalable architecture for neural network computing,” in2019 IEEE Hot Chips 31 Symposium (HCS). IEEE Computer Society, 2019, pp. 1–44

    2019

  24. [32]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  25. [33]

    Ramulator 2.0: A modern, modular, and extensible dram simulator,

    H. Luo, Y . C. Tu ˘grul, F. N. Bostancı, A. Olgun, A. G. Ya ˘glıkc ¸ı, and O. Mutlu, “Ramulator 2.0: A modern, modular, and extensible dram simulator,”IEEE Computer Architecture Letters, vol. 23, no. 1, pp. 112– 116, 2023

    2023

  26. [34]

    The design process for google’s training chips: Tpuv2 and tpuv3,

    T. Norrie, N. Patil, D. H. Yoon, G. Kurian, S. Li, J. Laudon, C. Young, N. Jouppi, and D. Patterson, “The design process for google’s training chips: Tpuv2 and tpuv3,”IEEE Micro, vol. 41, no. 2, pp. 56–63, 2021

    2021

  27. [35]

    Nvidia a100 tensor core gpu architecc ture,

    NVIDIA, “Nvidia a100 tensor core gpu architecc ture,” 2020, https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/ nvidia-ampere-architecture-whitepaper.pdf

    2020

  28. [36]

    Introduction to the nvidia dgx a100 system,

    NVIDIA, “Introduction to the nvidia dgx a100 system,” 2021. [Online]. Available: https://docs.nvidia.com/dgx/dgxa100-user-guide/ introduction-to-dgxa100.html

    2021

  29. [37]

    Nvidia h100 tensor core gpu architecture,

    NVIDIA, “Nvidia h100 tensor core gpu architecture,” 2023, https: //resources.nvidia.com/en-us-tensor-core/gtc22-whitepaper-hopper

    2023

  30. [38]

    Chatgpt,

    OpenAI, “Chatgpt,” September 2023. [Online]. Available: https: //chatgpt.com/blog/chatgpt

    2023

  31. [39]

    Gpt-oss-120b,

    OpenAI, “Gpt-oss-120b,” 2025, https://huggingface.co/openai/ gpt-oss-120b

    2025

  32. [40]

    Vespa: Vipt enhancements for superpage accesses,

    M. Parasar, A. Bhattacharjee, and T. Krishna, “Vespa: Vipt enhancements for superpage accesses,” inarXiv preprint arXiv:1701.03499, 2017

    2017 arXiv

  33. [41]

    Attacc! unleashing the power of pim for batched transformer- based generative model inference,

    J. Park, J. Choi, K. Kyung, M. J. Kim, Y . Kwon, N. S. Kim, and J. H. Ahn, “Attacc! unleashing the power of pim for batched transformer- based generative model inference,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages...

    2024

  34. [42]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

    2019

  35. [43]

    Splitwise: Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative llm inference using phase splitting,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 118–132

    2024

  36. [44]

    {DRAMA}: Exploiting{DRAM}addressing for{Cross-CPU}at- tacks,

    P. Pessl, D. Gruss, C. Maurice, M. Schwarz, and S. Mangard, “{DRAMA}: Exploiting{DRAM}addressing for{Cross-CPU}at- tacks,” in25th USENIX security symposium (USENIX security 16), 2016, pp. 565–581

    2016

  37. [45]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  38. [46]

    Ramulator 2.0,

    SAFARI Research Group, “Ramulator 2.0,” 2023, https://github.com/ CMU-SAFARI/ramulator2

    2023

  39. [47]

    Khaa44801b-mc16: 8gb hbm2e flashbolt,

    S. Semiconductor, “Khaa44801b-mc16: 8gb hbm2e flashbolt,”

  40. [48]

    Ianus: Integrated accelerator based on npu-pim unified memory system,

    M. Seo, X. T. Nguyen, S. J. Hwang, Y . Kwon, G. Kim, C. Park, I. Kim, J. Park, J. Kim, W. Shinet al., “Ianus: Integrated accelerator based on npu-pim unified memory system,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languag...

    2024

  41. [49]

    Facil: Flexible dram address mapping for soc-pim cooperative on-device llm inference,

    S. H. Seo, J. Kim, D. Lee, S. Yoo, S. Moon, Y . Park, and J. W. Lee, “Facil: Flexible dram address mapping for soc-pim cooperative on-device llm inference,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1720–1733

    2025

  42. [50]

    Stuart, Mitchell and Anita, Kean and Andrew, Mason and Michael, O’Sullivan and Antony, Phillips and Franco, Peschiera, “Pulp,” 2024, https://coin-or.github.io/pulp/

    2024

  43. [51]

    Design compiler,

    Synopsys, “Design compiler,” http://www.synopsys.com/Tools/ Implementation/RTLSynthesis/DesignCompiler/Pages

  44. [52]

    Qwen1.5-moe-a2.7b,

    Q. Team, “Qwen1.5-moe-a2.7b,” 2024. [Online]. Available: https: //huggingface.co/Qwen/Qwen1.5-MoE-A2.7B

    2024

  45. [53]

    Sglang: Efficient execution of structured language model programs,

    S. Team, “Sglang: Efficient execution of structured language model programs,” September 2024. [Online]. Available: https://github.com/ sgl-project/sglang

    2024

  46. [54]

    Introducing the next generation of claude

    T. C. Team, “Introducing the next generation of claude.” September 2024. [Online]. Available: https://www.anthropic.com/news/ claude-3-family

    2024

  47. [55]

    Upmem software development kit (sdk),

    UPMEM, “Upmem software development kit (sdk),” 2025. [Online]. Available: https://sdk.upmem.com/

    2025

  48. [56]

    Bloom: A 176b-parameter open-access multilingual language model,

    B. Workshop, T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili ´c, D. Hesslow, R. Castagn ´e, A. S. Luccioni, F. Yvonet al., “Bloom: A 176b-parameter open-access multilingual language model,”arXiv preprint arXiv:2211.05100, 2022

    2022 arXiv

  49. [57]

    Pim gpt a hybrid process in memory accelerator for autoregressive transformers,

    Y . Wu, Z. Wang, and W. D. Lu, “Pim gpt a hybrid process in memory accelerator for autoregressive transformers,”npj Unconventional Com- puting, vol. 1, no. 1, p. 4, 2024

    2024

  50. [58]

    Waitgpt: Monitoring and steering conversational llm agent in data analysis with on-the-fly code visualization,

    L. Xie, C. Zheng, H. Xia, H. Qu, and C. Zhu-Tian, “Waitgpt: Monitoring and steering conversational llm agent in data analysis with on-the-fly code visualization,” inProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 2024, pp. 1–14

    2024

  51. [59]

    Orders in chaos: Enhancing large-scale moe llm serving with data movement forecasting,

    Z. Yu, Y . Guan, Z. Yu, C. Zhou, S. Pei, Y . Kang, Y . Ding, and P.-A. Tsai, “Orders in chaos: Enhancing large-scale moe llm serving with data movement forecasting,” 2025. [Online]. Available: https://arxiv.org/abs/2510.05497

    2025 arXiv

  52. [60]

    Duplex: A device for large language models with mixture of experts, grouped query attention, and continuous batching,

    S. Yun, K. Kyung, J. Cho, J. Choi, J. Kim, B. Kim, S. Lee, K. Sohn, and J. H. Ahn, “Duplex: A device for large language models with mixture of experts, grouped query attention, and continuous batching,” in2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO)....

    2024

  53. [61]

    Llmcompass: Enabling efficient hardware design for large language model inference,

    H. Zhang, A. Ning, R. B. Prabhakar, and D. Wentzlaff, “Llmcompass: Enabling efficient hardware design for large language model inference,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 1080–1096

    2024

  54. [62]

    Um-pim: Dram-based pim with uniform & shared memory space,

    Y . Zhao, M. Gao, F. Liu, Y . Hu, Z. Wang, H. Lin, J. Li, H. Xian, H. Dong, T. Yanget al., “Um-pim: Dram-based pim with uniform & shared memory space,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 644– 659

    2024

  55. [63]

    Lmsys-chat-1m: A large-scale real-world llm conversation dataset,

    L. Zheng, W.-L. Chiang, Y . Sheng, T. Li, S. Zhuang, Z. Wu, Y . Zhuang, Z. Li, Z. Lin, E. P. Xing, J. E. Gonzalez, I. Stoica, and H. Zhang, “Lmsys-chat-1m: A large-scale real-world llm conversation dataset,” 2023

    2023

  56. [64]

    Hybrimoe: Hybrid cpu-gpu scheduling and cache management for efficient moe inference,

    S. Zhong, Y . Sun, L. Liang, R. Wang, R. Huang, and M. Li, “Hybrimoe: Hybrid cpu-gpu scheduling and cache management for efficient moe inference,”arXiv preprint arXiv:2504.05897, 2025

    2025 arXiv

  57. [65]

    {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193–210

    2024

  58. [66]

    Mixture-of-experts with expert choice routing,

    Y . Zhou, T. Lei, H. Liu, N. Du, Y . Huang, V . Zhao, A. M. Dai, Q. V . Le, J. Laudonet al., “Mixture-of-experts with expert choice routing,” Advances in Neural Information Processing Systems, vol. 35, pp. 7103– 7114, 2022

    2022

  59. [67]

    A comprehensive analysis of superpage management mechanisms and policies,

    W. Zhu, A. L. Cox, and S. Rixner, “A comprehensive analysis of superpage management mechanisms and policies,” in2020 USENIX Annual Technical Conference (USENIX ATC), 2020

    2020

  60. [2020]

    Available: https://semiconductor.samsung.com/dram/ hbm/hbm2e-flashbolt/khaa44801b-mc16/

    [Online]. Available: https://semiconductor.samsung.com/dram/ hbm/hbm2e-flashbolt/khaa44801b-mc16/

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.