Pith. sign in

REVIEW 4 major objections 6 minor 39 references

HPIM, a heterogeneous PIM accelerator pairing SRAM-PIM and HBM-PIM, claims up to a 34.3x latency reduction over the NVIDIA A100 GPU for single-batch LLM inference.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 16:28 UTC pith:IOSTOJNQ

load-bearing objection The architecture is worth a look, but the headline 34.3x speedup rests on an A100 baseline that the paper's own numbers make physically impossible, so the main quantitative claim is unsupported. the 4 major comments →

arxiv 2509.12993 v3 pith:IOSTOJNQ submitted 2025-09-16 cs.AR

HPIM: Heterogeneous Processing-In-Memory-based Accelerator for Large Language Models Inference

Cenlin Duan , Jianlei Yang , Rubing Yang , Yikun Wang , Yiou Wang , Lingkun Long , Yingjie Qi , Xiaolin He
show 3 more authors
Ao Zhou Xueyan Wang Weisheng Zhao
This is my paper
classification cs.AR
keywords large language modelsprocessing-in-memoryheterogeneous PIMSRAM-PIMHBM-PIMLLM inferenceautoregressive decodingmemory-centric accelerator
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM decoding is memory-bound and serial: each new token depends on the last, and the dominant operations are weight-heavy matrix-vector multiplications (GEMV). HPIM attacks this by combining two in-memory computing fabrics—an SRAM-PIM subsystem for fast, flexible attention and nonlinear operations, and an HBM-PIM subsystem for high-bandwidth, large-capacity GEMV—and mapping operations to whichever fabric fits their characteristics. A compiler partitions the workload and a tightly coupled pipeline overlaps QKV generation, attention, and feed-forward computation so that different parts of the same token run concurrently in the two memory domains. Cycle-accurate simulation across OPT models (350M–30B) reports up to a 34.3x latency speedup versus an A100 GPU, 1.50x lower latency than IANUS, and up to 5.76x higher throughput than CXL-PNM. The paper's central claim is that memory-centric heterogeneous PIM, rather than more compute, is the lever for latency-critical single-batch inference.

Core claim

HPIM is a heterogeneous processing-in-memory accelerator for single-batch LLM inference. Its SRAM-PIM subsystem contains 32 cores with tensor, vector, scalar, and in-array PIM units, handling GEMM-bound prefill operations, attention score computation (QxK^T, softmax, SxV), and nonlinear layers such as GELU and LayerNorm. Its HBM-PIM subsystem uses four HBM3 stacks with per-bank MAC units to compute the weight-heavy GEMVs of QKV generation, attention projection, and feed-forward networks directly in memory, exploiting the large internal bandwidth of HBM3. A compiler framework partitions the model graph: the prefill stage is dispatched to the tensor units, while decoding splits the work—GEMV-h

What carries the argument

The central mechanism is the heterogeneous two-fabric memory system, coordinated by a hardware-aware compiler and a pipelined execution schedule. On one side, the HBM3-PIM fabric stores the model weights and performs GEMV in per-bank MAC units near the data; on the other, the SRAM-PIM compute-in-memory fabric provides ultra-low-latency execution for attention and nonlinear operations. The load-bearing identity is the workload partition itself: latency-critical, small-matrix attention and elementwise operations are assigned to SRAM-PIM, while bandwidth-critical, weight-dominated matrix-vector multiplications are assigned to HBM-PIM, and the two run concurrently to create intra-token paralleli

Load-bearing premise

The speedup depends on the assumption that the HBM-PIM banks actually sustain the modeled 102.4 TB/s internal bandwidth under real GEMV traffic, with minimal bank conflicts, refresh, and synchronization overhead; if real bank-level throughput is lower, the latency advantage over the A100 shrinks.

What would settle it

Measure the sustainable bank-level bandwidth of the HBM3-PIM design under the exact GEMV access pattern used in the paper (e.g., read weights once, stream activations, accumulate partial sums in bank). If the measured throughput falls materially below the 102.4 TB/s used in the simulator—say by more than 30%—the reported 34.3x speedup over the A100 would decrease by the same factor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the simulator is faithful, single-batch LLM decoding latency can be reduced by more than a factor of 30 without adding compute throughput, by moving the computation into memory and overlapping subtasks across memory types.
  • The result would imply that memory-bound, serial decode stages are better served by memory-centric accelerators than by compute-centric GPUs, shifting the design target from peak TFLOPS to sustainable internal memory bandwidth and data placement.
  • A heterogeneous PIM approach suggests that different phases of inference (prefill GEMM vs decode GEMV) and different operation types (attention vs weights) should be mapped to different memory technologies, opening the door for future accelerators that mix SRAM, DRAM, and flash PIM.
  • The tightly coupled pipeline demonstrates a concrete way to expose intra-token parallelism, which other single-batch accelerators could adopt even if their memory technology choices differ.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 34.3x figure is measured against a single-batch A100 with no batching; in throughput-oriented scenarios with larger batches, GPUs amortize weight movement across requests, so the advantage is likely to shrink—the paper's case is specifically latency-critical single-request inference.
  • A natural extension is to combine HPIM with speculative decoding: the SRAM-PIM fabric could validate several candidate tokens while the HBM-PIM fabric processes their shared weight GEMVs, further hiding the serial dependency of autoregressive generation.
  • The paper does not address models that exceed the 96 GB HBM capacity; for such models a spilling strategy to a lower-bandwidth tier would break the simple two-fabric pipeline, so the speedup should be re-evaluated with capacity constraints.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes HPIM, a heterogeneous processing-in-memory architecture for single-batch LLM inference. It splits decode-stage work between an HBM3-PIM subsystem for weight-heavy GEMV operations and an SRAM-PIM subsystem for latency-critical attention and non-linear operations, supported by a hardware-aware compiler and a pipelined intra-token scheduler. Evaluation is performed with an in-house cycle-accurate simulator (a DRAMsim3 extension plus a custom SRAM-PIM model) on OPT models from 350M to 30B parameters. The paper claims up to 34.3x peak speedup over an NVIDIA A100 GPU, 1.50x lower latency than IANUS, and up to 5.76x higher throughput than CXL-PNM.

Significance. The architectural concept is timely and well motivated: partitioning attention (latency-critical, logic-flexible) onto SRAM-PIM and weight-heavy GEMVs (bandwidth- and capacity-bound) onto HBM-PIM is a sensible design point, and the proposed intra-token pipeline directly targets the serial-dependency bottleneck of autoregressive decoding. If the reported numbers were reproducible, the paper would make a valuable contribution to PIM-based LLM inference. The use of DRAMsim3 and Verilog-level SRAM-PIM modeling is a reasonable methodology start. However, the central quantitative claims currently rest on an implausible A100 baseline and unvalidated internal-bandwidth assumptions, and no simulator code is released. The significance is therefore conditional on substantial revision.

major comments (4)
  1. [§VII.B, Fig. 13, Table III] The A100 FFN baseline for OPT-13B is physically impossible. OPT-13B FFN weights are 2×5120×20480×40 = 8.39B FP16 values = 16.8 GB. In single-batch decode these must be read from DRAM for every token. Fig. 13 reports 7902 ms for 1024 output tokens, i.e., 7.72 ms/token, requiring 16.8 GB/7.72 ms = 2.18 TB/s of DRAM traffic. This exceeds both the paper's own A100 bandwidth (1.935 TB/s, Table III) and the A100-80GB SXM spec (~2.04 TB/s). The QKV-generation and projection numbers are below the bound and merely optimistic, but the FFN number cannot be reconciled with dense FP16 execution unless undisclosed quantization, sparsity, or weight sharing is assumed. Consequently, the 34.3x peak speedup in Fig. 11 and the 3.64x end-to-end speedup in §VII.B are not validated. Please re-measure the A100 with a properly optimized, fully disclosed inference stack and report GPU variant, software versions,
  2. [Abstract, §I, §VII.B] The paper reports contradictory peak speedup figures: 23.1x in the abstract, 22.8x in the introduction, and 34.3x in Section VII.B and Fig. 11. These must be reconciled, or the specific configuration (model, input/output length) corresponding to each number must be stated explicitly. As written, the inconsistency undermines confidence in the reported results.
  3. [§V.B, Table III] The central performance advantage is driven by the modeled HBM-PIM internal bandwidth of 102.4 TB/s. This is an assumed input to the simulator rather than an independently measured quantity, and the paper does not provide a sensitivity analysis or validation of this constant. It also does not explain how DRAMsim3 was extended for in-bank compute mode or how bank conflicts, refresh, row activation, and synchronization across 4 HBM stacks and 32 SRAM cores are accounted for. Please add a sensitivity analysis and a validation subsection, or release the simulator configuration so the results can be checked independently.
  4. [§VII.A] The A100 baseline is described only as 'HuggingFace Transformers framework' with no optimization level, GPU variant (SXM vs PCIe), CUDA version, or profiling methodology. Default HuggingFace generation is not representative of state-of-the-art optimized GPU inference, and comparisons against such a baseline inflate the reported speedups. This must be corrected together with the first major comment.
minor comments (6)
  1. [Fig. 10] The captions read 'prefill phrase' and 'decoding phrase'; these should be 'prefill phase' and 'decoding phase'.
  2. [Table III] The header 'LANUS [33]' is a typo; it should be 'IANUS [33]'.
  3. [§V] The text says the microarchitecture details are 'outlined in §VI-B'; the correct forward reference appears to be §IV-B.
  4. [Ref. [23]] The title contains 'Transfmer' (missing 'o'). Please proofread the reference list.
  5. [§VII.B] The phrase '1.50x lower latency than IANUS' should be expressed as '1.50x lower latency' or 'a 1.50x latency reduction' to avoid confusion between latency and throughput ratios.
  6. [Table I] Ref. [13] (SHMT) describes an SRAM+HBM hybrid CIM architecture and is cited in the paper, but it is omitted from the Table I comparison. The 'first' novelty claim should be positioned against this work explicitly.

Circularity Check

0 steps flagged

No significant circularity: HPIM's speedup numbers come from a cycle-accurate simulator with external baselines; the main caveats (A100 baseline plausibility, inconsistent peak-speedup figures) are correctness concerns, not circular derivations.

full rationale

The paper's central claim is an engineering performance estimate from simulation, not a mathematical derivation from first principles. HPIM's latency is produced by a cycle-accurate simulator: the HBM-PIM subsystem is modeled with an extended DRAMsim3 and the SRAM-PIM compute logic is described in Verilog with memory-compiler characteristics (Sec. VII.A). No fitted parameter is subsequently renamed as a prediction, and no sub-result is defined in terms of the headline speedup. The workload partitioning (attention to SRAM-PIM, weight-heavy GEMV to HBM-PIM) is a design choice that the simulator evaluates, rather than a conclusion equivalent to its own inputs. Comparisons against external baselines (measured A100 via HuggingFace Transformers, and IANUS/CXL-PNM from prior publications) provide independent reference points. Citations to the authors' earlier PIM work are background references and are not load-bearing: the architecture is not justified by a uniqueness theorem or by an ansatz imported solely from self-citations. The serious caveats are non-circular: Fig. 13's A100 FFN decode time for OPT-13B (7902 ms over 1024 tokens, i.e., 7.72 ms/token) appears to violate the DRAM-bandwidth floor computed from the paper's own Table III (1935 GB/s and ~16.8 GB of FFN weights gives ~8.67 ms/token), and the peak speedup varies across the abstract (23.1x), introduction (22.8x), and results (34.3x). These undermine confidence in the reported magnitude but do not constitute circular reasoning. Thus the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The reported speedups rest on design parameters selected by the authors (HBM internal bandwidth, bank-level PU count, 32 SRAM cores, 1 GHz clock), on standard assumptions that single-batch decode is memory bound, and on the comparability of baselines. No fabricated chip, formal proof, or public artifact is provided, so these parameters and assumptions carry the entire performance claim.

free parameters (4)
  • HBM-PIM internal bandwidth = 102.4 TB/s (Table III)
    Assumed maximum internal bandwidth of the HBM3 PIM subsystem; it directly sets the upper bound on GEMV speed and therefore the central speedup claim.
  • HBM-PIM processing unit configuration = 1 PU per bank, 16 FP16 multipliers per PU (Table IV)
    Chosen design point that determines HBM-PIM compute throughput and area assumptions.
  • SRAM-PIM core count and PIM unit throughput = 32 cores, 4.09 TFLOPS per core (Tables III, IV)
    Determines prefill GEMM throughput and attention GEMV throughput; both drive the latency results.
  • Clock frequency = 1 GHz (Tables III, IV)
    Used in all throughput and latency calculations for both subsystems.
axioms (4)
  • domain assumption Single-batch LLM decoding is memory-bound and dominated by GEMV operations.
    Section II and the roofline analysis assert this; it motivates the entire workload partitioning.
  • domain assumption HBM-PIM bank-level MAC units execute GEMV at near the modeled internal bandwidth without bank conflicts, refresh interference, or data transfer bottlenecks.
    Section V.B and Table III imply sustained 102.4 TB/s internal bandwidth; no contention or refresh modeling detail is given.
  • domain assumption HBM-PIM and SRAM-PIM can be tightly pipelined with synchronization and communication cost exactly as modeled.
    Section VI.B claims deep pipelining hides attention latency, but the simulator's deadlock, sync, and NoC overheads are not independently validated.
  • domain assumption The A100 baseline measured with HuggingFace Transformers is representative of state-of-the-art GPU inference.
    Section VII.A specifies only the framework; no optimized kernels, kernel libraries, or measurement methodology are described.

pith-pipeline@v1.3.0-alltime-deepseek · 20572 in / 12041 out tokens · 124349 ms · 2026-08-04T16:28:07.070877+00:00 · methodology

0 comments
read the original abstract

The deployment of large language models (LLMs) presents significant challenges due to their enormous memory footprints, low arithmetic intensity, and stringent latency requirements, particularly during the autoregressive decoding stage. Traditional compute-centric accelerators, such as GPUs, suffer from severe resource underutilization and memory bandwidth bottlenecks in these memory-bound workloads. To overcome these fundamental limitations, we propose HPIM, the first memory-centric heterogeneous Processing-In-Memory (PIM) accelerator that integrates SRAM-PIM and HBM-PIM subsystems designed specifically for LLM inference. HPIM employs a software-hardware co-design approach that combines a specialized compiler framework with a heterogeneous hardware architecture. It intelligently partitions workloads based on their characteristics: latency-critical attention operations are mapped to the SRAM-PIM subsystem to exploit its ultra-low latency and high computational flexibility, while weight-intensive GEMV computations are assigned to the HBM-PIM subsystem to leverage its high internal bandwidth and large storage capacity. Furthermore, HPIM introduces a tightly coupled pipeline strategy across SRAM-PIM and HBM-PIM subsystems to maximize intra-token parallelism, thereby significantly mitigating the serial dependency of the autoregressive decoding stage. Comprehensive evaluations using a cycle-accurate simulator demonstrate that HPIM significantly outperforms state-of-the-art accelerators, achieving a peak speedup of up to 23.1x compared to the NVIDIA A100 GPU. Moreover, HPIM exhibits superior performance over contemporary PIM-based accelerators, highlighting its potential as a highly practical and scalable solution for accelerating large-scale LLM inference.

Figures

Figures reproduced from arXiv: 2509.12993 by Ao Zhou, Cenlin Duan, Jianlei Yang, Lingkun Long, Rubing Yang, Weisheng Zhao, Xiaolin He, Xueyan Wang, Yikun Wang, Yingjie Qi, Yiou Wang.

Figure 1
Figure 1. Figure 1: The diverse requirements of LLM inference, the corresponding trade [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Model architecture and inference process of LLM. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Roofline model of OPT-6.7B (Bright), OPT-13B (Moderate), and OPT-30B (Dark) operations on an A100 GPU with a sequence length of 2048. The points plot the performance of Attention (circle) and QKV Generation (pentagram) during the prefill (orange) and decode (green) phases. specialized hardware acceleration tailored specifically to these distinct computational patterns. B. Traditional Hardware Accelerators … view at source ↗
Figure 5
Figure 5. Figure 5: Overview of the HPIM accelerator, including its two main components: HPIM compiler and HPIM architecture. The workflow starts from user-defined [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The microarchitecture of the SRAM-PIM subsystem, which is [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The microarchitecture of the HBM-PIM subsystem. It illustrates the hierarchical structure based on HBM3, from the 3D-stacked package down to [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Illustration of weight K of 16 heads allocation and tiling scheme for HBM-PIM subsystems. Algorithm 1 Hybrid Parallelism of Q/K/V Weight Allocation Require: Number of heads Nh, DRAM channels ND, SRAM cores NS, weight dimensions demb, tensors {Th} for each head h. Ensure: Partitioned tensor slices {Sc,r} to each channel c and round r. 1: // Calculate dimensions per head 2: dk ← demb/Nh 3: // Initialize coun… view at source ↗
Figure 9
Figure 9. Figure 9: Illustration of parallel processing of a single head on multiple SRAM-PIM cores. The figure illustrates the three main stages of attention: (a) GEMV [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Example execution timelines of 32-head for (a) prefill phrase, where [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: End-to-end inference latency comparison between HPIM and an NVIDIA A100 GPU across a range of OPT models ( [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance comparison of HPIM against SOTA accelerators IANUS [PITH_FULL_IMAGE:figures/full_fig_p011_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 6 linked inside Pith

  1. [1]

    Language Models are Few-shot Learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language Models are Few-shot Learners,” inProceed- ings of Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  2. [2]

    Language Models are Unsupervised Multitask Learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language Models are Unsupervised Multitask Learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  3. [3]

    The Llama 3 Herd of Models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The Llama 3 Herd of Models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    LLaMA: Open and Efficient Foundation Language Models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    LLAMA 2: Open Foundation and Fine-Tuned Chat Models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “LLAMA 2: Open Foundation and Fine-Tuned Chat Models,” arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Gorilla: Large Language Model Connected with Massive APIs,

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large Language Model Connected with Massive APIs,” in Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  7. [7]

    Toolformer: Language Models Can Teach Themselves to Use Tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language Models Can Teach Themselves to Use Tools,” inProceedings of Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  8. [8]

    FlashAt- tention: Fast and Memory-Efficient Exact Attention with IO- Awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “FlashAt- tention: Fast and Memory-Efficient Exact Attention with IO- Awareness,” inProceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  9. [9]

    CodeGen: An Open Large Lan- guage Model for Code with Multi-Turn Program Synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “CodeGen: An Open Large Lan- guage Model for Code with Multi-Turn Program Synthesis,” inProceedings of the International Conference on Learning Representations (ICLR), 2022

    2022

  10. [10]

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,” inPro- ceedings of Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  11. [11]

    Au- toCodeRover: Autonomous Program Improvement,

    Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Au- toCodeRover: Autonomous Program Improvement,” inPro- ceedings of International Symposium on Software Testing and Analysis (ISSTA), 2024

    2024

  12. [12]

    A 28nm 74.34 TFLOPS/W BF16 Heterogenous CIM-Based Accelerator Exploiting Denoising- Similarity for Diffusion Models,

    R. Guo, L. Wang, X. Chen, H. Sun, Z. Yue, Y . Qin, H. Han, Y . Wang, F. Tu, S. Weiet al., “A 28nm 74.34 TFLOPS/W BF16 Heterogenous CIM-Based Accelerator Exploiting Denoising- Similarity for Diffusion Models,” inProceedings of Interna- tional Solid-State Circuits Conference (ISSCC), 2024

    2024

  13. [13]

    SHMT: An SRAM and HBM Hybrid Computing-in-Memory Architec- ture With Optimized KV Cache for Multimodal Transformer,

    X. Fu, J. Yue, M. Faizan, Z. Li, Q. Huo, and F. Zhang, “SHMT: An SRAM and HBM Hybrid Computing-in-Memory Architec- ture With Optimized KV Cache for Multimodal Transformer,” Transactions on Circuits and Systems I: Regular Papers (TCAS- I), 2025

    2025

  14. [14]

    DDC-PIM: Efficient Algorithm/Architecture Co-Design for Doubling Data Capac- ity of SRAM-based Processing-in-Memory,

    C. Duan, J. Yang, X. He, Y . Qi, Y . Wang, Y . Wang, Z. He, B. Yan, X. Wang, X. Jia, and W. Zhao, “DDC-PIM: Efficient Algorithm/Architecture Co-Design for Doubling Data Capac- ity of SRAM-based Processing-in-Memory,”Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), vol. 43, no. 3, pp. 906–918, 2024

    2024

  15. [15]

    Towards Efficient SRAM- PIM Architecture Design by Exploiting Unstructured Bit-Level Sparsity,

    C. Duan, J. Yang, Y . Wang, Y . Wang, Y . Qi, X. He, B. Yan, X. Wang, X. Jia, and W. Zhao, “Towards Efficient SRAM- PIM Architecture Design by Exploiting Unstructured Bit-Level Sparsity,” inProceedings of Design Automation Conference (DAC), 2024

    2024

  16. [16]

    CIMFlow: An Integrated Framework for Systematic Design and Evaluation of Digital CIM Architec- tures,

    Y . Qi, J. Yang, Y . Wang, Y . Wang, D. Wang, L. Tang, C. Duan, X. He, and W. Zhao, “CIMFlow: An Integrated Framework for Systematic Design and Evaluation of Digital CIM Architec- tures,” inProceedings of Design Automation Conference (DAC), 2025

    2025

  17. [17]

    APIM: An Antiferromagnetic MRAM-Based Processing-In-Memory System for Efficient Bit-Level Opera- tions of Quantized Convolutional Neural Networks,

    Y . Li, J. Wang, D. Zhu, J. Li, A. Du, X. Wang, Y . Zhang, and W. Zhao, “APIM: An Antiferromagnetic MRAM-Based Processing-In-Memory System for Efficient Bit-Level Opera- tions of Quantized Convolutional Neural Networks,”Transac- tions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), vol. 43, no. 8, pp. 2405–2410, 2024

    2024

  18. [18]

    TensorCIM: A 28nm 3.7 nJ/Gather and 8.3 TFLOPS/W FP32 Digital-CIM Tensor Processor for MCM-CIM-Based Beyond- NN Acceleration,

    F. Tu, Y . Wang, Z. Wu, W. Wu, L. Liu, Y . Hu, S. Wei, and S. Yin, “TensorCIM: A 28nm 3.7 nJ/Gather and 8.3 TFLOPS/W FP32 Digital-CIM Tensor Processor for MCM-CIM-Based Beyond- NN Acceleration,” inProceedings of International Solid-State Circuits Conference (ISSCC), 2023

    2023

  19. [19]

    P 3ViT: A CIM-Based High-Utilization Architecture With Dynamic Pruning and Two-Way Ping-Pong Macro for Vision Transformer,

    X. Fu, Q. Ren, H. Wu, F. Xiang, Q. Luo, J. Yue, Y . Chen, and F. Zhang, “P 3ViT: A CIM-Based High-Utilization Architecture With Dynamic Pruning and Two-Way Ping-Pong Macro for Vision Transformer,”Transactions on Circuits and Systems I: Regular Papers (TCAS-I), vol. 70, no. 12, pp. 4938–4948, 2023

    2023

  20. [20]

    PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System,

    Y . He, H. Mao, C. Giannoula, M. Sadrosadati, J. G ´omez- Luna, H. Li, X. Li, Y . Wang, and O. Mutlu, “PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System,” inPro- 13 ceedings of International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2025

    2025

  21. [21]

    SpecPIM: Accelerating Speculative Inference on PIM-Enabled System via Architecture-Dataflow Co-Exploration,

    C. Li, Z. Zhou, S. Zheng, J. Zhang, Y . Liang, and G. Sun, “SpecPIM: Accelerating Speculative Inference on PIM-Enabled System via Architecture-Dataflow Co-Exploration,” inProceed- ings of International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024

    2024

  22. [22]

    An LPDDR-based CXL- PNM Platform for TCO-efficient Inference of Transformer- based Large Language Models,

    S.-S. Park, K. Kim, J. So, J. Jung, J. Lee, K. Woo, N. Kim, Y . Lee, H. Kim, Y . Kwonet al., “An LPDDR-based CXL- PNM Platform for TCO-efficient Inference of Transformer- based Large Language Models,” inProceeding of Interna- tional Symposium on High-Performance Computer Architecture (HPCA), 2024

    2024

  23. [23]

    Samsung PIM/PNM for Transfmer Based AI: Energy Efficiency on PIM/PNM Cluster,

    J. H. Kim, Y . Ro, J. So, S. Lee, S.-h. Kang, Y . Cho, H. Kim, B. Kim, K. Kim, S. Parket al., “Samsung PIM/PNM for Transfmer Based AI: Energy Efficiency on PIM/PNM Cluster,” inProceedings of Hot Chips Symposium (HCS), 2023

    2023

  24. [24]

    3D-FPIM: An Extreme Energy-Efficient DNN Acceleration System Using 3D NAND Flash-Based In- Situ PIM Unit,

    H. Lee, M. Kim, D. Min, J. Kim, J. Back, H. Yoo, J.-H. Lee, and J. Kim, “3D-FPIM: An Extreme Energy-Efficient DNN Acceleration System Using 3D NAND Flash-Based In- Situ PIM Unit,” inProceedings of International Symposium on Microarchitecture (MICRO), 2022

    2022

  25. [25]

    S- FLASH: A NAND Flash-Based Deep Neural Network Acceler- ator Exploiting Bit-Level Sparsity,

    M. Kang, H. Kim, H. Shin, J. Sim, K. Kim, and L.-S. Kim, “S- FLASH: A NAND Flash-Based Deep Neural Network Acceler- ator Exploiting Bit-Level Sparsity,”Transactions on Computers (TC), vol. 71, no. 6, pp. 1291–1304, 2021

    2021

  26. [26]

    Attention is All You Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is All You Need,” inProceedings of Advances in neural information processing systems (NeurIPS), 2017

    2017

  27. [27]

    Deepspeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters,

    J. Rasley, S. Rajbhandari, O. Ruwase, and Y . He, “Deepspeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters,” inProceedings of SIGKDD international conference on knowledge discovery & data mining (KDD), 2020

    2020

  28. [28]

    DeepSpeed-Inference: Enabling Efficient Inference of Trans- former Models at Unprecedented Scale,

    R. Y . Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasleyet al., “DeepSpeed-Inference: Enabling Efficient Inference of Trans- former Models at Unprecedented Scale,” inProceedings of International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2022

    2022

  29. [29]

    LLM Inference Unveiled: Survey and Roofline Model Insights,

    Z. Yuan, Y . Shang, Y . Zhou, Z. Dong, Z. Zhou, C. Xue, B. Wu, Z. Li, Q. Gu, Y . J. Leeet al., “LLM Inference Unveiled: Survey and Roofline Model Insights,”arXiv preprint arXiv:2402.16363, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing,

    G. Heo, S. Lee, J. Cho, H. Choi, S. Lee, H. Ham, G. Kim, D. Mahajan, and J. Park, “NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencing,” inProceedings of International Conference on Architectural Support for Program- ming Languages and Operating Systems (ASPLOS), 2024

    2024

  31. [31]

    AttAcc! Unleashing the Power of PIM for Batched Transformer-based Generative Model Inference,

    J. Park, J. Choi, K. Kyung, M. J. Kim, Y . Kwon, N. S. Kim, and J. H. Ahn, “AttAcc! Unleashing the Power of PIM for Batched Transformer-based Generative Model Inference,” inProceedings of International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024

    2024

  32. [32]

    TransPIM: A Memory-based Acceleration via Software-Hardware Co-Design for Transformer,

    M. Zhou, W. Xu, J. Kang, and T. Rosing, “TransPIM: A Memory-based Acceleration via Software-Hardware Co-Design for Transformer,” inProceedings of International Symposium on High-Performance Computer Architecture (HPCA), 2022

    2022

  33. [33]

    IANUS: Integrated Accelerator based on NPU-PIM Unified Memory System,

    M. Seo, X. T. Nguyen, S. J. Hwang, Y . Kwon, G. Kim, C. Park, I. Kim, J. Park, J. Kim, W. Shinet al., “IANUS: Integrated Accelerator based on NPU-PIM Unified Memory System,” inProceedings of International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024

    2024

  34. [34]

    Cambricon-LLM: A Chiplet- Based Hybrid Architecture for On-Device Lnference of 70B LLM,

    Z. Yu, S. Liang, T. Ma, Y . Cai, Z. Nan, D. Huang, X. Song, Y . Hao, J. Zhang, T. Zhiet al., “Cambricon-LLM: A Chiplet- Based Hybrid Architecture for On-Device Lnference of 70B LLM,” inProceedings of International Symposium on Microar- chitecture (MICRO), 2024

    2024

  35. [35]

    Newton: A DRAM-maker’s Accelerator-in-Memory (AiM) Architecture for Machine Learn- ing,

    M. He, C. Song, I. Kim, C. Jeong, S. Kim, I. Park, M. Thot- tethodi, and T. Vijaykumar, “Newton: A DRAM-maker’s Accelerator-in-Memory (AiM) Architecture for Machine Learn- ing,” inProceedings of International Symposium on Microar- chitecture (MICRO), 2020

    2020

  36. [36]

    OPT: Open Pre-trained Trans- former Language Models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “OPT: Open Pre-trained Trans- former Language Models,”arXiv preprint arXiv:2205.01068, 2022

    Pith/arXiv arXiv 2022

  37. [37]

    Hugging- Face’s Transformers: State-of-the-art Natural Language Process- ing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiczet al., “Hugging- Face’s Transformers: State-of-the-art Natural Language Process- ing,”arXiv preprint arXiv:1910.03771, 2019

    Pith/arXiv arXiv 1910

  38. [38]

    DRAM- sim3: A Cycle-Accurate, Thermal-Capable DRAM Simulator,

    S. Li, Z. Yang, D. Reddy, A. Srivastava, and B. Jacob, “DRAM- sim3: A Cycle-Accurate, Thermal-Capable DRAM Simulator,” Computer Architecture Letters (CAL), vol. 19, no. 2, pp. 106– 109, 2020

    2020

  39. [39]

    High Bandwidth Memory DRAM (HBM3),

    “High Bandwidth Memory DRAM (HBM3),”JEDEC, 2022. Cenlin Duanreceived the B.S. degree in Electronic Science and Technology from University of Elec- tronic Science and Technology of China, Chengdu, China, in 2015, and the M.S. degree in Software Engineering from Xidian University, Xi’an, China, in 2018. She is currently pursuing the Ph.D. degree at the Scho...

    2022