Pith. sign in

REVIEW 4 major objections 6 minor 66 references

FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FLARE is an AMS-PiM architecture that claims to run transformer self-attention end-to-end with integer-only arithmetic, no floating-point units, no division, and no high-ENOB ADCs.

desk verdict A genuinely integrated AMS-PiM attention accelerator with strong end-to-end accuracy, but the scale-accounting proof is missing and a fair baseline is omitted—needs a real referee, not a desk reject. read the letter →

arxiv 2411.14733 v1 pith:4ZRTEP2K submitted 2024-11-22 cs.LG cs.SYeess.IVeess.SY

Donghyeon Yi , Seoyoung Lee , Jongho Kim , Junyoung Kim , Sohmyung Ha , Ik Joon Chang , Minkyu Je This is my paper · ORCID
classification cs.LGcs.SYeess.IVeess.SY
keywords analog-mixed-signalprocessing-in-memorypost-trainingquantizationself-attentionaccelerationtransformerinferenceeMSB-Qinteger-onlysoftmaxbitwisesparsitylow-ENOBADC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FLARE is an analog-mixed-signal processing-in-memory architecture that claims to run transformer self-attention layers end-to-end on chip using only integer arithmetic and low-resolution ADCs. The central claim is that post-training quantization for attention does not need dequantization-quantization steps, floating-point units, division, or high-ENOB ADCs, if the exponent information of each token is carried through a fused quantization scheme and absorbed by an integer softmax. If true, attention accelerators can avoid the exponential area and energy cost of high-precision ADCs and the overhead of FPUs, while keeping accuracy close to FP32. The paper reports near-FP32 accuracy on GLUE and ImageNet benchmarks and better latency and energy than GPUs and conventional PiM baselines.

What carries the argument

The architecture is carried by three interlocking mechanisms. eMSB-Q (effective-most-significant-bit quantization) replaces FP division-based quantization with an MSB-position search, parsing, and bit-shifting; per-token eMSB information is passed to the softmax block rather than stored for all tokens. VDR-Softmax is an integer-only softmax that changes the exponential base from e to the n-th root of e so that exponent information is absorbed into the function's parameters, and it uses a second-order polynomial plus shifts instead of division. BitSift-GEMV is a sparse GEMV controller that counts the number of 1 bits in bit-serial input slices and fetches only the longest slices containing up to eight ones, supplementing dummy ones through a SAWLD controller so every analog column-sum sees a fixed 8 active wordlines; this fixed SAWL condition is what lets a low-ENOB ADC operate with a narrow, stable dynamic range despite the analog sum.

What would settle it

Run the same 8-bit FLARE pipeline on an encoder at sequence lengths 512 and 4096 without retraining and compare end-task accuracy to FP32; if the accuracy gap grows substantially with sequence length, the per-token exponent-dropping is losing scale information that the fused linear GEMVs need.

Watch

Extended reading notes

Core claim

The paper's central claim is that a PTQ-based AMS-PiM accelerator can execute a complete self-attention layer in integer arithmetic only, with no dequantization-quantization round trips, no floating-point units, no division operations, and no high-ENOB ADCs, while preserving near-FP32 accuracy. The discovery is that the exponent information lost by integer quantization can be carried forward in a separate per-token channel, absorbed into the softmax computation, and then dropped, so that linear projections and attention-weighted sums never need to see it. FLARE combines effective-MSB quantization (eMSB-Q), an integer-only softmax called VDR-Softmax that changes the exponential base from e to a token-dependent root of e, and a bitwise-sparse GEMV engine that keeps a fixed number of simultaneously activated wordlines so low-ENOB ADCs see a stable analog dynamic range. On GLUE and ImageNet benchmarks with 8-bit integer models, the reported accuracy is close to FP32 and often above the conventional FP-based DQ-Q baseline.

Load-bearing premise

The load-bearing premise is that dropping a token's exponent during the linear QK^T and attention-weighted-V sums, then restoring it only inside the softmax stage, never lets scale mismatches accumulate enough to hurt accuracy.

Editorial extensions

If this is right

  • Attention layers can be processed entirely on one hybrid MRAM-SRAM AMS-PiM device, cutting out-of-PiM tensor traffic from quadratic O(N^2) to linear O(N) in sequence length.
  • PTQ-based transformers no longer need high-ENOB ADCs, which the paper argues would otherwise require 18 or more bits, avoiding area and energy that scale as 2^ENOB.
  • Softmax can be computed with shifts, polynomial evaluation, and bit-parsing instead of FP exponentiation and division, with near-FP32 accuracy on GLUE and ImageNet.
  • GEMV latency reduces by roughly the bitwise-sparsity factor because bitwise zero activations are skipped, with measured boosting factors matching the predicted values.
  • The design is compatible with other optimizations such as FlashAttention-style tiling, so long-sequence or oversized models can be split across PEs without hardware changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If scale information is truly preserved only through the per-token eMSB channel, the scheme's accuracy should degrade gracefully under longer sequences and deeper stacks, so evaluating end-to-end accuracy at longer contexts would directly test whether the exponent path leaks.
  • BitSift-GEMV's skip rate is data-dependent; a token whose value bits are dense in every slice would see little speedup, and the fixed-SAWL dummy-one padding would add cycles, suggesting an adaptive slice-length extension worth testing.
  • The 6-sigma sensing-margin claim derives from Monte-Carlo simulation on a 28nm FD-SOI array, so moving to another process node or a different resistive memory technology would require re-characterizing the safe 8-wordline bound.
  • eMSB-Q and VDR-Softmax together imply that any nonlinearity depending on absolute scale, such as layer normalization with per-channel statistics, would need its own exponent-carrying scheme; the paper focuses on attention's softmax.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FLARE, an AMS-PiM architecture for encoder transformer self-attention. It introduces eMSB-Q, an integer-only quantization that replaces FP division with exponent detection and shifting; VDR-Softmax, an integer approximation of softmax without division or FPUs; and BitSift-GEMV, a bitwise-sparsity-driven GEMV scheme with low-ENOB ADCs and fixed SAWL=8 for 6-sigma reliability. The authors report near-FP32 accuracy on GLUE and ImageNet for BERT/ViT/DeiT and large gains in tokens/sec and tokens/Joule over GPUs and a PiM baseline, based on a 28nm FD-SOI post-layout design and Monte Carlo simulations.

Significance. If the claims hold, this is a potentially valuable contribution: it would remove high-ENOB ADCs, FPUs, and division from PTQ-based attention inference, and the fused dataflow reduces quadratic tensor traffic to linear. The paper includes concrete hardware implementation details (Table 1, Fig. 9-13), real activation bitwise sparsity measurements (Fig. 7), and end-to-end accuracy tables (Fig. 15), and the BitSift-GEMV speedup is measured against a fixed-length baseline rather than assumed. However, the paper's central accuracy guarantee rests on unproved scale-propagation and normalization invariants, which must be established before the claims are fully supported.

major comments (4)
  1. [§4.2, §4.5] Section 4.2 (and the fused dataflow of Section 4.5) does not establish how per-token eMSB-Q scales propagate through the attention computation. If K and V are quantized per token, then L_ij = s_Q_i * s_K_j * (q_i dot k_j); a single row-wise exponent n_e can restore the softmax temperature only when s_K_j is constant across j. The text states that KV parsing uses MSBs rather than eMSBs 'to preserve global context', but it never proves that this yields a token-independent scale or that the longer MSB-based parse avoids clipping for all tokens. Since the Fig. 15 accuracy claims depend on this, please add a lemma giving the scale-propagation equations for QK^T and for O_i = gamma_i * delta * sum_j A_int_ij V_int_j, with explicit conditions on s_K_j and s_V_j.
  2. [Algorithm 1] Algorithm 1 (VDR_Norm) is missing the softmax denominator. The function subtracts the row maximum, exponentiates via iEXP, and then applies eMSB-Q to the exponentiated scores, but no sum over j is computed or divided. This is harmless only if the downstream per-token eMSB-Q on the attention output absorbs an arbitrary per-token scale, so that exact sum-normalization is unnecessary. That invariant is not stated or proved anywhere in Section 4.3 or 4.5. Please state exactly how the denominator is replaced and verify that the weighted sum of V inherits only a token-dependent factor that is canceled by the output quantization.
  3. [§4.3] Section 4.3 presents no error analysis for the VDR-Softmax approximation. The coefficients a,b,c,S,l are taken from previous works [32, 37] and adjusted per token through the base change e -> n-th root of e, but there is no bound on the approximation error as a function of n_e, Q_I, and Q_O. The end-to-end results in Fig. 15 cannot separate this error from eMSB-Q quantization error. Please add an approximation-error analysis or an ablation (e.g., VDR-Softmax vs. exact FP softmax with identical eMSB-Q) to support the 'numerical stability' claim.
  4. [§4.1, §4.4, Table 1] The hardware-overhead claim is not quantitatively supported because the ADC resolution and partial-sum bit width are never reported. The motivation in Section 3.2 argues that high-ENOB ADCs are needed for PTQ partial sums, but Table 1 and Section 4.4 only say 'low-ENOB ADC' without giving the ENOB used in the MRAM and SRAM arrays, nor the accumulation width before the SHIFT/ADD-eMSB-Q fused arithmetic. Without these numbers, the advertised area/energy savings cannot be evaluated or reproduced.
minor comments (6)
  1. [Abstract] The abstract calls the architecture 'RAP' while the title, body, and conclusion call it 'FLARE'; unify the name.
  2. [Algorithm 1] Algorithm 1's notation is under-specified: define N, the indexing of x, and especially the computation of n_e; the expression 'Í(MSB-eMSB)' is not standard notation.
  3. [Table 1] Table 1 mixes per-module and per-array entries (e.g., 'FLARE Unit PE (≒1.86*)' appears alongside 29.665); clarify the units and report the number of PEs used in the evaluations.
  4. [Figures 16-18] Figures 16-18 are referenced, but the text does not describe their axes or the measurement methodology for the GPU and PiM baselines; add descriptions of how the speedup and energy numbers were obtained.
  5. [§3.3] The 10 ns per bitwise GEMV assumption in Section 3.3 is presented without a citation; give a source or a sensitivity analysis.
  6. [§3.1, §4.2] Equation (1) uses 'lossless' to mean ratio preservation, but later sections use 'lossless' for error-free analog computation (§4.1) and for quantization (§4.2); distinguish these meanings explicitly.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity in the BitSift-GEMV speedup validation; central accuracy claims rest on end-to-end benchmarks and external exponential approximations, not on self-citation.

  1. other [Section 4.4 (BitSift-GEMV) and Section 5.3 / Fig. 16]
    "Thus along the input vector of length [D], it takes processing cycles up to roundup(D×(1-bitwise_sparsity)/8). Therefore, our BitSift-GEMV technique reduces the processing latency by (1− bitwise_sparsity) ... The boosting factors were almost directly the same as our anticipated values, revealing our BitSift-GEMV’s efficacy."

    The claimed speedup is not an independent prediction: the cycle count is defined by the zero-skipping rule, so the boosting factor relative to a fixed-length baseline is exactly 1/(1−bitwise_sparsity), a direct transformation of the measured bitwise sparsity. The 'anticipated values' in Fig. 16 are therefore the same quantity computed from the same input measurement, and their agreement confirms controller arithmetic rather than providing an independently falsifiable benefit.

full rationale

The central accuracy chain is not circular. eMSB-Q is defined against Eq. (1) as ratio-preserving quantization, but the paper does not derive benchmark accuracy from that definition; accuracy is measured end-to-end on GLUE/ImageNet against an FP32 baseline. VDR-Softmax borrows its exponential approximations from I-BERT and I-ViT (refs [32,37]), which are external works, not self-citations, and its LUT parameters depend on the exponent n_e rather than on fitted model outputs. The omission of the softmax denominator and the unproved scale-cancellation through QK^T and the V-weighted sum are correctness/completeness gaps, not circularity: the accuracy results would stand or fall on the actual simulations, not on an equation that presupposes the result. Self-citations [20,56] are background rather than load-bearing: [56] is corroborated by external references [3,30] and by the paper's own Monte-Carlo simulation, while [20] is used only to motivate token-wise parsing. The only by-construction element is the BitSift-GEMV speedup, whose 'anticipated' boosting factor follows from the same bitwise-sparsity measurement used to report the speedup; this is a minor validation tautology and does not affect the independent end-to-end accuracy evaluations.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four classes of assumptions: linear analog summation with the stated 6-sigma margin, the accuracy of the integer softmax approximation, the ability to drop per-token scales outside softmax, and the stability of bitwise sparsity. No free physical constants are introduced and no new physical entities are postulated; the free parameters are the polynomial LUT coefficients, the SAWL=8 design point, and the chosen bit widths.

free parameters (3)
  • iPOLY coefficients a, b, c, S, l = not reported (100-byte LUT indexed by n_e)
    VDR-Softmax depends on these polynomial and shift parameters to approximate e^x; the paper says they come from prior exp approximations [32,37] but gives no derivation or error bounds.
  • SAWL cap = 8 = 8 activated word lines
    Chosen so Monte Carlo gives 6-sigma error-free analog computation; the Monte Carlo setup and margin model are not described, and the ADC error model is not released.
  • Quantization bit precisions (8+1, Q_I, Q_O, BP=8) = INT8 input/weight, INT9 for eMSB-Q output
    Base precision and extra exponent bit are design choices; the accuracy results assume these widths.
assumptions (4)
  • domain assumption Analog column summation in MRAM/SRAM arrays is linear and the 6-sigma Monte Carlo error model is representative.
    The low-ENOB ADC claim relies on the reliability simulation summarized in Fig. 9-(b); the simulation details are not in the paper.
  • domain assumption The integer polynomial with base-adjusted exponent (VDR-iEXP) approximates softmax closely enough over the logit distributions that occur after eMSB-Q.
    Algorithm 1 uses a polynomial and shifting scheme without a stated error bound; accuracy is only shown end-to-end on benchmarks.
  • ad hoc to paper Per-token scale information can be dropped during linear GEMVs and restored only inside softmax without corrupting downstream attention outputs.
    Section 4.2 discards per-token scales ('temporarily storing only a single-token value in compact registers'), but no scale-propagation proof is given for the QK^T and weighted-sum stages.
  • domain assumption The bitwise sparsity measured on the tested models transfers to the reported speedups.
    Fig. 7-(e,f) reports measured sparsity; the speedup formula in Section 4.4 assumes this sparsity is stable during deployment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration." pith.science (2026). https://pith.science/paper/4ZRTEP2K

@misc{pith2026241114733,
  author       = {Pith},
  title        = {Pith review of: FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ZRTEP2K}},
  note         = {Machine review of arXiv:2411.14733}
}
abstract

Encoder-based transformers, powered by self-attention layers, have revolutionized machine learning with their context-aware representations. However, their quadratic growth in computational and memory demands presents significant bottlenecks. Analog-Mixed-Signal Process-in-Memory (AMS-PiM) architectures address these challenges by enabling efficient on-chip processing. Traditionally, AMS-PiM relies on Quantization-Aware Training (QAT), which is hardware-efficient but requires extensive retraining to adapt models to AMS-PiMs, making it increasingly impractical for transformer models. Post-Training Quantization (PTQ) mitigates this training overhead but introduces significant hardware inefficiencies. PTQ relies on dequantization-quantization (DQ-Q) processes, floating-point units (FPUs), and high-ENOB (Effective Number of Bits) analog-to-digital converters (ADCs). Particularly, High-ENOB ADCs scale exponentially in area and energy ($2^{ENOB}$), reduce sensing margins, and increase susceptibility to process, voltage, and temperature (PVT) variations, further compounding PTQ's challenges in AMS-PiM systems. To overcome these limitations, we propose RAP, an AMS-PiM architecture that eliminates DQ-Q processes, introduces FPU- and division-free nonlinear processing, and employs a low-ENOB-ADC-based sparse Matrix Vector multiplication technique. Using the proposed techniques, RAP improves error resiliency, area/energy efficiency, and computational speed while preserving numerical stability. Experimental results demonstrate that RAP outperforms state-of-the-art GPUs and conventional PiM architectures in energy efficiency, latency, and accuracy, making it a scalable solution for the efficient deployment of transformers.

Figures

Figures reproduced from arXiv: 2411.14733 by the authors.

Figure 1
Figure 1. Limitations of PTQ during inference optimizations. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Design example for typical PiM devices, where the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of self-attention, with operation type [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Thus, our optimization strategy tackles these challenging [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: (a) Even for a lossless quantization, values lose the [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: AMS-PiM with high-ENOB ADCs are more suscep [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: (a) Overall FLARE Architecture. (b) Hardware con [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 7
Figure 7. Figure 7: Comparison of sparsity measured in various granu [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: (a) Attribute and size requirements of each memory [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Our FP- and division-less integer quantization [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: (a) SAWL𝐷 controller enabling a fixed number of SAWL(=8) when the input data is insufficient of “1”s. (b) Relation between SAWL and the dynamic range variation of analog-signal-domain popcount. Limiting flexibility with SAWL relaxes the ADC-design challenges for AMS-P…
Figure 12
Figure 12. Figure 12: Our BitSift-GEMV controller parses, selects, and [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 13
Figure 13. Figure 13: Visualized Dataflow and tensor traffic of FLARE: [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Design study on array-level device parallelism. [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 17
Figure 17. Figure 17: Comparison of normalized token/sec performance. [PITH_FULL_IMAGE:figures/full_fig_p011_17.png]
Figure 16
Figure 16. Figure 16: Average amount of how much GEMV operations [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 43 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Manoj Alwani, Han Chen, Michael Ferdman, and Peter Milder. 2016. Fused-layer CNN accelerators. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 1–12

    work page 2016

  3. [3]

    Tanner Andrulis, Joel S Emer, and Vivienne Sze. 2023. RAELLA: Reforming the arithmetic for efficient, low-resolution, and low-loss analog PIM: No retraining required!. In Proceedings of the 50th Annual International Symposium on Computer Architecture. 1–16

    work page 2023

  4. [4]

    Anonymous. 2024. QRazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring. In Submitted to The Thirteenth International Conference on Learning Representations . https://openreview.net/forum?id=lwcnZmyojm under review

    work page 2024

  5. [5]

    Ron Banner, Yury Nahshan, and Daniel Soudry. 2019. Post training 4-bit quan- tization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems 32 (2019)

    2019

  6. [6]

    Avishek Biswas and Anantha P Chandrakasan. 2018. CONV-SRAM: An energy- efficient SRAM with in-memory dot-product computation for low-power con- volutional neural networks. IEEE Journal of Solid-State Circuits 54, 1 (2018), 217–230

    work page 2018

  7. [7]

    L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, et al. 2002. An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Software 28, 2 (2002), 135–151

    work page 2002

  8. [8]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

    arXiv 2020

Show all 66 references
  1. [9]

    Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Alberto Nannarelli, Marco Re, and Sergio Spanò. 2021. A pseudo-softmax function for hardware-based high speed image classification. Scientific reports 11, 1 (2021), 15307

    2021

  2. [10]

    Wei-Hao Chen, Kai-Xiang Li, Wei-Yu Lin, Kuo-Hsiang Hsu, Pin-Yi Li, Cheng- Han Yang, Cheng-Xin Xue, En-Yu Yang, Yen-Kai Chen, Yun-Sheng Chang, et al

  3. [11]

    Ping Chi, Shuangchen Li, Cong Xu, Tao Zhang, Jishen Zhao, Yongpan Liu, Yu Wang, and Yuan Xie. 2016. PRIME: A Novel Processing-in-Memory Architec- ture for Neural Network Computation in ReRAM-Based Main Memory. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Ar...

    2016

  4. [12]

    Loai Danial, Nicolás Wainstein, Shraga Kraus, and Shahar Kvatinsky. 2018. Break- ing through the speed-power-accuracy tradeoff in ADCs using a memristive neuromorphic architecture. IEEE Transactions on Emerging Topics in Computa- tional Intelligence 2, 5 (2018), 396–409

    2018

  5. [13]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems 35 (2022), 16344–16359

    2022

  6. [14]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255

    2009

  7. [15]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems 36 (2024)

    2024

  8. [16]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  9. [17]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv prepri...

    2020 arXiv

  10. [18]

    Xinjie Guo, F Merrikh Bayat, M Bavandpour, M Klachko, MR Mahmoodi, M Prezioso, KK Likharev, and DB Strukov. 2017. Fast, energy-efficient, robust, and reproducible mixed-signal neuromorphic classifier based on embedded NOR flash memory technology. In 2017 IEEE International Ele...

    2017

  11. [19]

    Saransh Gupta, Mohsen Imani, Harveen Kaur, and Tajana Simunic Rosing. 2019. Nnpim: A processing in-memory architecture for neural network acceleration. IEEE Trans. Comput. 68, 9 (2019), 1325–1337

    2019

  12. [20]

    Nguyen-Dong Ho and Ik-Joon Chang. 2023. O-2A: Ourlier-Aware Compression for 8-bit Post-Training Quantization Model. IEEE Access (2023)

    2023

  13. [21]

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2018. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research 18, 187 (2018), 1–30

    2018

  14. [22]

    Mohsen Imani, Saransh Gupta, Yeseong Kim, Minxuan Zhou, and Tajana Rosing

  15. [23]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vis...

    2018

  16. [24]

    Zhewei Jiang, Shihui Yin, Jae-Sun Seo, and Mingoo Seok. 2020. C3SRAM: An in- memory-computing SRAM macro based on robust capacitive coupling computing mechanism. IEEE Journal of Solid-State Circuits 55, 7 (2020), 1888–1897

    2020

  17. [25]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351 (2019)

    2019 arXiv

  18. [26]

    Seungchul Jung, Hyungwoo Lee, Sungmeen Myung, Hyunsoo Kim, Seung Keun Yoon, Soon-Wan Kwon, Yongmin Ju, Minje Kim, Wooseok Yi, Shinhee Han, et al

  19. [27]

    Herve Jégou, Matthijs Douze, and Cordelia Schmid. 2011. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 1 (2011), 117–128. https://doi.org/10.1109/TPAMI.2010.57

    2011 doi

  20. [28]

    Sheng-Chun Kao, Suvinay Subramanian, Gaurav Agrawal, Amir Yazdanbakhsh, and Tushar Krishna. 2023. Flat: An optimized dataflow for mitigating attention bottlenecks. In Proceedings of the 28th ACM International Conference on Archi- tectural Support for Programming Languages and ...

    2023

  21. [29]

    Arman Kazemi, Mohammad Mehdi Sharifi, Zhuowen Zou, Michael Niemier, X Sharon Hu, and Mohsen Imani. 2021. Mimhd: Accurate and efficient hyperdi- mensional inference using multi-bit in-memory computing. In 2021 IEEE/ACM International Symposium on Low Power Electronics and Design...

    2021

  22. [30]

    Hyungjun Kim, Hyunmyung Oh, and Jae-Joon Kim. 2020. Energy-efficient XNOR-free in-memory BNN accelerator with input distribution regularization. In Proceedings of the 39th International Conference on Computer-Aided Design . 1–9

    2020

  23. [31]

    Hyeonuk Kim, Jaehyeong Sim, Yeongjae Choi, and Lee-Sup Kim. 2019. Nand-net: Minimizing computational complexity of in-memory processing for binary neural networks. In 2019 IEEE international symposium on high performance computer architecture (HPCA). IEEE, 661–673

    2019

  24. [32]

    Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer

  25. [33]

    Taehyun Kim, Kwanseok Choi, Youngmock Cho, Jaehoon Cho, Hyuk-Jae Lee, and Jaewoong Sim. 2024. MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models. arXiv preprint arXiv:2405.18832 (2024)

    2024 arXiv

  26. [34]

    Z Lan. 2019. Albert: A lite bert for self-supervised learning of language represen- tations. arXiv preprint arXiv:1909.11942 (2019)

    2019 arXiv

  27. [35]

    Hunjun Lee, Minseop Kim, Dongmoon Min, Joonsung Kim, Jongwon Back, Honam Yoo, Jong-Ho Lee, and Jangwoo Kim. 2022. 3D-FPIM: An extreme energy- efficient DNN acceleration system using 3D NAND flash-based in-situ PIM unit. In 2022 55th IEEE/ACM International Symposium on Microarc...

    2022

  28. [36]

    Huize Li, Zhaoying Li, Zhenyu Bai, and Tulika Mitra. 2024. ASADI: Accelerating Sparse Attention Using Diagonal-based In-Situ Computing. In 2024 IEEE Inter- national Symposium on High-Performance Computer Architecture (HPCA) . IEEE, 774–787

    2024

  29. [37]

    Zhikai Li and Qingyi Gu. 2023. I-vit: Integer-only quantization for efficient vision transformer inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 17065–17075

    2023

  30. [38]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

    2019 arXiv

  31. [39]

    Yun Long, Taesik Na, and Saibal Mukhopadhyay. 2018. ReRAM-Based Processing- in-Memory Architecture for Recurrent Neural Network Acceleration. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 26, 12 (2018), 2781–

    2018

  32. [40]

    Abhishek Moitra, Abhiroop Bhattacharjee, and Priyadarshini Panda. 2024. PIVOT- Input-aware Path Selection for Energy-efficient ViT Inference. In Proceedings of the 61st ACM/IEEE Design Automation Conference . 1–6

    2024

  33. [41]

    Sourjya Roy, Mustafa Ali, and Anand Raghunathan. 2021. PIM-DRAM: Accel- erating machine learning workloads using processing in commodity DRAM. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 11, 4 (2021), 701–710

    2021

  34. [42]

    V Sanh. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  35. [43]

    Ali Shafiee, Anirban Nag, Naveen Muralimanohar, Rajeev Balasubramonian, John Paul Strachan, Miao Hu, R Stanley Williams, and Vivek Srikumar. 2016. ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arith- metic in Crossbars. In 2016 ACM/IEEE 43rd Annual Inte...

    2016

  36. [44]

    Xin Si, Jia-Jing Chen, Yung-Ning Tu, Wei-Hsing Huang, Jing-Hong Wang, Yen- Cheng Chiu, Wei-Chen Wei, Ssu-Yen Wu, Xiaoyu Sun, Rui Liu, et al. 2019. 24.5 A twin-8T SRAM computation-in-memory macro for multiple-bit CNN-based ma- chine learning. In 2019 IEEE International Solid-St...

    2019

  37. [45]

    Xin Si, Jia-Jing Chen, Yung-Ning Tu, Wei-Hsing Huang, Jing-Hong Wang, Yen- Cheng Chiu, Wei-Chen Wei, Ssu-Yen Wu, Xiaoyu Sun, Rui Liu, et al. 2019. A twin-8T SRAM computation-in-memory unit-macro for multibit CNN-based AI edge processors. IEEE Journal of Solid-State Circuits 55...

    2019

  38. [46]

    Shrihari Sridharan, Jacob R Stevens, Kaushik Roy, and Anand Raghunathan. 2023. X-former: In-memory acceleration of transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 31, 8 (2023), 1223–1233

    2023

  39. [47]

    Fang Su, Wei-Hao Chen, Lixue Xia, Chieh-Pu Lo, Tianqi Tang, Zhibo Wang, Kuo-Hsiang Hsu, Ming Cheng, Jun-Yi Li, Yuan Xie, et al . 2017. A 462GOPs/J RRAM-based nonvolatile intelligent processor for energy harvesting IoE system featuring nonvolatile logics and processing-in-memor...

    2017

  40. [48]

    Baohua Sun, Daniel Liu, Leo Yu, Jay Li, Helen Liu, Wenhan Zhang, and Terry Torng. 2018. MRAM co-designed processing-in-memory CNN accelerator for mobile and IoT applications. arXiv preprint arXiv:1811.12179 (2018)

    2018 arXiv

  41. [49]

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In International conference on machine learning . PMLR, 10347–10357

    2021

  42. [50]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems (2017)

    2017

  43. [51]

    Naveen Verma, Hongyang Jia, Hossein Valavi, Yinqi Tang, Murat Ozatay, Lung- Yen Chen, Bonan Zhang, and Peter Deaville. 2019. In-memory computing: Ad- vances and prospects. IEEE Solid-State Circuits Magazine 11, 3 (2019), 43–55

    2019

  44. [52]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks ...

    2018

  45. [53]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning . PMLR, 38087–38099

    2023

  46. [54]

    Cheng-Xin Xue, Wei-Hao Chen, Je-Syu Liu, Jia-Fang Li, Wei-Yu Lin, Wei-En Lin, Jing-Hong Wang, Wei-Chen Wei, Ting-Wei Chang, Tung-Cheng Chang, et al

  47. [55]

    Xiaoxuan Yang, Bonan Yan, Hai Li, and Yiran Chen. 2020. ReTransformer: ReRAM- based processing-in-memory architecture for transformer acceleration. In Pro- ceedings of the 39th International Conference on Computer-Aided Design . 1–9

    2020

  48. [56]

    Donghyeon Yi, Seoyoung Lee, Injun Choi, Gichan Yun, Edward Jongyoon Choi, Jonghee Park, Jonghoon Kwak, Sung-Joon Jang, Sohmyung Ha, Ik-Joon Chang, et al. 2024. Skew-CIM: Process-Variation-Resilient and Energy-Efficient Computation-in-Memory Design Technique With Skewed Weights...

    2024

  49. [57]

    Shihui Yin, Zhewei Jiang, Jae-Sun Seo, and Mingoo Seok. 2020. XNOR-SRAM: In-memory computing SRAM macro for binary/ternary deep neural networks. IEEE Journal of Solid-State Circuits 55, 6 (2020), 1733–1743

    2020

  50. [58]

    Minxuan Zhou, Weihong Xu, Jaeyoung Kang, and Tajana Rosing. 2022. Transpim: A memory-based acceleration via software-hardware co-design for transformer. In 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 1071–1085

    2022

  51. [59]

    Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou

  52. [60]

    In 2019 IEEE International Solid-State Circuits Conference-(ISSCC)

    24.1 A 1Mb multibit ReRAM computing-in-memory macro with 14.6 ns parallel MAC computing time for CNN based AI edge processors. In 2019 IEEE International Solid-State Circuits Conference-(ISSCC) . IEEE, 388–390

    2019

  53. [2016]

    arXiv preprint arXiv:1606.06160 (2016)

    Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 (2016)

    2016 arXiv

  54. [2018]

    In 2018 IEEE International Solid-State Circuits Conference-(ISSCC)

    A 65nm 1Mb nonvolatile computing-in-memory ReRAM macro with sub- 16ns multiply-and-accumulate for binary DNN AI edge processors. In 2018 IEEE International Solid-State Circuits Conference-(ISSCC) . IEEE, 494–496

    2018

  55. [2019]

    In Proceedings of the 2019 on Great Lakes Symposium on VLSI

    Digitalpim: Digital-based processing in-memory for big data acceleration. In Proceedings of the 2019 on Great Lakes Symposium on VLSI . 429–434

    2019

  56. [2021]

    InInternational conference on machine learning

    I-bert: Integer-only bert quantization. InInternational conference on machine learning. PMLR, 5506–5518

  57. [2022]

    Nature 601, 7892 (2022), 211–216

    A crossbar array of magnetoresistive memory devices for in-memory computing. Nature 601, 7892 (2022), 211–216

    2022

  58. [2794]

    https://doi.org/10.1109/TVLSI.2018.2819190

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.