Pith. sign in

REVIEW 3 major objections 5 minor 57 references

A Flexible Template for Edge Generative AI with High-Accuracy Accelerated Softmax & GELU

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SoftEx lifts end-to-end ViT throughput to 310 GOPS by accelerating softmax and GELU.

desk verdict Solid post-layout architecture paper with a real but addressable gap between the validated exponent algorithm and the RTL coefficient encoding. read the letter →

arxiv 2412.06321 v1 pith:WDSHQWAW submitted 2024-12-09 cs.AR

Andrea Belano , Yvan Tortorella , Angelo Garofalo , Luca Benini , Davide Rossi , Francesco Conti This is my paper · ORCID
classification cs.AR
keywords softmaxacceleratorGELUapproximateexponentiationSchraudolph'smethodBFloat16transformerinferenceRISC-VclusteredgeAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when matrix multiplication gets hardware acceleration, the remaining softmax and GELU nonlinearities become the dominant bottleneck in transformer inference, and proposes a dedicated accelerator, SoftEx, to remove it. SoftEx is built on expp, a hardware-friendly approximate exponential that refines Schraudolph's method with a polynomial correction on the mantissa. The authors report that SoftEx speeds up softmax by up to 10.8x and GELU by 5.11x over optimized RISC-V software, raising end-to-end ViT throughput to 310 GOPS (72% of the tensor engine's theoretical peak) and energy efficiency to 1.34 TOPS/W, while occupying only 3.22% of the cluster area. If correct, this shows that unmodified BF16 transformers can be run efficiently at the edge without quantization-aware retraining.

What carries the argument

The load-bearing object is expp, a BF16 exponential approximation that starts from Schraudolph's method, which treats the bit pattern of a float as a scaled integer so that exp(x) becomes roughly a multiply and an add, and applies a piecewise second-order polynomial correction to the mantissa, with the [0,1) fraction split into two halves and each half fitted by a polynomial of the form 1 + αx(x + γ). In SoftEx, expp feeds a datapath that computes softmax as a two-phase accumulation and normalization with an online denominator update, and GELU as a sum of exponentials whose fixed-point lane accumulators are kept at 14 bits. The claimed effect is that the nonlinearity cost drops to the point where the tensor engine, not the activations, sets the cluster's throughput.

What would settle it

Synthesize the SoftEx RTL and run a bit-exact comparison against the paper's Python model over the reported test range [-88.7, 88.7]; if the mean relative error of the hardware exponential exceeds the claimed 0.14% (or the ViT label mismatch exceeds 0.27% when the accelerator is used end-to-end), the central high-accuracy claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that a relatively small parametric hardware unit, SoftEx, can make softmax and GELU nearly free in an otherwise MatMul-accelerated transformer cluster, letting the cluster reach about 72% of the theoretical peak throughput on end-to-end ViT inference. The accelerator computes softmax with an online maximum and denominator update that avoids a separate max pass, and computes GELU by approximating the Gaussian CDF as a sum of exponentials whose leading term is the same expp exponential; accumulation is done in fixed point with enough bits to keep model accuracy. On MobileBERT's attention layer SoftEx reaches up to 324 GOPS (75% of peak), and on ViT it achieves 310 GOPS at 0.8V and 1.34 TOPS/W at 0.55V, with a mean relative error of 0.14% for the exponential approximation and a 0.27% label mismatch on ImageNet1k for ViT.

Load-bearing premise

The central claim relies on the physical exponential unit in SoftEx having the same accuracy as the Python/PyTorch model of expp, since the paper states $\alpha$=0.21875 is stored as a 4-bit integer with scale $2^{-4}$, which cannot represent that value exactly, and no bit-accurate hardware-versus-algorithm co-simulation is reported.

Editorial extensions

If this is right

  • Unmodified BF16 transformers, including ViT and MobileBERT, can be executed at roughly 69 to 72 percent of the tensor engine's theoretical peak without quantization-aware fine-tuning.
  • Softmax and GELU, which dominate runtime when MatMul is accelerated, are reduced to a small fraction of total runtime, making further MatMul scaling worthwhile.
  • The same cluster design scales to an 8x8 mesh with a modeled 18.2 TOPS on GPT-2 XL, with per-cluster throughput at 82.6% of a standalone configuration.
  • Accuracy remains close to the base model: 0.27% label mismatch on ImageNet1k for ViT and a perplexity of 37.816 on WikiText-2 for GPT-2.
  • The accuracy claims are tied to the Python model of expp; if the synthesized RTL quantizes the correction coefficients differently, the reported error numbers could shift on silicon.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because expp is a generic exp approximator and GELU is built from a sum of exponentials, the same SoftEx datapath could plausibly accelerate other exponential-based nonlinearities such as SiLU or hard variants, though the paper does not evaluate them.
  • The paper's stated 4-bit encoding of alpha (0.21875 with scale 2^-4) cannot represent that value exactly, so the actual hardware behavior depends on the rounding scheme; a bit-exact comparison of the RTL against the modeled expp would settle whether the 0.14% mean relative error and the 0.27% label mismatch hold on silicon.
  • A testable extension would be applying SoftEx to decoder-heavy generative models with longer sequences, where the online softmax update may interact differently with memory bandwidth and where the GELU sum of exponentials might need more terms.
  • The scalability analysis assumes conservative NoC contention; a hardware prototype or cycle-accurate simulation of the 8x8 mesh would verify whether the modeled 18.2 TOPS holds, especially for small tiles where per-cluster overheads matter more.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents SoftEx, a parametric hardware accelerator for softmax and GELU in BFloat16, integrated into a PULP-based heterogeneous cluster with 8 RISC-V cores and a 24×8 systolic-array MatMul engine. The authors introduce expp, an approximate exponential based on Schraudolph's method with a polynomial correction, and report post-layout results in GlobalFoundries 12nm: SoftEx occupies 0.039 mm^2 (3.22% of the cluster), accelerates softmax and GELU by up to 10.8× and 5.11× over optimized RISC-V software, and improves end-to-end ViT throughput to 310 GOPS (1.58×) and energy efficiency to 1.34 TOPS/W (1.42×). A scalability analysis of an 8×8 mesh of such clusters on GPT-2 XL predicts 18.2 TOPS ensemble throughput.

Significance. If the accuracy and performance claims hold, the work is significant: it shows that a small dedicated unit can remove the softmax/GELU bottleneck for unmodified BF16 Transformer inference, a gap not addressed by prior integer-only accelerators that require quantization-aware fine-tuning. The paper's strengths are its concrete post-layout measurements, the parametric design with open-source RTL, the use of independent end-to-end benchmarks (MobileBERT, ViT, GPT-2), and the clear 0.14% mean relative error target. The main risk is that the accuracy numbers may not correspond to the actual silicon datapath, because the coefficients are validated only in Python/PyTorch and one stated coefficient is not representable in the claimed fixed-point format.

major comments (3)
  1. [Section IV, Eq. (14) and encoding paragraph] The stated parameter α = 0.21875 cannot be exactly represented in the specified 4-bit integer format with scaling 2^-4, since 0.21875·16 = 3.5 is not an integer; the RTL must therefore use a neighboring value such as 0.1875 or 0.25. Because the accuracy results in Section VI-A (mean relative error 0.14%, softmax error 0.44%) and Section VI-B (0.27% ViT label mismatch) are produced by a Python/PyTorch implementation (Section VII-A) rather than by bit-accurate RTL co-simulation, the reported accuracy of the actual hardware datapath is not established. The authors should either change the encoding so that α is exactly representable, or state the exact encoded coefficient and re-run the accuracy evaluation with that value; ideally they should also provide an RTL-in-the-loop or bit-accurate comparison.
  2. [Section VI-B / Algorithm 1] The specific a_i and b_i coefficients used for the four-term sum-of-exponentials GELU approximation are not listed in the paper, so the exact GELU function implemented in SoftEx is not reproducible from the manuscript. The paper states that Eq. 7 was solved for r(0) = -r_max and x_{2N+1} = 2.8, but the resulting coefficient values are not given; the open-source repository is a partial remedy, but a journal paper should include the parameters. Please add a table (or an appendix) with the coefficients for N=4 (and possibly the other term counts shown in Fig. 5), and indicate which quantization is applied to them in the RTL.
  3. [Section VII-A] The algorithms are validated only in Python using PyTorch; no RTL-level simulation of SoftEx is reported. The softmax datapath includes fixed-point conversions, a Newton-Raphson reciprocal, and a dynamic rescaling mechanism for online normalization, while the GELU path uses fixed-point lane accumulators; each of these can introduce error beyond the Python model. A bit-accurate RTL simulation comparing the hardware datapath against the Python model would be needed to support the end-to-end accuracy claims (e.g., 0.27% label mismatches on ViT). Without such a check, the 'high-accuracy' qualifier is not tied to the actual implementation.
minor comments (5)
  1. [Section VI-A1] The text says '0.03% lower than out algorithm'; 'out' should be 'our'.
  2. [Section VII-C and Table I] The text reports the tensor processing unit's peak efficiency as 1.72 TOPS/W, while Table I lists 1.61 TOPS/W for this work; the discrepancy should be reconciled or clarified (e.g., by specifying that the table value refers to a different operating point or to the whole cluster).
  3. [Table II] The 7nm scaling footnote gives an equation but no voltage values; with equal voltages the efficiency would scale by 12/7, not by the implied ≈2.6×, so the assumed voltage ratio should be stated explicitly.
  4. [Section VI-A2] The definition of 'mean relative error' for the softmax outputs is not given; because softmax outputs include very small probabilities, the error metric should specify how near-zero entries are treated (e.g., absolute-error weighting or a floor).
  5. [Section VIII] The scalability model's conflict-delay assumption (uniform [0,0.5] cycles per hop) and the Monte Carlo methodology are stated, but no sensitivity analysis is provided; please add a short discussion of how the results depend on these assumptions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SoftEx's accuracy and speedup results are measured against external baselines and independent workloads, not derived from its own fitted coefficients or self-citations.

full rationale

The paper's derivation chain is self-contained against external benchmarks. The expp coefficients are fitted by a Monte Carlo procedure to minimize error with respect to glibc's exp (Section IV), and the reported 0.14% mean relative error is then measured on 10^8 fresh samples in the same input range (Section VI-A.1); the softmax error and MobileBERT logit MSE are computed on real attention tensors, and the GELU/ViT/GPT-2 accuracy numbers are measured on ImageNet1k and WikiText-2 after selecting four terms and 14-bit accumulators from the Fig. 5 sweep. That selection is a post-hoc design tradeoff reported as a measured outcome, not a prediction forced by the fitted coefficients, so it does not reduce by construction. The cited RedMulE/PULP baselines are prior-group hardware whose properties are not invoked to forbid alternatives, and the central SoftEx speedups are benchmarked against software on the same cluster rather than derived from self-citations. Two flagged issues are correctness risks, not circularity: Section VII-A says 'The algorithms have been validated by implementing them in Python using the PyTorch library', with no bit-accurate RTL-in-the-loop co-simulation reported, and Section IV states 'To represent the α and β parameters we use a 4-bit integer number with a fixed scaling factor of 2−4' while α=0.21875 implies 0.21875*16=3.5, which is not an integer in that format. These concerns affect the hardware/algorithm equivalence of the accuracy claim, but no equation in the paper is shown to equal its own input by construction, so the circularity score remains 0.

Assumptions & free parameters 7 free parameters · 7 assumptions · 0 invented entities

The ledger is dominated by fitted coefficients for the exp approximation and post-hoc architectural choices (number of GELU terms, accumulator width, lane count). The mathematical core is standard; the riskier assumptions are the transfer of Python-level accuracy to the quantized RTL and the representativeness of the benchmark inputs.

free parameters (7)
  • alpha (expp first-half polynomial coefficient) = 0.21875
    Fitted by Monte Carlo (1e6 trials) to minimize relative error of expp vs exp; Section IV. Stated to be stored as a 4-bit integer with 2^-4 scaling, but 0.21875*16=3.5 is not representable in 4 bits.
  • beta (expp second-half polynomial coefficient) = 0.4375
    Fitted by the same Monte Carlo procedure; Section IV.
  • gamma1 (expp first-half polynomial offset) = 3.296875
    Fitted by the same Monte Carlo procedure; Section IV.
  • gamma2 (expp second-half polynomial offset) = 2.171875
    Fitted by the same Monte Carlo procedure; Section IV.
  • GELU terms Nw and lane-accumulator width = 4 terms, 14 bits
    Chosen from accuracy sweeps on ViT/ImageNet-1k and GPT-2/WikiText-2 (Fig. 5), a post-hoc design choice balancing model accuracy against area, not a first-principles derivation.
  • SoftEx lane count N = 16
    Chosen via area and latency scaling analysis (Fig. 8) as a compromise.
  • GELU approximation endpoint parameters = r(0)=-rmax, x_{2N+1}=2.8
    Chosen by the authors (Section VI.B) based on GELU's behavior near zero and saturation beyond 2.8; these set the error profile in the minmax Q-function approximation.
assumptions (7)
  • standard math Schraudolph's base-2 reinterpretation approximation is a valid starting point for exponentiation in hardware.
    The paper builds expp on Algorithm 2 from [27] and cites the original work for its proof; Section IV.
  • standard math The online softmax update equality Den(x,N)=Den(x,N-1)*e^{Max(x,N-1)-Max(x,N)}+e^{x_N-Max(x,N)} is exact.
    Used for the single-pass accumulation step; Eq. 2 in Section III-B.
  • domain assumption Two Newton-Raphson iterations on the FP32 FMA starting from the stated seed yield a reciprocal accurate enough for the reported softmax accuracy.
    The seed is the exact exponent 2B-1-E and mantissa estimate (1-M)^2/2; convergence and final accuracy are not quantified in the paper (Section V-B-2-b).
  • domain assumption The uniform distribution over [-88.7, 88.7] used to measure expp error is representative of the input distribution of exponential units in softmax and GELU in real models.
    Accuracy characterization in Section VI-A-1 uses 1e8 uniform samples; real attention logits and GELU pre-activations have different, model-dependent distributions, and only a single MobileBERT attention layer is used for softmax validation.
  • domain assumption For GELU inputs with |x| > 2.8, Phi(x) is effectively 0 or 1 and GELU(x) equals x (or 0), so truncating the approximation range at 2.8 is safe.
    Stated in Section VI-B; this is a modeling choice relying on the concentration of the activation distribution.
  • ad hoc to paper The RTL implementation of expp and SoftEx exactly realizes the Python/PyTorch-validated algorithm, including the quantized coefficient encoding.
    Accuracy is measured in Python (Section VI), while performance is measured on the placed-and-routed RTL (Section VII). No bit-accurate co-simulation is reported, and the alpha encoding inconsistency makes this assumption non-trivial.
  • domain assumption In the scalability model, per-hop conflict delays are independent uniform random variables in [0,0.5] cycles and worst-case path delay bounds the mesh slowdown.
    Listed as conservative assumptions in Section VIII; the model is not validated against cycle-accurate NoC simulation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Flexible Template for Edge Generative AI with High-Accuracy Accelerated Softmax & GELU." pith.science (2026). https://pith.science/paper/WDSHQWAW

@misc{pith2026241206321,
  author       = {Pith},
  title        = {Pith review of: A Flexible Template for Edge Generative AI with High-Accuracy Accelerated Softmax & GELU},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WDSHQWAW}},
  note         = {Machine review of arXiv:2412.06321}
}
abstract

Transformer-based generative Artificial Intelligence (GenAI) models achieve remarkable results in a wide range of fields, including natural language processing, computer vision, and audio processing. However, this comes at the cost of increased complexity and the need of sophisticated non-linearities such as softmax and GELU. Even if Transformers are computationally dominated by matrix multiplications (MatMul), these non-linearities can become a performance bottleneck, especially if dedicated hardware is used to accelerate MatMul operators. In this work, we introduce a GenAI BFloat16 Transformer acceleration template based on a heterogeneous tightly-coupled cluster containing 256KiB of shared SRAM, 8 general-purpose RISC-V cores, a 24x8 systolic array MatMul accelerator, and a novel accelerator for Transformer softmax and GELU non-linearities: SoftEx. SoftEx introduces an approximate exponentiation algorithm balancing efficiency (121x speedup over glibc's implementation) with accuracy (mean relative error of 0.14%). In 12nm technology, SoftEx occupies 0.039 mm$^2$, only 3.22% of the cluster, which achieves an operating frequency of 1.12 GHz. Compared to optimized software running on the RISC-V cores, SoftEx achieves significant improvements, accelerating softmax and GELU computations by up to 10.8x and 5.11x, respectively, while reducing their energy consumption by up to 10.8x and 5.29x. These enhancements translate into a 1.58x increase in throughput (310 GOPS at 0.8V) and a 1.42x improvement in energy efficiency (1.34 TOPS/W at 0.55V) on end-to-end ViT inference workloads.

Figures

Figures reproduced from arXiv: 2412.06321 by the authors.

Figure 1
Figure 1. Breakdown of one of ViT’s layers’ runtime running on a 8 core PULP [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The circuit implementing the correction proposed in Section [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Architecture of the enhanced PULP cluster proposed in this work. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: A detailed view of SoftEx and its Datapath. In the left image, the paths used in the calculation of Softmax are highlighted, with paths used in the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: The effects of changing the number of bits in the lane accumulators and number of terms in the sum of exponentials. From left to right: the number [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: SoftEx’s area breakdown and the annotated layout of the proposed [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: SoftEx’s average latency on 2048-long vectors (a, b) and area (c) with [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Runtime breakdown of GELU on 2 14 elements calculated either using only the cores or assisted with SoftEx. When run entirely in software, GELU is approximated using the sigmoid function (Eq. 5), when assisted with SoftEx, a four-term sum of exponentials is used. 128 25…
Figure 10
Figure 10. Figure 10: System throughput @0.80V (a) and energy efficiency @0.55V [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 13
Figure 13. Figure 13: Runtime breakdown of the kernels inside ViT using SoftEx or the [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: Dataflow modeled for scalability analysis. [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]
Figure 15
Figure 15. Figure 15: Performance figures of meshes of clusters of different sizes on GPT-2 XL. From left to right: The cumulative throughput achieved by the mesh [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 51 canonical work pages

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer et al., “Attention is all you need,” in Advances in Neural Information Processing Systems , vol. 30. Curran Associates, Inc., 2017

    work page 2017

  2. [2]

    Swin Transformer: Hierarchical Vision Trans- former Using Shifted Windows,

    Z. Liu, Y . Lin et al. , “Swin Transformer: Hierarchical Vision Trans- former Using Shifted Windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10 012–10 022

    work page 2021

  3. [3]

    Compressed-Domain Vision Transformer for Image Classification,

    R. Ji and L. J. Karam, “Compressed-Domain Vision Transformer for Image Classification,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 14, no. 2, pp. 299–310, Jun. 2024

    work page 2024

  4. [4]

    AST: Audio spectrogram trans- former,

    Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio spectrogram trans- former,” in Proc. Interspeech 2021 , 2021, pp. 571–575

    2021

  5. [5]

    LLaMA: Open and Efficient Foundation Language Models,

    H. Touvron, T. Lavril et al. , “LLaMA: Open and Efficient Foundation Language Models,” Feb. 2023. [Online]. Available: http://arxiv.org/abs/2302.13971

    arXiv 2023

  6. [6]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu et al., “Language models are unsupervised multitask learners,” 2019. [Online]. Available: https://insightcivic.s3.us-east-1. amazonaws.com/language-models.pdf

    work page 2019

  7. [7]

    CGVC-T: Contextual Generative Video Compression With Transformers,

    P. Du, Y . Liu, and N. Ling, “CGVC-T: Contextual Generative Video Compression With Transformers,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 14, no. 2, pp. 209–223, Jun. 2024

    work page 2024

  8. [8]

    FVIFormer: Flow-Guided Global-Local Aggre- gation Transformer Network for Video Inpainting,

    W. Yan, Y . Sun et al., “FVIFormer: Flow-Guided Global-Local Aggre- gation Transformer Network for Video Inpainting,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 14, no. 2, pp. 235–244, Jun. 2024

    work page 2024

Show all 57 references
  1. [9]

    TM-GAN: A Transformer- Based Multi-Modal Generative Adversarial Network for Guided Depth Image Super-Resolution,

    J. Zhu, V . K. Z. Koh, Z. Lin, and B. Wen, “TM-GAN: A Transformer- Based Multi-Modal Generative Adversarial Network for Guided Depth Image Super-Resolution,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems , vol. 14, no. 2, pp. 261–274, Jun. 2024

    2024

  2. [10]

    RT-1: Robotics Transformer for Real-World Control at Scale,

    A. Brohan, N. Brown et al. , “RT-1: Robotics Transformer for Real-World Control at Scale,” Aug. 2023. [Online]. Available: http://arxiv.org/abs/2212.06817

    2023 arXiv

  3. [11]

    Empowering generative AI through mobile edge computing,

    L. Ale, N. Zhang, S. A. King, and D. Chen, “Empowering generative AI through mobile edge computing,” Nature Reviews Electrical Engi- neering, vol. 1, no. 7, pp. 478–486, Jul. 2024

    2024

  4. [12]

    An Overview on Generative AI at Scale With Edge–Cloud Computing,

    Y .-C. Wang, J. Xue, C. Wei, and C. C. J. Kuo, “An Overview on Generative AI at Scale With Edge–Cloud Computing,” IEEE Open Journal of the Communications Society , vol. 4, pp. 2952–2971, 2023

    2023

  5. [13]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016, pp. 770–778. PRE-PRINT SUBMITTED TO IEEE JOURNAL OF EMERGING AND SELECTED TOPICS IN CIRCUITS AND SYSTEMS 14

    2016

  6. [14]

    MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,

    A. G. Howard, M. Zhu et al. , “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” Apr. 2017. [Online]. Available: http://arxiv.org/abs/1704.04861

    2017 arXiv

  7. [15]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, L. Beyer et al., “An image is worth 16x16 words: Trans- formers for image recognition at scale,” in International Conference on Learning Representations, Oct. 2020

    2020

  8. [16]

    QLoRA: Efficient Finetuning of Quantized LLMs,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” Advances in Neural Informa- tion Processing Systems , vol. 36, pp. 10 088–10 115, Dec. 2023

    2023

  9. [17]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,” Mar. 2023. [Online]. Available: http://arxiv.org/abs/2210.17323

    2023 arXiv

  10. [18]

    AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration,

    J. Lin, J. Tang et al. , “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration,” in Proceedings of Machine Learning and Systems , vol. 6, May 2024, pp. 87–100

    2024

  11. [19]

    SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models,

    G. Xiao, J. Lin et al. , “SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models,” in Proceedings of the 40th International Conference on Machine Learning . PMLR, Jul. 2023, pp. 38 087–38 099

    2023

  12. [20]

    Toward Attention-based TinyML: A Heterogeneous Accelerated Architecture and Automated Deployment Flow,

    P. Wiese, G. ˙Islamo˘glu et al. , “Toward Attention-based TinyML: A Heterogeneous Accelerated Architecture and Automated Deployment Flow,” Aug. 2024. [Online]. Available: http://arxiv.org/abs/2408.02473

    2024 arXiv

  13. [21]

    A 17–95.6 TOPS/W deep learning inference accelerator with per-vector scaled 4-bit quantization for trans- formers in 5nm,

    B. Keller, R. Venkatesan et al. , “A 17–95.6 TOPS/W deep learning inference accelerator with per-vector scaled 4-bit quantization for trans- formers in 5nm,” in 2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits) , Jun. 2022, pp. 16–17

    2022

  14. [22]

    Deeploy: Enabling Energy-Efficient Deployment of Small Language Models on Heterogeneous Microcon- trollers,

    M. Scherer, L. Macan et al. , “Deeploy: Enabling Energy-Efficient Deployment of Small Language Models on Heterogeneous Microcon- trollers,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 43, no. 11, pp. 4009–4020, Nov. 2024

    2024

  15. [23]

    RedMule: A mixed-precision matrix– matrix operation engine for flexible and energy-efficient on-chip linear algebra and TinyML training acceleration,

    Y . Tortorella, L. Bertaccini et al., “RedMule: A mixed-precision matrix– matrix operation engine for flexible and energy-efficient on-chip linear algebra and TinyML training acceleration,”Future Generation Computer Systems, vol. 149, pp. 122–135, Dec. 2023

    2023

  16. [24]

    Paulin, P

    G. Paulin, P. Scheffler et al. , “Occamy: A 432-Core 28.1 DP- GFLOP/s/W 83% FPU Utilization Dual-Chiplet, Dual-HBM2E RISC-V- Based Accelerator for Stencil and Sparse Linear Algebra Computations with 8-to-64-bit Floating-Point Support in 12nm FinFET,” in 2024 IEEE Symposium on ...

    2024

  17. [25]

    BEiT: BERT Pre-Training of Image Transformers,

    H. Bao, L. Dong, S. Piao, and F. Wei, “BEiT: BERT Pre-Training of Image Transformers,” in International Conference on Learning Repre- sentations, Apr. 2022

    2022

  18. [26]

    Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks,

    B. Xiao, H. Wu et al., “Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 4818–4829

    2024

  19. [27]

    A fast, compact approximation of the exponential function,

    N. N. Schraudolph, “A fast, compact approximation of the exponential function,” Neural Computation, vol. 11, no. 4, pp. 853–862, May 1999

    1999

  20. [28]

    Gaussian Error Linear Units (GELUs),

    D. Hendrycks and K. Gimpel, “Gaussian Error Linear Units (GELUs),” Jun. 2023. [Online]. Available: http://arxiv.org/abs/1606.08415

    2023 arXiv

  21. [29]

    Robust Speech Recognition via Large- Scale Weak Supervision,

    A. Radford, J. W. Kim et al. , “Robust Speech Recognition via Large- Scale Weak Supervision,” in Proceedings of the 40th International Conference on Machine Learning. PMLR, Jul. 2023, pp. 28 492–28 518

    2023

  22. [30]

    DaViT: Dual Attention Vision Transformers,

    M. Ding, B. Xiao et al., “DaViT: Dual Attention Vision Transformers,” in Computer Vision – ECCV 2022 , S. Avidan, G. Brostow et al. , Eds. Cham: Springer Nature Switzerland, 2022, pp. 74–92

    2022

  23. [31]

    Fast Exponential Computation on SIMD Architectures,

    C. Malossi, Y . Ineichen, C. Bekas, and A. Curioni, “Fast Exponential Computation on SIMD Architectures,” in HiPEAC, Jan. 2015

    2015

  24. [32]

    A high speed SoftMax VLSI architecture based on basic-split,

    Q. Sun, Z. Di et al., “A high speed SoftMax VLSI architecture based on basic-split,” in 2018 14th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT) , Oct. 2018, pp. 1–3

    2018

  25. [33]

    Design and Implementation of an Approximate Softmax Layer for Deep Neural Networks,

    Y . Gao, W. Liu, and F. Lombardi, “Design and Implementation of an Approximate Softmax Layer for Deep Neural Networks,” in 2020 IEEE International Symposium on Circuits and Systems (ISCAS) , Oct. 2020, pp. 1–5

    2020

  26. [34]

    Efficient Precision-Adjustable Architecture for Softmax Function in Deep Learning,

    D. Zhu, S. Lu et al. , “Efficient Precision-Adjustable Architecture for Softmax Function in Deep Learning,” IEEE Transactions on Circuits and Systems II: Express Briefs , vol. 67, no. 12, pp. 3382–3386, Dec. 2020

    2020

  27. [35]

    Efficient hardware architecture of softmax layer in deep neural network,

    B. Yuan, “Efficient hardware architecture of softmax layer in deep neural network,” in 2016 29th IEEE International System-on-Chip Conference (SOCC), Sep. 2016, pp. 323–326

    2016

  28. [36]

    22.9 a 12nm 18.1tflops/W sparse transformer processor with entropy-based early exit, mixed-precision predication and fine-grained power management,

    T. Tambe, J. Zhang et al., “22.9 a 12nm 18.1tflops/W sparse transformer processor with entropy-based early exit, mixed-precision predication and fine-grained power management,” in 2023 IEEE International Solid- State Circuits Conference (ISSCC) , Feb. 2023, pp. 342–344

    2023

  29. [37]

    Base-2 softmax function: Suitability for training and efficient hardware implementation,

    Y . Zhang, Y . Zhang et al. , “Base-2 softmax function: Suitability for training and efficient hardware implementation,” IEEE Transactions on Circuits and Systems I: Regular Papers , vol. 69, no. 9, pp. 3605–3618, Sep. 2022

    2022

  30. [38]

    NN-LUT: Neural approximation of non-linear operations for efficient transformer inference,

    J. Yu, J. Park et al. , “NN-LUT: Neural approximation of non-linear operations for efficient transformer inference,” in Proceedings of the 59th ACM/IEEE Design Automation Conference , ser. DAC ’22. New York, NY , USA: Association for Computing Machinery, Aug. 2022, pp. 577–582

    2022

  31. [39]

    ViTA: A Highly Efficient Dataflow and Architecture for Vision Transformers,

    C. Chen, L. Li, and M. M. Sabry Aly, “ViTA: A Highly Efficient Dataflow and Architecture for Vision Transformers,” in 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE) , Mar. 2024, pp. 1–6

    2024

  32. [40]

    Enabling Efficient Hardware Acceleration of Hybrid Vision Transformer (ViT) Networks at the Edge,

    J. Dumoulin, P. Houshmand, V . Jain, and M. Verhelst, “Enabling Efficient Hardware Acceleration of Hybrid Vision Transformer (ViT) Networks at the Edge,” in 2024 IEEE International Symposium on Circuits and Systems (ISCAS) , May 2024, pp. 1–5

    2024

  33. [41]

    SambaNova SN10 RDU: A 7nm Dataflow Architecture to Accelerate Software 2.0,

    R. Prabhakar, S. Jairath, and J. L. Shin, “SambaNova SN10 RDU: A 7nm Dataflow Architecture to Accelerate Software 2.0,” in 2022 IEEE International Solid-State Circuits Conference (ISSCC) , vol. 65, Feb. 2022, pp. 350–352

    2022

  34. [42]

    The Wormhole AI Training Processor,

    D. Ignjatovi ´c, D. W. Bailey, and L. Baji ´c, “The Wormhole AI Training Processor,” in 2022 IEEE International Solid-State Circuits Conference (ISSCC), vol. 65, Feb. 2022, pp. 356–358

    2022

  35. [43]

    Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point,

    B. Darvish Rouhani, D. Lo et al. , “Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point,” in Advances in Neural Information Processing Systems , vol. 33. Curran Associates, Inc., 2020, pp. 10 271–10 281

    2020

  36. [44]

    FlexBlock: A Flexible DNN Training Acceler- ator With Multi-Mode Block Floating Point Support,

    S.-H. Noh, J. Koo et al., “FlexBlock: A Flexible DNN Training Acceler- ator With Multi-Mode Block Floating Point Support,”IEEE Transactions on Computers, vol. 72, no. 9, pp. 2522–2535, Sep. 2023

    2023

  37. [45]

    FAST: DNN Training Un- der Variable Precision Block Floating Point with Stochastic Rounding,

    S. Qian Zhang, B. McDanel, and H. T. Kung, “FAST: DNN Training Un- der Variable Precision Block Floating Point with Stochastic Rounding,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA), Apr. 2022, pp. 846–860

    2022

  38. [46]

    MobileBERT: A Compact Task-Agnostic BERT for Resource-Limited Devices,

    Z. Sun, H. Yu et al. , “MobileBERT: A Compact Task-Agnostic BERT for Resource-Limited Devices,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Comput...

    2020

  39. [47]

    New exponential bounds and approximations for the computation of error probability in fading channels,

    M. Chiani, D. Dardari, and M. Simon, “New exponential bounds and approximations for the computation of error probability in fading channels,” IEEE Transactions on Wireless Communications, vol. 2, no. 4, pp. 840–845, Jul. 2003

    2003

  40. [48]

    Global Minimax Approximations and Bounds for the Gaussian Q-Function by Sums of Exponentials,

    I. M. Tanash and T. Riihonen, “Global Minimax Approximations and Bounds for the Gaussian Q-Function by Sums of Exponentials,” IEEE Transactions on Communications, vol. 68, no. 10, pp. 6514–6524, Oct. 2020

    2020

  41. [49]

    SQuAD: 100,000+ Questions for Machine Comprehension of Text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ Questions for Machine Comprehension of Text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Com...

    2016

  42. [50]

    Neural Network Accept- ability Judgments,

    A. Warstadt, A. Singh, and S. R. Bowman, “Neural Network Accept- ability Judgments,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 625–641, 2019

    2019

  43. [51]

    ImageNet: A large-scale hierarchical image database,

    J. Deng, W. Dong et al. , “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2009, pp. 248–255

    2009

  44. [52]

    Pointer Sentinel Mix- ture Models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer Sentinel Mix- ture Models,” in International Conference on Learning Representations, Apr. 2017

    2017

  45. [53]

    FlooNoC: A 645 Gbps/link 0.15 pJ/B/hop Open-Source NoC with Wide Physical Links and End-to-End AXI4 Parallel Multi-Stream Support,

    T. Fischer, M. Rogenmoser et al. , “FlooNoC: A 645 Gbps/link 0.15 pJ/B/hop Open-Source NoC with Wide Physical Links and End-to-End AXI4 Parallel Multi-Stream Support,” Sep. 2024. [Online]. Available: http://arxiv.org/abs/2409.17606

    2024 arXiv

  46. [54]

    LPDDR5 part detail

    “LPDDR5 part detail.” [Online]. Available: https: //www.micron.com/products/memory/dram-components/lpddr5/ part-catalog/part-detail/mt62f512m32d2ds-031-wt-b

  47. [55]

    Hardware-Efficient SoftMax Architecture With Bit-Wise Exponentiation and Reciprocal Calculation,

    J. Kim, S. Kim, K. Choi, and I.-C. Park, “Hardware-Efficient SoftMax Architecture With Bit-Wise Exponentiation and Reciprocal Calculation,” IEEE Transactions on Circuits and Systems I: Regular Papers, pp. 1–12, 2024

    2024

  48. [56]

    Optimizing Foundation Model Inference on a Many-Tiny-Core Open-Source RISC-V Platform,

    V . Potocnik, L. Colagrande et al. , “Optimizing Foundation Model Inference on a Many-Tiny-Core Open-Source RISC-V Platform,” IEEE Transactions on Circuits and Systems for Artificial Intelligence , vol. 1, no. 1, pp. 37–52, Sep. 2024. PRE-PRINT SUBMITTED TO IEEE JOURNAL OF EME...

    2024

  49. [57]

    to calculate PN i=1 aie−bix2 ; 3) if x >0, complement the result of 2); 4) multiply x by the result of 3)

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.