Pith. sign in

REVIEW 3 major objections 5 minor 40 references

SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SoftmAP implements integer-only Softmax on associative processors, reporting up to 1300x energy and 12.58x latency savings over A100 and RTX3090 GPUs for Llama2 models.

desk verdict Genuine precision study and a plausible AP mapping, but the headline efficiency claim rests on an unvalidated simulator and the 'no compromise' claim is undercut by the paper's own perplexity numbers. read the letter →

arxiv 2411.17847 v1 pith:U2FRI7X7 submitted 2024-11-26 cs.AR cs.AI

Mariam Rakka , Jinhao Li , Guohao Dai , Ahmed Eltawil , Mohammed E. Fouda , Fadi Kurdahi This is my paper · ORCID
classification cs.ARcs.AI
keywords largelanguagemodelsSoftmaxquantizationin-memorycomputingassociativeprocessorsinteger-onlyapproximationenergy-delayproductLlama2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that Softmax, a non-linear operator that increasingly bottlenecks large language models at long sequence lengths, can be approximated with integer-only arithmetic and run on in-memory associative processors without a meaningful perplexity penalty. If correct, this would remove a quantization-resistant obstacle to deploying LLMs on resource-constrained devices, because the approximation uses only multiplications, additions, and shifts. The authors evaluate the approximation on Llama2-7b, 13b, and 70b over WikiText-2, and report that the associative processor beats A100 and RTX3090 GPUs by up to 1300x in energy and 12.58x in latency for the Softmax operation, reducing the energy-delay product by up to three orders of magnitude.

What carries the argument

The load-bearing object is the integer-only Softmax approximation adapted from I-BERT: after subtracting the maximum for numerical stability, the negative input is split into a quotient and remainder relative to $\ln(2)$, the remainder is passed through the polynomial $(v_{\text{corr}} + v_b)^2 + v_c$, and the result is shifted by the quotient to approximate the exponential. Barrett reduction replaces the modulo division with a precomputed constant multiplication and shifts, keeping everything in integer arithmetic. On the hardware side, a 2D associative processor stores the vocabulary entries as rows and executes the step sequence in Fig. 5 bit-serially, word-parallel, so each Softmax over sequence-length/2 words is done in parallel; the paper identifies $v_{\text{corr}} = M$, $M = 6$, $N = 16$ as the best precision combination across the three Llama2 models.

What would settle it

Run the same integer-only Softmax dataflow on a cycle-accurate or fabricated 16nm associative processor and measure energy and latency; if the measured energy-delay product advantage over an A100 falls below an order of magnitude, the paper's central hardware claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that the exponential inside Softmax can be faithfully replaced by a second-order polynomial plus a shift, computed entirely in low-precision integers, and that the resulting approximation can be mapped onto a two-dimensional associative processor operating bit-serially across words in parallel. With a clipping threshold of -7, a Softmax input precision of $M=6$, and a sum precision of $N=16$, the approximation keeps WikiText-2 perplexity at 5.92 for Llama2-7b and 5.35 for Llama2-13b, against floating-point baselines of 5.47 and 4.88, respectively. The paper further claims that this mixed-precision integer-only implementation on a 16nm associative processor consumes up to 1300x less energy and runs up to 12.58x faster than the Softmax operation on A100 and RTX3090 GPUs, with the largest advantages at long sequence lengths and small batch sizes.

Load-bearing premise

The 1300x energy and 12.58x latency claims rest on unvalidated simulator formulas and a cross-process-node comparison against GPUs.

Editorial extensions

If this is right

  • Softmax can be computed in integer-only arithmetic with a second-order polynomial and Barrett reduction, enabling quantization-friendly LLM inference without floating-point units.
  • For sequence lengths above 1024, where Softmax can reach 38% of Llama2-7b's runtime on an A100, the AP's latency advantage translates into overall execution time reductions (e.g., 10.71% for Llama2-70b at length 4096).
  • The best mixed-precision setting ($M=6$, $N=16$, $v_{\text{corr}}=M$) keeps perplexity within 0.45-0.47 of floating-point for Llama2-7b/13b on WikiText-2, so the approximation is usable for generation tasks.
  • The AP's edge over GPUs grows with sequence length and shrinks at short lengths, meaning the technique pairs naturally with long-context LLM inference.
  • SoftmAP is orthogonal to parallelization methods like FlashAttention-2, so it can be applied per-partition to further accelerate Softmax.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Table II energy/latency formulas hold, the same integer polynomial approach could be extended to LayerNorm, the other non-linear quantization bottleneck, potentially removing all floating-point ops from a transformer block; the paper does not address this.
  • The 16nm AP vs 7nm/8nm GPU comparison is not process-node normalized; on a common node the 1300x energy claim would likely shrink, though the in-memory computing advantage may persist.
  • A testable extension is to run the same approximation on Llama3/3.1-70b and on long-context benchmarks like C4 or PG-19; the paper predicts no change for structurally similar models, but this has not been measured.
  • The AP is only latency-competitive at sequence lengths of 1024 or more, so a practical deployment would need a dispatcher that offloads Softmax to the AP only for long-context inference.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SoftmAP, a software-hardware co-design for integer-only low-precision Softmax on associative processors (APs). The method uses a second-order polynomial approximation of the exponential with Barrett reduction, maps the computation to a 2D-AP dataflow, and evaluates perplexity on Llama2-7b/13b/70b over WikiText-2 across precision combinations (vcorr, M, N). The paper reports up to 1300x energy and 12.58x latency reductions versus A100 and RTX3090 GPUs, and up to ~3 orders of magnitude EDP improvement, based on a Python AP simulator.

Significance. If substantiated, the work would be significant: it presents the first integer-only low-precision Softmax tailored to associative processors for LLMs, with a detailed hardware mapping and a precision sensitivity analysis. The idea of accelerating non-linear operators on APs is timely and the approximation is grounded in prior I-BERT work. However, the quantitative headline claims rest entirely on an unvalidated simulator and a non-technology-neutral comparison, and the accuracy claim is weakened by post-hoc precision selection and a non-trivial perplexity gap at the chosen configuration. The contribution is promising but not yet convincing as presented.

major comments (3)
  1. [Section V-B, Table II, Figs. 6-8] The advertised energy and latency reductions (up to 1300x energy, 12.58x latency, and EDP ratios in Table V) are computed with a Python AP simulator whose per-operation cycle and energy formulas are taken from prior work [34] without validation against fabricated hardware, RTL simulation, or an independent cycle-accurate model, and no sensitivity analysis is reported. Since these numbers constitute the central quantitative claim of the paper, the claim is not yet supported; please provide at least an independent cross-check or a documented sensitivity study of the Table II formulas.
  2. [Section V-A, Tables III and IV] The 'best precision combination' (vcorr=M, M=6, N=16) and the clipping threshold TC are selected using the same WikiText-2 perplexity numbers that are later used to support the 'without compromising performance' claim. This is a post-hoc selection on the evaluation benchmark. Moreover, the selected configuration shows perplexity 5.92 vs 5.47 for Llama2-7b and 5.35 vs 4.88 for Llama2-13b (8-10% degradation), while M=8 achieves 5.51 and 4.93, respectively, but is not used in the hardware evaluation. The abstract's 'without compromising performance' is therefore overstated.
  3. [Section V-B, Figs. 6-7] The GPU comparison is not process-node-neutral: the AP is modeled at 16nm while the A100 is 7nm and the RTX3090 is 8nm, and no voltage/technology scaling is applied. Because energy and delay depend strongly on the technology node, part of the claimed AP advantage may be an artifact of comparing an older-node model against modern GPUs. Please normalize to a common node or report a technology-scaled comparison.
minor comments (5)
  1. [Section V-B, after Table V] The sentence 'The ratios are 4× higher with A100 compared to RTX3090' contradicts Table V, which shows the RTX3090 EDP ratios are higher; it should read 'with RTX3090 compared to A100'.
  2. [Figures 4 and 5] Many annotations in Figures 4 and 5 are garbled (e.g., 'BKLLM($$%&'!(∗F/2#))', '()++)'), making the dataflow difficult to follow, especially for the shift and Barrett-reduction steps.
  3. [Section IV, AP simulator] The AP simulator description is a single sentence; please provide the energy model equations (or identify the specific equations in [34]) so that the evaluation is reproducible.
  4. [Section III, Algorithm 1] In line 11, the shift amount '⌊−vstable/vln2⌋' is non-negative because vstable is non-positive and vln2 is positive; stating this explicitly would help readers verify the integer-only property.
  5. [Section III and references] The text consistently spells 'Barret reduction' and 'Barret' (Algorithm 1, Section III); the standard spelling is 'Barrett reduction' (as in reference [27]).

Circularity Check

2 steps flagged · score 4.0 of 10

Accuracy claim is tuned on the same WikiText-2 benchmark used for evaluation, and the headline efficiency numbers depend on a self-cited AP simulator rather than independent hardware.

  1. fitted input called prediction [Section IV (Experimental Setup); Section V-A (Precision Sensitivity Analysis)]
    "By using the WikiText-2 dataset as a calibration dataset, we analyze the input range of Softmax and manually select TC = −7 with M ∈ {6, 8} and TC = −4 with M = 4 as global clipping parameters for the model. ... By observing the perplexity values, we choose vcorr = M, M = 6, and N = 16 as the 'best precision combination' that yields the lowest perplexity with the lowest bitwidths all three Llama models."

    The same WikiText-2 benchmark is used both as the calibration set for choosing the clipping threshold TC and the precision combination (M, N, vcorr) and as the evaluation set for the perplexity tables that support the claim 'without compromising performance.' The selected precision combination is chosen by inspecting those very perplexity tables, so the subsequent perplexity values are in-sample results of the selection process, not independent predictions. The conclusion that the chosen configuration does not sacrifice perplexity is therefore restating the selection criterion rather than validating it on unseen data.

  2. self citation load bearing [Section IV (Experimental Setup); Section V-B (Hardware Evaluation); Table II]
    "Our simulator models the SRAM-based AP assuming a 16nm technology and relies on the formulations in Table II to model the energy and latency of performing elementary operations (addition, multiplication, etc.) similar to [34]."

    The headline efficiency numbers in Section V-B (up to 1300x energy and 12.58x latency reduction, and the EDP ratios in Table V) are not measured on hardware; they are produced by a Python AP simulator whose per-operation runtime and energy formulas in Table II are 'similar to [34]', a prior arXiv paper by the same authors (Rakka, Eltawil, Fouda, Kurdahi). No fabricated chip, RTL model, or independent cross-check of these formulas is provided for the Softmax dataflow in Fig. 5. Thus the central quantitative claim reduces to a self-citation: the simulator's assumptions from [34] fully determine the AP-side energy and latency, so the reported AP advantage is an arithmetic consequence of those borrowed formulas rather than an externally validated result.

full rationale

The integer-only Softmax approximation (Algorithm 1) is explicitly based on the external I-BERT work [28], and the perplexity numbers in Tables III-IV are genuine measurements on Llama2 models, so the algorithmic core is not circular. However, two load-bearing steps tie the paper's claims back to their own inputs. First, the clipping threshold TC and the precision combination are calibrated on WikiText-2 and then the same WikiText-2 perplexity is used to support the 'without compromising performance' claim; this is in-sample selection presented as validation. Second, the headline AP energy/latency/EDP numbers are generated entirely by a simulator whose Table II formulas are taken from the authors' prior work [34], with no independent hardware or RTL validation, so the efficiency claim reduces to a self-citation chain. These issues give the paper partial circularity, but the approximation itself and the LLM accuracy measurements are independent content, so the score is 4, not 6-10. The unnormalized process-node comparison (16nm AP vs 7nm A100/8nm RTX3090) is a correctness risk but not itself a circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central contribution rests on several fitted or assumed components: the clipping threshold and precision combination are tuned to WikiText-2, and the hardware evaluation depends on an unvalidated simulator from the authors' prior work. There are no newly-invented physical entities.

free parameters (3)
  • Clipping threshold TC = -7 (for M=6,8); -4 (for M=4)
    Manually selected using WikiText-2 as a calibration dataset; directly affects quantization error and perplexity, and is the paper's own tuning parameter.
  • Precision combination (M, N, vcorr) = M=6, N=16, vcorr=M
    Chosen after evaluating perplexity on WikiText-2; it is a design decision fitted to the benchmark used to claim 'no performance compromise.'
  • Quantization scale S = not specified
    Computed offline from the input data range; the paper does not describe how it is chosen or its sensitivity, so it acts as an unexamined tunable parameter.
assumptions (5)
  • domain assumption The I-BERT second-order polynomial accurately approximates the exponential on [-ln2, 0]
    The approximation in Algorithm 1 is inherited from [28]; the paper does not re-derive or validate the polynomial beyond the perplexity results.
  • domain assumption Softmax inputs after max subtraction lie within the clipping range [TC, 0] with TC=-7
    The paper manually selects TC based on WikiText-2 and assumes it generalizes across sequence lengths, batch sizes, and other datasets.
  • domain assumption The AP runtime and energy formulas in Table II are valid for a real 16nm SRAM-based associative processor
    The simulator is 'similar to [34]' and the formulas are not validated against fabricated hardware or a cycle-accurate model.
  • domain assumption Comparing a 16nm AP against 7nm A100 and 8nm RTX3090 without process-node normalization is meaningful
    The energy and latency comparisons in Figures 6-8 are presented as raw ratios despite the different technology nodes.
  • domain assumption Llama3 and Llama3.1 behave identically to Llama2 because they are structurally similar
    Stated in Section V-B as an extrapolation without experiments: 'these results won't change for Llama3/Llama3.1 with similar Llama2 parameters.'

how reviews work

0 comments
Cite this review

Pith. "Pith review of SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors." pith.science (2026). https://pith.science/paper/U2FRI7X7

@misc{pith2026241117847,
  author       = {Pith},
  title        = {Pith review of: SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2FRI7X7}},
  note         = {Machine review of arXiv:2411.17847}
}
read the original abstract

Recent research efforts focus on reducing the computational and memory overheads of Large Language Models (LLMs) to make them feasible on resource-constrained devices. Despite advancements in compression techniques, non-linear operators like Softmax and Layernorm remain bottlenecks due to their sensitivity to quantization. We propose SoftmAP, a software-hardware co-design methodology that implements an integer-only low-precision Softmax using In-Memory Compute (IMC) hardware. Our method achieves up to three orders of magnitude improvement in the energy-delay product compared to A100 and RTX3090 GPUs, making LLMs more deployable without compromising performance.

Figures

Figures reproduced from arXiv: 2411.17847 by the authors.

Figure 1
Figure 1. Softmax runtime proportion for Llama2-7b on A100 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of transformer block in Llama2 model. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 5
Figure 5. AP data flow of the approximate Softmax. [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Approximate Softmax mapping on one AP inside one head. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: Our simulator models the SRAM-based AP assuming [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: AP normalized energy for (a) Llama2-7b, (b) Llama2-13b, and (c) Llama2-70b. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: AP normalized latency for (a) Llama2-7b, (b) Llama2-13b, and (c) Llama2-70b. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: AP normalized energy-delay product vs sequence [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 15 canonical work pages

  1. [34]

    Bf-imna: A bit fluid in-memory neural architecture for neural network acceleration,

    M. Rakka, R. Karami, A. M. Eltawil, M. E. Fouda, and F. Kurdahi, “Bf-imna: A bit fluid in-memory neural architecture for neural network acceleration,” arXiv preprint arXiv:2411.01417 , 2024

    arXiv 2024

  2. [1]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

    arXiv 2023

  3. [2]

    Sparks of artificial general intelligence: Early experiments with gpt-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Ka- mar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023

    arXiv 2023

  4. [3]

    Parameter, compute and data trends in machine learning,

    E. AI, “Parameter, compute and data trends in machine learning,” 2024, accessed: 2024-05-03. [Online]. Available: https://epochai.org/ data/epochdb/visualization

    work page 2024

  5. [4]

    A survey of resource-efficient llm and multimodal foundation models,

    M. Xu, W. Yin, D. Cai, R. Yi, D. Xu, Q. Wang, B. Wu, Y . Zhao, C. Yang, S. Wang et al. , “A survey of resource-efficient llm and multimodal foundation models,” arXiv preprint arXiv:2401.08092 , 2024

    arXiv 2024

  6. [5]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

    arXiv 2001

  7. [6]

    Efficiently Distilling LLMs for Edge Applications

    A. Kundu, F. Lim, A. Chew, L. Wynter, P. Chong, and R. D. Lee, “Efficiently distilling llms for edge applications,” arXiv preprint arXiv:2404.01353, 2024

    work page Pith review arXiv 2024

  8. [7]

    Large language model inference acceleration: A comprehensive hardware perspective,

    J. Li, J. Xu, S. Huang, Y . Chen, W. Li, J. Liu, Y . Lian, J. Pan, L. Ding, H. Zhou et al., “Large language model inference acceleration: A comprehensive hardware perspective,” arXiv preprint arXiv:2410.04466, 2024

    arXiv 2024

Show all 40 references
  1. [8]

    Model compression and efficient inference for large language models: A survey,

    W. Wang, W. Chen, Y . Luo, Y . Long, Z. Lin, L. Zhang, B. Lin, D. Cai, and X. He, “Model compression and efficient inference for large language models: A survey,” arXiv preprint arXiv:2402.09748 , 2024

    2024 arXiv

  2. [9]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 38 087–38 099

    2023

  3. [10]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  4. [11]

    Zeroquant: Efficient and affordable post-training quantization for large- scale transformers,

    Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He, “Zeroquant: Efficient and affordable post-training quantization for large- scale transformers,” Advances in Neural Information Processing Sys- tems, vol. 35, pp. 27 168–27 183, 2022

    2022

  5. [12]

    Awq: Activation-aware weight quantization for llm compression and accel- eration,

    J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “Awq: Activation-aware weight quantization for llm compression and accel- eration,” arXiv preprint arXiv:2306.00978 , 2023

    2023 arXiv

  6. [13]

    Fast and efficient 2-bit llm inference on gpu: 2/4/16-bit in a weight matrix with asynchronous dequantization,

    J. Li, J. Xu, S. Li, S. Huang, J. Liu, Y . Lian, and G. Dai, “Fast and efficient 2-bit llm inference on gpu: 2/4/16-bit in a weight matrix with asynchronous dequantization,” 2024. [Online]. Available: https://arxiv.org/abs/2311.16442

    2024 arXiv

  7. [14]

    Softmax bias correction for quantized generative models,

    N. P. Pandey, M. Fournarakis, C. Patel, and M. Nagel, “Softmax bias correction for quantized generative models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 1453–1458

    2023

  8. [15]

    Softermax: Hardware/software co-design of an efficient softmax for transformers,

    J. R. Stevens, R. Venkatesan, S. Dai, B. Khailany, and A. Raghunathan, “Softermax: Hardware/software co-design of an efficient softmax for transformers,” in 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 2021, pp. 469–474

    2021

  9. [16]

    Consmax: Hardware-friendly alternative softmax with learnable parameters,

    S. Liu, G. Tao, Y . Zou, D. Chow, Z. Fan, K. Lei, B. Pan, D. Sylvester, G. Kielian, and M. Saligane, “Consmax: Hardware-friendly alternative softmax with learnable parameters,” arXiv preprint arXiv:2402.10930 , 2024

    2024 arXiv

  10. [17]

    Nongemm bench: Understanding the performance horizon of the latest ml workloads with nongemm workloads,

    R. Karami, H. Kota, S.-C. Kao, and H. Kwon, “Nongemm bench: Understanding the performance horizon of the latest ml workloads with nongemm workloads,” arXiv preprint arXiv:2404.11788 , 2024

    2024 arXiv

  11. [18]

    Associative processing and processors,

    A. Krikelis and C. C. Weems, “Associative processing and processors,” Computer, vol. 27, no. 11, pp. 12–17, 1994

    1994

  12. [19]

    In-memory associative processors: Tutorial, potential, and challenges,

    M. E. Fouda, H. E. Yantır, A. M. Eltawil, and F. Kurdahi, “In-memory associative processors: Tutorial, potential, and challenges,” IEEE Trans- actions on Circuits and Systems II: Express Briefs , vol. 69, no. 6, pp. 2641–2647, 2022

    2022

  13. [20]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  14. [21]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022

    2022 arXiv

  15. [22]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  16. [23]

    Palm: Scal- ing language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., “Palm: Scal- ing language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  17. [24]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,

    M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530 , 2024

    2024 arXiv

  18. [25]

    H. E. Yantir, Efficient acceleration of computation using associative in- memory processing. University of California, Irvine, 2018

    2018

  19. [26]

    A two-dimensional as- sociative processor,

    H. E. Yantır, A. M. Eltawil, and F. J. Kurdahi, “A two-dimensional as- sociative processor,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 26, no. 9, pp. 1659–1670, 2018

    2018

  20. [27]

    Implementing the rivest shamir and adleman public key en- cryption algorithm on a standard digital signal processor,

    P. Barrett, “Implementing the rivest shamir and adleman public key en- cryption algorithm on a standard digital signal processor,” in Conference on the Theory and Application of Cryptographic Techniques . Springer, 1986, pp. 311–323

    1986

  21. [28]

    I-bert: Integer-only bert quantization,

    S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, “I-bert: Integer-only bert quantization,” in International conference on machine learning. PMLR, 2021, pp. 5506–5518

    2021

  22. [29]

    Online normalizer calculation for softmax,

    M. Milakov and N. Gimelshein, “Online normalizer calculation for softmax,” arXiv preprint arXiv:1805.02867 , 2018

    2018 arXiv

  23. [30]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong et al. , “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  24. [31]

    PyTorch, https://pytorch.org

  25. [32]

    Transformers,

    “Transformers,” https://huggingface.co/docs/transformers/main/index

  26. [33]

    HuggingFace, 2024, https://huggingface.co/

    2024

  27. [35]

    Design space exploration for softmax implementations,

    Z. Wei, A. Arora, P. Patel, and L. John, “Design space exploration for softmax implementations,” in 2020 IEEE 31st International Conference on Application-specific Systems, Architectures and Processors (ASAP) . IEEE, 2020, pp. 45–52

    2020

  28. [36]

    Efficient softmax hardware architecture for deep neural networks,

    G. Du, C. Tian, Z. Li, D. Zhang, Y . Yin, and Y . Ouyang, “Efficient softmax hardware architecture for deep neural networks,” in Proceedings of the 2019 on Great Lakes Symposium on VLSI , 2019, pp. 75–80

    2019

  29. [37]

    Hardware implementation of softmax function based on piecewise lut,

    X. Dong, X. Zhu, and D. Ma, “Hardware implementation of softmax function based on piecewise lut,” in 2019 IEEE International Workshop on Future Computing (IWOFC . IEEE, 2019, pp. 1–3

    2019

  30. [38]

    Svd-softmax: Fast softmax approximation on large vocabulary neural networks,

    K. Shim, M. Lee, I. Choi, Y . Boo, and W. Sung, “Svd-softmax: Fast softmax approximation on large vocabulary neural networks,” Advances in neural information processing systems , vol. 30, 2017

    2017

  31. [39]

    Spatten: Efficient sparse attention architecture with cascade token and head pruning,

    H. Wang, Z. Zhang, and S. Han, “Spatten: Efficient sparse attention architecture with cascade token and head pruning,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2021, pp. 97–110

    2021

  32. [40]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,” arXiv preprint arXiv:2307.08691 , 2023

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.