REVIEW 4 cited by
Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanism
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
There has been a rapid advance of custom hardware (HW) for accelerating the inference speed of deep neural networks (DNNs). Previously, the softmax layer was not a main concern of DNN accelerating HW, because its portion is relatively small in multi-layer perceptron or convolutional neural networks. However, as the attention mechanisms are widely used in various modern DNNs, a cost-efficient implementation of softmax layer is becoming very important. In this paper, we propose two methods to approximate softmax computation, which are based on the usage of LookUp Tables (LUTs). The required size of LUT is quite small (about 700 Bytes) because ranges of numerators and denominators of softmax are stable if normalization is applied to the input. We have validated the proposed technique over different AI tasks (object detection, machine translation, sentiment analysis, and semantic equivalence) and DNN models (DETR, Transformer, BERT) by a variety of benchmarks (COCO17, WMT14, WMT17, GLUE). We showed that 8-bit approximation allows to obtain acceptable accuracy loss below $1.0\%$.
Forward citations
Cited by 4 Pith papers
-
A Systolic Array Architecture for Nonlinear Activation Functions and Softmax Computation using Chebyshev Polynomials
A shared systolic array computes tanh, sigmoid, ReLU, and softmax via Chebyshev polynomial approximation, with reported error and area/power advantages over CORDIC.
-
CIMple: Standard-cell SRAM-based CIM with LUT-based split softmax for attention acceleration
CIMple delivers a 32 kb digital SRAM-based compute-in-memory accelerator for transformer self-attention that reaches 26.1 TOPS/W at 0.85 V in 28 nm with INT8 precision using dual-banked architecture and LUT-based spli...
-
Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
Presents quantization, checkpointing, softmax approximation, and logits masking to achieve substantial peak memory reductions in LoRA fine-tuning of 3B LLMs.
-
Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT
4-bit and 6-bit integer-only quantized Transformers implemented on Spartan-7 FPGA for AIoT time-series forecasting achieve 0.63% higher test loss than 8-bit baselines but up to 132x speedup and 48x lower energy.
Discussion (0). Sign in to comment.