SpeLLM converts a standard token-based LLM into a character-spelling model with multiple parallel output heads, achieving competitive downstream performance with a 5.1% average decoding speedup.
Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Transformers have transformed the field of natural language processing. This performance is largely attributed to the use of stacked self-attention layers, each of which consists of matrix multiplies as well as softmax operations. As a result, unlike other neural networks, the softmax operation accounts for a significant fraction of the total run-time of Transformers. To address this, we propose Softermax, a hardware-friendly softmax design. Softermax consists of base replacement, low-precision softmax computations, and an online normalization calculation. We show Softermax results in 2.35x the energy efficiency at 0.90x the size of a comparable baseline, with negligible impact on network accuracy.
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
support 1representative citing papers
citing papers explorer
-
SpeLLM: Character-Level Multi-Head Decoding
SpeLLM converts a standard token-based LLM into a character-spelling model with multiple parallel output heads, achieving competitive downstream performance with a 5.1% average decoding speedup.