An integer-only Softmax approximation from I-BERT, mapped onto associative processors, can cut Softmax energy by up to 1300x and latency by up to 12.58x versus GPUs, but with small perplexity loss at the advertised precision.
Efficiently Distilling LLMs for Edge Applications
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.
fields
cs.AR 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors
An integer-only Softmax approximation from I-BERT, mapped onto associative processors, can cut Softmax energy by up to 1300x and latency by up to 12.58x versus GPUs, but with small perplexity loss at the advertised precision.