Convolutional Neural Networks using Logarithmic Data Representation
read the original abstract
Recent advances in convolutional neural networks have considered model complexity and hardware efficiency to enable deployment onto embedded systems and mobile devices. For example, it is now well-known that the arithmetic operations of deep networks can be encoded down to 8-bit fixed-point without significant deterioration in performance. However, further reduction in precision down to as low as 3-bit fixed-point results in significant losses in performance. In this paper we propose a new data representation that enables state-of-the-art networks to be encoded to 3 bits with negligible loss in classification performance. To perform this, we take advantage of the fact that the weights and activations in a trained network naturally have non-uniform distributions. Using non-uniform, base-2 logarithmic representation to encode weights, communicate activations, and perform dot-products enables networks to 1) achieve higher classification accuracies than fixed-point at the same resolution and 2) eliminate bulky digital multipliers. Finally, we propose an end-to-end training procedure that uses log representation at 5-bits, which achieves higher final test accuracy than linear at 5-bits.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
FP8 Formats for Deep Learning
FP8 formats E4M3 and E5M2 match 16-bit training accuracy on CNNs, RNNs, and Transformers up to 175B parameters without hyperparameter changes.
-
Neural Network Training with Approximate Logarithmic Computations
16-bit log-based training achieves classification accuracy within approximately 1% of floating-point baselines on common datasets by replacing multiplications with approximate log-domain additions.
-
QUOTIENT: Two-Party Secure Neural Network Training and Prediction
QUOTIENT achieves 50X faster WAN training time and 6% higher absolute accuracy for secure two-party DNN training by jointly optimizing a discretized training algorithm with a tailored secure protocol.
-
PoTAcc: A Pipeline for End-to-End Acceleration of Power-of-Two Quantized DNNs
PoTAcc delivers an end-to-end pipeline and three shift-PE FPGA accelerators for PoT-quantized DNNs that deliver up to 3.6x speedup and 78% energy reduction versus CPU-only runs on PYNQ-Z2 and Kria boards.
-
On Efficient Variants of Segment Anything Model: A Survey
A survey that reviews efficient variants of the Segment Anything Model, categorizes acceleration strategies, and provides a unified hardware evaluation on benchmarks.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.