REVIEW 6 cited by
BiBERT: Accurate Fully Binarized BERT
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but is also computation and memory expensive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parameters and bitwise operations. Unfortunately, the full binarization of BERT (i.e., 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe performance drop can be mainly attributed to the information degradation and optimization direction mismatch respectively in the forward and backward propagation, and propose BiBERT, an accurate fully binarized BERT, to eliminate the performance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction-Matching Distillation (DMD) scheme to optimize the full binarized BERT accurately. Extensive experiments show that BiBERT outperforms both the straightforward baseline and existing state-of-the-art quantized BERTs with ultra-low bit activations by convincing margins on the NLP benchmark. As the first fully binarized BERT, our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world resource-constrained scenarios.
Forward citations
Cited by 6 Pith papers
-
COBRA: Algorithm-Architecture Co-optimized Binary Transformer Accelerator for Edge Inference
COBRA co-designs a true 1-bit matrix multiplication engine and a shifted polarized softmax attention substitute, achieving 3,894 GOPS and 448.7 GOPS/W on an edge FPGA with a small GLUE accuracy drop.
-
Perforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks
Perforated Backpropagation adds error-correlating dendrite nodes to trained deep networks, improving accuracy and enabling compression in some cases.
-
MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models
MPQ-DM combines kurtosis-based intra-layer mixed-precision weight quantization with time-smoothed relation distillation to keep diffusion models accurate at 2 to 4 bit widths.
-
BiVM: Accurate Binarized Neural Network for Efficient Video Matting
BiVM is a 1-bit binarized video matting network that beats prior binarized methods on accuracy and efficiency, with 11.82 MAD on VideoMatte240K versus 28.49 for ReActNet-binarized RVM.
-
QP-SNN: Quantized and Pruned Spiking Neural Networks
A quantized and pruned spiking neural network with weight rescaling and singular-value-based pruning reaches comparable accuracy at roughly a tenth of the model size.
-
BiDense: Binarization for Dense Prediction
BiDense uses an input-adaptive binarizer and a channel-adaptive full-precision bypass to make 1-bit neural networks accurate on semantic segmentation and monocular depth estimation.
Discussion (0). Continue with ORCID to comment.