Pith. sign in

REVIEW 5 major objections 4 minor 60 references

High-Fidelity Differential-information Driven Binary Vision Transformer

T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DIDB-ViT claims a fully binarized ViT can reach 60.7% top-1 accuracy on ImageNet-1K by rewriting the attention update in differential form, adding Haar-wavelet frequency decomposition, and giving activations a per-token shift.

desk verdict Strong empirical results for binary ViTs, but the differential-attention derivation is shakier than it looks and needs clarification. read the letter →

arxiv 2507.02222 v2 pith:5WGTCN24 submitted 2025-07-03 cs.CV

Tian Gao , Zhiyuan Zhang , Kaijie Yin , Xu-Cheng Zhong , Hui Kong This is my paper · ORCID
classification cs.CV
keywords binaryvisiontransformermodelbinarizationdifferentialattentionHaarwavelethigh-frequencyinformationactivationfunctionimageclassificationsemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a vision transformer can be pushed to 1-bit weights and activations without collapsing accuracy, and that the key is to stop discarding the information that binarization loses. The proposed DIDB-ViT rewrites the attention update in differential form, keeps high-frequency detail by adding 8-neighborhood token differences, computes binary Q-K similarity from Haar-wavelet high- and low-frequency components, and gives the activation function a per-token shift. On ImageNet-1K the fully binary DeiT-Small reaches $60.7\%$ top-1 accuracy, $5.0$ points above the previous best binary ViT, and the method also improves Swin-Tiny, BinaryViT, and segmentation. If the results hold, fully binary transformers become a realistic option for edge devices that currently need full-precision or mixed-precision models.

What carries the argument

The load-bearing identity is the differential form of the attention update, $v_i^t = v_i^{t-1} + \sum_j w_j (v_j - v_i)$, which follows only from the fact that each row of the attention matrix sums to 1. The paper binarizes this form so that the update becomes a residual term $\beta v_i$ plus the sum of selected binarized tokens minus the sum of the eight neighboring binarized tokens; the minus sign is implemented as a frozen group convolution that places $-1$ around each diagonal of the attention matrix, acting as negative attention. For similarity, the non-subsampled discrete Haar wavelet, a single $2 \times 2$ binary filter, splits the input into low- and high-frequency maps, and four binary linear layers produce enhanced Q and K whose binarized dot product gives the high-fidelity similarity. The improved RPReLU adds one learnable per-token parameter $t_j$ to the existing per-channel shift parameters $m,n,k$, giving $N+3C$ new parameters per activation layer and letting the activation reshape the full distribution rather than just shift it.

What would settle it

Measure the diagonal entries of the full-precision attention matrices in a trained DeiT-Small and compare them with the learned threshold $b$ in Equation 3; if a non-negligible fraction of diagonal weights fall below $b$, the binary update that DIDB-ViT's differential attention is built on breaks for those tokens. A second check would be to run the same architecture with token orders that violate the 2D-grid assumption: the 8-neighborhood term and Haar decomposition lose their meaning, so a large accuracy drop would confirm that the image-like arrangement is load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that the damage done by binarizing attention is not random noise: it systematically deletes the differential information carried by the off-diagonal structure of the attention matrix. Because each row of attention sums to 1, the update rule can be written as $v_i^t = v_i^{t-1} + \sum_j w_j (v_j^{t-1} - v_i^{t-1})$; binarizing the weights $w_j$ then reduces every selected token's contribution to the same sign, flattening importance. DIDB-ViT recovers a version of that differential structure by subtracting the sum of the 8-neighborhood binarized tokens, which is equivalent to writing $-1$ in the attention matrix around each diagonal, and it preserves binary Q-K similarity by concatenating Haar-wavelet low- and high-frequency projections into the binary linear layers that produce Q and K. A token-wise shift in the improved RPReLU then reshapes the activation distribution channel by channel and token by token. The paper's claim is that these three changes, all cheap in binary operations, close much of the gap to full precision while keeping the original ViT architecture and 1-bit compute.

Load-bearing premise

The derivation assumes that every token's diagonal attention weight is always larger than the binarization threshold, so it always binarizes to 1; if that fails for any token, the reformulated binary attention update in Equations 8-12 no longer matches the full-precision update.

Editorial extensions

If this is right

  • If the reported numbers hold, fully binarized DeiT-Small at $60.7\%$ top-1 on ImageNet-1K becomes the strongest 1-bit ViT result, 5.0 points above the prior best.
  • The same modules transfer across DeiT-Tiny, DeiT-Small, Swin-Tiny, and BinaryViT, so the mechanism is not tied to one attention variant.
  • The activation and attention changes add negligible parameters and FLOPs, so the accuracy gain does not trade away the binary model's efficiency.
  • On ADE20K and aerial road segmentation, the BinaryViT-based DIDB encoder reaches $83.6$ mIoU on road segmentation, above the full-precision ResNet-34 encoder's $77.8$, supporting dense-prediction use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The diagonal-assumption check is the natural next experiment: if actual diagonal attention weights often fall below the threshold $b$, a per-head or learned threshold could extend the method to long-range or non-image data.
  • The 8-neighborhood differential term is essentially a fixed graph Laplacian applied to tokens; replacing it with a learned graph or k-NN adjacency could adapt DIBA to irregular token layouts such as text or point clouds, a direction the paper leaves implicit.
  • Ablating the low- and high-frequency branches of the Haar-wavelet similarity separately would show whether the gain comes from low-frequency stability or high-frequency detail; the paper reports only the combined module.
  • Because the differential update adds a negative-attention neighborhood, it resembles a high-pass filtering operation, suggesting DIDB-ViT may excel on fine-grained local tasks such as segmentation and detection more than on global-texture classification.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes DIDB-ViT, a binary vision transformer that aims to reduce the performance gap between fully binarized ViTs and their full-precision counterparts while preserving the original architecture. The method has three main components: a differential-informative binary attention (DIBA) module that adds local neighborhood differences to the binary attention update, a high-fidelity similarity calculation (HFSC) that uses non-subsampled Haar wavelet decomposition to compute attention similarities from high- and low-frequency components, and an improved RPReLU activation with token-wise learnable shifts. The authors report state-of-the-art results on CIFAR-100, Tiny-ImageNet, and ImageNet-1K for classification, and on ADE20K and an aerial road-segmentation dataset, including a 60.7% top-1 accuracy with binary DeiT-Small on ImageNet-1K, surpassing prior binary ViTs by several points.

Significance. If the reported results are reproducible and the design choices are properly validated, the paper would be a significant advance in vision-transformer binarization, showing that fully binary ViTs can approach full-precision accuracy on large-scale tasks. The method is mostly architecture-preserving, and the proposed modules are motivated by identifiable information-loss mechanisms, which is a strength. The paper also includes machine-checkable algebraic steps in the differential reformulation and a small ablation study for each module. However, the theoretical justification of the differential attention module rests on an unverified assumption about attention-matrix diagonals, and the ablations do not isolate the specific contribution of the differential term from auxiliary full-precision shortcuts and convolution operations. These issues affect the interpretability of the reported gains, so the significance is contingent on the authors addressing them.

major comments (5)
  1. [Section 3.2, Eq. (6)] The derivation of Eq. (6) assumes that the diagonal element of each full-precision attention row always binarizes to 1, based on the claim that it is necessarily larger than the threshold b in Eq. (3). This is not guaranteed: the learnable threshold b and scale a can shift the effective threshold b+a/2 above the diagonal value, and in softmax attention the diagonal is not necessarily the largest entry. The paper should either provide empirical evidence (e.g., histograms of diagonal attention values versus the learned threshold across layers and heads) or explicitly reframe Eq. (6) as an idealized motivation rather than a faithful binarization. Since this assumption underpins the narrative leading to the DIBA module, it is load-bearing for the methodological claim.
  2. [Section 3.2, Eqs. (8)-(12)] The differential update rule in Eqs. (8)-(12) is not a consequence of Eq. (6); it is a different update rule obtained by binarizing the algebraically equivalent differential form in Eq. (7). In particular, Eq. (9) has a coefficient (1-k) on v_i^{t-1}, whereas Eq. (6) has coefficient 1, so the two rules are not interchangeable. The text should clearly state that DIBA is a new heuristic design rather than a faithful binary approximation of the original attention. Furthermore, the ablation in Table 5 attributes a 5.2% gain to DIBA, but DIBA as implemented in Eq. (12) includes a full-precision scaled shortcut beta*v_i and a 3x3 binary convolution; without ablating these subcomponents, the gain cannot be attributed specifically to the differential-information term.
  3. [Section 3.2 and Table 8] The paper states in the Introduction that the method maintains 'computational efficiency' and does not increase computational complexity, but Eq. (12) adds a group convolution with frozen weights and a full-precision residual connection. Table 8 explicitly shows that the additional OPs scale with the receptive field (0.33e6 for 3x3, up to 1.81e6 for 7x7), and the full-precision shortcut involves float-point operations. The authors should quantify the actual overhead of these additions relative to the baseline model and temper the 'no additional computational complexity' claim accordingly.
  4. [Section 4.2, Table 3 and text] The text claims state-of-the-art segmentation performance, but the reported mIoU for the proposed method (18.1, if the last row of Table 3 is read as 18.1) is lower than the mIoU of BiDense (18.75) in the same table. The claim should be restricted to pixel accuracy, or the table should be corrected and the discussion revised to acknowledge that another method achieves higher mIoU. As written, the abstract's assertion of 'superior segmentation performance' is not supported by the data in Table 3.
  5. [Section 4, experimental methodology] The experiments report single-run accuracies without error bars, and no code is provided. Given that the claimed improvements over prior work are sometimes small (e.g., +0.7% on BinaryViT, +0.5% on CIFAR-100 in the two-stage setting), the lack of variance estimates makes it difficult to assess whether differences are significant. The authors should report means and standard deviations over at least three random seeds for the main classification and segmentation tables, and ideally release code to support reproducibility.
minor comments (4)
  1. [Section 3.1, Eq. (2)] Equation (2) contains a typo: 'F orward' should be 'Forward', and the piecewise conditions in the backward pass are not aligned with the forward definition; please format the cases clearly.
  2. [Section 3.2, notation] The sets Phi and Psi are both introduced without clear, separate definitions; Phi is defined for the 8-neighborhood, but Psi is used in Eq. (11) as a 3x3 receptive field. Please define Psi explicitly and ensure the notation is consistent throughout the section.
  3. [Section 4.1, Table 2] Table 2 appears to have formatting problems: several rows list only partial entries (e.g., methods with a single number such as '28.7' without a Top-1 value), and the column alignment is unclear. Please provide a complete table with all columns (size, OPs, Top-1) filled for every method.
  4. [Section 3.3, Eq. (14)] The symbol X in Eq. (14) is used as the input to the attention module, but X is not defined in the main text before that point; please state explicitly that X denotes the token sequence input to the attention module in the current layer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: DIBA, HFSC, and IRPReLU are new components evaluated on external benchmarks; self-citations are not load-bearing.

full rationale

The claimed derivation chain is self-contained. The differential-informative attention update is an algebraic reformulation of the softmax-weighted token update (Eq. 5 through Eqs. 8-11), with new learnable components introduced and then evaluated on ImageNet-1K, CIFAR-100, Tiny-ImageNet, and segmentation benchmarks; no parameter is fitted to a target and then reported as a prediction. HFSC is a defined construction (Eqs. 13-14) rather than a derived prediction, and IRPReLU is an explicit activation modification with a formal observation in the supplement. The few self-citations (GSB [7] and the RS-LVF dataset attribution [46]) are used for training-strategy comparison and dataset credit, not as the sole justification of the central claim. The concern in Section 3.2 that the diagonal attention element always binarizes to 1 is an unverified premise and a correctness/robustness risk (and Eq. 9's coefficient 1-k differs from Eq. 6's coefficient 1), but this is an internal consistency issue, not a circular reduction: the paper does not define the diagonal binary weight as 1 by construction nor fit it from the reported accuracies. The latency limitation in the supplement is explicitly acknowledged. Overall, no load-bearing step reduces to its own input.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central claim relies on several learnable parameters (scales, shifts, thresholds) and hyperparameters (lambda, receptive field) fitted to data or tuned by ablation. The derivation depends on structural assumptions about attention weights and token arrangement. No entirely new physical or conceptual entities are introduced; the method consists of architectural modifications within the existing binary neural network paradigm.

free parameters (7)
  • beta (learnable scale for self-token residual) = learned during training, initial value 10-k
    Appears in Equations 11 and 12 as the coefficient of the token's own value. Its initial value depends on k, the number of binarized attention ones, and it is trained.
  • gamma (scale factor for neighborhood term) = learned during training
    Scales the negative neighborhood term in Equation 12; learned during optimization.
  • alpha (scale factor for binary attention output) = learned during training
    Standard channel-wise scaling factor in binary attention, carried over from prior work and trained.
  • lambda (distillation loss weight) = 0.9
    Chosen via ablation in Table 7; balances cross-entropy and distillation loss in Equation 17.
  • t_j (token-wise shift in improved RPReLU) = learned during training
    Added in Equation 16 to allow different per-token activation shifts; N parameters per activation layer.
  • a, b (binarization threshold parameters) = learned during training
    Appear in Equations 2 and 3 for binarizing activations and attention values; standard learnable parameters in ReActNet-style binarization.
  • Receptive field theta (size of neighborhood) = 3x3
    Chosen via ablation in Table 8; balances performance and computational cost of the differential term.
assumptions (5)
  • standard math Softmax attention rows sum to 1.
    Used in the reformulation from Equation 5 to Equation 7, where the diagonal weight is expressed as 1 minus the sum of off-diagonal weights.
  • domain assumption The diagonal element of the attention matrix is always binarized to 1.
    Stated after Equation 6: the diagonal attention weight is assumed to exceed the threshold b. If violated, the differential attention derivation breaks.
  • domain assumption Self-attention behaves as a low-pass filter, motivating high-frequency retention.
    Cites references [1,35] to justify adding local differential information to preserve high-frequency components.
  • domain assumption Tokens are arranged in an image-like 2D grid.
    Required for defining the 8-neighborhood in Equation 10 and for the non-subsampled Haar wavelet decomposition in Section 3.3.
  • domain assumption Frequency decomposition via Haar wavelet preserves or enhances similarity fidelity.
    The paper asserts that integrating similarities across frequency components yields a more faithful attention matrix, but this is an empirical design choice without a formal proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of High-Fidelity Differential-information Driven Binary Vision Transformer." pith.science (2026). https://pith.science/paper/5WGTCN24

@misc{pith2026250702222,
  author       = {Pith},
  title        = {Pith review of: High-Fidelity Differential-information Driven Binary Vision Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5WGTCN24}},
  note         = {Machine review of arXiv:2507.02222}
}
read the original abstract

The binarization of vision transformers (ViTs) offers a promising approach to addressing the trade-off between high computational/storage demands and the constraints of edge-device deployment. However, existing binary ViT methods often suffer from severe performance degradation or rely heavily on full-precision modules. To address these issues, we propose DIDB-ViT, a novel binary ViT that is highly informative while maintaining the original ViT architecture and computational efficiency. Specifically, we design an informative attention module incorporating differential information to mitigate information loss caused by binarization and enhance high-frequency retention. To preserve the fidelity of the similarity calculations between binary Q and K tensors, we apply frequency decomposition using the discrete Haar wavelet and integrate similarities across different frequencies. Additionally, we introduce an improved RPReLU activation function to restructure the activation distribution, expanding the model's representational capacity. Experimental results demonstrate that our DIDB-ViT significantly outperforms state-of-the-art network quantization methods in multiple ViT architectures, achieving superior image classification and segmentation performance.

Figures

Figures reproduced from arXiv: 2507.02222 by the authors.

Figure 1
Figure 1. The binary attention module based on differential in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The figure about mapping the eight neighborhoods of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The schematic diagram of modifying similarity map [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 1
Figure 1. Figure 1: PReLU and RPReLU versus our improved RPReLU. For an input vector [PITH_FULL_IMAGE:figures/full_fig_p013_1.png]
Figure 2
Figure 2. Figure 2: The multiplication between binary vectors can be implemented by the Xnor and popcount. The result is [PITH_FULL_IMAGE:figures/full_fig_p013_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 56 canonical work pages

  1. [1]

    Improving vision transformers by revisit- ing high-frequency components

    Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transformers by revisit- ing high-frequency components. In European Conference on Computer Vision, pages 1–18, 2022. 4

    work page 2022

  2. [2]

    Binarized spectral compressive imaging

    Yuanhao Cai, Yuxin Zheng, Jing Lin, Xin Yuan, Yulun Zhang, and Haoqian Wang. Binarized spectral compressive imaging. Advances in Neural Information Processing Sys- tems, 36, 2024. 7

    work page 2024

  3. [3]

    ” bnn-bn=?”: Train- ing binary neural networks without batch normalization

    Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, and Zhangyang Wang. ” bnn-bn=?”: Train- ing binary neural networks without batch normalization. In Proceedings of the IEEE/CVF conference on Computer Vi- sion and Pattern Recognition, pages 4619–4629, 2021. 1

    work page 2021

  4. [4]

    Binaryconnect: Training deep neural networks with binary weights during propagations

    Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems, volume 28, 2015. 2

    work page 2015

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 248–255,

    work page

  6. [6]

    Network pruning via performance maximization

    Shangqian Gao, Feihu Huang, Weidong Cai, and Heng Huang. Network pruning via performance maximization. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 9270–9280, 2021. 1

    work page 2021

  7. [7]

    Gsb: Group superposition binarization for vision transformer with limited training samples

    Tian Gao, Cheng-Zhong Xu, Le Zhang, and Hui Kong. Gsb: Group superposition binarization for vision transformer with limited training samples. Neural Networks, page 106133,

    work page

  8. [8]

    Towards optimization-friendly binary neu- ral network

    Nianhui Guo, Joseph Bethge, Hong Guo, Christoph Meinel, and Haojin Yang. Towards optimization-friendly binary neu- ral network. Transactions on Machine Learning Research ,

    work page

Show all 60 references
  1. [9]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 6, 7

    2016

  2. [10]

    Bivit: Extremely com- pressed binary vision transformers

    Yefei He, Zhenyu Lou, Luoming Zhang, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Bivit: Extremely com- pressed binary vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5651–5663, 2023. 1, 2, 6, 7

    2023

  3. [11]

    Bolt, 2020

    HuaWei-Noah. Bolt, 2020. 13

    2020

  4. [12]

    Binarized neural networks

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. Ad- vances in neural information processing systems , 29, 2016. 1, 2, 7

    2016

  5. [13]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009. 6

    2009

  6. [14]

    Binaryvit: Pushing binary vision transformers towards convolutional models

    Phuoc-Hoan Charles Le and Xinlin Li. Binaryvit: Pushing binary vision transformers towards convolutional models. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition workshop , pages 4664–4673,

  7. [15]

    Bfd: Binarized frequency-enhanced distillation for vision transformer

    Hanglin Li, Peng Yin, Xiaosu Zhu, Lianli Gao, and Jingkuan Song. Bfd: Binarized frequency-enhanced distillation for vision transformer. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2024. 2, 6, 7

    2024

  8. [16]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In European conference on computer vision , pages 280–296. Springer, 2022. 1

    2022

  9. [17]

    Bi-vit: Pushing the limit of vision transformer quantization

    Yanjing Li, Sheng Xu, Mingbao Lin, Xianbin Cao, Chuan- jian Liu, Xiao Sun, and Baochang Zhang. Bi-vit: Pushing the limit of vision transformer quantization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3243–3251, 2024. 1, 2, 3, 7

    2024

  10. [18]

    Q-vit: Accurate and fully quantized low-bit vision transformer

    Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. In Advances in Neural Informa- tion Processing Systems, 2022. 1, 6

    2022

  11. [19]

    Siman: Sign-to- magnitude network binarization

    Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Fei Chao, Chia-Wen Lin, and Ling Shao. Siman: Sign-to- magnitude network binarization. IEEE Transactions on Pat- tern Analysis and Machine Intelligence , 45(5):6277–6288,

  12. [20]

    Ro- tated binary neural network

    Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wen Lin. Ro- tated binary neural network. In Advances in Neural Infor- mation Processing Systems , volume 33, pages 7474–7485,

  13. [21]

    Towards accurate binary convolutional neural network

    Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. Advances in neural in- formation processing systems, 30, 2017. 2

    2017

  14. [22]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021. 1, 7

    2021

  15. [23]

    Bit: Robustly binarized multi-distilled trans- former

    Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, and Yashar Mehdad. Bit: Robustly binarized multi-distilled trans- former. In Advances in Neural Information Processing Sys- tems, 2022. 6, 7

    2022

  16. [24]

    Reactnet: Towards precise binary neural net- work with generalized activation functions

    Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural net- work with generalized activation functions. In Proceedings of the European Conference on Computer Vision, pages 143–

  17. [25]

    Bi-real net: Enhancing the perfor- mance of 1-bit cnns with improved representational capabil- ity and advanced training algorithm

    Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the perfor- mance of 1-bit cnns with improved representational capabil- ity and advanced training algorithm. In Proceedings of the European Conference on Computer Vision, pages 722–737,

  18. [26]

    Image matching from handcrafted to deep fea- tures: A survey

    Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep fea- tures: A survey. International Journal of Computer Vision, 129(1):23–79, 2021. 5

    2021

  19. [27]

    Low rank approximation: algorithms, im- plementation, applications, volume 906

    Ivan Markovsky. Low rank approximation: algorithms, im- plementation, applications, volume 906. Springer, 2012. 1

    2012

  20. [28]

    Rectified linear units im- prove restricted boltzmann machines

    Vinod Nair and Geoffrey E Hinton. Rectified linear units im- prove restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML- 10), pages 807–814, 2010. 5

    2010

  21. [29]

    Bibert: Accurate fully binarized bert

    Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. In International Con- ference on Learning Representations, 2022. 7

    2022

  22. [30]

    Binary neural networks: A survey

    Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: A survey. Pattern Recognition, 105:107281, 2020. 1

    2020

  23. [31]

    Forward and backward information retention for accurate binary neural networks

    Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 2250– 2...

    2020

  24. [32]

    Bibench: Benchmarking and analyzing network binarization

    Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Zhon- gang Cai, Ziwei Liu, Fisher Yu, and Xianglong Liu. Bibench: Benchmarking and analyzing network binarization. arXiv preprint arXiv:2301.11233, 2023. 1

    2023 arXiv

  25. [33]

    Xnor-net: Imagenet classification using bi- nary convolutional neural networks

    Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using bi- nary convolutional neural networks. In Proceedings of the European Conference on Computer Vision, pages 525–542. Springer, 2016. 1, 2, 6

    2016

  26. [34]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International of Medical image computing and computer- assisted intervention, pages 234–241, 2015. 8

    2015

  27. [35]

    An attentive inductive bias for sequential recom- mendation beyond the self-attention

    Yehjin Shin, Jeongwhan Choi, Hyowon Wi, and Noseong Park. An attentive inductive bias for sequential recom- mendation beyond the self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8984– 8992, 2024. 4

    2024

  28. [36]

    Segmenter: Transformer for semantic segmenta- tion

    Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmenta- tion. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 7262–7272, 2021. 1

    2021

  29. [37]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International Conference on Machine Learning , pages 10347–10357. PMLR, 2021. 1, 6, 7

    2021

  30. [38]

    Adabin: Improving binary neural networks with adaptive bi- nary sets

    Zhijun Tu, Xinghao Chen, Pengju Ren, and Yunhe Wang. Adabin: Improving binary neural networks with adaptive bi- nary sets. In European conference on computer vision, pages 379–395. Springer, 2022. 7

    2022

  31. [39]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , p...

    2021

  32. [40]

    Bvt-ima: Binary vision transformer with information-modified attention

    Zhenyu Wang, Hao Luo, Xuemei Xie, Fan Wang, and Guangming Shi. Bvt-ima: Binary vision transformer with information-modified attention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 15761–15769, 2024. 1, 2, 6, 7

    2024

  33. [41]

    Tiny imagenet challenge

    Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical Report, 2017. 6

    2017

  34. [42]

    Resilient binary neural network

    Sheng Xu, Yanjing Li, Teli Ma, Mingbao Lin, Hao Dong, Baochang Zhang, Peng Gao, and Jinhu Lu. Resilient binary neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10620–10628, 2023. 2

    2023

  35. [43]

    Recurrent bilinear optimization for binary neural networks

    Sheng Xu, Yanjing Li, Tiancheng Wang, Teli Ma, Baochang Zhang, Peng Gao, Yu Qiao, Jinhu L ¨u, and Guodong Guo. Recurrent bilinear optimization for binary neural networks. In Proceedings of the European Conference on Computer Vi- sion, pages 19–35. Springer, 2022. 7

    2022

  36. [44]

    Learning frequency domain approxima- tion for binary neural networks

    Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approxima- tion for binary neural networks. Advances in Neural Infor- mation Processing Systems, 34:25553–25565, 2021. 1, 2

    2021

  37. [45]

    Lavt: Language-aware vision transformer for referring image segmentation

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Heng- shuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155–18165, 2022. 1

    2022

  38. [46]

    Pathfinder for low- altitude aircraft with binary neural network

    Kaijie Yin, Tian Gao, and Hui Kong. Pathfinder for low- altitude aircraft with binary neural network. arXiv preprint arXiv:2409.08824, 2024. 7

    2024 arXiv

  39. [47]

    Si-bivit: Binarizing vision transformers with spatial interaction

    Peng Yin, Xiaosu Zhu, Jingkuan Song, Lianli Gao, and Heng Tao Shen. Si-bivit: Binarizing vision transformers with spatial interaction. In Proceedings of the 32nd ACM Interna- tional Conference on Multimedia , pages 8169–8178, 2024. 1, 2, 6, 7

    2024

  40. [48]

    Bidense: Binariza- tion for dense prediction

    Rui Yin, Haotong Qin, Yulun Zhang, Wenbo Li, Yong Guo, Jianjun Zhu, Cheng Wang, and Biao Jia. Bidense: Binariza- tion for dense prediction. arXiv preprint arXiv:2411.10346,

    arXiv

  41. [49]

    Pokebnn: A binary pursuit of lightweight accuracy

    Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475–12485, 2022. 1, 2

    2022

  42. [50]

    Vit-yolo: Transformer-based yolo for object detection

    Zixiao Zhang, Xiaoqiang Lu, Guojin Cao, Yuting Yang, Licheng Jiao, and Fang Liu. Vit-yolo: Transformer-based yolo for object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2799– 2808, 2021. 1

    2021

  43. [51]

    Semantic under- standing of scenes through the ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302–321, 2019. 7

    2019

  44. [52]

    Towards effective low-bitwidth convolutional neural networks

    Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low-bitwidth convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7920–7928,

  45. [55]

    Let the element of O be Ok,l and the element of X be Xm,n, we have ∂Ok,l ∂Xm,n = ∂V + ∂Q + ∂K

    The Gradient Backpropagation of High- Fidelity Similarity Calculation For instance, consider the gradient between the outputO and the input X in the attention module. Let the element of O be Ok,l and the element of X be Xm,n, we have ∂Ok,l ∂Xm,n = ∂V + ∂Q + ∂K. (1) ∂V = tX i=1...

  46. [56]

    1, we take an input vector X ∈ R2×2 as an example

    The Improved RPReLU To achieve shifting the overall distribution of the entire feature vector, we propose an improved RPReLU, formu- lated as, Fi,j = ( (Xi,j − mi) +ni + tj Xi,j > mi ki (Xi,j − mi) +ni + tj Xi,j < mi , (7) As shown in Fig. 1, we take an input vector X ∈ R2×2 a...

  47. [57]

    Let A ∈ Rm×n be the original matrix

    Adding a Unique Learnable Parameter to Each Ele- ment. Let A ∈ Rm×n be the original matrix. We add a unique learnable parameter matrix P ∈ Rm×n to A, where pij is an independent learning parameter and aij is the element of A. The updated matrix is, A′ = A + P (8) The mean and ...

  48. [58]

    In this situation, the learnable parameter becomes a scalar p ∈ R1×1

    Adding a Single Learnable Parameter to All Ele- ments. In this situation, the learnable parameter becomes a scalar p ∈ R1×1. The updated matrix is, A′′ = A + p · 1, (12) where 1 is a all-one matrix with the same shape as A. The mean and variance of the updated matrix A′ are, µ...

  49. [59]

    Xnor+Popcount During the inference process of the model, the primary concept of the model binarization technique involves con- verting each numerical value linked with matrix multipli- cation to either 1 or -1 and applying Xnor and popcount operations to replace the multiplica...

  50. [60]

    Then, we utilize the BOLT toolbox [11] to implement each method to the edge device based on an ARM Cortex-A76 CPU (without CUDA)

    Evaluation The ablation study about the latency To obtain a latency result comparison between our method and the other bi- nary ViT method, we first transfer the Pytorch code of each method to the ONNX version. Then, we utilize the BOLT toolbox [11] to implement each method to...

  51. [159]

    1, 2, 5, 6, 7, 8

    Springer, 2020. 1, 2, 5, 6, 7, 8

    2020

  52. [2018]

    1 High-Fidelity Differential-information Driven Binary Vision Transformer Supplementary Material

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.