Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Bayesian Optimized 1-Bit CNNs

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding two Bayesian losses to the training objective lets fully binarized CNNs reach state-of-the-art accuracy, with a 1-bit ResNet-18 hitting 59.3% top-1 on ImageNet.

desk verdict The empirical recipe works and the ImageNet results are strong, but the 'theoretical support' does not hold as written because Eq. 5 is inconsistent with Eq. 1; still, the paper deserves a referee and a reasonable chance at publication after the derivation is repaired. read the letter →

arxiv 1908.06314 v1 pith:5IE64L2G submitted 2019-08-17 cs.CV

Jiaxin Gu , Junhe Zhao , Xiaolong Jiang , Baochang Zhang , Jianzhuang Liu , Guodong Guo , Rongrong Ji This is my paper · ORCID
classification cs.CV
keywords 1-bitCNNsbinaryneuralnetworksBayesianlearningquantizationkernellossfeatureImageNetclassificationmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that extreme 1-bit CNNs can be made far more accurate by training them with two losses derived from Bayesian reasoning about quantization. The first loss, the Bayesian kernel loss, steers each full-precision kernel toward a two-mode Gaussian mixture centered on the two binarized values while also minimizing reconstruction error. The second loss, the Bayesian feature loss, pulls features from the same class together to reduce the disturbance caused by binarization. Combined with ordinary cross-entropy, these losses let a fully binarized ResNet-18 reach 59.3% top-1 and 81.6% top-5 accuracy on ImageNet, which the paper reports as the best among existing 1-bit CNNs. The claim matters because it suggests the accuracy gap of binary networks can be narrowed by reshaping distributions during training rather than by adding capacity.

What carries the argument

The carrying mechanism is the pair of Bayesian losses, Eq. 7 (kernel) and Eq. 8 (feature), added to the cross-entropy loss. The kernel loss is framed as the maximum-a-posteriori solution of a model in which the quantization error $y = w^{-1} \circ \hat{x} - x$ is Gaussian noise and the full-precision kernel $x$ has a two-mode Gaussian mixture prior; minimizing it simultaneously shrinks the reconstruction error $\|\hat{x} - w \circ x\|_2^2$ and drives the kernel distribution toward the two-mode form that binarization expects. The feature loss uses a Gaussian model of per-class features with a learned class mean, adding an intra-class compactness term with per-dimension variance weighting. Both losses are differentiable and fit into standard SGD, so the network is trained end-to-end with the same forward-backward pipeline as any CNN.

What would settle it

Re-derive the MAP objective using the paper's own error definition $y = w^{-1} \circ \hat{x} - x$ inside the Gaussian likelihood, train BONN on CIFAR-10 with WRN-22 using the corrected loss, and check whether the 92.36% accuracy survives; if it changes materially, the reported gain is not produced by the claimed Bayesian derivation.

Watch

Extended reading notes

Core claim

The central claim is that the gap between full-precision and 1-bit CNNs can be substantially closed by optimizing the distribution of full-precision kernels and features while training. Treating the quantization error as Gaussian noise and the full-precision weights as latent variables, the paper derives a MAP objective for each convolution layer: a reconstruction term that keeps the binarized output close to the full-precision output, plus a Gaussian-mixture prior term that pushes the weight distribution into two modes located at the two binary quantization values. A companion feature loss applies the same reasoning to class features, enforcing intra-class compactness around a learned class mean. The paper reports that on ImageNet with a ResNet-18 backbone, BONN reaches 59.3% top-1 accuracy, outperforming XNOR-Net (51.2%), Bi-Real Net (56.4%), and PCNN (57.3%), and on CIFAR-10 with a Wide-ResNet-22 backbone it reaches 92.36%, within 2.6 points of the full-precision model.

Load-bearing premise

The method rests on the claim that the kernel loss is the exact Bayesian MAP solution of the stated quantization model, but as written the likelihood drops the learned modulation vector from the quantization error, so that derivation does not hold as stated.

Editorial extensions

If this is right

  • A fully binarized ResNet-18 reaches 59.3% top-1 and 81.6% top-5 on ImageNet, roughly 8 points above XNOR-Net and about 3 points above the closest prior 1-bit method.
  • On CIFAR-10, a binarized WRN-22 with 64-64-128-256 channels reaches 92.36%, only 2.6 points below the full-precision WRN-22.
  • The two Bayesian losses are complementary: on ImageNet, each one alone improves accuracy over the cross-entropy-only baseline, and together they give the full gain.
  • The kernel loss visibly reshapes weight distributions into two modes during training, which is the distributional condition that makes binary quantization less destructive.
  • The method keeps 1-bit storage and XNOR-style efficient convolutions, with an 11.10x compression rate on ResNet-18 and an estimated 58x CPU speedup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the two-mode Gaussian prior is the active ingredient, the same Bayesian loss construction should transfer to ternary or 4-bit quantization by replacing the two-mode prior with a mixture matching the number of quantization levels.
  • Because the feature loss is applied only at the fully connected layer, a version that applies intra-class compactness to convolutional feature maps could test whether the benefit generalizes across layers.
  • The method's gains come on top of standard binarization, so it should compose with other 1-bit tricks, such as better gradient estimators or multi-bit activations, rather than competing with them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes BONNs (Bayesian Optimized 1-bit CNNs), a training method for binary-weight and binary-activation networks. The method adds two auxiliary losses to the standard cross-entropy loss: a "Bayesian kernel loss" that penalizes the difference between full-precision and quantized kernels and encourages the full-precision weights to form a two-mode Gaussian mixture around the two quantization values, and a "Bayesian feature loss" that encourages class features to be compact around per-class means. The authors motivate these losses through a MAP/Bayesian derivation, then evaluate the method on CIFAR-10/100 with WideResNet backbones and on ImageNet with ResNet-18, reporting state-of-the-art results among the compared 1-bit CNNs, notably 59.3% top-1 accuracy on ImageNet, and ablation experiments showing that both losses contribute to the improvement.

Significance. If the reported results hold, the empirical contribution is meaningful: the paper demonstrates that two relatively simple regularizing losses can improve 1-bit CNNs by several points on ImageNet and CIFAR, and the ablation study in Table 2 provides direct evidence that both losses are useful. The paper also gives a clear description of the loss formulas and the training procedure, and the comparisons in Table 4 cover a reasonable set of earlier binarization methods. However, the central theoretical claim that the losses are derived from a Bayesian MAP formulation is not supported as written, because the likelihood in Eq. (5) does not follow from the error definition in Eq. (1), and Eq. (3) mislabels the prior as the posterior. In addition, the claim of state-of-the-art performance on CIFAR is under-supported by Table 3, which only compares against XNOR-Net, and the paper provides no code or error bars, which limits verification of the small accuracy differences reported.

major comments (4)
  1. [§2.1, Eq. (5) with Eq. (1)] The likelihood in Eq. (5) does not follow from the model defined in Eq. (1). Eq. (1) defines the reconstruction error as y = w^{-1}∘\hat{x} - x. If the components of y are i.i.d. Gaussian with variance ν, then p(y|x) ∝ exp(-||w^{-1}∘\hat{x} - x||²/(2ν)), not exp(-||\hat{x} - w∘x||²/(2ν)) as written in Eq. (5). The expression in Eq. (5) is instead the likelihood for the different error \hat{x} - w∘x, or equivalently for w∘y up to a Jacobian factor. Consequently, Eq. (7), which is called the Bayesian kernel loss, is not the MAP solution of the model in Eq. (1), and the paper's claim that the Bayesian losses are achieved "with a theoretical support" is not supported as written. This is load-bearing because the two Bayesian losses are the paper's primary novelty. The derivation could be repaired by changing Eq. (1) to y = \hat{x} - w∘x, or by using the correct likelihood and re-deriving the resulting gradients, but the present text is internally inconsistent.
  2. [§2.1, Eq. (3)] Eq. (3) is labeled p(x|y), but the right-hand side is a Gaussian mixture in x, which is a prior distribution p(x), not a posterior conditioned on the observed error y. Indeed, Eq. (6) later explicitly defines p(x) as this kind of Gaussian mixture. Moreover, the mixture centers in Eq. (3) are set to \tilde{μ} = w^{-1}∘\hat{x}, whereas the loss in Eq. (7) uses separately learned μ₊ and μ₋ that are not connected to \tilde{μ}. The text therefore does not actually compute or use the posterior p(x|y), and the derivation chain from Eq. (2) to Eq. (7) is not established. The authors should either correct the notation to distinguish p(x) from p(x|y) and derive the correct posterior, or explicitly state that the GMM term is a regularizer rather than a posterior.
  3. [§2.1–§2.2, Eqs. (7)–(9)] The parameters called "priors" are not fixed in advance: μᵢ, σᵢ, c_m, and σ_m are all learned on the training data by optimizing the total loss L = L_S + L_B. Thus the Bayesian kernel loss and Bayesian feature loss are better described as regularizers that encourage the assumed distributional shapes (two-mode Gaussian kernels, compact per-class features) rather than as a MAP solution of a pre-specified Bayesian model. The phrase "Bayesian losses with theoretical support" overstates what the derivation provides. The empirical value of the regularizers may stand, but the paper should qualify the Bayesian interpretation accordingly, for example by presenting them as regularizers inspired by a Gaussian-mixture prior, or by specifying a proper generative model with fixed hyperparameters and explaining how the learned parameters fit into the Bayesian formalism.
  4. [Table 3 and Abstract] The abstract claims that BONNs "achieve the best classification performance compared to state-of-the-art 1-bit CNNs" on the ImageNet and CIFAR datasets, but the CIFAR comparisons in Table 3 only include XNOR-Net as a competing 1-bit method. Other 1-bit CNNs that appear in the ImageNet comparison, such as Bi-Real Net, PCNN, and ABC-Net, are not evaluated on CIFAR, so the claim of state-of-the-art performance on CIFAR is not supported by the presented evidence. The authors should either add comparisons with additional 1-bit methods on CIFAR or narrow the claim to the specific baselines that were actually compared.
minor comments (6)
  1. [§2.1, Eq. (2)] Eq. (2) writes \hat{x} = max p(x|y), but it should be \hat{x} = \arg\max_x p(x|y); the current notation is formally incorrect.
  2. [§2.3.1, Eq. (13)] The derivative of ||\hat{X}ᵢ - w∘Xᵢ||² with respect to Xᵢ is 2w∘(w∘Xᵢ - \hat{X}ᵢ), so Eq. (13) omits a factor of 2. Since λ is a free hyperparameter, this does not change the optimization, but the derivation should be consistent with the stated loss.
  3. [Introduction] The introduction contains unresolved citation placeholders, e.g., "[?]" in the sentence about high-level vision tasks and "[?, ?, ?]" in the discussion of related quantization practices; these must be completed in a journal version.
  4. [§3.2 and §3.3] The paper reports no error bars or multiple-seed results. In Table 2, the differences between the single-loss runs (58.3 vs. 58.4) and between the best run and the single-loss runs (59.3 vs. 58.4) are small, so without repeated runs it is difficult to assess whether the improvement is statistically reliable.
  5. [§2.3.3] The update rule for σ_m is mentioned but not given: the text says it is "straightforward" and "not elaborated here for brevity." For reproducibility, the update equations should be included.
  6. [§3.3] The phrase "only 2.6% left" in the discussion of the WRN22 64-64-128-256 CIFAR-10 result is ambiguous; the authors likely mean a 2.6% accuracy degradation relative to the full-precision model, and this should be stated more clearly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the BONN losses are explicit regularizers and the accuracy claims rest on external benchmarks; the Eq. 4–5 likelihood mismatch is a correctness issue, not a circular reduction.

full rationale

The paper's two Bayesian losses are constructed as MAP-style regularizers: a Gaussian-mixture prior over kernels and a Gaussian feature-compactness term, both combined with the cross-entropy loss. The accuracy results on CIFAR and ImageNet are external benchmark numbers, so the central empirical claim is not an output of the fitted parameters by construction. The ablation study in Table 2 and Figures 3–5 demonstrate that the losses move the kernel distribution toward the assumed two-mode GMM; this is a regularizer doing what it was designed to do, not a prediction derived from the loss. The paper does cite the authors' own prior work (e.g., PCNN [6] and modulated convolutional networks [20]), but those citations are used as baselines or related work, not as load-bearing justification for the BONN derivation. The one genuinely problematic passage is Eq. 5 in Section 2.1: Eq. 1 defines y = w^{-1}∘\hat{x} - x, so the likelihood should be exp(-||w^{-1}∘\hat{x} - x||^2/(2\nu)), not exp(-||\hat{x} - w\u2218x||^2/(2\nu)) as written. This makes the claimed MAP derivation internally inconsistent, but it is not circularity: the loss does not reduce to its own input by definition; it is simply not the MAP solution of the stated model. That is a correctness and reproducibility risk, not a circularity finding. Since the derivation is not self-referential in the sense of renaming a fit as a prediction, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method rests on several distributional assumptions that are built into the losses rather than verified: zero-mean Gaussian quantization error, a two-mode Gaussian mixture for kernels, Gaussian per-class features, and independence across kernel elements. The three scalar hyperparameters λ, θ, and ν are hand-set via ablations.

free parameters (3)
  • lambda = 1e-4 (ImageNet/CIFAR)
    Trade-off scalar weighting the Bayesian kernel loss; chosen via ablation on CIFAR-10 and fixed for ImageNet.
  • theta = 1e-3
    Trade-off scalar weighting the Bayesian feature loss; chosen via ablation.
  • nu = 1e-4 for WRN, 1e-3 for ResNet18
    Variance of the assumed zero-mean Gaussian quantization error; set by hand per backbone.
assumptions (5)
  • domain assumption Quantization error y in Eq. 1 is i.i.d. zero-mean Gaussian with variance ν.
    This motivates the likelihood in Eq. 5 and the MAP objective, but it is not checked against the actual error distribution and is inconsistent with the modulation vector w.
  • domain assumption Full-precision kernels follow a two-mode Gaussian mixture located at the binary values, with overlap between positive and negative parts neglected.
    This is the basis of the Bayesian kernel loss (Eqs. 6-7); the mode locations and widths are learned rather than fixed, which weakens the Bayesian interpretation.
  • domain assumption Features of each class are Gaussian-distributed around a class center c_m.
    Adopted from center loss [21]; supports the Bayesian feature loss in Eq. 8.
  • ad hoc to paper Parameters in the same kernel are independent and share a common scalar variance (σ_i)^2.
    Stated after Eq. 9 to diagonalize Ψ_i and speed up inversion; it is a computational simplification rather than an empirical fact.
  • ad hoc to paper The gradient of the non-differentiable sign function is estimated by the clip indicator function (straight-through estimator).
    Used in Eq. 12 to pass gradients through binarization; standard in the field [18] but not a mathematical derivative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bayesian Optimized 1-Bit CNNs." pith.science (2026). https://pith.science/paper/5IE64L2G

@misc{pith2026190806314,
  author       = {Pith},
  title        = {Pith review of: Bayesian Optimized 1-Bit CNNs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5IE64L2G}},
  note         = {Machine review of arXiv:1908.06314}
}
read the original abstract

Deep convolutional neural networks (DCNNs) have dominated the recent developments in computer vision through making various record-breaking models. However, it is still a great challenge to achieve powerful DCNNs in resource-limited environments, such as on embedded devices and smart phones. Researchers have realized that 1-bit CNNs can be one feasible solution to resolve the issue; however, they are baffled by the inferior performance compared to the full-precision DCNNs. In this paper, we propose a novel approach, called Bayesian optimized 1-bit CNNs (denoted as BONNs), taking the advantage of Bayesian learning, a well-established strategy for hard problems, to significantly improve the performance of extreme 1-bit CNNs. We incorporate the prior distributions of full-precision kernels and features into the Bayesian framework to construct 1-bit CNNs in an end-to-end manner, which have not been considered in any previous related methods. The Bayesian losses are achieved with a theoretical support to optimize the network simultaneously in both continuous and discrete spaces, aggregating different losses jointly to improve the model capacity. Extensive experiments on the ImageNet and CIFAR datasets show that BONNs achieve the best classification performance compared to state-of-the-art 1-bit CNNs.

Figures

Figures reproduced from arXiv: 1908.06314 by the authors.

Figure 1
Figure 1. The evolution of the prior p(x), the distribution of the observation y, and the posterior p(x|y) during learning, where x is the latent variable representing the full-precision parameters and y is the quantization error. At the beginning, the parameters x are initialized according to a single-mode Gaussian distribution. When our learning algorithm converges, the ideal case is that (i) p(y) becomes a Gaussian distrib… view at source ↗
Figure 2
Figure 2. By considering the prior distributions of the kernels and features in the Bayesian framework, we achieve two new Bayesian losses [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We demonstrate the kernel weight distribution of the [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Weight distribution of XNOR and BONN, both based [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The evolution of the binarized values, |x|s, during the training process of XNOR and BONN. They are both based on WRN22 (2nd, 3rd, 8th and 14th convolutional layers) and the curves are not sharing the same y-axis. The binarized values of XNOR Net tend to converge to sm…
Figure 6
Figure 6. Figure 6: Training and Test accuracies on ImageNet when [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 21 canonical work pages

  1. [1]

    Blundell, J

    C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613–1622, 2015

    2015

  2. [2]

    Courbariaux, Y

    M. Courbariaux, Y . Bengio, and J.-P. David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems, pages 3123–3131, 2015

    work page 2015

  3. [3]

    Courbariaux, I

    M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y . Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-

    work page

  4. [4]

    arXiv preprint arXiv:1602.02830, 2016

    arXiv 2016

  5. [5]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    work page 2009

  6. [6]

    C. Dong, C. C. Loy, K. He, and X. Tang. Learning a deep convolutional network for image super-resolution. In Proceedings of the European Conference on Computer Vision, pages 184–199. Springer, 2014

    work page 2014

  7. [7]

    J. Gu, C. Li, B. Zhang, J. Han, X. Cao, J. Liu, and D. Doermann. Projection convolutional neural networks for 1-bit cnns via discrete back propagation. InAAAI Conference on Artificial Intelligence, 2019

    work page 2019

  8. [8]

    S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems , pages 1135–1143, 2015

    2015

Show all 26 references
  1. [9]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016

    2016

  2. [10]

    Krizhevsky, V

    A. Krizhevsky, V . Nair, and G. Hinton. The cifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 2014

    2014

  3. [11]

    C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin. Extremely low bit neural network: Squeeze the last bit out with admm. In AAAI Conference on Artificial Intelligence, 2018

    2018

  4. [12]

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017

    2017

  5. [13]

    X. Lin, C. Zhao, and W. Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pages 345–353, 2017

    2017

  6. [14]

    G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision, pages 85–100, 2018

    2018

  7. [15]

    W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.- Y . Fu, and A. C. Berg. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision, pages 21–37. Springer, 2016

    2016

  8. [16]

    Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision, pages 747–763. Springer, 2018

    2018

  9. [17]

    D. J. MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448– 472, 1992

    1992

  10. [18]

    Mockus, V

    J. Mockus, V . Tiesis, and A. Zilinskas. The application of bayesian methods for seeking the extremum.Towards global optimization, 2(117-129):2, 1978

    1978

  11. [19]

    Rastegari, V

    M. Rastegari, V . Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Conference on Computer Vision, pages 525–542, 2016

    2016

  12. [20]

    D. Wan, F. Shen, L. Liu, F. Zhu, J. Qin, L. Shao, and H. Tao Shen. Tbn: Convolutional neural network with ternary inputs and binary weights. In Proceedings of the European Conference on Computer Vision, September 2018

    2018

  13. [21]

    X. Wang, B. Zhang, C. Li, R. Ji, J. Han, X. Cao, and J. Liu. Modulated convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition , pages 840–848, 2018

    2018

  14. [22]

    Y . Wen, K. Zhang, Z. Li, and Y . Qiao. A discriminative feature learning approach for deep face recognition. In Proceedings of the European Conference on Computer Vision, pages 499–515. Springer, 2016

    2016

  15. [23]

    S. Wu, G. Li, F. Chen, and L. Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations, 2018

    2018

  16. [24]

    Zagoruyko and N

    S. Zagoruyko and N. Komodakis. Wide residual networks. In British Machine Vision Conference. British Machine Vision Association, 2016

    2016

  17. [25]

    S. Zhou, Y . Wu, Z. Ni, X. Zhou, H. Wen, and Y . Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016

    2016 arXiv

  18. [26]

    Zhuang, C

    B. Zhuang, C. Shen, M. Tan, L. Liu, and I. Reid. Towards ef- fective low-bitwidth convolutional neural networks. InIEEE Conference on Computer Vision and Pattern Recognition , June 2018

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.