Pith. sign in

REVIEW 3 major objections 7 minor 32 references

The HSIC Bottleneck: Deep Learning without Back-Propagation

T0 review · 3 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep classification networks can be trained with an information-bottleneck objective and no backpropagation, matching backprop accuracy.

desk verdict A genuinely new no-backprop training idea with code, but the 'comparable to backprop' claim rests on single-seed curves and an under-tuned baseline, so the paper deserves a serious referee but not yet a pass. read the letter →

arxiv 1908.01580 v3 pith:4PRCML4C submitted 2019-08-05 cs.LG stat.ML

Wan-Duo Kurt Ma , J.P. Lewis , W. Bastiaan Kleijn This is my paper · ORCID
classification cs.LGstat.ML
keywords Hilbert-Schmidtindependencecriterioninformationbottleneckbackpropagation-freelearningkernelmethodslayerwisetrainingdeepclassificationHSIC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that deep classifiers can be trained without backpropagation by replacing the cross-entropy loss with a layerwise information-bottleneck objective. The objective, called the HSIC bottleneck, maximizes a kernel measure of dependence between each hidden layer and the class labels while minimizing dependence between that layer and the input, using the Hilbert-Schmidt independence criterion instead of mutual information. On MNIST, FashionMNIST, and CIFAR10 the layerwise training alone tends to produce one-hot-like outputs that can be read off by a fixed permutation, and appending one softmax layer trained with plain SGD matches or slightly beats an identical network trained by backpropagation. If correct, the method would decouple layer training from end-to-end error propagation, removing vanishing-gradient, weight-transport, and update-locking constraints.

What carries the argument

The load-bearing object is the normalized Hilbert-Schmidt independence criterion (nHSIC), used as a stand-in for mutual information in the information bottleneck. For a minibatch, nHSIC between two variables is $\operatorname{tr}(\tilde K_X \tilde K_Y)$, where $\tilde K$ are centered, regularized kernel matrices; the paper maximizes this between each layer's activation and the label matrix and minimizes it between the activation and the input, with a single kernel width $\sigma$ scaled by layer dimension and a fixed $\beta=500$. This quantity carries the argument: it replaces the intractable mutual information terms with a differentiable, $O(m^2)$ kernel statistic that can be optimized per layer without propagating gradients backward.

What would settle it

Train the same HSIC-bottleneck architecture while permuting the label matrix independently of the inputs in every minibatch; if the layerwise objective still drives $\mathrm{nHSIC}(Z_i,Y)$ upward and produces one-hot outputs with format-training accuracy near the labeled case, the label term is not the mechanism. A complementary check computes true mutual information $I(Z_i;X)$ and $I(Z_i;Y)$ on a small Gaussian mixture where MI is tractable: if the optimized nHSIC objective rises while $I(Z_i;Y)$ falls or $I(Z_i;X)$ stays constant, the bottleneck interpretation fails.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that the nHSIC objective $Z_i^* = \arg\min_{Z_i} \mathrm{nHSIC}(Z_i,X) - \beta\,\mathrm{nHSIC}(Z_i,Y)$, optimized independently at each layer by block coordinate descent, yields hidden representations that retain label information and discard input information, and that these representations are sufficient for classification. In experiments with fully connected networks and with a residual network, unformatted HSIC training tended to produce permuted one-hot outputs, and format training (a single appended softmax layer trained with SGD without backpropagation) reached final test accuracies of 98.8%, 88.3%, and 59.4% on MNIST, FashionMNIST, and CIFAR10, versus 98.4%, 87.6%, and 56.5% for the backpropagation-trained equivalents. The paper also claims that because gradients are computed layerwise, the method avoids the vanishing and exploding gradient problems that make very deep backpropagation-trained networks fail, and it removes the need for symmetric feedback and update locking.

Load-bearing premise

The load-bearing premise is that optimizing the normalized HSIC objective layer by layer, with a single kernel-width scaling rule and fixed beta, really creates representations that preserve label information and discard input information, even though the exact relation between HSIC and mutual information has not been established.

Editorial extensions

If this is right

  • Deep networks can be trained with no backward pass at all, so the vanishing- and exploding-gradient failure mode of backpropagation is sidestepped by construction.
  • Because each layer's objective is local, layers can in principle be updated in parallel rather than sequentially, a consequence of removing update locking.
  • The unformatted one-hot phenomenon means a classifier can be extracted from an HSIC-trained network by a fixed permutation of output units, with no learned output layer.
  • Appending a single SGD-trained layer on top of the frozen HSIC representation consistently improves accuracy and converges faster than training the same architecture with backpropagation.
  • Combining several kernel widths in parallel outperforms any single width, suggesting that the HSIC objective benefits from covering multiple scales.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the informal entropy-diameter analogy in the appendix could be made rigorous, HSIC might become a general computable proxy for mutual information in other bottleneck-style objectives.
  • The layerwise independence of the objective suggests a natural testbed for very deep or wide networks where backpropagation is infeasible; the paper's results hint, but do not prove, that the advantage over backpropagation grows with depth.
  • The sensitivity to kernel width and the multiple-scale improvement point toward a learnable or adaptive $\sigma$ per layer as a direct extension, and the fixed $\beta$ may need re-tuning for new tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces the HSIC bottleneck, a layer-wise training objective for deep networks that avoids backpropagation and the cross-entropy loss. The objective (6) minimizes the normalized HSIC between each layer activation and the input while maximizing nHSIC between the activation and the labels, optimized independently per layer by block coordinate descent (Algorithm 1). The resulting representations are used either directly through a fixed permutation when they are one-hot ('unformatted training') or as input to an appended softmax layer trained by SGD without backpropagation ('format training'). Experiments on MNIST, FashionMNIST, and CIFAR10 with fully connected networks and a ResNet report accuracy comparable to backpropagation counterparts; final ResNet numbers are (98.8, 88.3, 59.4) for format-trained versus (98.4, 87.6, 56.5) for backpropagation. The authors state that these are initial results, not state-of-the-art, and the appendix disclaims an exact HSIC-MI relation.

Significance. The paper's strongest asset is the demonstration that a nontrivial deep classifier can be trained without backpropagation using a simple, transparent kernel objective. Algorithm 1 is explicit and reproducible, code is provided, and the experimental setup compares identical architectures. The reported advantages—no update locking, no symmetric feedback, layer-local computation—are concrete and potentially influential. However, the headline claim of competitiveness with backpropagation is not yet supported at the current evidence level: the head-to-head curves are single trajectories, the one-hot behavior is explicitly seed-dependent with no selection policy reported, and the hyperparameters were tuned on the same datasets while the baseline received less tuning. These are fixable evidential gaps rather than flaws in the method's logic.

major comments (3)
  1. [§4.4, Fig. 7] The central claim that HSIC-bottleneck training is competitive with backpropagation rests on single-run accuracy curves with no error bars or multi-seed statistics. The final ResNet numbers (format 98.8/88.3/59.4 vs. backprop 98.4/87.6/56.5) differ by less than 3 points; without knowing seed-to-seed variance, these margins could easily be reversed. Please report mean and standard deviation over at least 5–10 seeds for both methods, and state the selection rule for the displayed runs.
  2. [§4.1] The paper states that unformatted training produces one-hot outputs for 'many (but not all) random weight initializations' yet does not report the fraction of successful initializations, how the displayed runs were chosen, or whether the format-training results in Sections 4.2–4.4 depend on this selection. Since the method's advertised ability to classify without an output layer depends on this behavior, please quantify its frequency and demonstrate that the main comparisons are not cherry-picked seeds.
  3. [§4, Eq. (6)] The hyperparameters beta=500 and sigma=5 are reported as set after 'a modest hyperparameter search' on the same benchmark datasets, and the text also concedes that 'relatively little effort' was spent tuning the backpropagation baseline. This makes the competitiveness comparison difficult to interpret: the improved accuracy of format-training over backpropagation could partly reflect asymmetric tuning. Please report the baseline's hyperparameter search (learning rate, batch size, epochs, and any regularization) and use a common validation-based protocol for both methods.
minor comments (7)
  1. [§2, Eq. (4)] The sentence introducing Eq. (4) is garbled ('the i'th of component'); please rewrite it and define the notation k_{Xi,j} precisely.
  2. [§4.1 and Fig. 4 caption] The caption states that the top and bottom rows use 5 and 50 hidden layers, but Section 4.1 describes a 784-256-256-256-256-256-10 network with five hidden layers; please clarify which architecture corresponds to the shallow and deep cases in Fig. 4.
  3. [§3.5, Footnote 4] The paper claims HSIC is O(m^2) but Footnote 4 says the released code uses an O(m^3) implementation; please state the computational cost of the actual artifact or improve the implementation to match the claim.
  4. [§4.4] Please report the ResNet architecture (number of residual blocks, filter sizes, and parameter count) and the exact epoch budget for Fig. 7; the text mentions 'initial epochs' but then quotes final accuracies.
  5. [§4] The sentence 'For complete details of the experiments refer to our github repository' makes the paper not fully self-contained; include the key hyperparameters (minibatch size, learning rate, epoch count, number of runs) in the main text or an appendix.
  6. [Supplementary material] The appendix titled 'Differences in Arxiv version 2' is a version-history note and should be removed from a published version.
  7. [Fig. 8 caption] 'tanh classed activation in R1 distribution' should read 'tanh activation distribution in R1'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the HSIC-bottleneck is an empirically evaluated training objective, and the paper's claims are supported by held-out test-set comparisons rather than by construction.

full rationale

The paper's central claim is that optimizing the nHSIC surrogate in Eq. (6) at each layer produces representations that support classification without backpropagation. This is an empirical claim: Algorithm 1 defines an optimization procedure, and the experimental sections (Figs. 4, 6, 7, Sec. 4.4) evaluate the resulting networks on held-out test sets against backpropagation-trained counterparts. The objective itself is not derived from the results; rather, the results are consequences of running the objective. The one-hot 'unformatted' behavior is reported as an empirical observation ('often results in non-overlapping one-hot output activations for many (but not all) random weight initializations'), and the fixed-permutation classifier is a standard way to read out a permuted representation, not a fitted quantity relabeled as a prediction. The kernel width scaling and fixed values of beta=500 and sigma=5 are hyperparameter choices, explicitly acknowledged as the result of a modest search, which does not constitute circular reasoning. The appendix's informal HSIC-to-entropy analogy is explicitly non-rigorous ('its exact relation to mutual information has not been established') and is not used to prove any benchmark result. There are no load-bearing self-citations: the method cites standard external work for HSIC estimation and the information bottleneck, and no uniqueness theorem from the authors is invoked to force the choice of objective. In short, every stated result is tested rather than assumed, and no equation reduces to its own input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on treating nHSIC as a mutual-information surrogate, on the validity of layer-wise block coordinate descent, and on kernel hyperparameters chosen for the target datasets. No new physical or mathematical entities are introduced; the leading free parameters are beta, sigma, epsilon, and minibatch size, with the latter two not reported in the text.

free parameters (4)
  • beta (bottleneck trade-off) = 500
    Balances minimization of nHSIC(Z_i,X) and maximization of nHSIC(Z_i,Y) in objective (6); chosen by a modest hyperparameter search on the target datasets, not derived.
  • sigma (Gaussian kernel width) = 5
    Kernel scale in the nHSIC computation; the paper states it empirically balances the objective and Figure 6c shows accuracy varies strongly with sigma.
  • epsilon (nHSIC regularization) = not reported
    Stabilizer in equation (5) for the normalized kernel matrices; its value is not stated, which is a small reproducibility gap.
  • minibatch size m = not reported
    The authors note the method's performance depends heavily on minibatch size, but the values used in the experiments are not stated in the paper.
assumptions (4)
  • ad hoc to paper nHSIC is an adequate surrogate for mutual information in the information bottleneck objective.
    The entire method replaces I(X;Z_i) and I(Z_i;Y) in equation (1) with nHSIC terms, while the paper states the exact relation between HSIC and mutual information is not established and offers only an informal analogy.
  • ad hoc to paper Layer-wise independent optimization by block coordinate descent converges to useful representations.
    Algorithm 1 updates each layer's weights from local HSIC gradients without global error backpropagation; no convergence proof or analysis is provided.
  • ad hoc to paper A single kernel scale sigma normalized by sqrt(d) is valid across layers and datasets.
    Section 2 assumes expected squared distance scales with dimension to justify k(x,y) ~ exp(-||x-y||^2/(2 sigma^2 d)), and this rule is used in all experiments.
  • standard math Empirical HSIC estimates converge to population values.
    The paper invokes the 1/sqrt(n) convergence result from Gretton et al. 2005 to justify finite-sample computation; this is a standard result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The HSIC Bottleneck: Deep Learning without Back-Propagation." pith.science (2026). https://pith.science/paper/4PRCML4C

@misc{pith2026190801580,
  author       = {Pith},
  title        = {Pith review of: The HSIC Bottleneck: Deep Learning without Back-Propagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PRCML4C}},
  note         = {Machine review of arXiv:1908.01580}
}
read the original abstract

We introduce the HSIC (Hilbert-Schmidt independence criterion) bottleneck for training deep neural networks. The HSIC bottleneck is an alternative to the conventional cross-entropy loss and backpropagation that has a number of distinct advantages. It mitigates exploding and vanishing gradients, resulting in the ability to learn very deep networks without skip connections. There is no requirement for symmetric feedback or update locking. We find that the HSIC bottleneck provides performance on MNIST/FashionMNIST/CIFAR10 classification comparable to backpropagation with a cross-entropy target, even when the system is not encouraged to make the output resemble the classification labels. Appending a single layer trained with SGD (without backpropagation) to reformat the information further improves performance.

Figures

Figures reproduced from arXiv: 1908.01580 by the authors.

Figure 1
Figure 1. The HSIC-trained network (Fig. 1a) is a standard [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. The MNIST output category distribution for a shal [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Visualization of the HSIC-bottleneck quantities [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Test accuracy versus epoch number for differently [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 4
Figure 4. Figure 4: The test accuracy of HSIC-bottleneck training on [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Format-trained training accuracy as a function of [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Test accuracy versus epoch number for ResNet. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: The tanh classed activation in R 1 distribution of MNIST between backpropagation Fig 8a, and unformatted￾training Fig 8b. Each curve represents the activation distri￾bution from particular image category. The distribution is generated by Gaussian Kernel Density Estimat…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 23 canonical work pages

  1. [1]

    A.; Fischer, I.; Dillon, J

    Alemi, A. A.; Fischer, I.; Dillon, J. V.; and Murphy, K. 2017. Deep variational information bottleneck. In ICLR . OpenReview.net

    work page 2017

  2. [2]

    A., and Geiger, B

    Amjad, R. A., and Geiger, B. C. 2018. How (not) to train your neural network using the information bottleneck principle. CoRR abs/1802.09766

    arXiv 2018

  3. [3]

    Baddeley, R.; Foldiak, P.; and Hancock, P., eds. 1999. Information Theory and the Brain . Cambridge University Press

    work page 1999

  4. [4]

    Balduzzi, D.; Vanchinathan, H.; and Buhmann, J. 2015. Kickback cuts backprop's red-tape: Biologically plausible credit assignment in neural networks. In Proc. AAAI

    work page 2015

  5. [5]

    The Variational Deficiency Bottleneck

    Banerjee, P. K., and Mont \' u far, G. 2018. The variational deficiency bottleneck. CoRR abs/1810.11677

    work page Pith review arXiv 2018

  6. [6]

    D.; and Courville, A

    Belghazi, I.; Rajeswar, S.; Baratin, A.; Hjelm, R. D.; and Courville, A. C. 2018. MINE: mutual information neural estimation. CoRR abs/1801.04062

    arXiv 2018

  7. [7]

    B., and Gretton, A

    Blaschko, M. B., and Gretton, A. 2008. A Hilbert-Schmidt dependence maximization approach to unsupervised structure discovery. In Proc. 6th Int. Workshop on Mining and Learning with Graphs

    work page 2008

  8. [8]

    Brakel, P., and Bengio, Y. 2018. Learning independent features with adversarial nets for non-linear ICA . https://openreview.net

    work page 2018

Show all 32 references
  1. [9]

    Choromanska, A.; Cowen, B.; Kumaravel, S.; Luss, R.; Rigotti, M.; Rish, I.; Diachille, P.; Gurev, V.; Kingsbury, B.; Tejwani, R.; and Bouneffouf, D. 2019. Beyond backprop: Online alternating minimization with auxiliary variables. In ICML , 1193--1202

    2019

  2. [10]

    M., and Thomas, J

    Cover, T. M., and Thomas, J. A. 2006. Elements of Information Theory . New York, NY, USA: Wiley-Interscience

    2006

  3. [11]

    de Souza Farias, T., and Maziero, J. 2018. Gradient target propagation. CoRR abs/1810.09284

    2018 arXiv

  4. [12]

    Fukumizu, K.; Gretton, A.; Sun, X.; and Sch\" o lkopf, B. 2008. Kernel measures of conditional dependence. In Platt, J. C.; Koller, D.; Singer, Y.; and Roweis, S. T., eds., NIPS . 489--496

    2008

  5. [13]

    H.; Melnyk, I.; Nguyen, N.; Kingsbury, B.; and Polyanskiy, Y

    Goldfeld, Z.; van den Berg, E.; Greenewald, K. H.; Melnyk, I.; Nguyen, N.; Kingsbury, B.; and Polyanskiy, Y. 2018. Estimating information flow in neural networks. CoRR abs/1810.05728

    2018 arXiv

  6. [14]

    Gretton, A.; Bousquet, O.; Smola, A.; and Sch\" o lkopf, B. 2005. Measuring statistical dependence with H ilbert- S chmidt norms. In Proc. Int. Conf. Algorithmic Learning Theory , 63--77. Springer-Verlag

    2005

  7. [15]

    Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167

    2015 arXiv

  8. [16]

    A.; Rietman, E

    Kohan, A. A.; Rietman, E. A.; and Siegelmann, H. T. 2018. Error forward-propagation: Reusing feedforward connections to propagate errors in deep learning. CoRR abs/1808.03357

    2018 arXiv

  9. [17]

    D.; and Wolpert, D

    Kolchinsky, A.; Tracey, B. D.; and Wolpert, D. H. 2017. Nonlinear information bottleneck. CoRR abs/1705.02436

    2017 arXiv

  10. [18]

    Kwak , N., and Chong-Ho Choi . 2002. Input feature selection by mutual information based on P arzen window. IEEE Tran. Pattern Analysis and Machine Intelligence 24(12):1667--1671

    2002

  11. [19]

    P.; Cownden, D.; Tweed, D

    Lillicrap, T. P.; Cownden, D.; Tweed, D. B.; and Akerman, C. J. 2016. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications 7

    2016

  12. [20]

    Lopez, R.; Regier, J.; Yosef, N.; and Jordan, M. I. 2018. Information constraints on auto-encoding variational bayes. CoRR abs/1805.08672

    2018 arXiv

  13. [21]

    H.; Litwin - Kumar, A.; and Abbott, L

    Moskovitz, T. H.; Litwin - Kumar, A.; and Abbott, L. F. 2018. Feedback alignment in deep convolutional networks. CoRR abs/1812.06488

    2018 arXiv

  14. [22]

    M.; Bansal, Y.; Dapello, J.; Advani, M.; Kolchinsky, A.; Tracey, B

    Saxe, A. M.; Bansal, Y.; Dapello, J.; Advani, M.; Kolchinsky, A.; Tracey, B. D.; and Cox, D. D. 2018. On the information bottleneck theory of deep learning. In International Conference on Learning Representations

    2018

  15. [23]

    K.; and Fukumizu, K

    Sejdinovic, D.; Gretton, A.; Sriperumbudur, B. K.; and Fukumizu, K. 2012. Hypothesis testing using pairwise distances and associated kernels (with appendix). CoRR abs/1205.0411

    2012 arXiv

  16. [24]

    Shwartz - Ziv, R., and Tishby, N. 2017. Opening the black box of deep neural networks via information. CoRR abs/1703.00810

    2017 arXiv

  17. [25]

    K.; Fukumizu, K.; and Lanckriet, G

    Sriperumbudur, B. K.; Fukumizu, K.; and Lanckriet, G. 2010. On the relation between universality, characteristic kernels and RKHS embedding of measures. In International Conference on Artificial Intelligence and Statistics , 773--780

    2010

  18. [26]

    Sugiyama, M., and Yamada, M. 2012. On kernel parameter selection in H ilbert- S chmidt independence criterion. IEICE Transactions on Information and Systems E95D

    2012

  19. [27]

    C.; and Bialek, W

    Tishby, N.; Pereira, F. C.; and Bialek, W. 1999. The information bottleneck method. In Allerton Conference on Communication, Control, and Computing

    1999

  20. [28]

    K.; Gelly, S.; and Lucic, M

    Tschannen, M.; Djolonga, J.; Rubenstein, P. K.; Gelly, S.; and Lucic, M. 2019. On mutual information maximization for representation learning. CoRR abs/1907.13625

    2019 arXiv

  21. [29]

    Vepakomma, P.; Gupta, O.; Dubey, A.; and Raskar, R. 2019. Reducing leakage in distributed deep learning for sensitive health data. In ICLR AI for social good workshop

    2019

  22. [30]

    Werbos , P. J. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10):1550--1560

    1990

  23. [31]

    D ependency B ottleneck

    Wu, D.; Zhao, Y.; Tsai, Y.-H. H.; Yamada, M.; and Salakhutdinov, R. 2018. " D ependency B ottleneck" in auto-encoding architectures: an empirical study. CoRR abs/1802.05408

    2018 arXiv

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.