{"id":"19c8d1bb-3031-4e8d-b326-56fa56a33b59","arxiv_id":"1908.01580","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An HSIC-based information-bottleneck objective trains deep networks layer-by-layer without backpropagation and matches backpropagation accuracy on small image benchmarks in the reported runs.","lead":"The paper introduces a way to train deep neural networks without backpropagation, using a kernel statistic called HSIC to measure how each layer relates to input and labels. On three standard image benchmarks the method reportedly reaches accuracy comparable to backpropagation, and it motivates a different computing pattern for deep learning training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that HSIC-bottleneck training is competitive with backpropagation relies on single-seed curves and an acknowledged under-tuned baseline; the reported margins are within the range where seed variance could flip the conclusion.","rationale":"The reader's weakest_assumption points at the theoretical link between HSIC and MI. I agree that link is informal, but it is not the load-bearing element: even if HSIC is not a faithful MI surrogate, the paper could still have made its empirical contribution. The load-bearing element is the reliability of the comparison that supports 'competitive'. Without error bars and seed selection criteria, a skeptic cannot distinguish a genuine advantage from run-to-run luck or a weak baseline. The code availability is real support, but the manuscript does not establish that the reported numbers are typical. This concern does not move the verdict away from the reader's CONDITIONAL; it strengthens the conditions already requested (multi-seed statistics, baseline tuning, success rates). I therefore recommend UNCHANGED.","tokens_in":13012,"tokens_out":11755,"duration_ms":116178,"concrete_test":"Re-run the ResNet experiment of Fig. 7 (and the 5-vs-50-layer experiments of Fig. 4) with at least 10 random seeds per condition for: (a) format-trained HSIC network, (b) unformatted HSIC network, (c) backprop-trained network with the same architecture and an equal or larger epoch budget and a learning rate chosen on a validation split. Report mean ± std test accuracy and the fraction of seeds that produce non-overlapping one-hot outputs. The concern is settled if the format-trained mean exceeds the backprop mean minus one standard deviation on the same benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: that the method trains without backpropagation, and that its results are competitive with backpropagation. The first part is convincingly demonstrated by Algorithm 1 and the provided code. The second part is the weak link. All head-to-head comparisons (Figs. 4 and 7, and the final ResNet numbers in Sec. 4.4) are single trajectories with no error bars. Sec. 3.2 admits that one-hot outputs occur for 'many (but not all) random weight initializations', so the method is seed-dependent; the paper does not report how the displayed runs were selected. Sec. 4 concedes the hyperparameters were set after 'a modest hyperparameter search' (v2 diff) and that 'neither the backpropagation baseline nor our method give state-of-the-art results' with 'relatively little effort towards finding optimal hyperparameters'. If the backprop baseline is under-tuned or the HSIC runs are lucky seeds, the claim of competitiveness is an artifact. The theoretical gap between HSIC and MI (appendix) is secondary because the empirical demonstration would stand on its own if the numbers are reliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the HSIC bottleneck, a layer-wise training objective for deep networks that avoids backpropagation and the cross-entropy loss. The objective (6) minimizes the normalized HSIC between each layer activation and the input while maximizing nHSIC between the activation and the labels, optimized independently per layer by block coordinate descent (Algorithm 1). The resulting representations are used either directly through a fixed permutation when they are one-hot ('unformatted training') or as input to an appended softmax layer trained by SGD without backpropagation ('format training'). Experiments on MNIST, FashionMNIST, and CIFAR10 with fully connected networks and a ResNet report accuracy comparable to backpropagation counterparts; final ResNet numbers are (98.8, 88.3, 59.4) for format-trained versus (98.4, 87.6, 56.5) for backpropagation. The authors state that these are initial results, not state-of-the-art, and the appendix disclaims an exact HSIC-MI relation.","tokens_in":13271,"tokens_out":4728,"duration_ms":44350,"significance":"The paper's strongest asset is the demonstration that a nontrivial deep classifier can be trained without backpropagation using a simple, transparent kernel objective. Algorithm 1 is explicit and reproducible, code is provided, and the experimental setup compares identical architectures. The reported advantages—no update locking, no symmetric feedback, layer-local computation—are concrete and potentially influential. However, the headline claim of competitiveness with backpropagation is not yet supported at the current evidence level: the head-to-head curves are single trajectories, the one-hot behavior is explicitly seed-dependent with no selection policy reported, and the hyperparameters were tuned on the same datasets while the baseline received less tuning. These are fixable evidential gaps rather than flaws in the method's logic.","major_comments":[{"comment":"The central claim that HSIC-bottleneck training is competitive with backpropagation rests on single-run accuracy curves with no error bars or multi-seed statistics. The final ResNet numbers (format 98.8/88.3/59.4 vs. backprop 98.4/87.6/56.5) differ by less than 3 points; without knowing seed-to-seed variance, these margins could easily be reversed. Please report mean and standard deviation over at least 5–10 seeds for both methods, and state the selection rule for the displayed runs.","section":"§4.4, Fig. 7"},{"comment":"The paper states that unformatted training produces one-hot outputs for 'many (but not all) random weight initializations' yet does not report the fraction of successful initializations, how the displayed runs were chosen, or whether the format-training results in Sections 4.2–4.4 depend on this selection. Since the method's advertised ability to classify without an output layer depends on this behavior, please quantify its frequency and demonstrate that the main comparisons are not cherry-picked seeds.","section":"§4.1"},{"comment":"The hyperparameters beta=500 and sigma=5 are reported as set after 'a modest hyperparameter search' on the same benchmark datasets, and the text also concedes that 'relatively little effort' was spent tuning the backpropagation baseline. This makes the competitiveness comparison difficult to interpret: the improved accuracy of format-training over backpropagation could partly reflect asymmetric tuning. Please report the baseline's hyperparameter search (learning rate, batch size, epochs, and any regularization) and use a common validation-based protocol for both methods.","section":"§4, Eq. (6)"}],"minor_comments":[{"comment":"The sentence introducing Eq. (4) is garbled ('the i'th of component'); please rewrite it and define the notation k_{Xi,j} precisely.","section":"§2, Eq. (4)"},{"comment":"The caption states that the top and bottom rows use 5 and 50 hidden layers, but Section 4.1 describes a 784-256-256-256-256-256-10 network with five hidden layers; please clarify which architecture corresponds to the shallow and deep cases in Fig. 4.","section":"§4.1 and Fig. 4 caption"},{"comment":"The paper claims HSIC is O(m^2) but Footnote 4 says the released code uses an O(m^3) implementation; please state the computational cost of the actual artifact or improve the implementation to match the claim.","section":"§3.5, Footnote 4"},{"comment":"Please report the ResNet architecture (number of residual blocks, filter sizes, and parameter count) and the exact epoch budget for Fig. 7; the text mentions 'initial epochs' but then quotes final accuracies.","section":"§4.4"},{"comment":"The sentence 'For complete details of the experiments refer to our github repository' makes the paper not fully self-contained; include the key hyperparameters (minibatch size, learning rate, epoch count, number of runs) in the main text or an appendix.","section":"§4"},{"comment":"The appendix titled 'Differences in Arxiv version 2' is a version-history note and should be removed from a published version.","section":"Supplementary material"},{"comment":"'tanh classed activation in R1 distribution' should read 'tanh activation distribution in R1'.","section":"Fig. 8 caption"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript's arXiv version includes a 'Differences in Arxiv version 2' appendix in the supplementary material; this is unusual for a submission and should be removed. I did not audit the GitHub repository, but a brief reproducibility statement or code inspection would strengthen the paper. The comparison with Vepakomma et al. (2019) is appropriately positioned: the missing second term in equation (6) is a real difference and the authors acknowledge it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is that a per-layer, two-term HSIC objective can stand in for the information bottleneck and train feedforward nets without any backward pass. That is not in the cited literature, and the authors show it more than once: unformatted training produces one-hot-ish outputs that classify directly, and a single appended SGD layer (still no backprop) gets results close to a same-architecture backprop baseline. Code is public and the algorithm is simple enough to reimplement in an afternoon. I believe the method does what they say it does; I am much less sure about the word \"comparable.\"\n\nThe honest part is good. The paper explicitly says neither method is state-of-the-art, concedes the backprop baseline is under-tuned, and admits the HSIC-MI connection is informal. That framing is fair. But the central comparison in Figs. 4 and 7 and the final ResNet numbers are single runs with no error bars. Section 3.2 says one-hot outputs occur for many but not all random initializations, and there is no description of how the displayed unformatted runs were selected. If those are lucky seeds, the margins in the ResNet table (98.8 vs 98.4, 88.3 vs 87.6, 59.4 vs 56.5) could flip. The stress-test note is right: the no-backprop part is proven, the competitive part is not.\n\nMinor soft spots: minibatch size and epsilon are not stated in the paper, only in code; the kernel scaling rule (sigma/d) is justified by a dimension-scaling argument but not checked against per-layer sigma selection; and there is no head-to-head against Kickback or feedback alignment, which would make the practical case much stronger given the same benchmark conditions.\n\nThe theory appendix is better than most such attempts, but it stays at the level of an analogy (volume vs diameter) and is not load-bearing. I do not count that against the paper, because the empirical demonstration stands or falls on its own.\n\nWho gets value from this: anyone working on backprop-free or biologically plausible learning, kernel dependence measures in deep nets, or information bottleneck training. It would be a fine reading-group paper because it is readable, provocative, and the flaw (missing error bars) is educational.\n\nRecommendation: engage with it. A serious editor should send this to peer review, not desk reject. But the reviewer should demand multi-seed results, explicit seed-selection criteria for unformatted training, and at least one comparison to an existing no-backprop method. If those hold up, the claim of competitiveness becomes real; if not, the method is still a useful contribution but a weaker one.","headline":"A genuinely new no-backprop training idea with code, but the 'comparable to backprop' claim rests on single-seed curves and an under-tuned baseline, so the paper deserves a serious referee but not yet a pass.","tokens_in":13779,"tokens_out":1345,"would_cite":true,"duration_ms":17255,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep classification networks can be trained with an information-bottleneck objective and no backpropagation, matching backprop accuracy.","keywords":["Hilbert-Schmidt independence criterion","information bottleneck","backpropagation-free learning","kernel methods","layerwise training","deep learning","classification","HSIC bottleneck"],"falsifier":"Train the same HSIC-bottleneck architecture while permuting the label matrix independently of the inputs in every minibatch; if the layerwise objective still drives $\\mathrm{nHSIC}(Z_i,Y)$ upward and produces one-hot outputs with format-training accuracy near the labeled case, the label term is not the mechanism. A complementary check computes true mutual information $I(Z_i;X)$ and $I(Z_i;Y)$ on a small Gaussian mixture where MI is tractable: if the optimized nHSIC objective rises while $I(Z_i;Y)$ falls or $I(Z_i;X)$ stays constant, the bottleneck interpretation fails.","tokens_in":12801,"feed_emoji":"🧠","tokens_out":9662,"duration_ms":88201,"temperature":0.7,"pith_summary":"The paper tries to show that deep classifiers can be trained without backpropagation by replacing the cross-entropy loss with a layerwise information-bottleneck objective. The objective, called the HSIC bottleneck, maximizes a kernel measure of dependence between each hidden layer and the class labels while minimizing dependence between that layer and the input, using the Hilbert-Schmidt independence criterion instead of mutual information. On MNIST, FashionMNIST, and CIFAR10 the layerwise training alone tends to produce one-hot-like outputs that can be read off by a fixed permutation, and appending one softmax layer trained with plain SGD matches or slightly beats an identical network trained by backpropagation. If correct, the method would decouple layer training from end-to-end error propagation, removing vanishing-gradient, weight-transport, and update-locking constraints.","feed_headline":"Deep learning without backpropagation matches backprop accuracy","feed_subtitle":"Layerwise kernel dependence trains networks that match or beat backprop on MNIST, FashionMNIST, and CIFAR10.","key_machinery":"The load-bearing object is the normalized Hilbert-Schmidt independence criterion (nHSIC), used as a stand-in for mutual information in the information bottleneck. For a minibatch, nHSIC between two variables is $\\operatorname{tr}(\\tilde K_X \\tilde K_Y)$, where $\\tilde K$ are centered, regularized kernel matrices; the paper maximizes this between each layer's activation and the label matrix and minimizes it between the activation and the input, with a single kernel width $\\sigma$ scaled by layer dimension and a fixed $\\beta=500$. This quantity carries the argument: it replaces the intractable mutual information terms with a differentiable, $O(m^2)$ kernel statistic that can be optimized per layer without propagating gradients backward.","core_discovery":"On its own terms, the central claim is that the nHSIC objective $Z_i^* = \\arg\\min_{Z_i} \\mathrm{nHSIC}(Z_i,X) - \\beta\\,\\mathrm{nHSIC}(Z_i,Y)$, optimized independently at each layer by block coordinate descent, yields hidden representations that retain label information and discard input information, and that these representations are sufficient for classification. In experiments with fully connected networks and with a residual network, unformatted HSIC training tended to produce permuted one-hot outputs, and format training (a single appended softmax layer trained with SGD without backpropagation) reached final test accuracies of 98.8%, 88.3%, and 59.4% on MNIST, FashionMNIST, and CIFAR10, versus 98.4%, 87.6%, and 56.5% for the backpropagation-trained equivalents. The paper also claims that because gradients are computed layerwise, the method avoids the vanishing and exploding gradient problems that make very deep backpropagation-trained networks fail, and it removes the need for symmetric feedback and update locking.","pith_inferences":["If the informal entropy-diameter analogy in the appendix could be made rigorous, HSIC might become a general computable proxy for mutual information in other bottleneck-style objectives.","The layerwise independence of the objective suggests a natural testbed for very deep or wide networks where backpropagation is infeasible; the paper's results hint, but do not prove, that the advantage over backpropagation grows with depth.","The sensitivity to kernel width and the multiple-scale improvement point toward a learnable or adaptive $\\sigma$ per layer as a direct extension, and the fixed $\\beta$ may need re-tuning for new tasks."],"forward_implications":["Deep networks can be trained with no backward pass at all, so the vanishing- and exploding-gradient failure mode of backpropagation is sidestepped by construction.","Because each layer's objective is local, layers can in principle be updated in parallel rather than sequentially, a consequence of removing update locking.","The unformatted one-hot phenomenon means a classifier can be extracted from an HSIC-trained network by a fixed permutation of output units, with no learned output layer.","Appending a single SGD-trained layer on top of the frozen HSIC representation consistently improves accuracy and converges faster than training the same architecture with backpropagation.","Combining several kernel widths in parallel outperforms any single width, suggesting that the HSIC objective benefits from covering multiple scales."],"supporting_citations":[{"why":"It supplies the information bottleneck tradeoff that the HSIC objective approximates.","marker":"Tishby, Pereira, and Bialek 1999"},{"why":"It defines the Hilbert-Schmidt independence criterion and its empirical trace estimator used in the objective.","marker":"Gretton et al. 2005"},{"why":"It provides the normalized cross-covariance operator that underlies the nHSIC formulation.","marker":"Fukumizu et al. 2008"},{"why":"It introduces the normalized dependence measure used in the paper's nHSIC equation.","marker":"Blaschko and Gretton 2008"},{"why":"It establishes that characteristic kernels make HSIC zero if and only if the variables are independent.","marker":"Sriperumbudur, Fukumizu, and Lanckriet 2010"},{"why":"It documents the weight transport and update locking problems that the proposed training removes.","marker":"Lillicrap et al. 2016"},{"why":"It is the prior HSIC-like input-dependence penalty without a label term, used to distinguish this contribution.","marker":"Vepakomma et al. 2019"},{"why":"It is an alternative backpropagation-free, layer-parallel training method used as a comparison point.","marker":"Choromanska et al. 2019"}],"fun_headline_variants":["HSIC bottleneck: deep learning without backprop","HSIC bottleneck trains deep nets without backprop","Backprop-free training with HSIC bottleneck matches BP accuracy","Deep learning without backprop: HSIC bottleneck rivals standard training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that optimizing the normalized HSIC objective layer by layer, with a single kernel-width scaling rule and fixed beta, really creates representations that preserve label information and discard input information, even though the exact relation between HSIC and mutual information has not been established.","fun_headline_variants_meta":{"raw":{"variants":["HSIC bottleneck: deep learning without backprop","HSIC bottleneck trains deep nets without backprop","Backprop-free training with HSIC bottleneck matches BP accuracy","Deep learning without backprop: HSIC bottleneck rivals standard training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000693,"raw_usage":{"total_tokens":3112,"prompt_tokens":896,"completion_tokens":2216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":2152}},"tokens_in":512,"tokens_out":2216,"duration_ms":16850,"temperature":1.0,"reasoning_tokens":2152,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:08:46.713530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same HSIC-bottleneck architecture while permuting the label matrix independently of the inputs in every minibatch; if the layerwise objective still drives $\\mathrm{nHSIC}(Z_i,Y)$ upward and produces one-hot outputs with format-training accuracy near the labeled case, the label term is not the mechanism. A complementary check computes true mutual information $I(Z_i;X)$ and $I(Z_i;Y)$ on a small Gaussian mixture where MI is tractable: if the optimized nHSIC objective rises while $I(Z_i;Y)$ falls or $I(Z_i;X)$ stays constant, the bottleneck interpretation fails.","supporting_citations":[{"cited_title":"C.; and Bialek, W","cited_arxiv_id":null,"evidence_quote":"It supplies the information bottleneck tradeoff that the HSIC objective approximates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the Hilbert-Schmidt independence criterion and its empirical trace estimator used in the objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the normalized cross-covariance operator that underlies the nHSIC formulation."},{"cited_title":"B., and Gretton, A","cited_arxiv_id":null,"evidence_quote":"It introduces the normalized dependence measure used in the paper's nHSIC equation."},{"cited_title":"K.; Fukumizu, K.; and Lanckriet, G","cited_arxiv_id":null,"evidence_quote":"It establishes that characteristic kernels make HSIC zero if and only if the variables are independent."},{"cited_title":"P.; Cownden, D.; Tweed, D","cited_arxiv_id":null,"evidence_quote":"It documents the weight transport and update locking problems that the proposed training removes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the prior HSIC-like input-dependence penalty without a label term, used to distinguish this contribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is an alternative backpropagation-free, layer-parallel training method used as a comparison point."}],"review_version":1}