{"id":"dbe36afa-4979-43b5-aa9f-05ee0af9d638","arxiv_id":"1908.09772","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Deep neural networks can be represented as Bayesian hierarchical models built from neuron-defined Gibbs distributions, yielding a new explanation of regularization and generalization.","lead":"The paper recasts deep neural networks as Bayesian hierarchical models, where each neuron defines an energy function of a Gibbs distribution and layers form a chain of conditional distributions. It argues this view explains why over-parametrized networks generalize and why deep networks fail on random labels.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The probabilistic representation is a post-hoc definition, not a derivation: any layer output defines a Gibbs distribution by Eq. (7), and the §5 validation compares distributions on different spaces, so the generalization explanation is unfalsifiable.","rationale":"The paper's central claim is that the new probabilistic representation is not just a relabeling but explains hierarchy and generalization. For that to hold, 'hidden layers formulate Gibbs distributions' must be a nontrivial property. It is not: Eq. (7) is the Boltzmann form applied to an arbitrary function, so any layer output, trained or untrained, can be cast as a Gibbs distribution. The reader's chosen weakest assumption—the missing expectation in the KL decomposition in Section 4.2—is a real mathematical slip, but inserting E_{p(x)} before the conditional KL would still leave the qualitative claim that the supervised loss does not explicitly optimize KL[p(X)||q(X)]. The deeper problem is that q(X) is not a prior in the Bayesian sense: it is a learned feature distribution, and calling it an explicit regularizer rests on the vacuous Gibbs construction. The experimental validation cannot rescue the paper because it compares a pixel-intensity distribution with a scalar-energy histogram and uses those KL values to support the generalization explanation. A random-weight rerun of Section 5.1 would expose the construction as post hoc. On this reading, the central claim is unsupported, so the reader's REJECT verdict stands unchanged; the concern is different from the reader's weakest_assumption, though the reader also noted that Proposition 2 is a definition rather than a derivation.","tokens_in":11139,"tokens_out":16368,"duration_ms":179205,"concrete_test":"Repeat the Section 5.1 validation procedure with CNN1 initialized at random, untrained weights: compute E_F1 = -Σ_{k=1}^{20} f_k(x) on the synthetic test images, build the same energy histogram, and compute the same KL against p(X) as in Figure 3. If an untrained network also yields a finite KL and a well-defined Gibbs distribution, Proposition 2 is a tautology and the validation cannot distinguish the proposed representation from any fixed feature map, so the generalization explanation in Section 4.2 is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 2 in Section 3 is the load-bearing step: the paper defines, for each neuron output g_k(x), p(F_k)=exp(g_k(x))/Z_F with energy E_F(k)=-g_k(x) (Eqs. (6)-(7)), and then concludes that hidden layers 'formulate Gibbs distributions.' The text concedes that 'hidden layers only formulate the corresponding energy functions, rather than directly formulating Gibbs distributions,' so the conclusion is a post-hoc definition. Any deterministic layer, trained or untrained, admits the same construction over a finite dataset, so the representation imposes no constraint and cannot explain why over-parametrized DNNs generalize. The sufficiency argument is also used incorrectly: a one-dimensional energy histogram is not the full layer distribution q(F1), and Section 5.1 computes KL[p(X)||q(F1)]=0.83 between the pixel-intensity distribution N(0,1024) and a histogram of scalar energy values—distributions on different spaces. The same mismatch underlies the 0.94 vs 1.59 comparison in Section 5.2.1. Since the 'Gibbs' label is assigned after the fact and the validation compares incomparable objects, the central claim that DNNs perform explicit regularization by learning a prior q(X) is an unfalsifiable relabeling rather than an explanation of generalization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a probabilistic representation of deep neural networks in which each neuron defines an energy function of a Gibbs distribution, each hidden layer is said to formulate a Gibbs distribution, and the whole network is interpreted as a Bayesian hierarchical model. On this basis, the authors argue that the hierarchy of DNNs corresponds to a split into a learned prior q(X) and a likelihood q(Y|X), that DNNs perform explicit regularization by learning q(X), and that the relaxed training loss used by backpropagation fails to infer q(X) and therefore limits generalization. The claims are supported by KL divergence calculations on a synthetic Gaussian pixel dataset and by experiments on over-parametrization and random labels.","tokens_in":11480,"tokens_out":10717,"duration_ms":107329,"significance":"If the proposed representation were correct, it would offer a unified energy-based and Bayesian view of DNN architecture and could connect regularization to learning the input distribution. The paper is clearly written, the synthetic setup is simple, and the idea of treating early layers as learning a data-dependent prior is intuitively appealing. However, the central steps are either definitional or mathematically incorrect: the Gibbs-distribution claim in Proposition 2 imposes no constraints because any neuron output can be normalized into a distribution, and the generalization argument in Section 4.2 uses an incorrect KL decomposition. The experimental validation compares distributions on different spaces and therefore cannot corroborate the representation. As a result, the claimed explanatory power is not established.","major_comments":[{"comment":"The construction p(F_k)=exp(g_k(x))/Z_F with energy E_F(k)=-g_k(x) defines a normalized distribution over the neuron index k for any function g_k, so every deterministic layer, trained or untrained, trivially admits this representation. The text itself concedes that hidden layers only formulate the corresponding energy functions, rather than directly formulating Gibbs distributions; since any finite energy function defines a Gibbs distribution after normalization, Proposition 2 is true by construction and imposes no testable constraint. Similarly, the Markov factorization in Eq. (5) holds for any feedforward network, and the conditional distributions q(F_i|F_{i-1}) are degenerate for deterministic layers, so calling the network a Bayesian hierarchical model is a relabeling rather than a derived result. This definitional step cannot support the subsequent explanation of generalization.","section":"Section 3, Proposition 2 and Eq. (7)"},{"comment":"The identity KL[p_theta(Y,X)||q(Y,X)] = KL[p(Y|X)||q(Y|X)] + KL[p(X)||q(X)] is mathematically incorrect. The chain rule for KL divergence is KL[p(x,y)||q(x,y)] = KL[p(x)||q(x)] + E_{p(x)} KL[p(y|x)||q(y|x)]. The missing expectation over X is load-bearing: without it, the conclusion that the relaxed loss merely corresponds to KL[p(Y|X)||q(Y|X)] and therefore ignores q(X) does not follow. This invalidates the paper's central claim that backpropagation is a reason for decreasing the generalization ability of DNNs.","section":"Section 4.2, KL decomposition before Eq. (13)"},{"comment":"The reported KL divergences compare incompatible objects. p(X)=N(0,1024) is a distribution over pixel intensities or pixel vectors, while q(F1) is estimated by a histogram of the scalar energy E_{F1}; these are distributions of different random variables, so KL[p(X)||q(F1)] is not a well-defined measure of how well f1 models p(X). The same problem affects KL[p(X)||qCNN1(F1)]=0.94 and =1.59 in Section 5.2.1 and =0.60 in Section 5.2.2. Additionally, using the histogram of a one-dimensional energy to represent the full joint distribution q(F1) is unjustified; sufficiency of the energy for the Gibbs family does not imply that its marginal histogram determines the layer's full distribution.","section":"Section 5.1, Figure 3 and Section 5.2.1"},{"comment":"The claim that a DNN cannot model random labels because q(X,Y)=q(Y|X)q(X) only models dependent variables is false: q(Y|X) can be chosen independent of X, giving q(X,Y)=q(Y)q(X), so the model class can represent independence. The toy example is also not an instance of random labels, since it posits that the labels are a deterministic function of the 'full' feature that the DNN detects. This weakens the paper's clarification of the random-label phenomenon.","section":"Section 5.2.2, random labels argument"},{"comment":"The division of layers into prior and likelihood is post hoc. The paper states there is no clear boundary, and in Section 5.1 f1 is designated the prior because max pooling compresses too much information; this criterion is not derived from the probabilistic representation. Because the split can be chosen after training to fit the data, the hierarchy claim is unfalsifiable and cannot serve as an explanation of DNN hierarchy.","section":"Section 4.1 and Section 5.1, prior/likelihood split"}],"minor_comments":[{"comment":"The phrases 'the truly posterior distribution' and 'the truly prior distribution' should be 'the true posterior distribution' and 'the true prior distribution'.","section":"Throughout"},{"comment":"'perspective histograms' should be 'respective histograms'.","section":"Section 5.1"},{"comment":"The expression for Z' is ambiguous in the text; it should be typeset as Z' = Z / prod_k Z_F^{beta_{lk}}.","section":"Equation (6)"},{"comment":"The notation KL[p(X)||p(x)] uses p for both the population distribution and the empirical histogram of a single image, which is confusing; please use distinct symbols.","section":"Figure 3 caption"},{"comment":"Calling CNN1 and CNN2 over-parametrized because they have 1680 and 1330 parameters while an image has 1024 pixels is nonstandard; over-parametrization usually refers to capacity relative to the number of training examples, so the term should be clarified.","section":"Section 5.2.1"},{"comment":"The experimental section does not specify how histograms are constructed or how the KL divergences are computed (bin width, support, normalization), which makes the quantitative claims difficult to reproduce.","section":"Section 5"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper wants to give DNNs an explicit probabilistic interpretation. The parts that are true are already in the cited literature; the parts that are new don't hold together. The Gibbs representation is a post-hoc labeling, and the generalization argument rests on a KL decomposition that is simply wrong as written.\n\nTo be fair, the paper does a service by collecting the known equivalences—softmax as Gibbs (Bridle 1990), DBM hidden layers as RBMs (Salakhutdinov and Hinton 2009), conv layers as MRFs (Zheng et al. 2015)—under one Markov-chain/Bayesian-hierarchical umbrella. Proposition 1 is a valid factorization, if somewhat trivial. The exposition is clear and the authors are upfront about intractability.\n\nThe load-bearing problems are serious. Section 4.2 writes\nKL[pθ(Y,X)||q(Y,X)] = KL[p(Y|X)||q(Y|X)] + KL[p(X)||q(X)],\nomitting the expectation over X. The correct chain rule is KL[p(x,y)||q(x,y)] = KL[p(x)||q(x)] + E_{p(x)} KL[p(y|x)||q(y|x)]. Without that expectation the equality doesn't hold, and the claim that the training loss ignores the prior doesn't follow. The entire regularization discussion collapses on this point.\n\nSecond, Proposition 2 defines p(F_k)=exp(g_k(x))/Z, with energy -g_k(x). Any deterministic layer output on a finite dataset admits this construction, so it imposes no constraint and cannot explain why some networks generalize. The text itself concedes that hidden layers only formulate energy functions, not Gibbs distributions directly. That is a definition, not a derivation.\n\nThird, the experiments compare KL[p(X)||q(F1)] where p(X) is a Gaussian over pixel intensities and q(F1) is a histogram of scalar energies from the first conv layer. Those are distributions on different spaces; the numbers 0.83, 0.94, 1.59 are not meaningful as stated. Same problem in the random-label experiment.\n\nFinally, the random-label explanation is an analogy (the shape/color/filled example), not a proof. The claim that DNNs can't fit random labels because they only model dependent variables is contradicted by the fact that they do fit such labels perfectly; the toy example merely explains why test error is 50%.\n\nNet: the ambition is fine, but the central claims are unsupported. This is not ready for peer review as a research contribution. If the authors reframed it as a perspective piece and dropped the generalization results, it could be discussable. As is, I'd desk-reject.","headline":"The paper's central generalization claim is undercut by a wrong KL decomposition, and the Gibbs representation is a post-hoc definition rather than a derivation.","tokens_in":11964,"tokens_out":3654,"would_cite":false,"duration_ms":39841,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62F15"],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep neural network is a Bayesian hierarchical model whose layers are Gibbs distributions, with early layers forming a prior and later layers a likelihood, this paper claims.","keywords":["probabilistic representation","Bayesian hierarchical model","Gibbs distribution","energy-based model","generalization","variational inference","backpropagation","over-parametrization"],"falsifier":"On a trained network from the paper's synthetic dataset, compute both sides of the claimed identity $KL[p_\\theta(Y,X)||q(Y,X)] = KL[p(Y|X)||q(Y|X)] + KL[p(X)||q(X)]$: if the two sides differ because the right-hand side omits an expectation over $X$, the decomposition used to blame backpropagation is not an equality. A second check is to add an explicit input-prior matching term to the loss and see whether generalization improves as the argument predicts.","tokens_in":10924,"feed_emoji":"🧠","tokens_out":11485,"duration_ms":99818,"temperature":0.7,"pith_summary":"This paper tries to establish an explicit probabilistic identity for deep neural networks: the whole architecture, from input to output, is a Bayesian hierarchical model built from Gibbs distributions. Each neuron is said to define an energy, each hidden layer (or block of layers) is said to form a Gibbs distribution through that energy, and the product of conditional layers is a joint distribution $q(X,Y)=q(Y|X)q(X)$. If this representation is right, the hierarchy of a network has a concrete meaning—early layers learn a prior $q(X)$ and later layers learn a likelihood $q(Y|X)$—and generalization has a concrete mechanism: learning the prior is an explicit regularizer, while backpropagation's relaxed loss only fits the likelihood term. The paper uses this picture to explain why over-parametrized networks can still generalize and why random-label training reaches zero training error with chance-level test error. A sympathetic reader would care because it turns 'why do deep nets work' into a question about a particular probabilistic model with identifiable parts.","feed_headline":"Deep nets are Bayesian hierarchies of Gibbs layers","feed_subtitle":"Each hidden layer defines an energy function; the whole network becomes a prior that regularizes a likelihood","key_machinery":"The carrying machinery is the Markov-chain factorization $q(F_1;\\ldots;F_I;F_Y)=q(F_1)\\prod_{i=1}^{I-1}q(F_{i+1}|F_i)q(F_Y|F_I)$, combined with the neuron-as-energy mapping $E_F(k)=-g_k(x)$, which turns any hidden layer into a Gibbs distribution $p(F_k)=\\exp\\{g_k(x)\\}/Z$. The softmax output is the same construction over label configurations, and known results identify convolutional-plus-nonlinear blocks with Markov random fields, another Gibbs family. Because an energy function is a sufficient statistic for a Gibbs distribution, defining the layer's energy is taken to define the layer's distribution; because the chain product is a Bayesian hierarchical model, the network splits into a prior part and a likelihood part. Conjugacy of Gibbs distributions and the backward flow of backpropagation are the mechanism said to let the prior be inferred indirectly through the likelihood.","core_discovery":"On the paper's account, a deep neural network is a probabilistic model in a literal sense. The forward pass is a Markov chain $F_1\\to\\cdots\\to F_I\\to F_Y$, so the joint distribution factorizes as $q(F_1)q(F_2|F_1)\\cdots q(F_Y|F_I)$. Each neuron $g_k(x)$ is treated as the negative energy of a Gibbs distribution, making each hidden layer a Gibbs distribution via its energy function, and the softmax output is a discrete Gibbs distribution over labels. Proposition 1 states that the whole architecture is therefore a Bayesian hierarchical model, and Proposition 2 states that hidden layers formulate Gibbs distributions. The hierarchy is a split between layers modeling a prior $q(X)$ and layers modeling a likelihood $q(Y|X)$. The generalization claim is that learning $q(X)$ is an explicit regularizer, while the standard relaxed loss—viewed as variational inference—optimizes only the label-conditional divergence, so the prior is learned only indirectly and can be a poor match to $p(X)$; this is offered as the reason backpropagation can reduce generalization, and as the explanation for why over-parametrized networks still generalize and why random labels give chance-level test error. Experiments on a synthetic Gaussian-distributed digit dataset show the first-layer energy histogram close to $p(X)$ and are offered as validation.","pith_inferences":["A direct testable extension is to add an explicit prior-matching penalty to the relaxed loss, such as a divergence between the early-layer feature distribution and an estimate of $p(X)$; the representation predicts this should improve generalization, though the paper does not run that experiment.","The energy-function representation makes layer densities explicit enough that information-theoretic quantities like mutual information between layers become computable from trained weights, which could connect this hierarchy picture to information-bottleneck analyses without altering the network.","The prior-likelihood split suggests a design rule the paper does not state: architectures that put more capacity in early layers should generalize better on small or noisy datasets, because that is where the regularizing prior is learned."],"forward_implications":["If the representation is correct, over-parametrization is not inherently dangerous: extra hidden units can improve the learned prior $q(X)$, which regularizes the likelihood, so larger networks can generalize better rather than overfit.","Standard backpropagation is an incomplete variational inference: it fits the likelihood term of the divergence and leaves the prior term unoptimized, which predicts that two networks with identical prior-layer formulas but different later layers will learn different priors, exactly the CNN1 versus CNN2 comparison in the paper.","Random-label experiments are consistent with generalization: the network still learns a good $q(X)$ and can always find some feature separating the training labels, but since the true labels are independent of $X$, test error stays at chance.","The hierarchy property gives a principled way to say which part of a network is doing prior modeling and which part is doing likelihood modeling, so layer-wise transfer or feature-extraction decisions can be guided by the probabilistic role of each layer."],"supporting_citations":[{"why":"Establishes that the softmax output layer is a discrete Gibbs distribution, which grounds Proposition 2.","marker":"[6]"},{"why":"Supplies the Markov-chain factorization, the exponential-family form, and the fact that an energy function is a sufficient statistic.","marker":"[8]"},{"why":"Shows hidden layers of a deep Boltzmann machine are restricted Boltzmann or Gibbs distributions, supporting the hidden-layer claim.","marker":"[31]"},{"why":"Shows convolutional layers with nonlinearities form a conditional random field, used as a Gibbs-distribution example for multi-layer blocks.","marker":"[40]"},{"why":"Provides the premise that stochastic gradient descent performs variational inference during training.","marker":"[7]"},{"why":"Supplies the stochastic variational inference framework in which backpropagation is cast as approximate posterior inference.","marker":"[14]"},{"why":"Defines the backpropagation algorithm whose backward parameter flow is blamed for limiting generalization.","marker":"[30]"},{"why":"Gives the Bayesian regularization theory that identifies a learned prior with an explicit regularizer.","marker":"[35]"},{"why":"Reports over-parametrized networks that generalize without overfitting, the empirical phenomenon the prior-learning explanation targets.","marker":"[24]"},{"why":"Reports zero training error with high test error on random labels, the second empirical phenomenon the paper explains.","marker":"[38]"}],"fun_headline_variants":["Deep nets are Bayesian hierarchies of Gibbs layers","Neurons as energies: DNNs as Gibbs distributions","Generalization loss from indirect prior learning in DNNs","A Bayesian decomposition of deep networks: prior and likelihood","Why random labels don't teach deep nets: the prior stays"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the assumption that the training objective separates exactly into a term for getting the labels right and a term for matching the input distribution; if that separation is not an equality—say, because the input-distribution term must be weighted by the data distribution inside an expectation—then the conclusion that backpropagation limits generalization by ignoring the input prior does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Deep nets are Bayesian hierarchies of Gibbs layers","Neurons as energies: DNNs as Gibbs distributions","Generalization loss from indirect prior learning in DNNs","A Bayesian decomposition of deep networks: prior and likelihood","Why random labels don't teach deep nets: the prior stays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000391,"raw_usage":{"total_tokens":2079,"prompt_tokens":992,"completion_tokens":1087,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1008}},"tokens_in":608,"tokens_out":1087,"duration_ms":11106,"temperature":1.0,"reasoning_tokens":1008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:02:23.544623+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a trained network from the paper's synthetic dataset, compute both sides of the claimed identity $KL[p_\\theta(Y,X)||q(Y,X)] = KL[p(Y|X)||q(Y|X)] + KL[p(X)||q(X)]$: if the two sides differ because the right-hand side omits an expectation over $X$, the decomposition used to blame backpropagation is not an equality. A second check is to add an explicit input-prior matching term to the loss and see whether generalization improves as the argument predicts.","supporting_citations":[{"cited_title":"Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters","cited_arxiv_id":null,"evidence_quote":"Establishes that the softmax output layer is a discrete Gibbs distribution, which grounds Proposition 2."},{"cited_title":"Wiley- Interscience, Hoboken, New Jersy, 2006","cited_arxiv_id":null,"evidence_quote":"Supplies the Markov-chain factorization, the exponential-family form, and the fact that an energy function is a sufficient statistic."},{"cited_title":"Deep boltzmann machines","cited_arxiv_id":null,"evidence_quote":"Shows hidden layers of a deep Boltzmann machine are restricted Boltzmann or Gibbs distributions, supporting the hidden-layer claim."},{"cited_title":"Conditional random fields as recurrent neural networks","cited_arxiv_id":null,"evidence_quote":"Shows convolutional layers with nonlinearities form a conditional random field, used as a Gibbs-distribution example for multi-layer blocks."},{"cited_title":"Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks","cited_arxiv_id":null,"evidence_quote":"Provides the premise that stochastic gradient descent performs variational inference during training."},{"cited_title":"Hoffman, David M","cited_arxiv_id":null,"evidence_quote":"Supplies the stochastic variational inference framework in which backpropagation is cast as approximate posterior inference."},{"cited_title":"Rumelhart, Geoffrey E","cited_arxiv_id":null,"evidence_quote":"Defines the backpropagation algorithm whose backward parameter flow is blamed for limiting generalization."},{"cited_title":"Jaakkola","cited_arxiv_id":null,"evidence_quote":"Gives the Bayesian regularization theory that identifies a learned prior with an explicit regularizer."},{"cited_title":"In search of the real inductive bias: On the role of implicit regularization in deep learning","cited_arxiv_id":null,"evidence_quote":"Reports over-parametrized networks that generalize without overfitting, the empirical phenomenon the prior-learning explanation targets."},{"cited_title":"Understanding deep learning requires rethinking generalization","cited_arxiv_id":null,"evidence_quote":"Reports zero training error with high test error on random labels, the second empirical phenomenon the paper explains."}],"review_version":1}