REVIEW 6 cited by
Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Deep linear networks have been extensively studied, as they provide simplified models of deep learning. However, little is known in the case of finite-width architectures with multiple outputs and convolutional layers. In this manuscript, we provide rigorous results for the statistics of functions implemented by the aforementioned class of networks, thus moving closer to a complete characterization of feature learning in the Bayesian setting. Our results include: (i) an exact and elementary non-asymptotic integral representation for the joint prior distribution over the outputs, given in terms of a mixture of Gaussians; (ii) an analytical formula for the posterior distribution in the case of squared error loss function (Gaussian likelihood); (iii) a quantitative description of the feature learning infinite-width regime, using large deviation theory. From a physical perspective, deep architectures with multiple outputs or convolutional layers represent different manifestations of kernel shape renormalization, and our work provides a dictionary that translates this physics intuition and terminology into rigorous Bayesian statistics.
Forward citations
Cited by 6 Pith papers
-
Statistical mechanics of extensive-width Bayesian neural networks near interpolation
Statistical mechanics formulas predict universal-to-specialisation phase transitions and Bayes-optimal generalization errors for extensive-width two-layer Bayesian networks at interpolation.
-
Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime
Deep Bayesian neural networks in the proportional scaling regime are asymptotically equivalent, in mutual information and generalization error, to generalized linear models.
-
Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
Closed dynamical mean field equations describe train and test loss trajectories of randomly initialized deep linear networks at large width and data, capturing hyperparameter transfer and power-law scaling.
-
Adaptive kernel predictors from feature-learning infinite limits of neural networks
Feature-learning infinite-width neural networks are kernel machines with data-dependent kernels, defined by a min-max saddle point (Bayesian/Langevin) or a DMFT fixed point (gradient flow with weight decay).
-
Proportional infinite-width infinite-depth limit for deep linear neural networks
Deep linear neural networks in the proportional depth-width limit converge to a nontrivial mixture of Gaussians, with posterior output correlations that depend on the observed labels.
-
Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidden-layer networks
Output-output correlations in finite Bayesian one-hidden-layer networks follow the kernel shape renormalization order parameter, with readout weight overlap equal to Q*_ab/λ1.
Discussion (0). Continue with ORCID to comment.