Pith. sign in

REVIEW 3 major objections 6 minor 24 references

Width-Robust Learnability in Mean-Field Bayesian Neural Networks

T0 review · 3 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read At the mean-field scaling, infinite-width Bayesian nets learn exactly the same polynomial-sample targets as polynomial-width nets.

desk verdict Solid width-robust equivalence for critical mean-field Bayesian nets; the active/lazy compression is the real contribution, with the usual black-box mean-field caveat. read the letter →

arxiv 2607.05735 v1 pith:T4K3NJZX submitted 2026-07-07 stat.ML cs.LG

Dmitry Vaintrob , Kaarel Hänni This is my paper
classification stat.MLcs.LG MSC 68T0762F1560G15
keywords mean-fieldBayesianneuralnetworksreducedentropywidth-robustlearnabilityfeaturelearningactive-lazysplitpointwisecompressionBooleancubeinductivebias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Infinite-width limits are a common idealization of overparameterized neural networks, but it is not obvious that the limiting learner keeps the same complexity bias as the finite networks one actually trains. This paper studies Bayesian networks at the critical mean-field (feature-learning) scaling and introduces a single intensive complexity measure: the reduced entropy, which is the prior cost per neuron of representing a target to a given mean-squared accuracy. The central theorem says that, at fixed depth on Boolean-cube targets, three statements are equivalent up to polynomial slack: the target is learnable from polynomially many samples at infinite width; it is learnable at polynomial width; and its reduced entropy is polynomially bounded. Equivalently, the mean-field Bayesian learner generalizes on precisely the targets that can already be computed by a polynomial-size network. The forward direction works by an explicit subsampling construction that keeps the data-dependent active directions and resamples the entropy-dominated lazy directions from the prior, producing a polynomial-width network that agrees with the infinite-width solution at every input simultaneously. The result therefore certifies that the infinite-width mean-field description is a clean analytic model of learning rather than an artificial source of extra generalization power.

What carries the argument

Reduced entropy s_∞(y,ε) = limsup_N (−1/N log π_N⁰(L ≤ ε)), the intensive prior cost of fitting y to population MSE ε. Width-robustness is proved by pointwise compression: an active–lazy spectral split plus a lazy-swap lemma that lets one keep polynomially many representatives of the data-dependent directions while resampling the rest from the prior, yielding a finite network that agrees with the infinite-width solution at every input at once.

What would settle it

Exhibit a fixed-depth Boolean-cube family whose reduced entropy is super-polynomial yet whose infinite-width empirical Gibbs posterior (or a poly-width one) still reaches o(1) population MSE from only polynomially many samples, or show that the cited mean-field limit does not hold for those Gibbs measures.

Watch

Extended reading notes

Core claim

At fixed depth and the mean-field scaling, a family of Boolean-cube targets is learnable from polynomially many samples by the infinite-width Bayesian Gibbs posterior if and only if it is learnable at polynomial width if and only if its reduced entropy is polynomially bounded; equivalently, up to polynomial slack in accuracy, the learner generalizes exactly on the targets representable by polynomial-size networks.

Load-bearing premise

The infinite-width mean-field description of the empirical Gibbs posterior (propagation of chaos and the vacuum mixture) is taken as a black-box input from prior work rather than re-derived; if that limit fails for the measures actually used, the equivalence does not go through.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies Bayesian fully connected networks at the mean-field (γ=1) scaling and introduces the reduced entropy s_∞(y,ε), the intensive prior cost of representing a target y to population MSE ε. Its main claim (Theorem 3.1, assembled in §7.3) is that, at fixed depth L on Boolean-cube targets, the following are equivalent up to poly(d) slack depending only on L: (i) the infinite-width empirical Gibbs posterior achieves population MSE o(1) from polynomially many samples; (ii) a polynomial-width Gibbs posterior does the same; (iii) a polynomial-width, polynomial-weight network achieves L=o(1); equivalently s_∞ is polynomially bounded. The engine is a pair of primitives (Theorem 3.2): pointwise compression of low-entropy mean-field solutions to poly width via an active/lazy spectral split and a Stein lazy-swap (Theorem 5.4, Appendix B), and conservation of reduced entropy by cloning a narrow teacher into a wide prior cylinder (Theorem 6.1, Appendix C). Generalization then follows by placing the compressed proxy in a fixed bounded-weight class and applying uniform convergence (Lemma 7.1, Theorem 7.2).

Significance. If correct, the result is a genuine complexity-theoretic sanity check on the mean-field limit: infinite width changes the analytic description of learning but not the polynomial-sample learnability class, so the inductive bias visible in the mean-field equations is not an artifact of infinite resources. The reduced-entropy measure, the active/lazy split at γ=1, and especially the pointwise (sup-norm) compression witness are substantive contributions; the Stein lazy-swap (Lemma 5.1/B.2) and the KL-budget regularity package (Proposition 4.4/A.10) are clean technical tools that other mean-field analyses can reuse. The paper is appropriately scoped (Bayesian Gibbs, fixed depth, Boolean cube) and honest about what it does not claim (algorithmic capture, practical constants). Strengths include a full theorem–lemma chain with detailed appendices and a self-contained conservation argument.

major comments (3)
  1. §4.1, Definitions 4.1–4.2, Lemma D.1, and the conditioning steps in the proof of Theorem 7.2: the forward half of the equivalence (infinite-width learnability ⇒ poly-width / low reduced entropy) treats the infinite-width limit of the empirical multi-layer Gibbs posterior π_D^{N,β̄} as a mixture of mean-field vacua supplying single-site laws, kernels, and Boltzmann tilts. Appendix D only sketches this via Sanov/Varadhan on the finite-dimensional field space R^{2^d}; the body cites [14,22,16,21,20,11] as a black box. Those references do not uniformly cover multi-layer μP networks, adaptive-temperature empirical MSE conditioning, and Boolean-cube inputs simultaneously. Because compression (Theorem 5.4/B.14) and the data-independent class H_{N0,B} are applied to those single-site laws, a gap here is load-bearing for Theorem 3.1(i)⇔(ii) and the forward direction of Theorem 7.2. Either expand
  2. Theorem 7.2, Steps 1–2, and the empirical free-energy bound (10): the argument first compresses a population Gibbs witness to a data-independent teacher f⋆, then clones that fixed teacher to lower-bound prior mass on low empirical loss, then re-compresses the empirical Gibbs posterior. This ordering is essential (the witness must not memorize the sample). The write-up is dense and mixes population and empirical active spaces (U^pop ⊕ U^emp) without a single self-contained statement of the empirical compression theorem that is actually invoked. Please isolate an explicit empirical version of Theorem 5.4 (or Theorem B.14) with the lazy-Gram bound (Lemma B.3) as a black-box input, and verify that the polynomial degrees remain controlled when the empirical evaluation top space is adjoined and n is only poly(s,d,1/Δ).
  3. Scope of “learnability” vs. claims in the abstract and §1: the theorems characterize the Bayesian empirical Gibbs posterior at a polynomially bounded intensive temperature, not SGD or other training algorithms. The abstract’s phrasing (“the Bayesian mean-field learner generalizes exactly on the targets…”) is accurate if read carefully, but the introduction’s language about inductive bias and “what is learned” can be read as applying to practical training. A short, prominent caveat in the abstract or at the statement of Theorem 3.1 that the learner is the tempered Gibbs posterior (and that algorithmic capture is out of scope) would prevent over-reading; this is load-bearing for how the community will cite the result.
minor comments (6)
  1. Theorem 3.1 is labeled “informal” but is the headline result; §7.3 makes it fully rigorous. Consider stating a fully quantitative version in the main text (with explicit poly exponents depending only on L) and demoting the informal phrasing to a corollary or discussion sentence.
  2. Figure 1 and Figure 2 are helpful but under-specified in the caption (e.g., the meaning of λ0 and the poly width N0 in Fig. 2). Adding the cutoff definition and a pointer to Lemma 5.1 would help non-specialist readers.
  3. Notation dictionary at the end of §8 is useful; consider moving a short version earlier (e.g., after §3) so that body-level V vs. appendix U_ℓ is less confusing on first reading.
  4. Lemma 2.1 (memorization) is only sketched; a one-paragraph proof or a precise citation for the finite-set ridge-feature construction with poly weight bounds would make the poly(2^d) upper bound on s_∞ fully self-contained.
  5. Typos / polish: “thereduced entropy” (abstract), “Atthecriticalscaling” (p.5), occasional missing spaces after periods in the introduction; also “Princint” in acknowledgments may need expansion.
  6. Related work: the connection to approximate description length [5,6] and to Ringel et al. on sample/width scales is well taken; a sentence contrasting reduced entropy with PAC-Bayes KL bounds in parameter space would further locate the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: reduced entropy is an independent prior-mass complexity, and the width-robust equivalence is proved by genuine compression and cloning arguments rather than by definition or self-citation of the target claim.

full rationale

The central quantity s_∞(y,ε) is defined directly from the prior mass of the population MSE ball (eq. 3), not from learnability, sample complexity, or any fitted parameter. Theorem 3.1 equates poly-bounded reduced entropy with infinite-width and poly-width Gibbs learnability and with existence of a poly-width poly-weight teacher; both directions are proved by independent constructions. Compression (Thm 5.4 / App. B) starts from a finite-KL mean-field tower, splits active/lazy spectrum, applies the lazy-swap Stein bound (Lemma 5.1), and produces an explicit poly-width network agreeing pointwise with the infinite-width function—none of these steps assumes the learnability conclusion. Conservation (Thm 6.1 / App. C) clones a poly-width teacher into a cylinder of poly KL cost and applies change of measure; again the conclusion is derived, not presupposed. The mean-field vacuum limit (Def. 4.1, Lemma D.1) is taken as a cited external input from the standard literature [14,22,16,21,20,11] (authors do not overlap) and sketched via Sanov/Varadhan on the finite field space; that is an assumption/correctness risk, not a circular reduction of the target identity to itself. Companion-paper references concern the γ>1 and general finite-width regimes and are not load-bearing for the γ=1 equivalence. No quantity is fitted to data and then re-presented as a prediction; no uniqueness theorem is imported from the same authors to force the result. The derivation is therefore self-contained against its stated inputs.

Assumptions & free parameters 0 free parameters · 7 assumptions · 2 invented entities

The paper is a pure theory result. Load-bearing inputs are standard analysis/probability tools, the cited mean-field infinite-width limit, architectural assumptions (bounded Lipschitz non-polynomial activation, fixed depth, Boolean cube, Gaussian μP prior at γ=1), and the modeling choice that the learner is the empirical Gibbs posterior. No free parameters are fitted to data. Invented quantities are definitional (reduced entropy, active/lazy split) rather than new physical entities.

assumptions (7)
  • domain assumption Infinite-width mean-field limit exists for the (empirical) Gibbs posterior: propagation of chaos to single-site laws / vacuum mixture (Def. 4.1, Lemma D.1), taken from prior literature rather than re-proved.
    All compression and learnability arguments condition on this limit; §4.1 and App. D cite [14,22,16,21,20,11] as input.
  • domain assumption Activation φ is bounded (|φ|≤1), Lipschitz, and non-polynomial (with biases), enabling finite-set interpolation / memorization (Lemma 2.1).
    Standing assumption in §2.1; used for universal approximation of teachers and weight-size bounds.
  • domain assumption Architecture is fixed-depth fully connected μP mean-field scaling γ=1 on the Boolean cube with uniform measure and Gaussian priors as in (1).
    Scope of all theorems; depth L fixed so poly exponents may depend on L only.
  • domain assumption Learner is the adaptive empirical Gibbs posterior π ∝ exp(−N β̄ L̂_D) π⁰_N, not SGD or other algorithms.
    Stated in §2.2 and §7; paper explicitly does not claim training algorithms find the Gibbs posterior.
  • domain assumption Teachers are measured by width and polynomial weight size; conservation entropy bounds use poly-bounded teacher weights.
    Theorem 3.2(b) and Theorem 6.1; without poly weights the cloning KL cost need not be poly(W,d).
  • standard math Standard tools: Stein/Mehler for Ornstein–Uhlenbeck, Donsker–Varadhan, Sanov/Varadhan mean-field free energy, Hoeffding/McDiarmid, Gaussian concentration for operator norms.
    Used throughout Lemmas 5.1, A.3–A.4, B.2, D.1 without novel measure theory.
  • ad hoc to paper Regularity package (readout moments, operator norms, Cameron–Martin coefficients, lazy scores) follows from intensive KL rate R and bounded activation via single-site KL decomposition and suffix invariants.
    Proposition 4.4 / A.10 is proved in App. A from the entropy budget; it is paper-specific infrastructure the compression consumes.
invented entities (2)
  • Reduced entropy s_∞(y,ε) independent evidence
    purpose: Width-independent intensive prior cost of representing target y to MSE ε; the complexity parameter for the equivalence.
    Definitional complexity measure (Eq. 3), not a new physical object; independent meaning as KL rate of the conditioned prior.
  • Active/lazy spectral split and lazy-swap mechanism at γ=1
    purpose: Separate data-dependent low-dimensional directions from entropy-dominated bulk so poly-width subsampling preserves the function pointwise.
    Conceptual and technical device of the paper (Fig. 2, Lemma 5.1); falsifiable only as part of the theorem package, not as an external entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Width-Robust Learnability in Mean-Field Bayesian Neural Networks." pith.science (2026). https://pith.science/paper/T4K3NJZX

@misc{pith2026260705735,
  author       = {Pith},
  title        = {Pith review of: Width-Robust Learnability in Mean-Field Bayesian Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T4K3NJZX}},
  note         = {Machine review of arXiv:2607.05735}
}
abstract

Infinite-width limits are a standard way to reason about neural networks, but it is not automatic that the limiting learner has the same complexity-theoretic inductive bias as large finite networks. We study this question for Bayesian neural networks at the mean-field, or critical feature-learning, scaling. The central quantity is the \emph{reduced entropy} \[ s_\infty(y,\varepsilon)=\limsup_N -\frac{1}{N}\log \pi_N^0(L\le \varepsilon), \] the intensive prior cost of representing a target function $y$ to population mean-squared error $\varepsilon$. Our main result is a width-robust learnability theorem. At fixed depth, a family of Boolean-cube targets is learnable from polynomially many samples at infinite width if and only if it is learnable at polynomial width, if and only if its reduced entropy is polynomially bounded. Equivalently, up to polynomial slack in accuracy, the Bayesian mean-field learner generalizes exactly on the targets that can be represented by polynomial-size networks. The forward direction is proved by a form of subsampling: from the infinitely many hidden neurons in the mean-field solution, one can select polynomially many representatives and still preserve the learned function on every input simultaneously. At the critical scaling this subsampling has both an ``active'' component, which keeps the data-dependent low-dimensional statistics, and a ``lazy'' component, which resamples the entropy-dominated directions from the prior. Thus the infinite-width mean-field limit gives a clean analytic description of learning without introducing spurious width-dependent generalization power.

Figures

Figures reproduced from arXiv: 2607.05735 by the authors.

Figure 1
Figure 1. The spectrum across scalings. The same decaying feature-kernel spectrum, re￾coloured by regime. At γ < 1 every direction is lazy (brown) and the learner is a fixed kernel (NTK/NNGP); at the mean-field scaling γ = 1 a few top directions become active (blue) above a lazy bulk; at γ > 1 the lazy bulk falls below the accuracy threshold and only active directions remain. This paper treats the critical scaling γ = 1. Whic… view at source ↗
Figure 2
Figure 2. Pointwise compression at γ = 1. Top: the feature-kernel spectrum splits at the cutoff λ0 into a low-dimensional active space V (dim V ≤ 1/λ0) and a lazy bulk C (∥C∥op ≤ λ0). Bottom: every neuron of the infinite-width learned network carries an active and a lazy part. The construction subsamples a few neurons (red), projects them onto the active directions through the fixed top eigenfunctions {ψk} — reconstructing th… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages

  1. [1]

    Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang

    Boaz Barak, Benjamin L. Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: SGD learns parities near the computational limit. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  2. [2]

    Self-consistent dynamical field theory of kernel evo- lution in wide neural networks

    Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evo- lution in wide neural networks. volume 2023, 2022

    work page 2023

  3. [3]

    How uniform random weights induce non-uniform bias: Typical interpolating neural networks generalize with narrow teachers

    Gon Buzaglo, Itamar Harel, Mor Shpigel Nacson, Alon Brutzkus, Nathan Srebro, and Daniel Soudry. How uniform random weights induce non-uniform bias: Typical interpolating neural networks generalize with narrow teachers. InProceedings of the 41st International Confer- ence on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 503...

    work page 2024

  4. [4]

    Lénaïc Chizat and Francis R. Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. InNeural Information Processing Systems, 2018

    work page 2018

  5. [5]

    Generalization bounds for neural networks via approximate description length

    Amit Daniely and Elad Granot. Generalization bounds for neural networks via approximate description length. InAdvances in Neural Information Processing Systems, 2019

    work page 2019

  6. [6]

    On the sample complexity of two-layer networks: Lipschitz vs

    Amit Daniely and Elad Granot. On the sample complexity of two-layer networks: Lipschitz vs. element-wise lipschitz activation. InProceedings of The 35th International Conference on Algorithmic Learning Theory, volume 237 ofProceedings of Machine Learning Research, pages 505–517, 2024

    work page 2024

  7. [7]

    Learning parities with neural networks

    Amit Daniely and Eran Malach. Learning parities with neural networks. InAdvances in Neural Information Processing Systems, 2020

    work page 2020

  8. [8]

    Algorithmic Task Capture, Computational Complexity, and Inductive Bias of Infinite Transformers

    Orit Davidovich and Zohar Ringel. Algorithmic task capture, computational complexity, and inductive bias of infinite transformers.arXiv preprint arXiv:2603.11161, 2026

    work page Pith review arXiv 2026

Show all 24 references
  1. [9]

    Lecture notes: From gaussian processes to feature learning.arXiv preprint arXiv:2602.12855, 2026

    Moritz Helias, Javed Lindner, Lars Schutzeichel, and Zohar Ringel. Lecture notes: From gaussian processes to feature learning.arXiv preprint arXiv:2602.12855, 2026

    2026

  2. [10]

    Neuraltangentkernel: convergenceand generalization in neural networks

    ArthurJacot, FranckGabriel, andClémentHongler. Neuraltangentkernel: convergenceand generalization in neural networks. InAdvances in Neural Information Processing Systems, 2018

    2018

  3. [11]

    Adaptive kernel predictors from feature-learning infinite limits of neural networks.ArXiv, abs/2502.07998, 2025

    Clarissa Lauditi, Blake Bordelon, and Cengiz Pehlevan. Adaptive kernel predictors from feature-learning infinite limits of neural networks.ArXiv, abs/2502.07998, 2025

    2025 arXiv

  4. [12]

    Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein

    Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes. InInternational Conference on Learning Representations, 2018

    2018

  5. [13]

    Lin, Allan Pinkus, and Shimon Schocken

    Moshe Leshno, Vladimir Ya. Lin, Allan Pinkus, and Shimon Schocken. Multilayer feed- forward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861–867, 1993. 35

    1993

  6. [14]

    A mean field view of the landscape of two-layer neural networks.Proceedings of the National Academy of Sciences of the United States of America, 115:E7665 – E7671, 2018

    Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks.Proceedings of the National Academy of Sciences of the United States of America, 115:E7665 – E7671, 2018

    2018

  7. [15]

    A self consistent theory of gaussian processes captures feature learning effects in finite CNNs.arXiv preprint arXiv:2106.04110, 2021

    Gadi Naveh and Zohar Ringel. A self consistent theory of gaussian processes captures feature learning effects in finite CNNs.arXiv preprint arXiv:2106.04110, 2021

    2021 arXiv

  8. [16]

    A rigorous framework for the mean field limit of multilayer neural networks.Mathematical Statistics and Learning, 6(3):201–357, 2023

    Phan-Minh Nguyen and Huy Tuan Pham. A rigorous framework for the mean field limit of multilayer neural networks.Mathematical Statistics and Learning, 6(3):201–357, 2023

    2023

  9. [17]

    Rotskoff and Eric Vanden-Eijnden

    Grant M. Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of artificial neural networks: An interacting particle system approach.Communications on Pure and Applied Mathematics, 75, 2018

    2018

  10. [18]

    Mitigating the curse of detail: Scaling arguments for feature learning and sample complexity.arXiv preprint arXiv:2512.04165, 2025

    Noa Rubin, Orit Davidovich, and Zohar Ringel. Mitigating the curse of detail: Scaling arguments for feature learning and sample complexity.arXiv preprint arXiv:2512.04165, 2025

    2025

  11. [19]

    From kernels to features: A multi-scale adaptive theory of feature learning

    Noa Rubin, Kirsten Fischer, Javed Lindner, Inbar Seroussi, Zohar Ringel, Michael Kr"amer, and Moritz Helias. From kernels to features: A multi-scale adaptive theory of feature learning. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProcee...

    2025

  12. [20]

    A unified approach to fea- ture learning in bayesian neural networks

    Noa Rubin, Zohar Ringel, Inbar Seroussi, and Moritz Helias. A unified approach to fea- ture learning in bayesian neural networks. InHigh-dimensional Learning Dynamics (HiLD) Workshop, 2024

    2024

  13. [21]

    Separation of scales and a thermodynamic description of feature learning in some cnns.Nature Communications, 14, 2021

    Inbar Seroussi and Zohar Ringel. Separation of scales and a thermodynamic description of feature learning in some cnns.Nature Communications, 14, 2021

    2021

  14. [22]

    Sirignano and Konstantinos V

    Justin A. Sirignano and Konstantinos V. Spiliopoulos. Mean field analysis of neural net- works: A central limit theorem.Stochastic Processes and their Applications, 2018

    2018

  15. [23]

    Edward Hu

    Greg Yang and J. Edward Hu. Tensor programs iv: Feature learning in infinite-width neural networks. InInternational Conference on Machine Learning, 2021

    2021

  16. [24]

    Zavatone-Veth, Blake Bordelon, and Cengiz Pehlevan

    Jacob A. Zavatone-Veth, Blake Bordelon, and Cengiz Pehlevan. Summary statistics of learning link changing neural representations to behavior.arXiv preprint arXiv:2504.16920, 2025. 36

    2025 arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.