REVIEW 4 major objections 4 minor 1 cited by
On Generalization Bounds for Neural Networks with Low Rank Layers
T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Deep networks with low-rank layers have generalization bounds where rank grows as Lr√h rather than compounding as r^{L/2}.
desk verdict A clever rank-aware bound, but Lemma 6 invalidates the main theorem as written; still worth a revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the chain rule for expected suprema of Gaussian processes, $G(F\circ Y) \le C_1 L(F)\,G(Y) + C_2 D(Y)\,R(F,Y) + G(F(y_0))$, which is used to peel one layer at a time. Two estimates feed into it. Lemma 5 bounds the diameter $D(Y)$ of the representation class by writing the difference of two networks' outputs as $(A_i-A'_i)x$ with $A_i$ a product of weight matrices and diagonal activation matrices; because every factor has rank at most its layer rank, $\mathrm{rank}(A_i)\le \min_j \mathrm{rank}(W_j)$ and $\|A_i\|_2\le \prod_j \|W_j\|_2$. Lemma 6 bounds the Gaussian average of Lipschitz quotients $R(F,Y)$ by $\|W_i\|_F\sqrt{h_i}$. These pieces, combined with a vector-valued Gaussian complexity that places one Gaussian variable on each output coordinate, are what prevent rank and width factors from multiplying across layers.
What would settle it
A concrete check is to estimate the empirical Gaussian complexity of ReLU networks with all layers of fixed spectral norm and rank $1$ at depths $L=5,10,20$; after factoring out the universal constant term $C_1^L$, the theorem predicts growth like $L\sqrt{h}$ in the normalized complexity, whereas rank multiplication across layers would predict growth like $2^{L/2}$. A direct algebraic check of Lemma 5 would also settle it: search for two low-rank ReLU networks whose output difference on a fixed sample exceeds the stated diameter bound $\|X\|_F\sqrt{2\min_j r_j}\,2\prod_j\|W_j\|_2$.
Extended reading notes
Core claim
The central result is Theorem 7, an upper bound on the Gaussian complexity of $F_L$. For a sample of $m$ points with $\|x_i\|_2\le R$, a piecewise-linear $1$-Lipschitz activation $\varphi$, and layer widths $h_i$, the bound reads approximately $$\hat{G}_S(F_L) \lesssim \frac{\|X\|_F}{m}\left( (\|W_1\|_F\sqrt{h_1})\prod_{i=2}^L C_1\|W_i\|_2 + \sum_{i=2}^L $C_1^{{L-i}}$ $C_2^{2}$ \sqrt{2 r_i \kappa_i}\, \|W_i\|_F\sqrt{h_i} \right),$$ where $r_i=\min_{j\le i}\mathrm{rank}(W_j)$ and $\kappa_i\propto\prod_{j\neq i}\|W_j\|_2$. With uniform rank and spectral bounds this simplifies to $O(C_1^L \prod_i B_i\, L\, r\sqrt{h}/\sqrt{m})$. The decisive point is that the rank enters through the running minimum of layer ranks in the disturbance terms, so low-rank layers stop the rank from compounding across depth, whereas earlier bounds evaluated on the same class carry $r^{L/2}$. The same chain-rule perspective shows that a rank-$1$ layer lets all layers above it be absorbed into a scalar Lipschitz map, which is the paper's explanation for why intermediate neural collapse should improve generalization.
Load-bearing premise
The proof depends on the activation being piecewise linear and $1$-Lipschitz (ReLU or Leaky ReLU), because that is what lets the difference between two networks' outputs be written through diagonal activation matrices with rank bounded by the smallest layer rank; for smooth activations the diameter estimate that halts rank accumulation has no stated proof, and the function class must also contain the zero function so the base term of the chain rule drops.
Editorial extensions
If this is right
- Low-rank inductive bias becomes provable generalization: on $F_L$ the gap is $O(C_1^L\prod_i B_i\, L\, r\sqrt{h}/\sqrt{m})$, so rank enters linearly and width only through $\sqrt{h}$.
- Intermediate neural collapse should tighten bounds: a rank-$1$ layer absorbs all layers above it into one scalar Lipschitz map, so the complexity is controlled by the layers below the collapsed layer.
- Compared with earlier norm-based bounds on the same class, which scale as $\sqrt{L}\,r^{L/2}$ or $\sqrt{L^3 r h}$, the new bound is better for deep low-rank networks but exposes a rank-depth tradeoff rather than a uniform win.
- The result gives a concrete mechanism by which the observed low-rank bias of training translates into better generalization, tying optimization findings to statistical guarantees.
Reading between the lines
- This suggests a testable architectural principle: after training, cutting the singular values of one intermediate layer down to the minimum rank in the chain should improve out-of-sample performance without retraining, and the improvement should not depend on the ranks of the other layers.
- The chain-rule treatment could plausibly be extended to Rademacher complexity using the recently proved Bernoulli-process analogue, which would bring the same rank-halting effect to architectures such as convolutional or residual networks.
- If the exponential-in-depth factor $C_1^L$ is removable, as similar depth factors were removed in earlier norm-based bounds, the remaining $L\,r\sqrt{h}$ dependence would make these bounds practically relevant for very deep low-rank models; this is an extrapolation beyond the paper's proof.
- One could also use the diameter argument to predict that generalization depends on the minimum rank along the chain rather than the average or maximum rank, a claim that empirical rank-reduction experiments could test.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes generalization bounds for deep neural networks whose weight matrices are constrained in spectral norm and rank. The authors apply Maurer's chain rule for Gaussian complexity to the class F_L = {f_W = W_L φ(...φ(W_1 x)...) : ||W_i||_2 ≤ B_i, rank(W_i) ≤ r_i} and claim a bound of the form O(C_1^L ∏_{i=1}^L B_i · L r √h / m), which would avoid the rank factor multiplying across layers that appears when prior norm-based bounds are evaluated on this class. The paper also discusses comparisons with Golowich et al., Bartlett et al., and Neyshabur et al., and argues that intermediate neural collapse can improve the bound via a rank-1 layer. The main proof is in Section 4, with key lemmas bounding the diameter of the function class (Lemma 5) and the Gaussian average of Lipschitz coefficients (Lemma 6), and a final assembly in Theorem 7.
Significance. If the main theorem were correct, the paper would offer a genuinely useful contribution: it would show that rank constraints control Gaussian complexity linearly in depth rather than exponentially, which is a non-obvious and potentially important improvement for the analysis of low-rank-trained deep networks. The use of Maurer's chain rule to sidestep the multiplicative rank accumulation is conceptually appealing, and the diameter bound in Lemma 5 is a solid and reusable ingredient. The paper is also clearly written in its high-level structure, with explicit comparisons in Table 1 and a self-contained derivation from standard results (Maurer's chain rule, Talagrand's contraction, Bartlett–Mendelson Gaussian complexity bounds). However, the central proof currently contains a serious flaw: Lemma 6, which is load-bearing for Theorem 7, is incorrect as stated. The significance of the contribution is therefore conditional on a successful repair of the proof.
major comments (4)
- [Section 4.3, Lemma 6] The proof of Lemma 6 is invalid as written. The quantity R(F, F_{ℓ-1}) is defined as a supremum over the class F = {W_ℓ : ||W_ℓ||_2 ≤ B}, but the proof replaces sup_{||W_ℓ||_2≤B} ⟨W_ℓ, Γ(Z−Z')⟩ with ||W_ℓ||_F ||Γ(Z−Z')||_F for a fixed matrix W_ℓ, without any justification. The correct value of this supremum is B ||Γ(Z−Z')||_* (the nuclear norm), which is not bounded by ||W_ℓ||_F ||Γ(Z−Z')||_F for a fixed W_ℓ. A concrete counterexample is B=1, A = Γ(Z−Z') = diag(1,1), and W_ℓ = diag(1,0): the left-hand side equals 2 while the right-hand side equals √2. Since R(F, Y) appears in the chain rule (Theorem 3) and in the recursion of Section 4.5, Lemma 6 is load-bearing and its failure invalidates the proof of Theorem 7 as it currently stands.
- [Theorem 7 and Section 4.5] The theorem statement and its proof conflate individual weight matrices with the function class. Eq. (4) is written in terms of ||W_i||_F and ||W_i||_2 for a particular network, but Gaussian complexity is a supremum over all f_W in F_L, so a valid upper bound must depend only on the class parameters B_i, r_i, h_i, and the data. In the recursion of Section 4.5, the chain rule's first term requires the Lipschitz constant of the class of linear maps, which is sup_{||W||_2≤B_i} ||W||_2 = B_i, not ||W_i||_2 for an arbitrary member. This is the same class-versus-instance conflation seen in Lemma 6 and is a separate obstruction to the correctness of the stated bound.
- [Eq. (4) and the simplified bound] The definition of κ_i in Eq. (4) is not consistent with the lemmas it is supposed to combine. Combining Lemma 5 with any corrected version of Lemma 6 would produce, up to universal constants, a term of the form √(r_{i-1} r_i) (∏_{j=1}^{i} ||W_j||_2) √h_i, rather than √(r_i κ_i) ||W_i||_F √h_i with κ_i ∝ ∏_{j≠i} ||W_j||_2. Moreover, the simplified bound O(C_1^L ∏_{i=1}^L B_i L r √h / m) does not follow algebraically from Eq. (4) with the printed expression for κ_i, even if the Frobenius norms are replaced by √r_i B_i. The authors should state a fully explicit final bound and verify the algebra that leads to the claimed O(L r √h) scaling.
- [Appendix, Lemma 10] Lemma 10 in the appendix, which is used by Lemma 4, contains the same error: the proof bounds sup_W ⟨W, ΓX⟩ by ||W||_F ||ΓX||_F for a fixed W, but for the class {||W||_2 ≤ B} the supremum is B||ΓX||_*, and if a rank constraint is imposed it is B times the sum of the r largest singular values of ΓX. This is not bounded by the fixed-matrix Frobenius expression used in the proof. The shallow-network bound should be rederived using the correct class-level supremum, and the deep linear bound of Lemma 4 must be revisited accordingly.
minor comments (4)
- [Throughout] The manuscript contains numerous typos and OCR-style artifacts (e.g., 'F ell−1' in Lemma 6, 'radicaltp' in Lemma 5's proof, and 'd' in the definition of κ_i). A careful proofreading pass is needed.
- [Table 1] Table 1 reports the denominator √m for the proposed bound, while Theorem 7's simplified bound has m in the denominator (and the abstract also writes √h/m). These should be reconciled, since the normalized Gaussian complexity scales as 1/m.
- [Theorem 3] Maurer's chain rule is stated for finite Y and finite F, but the function classes considered in the paper are infinite. The authors should justify the extension to compact or otherwise well-behaved infinite classes, or state the necessary approximation argument.
- [Section 5, Theorem 8] Theorem 8 is presented as 'Theorem 4 of Golowich et al. [14]' but includes a log^{3/2}(m) term and a constant cℓ; please verify the exact statement and ensure the reference is accurate.
Circularity Check
No significant circularity: the main derivation is self-contained against external tools (Maurer's chain rule, Talagrand contraction, Bartlett-Mendelson bounds), and self-citations are motivational only.
full rationale
The paper's central bound in Theorem 7 is obtained by applying Maurer's Gaussian complexity chain rule (Theorem 3), Talagrand's contraction lemma, and standard Gaussian complexity generalization bounds (Theorem 1). These are stated as external results and are not derived from the paper's own conclusions. The rank and spectral-norm function class in equation (1) is defined directly, and the diameter bound (Lemma 5) and Lipschitz-average bound (Lemma 6) are proved from assumptions rather than assumed in a form equivalent to the theorem. The only self-citations (e.g., refs. [12], [13], [15], [23]) motivate the relevance of low-rank structure and neural collapse; none of them supplies the proof of Theorem 7 or is invoked as an unverified premise that makes the result forced. The apparent issue in Lemma 6, where the sup over the spectral-norm ball is replaced by a Frobenius-norm term with a fixed W_l, is a potential correctness gap in the proof, not a circularity: it does not define the desired bound into the inputs. No parameter is fitted to data, no prediction is equivalent by construction to a fitted quantity, and no known result is merely renamed. I therefore find no significant circularity.
Assumptions & free parameters
assumptions (5)
- standard math Maurer's Gaussian complexity chain rule holds with universal constants C1 and C2, and the third term can be dropped because the zero function is in the class.
- standard math The loss is L-Lipschitz and the vector-valued Gaussian complexity generalization bound (Theorem 2) holds.
- domain assumption The activation function is 1-Lipschitz and piecewise linear, so differences of activations can be represented by diagonal matrices with entries bounded by 1.
- domain assumption Data points are bounded, max_i ||x_i|| ≤ R, and the class FL contains the zero network.
- domain assumption Rank and spectral norm bounds are known a priori: rank(W_i) ≤ r_i and ||W_i||2 ≤ B_i.
Cite this review
Pith. "Pith review of On Generalization Bounds for Neural Networks with Low Rank Layers." pith.science (2026). https://pith.science/paper/P6RTX77H
@misc{pith2026241113733,
author = {Pith},
title = {Pith review of: On Generalization Bounds for Neural Networks with Low Rank Layers},
year = {2026},
howpublished = {\url{https://pith.science/paper/P6RTX77H}},
note = {Machine review of arXiv:2411.13733}
}
read the original abstract
While previous optimization results have suggested that deep neural networks tend to favour low-rank weight matrices, the implications of this inductive bias on generalization bounds remain underexplored. In this paper, we apply Maurer's chain rule for Gaussian complexity to analyze how low-rank layers in deep networks can prevent the accumulation of rank and dimensionality factors that typically multiply across layers. This approach yields generalization bounds for rank and spectral norm constrained networks. We compare our results to prior generalization bounds for deep networks, highlighting how deep networks with low-rank layers can achieve better generalization than those with full-rank layers. Additionally, we discuss how this framework provides new perspectives on the generalization capabilities of deep networks exhibiting neural collapse.
Forward citations
Cited by 1 Pith paper
-
Parameter Symmetry Potentially Unifies Deep Learning Theory
This position paper argues that parameter symmetry breaking and restoration unify three hierarchies in deep learning: learning dynamics, model complexity, and representation formation.
Reference graph
Works this paper leans on
-
[1]
A chain rule for the expected suprema of g aussian processes
Andreas Maurer. A chain rule for the expected suprema of g aussian processes. Theoretical Computer Science, 650:109–122, 2016. 10
work page 2016
-
[2]
De ep residual learning for image recogni- tion
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. De ep residual learning for image recogni- tion. In 2016 IEEE Conference on Computer Vision and Pattern Recogni tion (CVPR), pages 770–778, 2016. doi: 10.1109/CVPR.2016.90
-
[3]
Very deep convolut ional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolut ional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014
arXiv 2014
-
[4]
Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszkor eit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you ne ed. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. ...
work page 2017
-
[5]
Language models are few-shot learners
T om Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry , Amanda Askell, S andhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T om Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin , Scot...
work page 1901
-
[6]
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, T imothy Lillicrap, Madeleine Leach, Ko- ray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Master ing...
-
[7]
Mark Chen, Jerry T worek, Heewoo Jun, Qiming Yuan, Henriq ue Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Broc kman, Alex Ray , Raul Puri, Gretchen Krueger, Michael Petrov , Heidy Khlaaf, Girish Sastry , Pame la Mishkin, Brooke Chan, Scott Gray , Nick Ryder, Mikhail Pavlov , Alethea Power, Lukasz Kaiser, M oham...
work page 2021
-
[8]
Benign overfitting in linear regression
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexan der Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063–30070, 2020
work page 2020
Show all 30 references
-
[9]
Harmless interpola- tion of noisy data in regression
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subraman ian, and Anant Sahai. Harmless interpola- tion of noisy data in regression. IEEE Journal on Selected Areas in Information Theory , 1(1):67–83, 2020
2020
-
[10]
No rm-based capacity control in neural net- works
Behnam Neyshabur, Ryota T omioka, and Nathan Srebro. No rm-based capacity control in neural net- works. In Conference on learning theory , pages 1376–1401. PMLR, 2015
2015
-
[11]
Advances in neural information processing systems , 30, 2017
Peter L Bartlett, Dylan J Foster, and Matus J T elgarsky.Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017
2017
-
[12]
Com plexity control by gradient descent in deep networks
T omaso Poggio, Qianli Liao, and Andrzej Banburski. Com plexity control by gradient descent in deep networks. Nature communications, 11(1):1027, 2020
2020
-
[13]
Dynamics in deep classifiers trained with the square loss: Normalization, lo w rank, neural collapse, and generalization bounds
Mengjia Xu, Akshay Rangamani, Qianli Liao, T omer Galan ti, and T omaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, lo w rank, neural collapse, and generalization bounds. Research, 6:0024, 2023. 11
2023
-
[14]
Siz e-independent sample complexity of neural networks
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Siz e-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297–299. PMLR, 2018
2018
-
[15]
T omer Galanti, Mengjia Xu, Liane Galanti, and T omaso A. Poggio. Norm-based generaliza- tion bounds for sparse neural networks. In Neural Information Processing Systems , 2023. URL https://api.semanticscholar.org/CorpusID:268042263
2023
-
[16]
Characterizing the implicit bias of regularized sgd in rank minimization
T omer Galanti, Zachary S Siegel, Aparna Gupte, and T oma so Poggio. Characterizing the implicit bias of regularized sgd in rank minimization. CoRR, abs/2206.05794 v6 , 2022
2022 arXiv
-
[17]
The low-rank simplicity bias in deep networks
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Che ung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. T ransactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=bCiNWDmlY2
2023
-
[18]
Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro
Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. 2018 Information Theory and Applications Workshop (ITA), pages 1–10, 2017. URL https://api.semanticscholar.org/CorpusID:3909231
2018
-
[19]
Directional convergence and alignment in deep learning
Ziwei Ji and Matus T elgarsky. Directional convergence and alignment in deep learning. ArXiv, abs/2006.06657, 2020. URL https://api.semanticscholar.org/CorpusID:219573767
2006 arXiv
-
[20]
Roberts, and Ethan Dyer
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradien t descent happens in a tiny subspace. ArXiv, abs/1812.04754, 2018. URL https://api.semanticscholar.org/CorpusID:54480858
2018 arXiv
-
[21]
Implicit regul arization towards rank minimiza- tion in relu networks
Nadav Timor, Gal V ardi, and Ohad Shamir. Implicit regul arization towards rank minimiza- tion in relu networks. In International Conference on Algorithmic Learning Theory , 2022. URL https://api.semanticscholar.org/CorpusID:246430285
2022
-
[22]
Prevalence of neural collapse during the terminal phase of deep learning training
V ardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652–24663, 2020
2020
-
[23]
Akshay Rangamani, Marius Lindegaard, T omer Galanti, a nd T omaso A. Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning ,
-
[24]
Rademacher and g aussian complexities: Risk bounds and structural results
Peter L Bartlett and Shahar Mendelson. Rademacher and g aussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463–482, 2002
2002
-
[25]
A chain rule for the expec ted suprema of bernoulli processes
Yifeng Chu and Maxim Raginsky. A chain rule for the expec ted suprema of bernoulli processes. arXiv preprint arXiv:2304.14474, 2023
2023 arXiv
-
[26]
Understanding machine learning: From theory to algorithms
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014
2014
-
[27]
Foundations of machine learning
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalk ar. Foundations of machine learning . MIT press, 2018
2018
-
[28]
A vector-contraction inequality for r ademacher complexities
Andreas Maurer. A vector-contraction inequality for r ademacher complexities. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 3–17. Springer, 2016
2016
-
[29]
A pac-bayesian approach to spectrally- normalized margin bounds for neural networks
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Sr ebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. In 6th International Conference on Learning Representa- tions, ICLR 2018, V ancouver , BC, Canada, April 30 - May 3, 201 8, Conferen...
2018
-
[2023]
URL https://api.semanticscholar.org/CorpusID:259289154
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.