On the inductive bias of infinite-depth ResNets and the bottleneck rank

Enric Boix-Adsera

REVIEW 3 major objections 4 minor 3 cited by

On the inductive bias of infinite-depth ResNets and the bottleneck rank

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper derives exact minimum-weight formulas for deep linear ResNets, proving their inductive bias interpolates between nuclear-norm and rank minimization and extends to low bottleneck rank in nonlinear ResNets.

desk verdict The linear half is a really solid, new result; the nonlinear upper bound has a load-bearing scaling bug that looks locally repairable but must be fixed before the bottleneck-rank claim can stand. read the letter →

arxiv 2501.19149 v1 pith:35LL3CRO submitted 2025-01-31 cs.LG cs.AIstat.ML

Enric Boix-Adsera This is my paper · ORCID

classification cs.LGcs.AIstat.ML MSC 68T0715A18

keywords inductivebiasResNetsbottleneckranknuclearnormminimizationweightdecaysingularvaluesdeeplinearnetworks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that residual networks, despite skip connections that make the identity map almost free, still prefer low-rank solutions. For deep linear ResNets it derives exact formulas for the minimum squared-Frobenius-norm cost of representing a linear map $A$: the cost splits as a sum over the singular values of $A$, and in the infinite-depth limit it reduces to the nuclear norm when the weight-decay parameter $\lambda$ is large and to the rank when $\lambda$ is small. The same mechanism is then transferred to nonlinear ReLU ResNets, where the normalized minimum cost is sandwiched between the Jacobian rank and the bottleneck rank as $\lambda\to 0$. The conclusion is that weight decay, rather than being a generic regularizer, selects between two classical low-rank inductive biases, and bottleneck rank remains a useful description of ResNet behavior.

What carries the argument

The load-bearing machinery is the cost functional $c_{L,n,\lambda}(A)$, the minimum sum of squared Frobenius norms of all weights that represent $A$, with $\lambda$ balancing embedding and unembedding cost against residual-block cost. The theorem is carried by a singular-value reduction: the $n$-dimensional problem splits into $n$ one-dimensional problems, one per singular value $\sigma_i(A)$. The upper bound is an explicit SVD-based construction of the weights; the lower bound uses the Gel'fand-Naimark product inequality for singular values, converted through a majorization lemma (Lemma 2.5) into a per-singular-value cost inequality. In the nonlinear case, the same cost functional is applied to the Jacobian $Jg(x)$ for the lower bound, and the upper bound is built by representing the factorization $g=h_2\circ h_1$ with two cheap subnetworks while the middle residual layers implement a large scaling $\tau x$ of the $k$-dimensional intermediate variable, the residual analogue of the identity bottleneck in fully-connected networks.

What would settle it

Compute the output of the constructed network in the upper bound of Theorem 3.3 with the printed scales $\alpha=\beta=1/\sqrt{\lambda}$, $\tau=1/\lambda$, and $W_u=\sqrt{\lambda}\,[I,0]$. Starting from embedded input $\sqrt{\lambda}x$, the first block, the intermediate $\tau$-scaling, the last block, and the unembedding produce $\lambda\,h_2(h_1(x)/\lambda)$ rather than $h_2(h_1(x))$, with the last block evaluated outside its stated domain; re-running the calculation with $\beta=\sqrt{\lambda}$ (or another scaling) and checking whether equality holds for all $x\in\Omega$ settles whether the quoted cost bound is valid.

Watch

Extended reading notes

Core claim

The central discovery is that the minimum-weight cost of a deep linear residual network is additive over singular values. The paper proves (Theorem 2.1) that for depth-1 blocks $$$c^{{1\text{-lin}}$}_{L,n,\$\lambda$}(A)=\sum_i\min_{\$\alpha$\ge 0}\left(\frac{\sigma_i(A)}{(1+\$\alpha$/L)^L}+\$\lambda$\$alpha^{2}$\right),$$ and for depth-2 blocks $$$c^{{2\text{-lin}}$}_{L,n,\$\lambda$}(A)=\sum_{\sigma_i(A)\le\$\lambda$}\sigma_i(A)+\sum_{\sigma_i(A)>\$\lambda$}\$\lambda$\left((L+1)(\sigma_i(A)/\$\lambda$)^{1/(L+1)}-L\right).$$ In the infinite-depth limit this cost converges to the nuclear norm $\|A\|_*$ as $\lambda\to\infty$, and after normalization by $\lambda(\log(1/\lambda))^2$ (depth-1) or $\lambda\log(1/\lambda)$ (depth-2) it converges to $\mathrm{rank}(A)$ as $\lambda\to 0$ (Corollary 2.3). For nonlinear ReLU ResNets the paper extends this to a sandwich theorem (Theorem 3.3): as $\lambda\to0$ the infinite-depth cost is at least the Jacobian rank and at most the bottleneck rank of the target finite piecewise-linear function, so with suitable hyperparameters the network is biased toward functions $g=h_2\circ h_1$ whose intermediate dimension is small.

Load-bearing premise

The nonlinear claim stands on the assumption that the explicitly constructed weights in the upper bound of Theorem 3.3 really compose to $h_2(h_1(x))$ for every input in the domain; if the printed scales do not compose correctly, that bound, and with it the claimed bottleneck-rank bias for nonlinear ResNets, is not established by the present proof.

Editorial extensions

If this is right

For a linear ResNet trained with weight decay, small $\lambda$ makes the minimum-cost fit to the data a minimum-rank linear map, so the learned transformation is explicitly driven to low rank.
For large $\lambda$ the same architecture minimizes nuclear norm, and $\lambda$ therefore acts as a tunable interpolation parameter between the two classical low-rank regularizers.
For nonlinear ReLU ResNets, the same weight-decay regime drives solutions toward functions that factor through a low-dimensional intermediate representation, preserving the bottleneck-rank bias despite skip connections.
The formulas give a sharp prediction for how the minimum cost scales with layer count $L$ and weight-decay strength $\lambda$, allowing the bias to be measured or tested in trained networks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: because the linear cost formula is exact, it could be inverted into a regularizer that targets a prescribed rank or nuclear-norm budget rather than tuning $\lambda$ heuristically.
Beyond the paper: the paper sends $L\to\infty$ before $\lambda\to0$; a testable question is whether finite networks with $L$ growing like $\log(1/\lambda)$ already display the rank bias, or whether the limits fail to commute.
Beyond the paper: a direct composition check of the printed upper-bound construction suggests the stated scales may not produce $h_2\circ h_1$; whether a small adjustment such as $\beta=\sqrt{\lambda}$ restores the equality is the immediate test of the nonlinear claim.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know before you read this one. The linear half is a genuine and, as far as I can tell, correct result: an explicit per-singular-value formula for the min-norm cost of deep linear ResNets with embedding/unembedding, interpolating between nuclear norm and rank as the regularizer ratio varies. That part is solid, and the proof is a clean SVD construction plus a Gelfand–Naimark-based lower bound. The nonlinear half, which is what the abstract and the title really point at, does not hold as printed.

The problem is in Theorem 3.3's upper-bound construction. The paper claims that with α = β = 1/√λ and τ = 1/λ, the network computes h2(h1(x)). Tracing the composition gives λ·h2(h1(x)/λ), not h2(h1(x)); the last block also receives input outside its stated domain. Changing β to √λ repairs the algebra, and the depth-2 construction is then plausible. But as submitted, the verification fails, and the claimed cost independence from α, β, λ is delegated to an unstated 'slight modification' of [ABMM16]. The depth-1 appendix has a further red flag: it postulates disjoint subsets S_i of [n] each of size n, which is impossible for more than one set. So the nonlinear claim is a plausible but unverified story.

If you are working on inductive bias or on the theory of ResNets, the linear section alone is worth your time, and it is a real contribution. The nonlinear result, if repaired, would rescue Jacot's bottleneck-rank picture for ResNets, and I would not be surprised if the local fix goes through. But the referee should ask for a rewritten Section 3 with an explicit simulation lemma and a correct scale calculation, not accept the current version.

My recommendation: send it to review, but treat the nonlinear theorem as unproven until the scaling is fixed. The paper deserves a serious referee; it just does not deserve acceptance as is.

Referee Report

3 major / 4 minor

Summary. The paper studies the inductive bias of deep ResNets with embedding and unembedding layers. For deep linear ResNets it derives explicit minimum-norm cost formulas (Theorems 2.1-2.3) and shows that, in the infinite-depth limit, the cost interpolates between the nuclear norm as λ→∞ and the rank as λ→0. The paper then claims a nonlinear analogue: as λ→0, the normalized minimum cost of a deep ReLU ResNet representing a finite piecewise-linear function is sandwiched between its Jacobian rank and its bottleneck rank (Theorem 3.3).

Significance. The linear section is a genuine contribution: Theorem 2.1 gives an exact, additive-over-singular-values characterization of the minimum squared-Frobenius-norm weights, with a constructive SVD upper bound and a lower bound via Gelfand-Naimark majorization; no constants are fitted and the asymptotic interpolation in Corollary 2.3 is clean. If the nonlinear claim can be made rigorous, it would substantially extend Jacot's bottleneck-rank theory to ResNets despite the presence of skip connections. However, the current proof of the nonlinear upper bound contains a concrete scaling error and relies on an unproved norm-independence assertion, so the paper cannot be accepted as submitted. The linear results alone are not the paper's headline, and the nonlinear construction appears locally repairable, so I do not recommend rejection.

major comments (3)

[Section 3, proof of Theorem 3.3 (depth-2 upper bound), 'Computing the total cost'] The verification at the end of the depth-2 upper-bound proof is incorrect as printed. With α=β=1/√λ, τ=1/λ, We=√λ[I_din;0], and Wu=√λ[I_dout,0], the embedded input is √λx; the first residual stack outputs h1(x)/α=√λh1(x); the middle block multiplies by τ=1/λ, giving h1(x)/√λ; the last residual stack, applied with β=1/√λ, outputs h2(β·h1(x)/√λ)/β=√λh2(h1(x)/λ); and the unembedding layer then outputs λh2(h1(x)/λ), not g(x)=h2(h1(x)). The last block also receives h1(x)/√λ, which lies outside its stated domain (h1(Ω)/β)=√λh1(Ω). Setting β=√λ repairs the composition, but as submitted the construction does not verify Theorem 3.3.
[Section 3, proof of Theorem 3.3, 'First and last residual layers'] The claim that the simulation of h1(αx)/α and h2(βx)/β can be performed with Frobenius-norm cost bounded by a constant C independent of α, β, and λ is not proven. It is only asserted via 'a slight modification of Theorem 2.1 of [ABMM16]', but that theorem provides existence of ReLU representations, not norm bounds under input and output rescaling. Moreover, a residual block contains a fixed skip connection and is not homogeneous under simultaneous scaling of all weights, so the assertion is not a formal consequence of ReLU homogeneity. This point is load-bearing: if the first and last simulation costs grow with 1/λ, the displayed total-cost bound would not scale as λ log(1/λ), and the upper bound (3.2) would not follow. The authors should supply a complete construction or lemma with explicit norm bounds for the rescaled simulations.
[Appendix A] The depth-1 upper-bound proof begins by choosing disjoint subsets S0,...,S_{L1}⊆[n] with |S_i|=n. For L1≥1 this is impossible in an n-element set. Since the depth-1 statement in Theorem 3.3 depends on this appendix, the depth-1 upper bound is not established as written. The construction needs to be corrected, for example by partitioning [n] into disjoint groups with the required roles, and the cost estimate must then be re-derived.

minor comments (4)

[Theorem 3.3 and Eq. (3.1)] The notation 'rankJ(x;Ω)' is used, but the definition in Section 3.1 is 'rankJ(g;Ω)'.
[Definition 3.2 and proof of Theorem 3.3] In Definition 3.2 the map h2 is said to be h2:R^k→R^{n1}, but in the proof and application it must map to R^{dout}; these codomains should be made consistent.
[Appendix A, final display] The network in the final display is denoted f1-lin(x;θ), but it should be f1-nonlin(x;θ).
[Corollary 2.3, small-λ proof] The sentence 'for any σ' in the small-λ analysis should explicitly distinguish σ=0 from σ>0; the current proof does this in the displayed formula but not in the surrounding text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the linear cost formula is a self-contained optimization result and the nonlinear bound is a constructive estimate, not a fitted prediction.

full rationale

The paper's central result for linear ResNets is derived from a constrained optimization over network weights, using an external singular-value inequality (Gel'fand–Naimark, as cited via Bhatia and Li–Mathias) and a convex majorization argument. No parameter is fitted to data, and the claimed limits are compared to independent mathematical objects (nuclear norm, rank). The nonlinear Theorem 3.3 upper bound is a constructive cost estimate built from a bottleneck-rank decomposition, not a quantity predicted from fitted values; the lower bound uses the Jacobian of the represented function. Reliance on [ABMM16] for expressivity of ReLU networks is an external, non-self citation. The manuscript contains possible correctness concerns in the explicit upper-bound construction (the printed scales do not verify g(x) = h2(h1(x))) and an impossible disjoint-set condition in Appendix A, but these are correctness issues rather than instances where a prediction reduces by definition to its inputs. No load-bearing self-citation chain or definitional equivalence was found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The linear results are a genuine constrained-optimization computation with no fitted constants; the hand-chosen objects are the hyperparameter λ (whose extremes define the interpolation endpoints) and the scale parameters α, β, τ, m in the nonlinear upper-bound construction, one of which (β) is inconsistent as printed. Axioms are standard inequality facts plus two modeling choices: the critical L-scaling of the cost, and the identification of inductive bias with min-weight cost, which the paper itself flags as approximate in footnote 1. No invented entities are postulated.

free parameters (5)

Regularizer ratio λ = analyzed in the limits λ→0 and λ→∞
Hyperparameter (not data-fitted) weighing embedding/unembedding cost against residual-block cost; its extremes define the nuclear-norm and rank endpoints of the interpolation.
Middle-layer scale τ = τ = 1/λ in Theorem 3.3 construction
Chosen by hand in the nonlinear upper-bound construction; the rate λlog(1/λ) of the cost bound comes from this choice, since the middle cost is approximately λk log τ.
First-block scale α = α = 1/√λ
Chosen so that α times the embedding scale √λ equals 1, making the first block evaluate h1 at the unembedded input; part of the repair-sensitive scaling trio.
Last-block scale β = printed as 1/√λ; the composition requires β = √λ
Needed to satisfy √λ/β = 1 and βτ/α = 1; with the printed value β = 1/√λ the network computes λ·h2(h1(x)/λ) rather than h2(h1(x)).
Simulation width factor m = m = ⌊L/log(1/λ)⌋
Chosen in Appendix A (depth-1 upper bound) so that the h1/h2 simulation cost λC(1+L/m) is negligible after normalization by λ(log(1/λ))².

assumptions (5)

standard math Gelfand-Naimark multiplicative singular-value inequality (Proposition 2.4)
Invoked in the proof of Lemma 2.5 to lower-bound the embedding, unembedding, and residual costs by ratios of singular values of partial products; the paper cites [GN50], [LM99], and [Bha96] Theorem III.4.5.
standard math Weak log-majorization implies the convex rearranged sum inequality (Fact 3.C.1.b of Marshall, Olkin, Arnold)
Bridges the submajorization of log singular values to the sum of convex g(σ) in Lemma 2.5.
ad hoc to paper Finite piecewise-linear functions can be simulated by ReLU ResNets of finite depth and width with cost independent of input/output rescaling, via 'a slight modification of Theorem 2.1 of [ABMM16]'
Load-bearing for the upper bound of Theorem 3.3 and for Appendix A; the claimed independence from α, β, λ is not proven and is not immediate because the output scaling 1/β would naively multiply the final weight norms by 1/β².
domain assumption Inductive bias is identified with the minimum squared-Frobenius-norm cost of representing a function
Defines cost(f) in Section 1; the paper's own footnote 1 says this is an approximation, since optimization dynamics may not reach min-cost solutions, so the practical 'inductive bias' claim inherits this caveat.
domain assumption Critical scaling of residual weights by λL (depth-1) or λ/2 (depth-2)
The L-scaling is chosen so the infinite-depth limit is nontrivial; with a different scaling the limits would degenerate, so the interpolation result is conditional on this normalization.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the inductive bias of infinite-depth ResNets and the bottleneck rank." pith.science (2026). https://pith.science/paper/35LL3CRO

@misc{pith2026250119149,
  author       = {Pith},
  title        = {Pith review of: On the inductive bias of infinite-depth ResNets and the bottleneck rank},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/35LL3CRO}},
  note         = {Machine review of arXiv:2501.19149}
}

read the original abstract

We compute the minimum-norm weights of a deep linear ResNet, and find that the inductive bias of this architecture lies between minimizing nuclear norm and rank. This implies that, with appropriate hyperparameters, deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks
math.FA 2026-06 unverdicted novelty 7.0 of 10

Develops general framework for representation costs of parametric models, proving that depth-L ReLU networks induce p-normable quasi-Banach spaces with p=2/L.
Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
cs.LG 2025-05 conditional novelty 7.0 of 10

Neural collapse is globally optimal in deep regularized ResNets and transformers, with the approximation improving as depth grows.
Differentially Private Natural Gradient Descent
cs.LG 2026-07 conditional novelty 6.0 of 10

DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigen...

Reference graph

Works this paper leans on

19 extracted references · 14 canonical work pages · cited by 3 Pith papers

[1]

Understanding deep neural networks with rectified linear units

Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv:1611.01491 , 2016

arXiv 2016
[2]

Implicit regularization in deep matrix factorization

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019

work page 2019
[3]

Matrix Analysis , volume 169

Rajendra Bhatia. Matrix Analysis , volume 169. Springer Science & Business Media, 1996

work page 1996
[4]

Neural ordinary differential equations

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018

2018
[5]

A mathematical perspective on transformers

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794 , 2023

arXiv 2023
[6]

Implicit bias of gradient descent on linear convolutional networks

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems , 31, 2018

work page 2018
[7]

The relation between the unitary representations of the complex unimodular group and its unitary subgroup

Izrail Moiseevich Gel'fand and Mark Aronovich Naimark. The relation between the unitary representations of the complex unimodular group and its unitary subgroup. Izvestiya Rossiiskoi Akademii Nauk. Seriya Matematicheskaya , 14(3):239--260, 1950

work page 1950
[8]

Implicit regularization in matrix factorization

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems , 30, 2017

work page 2017

Show all 19 references

[9]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778, 2016

2016
[10]

Implicit bias of large depth networks: a notion of rank for nonlinear functions

Arthur Jacot. Implicit bias of large depth networks: a notion of rank for nonlinear functions. In The Eleventh International Conference on Learning Representations , 2022

2022
[11]

Bottleneck structure in learned features: Low-dimension vs regularity tradeoff

Arthur Jacot. Bottleneck structure in learned features: Low-dimension vs regularity tradeoff. Advances in Neural Information Processing Systems , 36:23607--23629, 2023

2023
[12]

Hamiltonian mechanics of feature learning: Bottleneck structure in leaky resnets

Arthur Jacot and Alexandre Kaiser. Hamiltonian mechanics of feature learning: Bottleneck structure in leaky resnets. arXiv preprint arXiv:2405.17573 , 2024

2024
[13]

Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. arXiv preprint arXiv:2012.09839 , 2020

2012 arXiv
[14]

The lidskii-mirsky-wielandt theorem--additive and multiplicative versions

Chi-Kwong Li and Roy Mathias. The lidskii-mirsky-wielandt theorem--additive and multiplicative versions. Numerische Mathematik , 81:377--413, 1999

1999
[15]

Inequalities: Theory of majorization and its applications

Albert W Marshall, Ingram Olkin, and Barry C Arnold. Inequalities: Theory of majorization and its applications. Springer Series in Statistics , 2011

2011
[16]

Do ideas have shape? plato’s theory of forms as the continuous limit of artificial neural networks

Houman Owhadi. Do ideas have shape? plato’s theory of forms as the continuous limit of artificial neural networks. arXiv preprint arXiv:2008.03920 , pages 2--3, 2020

2008 arXiv
[17]

Implicit regularization in deep learning may not be explainable by norms

Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems , 33:21174--21187, 2020

2020
[18]

Gomez, ukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , page 6000–6010, 2017

2017
[19]

Which frequencies do cnns need? emergent bottleneck structure in feature learning

Yuxiao Wen and Arthur Jacot. Which frequencies do cnns need? emergent bottleneck structure in feature learning. arXiv preprint arXiv:2402.08010 , 2024

2024 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Understanding deep neural networks with rectified linear units

Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding deep neural networks with rectified linear units. arXiv preprint arXiv:1611.01491 , 2016

arXiv 2016

[2] [2]

Implicit regularization in deep matrix factorization

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019

work page 2019

[3] [3]

Matrix Analysis , volume 169

Rajendra Bhatia. Matrix Analysis , volume 169. Springer Science & Business Media, 1996

work page 1996

[4] [4]

Neural ordinary differential equations

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018

2018

[5] [5]

A mathematical perspective on transformers

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794 , 2023

arXiv 2023

[6] [6]

Implicit bias of gradient descent on linear convolutional networks

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems , 31, 2018

work page 2018

[7] [7]

The relation between the unitary representations of the complex unimodular group and its unitary subgroup

Izrail Moiseevich Gel'fand and Mark Aronovich Naimark. The relation between the unitary representations of the complex unimodular group and its unitary subgroup. Izvestiya Rossiiskoi Akademii Nauk. Seriya Matematicheskaya , 14(3):239--260, 1950

work page 1950

[8] [8]

Implicit regularization in matrix factorization

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems , 30, 2017

work page 2017