Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

Qi Zhou; Xiao-Song Yang; Xuan Zhou

Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

T0 review · 0 major / 8 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Residual nets with one-neuron residual branches need block width exactly max of input and output dimension for L^p universal approximation, and a slightly larger but explicit width for uniform approximation.

desk verdict Exact min block width max{dx,dy} for L^p under inner-width-one ResNets, with a tighter uniform upper bound and an activation-independent lower bound that holds for any inner width. read the letter →

arxiv 2607.04597 v2 pith:ENNCSJ4C submitted 2026-07-06 cs.LG stat.ML

Qi Zhou , Xuan Zhou , Xiao-Song Yang This is my paper · ORCID

classification cs.LGstat.ML MSC 41A4641A6341A6568T07

keywords minimumwidthresidualneuralnetworkscompactuniformapproximationuniversalpropertyblockinneroneReLU-likeactivations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Residual networks are compositions of residual blocks that add a small nonlinear branch to the identity. This paper asks how wide those blocks must be when every residual branch is restricted to a single neuron. For LeakyReLU, ReLU and a large family of ReLU-like activations, the answer for L^p approximation on compact domains is exact: the block width must be at least the larger of the input and output dimensions, and that same width is already enough. For uniform (sup-norm) approximation the same lower bound holds, while an explicit upper bound min{dx+dy, max{2dx+1, dy}} is proved by lifting the input into a higher-dimensional space, approximating diffeomorphisms by residual compositions, and projecting back. The lower bound is universal: no activation family and no larger inner width can succeed with block width smaller than max{dx, dy}. The results remove earlier dimensional restrictions and improve several known sufficient widths, showing that residual architecture can keep universal approximation even under the most severe restriction on the residual branches.

What carries the argument

Compositions of residual blocks of block width w and inner width 1 (denoted CRB). The paper shows that these compositions can compactly approximate all affine maps and all coordinate-wise piecewise-linear maps of the form fa, and, via single-coordinate diffeomorphisms and embedding width, all C^{2}-diffeomorphisms of R^w; the ResNet is then obtained by sandwiching such a composition between two affine maps.

What would settle it

Either exhibit a continuous map that cannot be uniformly approximated by any residual network of block width min{dx+dy, max{2dx+1, dy}} and inner width one (with the listed activations), or prove that some C^{2}-diffeomorphism of R^w cannot be compactly approximated by such residual compositions.

Watch

Extended reading notes

Core claim

When each residual branch has inner width one and the activations are LeakyReLU, ReLU or ReLU-like, the minimal block width for L^p universal approximation of maps from R^{dx} to R^{dy} on compact sets is exactly max{dx, dy}. For uniform approximation the same lower bound holds and the upper bound is min{dx+dy, max{2dx+1, dy}}. Independently of activation family and of inner width, block width strictly less than max{dx, dy} is never sufficient.

Load-bearing premise

The uniform upper bound rests on residual compositions of inner width one being able to approximate every C^{2}-diffeomorphism on compact sets; if that approximation step fails, the uniform width bound is unproved.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper settles a clean architectural question: for residual nets whose residual branches have inner width one, the minimum block width for L^p approximation on compact sets is exactly max{dx, dy} (LeakyReLU / ReLU / ReLU-like). The same lower bound holds for uniform approximation and, importantly, for any activation family and any inner width. The uniform upper bound they get is min{dx+dy, max{2dx+1, dy}}, which improves Tabuada et al. and removes the dx ≥ dy restriction that appeared in earlier ResNet/ODENet work.

What is new is the exact equality for the L^p case under the most restrictive residual branch, the extension of the activation class to a long list of ReLU-like maps, and the activation-independent lower bound that does not depend on inner width. The lower-bound argument (Theorem 4) is elementary and solid: when the ambient dimension is too small you either lose injectivity along a line or land in a proper affine subspace, so you cannot hit simple targets such as the squared norm or the moment curve. The L^p upper bound reduces to known MLP constructions once they show residual blocks of width one can approximate affine maps and coordinate-wise piecewise-linear maps; those reductions (Theorems 1–2) look carefully done.

The uniform upper bound is longer: it routes through embedding width, Palais extension, and approximation of C^{2}-diffeomorphisms by residual compositions. That chain leans on published ingredients (Teshima, Hwang, their own earlier lemmas) and each link is standard, so I do not see a break, but it is the softest part of the paper. The numerical self-intersection example for the 2-to-2 map is only illustrative; it does not carry the theorems.

Math and citations look careful; no free parameters, no circularity. This is for people who care about width theory of residual and neural-ODE architectures. I would send it to a serious referee and I would cite the exact L^p width and the lower bound myself.

Referee Report

0 major / 8 minor

Summary. The paper determines the minimum ambient (block) width of residual networks whose residual branches have inner width one, for universal approximation on compact domains. For LeakyReLU, ReLU and a large class of ReLU-like activations, the exact minimum block width for L^p approximation (1 ≤ p < ∞) is max{d_x, d_y} (Theorem 5). For uniform approximation the same lower bound holds and an upper bound min{d_x + d_y, max{2 d_x + 1, d_y}} is proved (Theorem 7). Independently of the activation family and of the inner width, block width strictly less than max{d_x, d_y} is shown to be insufficient in both topologies (Theorem 4). The arguments combine elementary rank/image-dimension lower bounds with residual-block approximations of affine maps and coordinate-wise piecewise-linear maps, plus an embedding-plus-diffeomorphism route for the uniform upper bound.

Significance. The work gives the first exact minimum-block-width result for residual networks under the most restrictive inner-width constraint, removes the dimensional restriction d_x ≥ d_y present in earlier ResNet/ODENet analyses, and improves the best previously published uniform upper bounds for a broad activation class. The activation-independent lower bound of Theorem 4 is elementary and sharp. The L^p characterization (Theorem 5) is clean and load-bearing; the uniform bounds, while not sharp, already improve Tabuada–Gharesifard and Aizawa et al. in several regimes. The proofs are self-contained once standard MLP and diffeomorphism-approximation lemmas are granted, and the numerical illustration that a width-3 ResNet can approximate a map obstructing width-3 MLPs is a useful geometric observation. These contributions are of clear interest to the approximation-theory community in deep learning.

minor comments (8)

The chain establishing Proposition 8 (D^{2} ≺ INN ≺ MLP_LeakyReLU ≺ residual compositions) is long and spans several external lemmas. A short roadmap paragraph at the start of §4.2, listing the three external ingredients and the two internal theorems used, would make the uniform upper bound easier to audit.
§1.1, last paragraph: “Then purpose of this paper” → “The purpose of this paper”.
Table 1 footnotes are dense; moving the activation-list footnote (3) into the caption or a short remark would improve readability.
Definition 10 (ReLU-like) and Proposition 4 are clear, but a one-line remark that the definition is uniform on every compact interval (not merely pointwise) would prevent a possible misreading.
In the proof of Theorem 6 the appeal to Palais’ extension theorem is correct, yet the text does not explicitly record that the ambient dimension Ω(n,m) always satisfies n ≤ Ω, so the hypothesis of that theorem is met. Adding this observation would close a small expository gap.
Figure 3 is informative; stating the precise training loss / max-norm error in the caption (already given in the text as < 3·10^{-5}) would make the figure self-contained.
Several matrix displays in Appendix B.1 use “*” for multiplication; standard juxtaposition or “·” would be more conventional.
The open gap between the uniform lower and upper bounds is correctly left as future work; a one-sentence explicit statement in §5 that “whether w_sup_min = max{d_x,d_y} remains open” would help readers.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: central min-width theorems rest on elementary constructions plus external approximation lemmas; one technical self-citation is non-load-bearing.

self citation load bearing [Proposition 5 (and its use in Theorem 2)]
"For any fixed k∈(0,1)∪(1,+∞), we have x+LeakyReLUk(x)=2∗LeakyReLU(1+k)/2(x). For any ε>0 and compact set K⊂R, the proof of Theorem 14 in [26] implies that there exists affine transformations {TWi,Bi}i∈{1,2,⋯,N} such that ||LeakyReLUa−TWN,BN∘LeakyReLU(1+k)/2∘⋯∘LeakyReLU(1+k)/2∘TW1,B1||sup,K<ε."

The only self-citation supplies a technical scalar approximation lemma used to lift LeakyReLU into residual-block compositions. It is not a uniqueness claim, does not define the target width, and is not required for the lower bound or the ReLU/ReLU-like cases; the central width equalities remain independently proved.

full rationale

The paper is a pure existence/approximation-theory work. Lower bounds (Theorem 4) are self-contained linear-algebra and measure arguments showing that block width < max{dx,dy} forces either non-injectivity or image contained in a hyperplane, independent of any citation. Upper bounds for Lp (Theorems 3/5) reduce known MLP constructions (Cai 2023) to residual blocks via the paper’s own Theorems 1–2 (affine maps by SVD/PLU + elementary residual updates; coordinate-wise piecewise-linear maps by direct residual constructions). Uniform upper bounds (Theorems 6/7) chain external results (Teshima et al. on diffeomorphism approximation by invertible nets; Hwang on single-coordinate maps by LeakyReLU MLPs) through the same Theorems 1–2 and Proposition 8. No parameters are fitted, no uniqueness theorem is imported to forbid alternatives, no ansatz is smuggled, and no quantity is redefined as its own prediction. The sole self-citation ([26], same authors) supplies only a technical scalar approximation of LeakyReLU by compositions of fixed-slope LeakyReLUs (used inside Proposition 5); that lemma is elementary and independently verifiable, so it does not force the width claims. Numerical illustration of a map from Hwang is likewise non-circular. Score 1 reflects only the minor self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is pure approximation theory. It relies only on standard measure-theoretic and differential-topology facts together with previously published approximation lemmas for MLPs and invertible networks; no free parameters are fitted and no new physical or mathematical entities are postulated.

assumptions (4)

standard math Singular-value and PLU factorizations of real matrices exist and can be used to reduce general affine maps to products of diagonal, triangular and permutation maps.
Invoked in the proof of Theorem 1 to approximate arbitrary affine transformations by residual blocks.
standard math Continuous maps on compact sets can be uniformly approximated by smooth maps (mollification).
Proposition 6, used to reduce continuous targets to smooth embeddings.
domain assumption Every C^{2}-diffeomorphism of R^w can be approximated by invertible neural networks built from single-coordinate transformations (Teshima et al., Hwang).
Proposition 8 chains this external result with residual-block approximations of LeakyReLU maps.
standard math For n 中 m a smooth embedding of the cube into R^m extends to a diffeomorphism of the ambient space (Palais).
Used in Theorem 6 to lift embeddings to residual compositions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One." pith.science (2026). https://pith.science/paper/ENNCSJ4C

@misc{pith2026260704597,
  author       = {Pith},
  title        = {Pith review of: Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENNCSJ4C}},
  note         = {Machine review of arXiv:2607.04597}
}

abstract

In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions $d_x$ and $d_y$, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve $L^p$ approximation $(1\leq p <+\infty)$ on any compact domain, we show that the exact minimum block width is $\max\{d_x,d_y\}$ when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width $\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}$ can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than $\max\{d_x, d_y\}$, both in the $L^p$ sense and the uniform sense, regardless of inner width.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 4 linked inside Pith

[1]

Discrete and Continuous Dynamical Systems - B 29(1):351–376

Aizawa Y, Kimura M, Matsui K (2024) Universal approximation properties for an odenet and a resnet: Mathematical analysis and numerical experiments. Discrete and Continuous Dynamical Systems - B 29(1):351–376. https://doi.org/10.3934/ dcdsb.2023099

2024
[2]

Springer, Cham, https://doi.org/10.1007/978-3-030-33143-6

Axler S (2020) Measure, Integration & Real Analysis, Graduate Texts in Mathe- matics, vol 282. Springer, Cham, https://doi.org/10.1007/978-3-030-33143-6

doi:10.1007/978-3-030-33143-6 2020
[3]

In: The Eleventh International Conference on Learning Representations

Cai Y (2023) Achieve the minimum width of neural networks for universal approx- imation. In: The Eleventh International Conference on Learning Representations

2023
[4]

In: Advances in Neural Information Processing Systems, pp 3134–3144

Dupont E, Doucet A, Teh YW (2019) Augmented neural odes. In: Advances in Neural Information Processing Systems, pp 3134–3144

2019
[5]

Com- munications in Mathematics and Statistics 5(1):1–11

E W (2017) A proposal on machine learning via dynamical systems. Com- munications in Mathematics and Statistics 5(1):1–11. https://doi.org/10.1007/ s40304-017-0103-z

2017
[6]

American Mathematical Society, Providence, RI, https: //doi.org/10.1090/gsm/019

Evans LC (2010) Partial Differential Equations, Graduate Studies in Mathe- matics, vol 19, 2nd edn. American Mathematical Society, Providence, RI, https: //doi.org/10.1090/gsm/019

doi:10.1090/gsm/019 2010
[7]

arXiv preprint arXiv:1710.11278 [stat.ML]

Hanin B, Sellke M (2017) Approximating continuous functions by ReLU nets of minimal width. arXiv preprint arXiv:1710.11278 [stat.ML]

arXiv 2017
[8]

In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recogni- tion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

2016

Show all 28 references

[9]

In: Proceedings of the IEEE International Conference on Computer Vision, pp 2961–2969

He K, Gkioxari G, Doll´ ar P, et al (2017) Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2961–2969

2017
[10]

In: Belgrave D, Zhang C, Lin H, et al (eds) Advances in Neural Infor- mation Processing Systems, vol 38

Hwang G (2025) Minimum width for deep, narrow mlp: A diffeomorphism approach. In: Belgrave D, Zhang C, Lin H, et al (eds) Advances in Neural Infor- mation Processing Systems, vol 38. Curran Associates, Inc., Red Hook, NY, pp 145986–146011

2025
[11]

arXiv preprint arXiv:1810.00393 [cs.LG]

Johnson J (2018) Deep, skinny neural networks are not universal approximators. arXiv preprint arXiv:1810.00393 [cs.LG]

2018 arXiv
[12]

In: Conference on learning theory, PMLR, pp 2306–2327

Kidger P, Lyons T (2020) Universal approximation with deep narrow networks. In: Conference on learning theory, PMLR, pp 2306–2327

2020
[13]

In: Kim B, Yue Y, Chaudhuri S, et al (eds) International Conference on Representation Learning, pp 3097–3130 32

Kim N, Min C, Park S (2024) Minimum width for universal approximation using relu networks on compact domain. In: Kim B, Yue Y, Chaudhuri S, et al (eds) International Conference on Representation Learning, pp 3097–3130 32

2024
[14]

In: Inter- national Conference on Learning Representations, San Diego, CA, USA, URL https://arxiv.org/abs/1412.6980

Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Inter- national Conference on Learning Representations, San Diego, CA, USA, URL https://arxiv.org/abs/1412.6980

2015 arXiv
[15]

Journal of the European Mathematical Society 25(5):1671–1709

Li Q, Lin T, Shen Z (2023) Deep learning via dynamical systems: An approxima- tion perspective. Journal of the European Mathematical Society 25(5):1671–1709. https://doi.org/10.4171/JEMS/1221

2023 doi
[16]

In: Advances in Neural Information Processing Systems, vol 31

Lin H, Jegelka S (2018) Resnet with one-neuron hidden layers is a universal approximator. In: Advances in Neural Information Processing Systems, vol 31. Curran Associates, Inc., Red Hook, NY, pp 6169–6178

2018
[17]

In: Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 235

Liu C, Liang E, Chen M (2024) Characterizing ResNet’s universal approxima- tion capability. In: Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, vol 235. PMLR, Vienna, Austria, pp 31477–31515, URL https://proceedin...

2024
[18]

Pearson Education, Harlow

Munkres J (2014) Topology. Pearson Education, Harlow

2014
[19]

Lyryx, Calgary, AB, Canada

Nicholson WK (2019) Linear Algebra with Applications. Lyryx, Calgary, AB, Canada

2019
[20]

Proceedings of the American Mathematical Society 11(2):274–277

Palais RS (1960) Extending diffeomorphisms. Proceedings of the American Mathematical Society 11(2):274–277. https://doi.org/10.2307/2032968

1960 doi
[21]

In: International Conference on Learning Representations

Park S, Yun C, Lee J, et al (2021) Minimum width for universal approximation. In: International Conference on Learning Representations. OpenReview.net, Virtual Event, Austria, URL https://openreview.net/forum?id=O-XJwyoIF-k

2021
[22]

arXiv preprint arXiv:2407.18384 [cs.LG]

Petersen P, Zech J (2024) Mathematical theory of deep learning. arXiv preprint arXiv:2407.18384 [cs.LG]

2024
[23]

arXiv preprint arXiv:2007.06007 [cs.LG]

Tabuada P, Gharesifard B (2020) Universal approximation power of deep residual neural networks via nonlinear control theory. arXiv preprint arXiv:2007.06007 [cs.LG]

2020 arXiv
[24]

In: Larochelle H, Ranzato M, Hadsell R, et al (eds) Advances in Neural Information Processing Systems, vol 33

Teshima T, Ishikawa I, Tojo K, et al (2020) Coupling-based invertible neural networks are universal diffeomorphism approximators. In: Larochelle H, Ranzato M, Hadsell R, et al (eds) Advances in Neural Information Processing Systems, vol 33. Curran Associates, Inc., Red Hook, N...

2020
[25]

Cambridge Mathematical Textbooks, Cambridge University Press, Cambridge

Williams D (1991) Probability with Martingales. Cambridge Mathematical Textbooks, Cambridge University Press, Cambridge

1991
[26]

arXiv preprint arXiv:2511.06837 [cs.LG] 33

Yang XS, Zhou Q, Zhou X (2025) Minimum width of deep narrow networks for universal approximation. arXiv preprint arXiv:2511.06837 [cs.LG] 33

2025
[27]

In: Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol

Zhang H, Gao X, Unterman J, et al (2020) Approximation capabilities of neural ODEs and invertible residual networks. In: Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol

2020
[28]

PMLR, Online, pp 11086–11095 34

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.

[1] [1]

Discrete and Continuous Dynamical Systems - B 29(1):351–376

Aizawa Y, Kimura M, Matsui K (2024) Universal approximation properties for an odenet and a resnet: Mathematical analysis and numerical experiments. Discrete and Continuous Dynamical Systems - B 29(1):351–376. https://doi.org/10.3934/ dcdsb.2023099

2024

[2] [2]

Springer, Cham, https://doi.org/10.1007/978-3-030-33143-6

Axler S (2020) Measure, Integration & Real Analysis, Graduate Texts in Mathe- matics, vol 282. Springer, Cham, https://doi.org/10.1007/978-3-030-33143-6

doi:10.1007/978-3-030-33143-6 2020

[3] [3]

In: The Eleventh International Conference on Learning Representations

Cai Y (2023) Achieve the minimum width of neural networks for universal approx- imation. In: The Eleventh International Conference on Learning Representations

2023

[4] [4]

In: Advances in Neural Information Processing Systems, pp 3134–3144

Dupont E, Doucet A, Teh YW (2019) Augmented neural odes. In: Advances in Neural Information Processing Systems, pp 3134–3144

2019

[5] [5]

Com- munications in Mathematics and Statistics 5(1):1–11

E W (2017) A proposal on machine learning via dynamical systems. Com- munications in Mathematics and Statistics 5(1):1–11. https://doi.org/10.1007/ s40304-017-0103-z

2017

[6] [6]

American Mathematical Society, Providence, RI, https: //doi.org/10.1090/gsm/019

Evans LC (2010) Partial Differential Equations, Graduate Studies in Mathe- matics, vol 19, 2nd edn. American Mathematical Society, Providence, RI, https: //doi.org/10.1090/gsm/019

doi:10.1090/gsm/019 2010

[7] [7]

arXiv preprint arXiv:1710.11278 [stat.ML]

Hanin B, Sellke M (2017) Approximating continuous functions by ReLU nets of minimal width. arXiv preprint arXiv:1710.11278 [stat.ML]

arXiv 2017

[8] [8]

In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recogni- tion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

2016