Shallow ReLU$^s$ Networks in $L^p$-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

Fanghui Liu; Lei Shi; Weizhao Li

arxiv: 2605.18468 · v4 · pith:FC6QDEVHnew · submitted 2026-05-18 · 📊 stat.ML · cs.LG

Shallow ReLU^s Networks in L^p-Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization

Weizhao Li , Fanghui Liu , Lei Shi This is my paper

Pith reviewed 2026-05-25 06:00 UTC · model grok-4.3

classification 📊 stat.ML cs.LG

keywords shallow neural networksReLU^s activationpath-norm regularizationapproximation ratesnonparametric regressionSobolev spacesL^p spacesminimax rates

0 comments

The pith

Path-norm regularized shallow ReLU^s networks achieve minimax-optimal rates in nonparametric regression over B_s and Sobolev spaces.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper derives approximation rates for shallow networks using the ReLU^s activation in L^p-type integral spaces via spherical harmonic analysis, with explicit exponents that change at a critical p value. It obtains Sobolev space bounds by embedding into spectral Barron spaces. For nonparametric regression with sub-Gaussian noise, controlling the l1 path-norm produces generalization rates that match the minimax lower bounds up to logarithmic factors. A sympathetic reader would care because the results give concrete rates that shallow networks can attain under regularization without requiring deeper architectures.

Core claim

For nonparametric regression with sub-Gaussian noise, path-norm-regularized shallow ReLU^s networks achieve minimax-optimal rates O(n^{-(d+2s+1)/(2d+2s+1)} log n) over B_s and O(n^{-2 alpha/(2 alpha + d)} log n) over W^{alpha, infty}, with matching lower bounds up to logarithmic factors. Approximation bounds in the L^p-type spaces are O(m^{-p(2s+2d+1)-2d/(2dp)}) for 1 <= p <= p* and O(m^{-p(4s+3d-1)-2d+2/(4dp)}) for p* < p < 2, where p* = (2d+2)/(d+3).

What carries the argument

Spherical harmonic analysis yielding the L^p approximation rates, together with the l1 path-norm that regularizes the network for the generalization analysis.

If this is right

The rates are optimal for both the space B_s and the Sobolev space W^{alpha, infty} up to log factors.
The approximation exponent in L^p spaces changes at the threshold p* = (2d+2)/(d+3).
Path-norm regularization alone suffices to reach the minimax rates for the given function spaces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If path-norm can be efficiently estimated or optimized, the results suggest a practical route to optimal rates using only shallow networks.
Similar harmonic-analysis techniques might yield rates for other smooth activations beyond ReLU^s.

Load-bearing premise

The approximation bounds in L^p-type spaces are obtained via spherical harmonic analysis, and Sobolev bounds follow from embeddings into spectral Barron spaces.

What would settle it

An experiment or calculation showing that the observed regression rate over B_s is slower than n^{-(d+2s+1)/(2d+2s+1)} log n for large n would falsify the optimality claim.

Figures

Figures reproduced from arXiv: 2605.18468 by Fanghui Liu, Lei Shi, Weizhao Li.

**Figure 1.** Figure 1: Hierarchy of function spaces considered in this paper, ordered from smaller to larger in generality. and let Σs = S∞ m=1 Σ s m denote the space of shallow ReLUs networks of arbitrary width. Studying approximation over Fep,ν,s clarifies how the difficulty of learning changes along the spectrum from RKHSs to Barron spaces. For Sobolev spaces, the theory is relatively complete when p ≥ 2, whereas the regime 1… view at source ↗

read the original abstract

This paper studies approximation by shallow ReLU$^s$ networks, $\sigma_s(t)=\max\{0,t\}^s$, together with their generalization behavior under $\ell_1$ path-norm control. For the $L^p$-type integral spaces $\widetilde{\mathcal{F}}_{p,\tau_d,s}$, $1\le p\le2$, spherical harmonic analysis yields approximation bounds for shallow networks. In particular, when $\tau_d$ is the uniform measure and $1\le p<2$, the approximation rate is $O\!\left(m^{-\frac{p(2s+2d+1)-2d}{2dp}}\right)$ for $1\le p\le p^*$ and $O\!\left(m^{-\frac{p(4s+3d-1)-2d+2}{4dp}}\right)$ for $p^*<p<2$, where $p^*=\frac{2d+2}{d+3}$. Approximation bounds for Sobolev spaces $W^{\alpha,p}$, $1\le p<2$, are obtained through embeddings into spectral Barron spaces. For nonparametric regression with sub-Gaussian noise, path-norm-regularized shallow ReLU$^s$ networks achieve minimax-optimal rates $O\!\left(n^{-\frac{d+2s+1}{2d+2s+1}}\log n\right)$ over $\mathscr{B}_s$ and $O\!\left(n^{-\frac{2\alpha}{2\alpha+d}}\log n\right)$ over $W^{\alpha,\infty}$, with matching lower bounds up to logarithmic factors.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives explicit approximation rates for shallow ReLU^s nets in L^p-type spaces via spherical harmonics, plus path-norm generalization bounds that hit minimax rates with matching lower bounds.

read the letter

This paper works out approximation rates for shallow networks using ReLU^s activations in certain L^p integral spaces and then ties those to generalization under path-norm control. The main new pieces are the rates that split at p* = (2d+2)/(d+3), with one form for p below that and another above, derived from spherical harmonic analysis when the measure is uniform. They also get Sobolev bounds by embedding into spectral Barron spaces. On the statistical side, path-norm regularized shallow ReLU^s networks reach the rate O(n^-(d+2s+1)/(2d+2s+1) log n) over the Barron ball B_s and the standard Sobolev rate over W^alpha,infty, with lower bounds that match up to logs for nonparametric regression under sub-Gaussian noise. That matching lower bound is useful and not automatic. The claims line up with prior Barron-type work once the embeddings are accepted, and the p* transition looks consistent with the analysis. The abstract states the results cleanly but does not include the derivations or error bounds, so the technical steps need checking in the full text. No obvious circularity or mismatched exponents show up. This is aimed at readers who follow approximation theory for neural nets and its statistical consequences. It has enough concrete rates and optimality statements to deserve a serious referee, though the proofs will need close reading.

Referee Report

0 major / 3 minor

Summary. The manuscript studies approximation properties of shallow ReLU^s networks (with activation max{0,t}^s) in the L^p-type integral spaces F̃_{p,τ_d,s} for 1≤p≤2, deriving rates via spherical harmonic analysis that exhibit a transition at p^*=(2d+2)/(d+3). It obtains Sobolev-space bounds W^{α,p} via embeddings into spectral Barron spaces, and shows that ℓ1 path-norm regularized shallow ReLU^s networks attain the stated minimax-optimal nonparametric regression rates O(n^{-(d+2s+1)/(2d+2s+1)} log n) over B_s and O(n^{-2α/(2α+d)} log n) over W^{α,∞} (with sub-Gaussian noise), together with matching lower bounds up to logarithmic factors.

Significance. If the central claims hold, the work supplies a coherent extension of Barron-type approximation theory to ReLU^s activations and L^p-type spaces, with explicit phase-transition exponents and embeddings that enable optimal statistical rates under path-norm control. The presence of matching lower bounds (up to logs) and the use of harmonic-analysis tools constitute a clear technical strength for the nonparametric regression setting.

minor comments (3)

[Abstract / §3] The transition point p^* and the two distinct approximation exponents in the L^p-type spaces are stated in the abstract; the manuscript should explicitly reference the spherical-harmonic lemmas or propositions that produce the precise algebraic forms of these exponents.
[Abstract] Notation for the spaces B_s and the precise definition of the path-norm regularizer should be introduced with a forward reference to the relevant section before the statistical-rate statements.
[§4] The embedding argument from W^{α,∞} into the spectral Barron space is invoked to transfer approximation rates; a short self-contained statement of the embedding constant or the precise norm comparison would improve readability.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary, significance assessment, and recommendation of minor revision. No specific major comments appear in the report, so we have no point-by-point responses to provide. We will address any minor issues identified during the revision process.

Circularity Check

0 steps flagged

No significant circularity; derivation relies on external harmonic analysis and embeddings

full rationale

The paper derives approximation rates for shallow ReLU^s networks in L^p-type spaces via spherical harmonic analysis and obtains Sobolev bounds through embeddings into spectral Barron spaces. Statistical rates under path-norm regularization follow from these approximation results combined with standard nonparametric regression analysis for sub-Gaussian noise, with matching lower bounds. No step reduces a claimed prediction or uniqueness result to a fitted parameter, self-citation chain, or definitional equivalence; the central claims remain independent of the paper's own fitted quantities or prior self-references.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Based on the abstract alone, the claims rest on standard tools from harmonic analysis and function space embeddings with no free parameters or invented entities identified.

axioms (2)

domain assumption Spherical harmonic analysis yields the stated approximation bounds for the L^p-type spaces
Explicitly invoked in the abstract as the source of the rates.
domain assumption Sobolev spaces embed into spectral Barron spaces allowing transfer of approximation bounds
Stated as the route to Sobolev bounds in the abstract.

pith-pipeline@v0.9.0 · 5832 in / 1234 out tokens · 31316 ms · 2026-05-25T06:00:16.733657+00:00 · methodology

Review history (3 revisions) →