REVIEW 4 major objections 5 minor 17 references
Optimization for Neural Operators can Benefit from Width
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read The paper proves that gradient descent lowers the loss of Deep Operator Networks and Fourier Neural Operators whenever the networks are wide enough.
desk verdict The FNO half is plausible, but the DON convergence guarantee rests on an invalid cosine reduction; the paper needs a real fix before the headline claim holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the restricted strong convexity (RSC) condition posed over a carefully chosen restricted set $N_t = B_t^\kappa$, paired with a $\beta$-smoothness bound on the loss. For a DON, $B_t^\kappa$ intersects a neighborhood of the initialization, a small ball around the current iterate, and a cone-like set $Q_t^\kappa$ that enforces a cosine condition on the direction of decrease together with sign conditions on the branch–trunk cross terms; for an FNO, $Q_t^\kappa$ is just the set of directions whose cosine with the average gradient $\nabla_\theta \bar G_t$ is at least $\kappa$, because the FNO is a single network. The argument verifies RSC through the second-order Taylor expansion of the empirical loss, lower-bounding the Hessian quadratic form by an alignment term $2\kappa^2\|\nabla_\theta \bar G_t\|^2$ minus fluctuation terms that decay as $1/\sqrt{m_f} + 1/\sqrt{m_g}$ (DON) or $1/\sqrt{m}$ (FNO). This split — an alignment term that scales with the squared gradient norm plus a fluctuation that shrinks with width — is the mechanism that lets width ease the RSC condition and enlarge the valid initialization neighborhood.
What would settle it
Train a DON and an FNO while logging each iterate and the squared average gradient norm $\|\nabla_\theta \bar G_t\|^2$; at a few early steps, sample candidate points $\theta'$ from the restricted set $B_t^\kappa$ and measure the largest $\alpha_t$ for which the RSC inequality $L(\theta') \geq L(\theta_t) + \langle \theta' - \theta_t, \nabla L(\theta_t)\rangle + (\alpha_t/2)\|\theta' - \theta_t\|^2$ holds, comparing it with $2\kappa^2\|\nabla_\theta \bar G_t\|^2 - c_1 K^2(1/\sqrt{m_f} + 1/\sqrt{m_g})$ for DONs and $2\kappa^2\|\nabla_\theta \bar G_t\|^2 - c_1/\sqrt{m}$ for FNOs. A measured $\alpha_t$ that disagrees with the formula, a step where the gradient norm is above the $\Omega(\cdot)$ threshold yet the loss rises, or iterates that leave $B(\theta_0)$ while the loss keeps falling would each contradict the theorems.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the empirical losses of DONs and FNOs are not arbitrary nonconvex objectives: they satisfy two conditions that make gradient descent provably contract the loss. For a DON, whose output is the inner product of a branch and a trunk network, the RSC parameter at step $t$ is $\alpha_t = 2\kappa^2\|\nabla_\theta \bar G_t\|^2 - c_1 K^2(1/\sqrt{m_f} + 1/\sqrt{m_g})$, so $\alpha_t > 0$ whenever $\|\nabla_\theta \bar G_t\|^2 = \Omega(1/\sqrt{m_f} + 1/\sqrt{m_g})$; for an FNO, whose hidden blocks contain learnable Fourier-domain weights, the bound is $\alpha_t = 2\kappa^2\|\nabla_\theta \bar G_t\|^2 - c_1/\sqrt{m}$, positive whenever $\|\nabla_\theta \bar G_t\|^2 = \Omega(1/\sqrt{m})$. Together with a width-dependent smoothness bound and the ratio condition $\alpha_t/\beta < 1$ (proved as Propositions 1 and 3), the general Theorem 1 yields a per-step multiplicative loss decrease toward the infimum within the initialization neighborhood. A feature the authors emphasize is that the two architectures satisfy the same two conditions for different technical reasons: the DON analysis must control cross-interaction terms between the branch and trunk networks in the Hessian, while the FNO analysis must control cross-derivatives between parameters in the data and Fourier domains.
Load-bearing premise
The load-bearing premise is that every gradient-descent iterate stays strictly inside the fixed neighborhood $B(\theta_0)$ of the initialization (Assumptions 4 and 7 for DONs and FNOs); the paper assumes this holds without proving that gradient descent maintains it or checking it in the experiments, and the RSC and smoothness bounds, and hence the guaranteed loss decrease, would fail if an iterate left the set.
Editorial extensions
If this is right
- Gradient descent on a sufficiently wide DON or FNO is guaranteed to reduce the empirical loss at every step, at a linear rate, once the average squared gradient norm is $\Omega(1/\sqrt{m_f} + 1/\sqrt{m_g})$ or $\Omega(1/\sqrt{m})$, respectively.
- Increasing width lowers the gradient-norm threshold required for RSC to hold, so over-parameterization strictly relaxes the optimization condition rather than merely improving approximation capacity.
- The initialization-scale condition allows the neighborhood radius to grow with width (the paper derives $\rho \leq \min\{m_f, m_g\}$ for DONs and $\rho_w + \rho_r \leq \sqrt{m}$ for FNOs), so wider networks can train with a larger certificate region around the initialization.
- On the antiderivative, diffusion-reaction, and Burgers' operator benchmarks, wider DONs and FNOs attain lower training losses and generally converge faster, matching the theory's prediction.
Reading between the lines
- Because the RSC threshold shrinks with width, very wide neural operators should enter a regime in which their training loss decreases, along the GD trajectory, like a strongly convex objective; a directly testable prediction is that the empirically measured per-step loss contraction factor tracks $1 - \alpha_t/\beta$.
- The iterate-confinement assumption (Assumptions 4 and 7) is stated without proof or empirical verification, so the theorems as written certify convergence only for trajectories that never leave $B(\theta_0)$; recording whether GD actually stays inside for the tested widths and step sizes would reveal how much of the observed convergence the theory covers.
- Because the framework's two conditions are architecture-agnostic, any future neural operator whose loss is shown to satisfy RSC and smoothness would inherit the same linear-decrease guarantee, so the same proof strategy likely transfers to spectral, graph-kernel, or multiscale operator architectures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper develops a general framework under which gradient descent decreases a loss whenever the loss satisfies a restricted strong convexity (RSC) condition and a smoothness condition at each iterate (Theorem 1). The authors then analyze two neural operator architectures. For DONs, Theorem 2 gives α_t = 2κ²∥∇θ Ḡt∥² - O(1/√mf + 1/√mg) and Theorem 3 gives smoothness, subject to an assumption that all GD iterates stay in a fixed neighborhood; for FNOs, Theorems 4 and 5 give analogous statements with α_t = 2κ²∥∇θ Ḡt∥² - O(1/√m). The width enters by lowering the gradient-norm threshold for RSC and by allowing a larger neighborhood. Experiments on antiderivative, diffusion-reaction, and Burgers' equation show lower training loss with increasing width, although the experiments use Adam rather than GD.
Significance. If the proof gaps are closed, the paper would be a meaningful contribution: it provides the first optimization guarantees for DONs and FNOs under a unified RSC/smoothness framework, and the Hessian bounds for the inner-product branch/trunk model and Fourier-domain layers are nontrivial. The clean abstraction of Theorem 1 and the explicit width dependence in Eqs. (13) and (14) are strengths. However, the advertised convergence is currently conditional on an unproved invariant, and the DON nonemptiness proof contains a substantive gap; until these are fixed, the central claim is not established.
major comments (4)
- [Section 5, Theorem 2 proof] The reduction of membership in Q_t^κ to the branch subvector, around Eq. (28), is invalid. Setting θ'_g = θ_{g,t} gives |cos(θ' - θ_t, ∇_θ Ḡ_t)| = |δ_f^T ∇_{θ_f} Ḡ_t| / (||δ_f|| sqrt(||∇_{θ_f} Ḡ_t||^2 + ||∇_{θ_g} Ḡ_t||^2)), which equals the branch cosine only when ∇_{θ_g} Ḡ_t = 0; this is generically false. Consequently the existence of θ' ∈ B_t^κ satisfying Condition 1(b) is not proved, and step (b) of Eq. (36), which uses the full-vector cosine condition from Definition 2, is not justified. A valid construction that also satisfies the average and output cross-term constraints in Definition 2 is needed.
- [Section 4, Assumption 1; Sections 5-6, Assumptions 4 and 7] The convergence guarantee is conditional on all GD iterates remaining strictly inside the neighborhood B(θ_0), but the paper never proves this invariant. Theorems 2-5 only provide per-iterate RSC and smoothness at a point already assumed to lie in the neighborhood; nothing shows θ_{t+1} remains there for the chosen step size and widths. Since Theorem 1 is applied recursively, the central optimization guarantee is not closed. The authors need an induction argument, or an explicit construction of step size and width conditions, showing that the GD step does not leave B(θ_0).
- [Section 5, Theorem 2 proof (angle construction)] The inequality |cos(ν - ν_1)| ≥ |cos(π/2 - ν_1)| = sqrt(1 - κ^2) is not valid for the stated choices; for example, with κ = 0.1, ν = 0, and ν_1 = arccos(κ), the left side equals 0.1, which is smaller than 0.995. This bound is used to ensure that the constructed θ' satisfies L(θ') < L(θ_t). The construction may be repairable by choosing the angle differently, but as written the proof of Condition 1(b.2) does not go through.
- [Section 6, Theorem 4 proof] The RSC part of Theorem 4 is deferred to a 'direct adaptation' of Theorem 5.1 of Banerjee et al. (2023b) using Lemma E.2. Because the FNO contains learnable parameters acting in the Fourier domain and uses a nonstandard Q_t^κ set, the adaptation should be shown explicitly; otherwise the FNO RSC claim is not verifiable. At minimum, the authors should state which hypotheses of the referenced theorem are satisfied and provide the proof in an appendix.
minor comments (5)
- [Section 8] The experiments use the Adam optimizer while the theoretical results are for gradient descent; this should be acknowledged, and ideally GD runs should be reported to directly test the predicted behavior.
- [Definition 2] The notation ℓ_{i,j} is defined as the squared residual at θ_t, while the proof later uses ℓ′ and ℓ″; please define these derivatives explicitly and consistently.
- [Eq. (36)] The notation ∇_θ Ḡ_θ is inconsistent with ∇_θ Ḡ_{θ_t}; also, the application of Jensen's inequality should identify the average operator explicitly.
- [Section 2] The FNO paragraph repeats 'optimization guarantees for DONs is also an open problem' in the FNO discussion; this is likely a typo for FNOs.
- [Figures 1 and 2] The horizontal axis is labeled 'Epochs % 100'; either state that ticks are every 100 epochs or relabel the axis to avoid ambiguity.
Circularity Check
No circular derivation found: the DON and FNO convergence guarantees follow from model-specific Hessian, gradient, and predictor bounds rather than from the conclusions they are used to prove.
full rationale
The paper's derivation chain is: (i) Theorem 1 reduces GD loss decrease to Conditions 1 and 2 and is proved in Appendix C from a restricted PL inequality and smoothness; (ii) Theorems 2-3 and 4-5 establish RSC and smoothness for the DON and FNO losses using model-specific Hessian, gradient, and predictor bounds; and (iii) the width-benefit statements are algebraic consequences of the displayed RSC parameters, not fitted quantities. No parameter is fitted to the experiments and then renamed as a prediction. The restricted sets Q_t^kappa are constructed to make cross-interaction terms in the Hessian tractable, but the construction is not equivalent to the RSC inequality; obtaining the RSC bound still requires substantive Hessian estimates and probability arguments. The self-citations to Banerjee et al. (2023b) provide the RSC proof template and are external published results, while the DON proof is carried out in the paper and the FNO proof adapts the prior theorem using a new Lemma E.2. The potential gap in the nonemptiness proof of Q_t^kappa for DONs, concerning reduction of the full-parameter cosine condition to the branch subvector, is a correctness concern rather than a circular reduction, so it does not raise the circularity score. The unproven iterate-invariant assumptions are assumptions, not circular steps. Overall, the central claims do not reduce by construction to their inputs or to a load-bearing self-citation chain.
Assumptions & free parameters
free parameters (3)
- κ
- ρ, ρ1, ρ2
- σ0 (DON) / σ1,w, σ1,r (FNO)
assumptions (5)
- domain assumption Activation functions are 1-Lipschitz and βϕ-smooth
- domain assumption Weights initialized as independent Gaussians with variance scaling ~1/m and last-layer vectors with unit norm
- domain assumption Inputs are normalized: ∥u(x)∥₂ = √R, ∥y∥₂ = √d_y, and encoder output norm √d
- domain assumption All GD iterates stay strictly inside the initialized neighborhood
- standard math RSC-based optimization framework of Banerjee et al. (2023b)
Cite this review
Pith. "Pith review of Optimization for Neural Operators can Benefit from Width." pith.science (2026). https://pith.science/paper/OSLZA6DX
@misc{pith2026250200705,
author = {Pith},
title = {Pith review of: Optimization for Neural Operators can Benefit from Width},
year = {2026},
howpublished = {\url{https://pith.science/paper/OSLZA6DX}},
note = {Machine review of arXiv:2502.00705}
}
read the original abstract
Neural Operators that directly learn mappings between function spaces, such as Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs), have received considerable attention. Despite the universal approximation guarantees for DONs and FNOs, there is currently no optimization convergence guarantee for learning such networks using gradient descent (GD). In this paper, we address this open problem by presenting a unified framework for optimization based on GD and applying it to establish convergence guarantees for both DONs and FNOs. In particular, we show that the losses associated with both of these neural operators satisfy two conditions -- restricted strong convexity (RSC) and smoothness -- that guarantee a decrease on their loss values due to GD. Remarkably, these two conditions are satisfied for each neural operator due to different reasons associated with the architectural differences of the respective models. One takeaway that emerges from the theory is that wider networks should lead to better optimization convergence for both DONs and FNOs. We present empirical results on canonical operator learning problems to support our theoretical results.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[3]
r r R n nqi Trunk Net: Branch Net: Output Locations Input Functions “inner product” 1 1 g f mf mg mf mg width of the branch net width of the trunk net K K Figure 3: A schematic of the DON architecture by Lu et al. (2021) used in our study. We refer to the notation used in our paper. Note that the input functions need not be sampled on a structured grid of...
work page 2021
-
[5]
This is the same setup as in (Wang et al., 2021a; Lu et al., 2021)
along with a forcing function u(x) defined by a GRF. This is the same setup as in (Wang et al., 2021a; Lu et al., 2021). The corresponding solutions for DONs and FNOs are presented in Figures 7 and 8 respectively. Again, a larger width m leads to a more accurate solution. The neural operator aims to learn a mapping from the forcing function to the solutio...
work page 2021
-
[9]
arXiv: 2202.11214. S. Qin, F. Lyu, W. Peng, D. Geng, J. Wang, N. Gao, X. Liu, and L. L. Wang. Toward a better understanding of fourier neural operators: Analysis and improvement from a spectral perspective. arXiv preprint arXiv:2404.07200,
-
[10]
Y. Qiu, N. Bridges, and P. Chen. Derivative-enhanced deep operator network.arXiv preprint arXiv:2402.19242,
-
[11]
A. Tran, A. Mathews, L. Xie, and C. S. Ong. Factorized fourier neural operators. arXiv preprint arXiv:2111.13802,
-
[13]
arXiv: 2106.05384. S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets, 2021a. S. Wang, H. Wang, and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets. arXiv:2103.10974 [cs, math, stat] , 202...
-
[14]
Q. Yang, A. Hernandez-Garcia, P. Harder, V. Ramesh, P. Sattegeri, D. Szwarcman, C. D. Watson, and D. Rolnick. Fourier neural operators for arbitrary resolution climate data downscaling. arXiv preprint arXiv:2305.14452,
-
[15]
14 A Related Work Learning Operators. Constructing operator networks for ordinary differential equations using learning- based approaches was first studied in (Chen and Chen, 1995), where a neural network with a single hidden layer was shown to approximate a nonlinear continuous functional. This was, in essence, akin to the Universal Approximation Theorem...
work page 2021
Show all 17 references
-
[17]
data” refers to the ground truth (obtained by a standard numerical solver) and “pred
We also present further results on how the accuracy of each neural operator model improves as the width m increases. Finally, we provide details about the hyperparameters and datasets used in the training of the corresponding models. We remark that all experiments with widths ...
2000
-
[18]
= 0 , 3During training; however, for each i ∈ [n], instead of averaging the loss over all the qi points, we simply randomly choose one of the qi points and evaluate the loss on it. This is strictly done in the interest of computational efficiency, since it is known to not redu...
2021
-
[20]
All solutions are calculated for a single viscosity ν = 0.01
We make use of the datasets publicly available at https://github.com/neuraloperator/ neuraloperator, specifically the Burgers R10.mat dataset available at https://drive.google.com/drive/ folders/1UnbQh2WWc6knEHbLn-ZaXrKUZhp7pjt-, which comprises of 2048 input functions and cor...
-
[2004]
Centofanti, M
E. Centofanti, M. Ghiotto, and L. F. Pavarino. Learning the hodgkin-huxley model with operator learning techniques. arXiv preprint arXiv:2406.02173 ,
-
[2017]
Curran Associates Inc. K. Kontolati, S. Goswami, M. D. Shields, and G. E. Karniadakis. On the influence of over-parameterization in manifold based surrogates and deep neural operators. arXiv preprint arXiv:2203.05071 ,
-
[2019]
arXiv:1811.03962 [cs, math, stat] type: article. S. Arora, S. Du, W. Hu, Z. Li, and R. Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning . 2019a. S. Arora, S. S. Du, W...
2005 arXiv
-
[2022]
arXiv: 2108.06905. P. Harder, A. Hernandez-Garcia, V. Ramesh, Q. Yang, P. Sattegeri, D. Szwarcman, C. Watson, and D. Rolnick. Hard-constrained deep learning for climate downscaling. Journal of Machine Learning Research , 24(365): 1–40,
-
[2023]
B. Liu, N. Kovachki, Z. Li, K. Azizzadenesheli, A. Anandkumar, A. Stuart, and K. Bhattacharya. A learning- based multiscale method and its application to inelastic impact problems. Journal of the Mechanics and Physics of Solids , 158:104668, 2022a. arXiv: 2102.07256. C. Liu, L...
2010 arXiv
-
[2024]
Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Multipole Graph Neural Operator for Parametric Partial Differential Equations. arXiv:2006.09535 [cs, math, stat] , 2020a. arXiv: 2006.09535. 12 Z. Li, N. Kovachki, K. Azizzadenesheli...
2006 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.