Pith. sign in

REVIEW 3 major objections 6 minor 56 references

Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that Fourier Neural Operators have a Maximal Update Parametrization under Fourier-mode scaling, making small-model hyperparameters transfer to billion-parameter FNOs.

desk verdict The scaling analysis for FNO is genuinely new, but Theorem 3.5 builds on a false premise about Adam's update tensor: the entries are bounded, so the max cannot grow like sqrt(d log K), and the derived learning-rate scaling actually makes feature updates vanish. read the letter →

arxiv 2506.19396 v1 pith:C3JBBNZG submitted 2025-06-24 cs.LG

Shanda Li , Shinjae Yoo , Yiming Yang This is my paper · ORCID
classification cs.LG
keywords FourierNeuralOperatorsoperatorlearningMaximalUpdateParametrizationzero-shothyperparametertransferAdamoptimizerkernelintegralPDEsurrogatesNavier-Stokesequations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that Fourier Neural Operators (FNOs), a standard deep-learning architecture for solving partial differential equations, have a Maximal Update Parametrization when the number of Fourier modes $K$ is scaled up. The paper derives the specific rule: as $K$ grows, the standard deviation of the kernel-integral parameter initialization and the learning rate for those parameters should both scale like $1/\sqrt{d\log K}$, where $d$ is the PDE dimension. If this rule is correct, the optimal hyperparameters found on a small, inexpensive FNO (learning rate, batch size, optimizer settings) remain optimal for models with far more Fourier modes, up to roughly a billion parameters, so no retuning is needed at scale. The paper validates the rule across Burgers' equation, Darcy flow, and Navier-Stokes, and reports that on Navier-Stokes the transferred configuration beats direct large-model tuning in accuracy while using 0.30x of the training compute. A correct maximal update parametrization would make large neural PDE solvers practical to train instead of prohibitively expensive to tune.

What carries the argument

The load-bearing object is the spectral norm of the Fourier kernel integral operator $\mathcal{K}_\ell$. For a discretized input, $\mathcal{K}_\ell$ has an eigen-decomposition via the discrete Fourier transform with a diagonal matrix, so $\|\mathcal{K}_\ell\|_2 = \max_{k\in[K]^d}|r_k^\ell|$. This turns the analysis of feature growth into a question about the maximum of $K^d$ i.i.d. sub-Gaussian variables, which scales like $b(K)\sqrt{d\log K}$; requiring this norm to be $\Theta(1)$ fixes $b(K)=\Theta(1/\sqrt{d\log K})$. The same max-of-sub-Gaussians estimate is applied to the Adam update tensor, which yields the matching learning-rate scale $c(K)=\Theta(1/\sqrt{d\log K})$ and guarantees that each Adam step changes the operator by $\Theta(1)$. The generalized abc-parametrization (Definition 3.1) is the framework object that assigns the three scales to the kernel tensor, and Lemma A.3 is what connects the two norm conditions to the Maximal Update Parametrization definition.

What would settle it

Measure the spectral norm of one Adam update to the Fourier-kernel tensor at $K=3$ and $K=24$ under the proposed scaling: the theorem predicts it stays $\Theta(1)$ across $K$, while if the largest absolute entry of Adam's normalized update is empirically flat in $K$, the norm should shrink like $1/\sqrt{\log K}$ and the maximal-update condition fails.

Watch

Extended reading notes

Core claim

The paper's central claim is Theorem 3.5: under Assumptions 3.3 and 3.4, the abc-parametrization $a(K)=1$, $b(K)=c(K)=\Theta(1/\sqrt{d\log K})$ is a Maximal Update Parametrization of the FNO with Adam when the number of Fourier modes $K$ grows. Here $b(K)$ scales the standard deviation of the kernel-tensor initialization, so the initialization variance is $\Theta(1/(d\log K))$, and $c(K)$ scales the master learning rate for that tensor. The paper derives this by computing the spectral norm of the kernel integral operator: after the discrete Fourier transform the operator is diagonal, so its norm is the largest absolute value among $K^d$ independent sub-Gaussian parameters, which grows like $\sqrt{d\log K}$; rescaling by that factor keeps hidden features $\Theta(1)$ at initialization while still allowing each layer's feature update to be $\Theta(1)$, the definition of maximal feature learning. The corollary the paper emphasizes is zero-shot transfer: with this parametrization, hyperparameters tuned on a small-$K$ proxy transfer to a large-$K$ FNO, and the paper reports stable optimal learning rates, batch sizes, and Adam $\beta_2$ values across $K$, and on the Navier-Stokes benchmark a $5.34\%$ relative error at $0.30\times$ of the full-tuning compute.

Load-bearing premise

The load-bearing premise is that the per-parameter optimizer updates have Gaussian-like tails, so the largest of the $K^d$ updates grows like $\sqrt{d\log K}$; Adam's normalized updates are in fact bounded, which would keep that maximum constant and could invalidate the derived scaling.

Editorial extensions

If this is right

  • A practitioner can tune a small-$K$ proxy FNO and transfer the configuration to a large-$K$ FNO with no additional search; in the paper's FNO-3D experiment the optimal learning rate stays essentially fixed as the model grows from 1.7M to 906M parameters.
  • The recipe is explicit: when moving from proxy $K_p$ to target $K^*$, scale the kernel-tensor learning rate by $\sqrt{\log K_p/\log K^*}$ and the initialization variance by $\log K_p/\log K^*$.
  • The scaling is independent of the discretization grid size $N_j$, so it does not break the resolution-agnostic nature of FNOs.
  • The same parametrization works with physics-informed objectives: $\mu$Transfer-PINO stabilizes the learning-rate landscape on Darcy flow, showing the result does not depend on the supervised-loss form.
  • On the Navier-Stokes problem, the transferred configuration reaches $5.34\%$ relative error versus $5.69\%$ for direct full-model tuning while using $0.30\times$ the compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to derive the analogous maximal update parametrization for other spectral or kernel-based operator architectures, such as wavelet-based, multipole, or transformer-based neural operators, whose parameter count also grows with the number of modes; the paper does not claim these, but the max-of-sub-Gaussians mechanism would apply directly.
  • A sharper version of the argument could replace Assumption 3.4 with the deterministic bound that Adam's normalized update already satisfies; this might recover a maximal-update parametrization with a different $K$-dependence, and checking it would settle the main theoretical vulnerability.
  • Because the derived scaling depends only on the logarithm of $K$, the transfer window may extend beyond the tested range $K=3$ to $K=24$; the paper's experiments do not establish the upper end, so this remains an open empirical question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes μTransfer-FNO, a zero-shot hyperparameter transfer method for Fourier Neural Operators (FNOs) when scaling the number of Fourier modes K. It defines an abc-parametrization for the kernel integral operator, and claims in Theorem 3.5 that a(K)=1, b(K)=c(K)=Θ(1/√(d log K)) is a Maximal Update Parametrization for FNO trained with Adam. The proof in Appendix A.2 analyzes the spectral norm of the kernel operator as the maximum of K^d sub-Gaussian entries, leading to the √(d log K) scaling. The paper then presents Algorithm 1 for hyperparameter transfer and reports experiments on Burgers' equation, Darcy flow, Navier-Stokes equations, and PINO, showing stable optimal learning rates and a 0.30x compute saving on a near-billion-parameter FNO.

Significance. If the main theorem were correct, the work would be a meaningful extension of the Maximal Update Parametrization framework to neural operators, introducing a new scaling law with a √(d log K) dependence and providing a principled zero-shot tuning recipe for large FNOs. The empirical study is extensive, covers several PDEs and training objectives, and includes a near-billion-parameter demonstration with released code. However, the central theoretical claim is not supported: the proof's key concentration step treats Adam update entries as unbounded sub-Gaussians whose maximum grows like Θ(√(d log K)), whereas Adam's normalized updates are actually bounded by a constant independent of K. This invalidates the derived learning-rate scaling and the proof of the maximal-update condition. The empirical results do not compensate for the unsupported central derivation, because the paper's contribution is framed as a mathematical derivation of the parametrization.

major comments (3)
  1. [Theorem 3.5 / Appendix A.2] The proof of Theorem 3.5 relies on the step in Appendix A.2: 'by Assumption 3.4, using an argument similar to the above proof of stability at initialization, we have ∥Kℓ,t+1−Kℓ,t∥2 = Θ(1)'. This step requires that the maximum of the K^d entries of the Adam update tensor g_t(Rℓ) grows like Θ(√(d log K)). But for Adam, g_t is the normalized update m̂/(√v̂+ε), whose entries are bounded by a constant depending only on β1, β2, and ε, not on K (pointwise, |m̂| is bounded by a K-independent multiple of √v̂ up to bias-correction factors). Therefore the maximum of K^d entries is O(1) with high probability, not Θ(√(d log K)). With c(K)=Θ(1/√(d log K)), the per-step operator update satisfies ||Δ_t K_ℓ||_2 = η0 c(K)·O(1) = O(1/√(d log K)) = o(1), which violates the feature-learning condition Δ_t w_ℓ = Θ(1) in Definition 3.2. Assumption 3.4 is not false—bounded variables are sub-Gaussian—but the maximum concentration theorem used to justify the cancellation does not apply to bounded entries with a fixed proxy constant.
  2. [Section 3.3] The discussion of Assumption 3.4 states that 'the sub-Gaussian condition can be explicitly enforced in practice through element-wise gradient clipping, which bounds the updates and thereby ensures their sub-Gaussianity'. This points in the wrong direction: clipping or the intrinsic normalization in Adam makes the entries of g_t uniformly bounded by a constant, forcing max_k |g_t^k| = O(1). The proof needs this maximum to be Θ(√(d log K)) to cancel c(K)=Θ(1/√(d log K)). Thus the proposed practical enforcement removes exactly the K-dependence that the proof's concentration argument requires, making the proof and the practical recommendation mutually inconsistent.
  3. [Definition 3.2 / Theorem 3.5] Even under the paper's own definitions, the theorem does not establish the 'feature learning in every layer' condition. The induction step showing h_{ℓ,t}=Θ(1) depends on the invalid claim that ||Δ_t K_ℓ||_2 = Θ(1) per step. With the derived c(K)=Θ(1/√(d log K)) and bounded Adam update entries, the per-step spectral-norm update is O(1/√(d log K)), so after any fixed number of steps t the feature update is o(1), contradicting Definition 3.2's requirement that Δ_t w_ℓ = Θ(1). The parametrization in Theorem 3.5 therefore does not satisfy the maximal-update definition it is claimed to meet.
minor comments (6)
  1. [Section 3.2] The sentence 'We extends and generalizes the abc-Parametrization' has a grammatical error; it should be 'We extend and generalize'.
  2. [Section 4.2] The text reads 'for FNO-2D on the the incompressible Navier-Stokes Equation, the optimal learning rate align approximately at 4.2×10−3'; FNO-2D is used for Darcy Flow rather than Navier-Stokes, and 'the the' is a typo.
  3. [Figure 2c] The label in Figure 2c says 'Transfer-PINO' while the text consistently uses 'μTransfer-PINO'; the abstract similarly uses 'Transfer-FNO' without the μ prefix.
  4. [Section 2.3] The sentence 'the training dynamics and hence the parameter landscape across remain consistent as the model size scales under μP' is missing a noun after 'across'.
  5. [Section 3.3] The heading 'Regarding Assumption 3.4 and proactical applications' contains a typo: 'proactical' should be 'practical'.
  6. [Table 1 caption] The caption 'Test relative error comparisonsbetween directly tuning' is missing a space: 'comparisons between'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the μP scaling rates are solved from Definition 3.2 under stated stochastic assumptions and validated empirically; the flagged sub-Gaussian-max issue is a correctness concern, not a reuse of inputs.

full rationale

The paper's central derivation is self-contained in the sense required here. Definition 3.2 fixes the target behavior (spectral norms and feature updates of order Θ(1)); Theorem 3.5 and Appendix A.2 solve for the initialization multiplier b(K) and learning-rate multiplier c(K) from this target plus Assumptions 3.3 and 3.4, using the maximum-of-sub-Gaussian bound for K^d entries. No parameter in the theorem is fitted to the experimental data that is later used to claim zero-shot transfer; the experiments are external validation rather than an input to the derivation. Citations to Yang & Hu (2021) and Yang et al. (2022) supply the background μP/μTransfer framework but are not load-bearing: the FNO-specific spectral-norm argument is carried out in the present paper. The reviewer concern about Adam's normalized updates being bounded is a legitimate mathematical objection to whether Assumption 3.4 yields Θ(sqrt(d log K)) maxima, but that is a correctness issue about the assumption, not a circular reuse of the conclusion; the assumption is stated, not derived from the theorem. Therefore no circularity score above zero is warranted.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on Assumption 3.4 (sub-Gaussian, independent Adam updates), which is not established for Adam and is in tension with the boundedness of Adam's normalized updates. The other axioms are standard or domain-specific. No parameters are fitted to data in the derivation; the scaling rates come from Theta-cancellation.

assumptions (5)
  • domain assumption Activation functions are tanh or sigma-GELU (Assumption 3.3)
    Standard in uP literature; ensures h_l = phi(w_l) = Theta(1) when w_l = Theta(1).
  • ad hoc to paper Adam optimizer update tensor entries are independent C-sub-Gaussian, conditional on previous parameters (Assumption 3.4)
    This is the load-bearing assumption. It is not proven and is questionable for Adam, whose normalized updates are bounded. If the entries are bounded, the max over K^d entries does not grow as sqrt(d log K), invalidating the derived learning-rate scaling c(K).
  • domain assumption The hidden dimensionality m and depth L are fixed while K scales
    The paper explicitly considers scaling only the number of Fourier modes K; width/depth scaling is left to prior work.
  • domain assumption The domain discretization satisfies min_j N_j > K
    Needed for the truncated Fourier representation and for treating the operator as a finite matrix.
  • standard math uP is invariant under constant rescaling (a,b,c) -> (a/psi, b*psi, c*psi) for Adam
    Taken from Yang et al. (2022); used to justify setting a(K)=1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators." pith.science (2026). https://pith.science/paper/C3JBBNZG

@misc{pith2026250619396,
  author       = {Pith},
  title        = {Pith review of: Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3JBBNZG}},
  note         = {Machine review of arXiv:2506.19396}
}
abstract

Fourier Neural Operators (FNOs) offer a principled approach for solving complex partial differential equations (PDEs). However, scaling them to handle more complex PDEs requires increasing the number of Fourier modes, which significantly expands the number of model parameters and makes hyperparameter tuning computationally impractical. To address this, we introduce $\mu$Transfer-FNO, a zero-shot hyperparameter transfer technique that enables optimal configurations, tuned on smaller FNOs, to be directly applied to billion-parameter FNOs without additional tuning. Building on the Maximal Update Parametrization ($\mu$P) framework, we mathematically derive a parametrization scheme that facilitates the transfer of optimal hyperparameters across models with different numbers of Fourier modes in FNOs, which is validated through extensive experiments on various PDEs. Our empirical study shows that Transfer-FNO reduces computational cost for tuning hyperparameters on large FNOs while maintaining or improving accuracy.

Figures

Figures reproduced from arXiv: 2506.19396 by the authors.

Figure 1
Figure 1. Loss values of FNO-3D on the incompressible Navier-Stokes Equation, with varying numbers of Fourier modes K and different learning rates. The left and right panels correspond to models trained with standard techniques and µTransfer-FNO, respectively. The star on each curve marks the optimal learning rate which leads to the lowest loss value. Under standard parametrization, the optimal learning rates shifts as model … view at source ↗
Figure 2
Figure 2. Loss values of FNOs/PINOs with varying numbers of Fourier modes K and different learning rates. The star on each curve marks the optimal learning rate which leads to the lowest loss value. We observe that the optimal learning rate is more consistent under µTransfer, compared to the standard parametrization. under both standard parametrization and µTransfer-FNO trained with different learning rates. Under standard pa… view at source ↗
Figure 3
Figure 3. Loss values of FNO-2D on the Darcy Flow problem with different hyperparameters and varying numbers of Fourier modes K. The star on each curve marks the optimal setting which leads to the lowest loss value. We observe that the optimal hyperparameter is more consistent under µTransfer, compared to the standard parametrization. timal learning rates aligning at approximately 5.6 × 10−3 . This result is noteworthy given … view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 27 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Gruber, L., Holzleitner, M., and Brandstetter, J

    Alkin, B., F \"u rst, A., Schmid, S. L., Gruber, L., Holzleitner, M., and Brandstetter, J. Universal physics transformers: A framework for efficiently scaling neural operators. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=oUXiNX5KRm

    work page 2024

  3. [3]

    Y., Deiseroth, B., Cruz-Salinas, A

    Blake, C., Eichenberg, C., Dean, J., Balles, L., Prince, L. Y., Deiseroth, B., Cruz-Salinas, A. F., Luschi, C., Weinbach, S., and Orr, D. u- p: The unit-scaled maximal update parametrization. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024. URL https://openreview...

    2024

  4. [4]

    B., Hanin, B., and Pehlevan, C

    Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehlevan, C. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KZJehvRKGD

    work page 2024

  5. [5]

    B., Levine, M

    Calvello, E., Kovachki, N. B., Levine, M. E., and Stuart, A. M. Continuum attention for neural operators. arXiv preprint arXiv:2406.06486, 2024

    arXiv 2024

  6. [6]

    Choose a transformer: Fourier or galerkin

    Cao, S. Choose a transformer: Fourier or galerkin. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=ssohLcmn4-r

    work page 2021

  7. [7]

    Principled architecture-aware scaling of hyperparameters

    Chen, W., Wu, J., Wang, Z., and Hanin, B. Principled architecture-aware scaling of hyperparameters. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=HZndRcfyNI

    work page 2024

  8. [8]

    and Mishra, S

    De Ryck, T. and Mishra, S. Generic bounds on the approximation error for physics-informed (and) operator learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 10945--10958. Curran Associates, Inc., 2022

    work page 2022

Show all 56 references
  1. [9]

    Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster

    Dey, N., Gosal, G., Khachane, H., Marshall, W., Pathria, R., Tom, M., Hestness, J., et al. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. arXiv preprint arXiv:2304.03208, 2023

    2023 arXiv

  2. [10]

    Physics-informed inference time scaling via simulation-calibrated scientific machine learning

    Fan, Z., Sun, Y., Yang, S., and Lu, Y. Physics-informed inference time scaling via simulation-calibrated scientific machine learning. arXiv preprint arXiv:2504.16172, 2025

    2025

  3. [11]

    J., Zhao, J., Kossaifi, J., Li, Z., and Anandkumar, A

    George, R. J., Zhao, J., Kossaifi, J., Li, Z., and Anandkumar, A. Incremental spatial and spectral learning of neural operators for solving large-scale PDE s. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=xI6cPQObp0

    2024

  4. [12]

    Multiwavelet-based operator learning for differential equations

    Gupta, G., Xiao, X., and Bogdan, P. Multiwavelet-based operator learning for differential equations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 24048--24062. Curran Asso...

    2021

  5. [13]

    Solving high-dimensional partial differential equations using deep learning

    Han, J., Jentzen, A., and Weinan, E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115 0 (34): 0 8505--8510, 2018

    2018

  6. [14]

    Learning physics-informed neural networks without stacked back-propagation

    He, D., Li, S., Shi, W., Gao, X., Zhang, J., Bian, J., Wang, L., and Liu, T.-Y. Learning physics-informed neural networks without stacked back-propagation. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intel...

    2023

  7. [15]

    B., Dhariwal, P., Gray, S., et al

    Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020

    2010 arXiv

  8. [16]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  9. [17]

    Meta-auto-decoder for solving parametric partial differential equations

    Huang, X., Ye, Z., Liu, H., Ji, S., Wang, Z., Yang, K., Li, Y., Wang, M., Chu, H., Yu, F., et al. Meta-auto-decoder for solving parametric partial differential equations. Advances in Neural Information Processing Systems, 35: 0 23426--23438, 2022

    2022

  10. [18]

    and Karakida, R

    Ishikawa, S. and Karakida, R. On the parameterization of second-order optimization effective towards the infinite width. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=g8sGBSQjYk

    2024

  11. [19]

    Jones, A. L. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03113, 2021

    2021 arXiv

  12. [20]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  13. [21]

    Solving parametric pde problems with artificial neural networks

    Khoo, Y., Lu, J., and Ying, L. Solving parametric pde problems with artificial neural networks. European Journal of Applied Mathematics, 32 0 (3): 0 421--435, 2021

    2021

  14. [22]

    and Kang, M

    Kim, T. and Kang, M. Bounding the rademacher complexity of fourier neural operators. Machine Learning, 113 0 (5): 0 2467--2498, 2024

    2024

  15. [23]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  16. [24]

    A., Alieva, A., Wang, Q., Brenner, M

    Kochkov, D., Smith, J. A., Alieva, A., Wang, Q., Brenner, M. P., and Hoyer, S. Machine learning--accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118 0 (21): 0 e2101784118, 2021

    2021

  17. [25]

    Understanding the expressivity and trainability of fourier neural operator: A mean-field perspective

    Koshizuka, T., Fujisawa, M., Tanaka, Y., and Sato, I. Understanding the expressivity and trainability of fourier neural operator: A mean-field perspective. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?i...

    2024

  18. [26]

    On universal approximation and error bounds for fourier neural operators

    Kovachki, N., Lanthaler, S., and Mishra, S. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22 0 (290): 0 1--76, 2021

    2021

  19. [27]

    Neural operator: Learning maps between function spaces with applications to pdes

    Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24 0 (89): 0 1--97, 2023

    2023

  20. [28]

    and Dik, M

    Le, V.-A. and Dik, M. A mathematical analysis of neural operator behaviors. arXiv preprint arXiv:2410.21481, 2024

    2024

  21. [29]

    Codepde: An inference framework for llm-driven pde solver generation

    Li, S., Marwah, T., Shen, J., Sun, W., Risteski, A., Yang, Y., and Talwalkar, A. Codepde: An inference framework for llm-driven pde solver generation. arXiv preprint arXiv:2505.08783, 2025

    2025

  22. [30]

    Multipole graph neural operator for parametric partial differential equations

    Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., and Anandkumar, A. Multipole graph neural operator for parametric partial differential equations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural I...

    2020

  23. [31]

    B., Azizzadenesheli, K., liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A

    Li, Z., Kovachki, N. B., Azizzadenesheli, K., liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=c8P9NQVtmnO

    2021

  24. [32]

    Physics-informed neural operator for learning partial differential equations

    Li, Z., Zheng, H., Kovachki, N., Jin, D., Chen, H., Liu, B., Azizzadenesheli, K., and Anandkumar, A. Physics-informed neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 1 0 (3): 0 1--27, 2024

    2024

  25. [33]

    A large-scale exploration of -transfer

    Lingle, L. A large-scale exploration of -transfer. arXiv preprint arXiv:2404.05728, 2024

    2024 arXiv

  26. [34]

    PDE -net: Learning PDE s from data

    Long, Z., Lu, Y., Ma, X., and Dong, B. PDE -net: Learning PDE s from data. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 3208--3216. PMLR, 10--15 Jul 2018. URL ...

    2018

  27. [35]

    Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network

    Long, Z., Lu, Y., and Dong, B. Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network. Journal of Computational Physics, 399: 0 108925, 2019

    2019

  28. [36]

    Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3 0 (3): 0 218--229, 2021

    2021

  29. [37]

    Super consistency of neural network landscapes and learning rate transfer

    Noci, L., Meterez, A., Hofmann, T., and Orvieto, A. Super consistency of neural network landscapes and learning rate transfer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=rgwhJ7INtZ

    2024

  30. [38]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  31. [39]

    On the difficulty of training recurrent neural networks

    Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp.\ 1310--1318. Pmlr, 2013

    2013

  32. [40]

    Py T orch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Py T orch: An impe...

    2019

  33. [41]

    and Xie, S

    Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4195--4205, 2023

    2023

  34. [42]

    A., George, R

    Rahman, M. A., George, R. J., Elleithy, M., Leibovici, D., Li, Z., Bonev, B., White, C., Berner, J., Yeh, R. A., Kossaifi, J., Azizzadenesheli, K., and Anandkumar, A. Pretraining codomain attention neural operators for solving multiphysics PDE s. In The Thirty-eighth Annual Co...

    2024

  35. [43]

    Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378: 0 686--707, 2019

    2019

  36. [44]

    Saad, N., Gupta, G., Alizadeh, S., and Maddix, D. C. Guiding continuous operator learning through physics-based boundary constraints. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=gfWNItGOES6

    2023

  37. [45]

    and Spiliopoulos, K

    Sirignano, J. and Spiliopoulos, K. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375: 0 1339--1364, 2018

    2018

  38. [46]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  39. [47]

    High-dimensional probability: An introduction with applications in data science, volume 47

    Vershynin, R. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018

    2018

  40. [48]

    Is L^2 physics-informed loss always suitable for training physics-informed neural network? In Advances in Neural Information Processing Systems, 2022

    Wang, C., Li, S., He, D., and Wang, L. Is L^2 physics-informed loss always suitable for training physics-informed neural network? In Advances in Neural Information Processing Systems, 2022

    2022

  41. [49]

    Beyond closure models: Learning chaotic-systems via physics-informed neural operators

    Wang, C., Berner, J., Li, Z., Zhou, D., Wang, J., Bae, J., and Anandkumar, A. Beyond closure models: Learning chaotic-systems via physics-informed neural operators. arXiv preprint arXiv:2408.05177, 2024

    2024 arXiv

  42. [50]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

    Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024

    2024 arXiv

  43. [51]

    and Hu, E

    Yang, G. and Hu, E. J. Tensor programs IV : Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pp.\ 11727--11737. PMLR, 2021

    2021

  44. [52]

    J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

    Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V : Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022

    2022 arXiv

  45. [53]

    Tensor programs VI : Feature learning in infinite depth neural networks

    Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI : Feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=17pVDnpwwl

    2024

  46. [54]

    Pdeformer: Towards a foundation model for one-dimensional partial differential equations

    Ye, Z., Huang, X., Chen, L., Liu, H., Wang, Z., and Dong, B. Pdeformer: Towards a foundation model for one-dimensional partial differential equations. In ICLR 2024 Workshop on AI4DifferentialEquations In Science, 2024

    2024

  47. [55]

    Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B

    Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2 0 (3): 0 5, 2022

    2022 arXiv

  48. [56]

    Weak adversarial networks for high-dimensional partial differential equations

    Zang, Y., Bao, G., Ye, X., and Zhou, H. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411: 0 109409, 2020

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.