{"id":"facbe564-ef99-44f8-a9a5-efb1e59262cf","arxiv_id":"2506.19396","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors derive and test a parametrization for FNOs that keeps optimal hyperparameters stable as the number of Fourier modes grows, enabling zero-shot transfer from small proxy models to near-billion-parameter models.","lead":"This paper proposes a scaling scheme for Fourier Neural Operators that lets hyperparameters tuned on small models transfer to near-billion-parameter models without retuning. It derives a Maximal Update Parametrization for scaling the number of Fourier modes and tests it on Burgers, Darcy, and Navier-Stokes equations.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adam's normalized updates are bounded, so the max over K^d entries cannot grow like sqrt(d log K); Theorem 3.5's c(K)=1/sqrt(d log K) makes the feature update o(1) and does not establish μP.","rationale":"The reader's weakest assumption identifies the same load-bearing gap I find. The paper's empirical recipe appears useful and the code release is valuable, but the theoretical result as stated does not go through: the proof of Theorem 3.5 relies on the maximum of K^d independent C-sub-Gaussian Adam updates growing like sqrt(d log K), while Adam's normalized update is bounded by a constant. The paper itself notes in Section 3.3 that clipping bounds the updates; bounding is precisely what destroys the needed max-growth. Thus the derived c(K)=Theta(1/sqrt(d log K)) makes the per-step feature update o(1), violating the maximal-update condition in Definition 3.2. A corrected derivation would likely require c(K)=Theta(1) if updates are bounded, or a different argument that recovers a K-dependent scale from the gradient structure before Adam normalization. Neither is present. Since this is the central theoretical claim, the reader's REJECT verdict is appropriate; no change is needed. I do not see another concern that would alter the verdict, and I do not think a non-finding is honest here.","tokens_in":17596,"tokens_out":8093,"duration_ms":82500,"concrete_test":"Instrument the first Adam step in the released code for FNO-2D or FNO-3D at K=3,6,12,24: record the tensor g_t(R_ell) = m_hat/(sqrt(v_hat)+epsilon) before learning-rate multiplication and compute M(K) = max_k |g_t^k|. If M(K) is flat in K (e.g., roughly 1-10) rather than growing like sqrt(d log K), the max bound used in Appendix A.2 is contradicted empirically and the c(K) scaling makes the first feature update vanish as K grows. An analytic companion is to bound M(K) <= (1-beta_1)^{-1/2}(1-beta_2)^{-1/2} for the paper's beta_1=0.9, beta_2=0.999, showing the O(1) bound invalidates the proof line '||K_{ell,t+1}-K_{ell,t}||_2 = Theta(1)'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Theorem 3.5, whose proof in Appendix A.2 hinges on treating the entries of the Adam update tensor g_t(R_ell) as independent C-sub-Gaussian and then using max_{k in [K]^d} |g_t^k| = Theta(sqrt(d log K)) to cancel c(K) = Theta(1/sqrt(d log K)). The flaw is that for Adam, g_t is the normalized update m_hat/sqrt(v_hat)+epsilon, which is bounded entrywise by a constant independent of K (roughly 1/sqrt(1-beta_2) up to bias-correction factors). Bounded variables are sub-Gaussian, so Assumption 3.4 is not false, but the relevant maximum cannot grow like sqrt(d log K): the maximum of K^d bounded iid variables is O(1), not Theta(sqrt(d log K)). Consequently the per-step operator update satisfies ||Delta_t K_ell||_2 = eta_0 c(K) ||K_{g_t}||_2 = O(1/sqrt(d log K)) -> 0, whereas Definition 3.2 requires Delta_t w_ell = Theta(1). The initialization part using b(K) is unaffected because initial entries are truly Gaussian; the failure is specifically in the feature-learning update part. Section 3.3's claim that gradient clipping 'ensures their sub-Gaussianity' points in the wrong direction: clipping makes the bound constant and removes exactly the K-dependence of the maximum that the proof needs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes μTransfer-FNO, a zero-shot hyperparameter transfer method for Fourier Neural Operators (FNOs) when scaling the number of Fourier modes K. It defines an abc-parametrization for the kernel integral operator, and claims in Theorem 3.5 that a(K)=1, b(K)=c(K)=Θ(1/√(d log K)) is a Maximal Update Parametrization for FNO trained with Adam. The proof in Appendix A.2 analyzes the spectral norm of the kernel operator as the maximum of K^d sub-Gaussian entries, leading to the √(d log K) scaling. The paper then presents Algorithm 1 for hyperparameter transfer and reports experiments on Burgers' equation, Darcy flow, Navier-Stokes equations, and PINO, showing stable optimal learning rates and a 0.30x compute saving on a near-billion-parameter FNO.","tokens_in":17883,"tokens_out":8458,"duration_ms":77999,"significance":"If the main theorem were correct, the work would be a meaningful extension of the Maximal Update Parametrization framework to neural operators, introducing a new scaling law with a √(d log K) dependence and providing a principled zero-shot tuning recipe for large FNOs. The empirical study is extensive, covers several PDEs and training objectives, and includes a near-billion-parameter demonstration with released code. However, the central theoretical claim is not supported: the proof's key concentration step treats Adam update entries as unbounded sub-Gaussians whose maximum grows like Θ(√(d log K)), whereas Adam's normalized updates are actually bounded by a constant independent of K. This invalidates the derived learning-rate scaling and the proof of the maximal-update condition. The empirical results do not compensate for the unsupported central derivation, because the paper's contribution is framed as a mathematical derivation of the parametrization.","major_comments":[{"comment":"The proof of Theorem 3.5 relies on the step in Appendix A.2: 'by Assumption 3.4, using an argument similar to the above proof of stability at initialization, we have ∥Kℓ,t+1−Kℓ,t∥2 = Θ(1)'. This step requires that the maximum of the K^d entries of the Adam update tensor g_t(Rℓ) grows like Θ(√(d log K)). But for Adam, g_t is the normalized update m̂/(√v̂+ε), whose entries are bounded by a constant depending only on β1, β2, and ε, not on K (pointwise, |m̂| is bounded by a K-independent multiple of √v̂ up to bias-correction factors). Therefore the maximum of K^d entries is O(1) with high probability, not Θ(√(d log K)). With c(K)=Θ(1/√(d log K)), the per-step operator update satisfies ||Δ_t K_ℓ||_2 = η0 c(K)·O(1) = O(1/√(d log K)) = o(1), which violates the feature-learning condition Δ_t w_ℓ = Θ(1) in Definition 3.2. Assumption 3.4 is not false—bounded variables are sub-Gaussian—but the maximum concentration theorem used to justify the cancellation does not apply to bounded entries with a fixed proxy constant.","section":"Theorem 3.5 / Appendix A.2"},{"comment":"The discussion of Assumption 3.4 states that 'the sub-Gaussian condition can be explicitly enforced in practice through element-wise gradient clipping, which bounds the updates and thereby ensures their sub-Gaussianity'. This points in the wrong direction: clipping or the intrinsic normalization in Adam makes the entries of g_t uniformly bounded by a constant, forcing max_k |g_t^k| = O(1). The proof needs this maximum to be Θ(√(d log K)) to cancel c(K)=Θ(1/√(d log K)). Thus the proposed practical enforcement removes exactly the K-dependence that the proof's concentration argument requires, making the proof and the practical recommendation mutually inconsistent.","section":"Section 3.3"},{"comment":"Even under the paper's own definitions, the theorem does not establish the 'feature learning in every layer' condition. The induction step showing h_{ℓ,t}=Θ(1) depends on the invalid claim that ||Δ_t K_ℓ||_2 = Θ(1) per step. With the derived c(K)=Θ(1/√(d log K)) and bounded Adam update entries, the per-step spectral-norm update is O(1/√(d log K)), so after any fixed number of steps t the feature update is o(1), contradicting Definition 3.2's requirement that Δ_t w_ℓ = Θ(1). The parametrization in Theorem 3.5 therefore does not satisfy the maximal-update definition it is claimed to meet.","section":"Definition 3.2 / Theorem 3.5"}],"minor_comments":[{"comment":"The sentence 'We extends and generalizes the abc-Parametrization' has a grammatical error; it should be 'We extend and generalize'.","section":"Section 3.2"},{"comment":"The text reads 'for FNO-2D on the the incompressible Navier-Stokes Equation, the optimal learning rate align approximately at 4.2×10−3'; FNO-2D is used for Darcy Flow rather than Navier-Stokes, and 'the the' is a typo.","section":"Section 4.2"},{"comment":"The label in Figure 2c says 'Transfer-PINO' while the text consistently uses 'μTransfer-PINO'; the abstract similarly uses 'Transfer-FNO' without the μ prefix.","section":"Figure 2c"},{"comment":"The sentence 'the training dynamics and hence the parameter landscape across remain consistent as the model size scales under μP' is missing a noun after 'across'.","section":"Section 2.3"},{"comment":"The heading 'Regarding Assumption 3.4 and proactical applications' contains a typo: 'proactical' should be 'practical'.","section":"Section 3.3"},{"comment":"The caption 'Test relative error comparisonsbetween directly tuning' is missing a space: 'comparisons between'.","section":"Table 1 caption"}],"recommendation":"reject","confidential_remarks":"The flaw in the proof of Theorem 3.5 is load-bearing and not a presentation issue: the derived learning-rate scaling does not yield a Maximal Update Parametrization under Adam because the relevant maximum of the update tensor is O(1), not Θ(√(d log K)). The empirical study is extensive and may be of independent interest, but the paper's primary contribution is the claimed mathematical derivation, which is unsupported as written. A rejection is warranted unless the authors can prove the required maximum growth for Adam updates or substantially reframe the contribution as purely empirical."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2506.19396 (muTransfer for FNO). The paper is the first to attempt a muP derivation for Fourier Neural Operators under Fourier-mode scaling, and the claimed rate Theta(1/sqrt(d log K)) for both init variance and learning rate is new relative to the width/depth muP literature. The experiments are broad -- Burgers, Darcy, Navier-Stokes, plus PINO, batch size, beta2 -- and the code is released. If the recipe works, it would save real compute: the 0.30x training cost number on Navier-Stokes is attractive.\n\nBut the central theorem has a load-bearing gap. The proof (Appendix A.2) treats the Adam update tensor g_t(R_l) as C-sub-Gaussian and then uses the standard max-of-sub-Gaussians bound to say its spectral norm grows like sqrt(d log K). That step is where it breaks. Adam's normalized update m_hat/sqrt(v_hat+epsilon) is bounded entrywise by a constant that depends only on beta1 and beta2 (roughly 1/sqrt(1-beta2) after bias correction, up to a small factor). Bounded variables are sub-Gaussian, so Assumption 3.4 isn't false -- it's just insufficient. The maximum of K^d bounded variables is O(1), not Theta(sqrt(d log K)). Consequently, with c(K)=Theta(1/sqrt(d log K)), the per-step operator update is O(1/sqrt(d log K)) -> 0, which violates the paper's own Definition 3.2 that requires Theta(1) feature updates. The initialization side with b(K) is fine because the initial entries really are Gaussian; the failure is purely in the feature-learning update. The paper's Section 3.3 suggestion that gradient clipping enforces sub-Gaussianity makes this worse -- clipping bounds the entries and kills the K-dependence the proof needs.\n\nSo Theorem 3.5, as stated, is not proven. The empirical transfer might still be a useful heuristic -- the learning-rate landscapes in Figures 1-3 do look more stable under their recipe -- but the paper currently sells itself as a theoretical result, and that claim doesn't hold.\n\nMinor issues: no error bars or repeated seeds reported, and the abstract says \"billion-parameter\" while the largest model is 906M (the text later says \"near 1B\", which is honest). Citation patterns look fine; the relevant muP literature is cited.\n\nMy recommendation: send it to peer review, not desk reject. The problem is important, the empirical work is substantial, and a repair attempt is worth referee time. But I'd expect heavy revision: either fix the proof with a different argument (maybe using the raw gradient's sub-Gaussian structure before Adam normalization, if that can be made rigorous) or reframe the paper as an empirical method with a heuristic scaling rule and drop the theorem. As it stands, the central theoretical claim is unsupported.\n\nBest,\n[Your name]","headline":"The scaling analysis for FNO is genuinely new, but Theorem 3.5 builds on a false premise about Adam's update tensor: the entries are bounded, so the max cannot grow like sqrt(d log K), and the derived learning-rate scaling actually makes feature updates vanish.","tokens_in":18449,"tokens_out":6149,"would_cite":false,"duration_ms":55654,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that Fourier Neural Operators have a Maximal Update Parametrization under Fourier-mode scaling, making small-model hyperparameters transfer to billion-parameter FNOs.","keywords":["Fourier Neural Operators","operator learning","Maximal Update Parametrization","zero-shot hyperparameter transfer","Adam optimizer","kernel integral operator","PDE surrogates","Navier-Stokes equations"],"falsifier":"Measure the spectral norm of one Adam update to the Fourier-kernel tensor at $K=3$ and $K=24$ under the proposed scaling: the theorem predicts it stays $\\Theta(1)$ across $K$, while if the largest absolute entry of Adam's normalized update is empirically flat in $K$, the norm should shrink like $1/\\sqrt{\\log K}$ and the maximal-update condition fails.","tokens_in":17340,"feed_emoji":"🌊","tokens_out":14074,"duration_ms":124896,"temperature":0.7,"pith_summary":"This paper tries to establish that Fourier Neural Operators (FNOs), a standard deep-learning architecture for solving partial differential equations, have a Maximal Update Parametrization when the number of Fourier modes $K$ is scaled up. The paper derives the specific rule: as $K$ grows, the standard deviation of the kernel-integral parameter initialization and the learning rate for those parameters should both scale like $1/\\sqrt{d\\log K}$, where $d$ is the PDE dimension. If this rule is correct, the optimal hyperparameters found on a small, inexpensive FNO (learning rate, batch size, optimizer settings) remain optimal for models with far more Fourier modes, up to roughly a billion parameters, so no retuning is needed at scale. The paper validates the rule across Burgers' equation, Darcy flow, and Navier-Stokes, and reports that on Navier-Stokes the transferred configuration beats direct large-model tuning in accuracy while using 0.30x of the training compute. A correct maximal update parametrization would make large neural PDE solvers practical to train instead of prohibitively expensive to tune.","feed_headline":"A scaling rule makes FNO tuning transfer to billion-parameter models","feed_subtitle":"A small proxy Fourier neural operator's best settings carry to billion-parameter models at 0.30x training compute.","key_machinery":"The load-bearing object is the spectral norm of the Fourier kernel integral operator $\\mathcal{K}_\\ell$. For a discretized input, $\\mathcal{K}_\\ell$ has an eigen-decomposition via the discrete Fourier transform with a diagonal matrix, so $\\|\\mathcal{K}_\\ell\\|_2 = \\max_{k\\in[K]^d}|r_k^\\ell|$. This turns the analysis of feature growth into a question about the maximum of $K^d$ i.i.d. sub-Gaussian variables, which scales like $b(K)\\sqrt{d\\log K}$; requiring this norm to be $\\Theta(1)$ fixes $b(K)=\\Theta(1/\\sqrt{d\\log K})$. The same max-of-sub-Gaussians estimate is applied to the Adam update tensor, which yields the matching learning-rate scale $c(K)=\\Theta(1/\\sqrt{d\\log K})$ and guarantees that each Adam step changes the operator by $\\Theta(1)$. The generalized abc-parametrization (Definition 3.1) is the framework object that assigns the three scales to the kernel tensor, and Lemma A.3 is what connects the two norm conditions to the Maximal Update Parametrization definition.","core_discovery":"The paper's central claim is Theorem 3.5: under Assumptions 3.3 and 3.4, the abc-parametrization $a(K)=1$, $b(K)=c(K)=\\Theta(1/\\sqrt{d\\log K})$ is a Maximal Update Parametrization of the FNO with Adam when the number of Fourier modes $K$ grows. Here $b(K)$ scales the standard deviation of the kernel-tensor initialization, so the initialization variance is $\\Theta(1/(d\\log K))$, and $c(K)$ scales the master learning rate for that tensor. The paper derives this by computing the spectral norm of the kernel integral operator: after the discrete Fourier transform the operator is diagonal, so its norm is the largest absolute value among $K^d$ independent sub-Gaussian parameters, which grows like $\\sqrt{d\\log K}$; rescaling by that factor keeps hidden features $\\Theta(1)$ at initialization while still allowing each layer's feature update to be $\\Theta(1)$, the definition of maximal feature learning. The corollary the paper emphasizes is zero-shot transfer: with this parametrization, hyperparameters tuned on a small-$K$ proxy transfer to a large-$K$ FNO, and the paper reports stable optimal learning rates, batch sizes, and Adam $\\beta_2$ values across $K$, and on the Navier-Stokes benchmark a $5.34\\%$ relative error at $0.30\\times$ of the full-tuning compute.","pith_inferences":["A testable extension is to derive the analogous maximal update parametrization for other spectral or kernel-based operator architectures, such as wavelet-based, multipole, or transformer-based neural operators, whose parameter count also grows with the number of modes; the paper does not claim these, but the max-of-sub-Gaussians mechanism would apply directly.","A sharper version of the argument could replace Assumption 3.4 with the deterministic bound that Adam's normalized update already satisfies; this might recover a maximal-update parametrization with a different $K$-dependence, and checking it would settle the main theoretical vulnerability.","Because the derived scaling depends only on the logarithm of $K$, the transfer window may extend beyond the tested range $K=3$ to $K=24$; the paper's experiments do not establish the upper end, so this remains an open empirical question."],"forward_implications":["A practitioner can tune a small-$K$ proxy FNO and transfer the configuration to a large-$K$ FNO with no additional search; in the paper's FNO-3D experiment the optimal learning rate stays essentially fixed as the model grows from 1.7M to 906M parameters.","The recipe is explicit: when moving from proxy $K_p$ to target $K^*$, scale the kernel-tensor learning rate by $\\sqrt{\\log K_p/\\log K^*}$ and the initialization variance by $\\log K_p/\\log K^*$.","The scaling is independent of the discretization grid size $N_j$, so it does not break the resolution-agnostic nature of FNOs.","The same parametrization works with physics-informed objectives: $\\mu$Transfer-PINO stabilizes the learning-rate landscape on Darcy flow, showing the result does not depend on the supervised-loss form.","On the Navier-Stokes problem, the transferred configuration reaches $5.34\\%$ relative error versus $5.69\\%$ for direct full-model tuning while using $0.30\\times$ the compute."],"supporting_citations":[{"why":"Defines the abc-parametrization and the maximal feature-learning conditions that this paper extends to FNOs.","marker":"Yang & Hu, 2021"},{"why":"Establishes zero-shot hyperparameter transfer via muTransfer and the Adam scale-invariance used to set a(K)=1.","marker":"Yang et al., 2022"},{"why":"Supplies the sub-Gaussian maximum bound Theta(sqrt(d log K)) that yields the kernel operator's spectral norm.","marker":"Vershynin, 2018"},{"why":"Introduces the Fourier Neural Operator and the kernel integral operator whose K-scaling is the object of study.","marker":"Li et al., 2021"},{"why":"Defines Adam, the optimizer whose update scaling the theorem addresses.","marker":"Kingma & Ba, 2015"},{"why":"Formulates the feature-learning conditions that Lemma A.3 uses to translate norm bounds into the Maximal Update Parametrization definition.","marker":"Ishikawa & Karakida, 2024"}],"fun_headline_variants":["Scaling rule transfers FNO hyperparameters to billion-parameter models","μTransfer-FNO: tune small, transfer to billion-parameter FNOs","Zero-shot hyperparameter transfer for FNOs: tune small, run large","Scaling rule lets FNO hyperparameters transfer with zero tuning","Tune small FNO, get billion-scale at 0.30x compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-parameter optimizer updates have Gaussian-like tails, so the largest of the $K^d$ updates grows like $\\sqrt{d\\log K}$; Adam's normalized updates are in fact bounded, which would keep that maximum constant and could invalidate the derived scaling.","fun_headline_variants_meta":{"raw":{"variants":["Scaling rule transfers FNO hyperparameters to billion-parameter models","μTransfer-FNO: tune small, transfer to billion-parameter FNOs","Zero-shot hyperparameter transfer for FNOs: tune small, run large","Scaling rule lets FNO hyperparameters transfer with zero tuning","Tune small FNO, get billion-scale at 0.30x compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000899,"raw_usage":{"total_tokens":3909,"prompt_tokens":1018,"completion_tokens":2891,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":2793}},"tokens_in":634,"tokens_out":2891,"duration_ms":20534,"temperature":1.0,"reasoning_tokens":2793,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:33:11.716435+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the spectral norm of one Adam update to the Fourier-kernel tensor at $K=3$ and $K=24$ under the proposed scaling: the theorem predicts it stays $\\Theta(1)$ across $K$, while if the largest absolute entry of Adam's normalized update is empirically flat in $K$, the norm should shrink like $1/\\sqrt{\\log K}$ and the maximal-update condition fails.","supporting_citations":[{"cited_title":"and Hu, E","cited_arxiv_id":null,"evidence_quote":"Defines the abc-parametrization and the maximal feature-learning conditions that this paper extends to FNOs."},{"cited_title":"and Karakida, R","cited_arxiv_id":null,"evidence_quote":"Formulates the feature-learning conditions that Lemma A.3 uses to translate norm bounds into the Maximal Update Parametrization definition."}],"review_version":2}