Pith. sign in

REVIEW 4 major objections 6 minor 16 references

Are Deep Learning Based Hybrid PDE Solvers Reliable? Why Training Paradigms and Update Strategies Matter

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Deep learning-based hybrid PDE solvers can stall at false fixed points where updates vanish but residuals stay large; the paper shows that aligning training objectives and using residual-based acceleration restores reliable convergence.

desk verdict Plausible, useful empirical study of DL-HIM training/update strategies, but the central 'false fixed point' claim needs a longer-horizon check before it carries the weight the paper puts on it. read the letter →

arxiv 2602.06842 v2 pith:O5HCGOIG submitted 2026-02-06 math.NA cs.LGcs.NA

Yuhan Wu , Jan Willem van Beek , Victorita Dolean , Alexander Heinlein This is my paper · ORCID
classification math.NAcs.LGcs.NA MSC 65F1065N2268T07
keywords deeplearning-basedhybriditerativemethodsfalsefixedpointsAndersonaccelerationneuraloperatorsHelmholtzequationtrainingobjectivesphysics-awareconvergencestagnation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether deep learning-based hybrid iterative solvers (DL-HIMs), which alternate classical smoothers with neural operators, can be trusted to solve PDEs. It argues that their frequent stagnation arises not from network architecture but from two design choices: the training objective and the update rule. Through experiments on a DeepONet-based solver (HINTS) and an FFT-based solver (FNS), it shows that residual-based losses work best for the former while error-based losses are safer for the latter under static training, and that fixed step sizes or standard Anderson acceleration can push a nonlinear solver toward a false fixed point. The paper introduces physics-aware Anderson acceleration (PA-AA), which minimizes the physical residual rather than the update magnitude, and demonstrates that it lets HINTS converge to residuals around 1e-9 on Helmholtz problems where standard acceleration stalls above 1e-3. If correct, this reframes the reliability debate: hybrid solvers are not inherently unreliable, but demand physically informed training and iteration design.

What carries the argument

The key objects are the DL-HIM fixed-point map G_theta = M_{N_theta} composed with M_S^n (a cycle of n smoother steps followed by one neural correction) and the false-fixed-point phenomenon where delta_k = G_theta(u_k) - u_k tends to 0 while r_k = f - A u_k stays large. The main new mechanism is physics-aware Anderson acceleration (PA-AA), a multisecant acceleration that solves the constrained least-squares problem for coefficients alpha_j minimizing ||sum alpha_j (f - A g_{k-j})|| instead of ||sum alpha_j (g_{k-j} - u_{k-j})||, thereby forcing the accelerated iterate to reduce the PDE residual directly.

What would settle it

Run the HINTS iteration from Figure 1 for a much longer horizon (e.g., 10^5 cycles) and track the physical residual; if it eventually drops below, say, 1e-2, the plateau is slow convergence rather than a fixed point, which would weaken the mechanistic motivation for PA-AA.

Watch

Extended reading notes

Core claim

The central claim is that a DL-HIM is a fixed-point iteration u^{k+1} = G_theta(u^k) whose mathematical fixed points need not coincide with the PDE solution. Because the classical smoother and the neural operator have complementary spectral biases, there is a spectral gap: on fine grids the smoother barely moves low-frequency error, and the neural network may fail to correct certain modes, making the full-cycle update tiny while the physical residual r = f - A u remains large. This 'false fixed point' explains the stagnation plateau observed in HINTS. The paper substantiates this with a systematic comparison of training objectives and update strategies, and shows that PA-AA—which replaces th

Load-bearing premise

The load-bearing premise is that the plateau in Figure 1—where the update norm is about 1e-4 but the residual is about 1e1—is a genuine false fixed point, not merely slow convergence; the paper never tests whether the iteration would escape the plateau with longer runs or different tolerances.

Editorial extensions

If this is right

  • Practitioners should monitor the physical residual rather than the update norm when judging whether a hybrid solver has converged.
  • Training objectives should be chosen alongside the architecture: residual-based losses are a better default for DeepONet-based hybrids, while FFT-based solvers like FNS may need error-based objectives if trained statically.
  • The cost of dynamic training (unrolling K iterations) may not be justified: in these tests K=5 increased training time by about 8.6 times and memory by 1.27 times with little convergence benefit.
  • Standard Anderson acceleration can be counterproductive for nonlinear neural operators; a residual-aware variant (PA-AA) accelerates both linear and nonlinear solvers to much smaller residuals in fewer iterations.
  • The results give a concrete answer to the reliability debate: DL-HIMs are not inherently unreliable, but require coherent design of architecture, training objective, and update strategy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The false-fixed-point diagnosis suggests a general lesson: any learned preconditioner or iterator used inside a fixed-point loop should be certified by residual monitoring, not by update norm alone; this could apply to other learned iterative schemes beyond DL-HIMs.
  • Because PA-AA only needs the operator A and the residual f - A u, it is a drop-in acceleration for any nonlinear fixed-point iterator; testing it on classical nonlinear solvers (e.g., domain-decomposition or multigrid with inexact coarse solves) that exhibit stagnation would be a natural next step.
  • The paper compares static and dynamic training only for HINTS with a short horizon K=5; one could test whether longer unrolling horizons or finer target grids make dynamic training competitive, which the current cost analysis leaves open.
  • The observed preference of FNS for error-based losses under static training hints that the architecture's spectral operating mode changes how residual-based losses stress the network; a frequency-domain analysis of the loss landscape might predict such preferences a priori.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that the reliability of deep-learning-based hybrid iterative methods (DL-HIMs) for PDEs depends not only on neural architecture but also on training paradigms and update strategies. It formalizes a HINTS-type cycle as a fixed-point iteration and claims that a spectral gap between the classical smoother and the neural operator can produce 'false fixed points' where the full-cycle update vanishes while the physical residual remains large. It compares static and dynamic training, error- and residual-based losses, and several update strategies, including a new physics-aware Anderson acceleration (PA-AA) that minimizes the physical residual instead of the fixed-point update. Numerical experiments on one-dimensional diffusion and Helmholtz problems suggest that residual-based losses help DeepONet-based HINTS, that static residual losses can destabilize FNS, and that PA-AA breaks stagnation and reaches much smaller residuals than standard Anderson acceleration.

Significance. If the claims are fully substantiated, the paper makes a useful contribution to an active debate: it shifts attention from architecture design to the joint design of training objectives and iteration strategies, and it offers a simple, practical acceleration variant. The explicit comparison of two structurally different neural operators and the public code repository are strengths. However, the empirical support is currently narrower than the claims: the central false-fixed-point mechanism is inferred from a single plateau, no multiple seeds or error bars are reported, and the PA-AA advantage is demonstrated on single trajectories without a convergence theory or cost analysis. The paper is therefore promising but not yet conclusive.

major comments (4)
  1. [Section 5.2, Figure 1] The false-fixed-point diagnosis is not established. The update norm plateaus near 1e-4, not zero, and no long-horizon run is shown. For the same HINTS construction at N=201, Table 1 reports a final residual of 6.9e-5 after 700 cycles, so the N=801 plateau may be a long transient rather than a true fixed point. Since the motivation for PA-AA and the claim that standard AA is ill-suited both rest on the iteration being trapped, the authors should (i) run the unaccelerated iteration to a convergence floor over a much longer horizon, (ii) report multiple independent random-field instances/seeds, and (iii) if possible verify numerically that G(u_*) is indistinguishable from u_* while f - A u_* remains large. If the residual is still decreasing, the phenomenon should be relabeled slow convergence rather than a false fixed point.
  2. [Section 5.5, Figure 4 and Conclusion] The central PA-AA claim, 'converges robustly to 1e-9', is based on a single trajectory plot with no stopping criterion, no seeds, and no error bars. Because PA-AA minimizes ||f - A u||_2, the reported residual is the same objective being optimized; for an indefinite Helmholtz problem this may overstate accuracy, so solution-error curves are needed as an independent metric. The paper also does not report wall-clock time or per-iteration cost; PA-AA adds a least-squares solve and additional residual evaluations each cycle, so 'fewer iterations' does not yet demonstrate lower cost. Please report statistics over the test set, define a convergence threshold, include error curves, give a cost comparison in time or FLOPs, and state clearly whether PA-AA is heuristic or has convergence guarantees.
  3. [Section 5.4, Table 1] The cost-efficiency conclusion ('dynamic training should be adopted with caution') rests on a single comparison with K=5. The 8.63x training-time and 1.27x memory figures are reported without measurement methodology or variance, and the final errors/residuals differ by only a small factor (3.4e-7 vs 4.4e-7; 6.9e-5 vs 1.0e-4). These results do not eliminate the possibility that a larger unrolling horizon K would substantively improve convergence, nor do they establish that the static model's advantage is significant. The authors should either report multiple independent training runs with standard deviations or weaken the conclusion to a statement about this specific configuration.
  4. [Section 5.3, Figure 3 and Section 5.5] The claim that residual-based objectives are architecture-dependent is supported by only a single experiment per architecture, all in 1D. The FNS 'diverge' curves are not quantified (divergence threshold? blow-up or slow growth?), and no seeds are reported. Given the broad title and the conclusion about general DL-HIM reliability, the conclusions need either more systematic variation (problem dimension, n, wavenumber, loss norm, lambda, network initialization) or an explicit restriction of the claims to the tested 1D settings.
minor comments (6)
  1. [General] The paper is not consistently formatted; the opening sentence of Section 1 has missing spaces between words, and the Conclusion contains oddly capitalized headings such as 'T raining Objectives'. A careful copyedit is needed.
  2. [Eq. (5)] The notation 'δ(k) = 0⇐ ⇒r(k) = 0' is typeset confusingly; use a proper double implication symbol.
  3. [Algorithm 2] PA-AA is presented as new, but its structure is a residual-minimizing Anderson type. The discussion should relate PA-AA to existing nonlinear Krylov/Anderson variants and state precisely what is new.
  4. [Section 4.1] When the adaptive step size is used for non-SPD problems, the denominator p_k^T A p_k may vanish or change sign. The text says this is treated heuristically, but the experiments should report how such cases were handled numerically.
  5. [Section 5.1] The choice n=19 Jacobi steps per cycle is fixed throughout. A sensitivity test with respect to n would help determine whether the observed stagnation is robust or an artifact of this specific cycle split.
  6. [Section 5.1] Training hyperparameters such as network widths, training epochs, optimizer, learning rate, and the H1 loss weight lambda are not reported in the text. The code repository helps, but the manuscript should state these for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons, and the self-citations are not load-bearing.

full rationale

The paper's derivation chain is not circular. The DL-HIM iteration is defined as a fixed-point map (Eq. 4), the false-fixed-point phenomenon is an empirical observation (Fig. 1), and training losses are proxies for contraction factors (Eqs. 14–17). The introduction of PA-AA changes the least-squares objective from the fixed-point update to the physical residual (Algorithms 1 and 2), and then the paper evaluates this choice by comparing against fixed step size, adaptive step size, and standard AA (Fig. 4). This is an algorithmic design choice tested empirically, not a fitted parameter renamed as a prediction. PA-AA does minimize, by construction, the physical residual over the current affine subspace at each step, but the reported convergence to 1e-9 is a global dynamical outcome that is not forced by that local minimization; standard AA also minimizes its own residual locally and yet stagnates above 1e-3 in the same experiments. Thus the comparison provides independent evidence. The only self-citation is reference [4], a standard domain-decomposition textbook (Dolean et al.), and it is used only as background on classical preconditioners, not as load-bearing support for the paper's central claims. The concern that the plateau in Fig. 1 might be slow convergence rather than a true false fixed point is a correctness or robustness question about the empirical diagnosis, not a circularity of the derivation. No equation in the paper is shown to be equivalent to its own input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

This is an empirical paper with no new physical entities. The free parameters are primarily experimental hyperparameters that shape the reported comparisons. The most notable assumption is the interpretation of the stagnation plateau as a false fixed point, which underpins the motivation for PA-AA.

free parameters (5)
  • PA-AA memory size m = 10
    Chosen because it was 'sufficient' for speeding up convergence; no sensitivity study; results may depend on m.
  • Jacobi steps per DL-HIM cycle n = 19
    Empirically set without ablation; affects smoother-neural spectral balance.
  • Dynamic training unrolling horizon K = 5
    Used in cost-efficiency comparison; conclusions about dynamic training may be K-dependent.
  • H1 loss weight λ = not specified
    Weights high-frequency penalty in H1-type losses; no value given, affecting reproducibility of loss-function comparisons.
  • Data generation constants (μ_k, k_min, σ_k, l_k, σ_f, l_f) = diffusion (1.0,0.3,0.3,0.1; σ_f=1.0, l_f=0.1), Helmholtz (8.0,3.0,2.0,0.2)
    Define the problem distribution; not fitted but directly shape which PDE instances are tested.
assumptions (5)
  • domain assumption The parametric PDE (1) is well posed and has a unique solution
    Stated in §2.1; ensures discrete linear system Au=f is solvable.
  • domain assumption Neural networks exhibit spectral bias: low frequencies learned faster than high frequencies
    Invoked in §1 and §5.2 to justify the complementary spectral role of the neural operator versus the classical smoother (citing [10]).
  • ad hoc to paper The observed stagnation plateau (Figure 1) is a false fixed point rather than merely slow convergence
    The paper does not prove the iteration converges to a fixed point with nonzero residual; the plateau is interpreted as a false fixed point, and this interpretation motivates PA-AA.
  • standard math The least-squares problems in AA and PA-AA are well-posed and stable
    The algorithms assume a solution to the constrained least-squares problem at each step; standard Anderson theory provides background but is not re-derived.
  • domain assumption Residual-based losses trained on coarse grids transfer to the fine-grid inference distribution
    Training uses N_train=31 and inference on N=101 to 801; the paper assumes the neural operator generalizes across resolutions without explicit verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are Deep Learning Based Hybrid PDE Solvers Reliable? Why Training Paradigms and Update Strategies Matter." pith.science (2026). https://pith.science/paper/O5HCGOIG

@misc{pith2026260206842,
  author       = {Pith},
  title        = {Pith review of: Are Deep Learning Based Hybrid PDE Solvers Reliable? Why Training Paradigms and Update Strategies Matter},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5HCGOIG}},
  note         = {Machine review of arXiv:2602.06842}
}
read the original abstract

Deep learning-based hybrid iterative methods (DL-HIMs) integrate classical numerical solvers with neural operators, utilizing their complementary spectral biases to accelerate convergence. Despite this promise, many DL-HIMs stagnate at false fixed points where neural updates vanish while the physical residual remains large, raising questions about reliability in scientific computing. In this paper, we provide evidence that performance is highly sensitive to training paradigms and update strategies, even when the neural architecture is fixed. Through a detailed study of a DeepONet-based hybrid iterative numerical transferable solver (HINTS) and an FFT-based Fourier neural solver (FNS), we show that significant physical residuals can persist when training objectives are not aligned with solver dynamics and problem physics. We further examine Anderson acceleration (AA) and demonstrate that its classical form is ill-suited for nonlinear neural operators. To overcome this, we introduce physics-aware Anderson acceleration (PA-AA), which minimizes the physical residual rather than the fixed-point update. Numerical experiments confirm that PA-AA restores reliable convergence in substantially fewer iterations. These findings provide a concrete answer to ongoing controversies surrounding AI-based PDE solvers: reliability hinges not only on architectures but on physically informed training and iteration design.

Figures

Figures reproduced from arXiv: 2602.06842 by the authors.

Figure 1
Figure 1. Update and Residual Norms of HINTS on the 1D Diffusion Equation with [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Convergence Behavior of HINTS with DeepONet Models Trained in the Static Training [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Convergence Behavior of FNS Statically Trained by Residual-Based and Error-Based [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Convergence of DL-HIMs and Classic Numerical Methods with Different Update [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 2 linked inside Pith

  1. [1]

    Number 13

    Henk A Van der Vorst.Iterative Krylov methods for large linear systems. Number 13. Cambridge University Press, 2003

    2003

  2. [2]

    Preconditioning techniques for large linear systems: a survey.Journal of computational Physics, 182(2):418–477, 2002

    Michele Benzi. Preconditioning techniques for large linear systems: a survey.Journal of computational Physics, 182(2):418–477, 2002

    2002

  3. [3]

    Oosterlee, and Anton Schuller.Multigrid

    Ulrich Trottenberg, Cornelius W. Oosterlee, and Anton Schuller.Multigrid. Elsevier, November 2000

    2000

  4. [4]

    SIAM, 2015

    Victorita Dolean, Pierre Jolivet, and Frédéric Nataf.An introduction to domain decompo- sition methods: algorithms, theory, and parallel implementation. SIAM, 2015

    2015

  5. [5]

    Chebyshev semi-iterative methods, successive over- relaxation iterative methods, and second order Richardson iterative methods.Numerische Mathematik, 3(1):157–168, 1961

    Gene H Golub and Richard S Varga. Chebyshev semi-iterative methods, successive over- relaxation iterative methods, and second order Richardson iterative methods.Numerische Mathematik, 3(1):157–168, 1961

    1961

  6. [6]

    Anderson acceleration for fixed-point iterations.SIAM Journal on Numerical Analysis, 49(4):1715–1735, 2011

    Homer F Walker and Peng Ni. Anderson acceleration for fixed-point iterations.SIAM Journal on Numerical Analysis, 49(4):1715–1735, 2011

    2011

  7. [7]

    Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural net- works: A deep learning framework for solving forward and inverse problems involving non- linear partial differential equations.Journal of Computational physics, 378:686–707, 2019

    2019

  8. [8]

    Fourier neural operator for parametric partial differential equations.arXiv preprint arXiv:2010.08895, 2020

    Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations.arXiv preprint arXiv:2010.08895, 2020

    arXiv 2010

Show all 16 references
  1. [9]

    Learn- ing nonlinear operators via DeepONet based on the universal approximation theorem of operators.Nature machine intelligence, 3(3):218–229, 2021

    Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learn- ing nonlinear operators via DeepONet based on the universal approximation theorem of operators.Nature machine intelligence, 3(3):218–229, 2021

    2021

  2. [10]

    Ham- precht, Yoshua Bengio, and Aaron Courville

    Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Ham- precht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks, May 2019. arXiv:1806.08734 [cs, stat]

    2019 arXiv

  3. [11]

    Blending neural operators and relaxation methods in PDE numerical solvers.Nature Machine Intelligence, pages 1–11, 2024

    Enrui Zhang, Adar Kahana, Alena Kopaničáková, Eli Turkel, Rishikesh Ranade, Jay Pathak, and George Em Karniadakis. Blending neural operators and relaxation methods in PDE numerical solvers.Nature Machine Intelligence, pages 1–11, 2024. 16

    2024

  4. [12]

    A hybrid iterative method based on mionet for PDEs: Theory and numerical examples.Mathematics of Computation, 2025

    Jun Hu and Pengzhan Jin. A hybrid iterative method based on mionet for PDEs: Theory and numerical examples.Mathematics of Computation, 2025

    2025

  5. [13]

    MIONet: Learning multiple-input operators via tensor product.SIAM Journal on Scientific Computing, 44(6):A3490–A3514, 2022

    Pengzhan Jin, Shuai Meng, and Lu Lu. MIONet: Learning multiple-input operators via tensor product.SIAM Journal on Scientific Computing, 44(6):A3490–A3514, 2022

    2022

  6. [14]

    Learning singularity-encoded Green’s functions with application to iterative methods.arXiv preprint arXiv:2509.11580, 2025

    Qi Sun, Shengyan Li, Bowen Zheng, Lili Ju, and Xuejun Xu. Learning singularity-encoded Green’s functions with application to iterative methods.arXiv preprint arXiv:2509.11580, 2025

    2025

  7. [15]

    A hybrid iterative neural solver based on spectral analysis for parametric PDEs.Journal of Computational Physics, page 114165, 2025

    Chen Cui, Kai Jiang, Yun Liu, and Shi Shu. A hybrid iterative neural solver based on spectral analysis for parametric PDEs.Journal of Computational Physics, page 114165, 2025

    2025

  8. [16]

    Deeponet based preconditioning strate- gies for solving parametric linear systems of equations.SIAM Journal on Scientific Com- puting, 47(1):C151–C181, 2025

    Alena Kopaničáková and George Em Karniadakis. Deeponet based preconditioning strate- gies for solving parametric linear systems of equations.SIAM Journal on Scientific Com- puting, 47(1):C151–C181, 2025. 17

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.