Pith. sign in

REVIEW 4 major objections 6 minor 29 references

Attention-based hybrid solvers for linear equations that are geometry aware

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Masked self-attention lets a pretrained DeepONet precondition unseen Helmholtz geometries.

desk verdict A real new architecture with a supported core claim, but the transferability claim needs a resolution caveat. read the letter →

arxiv 2411.13341 v1 pith:EFDBHBBM submitted 2024-11-20 math.NA cs.NA

Idan Versano , Eli Turkel This is my paper · ORCID
classification math.NAcs.NA MSC 65F1065N2268T0765N06
keywords HelmholtzequationpreconditionerDeepONetgeometrytransferabilitymaskedself-attentionhybriditerativesolverGauss-SeidelGMRES
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a learned preconditioner for the Helmholtz equation can be trained on a few simple geometries and then solve linear systems on very different geometries without retraining. The payoff would be that expensive geometry-specific data generation is unnecessary: one network, trained on rectangles and simple polygons, can sit inside an iterative solver and make it converge on domains with cracks, bumps, and holes. The paper reports that its masked-attention DeepONet converges in cases where the vanilla DeepONet with zero extension diverges, and that a simple CNN-based geometry-aware variant performs worse. If the claim holds, it would give a cheap and robust preconditioning option for non-symmetric, non-positive-definite PDE problems.

What carries the argument

The load-bearing object is the masked attention operation $M$-attention$(v, M^\Omega) = \mathrm{softmax}((vv^T + M^\Omega)/\sqrt{N})v$, where $M^\Omega$ is $0$ inside the domain and $-\infty$ outside. It lets the branch network see only grid points belonging to the current geometry, avoiding any artificial zero extension of $f$ outside $\Omega$. The branch and trunk networks form a DeepONet-style product, and the whole network is used as the preconditioner $N$ inside the HINTS update: every $J$ classical iterations (Gauss-Seidel or GMRES) are followed by one normalized network correction.

What would settle it

Run HINTS-Gauss-Seidel with $N_{\mathrm{masked}}$ on a domain containing a crack whose width is less than one training-grid spacing ($1/15$) while the test resolution is $h=1/57$; if the crack falls entirely between the 225 fixed training points, the masking matrix $M^\Omega$ carries no inside-point information there. The experiment would falsify the paper's claim if the method then behaves like the zero-extension $N_{\mathrm{non-masked}}$ and fails to converge, since it would show the coarse grid, not the mask, is responsible for geometry transfer.

Watch

Extended reading notes

Core claim

The paper claims that a DeepONet preconditioner equipped with masked self-attention ($N_{\mathrm{masked}}$) can be trained only on simple rectangular and polygonal geometries, then used inside the HINTS hybrid iteration to solve Helmholtz systems on different, more complicated geometries without retraining. It reports that on domains with abrupt boundary changes such as cracks, bumps, and holes, $N_{\mathrm{masked}}$ converges in cases where the vanilla DeepONet with zero extension diverges, and that a naive CNN-based geometry-aware extension usually performs worst. The central discovery is that masking the attention so that points outside the domain are ignored is enough to make geometry transfer work, even though the network input remains a fixed 15-by-15 grid.

Load-bearing premise

The load-bearing premise is that every geometry, no matter how complicated, can be represented adequately by the same fixed 15-by-15 grid, and that interpolating the network input transfers the trained network to the much finer test grids.

Editorial extensions

If this is right

  • A HINTS user can train once on simple geometries and deploy the same network as a preconditioner for new geometries, avoiding the cost of generating training data for many domains.
  • For domains with narrow cracks or holes, the masked model should be chosen over vanilla DeepONet with zero extension, since the mask removes the outside-domain values that mislead the network.
  • Because the network is non-convolutional and its inputs are point values, the same architecture should apply to non-rectangular grids and point clouds, not only uniform Cartesian grids.
  • Combined with Gauss-Seidel, the approach keeps a solver that would otherwise diverge converging; combined with GMRES, it reduces both total iterations and the number of expensive network calls.
  • Each network evaluation costs constant computation because the input dimension is fixed at 225 points, so the hybrid iteration remains $O(N)$ per step for sparse systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the masking idea should transfer to other indefinite or nonsymmetric second-order PDEs, such as convection-diffusion or time-harmonic Maxwell problems, because the architecture never assumes symmetry or definiteness.
  • The paper leaves open how the method behaves for features finer than the 15-by-15 training grid; a crack that falls entirely between the 225 fixed points may be invisible to the mask, so a natural next test would use a finer training grid or an adaptive point selection.
  • A practical user currently has to keep both the masked and non-masked networks and choose per geometry, since neither dominates in all tables; unifying or automatically switching between them is a plausible follow-up suggested by the numerical results.
  • Because the network acts on residuals only every $J$ steps, the experiments suggest that a moderately accurate operator network suffices to steer convergence, which could loosen accuracy requirements for geometry-transfer training in other applications.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a geometry-aware DeepONet preconditioner that combines masked self-attention with the HINTS iterative framework for solving Helmholtz-type linear systems. The network is trained on a fixed 15x15 grid over [0,1]^2 and evaluated on different geometries without fine-tuning, using Richardson iterations hybridized with Gauss-Seidel or GMRES. The main reported evidence is a set of iteration-count tables on 12 polygonal domains and several obstacle/exterior domains, where the masked model converges in cases where vanilla DeepONet or classical iterations diverge.

Significance. If the transferability claim is supported, the architecture would be a useful addition to the toolkit of learned preconditioners, especially for users who cannot rely on convolutional grids. The paper has the merit of evaluating on out-of-distribution geometries against independent baselines (vanilla DeepONet, GMRES, a CNN-based geometry-aware variant) and makes code available. However, the significance is currently limited by the fixed-grid geometry representation, the absence of a protocol for the user-selected skip factor, and the lack of statistical and timing evidence. The contribution is empirical and would become substantially stronger with controlled experiments.

major comments (4)
  1. [Section 2.1 and Section 3.1] The geometry-aware inputs O^Ω, f^Ω, and M^Ω are all defined on the same fixed 15x15 grid used in training. Geometric features that fall strictly between the 225 grid points, such as narrow cracks or small holes, are therefore invisible to the network, and the mask degenerates to the same zero-extension input used by N_non-masked. Section 3.1 claims that N_masked performs best for geometries with abrupt changes (cracks/bumps/holes), but this can only hold if the tested features are resolved by h=1/14. No such resolution condition is stated or tested. Remark 1.1 addresses interpolation of the input function f, but not refinement of the geometry representation. For the h=1/57 and h=1/224 test grids, none of the training grid points coincide with test grid points, so the mechanism by which the fixed-grid branch is applied to a test residual is unspecified. The paper should either demonstrate that the reported narrow-crack results are insensitive to the fixed sampling grid or restrict the transferability claim to features resolvable on the training grid.
  2. [Equation (2.3), Tables 1-5] The method depends critically on the user-selected skip factor J and preconditioner scale α, but no protocol for choosing these parameters is given. The tables use different J values across geometries (e.g., J=40, 60, and 80 in Tables 1-2; J=15 and 100 in Tables 4-5), and the qualitative ranking of methods changes with J: Section 3.1 reports that for the Figure 4 geometries with GMRES, vanilla is better at J=15, whereas at J=5 H-GMRES with vanilla diverges while N_masked converges. Without a fixed hyperparameter protocol or a sensitivity analysis, the comparison across models is not controlled, and the robustness claim is weaker than stated.
  3. [Tables 1-5] The paper reports only averaged iteration counts and, with one exception (Table 1, geometry 5: 1620 ± 170), gives no error bars, number of runs, seeds, or per-run statistics. The word "averaged" is not sufficient to support a robustness claim. In addition, no wall-clock times are reported despite Remark 3.2 discussing CPU time; iteration count alone is insufficient because the networks have different per-iteration costs. Multi-run statistics and timings should be provided for the main comparisons.
  4. [Section 3.1 and Abstract] The paper's own tables show that vanilla DeepONet is often faster or similar in iteration count (e.g., Table 1 geometries 3, 4, 5, 8; Table 2 geometries 10, 11, 12; Table 3 geometries 1, 4, 11). The abstract claims that the proposed network 'remains a robust preconditioner' and Section 3.1 states N_masked 'performed better than all methods' for a subset of geometries, but the conditions under which the masked model is preferable are not characterized. The central claim should be scoped precisely: the experiments support an existence claim (there are geometries where N_masked converges when vanilla diverges), not a general superiority claim. A criterion or at least a discussion of when the masked model is expected to help would be needed to support the broader wording.
minor comments (6)
  1. [Abstract and Section 1.2] The abstract says 'a simple geometry' but the training data in Figure 1 contain multiple geometries; the plural wording in Section 1.2 is more accurate and should be used consistently.
  2. [Equations (1.7) and (2.3)] Equation (1.7) writes the network update as un+1 = un + D(rn), while (2.3) uses a normalized argument std(rn)/α · N(α rn/std(rn)); the notation should be aligned to avoid confusion about what D is.
  3. [Tables 4 and 5] The H-GMRES rows in Tables 4 and 5 do not specify the restart parameter m, unlike Table 3 which states m=30. This makes the results difficult to reproduce and should be corrected.
  4. [Figure 2 and Tables 1-2] The tables refer to 'geometry 1' through 'geometry 12' but Figure 2 labels the domains as 'Polygon 1' through 'Polygon 12'; the mapping should be made explicit in the text or captions.
  5. [Remark 1.1] Remark 1.1 states that using grids that contain the training grid is less expensive, but the test grids h=1/57 and h=1/224 do not contain the training grid points; the remark should be reconciled with the actual evaluation procedure.
  6. [Title] There is a typo in the title on the first page: 'a re' should be 'are'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the masked-attention DeepONet is trained on simple reference geometries and evaluated on disjoint test geometries and forcings against independent baselines.

full rationale

The paper's central claim is that a masked-attention DeepONet, trained only on the simple geometries of Fig. 1, remains a robust preconditioner on the unseen geometries of Figs. 2-4 without fine-tuning. The evaluation is genuinely out-of-sample: training forcings are Gaussian random fields with kernel (2.4), while test forcings are drawn from a normal distribution with mean 10 and variance 10 (Sec. 3), and the test domains and grid resolutions (h=1/57, h=1/224) differ from the training grid (h=1/14). The geometry-aware inputs O^Omega, f^Omega, and M^Omega are computed from the test geometry at evaluation time, but the network weights are fixed. No equation defines the predicted quantity in terms of the training target, and no fitted parameter is renamed as a prediction. The HINTS update (2.3) and the vanilla DeepONet baseline are taken from prior work [27], which includes a co-author, but they serve as the solver framework and baseline rather than as the justification for the geometry-transferability claim; the novel masked architecture is compared against them instead of being derived from them. The user-selected skip factor J and scaling alpha are explicitly acknowledged as non-trained hyperparameters (Sec. 2.3), so their per-test selection is a comparison-protocol concern, not circularity: the network output itself is not fitted to the test data. The paper does contain genuine limitations that are weighed but are not circular: Remark 1.1 addresses interpolation of the input function but not refinement of the fixed 15x15 geometry representation, and Remark 3.2 reports that higher-resolution H-GS is slow or diverges on the authors' machine. These are unproven-generalization and implementation risks, not reductions of the result to its inputs. Accordingly, no circular step meeting the quoted-reduction standard is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim does not introduce new physical or mathematical entities. It rests on standard numerical analysis background (well-posed Helmholtz problem, GMRES convergence, high-frequency smoothing by Gauss-Seidel), on the HINTS framework from [27], and on user-chosen hyperparameters. The most notable modeling choice is representing every domain on the same 15x15 grid with signed distances and a mask, then interpolating for finer grids. The per-geometry skip factor J acts as a hand-fitted parameter in the reported convergence results.

free parameters (4)
  • skip factor J = J=40,60,80,100,200 per geometry; J=10 for H-GMRES
    User-selected in (2.3); iteration counts and convergence depend on it; no fixed protocol or sensitivity study.
  • preconditioner scale alpha = 0.3
    Factor in (2.3) normalizing residual to training distribution; fixed for all tests; no sensitivity analysis.
  • Gaussian random field parameters l and sigma = l = sigma = 0.1
    Training data covariance (2.4); chosen by hand.
  • training hyperparameters = 1000 epochs, batch size 64, initial LR 1e-4 decay 0.5
    Appendix A.2; optimizer and seeds unspecified.
assumptions (5)
  • domain assumption Helmholtz problem (1.4) with absorbing BC is well-posed for k^2=21 on piecewise smooth domains.
    Invoked in Section 1.1 citing [4]; not rederived.
  • domain assumption HINTS with a DeepONet preconditioner converges for linear systems when the network approximates the inverse; spectral bias balances low and high frequency errors.
    Carried over from [27] and [17]; central to why (2.3) should work.
  • domain assumption Gauss-Seidel reduces high-frequency errors even when the overall iteration diverges.
    Stated in Section 3 to justify H-GS comparisons; not proven here.
  • ad hoc to paper A fixed 15x15 grid plus interpolation faithfully represents finer test grids and arbitrary subdomains.
    The architecture is fixed-size; transfer to h=1/57 and h=1/224 relies on this assumption.
  • domain assumption Complex Helmholtz solutions are handled by applying the same real network to real and imaginary parts.
    Remark 2.1; no analysis of cross-coupling.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attention-based hybrid solvers for linear equations that are geometry aware." pith.science (2026). https://pith.science/paper/EFDBHBBM

@misc{pith2026241113341,
  author       = {Pith},
  title        = {Pith review of: Attention-based hybrid solvers for linear equations that are geometry aware},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EFDBHBBM}},
  note         = {Machine review of arXiv:2411.13341}
}
read the original abstract

We present a novel architecture for learning geometry-aware preconditioners for linear partial differential equations (PDEs). We show that a deep operator network (Deeponet) can be trained on a simple geometry and remain a robust preconditioner for problems defined by different geometries without further fine-tuning or additional data mining. We demonstrate our method for the Helmholtz equation, which is used to solve problems in electromagnetics and acoustics; the Helmholtz equation is not positive definite, and with absorbing boundary conditions, it is not symmetric.

Figures

Figures reproduced from arXiv: 2411.13341 by the authors.

Figure 1
Figure 1. Geometries used for training . on the geometries in [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Tested domains Ω used in Tables 1, 2 and 3 for solving (1.2). [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Tested exterior domains used in Table 4 for solving ∆ [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Tested exterior domains used in Table 5 solving ∆ [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    and Miranda, R., 1995

    Alciatore, D. and Miranda, R., 1995. A winding number and point-in- polygon algo- rithm. Glaxo Virtual Anatomy Project Research Report, Departm ent of Mechanical Engineering, Colorado State University

    work page 1995

  2. [2]

    and Manteuffel, T., 2005

    Baker, A.H., Jessup, E.R. and Manteuffel, T., 2005. A technique fo r accelerating the convergence of restarted GMRES. SIAM Journal on Matrix Analysis and Applications, 26(4), pp.962-984

    work page 2005

  3. [3]

    and Nataf, F., 2015

    Dolean, V., Jolivet, P. and Nataf, F., 2015. An introduction to dom ain decomposition methods: algorithms, theory, and parallel implementation. Society for Industrial and Applied Mathematics

    work page 2015

  4. [4]

    Finite Elements II: Galerkin Approximation, Elliptic and Mixed PDEs

    Alexandre Ern, Jean-Luc Guermond. Finite Elements II: Galerkin Approximation, Elliptic and Mixed PDEs. Springer, 2021, 10.1007/978-3-030-56923 -5 . hal-03226050

    work page doi:10.1007/978-3-030-56923 2021

  5. [5]

    and Gander, M.J., 2011

    Ernst, O.G. and Gander, M.J., 2011. Why it is difficult to solve Helmholt z problems with classical iterative methods. Numerical analysis of multiscale pro blems, pp.325- 363

    work page 2011

  6. [6]

    and Spence, E.A., 2015

    Gander, M.J., Graham, I.G. and Spence, E.A., 2015. Applying GMRES to the Helmholtz equation with shifted Laplacian preconditioning: what is the largest shift for which wavenumber-independent convergence is guaranteed? . Numerische Mathe- matik, 131(3), pp.567-614

    work page 2015

  7. [7]

    and Zhang, H., 2019

    Gander, M.J. and Zhang, H., 2019. A class of iterative solvers for the Helmholtz equa- tion: Factorizations, sweeping preconditioners, source transfer, single layer potentials, polarized traces, and optimized Schwarz methods. Siam Review, 61( 1), pp.3-76

    work page 2019

  8. [8]

    and Nataf, F., 2002

    Gander, M.J., Magoules, F. and Nataf, F., 2002. Optimized Schwar z methods without overlap for the Helmholtz equation. SIAM Journal on Scientific Comp uting, 24(1), pp.38-60

    work page 2002

Show all 29 references
  1. [9]

    Optimized schwarz methods

    Gander, M.J., 2006. Optimized schwarz methods. SIAM Journal o n Numerical Anal- ysis, 44(2), pp.699-731

    2006

  2. [10]

    and Zareei, A., 2024

    Gladstone, R.J., Rahmani, H., Suryakumar, V., Meidani, H., D’Elia, M. and Zareei, A., 2024. Mesh-based GNN surrogates for time-independent PDEs. Scientific Reports, 14(1), p.3394

    2024

  3. [11]

    and Karniadakis, G.E., 2022

    Goswami, S., Kontolati, K., Shields, M.D. and Karniadakis, G.E., 2022 . Deep trans- fer operator learning for partial differential equations under con ditional shift. Nature Machine Intelligence, 4(12), pp.1155-1164. 20

    2022

  4. [12]

    and Jasiuk , I., 2023

    He, J., Koric, S., Kushwaha, S., Park, J., Abueidda, D. and Jasiuk , I., 2023. Novel DeepONet architecture to predict stresses in elastoplastic struc tures with variable complex geometries and loads. Computer Methods in Applied Mechanic s and Engi- neering, 415, p.116277

    2023

  5. [13]

    and Jasiuk, I., 2024

    He, J., Kushwaha, S., Park, J., Koric, S., Abueidda, D. and Jasiuk, I., 2024. Sequential Deep Operator Networks (S-DeepONet) for predicting full-field so lutions under time- dependent loads. Engineering Applications of Artificial Intelligence, 127, p.107258

    2024

  6. [14]

    and Jasiuk, I., 2024

    He, J., Koric, S., Abueidda, D., Najafi, A. and Jasiuk, I., 2024. Ge om-DeepONet: A Point-cloud-based Deep Operator Network for Field Predictions on 3D Parameterized Geometries. arXiv preprint arXiv:2403.14788

    2024 arXiv

  7. [15]

    a nd Pathak, J.,

    Kahana, A., Zhang, E., Goswami, S., Karniadakis, G., Ranade, R. a nd Pathak, J.,

  8. [16]

    and Guibas, L.J., 2021

    Kashefi, A., Rempe, D. and Guibas, L.J., 2021. A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries. Physics of F luids, 33(2)

    2021

  9. [17]

    and Karniadakis, G.E., 2024

    Kopanicakova, A. and Karniadakis, G.E., 2024. Deeponet based precondition- ing strategies for solving parametric linear systems of equations. a rXiv preprint arXiv:2401.02016

    2024 arXiv

  10. [18]

    On the Schwarz alternating method

    Lions, P.L., 1990, March. On the Schwarz alternating method. I II: a variant for nonoverlapping subdomains. In Third international symposium on do main decom- position methods for partial differential equations (Vol. 6, pp. 202 -223). Philadelphia: SIAM

    1990

  11. [19]

    and Karniadakis, G.E., 2021

    Lu, L., Jin, P., Pang, G., Zhang, Z. and Karniadakis, G.E., 2021. Le arning nonlinear operators via DeepONet based on the universal approximation the orem of operators. Nature machine intelligence, 3(3), pp.218-229

    2021

  12. [20]

    and Courville, A., 2019, May

    Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprech t, F., Bengio, Y. and Courville, A., 2019, May. On the spectral bias of neural networks. In International conference on machine learning (pp. 5301-5310). PMLR

    2019

  13. [21]

    and Ding, C., 2019

    Raza, S. and Ding, C., 2019. Progress in context-aware recom mender systems—An overview. Computer Science Review, 31, pp.84-97

    2019

  14. [22]

    and Polosukhin, I., 2017

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., G omez, A.N., Kaiser, /suppress L. and Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems, 30. 21

    2017

  15. [23]

    and Schultz, M.H., 1986

    Saad, Y. and Schultz, M.H., 1986. GMRES: A generalized minimal re sidual algorithm for solving nonsymmetric linear systems. SIAM Journal on scientific and statistical computing, 7(3), pp.856-869

    1986

  16. [24]

    and Maggioni, M., 2024

    Yin, M., Charon, N., Brody, R., Lu, L., Trayanova, N. and Maggioni, M., 2024. Dimon: Learning solution operators of partial differential equations on a d iffeomorphic family of domains. arXiv preprint arXiv:2402.07250

    2024 arXiv

  17. [25]

    GMRES algorithms over 35 years

    Zou, Q., 2023. GMRES algorithms over 35 years. Applied Mathema tics and Compu- tation, 445, p.127869

    2023

  18. [26]

    and Tay, Y., 2019

    Zhang, S., Yao, L., Sun, A. and Tay, Y., 2019. Deep learning base d recommender system: A survey and new perspectives. ACM computing surveys ( CSUR), 52(1), pp.1-38

    2019

  19. [27]

    Zhang, A

    E. Zhang, A. Kahana, A. Kopanicakova, E. Turkel, R. Ranade, J. Pathak and G.E. Karniadakis ”Blending Neural Operators and Relaxation Methods in P DE Numerical Solvers” accepted Nature Machine Intelligence

  20. [28]

    and Lu, L., 2023

    Zhu, M., Feng, S., Lin, Y. and Lu, L., 2023. Fourier-DeepONet: Fourier-enhanced deep operator networks for full waveform inversion with improved accuracy, generalizability, and robustness. arXiv preprint arXiv:2305.17289. 22

    2023 arXiv

  21. [2023]

    Computational Mechanics, 72(3), pp.471- 484

    On the geometry transferability of the hybrid iterative nume rical solver for differential equations. Computational Mechanics, 72(3), pp.471- 484

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.