arxiv: 2605.08318 · v1 · submitted 2026-05-08 · 💻 cs.LG · cs.AI· cs.NA· math.NA· physics.comp-ph· stat.ML

Recognition: no theorem link

When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains

Brandon Yee , Pairie Koh , Jack Rodriguez , Mihir Tekal

Authors on Pith no claims yet

Pith reviewed 2026-05-12 01:04 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.NAmath.NAphysics.comp-phstat.ML

keywords multi-scale attention transformerPDE solvingirregular domainsneural operatorsFourier neural operatorphysics-informed regularizationapproximation error bounds

0 comments

The pith

Multi-scale attention transformers outperform Fourier operators on PDEs with complex irregular domains.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper investigates when transformer architectures with attention outperform Fourier-based neural operators for solving partial differential equations on irregular domains. It proposes the Multi-Scale Attention Transformer, which represents solution histories as tokens and learns attention across scales, optionally incorporating physics constraints. Empirical tests on five benchmarks show superior accuracy on complex geometries and much faster inference than competing models. Theoretical bounds link prediction error to boundary complexity, explaining the performance differences and guiding model choice. The findings highlight a tradeoff where physics regularization aids simple diffusion but harms more turbulent problems.

Core claim

MSAT encodes spatiotemporal solution histories as token sequences and uses learned attention to solve PDEs, achieving an L2 relative error of 0.0101 on Heat2D-CG, a 3.7 times improvement over FNO. It runs inference in 34 seconds total compared to 120812 seconds for Mamba-NO. Approximation error bounds as a function of domain boundary complexity kappa provide a theoretical basis for when attention beats Fourier methods and a rule for architecture selection.

What carries the argument

The Multi-Scale Attention Transformer (MSAT) that encodes PDE solution histories as token sequences with multi-scale attention mechanisms and optional physics-informed regularization.

If this is right

Error bounds grow with boundary complexity kappa, favoring attention-based models for high-complexity domains.
Physics regularization improves accuracy on diffusion-dominated problems but degrades it on chaotic and recirculating-flow regimes.
MSAT achieves state-of-the-art generalization on complex geometry problems in the PINNacle benchmarks.
Inference is dramatically faster than state-space models like Mamba-NO.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The kappa-based selection rule could be implemented in PDE solver software to automatically pick between attention and Fourier architectures.
The regularization tradeoff suggests adaptive or regime-detecting physics priors for broader applicability.
Extending the analysis to three-dimensional or highly nonlinear PDEs would test the robustness of the error bounds.
Attention mechanisms may offer similar advantages in other scientific ML tasks involving irregular or unstructured data.

Load-bearing premise

The five PINNacle benchmark problems with their fixed train/test splits and reference data are representative of real irregular-domain PDEs.

What would settle it

Observing whether the L2 relative error on a new problem with quantified boundary complexity kappa follows the paper's approximation bound or whether MSAT fails to outperform FNO on high-kappa domains.

read the original abstract

We study the problem of \emph{architecture selection} for deep learning models trained to solve partial differential equations (PDEs), asking when transformer-based architectures with learned attention outperform Fourier-domain neural operators. We introduce the \textbf{Multi-Scale Attention Transformer} (\msat{}), a deep learning architecture that encodes spatiotemporal solution histories as token sequences and trains end-to-end via a composite supervised objective with optional physics-informed regularization terms. We conduct a comprehensive empirical evaluation against nine baselines -- including physics-informed neural networks (PINNs), neural operators (FNO, DeepONet, GNOT), and state-space models (Mamba-NO) -- across five benchmark problems from the PINNacle suite, using identical train/test splits and reference data for all methods. \msat{} achieves state-of-the-art generalization on complex geometry problems ($L^2_\mathrm{rel} = 0.0101$ on Heat2D-CG, a $3.7\times$ improvement over FNO) at $34\,\mathrm{s}$ total inference vs.\ $120{,}812\,\mathrm{s}$ for Mamba-NO. Ablation studies over the physics regularization component reveal a precise inductive bias tradeoff: physics priors reduce test error on diffusion-dominated problems but degrade generalization on chaotic and recirculating-flow regimes, directly characterizing the prior misspecification boundary. Approximation error bounds as a function of domain boundary complexity $\kappa$ provide a theoretical basis for these empirical findings and a principled rule for architecture selection.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

MSAT shows a practical edge for attention-based models over Fourier operators on irregular PDE domains, backed by controlled benchmarks and a regularization tradeoff, though the supporting bounds need closer scrutiny.

read the letter

The paper's core result is that their Multi-Scale Attention Transformer beats FNO by a factor of 3.7 on the Heat2D-CG problem from PINNacle, reaching L2_rel of 0.0101 while running inference in 34 seconds total versus over 120k seconds for Mamba-NO. They also map out when optional physics regularization helps versus hurts, and tie performance to a boundary complexity measure kappa with some error bounds. That combination gives a usable rule for picking architectures on complex geometries.

Referee Report

2 major / 2 minor

Summary. The paper introduces the Multi-Scale Attention Transformer (MSAT) for solving PDEs on irregular domains. It claims superior performance over Fourier-based methods like FNO on complex geometry problems from the PINNacle suite, with L^2_rel = 0.0101 on Heat2D-CG (3.7× improvement), faster inference times, ablations showing tradeoffs in physics-informed regularization, and approximation error bounds depending on domain boundary complexity κ to guide architecture selection.

Significance. If the empirical results and theoretical bounds hold, this provides a significant contribution to neural operator literature by offering both a new architecture and a principled way to choose between attention and Fourier approaches based on geometry complexity. The detailed ablations on regularization misspecification are particularly useful for practitioners.

major comments (2)

[Theoretical analysis] The approximation error bounds as a function of κ are central to the architecture selection rule; the manuscript must clarify whether these are derived rigorously from first principles or involve empirical fitting, including all assumptions (theoretical analysis section).
[§4 Experiments] While the paper uses identical train/test splits and reference data for all methods, the representativeness of the five PINNacle benchmarks for broader irregular-domain PDEs should be justified more explicitly, as this underpins the generalization claims (§4 Experiments).

minor comments (2)

Ensure all baseline implementations (including Mamba-NO and GNOT) are detailed sufficiently for reproducibility, including any hyperparameter choices.
[Abstract] The inference time comparison (34s vs 120,812s) is striking; confirm if this includes only inference as stated or any preprocessing.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive recommendation of minor revision and for the constructive comments. These points help improve the clarity of our theoretical and experimental claims. We address each major comment below.

read point-by-point responses

Referee: [Theoretical analysis] The approximation error bounds as a function of κ are central to the architecture selection rule; the manuscript must clarify whether these are derived rigorously from first principles or involve empirical fitting, including all assumptions (theoretical analysis section).

Authors: We thank the referee for this important clarification request. The bounds presented in the theoretical analysis section are derived rigorously from first principles using approximation theory for attention-based operators on irregular domains. Specifically, we start from the Lipschitz continuity of the PDE solution operator and bound the approximation error of the multi-scale attention mechanism in terms of the domain boundary complexity measure κ, with all constants obtained analytically. No empirical fitting is performed. The derivation assumes (i) bounded domain irregularity (finite κ), (ii) Lipschitz continuity of the solution map, and (iii) sufficient smoothness of the input functions. We will revise the theoretical analysis section to explicitly enumerate these assumptions and restate the rigorous derivation steps. revision: yes
Referee: [§4 Experiments] While the paper uses identical train/test splits and reference data for all methods, the representativeness of the five PINNacle benchmarks for broader irregular-domain PDEs should be justified more explicitly, as this underpins the generalization claims (§4 Experiments).

Authors: We agree that an explicit justification of benchmark representativeness will strengthen the generalization claims. The five PINNacle problems were chosen precisely because they span a wide range of boundary complexities (low to high κ) and PDE regimes (diffusion, advection, and chaotic flows), directly testing the architecture selection rule. In the revised §4 we will add a dedicated paragraph that (i) summarizes the geometric and physical diversity of the suite, (ii) references the original PINNacle paper for benchmark construction details, and (iii) explains why these instances capture the essential challenges of irregular-domain PDE solving, thereby supporting broader applicability. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper's central claims rest on direct empirical comparisons across fixed benchmark splits (PINNacle suite), ablation studies on regularization tradeoffs, and reported approximation bounds in terms of boundary complexity κ. No load-bearing derivation reduces by the paper's own equations to a fitted input renamed as prediction, nor to a self-citation chain that is itself unverified. The architecture selection rule is presented as following from the empirical and bounding results rather than being presupposed by them. This is the expected outcome for a primarily empirical architecture-comparison study with stated splits and reference data.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; the model is described as a standard transformer with multi-scale token encoding and optional physics terms.

pith-pipeline@v0.9.0 · 5591 in / 1387 out tokens · 60657 ms · 2026-05-12T01:04:57.097351+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

[1]

Journal of Computational Physics , volume =

Raissi, Maziar and Perdikaris, Paris and Karniadakis, George Em , title =. Journal of Computational Physics , volume =. 2019 , doi =

work page 2019
[2]

and Anandkumar, Anima , title =

Li, Zongyi and Kovachki, Nikola Borislavov and Azizzadenesheli, Kamyar and Liu, Burigede and Bhattacharya, Kaushik and Stuart, Andrew M. and Anandkumar, Anima , title =. International Conference on Learning Representations , year =

work page
[3]

Nature Machine Intelligence , volume =

Lu, Lu and Jin, Pengzhan and Pang, Guofei and Zhang, Zhongqiang and Karniadakis, George Em , title =. Nature Machine Intelligence , volume =. 2021 , doi =

work page 2021
[4]

SIAM Review , volume =

Lu, Lu and Meng, Xuhui and Mao, Zhiping and Karniadakis, George Em , title =. SIAM Review , volume =. 2021 , doi =

work page 2021
[5]

International Conference on Machine Learning , year =

Hao, Zhongkai and Wang, Zhengyi and Su, Hang and Ying, Chengyang and Dong, Yinpeng and Liu, Songming and Cheng, Ze and Song, Jian and Zhu, Jun , title =. International Conference on Machine Learning , year =

work page
[6]

Advances in Neural Information Processing Systems , year =

Hao, Zhongkai and Yao, Jiachen and Su, Chang and Su, Hang and Wang, Ziao and Lu, Fanzhi and Xia, Zeyu and Zhang, Yichi and Liu, Songming and Lu, Lu and Zhu, Jun , title =. Advances in Neural Information Processing Systems , year =

work page
[7]

SIAM Journal on Scientific Computing , volume =

Wang, Sifan and Teng, Yujun and Perdikaris, Paris , title =. SIAM Journal on Scientific Computing , volume =. 2021 , doi =

work page 2021
[8]

Computer Methods in Applied Mechanics and Engineering , volume =

Wang, Sifan and Sankaran, Shyam and Perdikaris, Paris , title =. Computer Methods in Applied Mechanics and Engineering , volume =. 2024 , doi =

work page 2024
[9]

and Gholami, Amir and Zhe, Shandian and Kirby, Robert M

Krishnapriyan, Aditi S. and Gholami, Amir and Zhe, Shandian and Kirby, Robert M. and Mahoney, Michael W. , title =. Advances in Neural Information Processing Systems , volume =

work page
[10]

Journal of Machine Learning Research , volume =

Kovachki, Nikola and Li, Zongyi and Liu, Burigede and Azizzadenesheli, Kamyar and Bhattacharya, Kaushik and Stuart, Andrew and Anandkumar, Anima , title =. Journal of Machine Learning Research , volume =

work page
[11]

and Kaiser,

Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser,. Attention Is All You Need , booktitle =

work page
[12]

Conference on Language Modeling , year =

Gu, Albert and Dao, Tri , title =. Conference on Language Modeling , year =

work page
[13]

2024 , doi =

Li, Zongyi and Zheng, Hongkai and Kovachki, Nikola and Jin, David and Chen, Haoxuan and Liu, Burigede and Azizzadenesheli, Kamyar and Anandkumar, Anima , title =. 2024 , doi =

work page 2024
[14]

Advances in Neural Information Processing Systems , volume =

Cao, Shuhao , title =. Advances in Neural Information Processing Systems , volume =

work page
[15]

Advances in Computational Mathematics , volume =

Moseley, Ben and Markham, Andrew and Nissen-Meyer, Tarje , title =. Advances in Computational Mathematics , volume =. 2023 , doi =

work page 2023