Pith. sign in

REVIEW 5 major objections 6 minor 67 references

Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MANO treats self-attention as an n-body interaction and computes it across scales, keeping a global receptive field with linear time and memory.

desk verdict MANO is a plausible FMM-style attention for 2D grids with open code and competitive accuracy, but the headline efficiency claims are asserted from asymptotics, not measured, and the baselines are not apples-to-apples. read the letter →

arxiv 2507.02748 v1 pith:WWK5VIB4 submitted 2025-07-03 cs.CV cs.AIcs.LG

Alex Colagrande , Paul Caillon , Eva Feillet , Alexandre Allauzen This is my paper
classification cs.CVcs.AIcs.LG
keywords multipoleattentionfastmethodlinearneuraloperatorDarcyflowvisiontransformermulti-scaleglobalreceptivefield
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MANO, a Multipole Attention Neural Operator that casts self-attention as an n-body interaction problem between grid points and accelerates it with a Fast Multipole Method style hierarchy. Instead of a single quadratic attention pass, MANO runs the same windowed attention at several progressively downsampled resolutions and sums the results, so every head keeps a global receptive field at $O(N)$ time and memory. The authors claim this matches or beats ViT and Swin Transformer on fine-grained image classification and on Darcy-flow operator learning, while cutting runtime and peak memory by orders of magnitude. If true, attention-based models become practical for high-resolution images and PDE grids without sacrificing global context.

What carries the argument

The mechanism is the Fast Multipole Method analogy: near-field interactions run at full resolution inside windows, while far-field interactions are captured by progressively coarser summaries of the input. Concretely, MANO builds levels $X_0$ through $X_L$ with a shared downsampling convolution, computes windowed softmax attention at each level, upsamples each attended level with a shared transposed convolution, and sums all levels. Because the same kernels and attention weights are reused at every scale, the total cost is dominated by the finest level, giving $O(N M d)$ for $N$ tokens, window size $w$ ($M=w^2$), and embedding dimension $d$.

What would settle it

Train MANO on Darcy flow or CIFAR-100 with randomly initialized attention weights instead of pretrained SwinV2 weights, and compare against a second variant with per-level (not shared) convolution kernels; if the multiscale advantage over single-scale attention disappears or per-level kernels do just as well, the shared scale-agnostic projector is what carries the result.

Watch

Extended reading notes

Core claim

The paper's central claim is that one attention-weight set, applied inside sliding windows on a pyramid of feature maps built by shared convolutional downsampling, gives every head global context at linear cost. The same convolutional kernel produces every coarser level, and the same transposed-convolution kernel returns each level to full resolution, so the parameter count does not grow with the number of scales. Attention maps learned at the finest scale are assumed to stay effective at coarser scales, which is what lets MANO inherit pretrained SwinV2 attention weights and fine-tune only the sampling convolutions. Empirically, MANO outperforms SwinV2-T and several ViT baselines on five of six linear-probing benchmarks and roughly halves the relative MSE of FNO and patch-ViT on Darcy flow at every tested resolution.

Load-bearing premise

The load-bearing premise is that the same downsampling and upsampling convolution kernels, together with the same attention weights, are scale-agnostic: an attention map learned at the finest scale remains effective at every coarser scale, and no experiment trains the attention weights from scratch to isolate this assumption.

Editorial extensions

If this is right

  • High-resolution images and fine grids become tractable for attention-based models, since memory and time scale linearly in the number of tokens.
  • MANO can be inserted into an existing Swin-style backbone: pretrained attention weights are reused, and only the shared sampling convolutions are trained, adding about 2.7% parameters.
  • On Darcy flow, the same architecture yields roughly half the relative MSE of FNO and patch-based ViT at every tested resolution, with sharper reconstructions.
  • Because cost is dominated by the finest scale, adding coarser levels is nearly free and improves accuracy, so users can trade levels against accuracy without changing asymptotic complexity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the shared-convolution design keeps parameters constant in the number of levels, the same mechanism could in principle extend to 3D grids, where the multipole hierarchy would cut cost from quadratic in voxel count to linear.
  • A clean way to test the transfer claim is to evaluate a MANO model trained at 32x32 directly on 64x64 and 128x128 Darcy inputs without retraining; if accuracy degrades, the scale-agnostic premise is weaker than assumed.
  • The classification gains come from a linear-probing protocol with frozen attention; training attention from scratch would show whether the benefit is the multiscale structure itself or the inherited SwinV2 features.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes MANO (Multipole Attention Neural Operator), an attention mechanism inspired by the Fast Multipole Method that computes windowed self-attention at multiple downsampled levels of the input grid, using shared down/up-sampling convolutions and shared attention weights across levels. The authors claim that this maintains a global receptive field in each head while achieving linear time and memory complexity in the number of grid points, and present experiments on image classification (with a frozen SwinV2-T backbone plus linear probing) and Darcy flow regression. The paper also open-sources the code.

Significance. If the claims held, MANO would be a valuable contribution: it connects FMM ideas to attention, provides a clean asymptotic derivation (Eqs. 5-6), and reports large accuracy gains over baseline transformers in a linear-probing setup as well as lower relative MSE on Darcy flow. The authors also include an ablation comparing average pooling with learned convolutions and release code, which are positive reproducibility elements. However, the headline efficiency claim ('reducing runtime and peak memory usage by orders of magnitude') is not supported by any measurement, and the empirical comparisons are not fully controlled, as MANO adds trainable convolutions while baselines are frozen. The central idea is plausible and the paper is worth revising, but the evidence as presented is insufficient for the strength of the claims.

major comments (5)
  1. [Section 3.2, Eqs. (5)-(6) and Appendix B.2 (Table 4)] The complexity analysis assumes non-overlapping windows, but the Darcy flow experiments use overlapping windows with stride 1 and window size 2 (Appendix Table 4). With stride 1, the number of windows per level is N_l rather than N_l/M, so the per-level cost is O(N_l M^2 d), making the total O(N M^2 d) rather than O(N M d) as stated in Eq. (6). For M=4 and k=2, this changes the constant factor from 4 to 16 per level and weakens the claim that coarser levels add negligible overhead. Please either restrict the complexity statement to non-overlapping windows or give the correct constants for the overlapping case used in the physics experiments.
  2. [Abstract and Section 7 (Conclusion)] The abstract and conclusion state that 'Empirical results ... demonstrate ... reducing runtime and peak memory usage by orders of magnitude,' but the manuscript contains no wall-clock time, peak memory, FLOPs, or resolution-sweep measurements anywhere. Table 1 lists the same O(N) asymptotic complexity for SwinV2-T and MANO, so the asymptotic analysis cannot support an orders-of-magnitude advantage over SwinV2-T; against ViT, it would only appear at sufficiently large N. Please add actual runtime and memory benchmarks, or revise the claim to state that the complexity is linear and that runtime/memory gains are expected but not yet measured.
  3. [Section 4.1 and Table 1] The classification comparison is not controlled: MANO adds 740,356 trainable parameters (the down/up-sampling convolutions) and a warm-up phase to train them, while all baselines are linear-probed with completely frozen backbones. The large accuracy improvements over SwinV2-T (e.g., 10 points on CIFAR-100) could therefore come from the extra trainable parameters or the additional training budget, not from the multipole attention mechanism itself. Please add a controlled baseline (e.g., SwinV2-T with a comparable number of trainable parameters added, or MANO with attention weights fine-tuned) and an ablation that removes the learned convolutions, so the effect of the multipole attention is isolated.
  4. [Section 3.2 and Figure 3] The paper's parameter-efficiency and transfer story relies on the assumption that shared convolutions act as 'scale-agnostic projectors' and that an attention map learned at the finest scale remains effective at coarser scales, even with frozen attention weights. No experiment directly tests this assumption: the ablation in Figure 3 compares average pooling versus learned convolutions, but both use the same pretrained attention weights, so it does not reveal whether the attention weights themselves need to be retrained or fine-tuned for coarser levels. Please add an experiment that trains the attention weights from scratch or fine-tunes them on the target task and compares with the frozen-attention configuration, which would also help interpret the transfer results in Table 1.
  5. [Section 5.2, Table 2] The Darcy flow results are reported without error bars, seeds, or the number of runs, and the training details for baselines (FNO and ViT) are not fully specified. The improvements over the next-best method (e.g., 0.0020 vs 0.0038 at 32x32) may be within run-to-run variance. Please report the mean and standard deviation over at least 3-5 seeds, and state the data split and training protocols for all models.
minor comments (6)
  1. [Section 5.1] The text says 'The results for TinyViT are taken from [58], while all other baselines are fine-tuned via linear probing using pretrained backbones' but later states 'The reported results for SwinV2-T are taken from [36]'—please clarify which numbers come from the authors' own linear probing and which are cited from prior work.
  2. [Section 5.1] There is a duplicated phrase: 'the dataset with the smallest training set among those considered' appears twice in the discussion of Flowers-102.
  3. [Figure 1 caption] The caption says 'The higher the level, the shorter the range of the interaction,' but in the described hierarchy, coarser levels (higher ell) correspond to larger effective ranges after upsampling to the original resolution; please correct this description.
  4. [Section 3.2] Typo: 'whithin' should be 'within' in the sentence following Eq. (5); also 'paramters' in Section 4.1, and 'expect'/'accross' in Section 5.1 should be 'except'/'across'.
  5. [Section 5.2] The complexity of FNO is written as 'O(N^2 log N) (for an N x N grid)'—this is confusing because N is used for both the side length and the number of tokens elsewhere; using M = N^2 grid points and writing O(M log M) would be clearer.
  6. [Appendix B.1 (Table 3)] The number-of-levels hyperparameter is listed as [3, 2, 1, 1], but the text in Section 5.2 (hyperparameters paragraph) describes this as '3 levels across the first two layers, 2 levels for the next two layers'—please make the stage/layer terminology consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MANO's linear-complexity and global-receptive-field claims follow directly from a self-contained multiscale construction (Eqs. 5–6), and the benchmark results in Tables 1–2 are external evaluations; the unmeasured 'orders of magnitude' runtime claim is a verification gap, not a circular step.

full rationale

The paper's derivation chain is self-contained. The O(N) complexity claim follows from the geometric-series argument in Eq. (6), which sums O(N/k^{2l} M d) over levels to O(N M d) for fixed window size M and downsample factor k; no fitted parameter enters, and Table 1 assigns the same O(N) class to SwinV2-T. The global receptive field is a direct consequence of the multiscale construction (at the coarsest level a single window covers the entire grid) and is presented as a design property, not as an empirical prediction that is then confirmed by the model's own output. The benchmark results are externally grounded: Table 1 uses linear probing of standard pretrained checkpoints (ViT-base, DeiT-small, SwinV2-T) and literature values (TinyViT), and Table 2 reports relative MSE on the standard Darcy-flow benchmark of [25] with FNO, ViT, and local-attention baselines trained under the same protocol; no reported metric is a function of the paper's own claims. There are no self-citations (no reference shares an author with this paper) and no uniqueness theorem or ansatz is imported from the authors' prior work. Reviewing-rule flags that are verification gaps, not circularity: (i) the Abstract and Conclusion claim MANO reduces 'runtime and peak memory usage by orders of magnitude,' but the manuscript contains no runtime table, memory trace, FLOPs comparison, or resolution sweep; since MANO and SwinV2-T are both O(N), the asymptotic analysis of Eqs. (5)–(6) alone cannot justify that claim over SwinV2-T (the multiscale sum adds only a constant factor, roughly 1.33 for k=2), so the efficiency claim is unmeasured. (ii) Section 3.2 asserts that shared convolutions act as 'scale-agnostic projectors'; this assumption is supported only indirectly by the frozen-attention transfer results in Table 1 and the pooling-vs-convolution ablation in Figure 3. (iii) Appendix A.2 states that a neural operator should exhibit discretization convergence, yet Table 2 trains and evaluates at matched resolutions only, so that property is not demonstrated. These gaps lower confidence in the headline efficiency and resolution-agnostic claims but do not constitute circularity; hence score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on the design choices above: the FMM analogy is qualitative, the uniform-grid assumption is explicit, the shared-convolution scale-transfer premise is unproven, and performance depends on hand-tuned levels and window sizes. No formal guarantees are provided beyond the non-overlapping-window complexity calculation.

free parameters (4)
  • number_of_levels = image: [3, 2, 1, 1]; Darcy: maximum possible, coarsest 2x2
    Authors tuned the number of levels, found the maximum best in vision, and state that in physics the number of levels can be treated as a hyperparameter (Section 5.2 and Appendix B.1).
  • window_size = 8 for image classification, 2 for Darcy flow
    Window size controls the locality of attention; it is chosen per task (Tables 3 and 4 in Appendix B).
  • attention_sampling_rate = 2
    Feature maps are downsampled by a factor of 2 between multipole levels (Appendix B.1 and B.2).
  • down_up_conv_kernel_stride = kernel 2, stride 2 for image; kernel 2, stride 1 for Darcy
    Down and up sampling convolutions have these kernels; the authors report learned convolutions outperform pooling, but no kernel-size sweep is shown (Appendix B.1, B.2, Figure 3).
assumptions (5)
  • domain assumption Attention can be approximated by local windowed attention on progressively coarsened grids, analogous to Fast Multipole Method far-field summaries.
    Invoked in Sections 3.1 and 3.2; no error bound or approximation guarantee is given for the softmax attention kernel.
  • domain assumption A single shared convolution and deconvolution pair maps features between scales in a scale-agnostic way.
    Section 3.2 claims the shared kernels act as scale-agnostic projectors; without this, the fixed parameter-count and weight-sharing claims fail.
  • ad hoc to paper Pretrained SwinV2 attention weights remain effective when applied to downsampled feature maps after fine-tuning only the convolution parameters.
    Section 4.1 freezes the encoder and trains only the MANO convolutions and classifier; this premise is load-bearing for Table 1 but is not tested by an ablation that trains attention weights from scratch.
  • domain assumption Input data live on uniform Cartesian grids.
    Explicitly acknowledged in Section 6 Limitations; all method and complexity statements depend on regular grids.
  • domain assumption Complexity can be analyzed in the non-overlapping window regime even though the implementation uses overlapping sliding windows.
    Section 3.2 states 'We analyze the cost under non-overlapping windows'; the overlap in experiments (Darcy window size 2, stride 1) changes the constant factor and violates the analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics." pith.science (2026). https://pith.science/paper/WWK5VIB4

@misc{pith2026250702748,
  author       = {Pith},
  title        = {Pith review of: Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWK5VIB4}},
  note         = {Machine review of arXiv:2507.02748}
}
abstract

Transformers have become the de facto standard for a wide range of tasks, from image classification to physics simulations. Despite their impressive performance, the quadratic complexity of standard Transformers in both memory and time with respect to the input length makes them impractical for processing high-resolution inputs. Therefore, several variants have been proposed, the most successful relying on patchification, downsampling, or coarsening techniques, often at the cost of losing the finest-scale details. In this work, we take a different approach. Inspired by state-of-the-art techniques in $n$-body numerical simulations, we cast attention as an interaction problem between grid points. We introduce the Multipole Attention Neural Operator (MANO), which computes attention in a distance-based multiscale fashion. MANO maintains, in each attention head, a global receptive field and achieves linear time and memory complexity with respect to the number of grid points. Empirical results on image classification and Darcy flows demonstrate that MANO rivals state-of-the-art models such as ViT and Swin Transformer, while reducing runtime and peak memory usage by orders of magnitude. We open source our code for reproducibility at https://github.com/AlexColagrande/MANO.

Figures

Figures reproduced from arXiv: 2507.02748 by the authors.

Figure 1
Figure 1. (Left) The multi-scale grid structure. (Center) The V-cycle structure for computing multipole attention with the fast multipole [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Darcy flow reconstruction: from left to right—input coefficient field, ground truth solution, MANO prediction, and ViT predic [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation study comparing average pooling (green) and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 52 canonical work pages

  1. [1]

    Univer- sal physics transformers: A framework for efficiently scaling neural operators

    Benedikt Alkin, Andreas F ¨urst, Simon Schmid, Lukas Gru- ber, Markus Holzleitner, and Johannes Brandstetter. Univer- sal physics transformers: A framework for efficiently scaling neural operators. Advances in Neural Information Process- ing Systems, 37:25152–25194, 2024. 2, 3, 12

    work page 2024

  2. [2]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Rep- resentations, ICLR 2015, 2015. 1

    work page 2015

  3. [3]

    Bruinsma, Ana Lucic, Megan Stanley, Anna Allen, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan A

    Cristian Bodnar, Wessel P. Bruinsma, Ana Lucic, Megan Stanley, Anna Allen, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan A. Weyn, Haiyu Dong, Jayesh K. Gupta, Kit Thambiratnam, Alexander T. Archibald, Chun- Chieh Wu, Elizabeth Heider, Max Welling, Richard E. Turner, and Paris Perdikaris. A foundation model for the earth system. Nature, 64...

    work page 2025

  4. [4]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pages 446–461. Springer, 2014. 5

    work page 2014

  5. [5]

    Learning a neural solver for parametric PDEs to en- hance physics-informed methods

    Lise Le Boudec, Emmanuel de Bezenac, Louis Serrano, Ra- mon Daniel Regueiro-Espino, Yuan Yin, and Patrick Galli- nari. Learning a neural solver for parametric PDEs to en- hance physics-informed methods. In The Thirteenth Inter- national Conference on Learning Representations, 2025. 12

    work page 2025

  6. [6]

    Bracewell

    R.N. Bracewell. The Fourier Transform and Its Applications. McGraw Hill, 2000. 1

    work page 2000

  7. [7]

    Multi-level adaptive solutions to boundary- value problems

    Achi Brandt. Multi-level adaptive solutions to boundary- value problems. Mathematics of computation, 31(138):333– 390, 1977. 3

    1977

  8. [8]

    A multigrid tutorial

    William L Briggs, Van Emden Henson, and Steve F Mc- Cormick. A multigrid tutorial. SIAM, 2000. 3

    work page 2000

Show all 67 references
  1. [9]

    Continuum attention for neural oper- ators

    Edoardo Calvello, Nikola B Kovachki, Matthew E Levine, and Andrew M Stuart. Continuum attention for neural oper- ators. arXiv preprint arXiv:2406.06486, 2024. 3, 12

    2024

  2. [10]

    Choose a transformer: Fourier or galerkin

    Shuhao Cao. Choose a transformer: Fourier or galerkin. Ad- vances in neural information processing systems, 34:24924– 24940, 2021. 3, 12

    2021

  3. [11]

    When vision transformers outperform resnets without pre- training or strong data augmentations

    Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre- training or strong data augmentations. arXiv preprint arXiv:2106.01548, 2021. 6

    2021 arXiv

  4. [12]

    Rethinking attention with performers

    Krzysztof Choromanski, Valerii Likhosherstov, David Do- han, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020. 2, 11

    2009 arXiv

  5. [13]

    P.G. Ciarlet. The Finite Element Method for Elliptic Prob- lems. Society for Industrial and Applied Mathematics, 2002. 1

    2002

  6. [14]

    Courant, K

    R. Courant, K. Friedrichs, and H. Lewy. On the partial dif- ference equations of mathematical physics.IBM J. Res. Dev., 11(2):215–234, 1967. 1

    1967

  7. [15]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 5, 11, 14

    2009

  8. [16]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, 8 Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInternationa...

    2021

  9. [17]

    The emergence of clusters in self-attention dynamics, 2024

    Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics, 2024. 3

    2024

  10. [18]

    A mathematical perspective on transform- ers, 2024

    Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transform- ers, 2024. 3

    2024

  11. [19]

    A new version of the fast multipole method for the laplace equation in three dimensions

    Leslie Greengard and Vladimir Rokhlin. A new version of the fast multipole method for the laplace equation in three dimensions. Acta Numerica, 6:229–269, 1997. 3, 4

    1997

  12. [20]

    Multiwavelet-based operator learning for differential equations

    Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems, 34:24048–24062, 2021. 12

    2021

  13. [21]

    Multi-grid methods and applications

    Wolfgang Hackbusch. Multi-grid methods and applications. Springer Science & Business Media, 2013. 3

    2013

  14. [22]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1

    2016

  15. [23]

    Fast mul- tipole attention: A divide-and-conquer attention mechanism for long sequences, 2024

    Yanming Kang, Giang Tran, and Hans De Sterck. Fast mul- tipole attention: A divide-and-conquer attention mechanism for long sequences, 2024. 3, 12

    2024

  16. [24]

    Physics-informed ma- chine learning

    George Karniadakis, Yannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed ma- chine learning. Nature Reviews Physics, pages 1–19, 2021. 1, 12

    2021

  17. [25]

    Neural operator: Learning maps between function spaces with applications to pdes

    Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Aziz- zadenesheli, Kaushik Bhattacharya, Andrew Stuart, and An- ima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Ma- chine Learning Research, 24(89):1–97, 2023. 1, 2, 5, 6, 12

    2023

  18. [26]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013. 5

    2013

  19. [27]

    Characterizing possible failure modes in physics-informed neural networks

    Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. In Ad- vances in Neural Information Processing Systems , pages 26548–26560. Curran Associates, Inc., 2021. 12

    2021

  20. [28]

    Learning multiple layers of features from tiny images, 2009

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009. 5

    2009

  21. [29]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. In Advances in Neural Information Processing Sys- tems. Curran Associates, Inc., 2012. 1

    2012

  22. [30]

    Tiny imagenet visual recognition challenge

    Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 5

    2015

  23. [31]

    Backpropagation applied to handwrit- ten zip code recognition.Neural computation, 1(4):541–551,

    Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwrit- ten zip code recognition.Neural computation, 1(4):541–551,

  24. [32]

    Multipole graph neural operator for parametric partial differential equations

    Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020. 3, 12

    2020

  25. [33]

    Fourier neural operator for paramet- ric partial differential equations, 2021

    Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for paramet- ric partial differential equations, 2021. 2, 7, 12

    2021

  26. [34]

    Geometry-informed neural operator for large-scale 3d pdes

    Zongyi Li, Nikola Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Otta, Mohammad Amin Nabian, Maxi- milian Stadler, Christian Hundt, Kamyar Azizzadenesheli, et al. Geometry-informed neural operator for large-scale 3d pdes. Advances in Neural Information Processing System...

    2023

  27. [35]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 2, 11

    2021

  28. [36]

    Swin transformer v2: Scaling up capacity and resolution

    Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 12009–12019,...

    2022

  29. [37]

    Deep- onet: Learning nonlinear operators for identifying differen- tial equations based on the universal approximation theorem of operators

    Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deep- onet: Learning nonlinear operators for identifying differen- tial equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019. 2, 12

    1910 arXiv

  30. [38]

    Learning nonlinear operators via deeponet based on the universal approximation theorem of operators

    Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021. 12

    2021

  31. [39]

    Deepxde: A deep learning library for solving differ- ential equations

    Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karni- adakis. Deepxde: A deep learning library for solving differ- ential equations. SIAM review, 63(1):208–228, 2021. 1

    2021

  32. [40]

    Multi-scale physical representations for approxi- mating pde solutions with graph neural operators

    Leon Migus, Yuan Yin, Jocelyn Ahmed Mazari, and Patrick Gallinari. Multi-scale physical representations for approxi- mating pde solutions with graph neural operators. In Topo- logical, Algebraic and Geometric Learning Workshops 2022, pages 332–340. PMLR, 2022. 3, 12

    2022

  33. [41]

    Nguyen, Vai Suliafu, Stanley J

    Tan M. Nguyen, Vai Suliafu, Stanley J. Osher, Long Chen, and Bao Wang. Fmmformer: Efficient and flexible trans- former via decomposed near-field and far-field attention,

  34. [42]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 5

    2008

  35. [43]

    G. Of. An efficient algebraic multigrid preconditioner for a fast multipole boundary element method. Computing, 82(2): 139–155, 2008. 3

    2008

  36. [44]

    9 Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 9 Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1, 2, 11

    2023 arXiv

  37. [45]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 5

    2012

  38. [46]

    Physics informed deep learning (part i): Data-driven solu- tions of nonlinear partial differential equations, 2017

    Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solu- tions of nonlinear partial differential equations, 2017. 1, 12

    2017

  39. [47]

    An operator preconditioning perspective on training in physics-informed machine learning

    Tim De Ryck, Florent Bonnet, Siddhartha Mishra, and Em- manuel de Bezenac. An operator preconditioning perspective on training in physics-informed machine learning. In The Twelfth International Conference on Learning Representa- tions, 2024. 12

    2024

  40. [48]

    Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyr ´e

    Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyr ´e. Sinkformers: Transformers with doubly stochastic attention, 2022. 3

    2022

  41. [49]

    Synthesizer: Rethinking self-attention for transformer models

    Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In International conference on ma- chine learning, pages 10183–10192. PMLR, 2021. 2

    2021

  42. [50]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lu- cas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:2...

  43. [51]

    Going deeper with im- age transformers

    Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herv´e J´egou. Going deeper with im- age transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 32–42, 2021. 2, 11

    2021

  44. [52]

    Deit iii: Revenge of the vit

    Hugo Touvron, Matthieu Cord, and Herv ´e J ´egou. Deit iii: Revenge of the vit. In European conference on computer vision, pages 516–533. Springer, 2022. 2, 6, 11

    2022

  45. [53]

    Wavelet neural op- erator: a neural operator for parametric partial differential equations

    Tapas Tripura and Souvik Chakraborty. Wavelet neural op- erator: a neural operator for parametric partial differential equations. arXiv preprint arXiv:2205.02191, 2022. 12

    2022 arXiv

  46. [54]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neu- ral Information Processing Systems. Curran Associates, Inc.,

  47. [55]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. 2, 11

    2006 arXiv

  48. [56]

    Cvit: Contin- uous vision transformer for operator learning

    Sifan Wang, Jacob H Seidman, Shyam Sankaran, Hanwen Wang, George J Pappas, and Paris Perdikaris. Cvit: Contin- uous vision transformer for operator learning. arXiv preprint arXiv:2405.13998, 2024. 3, 12

    2024 arXiv

  49. [57]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision , p...

    2021

  50. [58]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European con- ference on computer vision, pages 68–85. Springer, 2022. 2, 6, 11

    2022

  51. [59]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neu- ral information processing systems, 33:17283–17297, 2020. 2

    2020

  52. [60]

    Multi resolution analysis (mra) for approx- imate self-attention, 2022

    Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, and Vikas Singh. Multi resolution analysis (mra) for approx- imate self-attention, 2022. 3, 12

    2022

  53. [61]

    Erwin: A tree-based hierarchical transformer for large-scale physical systems

    Maksim Zhdanov, Max Welling, and Jan-Willem van de Meent. Erwin: A tree-based hierarchical transformer for large-scale physical systems. CoRR, 2025. 3, 12

    2025

  54. [62]

    H-transformer-1d: Fast one- dimensional hierarchical attention for sequences, 2021

    Zhenhai Zhu and Radu Soricut. H-transformer-1d: Fast one- dimensional hierarchical attention for sequences, 2021. 3, 11 10 A. Extended related work section In this section, we provide a more comprehensive overview of the related literature, expanding upon the works briefly men...

    2021

  55. [63]

    quadratic computational complexity O(N 2) with respect to the number of input patches (N, where typically N ≈ 196 for a 224 × 224 image)

  56. [64]

    Absence of built-in locality and translation equivariance, in contrast to CNNs, which makes ViTs more dependent on large training datasets

  57. [65]

    These drawbacks have spurred the development of more ef- ficient ViT variants

    High computational and memory demands—for in- stance, ViT-Large/ 16 contains roughly 300 million parameters and requires thousands of GPU-hours to train [16]. These drawbacks have spurred the development of more ef- ficient ViT variants. A.1. Efficient Vision Transformer Varia...

  58. [66]

    patch merging

    uses a randomized feature map to approximate softmax(QK ⊤) ≈ Φ(Q)Φ(K)⊤, achieving true O(N ) time and memory with bounded error. When applied to ViT backbones, these methods handle larger images with much lower memory cost. All-MLP Architectures: MLP-Mixer [50] differs from bo...

  59. [67]

    introduces wavelet-based multiscale localization, en- abling kernels that simultaneously capture global structures and fine-grained details. The Multiwavelet Neural Opera- tor (MWNO) [20] further extends this approach by incorpo- rating multiple resolution components, leading ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.