pith. machine review for the scientific record. sign in

arxiv: 2604.14724 · v1 · submitted 2026-04-16 · 💻 cs.CV · cs.LG· eess.IV

Recognition: unknown

HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet

Badri N. Patro , Vijay S. Agneeswaran
Authors on Pith no claims yet

Pith reviewed 2026-05-10 11:35 UTC · model grok-4.3

classification 💻 cs.CV cs.LGeess.IV
keywords state space modelsspectral convolutionscanning-free SSMSpectralPulseNetImageNet classificationFFT-based visionefficient vision modelsfrequency domain gating
0
0 comments X

The pith

HAMSA processes 2D images directly in the spectral domain to eliminate scanning from vision state space models while reaching 85.7% ImageNet top-1 accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that traditional vision SSMs require complex scanning strategies to handle 2D images, which adds overhead and complexity. HAMSA instead uses FFT-based convolution with a single Gaussian complex kernel and input-dependent frequency gating to model sequences in the frequency domain. This produces a simpler architecture that runs faster, uses less memory and energy, and still delivers the highest accuracy among SSMs on ImageNet-1K. A sympathetic reader would care because the result suggests scanning may not be necessary if spectral operations can preserve the core sequential modeling ability of SSMs.

Core claim

HAMSA shows that a scanning-free SSM can be built by replacing the standard (A, B, C) parameterization with a single Gaussian-initialized complex kernel, adding SpectralPulseNet for input-dependent frequency gating, and using the Spectral Adaptive Gating Unit for stable magnitude-based modulation; the resulting model achieves 85.7% top-1 accuracy on ImageNet-1K, 2.2x faster inference than DeiT-S, and 1.4-1.9x speedup over scanning SSMs while consuming less memory and energy.

What carries the argument

SpectralPulseNet, the input-dependent spectral gating mechanism that performs FFT convolution with a single Gaussian complex kernel and magnitude-based adaptive gating to enable direct frequency-domain modeling of 2D images without any scanning path.

If this is right

  • Vision SSMs can reach transformer-level accuracy with O(L log L) complexity and no sequential scanning steps.
  • Simplified kernel parameterization removes discretization instabilities that affect conventional SSM training.
  • FFT-based operations reduce both memory footprint and energy use compared with attention or scanning layers.
  • The same spectral design generalizes from classification to transfer learning and dense prediction tasks.
  • Inference latency drops to 4.2 ms per image on standard hardware, enabling real-time applications.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The spectral approach may remove the need for explicit spatial ordering in other structured data such as video or medical volumes.
  • Hybrid models could combine the lightweight spectral backbone with local convolutional branches for tasks requiring fine spatial detail.
  • Scaling the method to higher-resolution inputs becomes straightforward because complexity grows only logarithmically with sequence length.
  • The removal of scanning may simplify hardware mapping on accelerators that already optimize FFT operations.

Load-bearing premise

That a single Gaussian complex kernel plus input-dependent spectral gating can preserve the sequential dependency modeling power that scanning was introduced to provide for two-dimensional image data.

What would settle it

An ablation on ImageNet-1K where removing SpectralPulseNet and the Gaussian kernel drops top-1 accuracy below 84% while keeping the same parameter count and inference budget would falsify the claim that the spectral design fully replaces scanning.

Figures

Figures reproduced from arXiv: 2604.14724 by Badri N. Patro, Vijay S. Agneeswaran.

Figure 1
Figure 1. Figure 1: Comparison of scanning strategies: (a) Mamba uses [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of learnable filter weights. HAMSA [PITH_FULL_IMAGE:figures/full_fig_p001_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: HAMSA architecture overview. Our model replaces traditional SSM components with a simplified Gaussian-initialized kernel. Both input and kernel are transformed to the spectral domain, where SpectralPulseNet enables adaptive frequency intelligence for efficient global information mixing without scanning. Theoretical Justification. Why is scanning unnecessary? SSMs model Linear Time-Invariant systems where o… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics and convergence analysis. (Left) Training and validation loss curves over 300 epochs comparing HAMSA with transformer baselines (Swin, DeiT, PVT) and SSM variants (VMamba, SiMBA, LocalVMamba). HAMSA achieves faster convergence and lower final loss. (Middle) Top-1 accuracy progression showing HAMSA reaches 85.7% on ImageNet-1K with 3.5× faster training than transformers. (Right) Learning r… view at source ↗
Figure 7
Figure 7. Figure 7: Error analysis and per-class performance break￾down. (Top row) Per-class accuracy comparison on 16 challenging ImageNet-1K categories, showing HAMSA’s robustness across di￾verse object types. (Middle row) Confusion matrix analysis re￾vealing misclassification patterns. (Bottom row) Failure case anal￾ysis categorized by error types (occlusion, scale variation, view￾point, texture similarity, lighting condit… view at source ↗
Figure 8
Figure 8. Figure 8: Gradient stability analysis during training. (Left) Gradient norm evolution over 300 training epochs comparing HAMSA (stable, blue) with VMamba (unstable with explosions at epochs 30, 45, 78, 120, 180, 220, marked in red). (Middle) Layer￾wise gradient distribution across 24 network layers, showing con￾sistent gradient flow in HAMSA versus vanishing/exploding gra￾dients in baseline SSMs. (Right) Training lo… view at source ↗
Figure 9
Figure 9. Figure 9: Performance vs. computational complexity for vi￾sual Mamba backbones. Accuracy-Efficiency Trade-off Com￾parison on ImageNet-1K : Hamsa demonstrates superior per￾formance across all model scales, achieving the highest Top￾1 accuracy with competitive computational cost. Hamsa￾L achieves 85.7% accuracy at 16.5 GFLOPs, significantly outperforming all state-of-the-art vision models, including VMamba (83.9% at 1… view at source ↗
Figure 12
Figure 12. Figure 12: SpectralPulseNet (SPN) frequency response and gating mechanism. (Top row) Learned spectral filters across 12 layers showing progressive frequency selectivity from low￾frequency (early layers) to high-frequency (deep layers). (Mid￾dle row) Frequency response curves demonstrating adaptive band￾pass filtering with learnable center frequencies and bandwidths. (Bottom row) Spectral gating activation patterns r… view at source ↗
Figure 13
Figure 13. Figure 13 [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗
read the original abstract

Vision State Space Models (SSMs) like Vim, VMamba, and SiMBA rely on complex scanning strategies to adapt sequential SSMs to process 2D images, introducing computational overhead and architectural complexity. We propose HAMSA, a scanning-free SSM operating directly in the spectral domain. HAMSA introduces three key innovations: (1) simplified kernel parameterization-a single Gaussian-initialized complex kernel replacing traditional (A, B, C) matrices, eliminating discretization instabilities; (2) SpectralPulseNet (SPN)-an input-dependent frequency gating mechanism enabling adaptive spectral modulation; and (3) Spectral Adaptive Gating Unit (SAGU)-magnitude-based gating for stable gradient flow in the frequency domain. By leveraging FFT-based convolution, HAMSA eliminates sequential scanning while achieving O(L log L) complexity with superior simplicity and efficiency. On ImageNet-1K, HAMSA reaches 85.7% top-1 accuracy (state-of-the-art among SSMs), with 2.2 X faster inference than transformers (4.2ms vs 9.2ms for DeiT-S) and 1.4-1.9X speedup over scanning-based SSMs, while using less memory (2.1GB vs 3.2-4.5GB) and energy (12.5J vs 18-25J). HAMSA demonstrates strong generalization across transfer learning and dense prediction tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes HAMSA, a scanning-free vision state space model operating directly in the spectral domain. It replaces the standard (A, B, C) SSM parameterization with a single Gaussian-initialized complex kernel, introduces SpectralPulseNet (SPN) as an input-dependent frequency gating mechanism, and Spectral Adaptive Gating Unit (SAGU) for magnitude-based gating to stabilize gradients. Using FFT-based convolution for O(L log L) complexity, the model eliminates multi-directional scanning. On ImageNet-1K it reports 85.7% top-1 accuracy (SOTA among SSMs), 2.2× faster inference than DeiT-S, 1.4–1.9× speedup over scanning SSMs, and lower memory/energy use, with claimed generalization to transfer learning and dense prediction tasks.

Significance. If the empirical claims are reproducible, the work offers a meaningful simplification of vision SSMs by removing scanning overhead while preserving competitive accuracy and improving efficiency metrics. The spectral-domain approach with adaptive gating could influence efficient vision architecture design. The manuscript does not mention machine-checked proofs, open code, or parameter-free derivations, so credit is limited to the reported efficiency gains and empirical results.

major comments (2)
  1. [Abstract] Abstract: the central claim that a single Gaussian-initialized complex kernel plus SPN/SAGU fully preserves the sequential modeling capacity of standard SSMs for 2D images (without scanning) is load-bearing for all performance assertions, yet the abstract supplies no equations, no comparison to learnable A/B/C dynamics, and no ablation isolating the kernel's expressivity. A fixed Gaussian kernel corresponds to a narrow family of smooth isotropic decays; without explicit verification that this spans the required function class for spatial dependencies, the 85.7% accuracy cannot be confidently attributed to the architectural advance rather than dataset-specific tuning.
  2. [Abstract] Abstract and experimental claims: no training details, hyper-parameters, ablation studies, or error analysis are provided to support the reported accuracy, speed (4.2 ms vs 9.2 ms), memory (2.1 GB), or energy figures. This absence makes it impossible to verify whether the efficiency and accuracy numbers are robust or sensitive to implementation choices, directly undermining the soundness of the efficiency and SOTA claims.
minor comments (1)
  1. The abstract refers to 'strong generalization across transfer learning and dense prediction tasks' without naming the specific datasets or metrics used, reducing clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, providing clarifications from the full paper and indicating where revisions will strengthen the presentation. We believe these changes will resolve the concerns while preserving the core contributions.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that a single Gaussian-initialized complex kernel plus SPN/SAGU fully preserves the sequential modeling capacity of standard SSMs for 2D images (without scanning) is load-bearing for all performance assertions, yet the abstract supplies no equations, no comparison to learnable A/B/C dynamics, and no ablation isolating the kernel's expressivity. A fixed Gaussian kernel corresponds to a narrow family of smooth isotropic decays; without explicit verification that this spans the required function class for spatial dependencies, the 85.7% accuracy cannot be confidently attributed to the architectural advance rather than dataset-specific tuning.

    Authors: We agree the abstract is concise and omits equations. The full manuscript (Section 3.1) derives the single complex Gaussian kernel as a fixed-A parameterization of the SSM recurrence, with SPN providing input-dependent frequency modulation equivalent to adaptive B/C dynamics and SAGU ensuring stable gradients. This combination enables spectral convolution to capture 2D spatial dependencies without scanning, as the FFT-based operation models arbitrary frequency responses. Ablations isolating the kernel (Table 3) show a 3.2% drop without SPN/SAGU, supporting attribution to the architecture. We will revise the abstract to reference the expressivity argument from the introduction and add a high-level equation for the kernel. revision: partial

  2. Referee: [Abstract] Abstract and experimental claims: no training details, hyper-parameters, ablation studies, or error analysis are provided to support the reported accuracy, speed (4.2 ms vs 9.2 ms), memory (2.1 GB), or energy figures. This absence makes it impossible to verify whether the efficiency and accuracy numbers are robust or sensitive to implementation choices, directly undermining the soundness of the efficiency and SOTA claims.

    Authors: Training details and hyperparameters are specified in Section 4.1 and Appendix A (including optimizer, learning rate schedule, and data augmentation). Ablation studies appear in Section 4.3 (Tables 2-4), and inference metrics (latency, memory, energy on A100 GPU with batch size 1) are reported in Section 4.2 with hardware details. Multi-run error analysis is in the supplementary material. To improve verifiability, we will add a concise hyperparameter summary to the main text near the results and expand sensitivity analysis in the revision. The reported figures follow standard ImageNet-1K protocols. revision: yes

Circularity Check

0 steps flagged

No circularity: performance claims rest on empirical benchmarks, not self-referential derivations or fitted predictions

full rationale

The paper presents HAMSA as a scanning-free spectral SSM with a single Gaussian complex kernel, SPN input-dependent gating, and SAGU magnitude gating. No equations, derivations, or parameter-fitting steps are described in the provided text that would reduce a claimed prediction or uniqueness result back to the inputs by construction. The 85.7% ImageNet accuracy and efficiency numbers are reported as measured outcomes on standard benchmarks, not as outputs of a self-defined or self-cited normalization. No self-citation load-bearing steps, ansatz smuggling, or renaming of known results appear in the abstract or summary. The architecture choices are presented as design decisions whose validity is tested empirically rather than proven by internal equivalence.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 2 invented entities

Review is limited to the abstract; no explicit free parameters, axioms, or independent evidence for new entities are provided in the text.

invented entities (2)
  • SpectralPulseNet (SPN) no independent evidence
    purpose: input-dependent frequency gating mechanism
    New component introduced to enable adaptive spectral modulation.
  • Spectral Adaptive Gating Unit (SAGU) no independent evidence
    purpose: magnitude-based gating for stable gradient flow in frequency domain
    New unit proposed to replace or augment traditional gating.

pith-pipeline@v0.9.0 · 5564 in / 1288 out tokens · 75894 ms · 2026-05-10T11:35:03.154259+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 25 canonical work pages · 2 internal anchors

  1. [1]

    Mambamixer: Efficient selective state space models with dual token and channel selection

    Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection.arXiv preprint arXiv:2403.19888, 2024

    work page arXiv 2024

  2. [2]

    Cascade r-cnn: Delv- ing into high quality object detection

    Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delv- ing into high quality object detection. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 6154–6162, 2018

    2018

  3. [3]

    Vision transformer adapter for dense predictions,

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions.arXiv preprint arXiv:2205.08534, 2022

    work page arXiv 2022

  4. [4]

    Language modeling with gated convolutional net- works

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional net- works. InInternational conference on machine learning, pages 933–941. PMLR, 2017

    2017

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  6. [6]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. InInternational Con- ference on Learning Representations, 2020

    2020

  7. [7]

    Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures, March 2024

    Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable vi- sual perception with rwkv-like architectures.arXiv preprint arXiv:2403.02308, 2024

    work page arXiv 2024

  8. [8]

    Hypergraph vision transformers: Images are more than nodes, more than edges

    Joshua Fixelle. Hypergraph vision transformers: Images are more than nodes, more than edges. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9751–9761, 2025

    2025

  9. [9]

    Monarch mixer: A simple sub-quadratic gemm-based architecture

    Dan Fu, Simran Arora, Jessica Grogan, Isys Johnson, Evan Sabri Eyuboglu, Armin Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher R ´e. Monarch mixer: A simple sub-quadratic gemm-based architecture. Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [10]

    Bhvit: Binarized hy- brid vision transformer

    Tian Gao, Yu Zhang, Zhiyuan Zhang, Huajun Liu, Kaijie Yin, Chengzhong Xu, and Hui Kong. Bhvit: Binarized hy- brid vision transformer. InProceedings of the Computer Vi- sion and Pattern Recognition Conference, pages 3563–3572, 2025

    2025

  11. [11]

    Understanding the diffi- culty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the diffi- culty of training deep feedforward neural networks. InPro- ceedings of the thirteenth international conference on artifi- cial intelligence and statistics, pages 249–256. JMLR Work- shop and Conference Proceedings, 2010

    2010

  12. [12]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    work page Pith review arXiv 2023

  13. [13]

    Efficiently mod- eling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently mod- eling long sequences with structured state spaces. InInter- national Conference on Learning Representations, 2021

    2021

  14. [14]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12175–12185, 2022

    2022

  15. [15]

    Diagonal state spaces are as effective as structured state spaces.Advances in Neural Information Processing Systems, 35:22982–22994, 2022

    Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces.Advances in Neural Information Processing Systems, 35:22982–22994, 2022

    2022

  16. [16]

    Flatten transformer: Vision transformer using focused linear attention

    Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 5961– 5971, 2023

    2023

  17. [17]

    MambaVision: A hybrid Mamba-Transformer vision back- bone,

    Ali Hatamizadeh and Jan Kautz. Mambavision: A hy- brid mamba-transformer vision backbone.arXiv preprint arXiv:2407.08083, 2024

    work page arXiv 2024

  18. [18]

    Fastervit: Fast vision transformers with hierarchical attention

    Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hierarchical attention. InThe Twelfth International Conference on Learning Representa- tions

  19. [19]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  20. [20]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. InProceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017

    2017

  21. [21]

    Vision transformer with super token sampling

    Huaibo Huang, Xiaoqiang Zhou, Jie Cao, Ran He, and Tie- niu Tan. Vision transformer with super token sampling. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 22690–22699, 2023

    2023

  22. [22]

    LocalMamba: Visual state space model with windowed selective scan

    Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan.arXiv preprint arXiv:2403.09338, 2024

    work page arXiv 2024

  23. [23]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013

    2013

  24. [24]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009

    2009

  25. [25]

    Fnet: Mixing tokens with fourier transforms.arXiv preprint arXiv:2105.03824,

    James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santi- ago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824, 2021

    work page arXiv 2021

  26. [26]

    Videomamba: State space model for efficient video understanding

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. InEuropean Conference on Computer Vision, pages 237–255. Springer, 2024

    2024

  27. [27]

    Mamba- nd: Selective state space modeling for multi-dimensional data.arXiv preprint arXiv:2402.05892, 2024

    Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba- nd: Selective state space modeling for multi-dimensional data.arXiv preprint arXiv:2402.05892, 2024

    work page arXiv 2024

  28. [28]

    Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.Advances in Neural Information Processing Systems, 33:21002–21012, 2020

    Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.Advances in Neural Information Processing Systems, 33:21002–21012, 2020

    2020

  29. [29]

    Effi- cientformer: Vision transformers at mobilenet speed.arXiv preprint arXiv:2206.01191, 2022

    Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evan- gelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Effi- cientformer: Vision transformers at mobilenet speed.arXiv preprint arXiv:2206.01191, 2022

    work page arXiv 2022

  30. [30]

    Vig: Linear-complexity visual sequence learning with gated linear attention

    Bencheng Liao, Xinggang Wang, Lianghui Zhu, Qian Zhang, and Chang Huang. Vig: Linear-complexity visual sequence learning with gated linear attention. InProceed- ings of the AAAI Conference on Artificial Intelligence, pages 5182–5190, 2025

    2025

  31. [31]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014

    2014

  32. [32]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. InPro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017

    2017

  33. [33]

    Mixmae: Mixed and masked autoencoder for effi- cient pretraining of hierarchical vision transformers

    Jihao Liu, Xin Huang, Jinliang Zheng, Yu Liu, and Hong- sheng Li. Mixmae: Mixed and masked autoencoder for effi- cient pretraining of hierarchical vision transformers. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6252–6261, 2023

    2023

  34. [34]

    VMamba: Visual state space model,

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model.arXiv preprint arXiv:2401.10166, 2024

    work page arXiv 2024

  35. [35]

    Swin transformer v2: Scaling up capacity and resolution

    Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 12009–12019, 2022

    2022

  36. [36]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 11976–11986, 2022

    2022

  37. [37]

    Long range language modeling via gated state spaces

    Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. InThe Eleventh International Conference on Learn- ing Representations, 2022

    2022

  38. [38]

    Springer Science & Business Media, 2003

    Yurii Nesterov.Introductory Lectures on Convex Optimiza- tion: A Basic Course. Springer Science & Business Media, 2003

    2003

  39. [39]

    S4nd: Modeling images and videos as multidimensional signals with state spaces.Advances in neural information processing systems, 35:2846–2861, 2022

    Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher R ´e. S4nd: Modeling images and videos as multidimensional signals with state spaces.Advances in neural information processing systems, 35:2846–2861, 2022

    2022

  40. [40]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008

    2008

  41. [41]

    Scat- tering vision transformer: Spectral mixing matters

    Badri Narayana Patro and Vijay Srinivas Agneeswaran. Scat- tering vision transformer: Spectral mixing matters. InThirty- seventh Conference on Neural Information Processing Sys- tems, 2023

    2023

  42. [42]

    Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges,

    Badri Narayana Patro and Vijay Srinivas Agneeswaran. Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, appli- cations, and challenges.arXiv preprint arXiv:2404.16112, 2024

    work page arXiv 2024

  43. [43]

    Simba: Simplified mamba-based architecture for vision and multivariate time series,

    Badri N Patro and Vijay S Agneeswaran. Simba: Simplified mamba-based architecture for vision and multivariate time series.arXiv preprint arXiv:2403.15360, 2024

    work page arXiv 2024

  44. [44]

    arXiv preprint arXiv:2304.06446 , year=

    Badri N Patro, Vinay P Namboodiri, and Vijay Srinivas Agneeswaran. Spectformer: Frequency and attention is what you need in a vision transformer.arXiv preprint arXiv:2304.06446, 2023

    work page arXiv 2023

  45. [45]

    Heracles: A hybrid ssm-transformer model for high-resolution image and time-series analysis

    Badri N Patro, Suhas Ranganath, Vinay P Namboodiri, and Vijay S Agneeswaran. Heracles: A hybrid ssm-transformer model for high-resolution image and time-series analysis. arXiv preprint arXiv:2403.18063, 2024

    work page arXiv 2024

  46. [46]

    Efficientvmamba: Atrous selective scan for light weight visual mamba.arXiv preprint arXiv:2403.09977, 2024

    Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba.arXiv preprint arXiv:2403.09977, 2024

    work page arXiv 2024

  47. [47]

    Hyena hierarchy: Towards larger con- volutional language models

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Re. Hyena hierarchy: Towards larger con- volutional language models. 2023

    2023

  48. [48]

    Toeplitz neural network for sequence modeling

    Zhen Qin, Xiaodong Han, Weixuan Sun, Bowen He, Dong Li, Dongxu Li, Yuchao Dai, Lingpeng Kong, and Yiran Zhong. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Repre- sentations, 2022

    2022

  49. [49]

    Hierarchically gated recurrent neural network for sequence modeling.Ad- vances in Neural Information Processing Systems, 36, 2024

    Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling.Ad- vances in Neural Information Processing Systems, 36, 2024

    2024

  50. [50]

    Designing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll ´ar. Designing network design spaces. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 10428–10436, 2020

    2020

  51. [51]

    Global filter networks for image classification.Ad- vances in Neural Information Processing Systems, 34:980– 993, 2021

    Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification.Ad- vances in Neural Information Processing Systems, 34:980– 993, 2021

    2021

  52. [52]

    Hornet: Efficient high- order spatial interactions with recursive gated convolutions

    Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser Nam Lim, and Jiwen Lu. Hornet: Efficient high- order spatial interactions with recursive gated convolutions. Advances in Neural Information Processing Systems, 35: 10353–10366, 2022

    2022

  53. [53]

    Sg-former: Self-guided transformer with evolving token reallocation

    Sucheng Ren, Xingyi Yang, Songhua Liu, and Xinchao Wang. Sg-former: Self-guided transformer with evolving token reallocation. InProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 6003–6014, 2023

    2023

  54. [54]

    McGraw-Hill, New York, 3rd edition, 1976

    Walter Rudin.Principles of Mathematical Analysis. McGraw-Hill, New York, 3rd edition, 1976

    1976

  55. [55]

    Groupmamba: Parameter-efficient and accurate group visual state space model.arXiv preprint arXiv:2407.13772, 2024

    Abdelrahman Shaker, Syed Talal Wasim, Salman Khan, Juergen Gall, and Fahad Shahbaz Khan. Groupmamba: Parameter-efficient and accurate group visual state space model.arXiv preprint arXiv:2407.13772, 2024

    work page arXiv 2024

  56. [56]

    GLU Variants Improve Transformer

    Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    work page internal anchor Pith review arXiv 2002

  57. [57]

    Multi-scale vmamba: Hierarchy in hierarchy visual state space model

    Yuheng Shi, Minjing Dong, and Chang Xu. Multi-scale vmamba: Hierarchy in hierarchy visual state space model. 2024

    2024

  58. [58]

    Inception transformer

    Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Y AN. Inception transformer. InAd- vances in Neural Information Processing Systems, 2022

    2022

  59. [59]

    Distillation-free scaling of large ssms for im- ages and videos.arXiv preprint arXiv:2409.11867, 2024

    Hamid Suleman, Syed Talal Wasim, Muzammal Naseer, and Juergen Gall. Distillation-free scaling of large ssms for im- ages and videos.arXiv preprint arXiv:2409.11867, 2024

    work page arXiv 2024

  60. [60]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. InInternational conference on machine learning, pages 6105–6114. PMLR, 2019

    2019

  61. [61]

    Scalable visual state space model with fractal scanning.arXiv preprint arXiv:2405.14480, 2024

    Lv Tang, HaoKe Xiao, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, and Bo Li. Scalable visual state space model with fractal scanning.arXiv preprint arXiv:2405.14480, 2024

    work page arXiv 2024

  62. [62]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. InInternational Conference on Machine Learning, pages 10347–10357. PMLR, 2021

    2021

  63. [63]

    Resmlp: Feedforward networks for image classification with data-efficient training.IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2022

    Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izac- ard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training.IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2022

    2022

  64. [64]

    Deit iii: Revenge of the vit.arXiv preprint arXiv:2204.07118, 2022

    Hugo Touvron, Matthieu Cord, and Herv ´e J ´egou. Deit iii: Revenge of the vit.arXiv preprint arXiv:2204.07118, 2022

    work page arXiv 2022

  65. [65]

    Maxvit: Multi-axis vision transformer

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. InComputer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23–27, 2022, Proceedings, Part XXIV, pages 459–479. Springer, 2022

    2022

  66. [66]

    Mamba-r: Vision mamba also needs registers.arXiv preprint arXiv:2405.14858, 2024

    Feng Wang, Jiahao Wang, Sucheng Ren, Guoyizhe Wei, Jieru Mei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Mamba-r: Vision mamba also needs registers.arXiv preprint arXiv:2405.14858, 2024

    work page arXiv 2024

  67. [67]

    Scaled relu mat- ters for training vision transformers

    Pichao Wang, Xue Wang, Hao Luo, Jingkai Zhou, Zhipeng Zhou, Fan Wang, Hao Li, and Rong Jin. Scaled relu mat- ters for training vision transformers. InProceedings of the AAAI Conference on Artificial Intelligence, pages 2495– 2503, 2022

    2022

  68. [68]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415–424, 2022

    2022

  69. [69]

    Dynamixer: a vision mlp architecture with dynamic mixing

    Ziyu Wang, Wenhao Jiang, Yiming M Zhu, Li Yuan, Yibing Song, and Wei Liu. Dynamixer: a vision mlp architecture with dynamic mixing. InInternational Conference on Ma- chine Learning, pages 22691–22701. PMLR, 2022

    2022

  70. [70]

    Unified perceptual parsing for scene understand- ing

    Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understand- ing. InProceedings of the European conference on computer vision (ECCV), pages 418–434, 2018

    2018

  71. [71]

    Mambatree: Tree topology is all you need in state space model.Advances in Neural Information Processing Systems, 37:75329–75354, 2024

    Yicheng Xiao, Lin Song, Jiangshan Wang, Siyu Song, Yixiao Ge, Xiu Li, Ying Shan, et al. Mambatree: Tree topology is all you need in state space model.Advances in Neural Information Processing Systems, 37:75329–75354, 2024

    2024

  72. [72]

    Plainmamba: Improving non-hierarchical mamba in visual recognition,

    Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Er- icsson, Zhenyu Wang, Jiaming Liu, and Elliot J Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition.arXiv preprint arXiv:2403.17695, 2024

    work page arXiv 2024

  73. [73]

    Wave-vit: Unifying wavelet and transformers for visual representation learning

    Ting Yao, Yingwei Pan, Yehao Li, Chong-Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. InComputer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23– 27, 2022, Proceedings, Part XXV, pages 328–345. Springer, 2022

    2022

  74. [74]

    Mambaout: Do we really need mamba for vision?

    Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision?arXiv preprint arXiv:2405.07992, 2024

    work page arXiv 2024

  75. [75]

    V olo: Vision outlooker for visual recog- nition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, and Shuicheng Yan. V olo: Vision outlooker for visual recog- nition.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    2022

  76. [76]

    Vim-f: Visual state space model benefiting from learning in the frequency domain

    Juntao Zhang, Kun Bian, Peng Cheng, Wenbo An, Jianning Liu, and Jun Zhou. Vim-f: Visual state space model bene- fiting from learning in the frequency domain.arXiv preprint arXiv:2405.18679, 2024

    work page arXiv 2024

  77. [77]

    Multi-scale vision long- former: A new vision transformer for high-resolution image encoding

    Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision long- former: A new vision transformer for high-resolution image encoding. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3008, 2021

    2021

  78. [78]

    Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fi- dler, Adela Barriuso, and Antonio Torralba. Semantic under- standing of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    2019

  79. [79]

    Biformer: Vision transformer with bi-level routing attention

    Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson WH Lau. Biformer: Vision transformer with bi-level routing attention. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 10323–10333, 2023

    2023

  80. [80]

    Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model.arXiv preprint arXiv:2401.09417, 2024. A. Introduction We present additional evidence and analyses that substanti- ate the claims made in our main paper regarding HAMSA’s performan...

    work page internal anchor Pith review arXiv 2024