Pith. sign in

REVIEW 2 major objections 5 minor 28 references

A Separable Self-attention Inspired by the State Space Model for Computer Vision

T0 review · 2 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VMINet shows that a Mamba-inspired separable self-attention with a causal mask beats Vision Mamba at linear cost.

desk verdict Solid empirical contribution with a shaky theoretical motivation: the mask works, likely for positional reasons, not the rank argument in Eq. (8). read the letter →

arxiv 2501.02040 v2 pith:CRILYR6M submitted 2025-01-03 cs.CV cs.AI

Juntao Zhang , Shaogeng Liu , Kun Bian , You Zhou , Pei Zhang , Jianning Liu , Jun Zhou , Bingyan Liu This is my paper · ORCID
classification cs.CVcs.AI
keywords separableself-attentionstatespacemodelVisionMambalinearcomplexityattentioncausalmaskdepthwiseconvolutionimageclassificationVMINet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the design ideas behind Mamba, a linear-complexity state space model, can be transplanted into separable self-attention without paying quadratic cost. It proposes VMI-SA, which combines a depthwise-convolution local branch, a causal, banded, or block-diagonal mask over token interactions, and trainable per-token weights, and builds VMINet by stacking this module with simple downsampling layers. The reported result is that VMINet consistently outperforms Vision Mamba (Vim) on ImageNet classification, COCO detection, and ADE20K segmentation at comparable or lower parameter counts, and stays competitive with strong CNN and Transformer backbones. If the claim holds, it points to a route to linear-complexity global modeling that avoids Mamba's causal scanning and specialized SSM kernels.

What carries the argument

The central object is the sparse mask matrix $M$ applied to the elementwise token product $Q \odot K$ before summation into a context vector. Equation (8) shows that zeroing the upper triangle of a nonzero matrix yields rank $\min\{L,D\}$, and the paper treats this maximum rank as richer contextual information. Around this, the module has a depthwise-convolution branch for local correlation, learnable scalars $\alpha_i$ controlling token importance and $\beta_i$ balancing local versus global information, and two equivalent forms: a recurrent form restricted to previous tokens and a parallelizable matrix form with a global receptive field. The matrix form is what VMINet stacks.

What would settle it

Train VMINet-XS with the mask removed but with learned positional encodings added to tokens; if that model matches or exceeds the masked model's 78.6 top-1 accuracy, the causal mask's benefit is positional, not rank-based.

Watch

Extended reading notes

Core claim

For any image sequence, VMI-SA computes a global context vector by elementwise-multiplying token-wise query/key features, weighting each token with learnable scalars, applying a sparse mask (lower-triangular, banded, block-diagonal, or hybrid), and broadcast-multiplying the result across values. The paper argues that elementwise multiplication already maps features into a higher-dimensional nonlinear space, that the mask raises the rank of the attention information to $\min\{L,D\}$, and that restoring the global receptive field in matrix form keeps the complexity linear. Its central empirical discovery is that a plain VMINet with 2.0–28.4M parameters outperforms Vim at matching scales: 78.6 versus 76.1 top-1 for the small variant, and 82.4 for VMINet-B against 82.2 for VMamba-T, with similar gains on COCO and ADE20K.

Load-bearing premise

The argument that the causal mask helps because it maximizes the rank of the attention information assumes the trained token products are all nonzero and that higher rank, rather than positional information, is what actually boosts accuracy.

Editorial extensions

If this is right

  • Linear-complexity global modeling can be applied to high-resolution dense prediction without quadratic attention cost or Mamba's selective-scan mechanism.
  • Images need not be flattened into 1D sequences: the mask and context vector act on 2D features, preserving spatial correlation while keeping the same complexity.
  • The exact triangular causal mask is not essential: banded and hybrid masks match or exceed it, so the benefit is a tunable positional and importance bias rather than a uniquely causal one.
  • The recurrent form, being an RNN-like linear attention, provides a natural route to causal sequence modeling outside images, as the paper notes for other fields.
  • The approach transfers well to smaller models: VMINet-XS reaches 78.6 top-1 with 7M parameters and 1.4 GFLOPs, making it attractive for efficient vision backbones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If adding learned positional encodings without any mask recovers the roughly two-point gap between masked and unmasked VMINet, the rank-maximization story would be a post-hoc explanation rather than the mechanism; the ablation data alone do not separate rank from position.
  • The fact that a banded matrix with bandwidth $B/2$ matches the triangular matrix in accuracy suggests the practical benefit is mostly local and global position bias, not literal maximal rank, because the banded matrix has lower rank.
  • Because the module is a drop-in attention layer using only depthwise convolution, elementwise products, and masks, a direct stress test would be to vary the mask bandwidth continuously from zero to full length and record accuracy; if accuracy saturates before full triangular context, the 'maximal rank' design point is not the operating one.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes VMI-SA, a linear-complexity separable self-attention variant that adds a depthwise-convolution branch, trainable token weights, and a lower-triangular/banded mask, together with a local residual term. It derives a recurrent form and a parallel matrix form, then builds VMINet by stacking VMI-SA blocks with simple downsampling. Experiments on ImageNet-1K, COCO object detection/instance segmentation, and ADE20K semantic segmentation report competitive results and consistent gains over Vim, with ablations isolating the contributions of the attention operation and different mask matrices. Code is publicly released.

Significance. The empirical contribution is meaningful if reproducible: VMINet is a simple, lightweight backbone whose ablations clearly show that the attention-like global term and the mask both matter (removing attention costs 2.2 points, removing the mask costs 2.1 points). The algebraic expansion in Eqs. (4)-(6) is correct, and the mask-form ablation in Table 4 is informative. The main weakness is the theoretical motivation: the rank-increase argument in Section 3.3.2 is not a valid derivation of the mask's benefit in the trained regime, and the recurrent-form indexing does not match standard causal sequence modeling. These issues affect the paper's central conceptual claim, but they are reparable by reframing or by adding measurements.

major comments (2)
  1. [3.3.2, Eq. (8)] The statement that introducing causality 'can theoretically increase the diversity of contextual information, thereby enhancing performance' is not established by the rank argument. For A = softmax(Q) ⊙ K with K ∈ R^{L,D}, one already has rank(A) ≤ D from Eq. (7); if K is full column rank, which is the generic case in a trained network, rank(A) = D before masking, so M cannot increase the rank to min{L,D} = D. The all-nonzero premise is unverified, and in the actual VMI-SA of Eq. (10) Q is not a scalar softmax score, so the formal analogy is loose. Eq. (11) shows what M actually does: it changes which tokens are summed into each context entry, with higher-index tokens encoded more frequently, and Section 4.2 attributes the gain to loss of positional information. Furthermore, Table 4 is inconsistent with a pure rank mechanism: banded and lower-triangular masks can both have full column rank D, yet the hybrid mask gives a further gain. Please remove the rank-increase claim and describe the mask as a positional/order bias, or supply direct evidence such as measured ranks of Q⊙K before and after masking and an experiment that varies rank while controlling position.
  2. [3.3.3, Eq. (9)] The recurrent form does not implement the stated 'restrict the receptive field to the previous token' in the usual sequence sense. In Eq. (9), h_i accumulates Q_i ⊙ K_i over tokens, but M_i is the i-th row of the L×D matrix M, so M_i ⊙ h_i gates feature dimensions according to the row index i rather than limiting h_i to the previous token. Eq. (11) confirms this reading by expressing e_n as a sum over t ≥ n, meaning the mask is lower-triangular in the (token, feature-dimension) plane and not a causal mask over the token sequence. The authors should either define M with a proper token-causal indexing or revise the recurrent-form discussion so that the analogy to Eq. (3) is not overstated.
minor comments (5)
  1. [3.1, Eq. (4)] The text says element-wise multiplication projects the feature vector into a 'C^2 dimensional' space, but after collecting symmetric terms the number of distinct monomials is C(C+1)/2; please correct the dimension statement.
  2. [3.3.3, Eq. (9) and Eq. (10)] The notation for β is inconsistent: Eq. (9) uses β_i per token while Eq. (10) uses a scalar β with no index; please specify whether β is per-token or shared.
  3. [3.3.2, Eq. (8)] The matrix M is stated to be in R^{L,D} but the displayed matrix appears square; for L > D, please clarify which D columns of the lower-triangular pattern are used.
  4. [Abstract and Introduction] The claim 'for the first time introducing some excellent design concepts of Mamba into separable self-attention' is a strong novelty statement that is hard to verify; consider softening it.
  5. [3.2] The assertion that the attention information in softmax(Q) ⊙ K is 'severely homogenized' is not quantified; please provide a concrete measure or remove the adjective.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VMINet's empirical claims are tested against external benchmarks, and its architectural components (mask, learned weights) are design choices rather than fitted predictions.

full rationale

The paper's central claim is that VMINet, built from VMI-SA, consistently outperforms Vim and is competitive with other state-of-the-art models (Section 4, Tables 2, 5, 6). These results are obtained by training on ImageNet, COCO, and ADE20K and evaluating on held-out validation sets; no benchmark number is produced by plugging fitted constants into the model. The trainable parameters alpha_i and beta_i are ordinary learned weights, not constants tuned to the reported test sets. The mask matrix is a hand-chosen architectural component, and its variants (triangular, banded, block diagonal, hybrid) are explicitly ablated in Table 4; this is experimental design, not a prediction that reduces to its input. The rank-based motivation in Eqs. (7)-(8) is an a priori mathematical statement about masked matrices. Its assumption that the trained softmax(Q) ⊕ K is rank-deficient, and its implicit claim that rank drives accuracy, may be questionable, but that is a correctness concern rather than circularity: the paper does not define the empirical outcome in terms of the rank argument, nor does it fit the mask to the measured improvement. Citations to Mamba, Vim, Flatten Transformer, and Demystify Mamba are external prior work by other authors; there is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in solely through the authors' own prior publications. Overall, the derivation chain is not circular and the central results stand on independent empirical evidence.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on learned weights (alpha, beta) and hand-selected mask forms rather than on newly postulated physical entities. The main theoretical axiom is the rank-to-performance link, which is plausible but unproven. No independent external evidence is offered for the mask design beyond the reported validation-set ablations.

free parameters (3)
  • alpha_i (token contribution weights) = learned during training
    Introduced in Eqs. (9) and (10) to weight each token's contribution to the context vector; learned with the rest of the network on ImageNet-1K.
  • beta (local/global balance weight) = learned during training
    Controls the mix of local Q element-wise K and global context in Eq. (10); learned during training.
  • Mask matrix form and bandwidth = lower triangular, banded with bandwidth B/2, block diagonal, or hybrid
    Selected by hand and ablation (Table 4); the hybrid stage assignment is an ad hoc design choice rather than derived.
assumptions (4)
  • standard math For any L by D matrix A with all entries nonzero and L greater than D, element-wise multiplication with a lower triangular mask yields rank D.
    Linear algebra fact; the nonzero-entry condition is stated by the authors in Eq. (8).
  • domain assumption Higher rank of the attention-equivalent matrix causes better downstream accuracy.
    The paper uses this to justify the mask but provides no controlled test; the ablation only shows that the mask helps, not that rank is the causal channel.
  • domain assumption Depthwise convolution before element-wise multiplication establishes useful local correlations for images.
    Standard inductive bias in vision backbones; adopted from Mamba and ConvNeXt design, with no dedicated ablation isolating this component in 2D.
  • domain assumption A triangular mask over a flattened ordering provides positional information sufficient to replace explicit position embeddings.
    The paper asserts this in Sections 3.3.3 and 3.3.4 and tests it indirectly through the no-mask ablation, but does not compare with position embeddings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Separable Self-attention Inspired by the State Space Model for Computer Vision." pith.science (2026). https://pith.science/paper/CRILYR6M

@misc{pith2026250102040,
  author       = {Pith},
  title        = {Pith review of: A Separable Self-attention Inspired by the State Space Model for Computer Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CRILYR6M}},
  note         = {Machine review of arXiv:2501.02040}
}
read the original abstract

Mamba is an efficient State Space Model (SSM) with linear computational complexity. Although SSMs are not suitable for handling non-causal data, Vision Mamba (ViM) methods still demonstrate good performance in tasks such as image classification and object detection. Recent studies have shown that there is a rich theoretical connection between state space models and attention variants. We propose a novel separable self attention method, for the first time introducing some excellent design concepts of Mamba into separable self-attention. To ensure a fair comparison with ViMs, we introduce VMINet, a simple yet powerful prototype architecture, constructed solely by stacking our novel attention modules with the most basic down-sampling layers. Notably, VMINet differs significantly from the conventional Transformer architecture. Our experiments demonstrate that VMINet has achieved competitive results on image classification and high-resolution dense prediction tasks.Code is available at: https://github.com/yws-wxs/VMINet.

Figures

Figures reproduced from arXiv: 2501.02040 by the authors.

Figure 1
Figure 1. Comparison with different modules. To facilitate a clear comparison, we uniformly adapt one-dimensional sequences as input, although this is not necessary for VMI-SA. the encoding of the m-th token, while E ′ m,n is the encoding of both the m-th and n-th tokens. The softmax and summation operations provide a global receptive field for separable self-attention, but the performance difference between separable self-at… view at source ↗
Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Grad-CAM activation maps of the models trained on ImageNet￾1K. The visualized images are from validation set. 4.2 Empirical studies on ImageNet-1K Recurrent form vs. matrix form. Given that the computational complexity difference between the matrix form and the recurrent form of VMI-SA is negligible, we use latency to measure the ac￾tual runtime efficiency difference between them. For comparison, we also report the … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The VMI-SA after removing attention-related operations. It can be observed that it shares the same overall structure as the ConvNeXt block, but differs in normalization methods and activation functions. Impact of mask matrices. For matrix-form VMI-SA, the mask matrix M…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 23 canonical work pages

  1. [1]

    Chen, S.-h

    J. Chen, S.-h. Kao, H. He, W. Zhuo, S. Wen, C.-H. Lee, and S.-H. G. Chan. Run, don’t walk: chasing higher flops for faster neural networks. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 12021–12031

    work page 2023

  2. [2]

    Dao and A

    T. Dao and A. Gu. Transformers are ssms: generalized models and ef- ficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    arXiv 2024

  3. [3]

    Gu and T

    A. Gu and T. Dao. Mamba: linear-time sequence modeling with selec- tive state spaces. Preprint arxiv:2312.00752, 2023

    arXiv 2023

  4. [4]

    D. Han, X. Pan, Y . Han, S. Song, and G. Huang. Flatten transformer: vi- sion transformer using focused linear attention. In 2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 5938–5948

    work page 2023

  5. [5]

    D. Han, Z. Wang, Z. Xia, Y . Han, Y . Pu, C. Ge, J. Song, S. Song, B. Zheng, and G. Huang. Demystify mamba in vision: A linear attention perspective. In NeurIPS, volume 37, pages 127181–127203, 2024

    work page 2024

  6. [6]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

    2016

  7. [7]

    Huang, X

    T. Huang, X. Pei, S. You, F. Wang, C. Qian, and C. Xu. Localmamba: visual state space model with windowed selective scan. Preprint arXiv:2403.09338, 2024

    arXiv 2024

  8. [8]

    Katharopoulos, A

    A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 5156–5165, 2020

    work page 2020

Show all 28 references
  1. [9]

    Y . Li, S. Xie, X. Chen, P. Dollár, K. He, and R. B. Girshick. Bench- marking detection transfer learning with vision transformers. Preprint arXiv:2111.11429, 2021

    2021 arXiv

  2. [10]

    X. Liu, H. Peng, N. Zheng, Y . Yang, H. Hu, and Y . Yuan. Efficientvit: memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14420–14430,

  3. [11]

    Y . Liu, Y . Tian, Y . Zhao, H. Yu, L. Xie, Y . Wang, Q. Ye, J. Jiao, and Y . Liu. Vmamba: Visual state space model. In NeurIPS, volume 37, pages 103031–103063, 2024

    2024

  4. [12]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted win- dows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992–10002,

    2021

  5. [13]

    Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A con- vnet for the 2020s. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11966–11976,

  6. [14]

    X. Ma, X. Dai, Y . Bai, Y . Wang, and Y . Fu. Rewrite the stars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5694–5703, 2024

    2024

  7. [15]

    Mehta, A

    H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur. Long range lan- guage modeling via gated state spaces. In The Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  8. [16]

    Mehta and M

    S. Mehta and M. Rastegari. Separable self-attention for mobile vision transformers. Transactions on Machine Learning Research, 2023. ISSN 2835-8856

    2023

  9. [17]

    X. Pei, T. Huang, and C. Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6443–6451, 2025

    2025

  10. [18]

    Radosavovic, R

    I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár. De- signing network design spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10428– 10436, 2020

    2020

  11. [19]

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis., 128(2):336–359, 2020

    2020

  12. [20]

    Shaker, M

    A. Shaker, M. Maaz, H. Rasheed, S. Khan, M.-H. Yang, and Khan. Swiftformer: efficient additive attention for transformer-based real-time mobile vision applications. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 17379–17390, 2023

    2023

  13. [21]

    Touvron, M

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jé- gou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (ICML) , pages 10347–10357, 2021

    2021

  14. [22]

    P. K. A. Vasu, J. Gabriel, J. Zhu, O. Tuzel, and A. Ranjan. Mobileone: an improved one millisecond nobile backbone. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7907–7917

  15. [23]

    W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In2021 IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 548–558, 2021

    2021

  16. [24]

    W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415–424, 2022

    2022

  17. [25]

    T. Xiao, Y . Liu, B. Zhou, Y . Jiang, and J. Sun. Unified perceptual pars- ing for scene understanding. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Pro- ceedings, Part V, volume 11209, pages 432–448, 2018

    2018

  18. [26]

    C. Yang, Z. Chen, M. Espinosa, L. Ericsson, Z. Wang, J. Liu, and E. J. Crowley. Plainmamba: improving non-hierarchical mamba in visual recognition. In 35th British Machine Vision Conference 2024, BMVC 2024

    2024

  19. [27]

    C. Yang, Y . Wang, J. Zhang, H. Zhang, Z. Wei, Z. Lin, and A. Yuille. Lite vision transformer with enhanced self-attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11998–12008, 2022

    2022

  20. [28]

    L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang. Vision mamba: efficient visual representation learning with bidirectional state space model. In Forty-first International Conference on Machine Learn- ing (ICML) 2024

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.