Pith. sign in

REVIEW 4 major objections 4 minor 31 references

V"Mean"ba: Visual State Space Models only need 1 hidden dimension

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a mean transform over channels can replace the hidden dimension in visual state space models, giving up to a 1.12x speedup with under 3% accuracy loss.

desk verdict The paper's real contribution is the observation that VMamba's inner scan channels are near-constant and the resulting training-free mean-reduction, but the headline accuracy claim is fitted to the same validation set used to select layers, so the empirical case is not yet made. read the letter →

arxiv 2412.16602 v1 pith:S3TWNTUT submitted 2024-12-21 cs.CV cs.AI

Tien-Yu Chi , Hung-Yueh Chiang , Chi-Chih Chang , Ning-Chi Huang , Kai-Chiang Wu This is my paper · ORCID
classification cs.CVcs.AI
keywords StateSpaceModelsVMambaselectivescanmodelcompressiontraining-freeinferenceaccelerationchannelredundancysemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that visual state space models such as VMamba waste computation in their selective-scan blocks: each block's output is nearly constant across the inner channel dimension, so the scan inputs $\bar{A}$, $\bar{B}u$, and $C$ can be replaced by their channel-wise mean without retraining. The proposed VMeanba transform collapses the inner dimension from $D$ to $1$, runs the scan in the reduced space, and broadcasts the result back. On ImageNet-1k classification and ADE20k semantic segmentation, the paper reports up to a $1.12\times$ end-to-end speedup with less than 3% accuracy loss, and compatibility with 40% unstructured pruning. If the paper is right, a large fraction of the channel dimension in these scan blocks is redundant at inference time, making post-training acceleration cheaper than quantization or pruning.

What carries the argument

The load-bearing mechanism is a transform pair $(T, T^{-1})$ around the selective-scan recurrence, the inner state-space scan over sequence length $L$ that dominates Mamba-block FLOPs. $T$ is the mean operator along the inner channel dimension $D$: it maps the three scan inputs $\bar{A}$, $\bar{B}u$, $C$ to single-channel representatives, the scan runs with $D=1$, and $T^{-1}$ broadcasts the output back to the full width. In FLOP terms the scan and state update go from $9BDL$ to about $B(D+10)L$, an 89% reduction for large $D$, which is why collapsing the channel axis rather than pruning weights or quantizing values is the paper's chosen lever.

What would settle it

Measure, for every Mamba block of a pretrained VMamba model on ImageNet validation, the channel-wise variance of the block's output activations, and compare the model's accuracy before and after mean-collapsing only those blocks whose variance is near zero; if replacing a near-zero-variance block still costs more than 3 percent accuracy, the presumed link between channel constancy and scan fidelity is broken.

Watch

Extended reading notes

Core claim

The central claim is that the hidden dimension of the Mamba scan inside VMamba is essentially redundant. For each layer, the paper asserts $y_{\text{layer}}[:, d, :] \approx y_{\text{layer}}[:, d', :]$ for all inner channels $d, d'$, and it attributes this to the scan inputs $\bar{A}$, $\bar{B}u$, $C$ being jointly low-rank along the channel axis. The mechanism is a transform $T$ that averages those inputs over the channel dimension to dimension $1$, followed by the original Mamba scan and a broadcast $T^{-1}$, so that $y_{\text{layer}} = T^{-1}(\mathrm{Mamba}(T(\bar{A}, \bar{B}u, C)))$. A validation-set pipeline replaces only the $K$ layers whose substitution costs the least accuracy. With this setup the paper reports under 3% top-1 accuracy loss on ImageNet classification and under 3% all-pixel accuracy loss on ADE20k, up to $1.12\times$ end-to-end batch-inference speedup, and a similar accuracy bound when combined with 40% unstructured pruning of linear or convolutional weights.

Load-bearing premise

The argument rests on the claim that each Mamba block's output is nearly identical across its inner channel dimension, so averaging the scan inputs along that dimension and broadcasting the output back loses almost nothing.

Editorial extensions

If this is right

  • VMamba inference can be accelerated without retraining simply by mean-collapsing the scan inputs of the selected layers, making the method a drop-in post-training step.
  • The accuracy-versus-$K$ curve lets a user trade speed for accuracy per model, selecting the $K$ lowest-impact layers from a validation set.
  • Since VMeanba operates on the scan inputs while pruning operates on weights, the two optimizations are orthogonal in the paper's experiments, so they can be stacked for further speedup.
  • Kernel memory traffic drops by orders of magnitude (e.g., from 3.3G/823.5M bytes to 6.4M/1.3M bytes at one configuration), enabling longer scan sequences or larger batches on the same GPU.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension of the paper's mechanism is that other selective-scan vision backbones with similar channel-invariant activations would enjoy the same mean-collapse speedup; a quick test on a different SSM backbone would show whether the redundancy is architectural or specific to VMamba.
  • The paper's layer-selection pipeline needs a validation forward pass per candidate layer; that calibration cost is not included in the reported speedup, so a variance-based heuristic would be needed for truly one-shot deployment.
  • The paper does not examine whether channel variance stays low after pruning or at higher input resolutions, so the 3% accuracy bound should be expected to degrade in regimes where channel invariance breaks down.
  • The gap between the appendix's kernel-level speedups (up to 293x) and the end-to-end 1.12x suggests that fusing the reduced scan with discretization, and removing the mean/broadcast overhead, is the concrete next step to turn this into a larger wall-clock win.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VMeanba, a training-free compression method for the selective scan operation in VMamba. The method replaces the inner channel dimension of the scan inputs (A, Bu, C) with their mean along the channel axis, runs the scan with dimension 1, and broadcasts the output back to the original dimension. Layer selection is performed by scoring each layer with Eq. (3) on the validation set and choosing the K layers with the smallest score (Algorithm 1). Experiments on ImageNet classification and ADE20k segmentation report up to 1.12x end-to-end speedup with less than 3% accuracy loss, and a combination with 40% unstructured pruning is claimed to keep the drop under 3%.

Significance. If the empirical claims survive a properly held-out evaluation, the method would be practically useful: it is simple, training-free, addresses a real bottleneck (the selective scan kernel), and appears orthogonal to pruning. The complexity and memory reductions in Appendix B and Tables 2-3 are substantial in principle. The paper also ships a concrete selection algorithm rather than leaving layer choice entirely unspecified. However, the current evaluation protocol prevents assessing the central claim: all accuracy numbers are computed on the same validation set used for selection, and the kernel speedup comparison uses kernels with different scopes. These issues are fixable with additional experiments, so the work is not beyond repair.

major comments (4)
  1. [§2.2, Eq. (3), Algorithm 1, Appendix D, Table 4, Figure 4] The headline accuracy claim is in-sample by construction. Eq. (3) defines the layer impact score using Acc(VMeanba on layer) on the validation set, Algorithm 1 selects the K layers from those scores, and Figure 4/Table 4 report accuracy for the selected K on the same validation set. Appendix D explicitly states that only the validation set is used. Thus the reported '<3% accuracy loss' is the loss of a configuration already optimized on the exact images used for evaluation, and K is chosen post hoc from the same curve. This does not provide an independent estimate of generalization. Please report accuracy on the held-out test set (or a cross-validated estimate), with the layer-selection procedure fixed before evaluating on the test set, and include variance or confidence intervals.
  2. [§3.2, Table 1, abstract] The abstract's claim that combining VMeanba with 40% pruning keeps the accuracy drop under 3% is not supported by Table 1 for the Conv2D case. In Table 1, the Conv2D-pruned baseline is 80.1% and VMeanba+pruning is 77.5%, a drop of 2.6 points relative to the pruned baseline, but relative to the original Base VMamba accuracy of 83.7% (Table 4, K=0) the drop is 6.2 points. The Linear Layers case stays under 3% against either baseline, but the current table does not distinguish these baselines. Please state the comparison baseline explicitly and correct the abstract or add the missing comparison against the unpruned model.
  3. [Appendix D, Tables 2 and 3] The kernel speedup numbers compare kernels with different scopes. The original CUDA kernel includes both discretization and the scan operation, while the new kernel handles only the scan operation, with discretization executed outside the kernel. Reporting a 273x speedup for the scan kernel alone against a kernel that also performs discretization is not an apples-to-apples comparison and does not by itself justify the end-to-end speedups in Table 4. Please report kernel-level timings for identical work scopes, or a full-block comparison that includes the discretization cost.
  4. [§2.1, Eq. (1), Figure 3] The core modeling assumption that y_layer[:, d, :] ≈ y_layer[:, d', :] is supported only by visual inspection of Figure 3. There is no quantitative variance threshold, no error metric between the full scan and the reduced scan, and no evidence that the property transfers across model sizes, resolutions, or after pruning. Because the entire method rests on this near-constant-channel property, please provide quantitative measurements, for example the ratio of channel variance to total activation norm per layer, the relative output error of the reduced scan versus the full scan, and the per-layer accuracy sensitivity. Without such evidence, the claim that 'only 1 hidden dimension' is sufficient is not established.
minor comments (4)
  1. [§2.2] There is a typo: 'indroduce' should be 'introduce'. Also Eq. (2) writes y_layer = T^{-1}(Mamba(T(A, Bu, C))) but does not specify how T acts on the product Bu; since u is computed inside the block, the notation should be clarified.
  2. [Table 4] The column header 'Accuracy (Acc@1 / aAcc)' is confusing for an image classification table; aAcc is only used for segmentation. Please separate the metrics for the two tasks.
  3. [Appendix B, Eq. (10)] The FLOP notation in Eq. (10) is typeset inconsistently (e.g., 'F LOPM amba'), and the variable d in '9BLd' is not defined at that point. Please define all symbols and give a step-by-step derivation of the reduced FLOP count, including the mean and broadcast costs.
  4. [Appendix D] The sentence 'We only use the validation set of them for the experiments' is a significant protocol statement; it should be either removed after switching to test-set evaluation or accompanied by an explicit explanation of why no test set is used. The paper also does not provide code or checkpoints, which would help reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

The <3% accuracy-loss claim is fitted to the same validation set used to select layers via Eq. (3) and Algorithm 1, so it is an in-sample statistic, not an independent prediction.

  1. fitted input called prediction [Section 2.2 (Eq. 3), Algorithm 1 (Appendix C), Appendix D, Table 4; abstract.]
    "We treat the choices of layers as a hyperparameter, determined using the validation set. Specifically, we calculate the layer impact score S_layer for each layer, and select the layers with the K smallest scores to apply the VMeanba optimization. The impact score is defined by equation (3): S_layer = Acc(OriginalModel) - Acc(VMeanba on layer) ... where Acc represents the model accuracy on the validation set. ... We only use the validation set of them for the experiments."

    Equation (3) defines the layer-selection score as the accuracy drop produced by applying VMeanba to that layer, and Algorithm 1 selects the K layers with the smallest scores using D_val. Appendix D states that all experiments use only the validation set. Table 4 then reports Acc@1 for the resulting K=2,4,8 configurations on that same validation set. Hence the headline 'less than 3% accuracy loss' is not an independent prediction: the method was explicitly tuned to minimize accuracy loss on the exact images used to evaluate it. This is selection bias / fitting the reported metric by construction; a held-out test set would be required to verify the claim.

full rationale

The central circularity is in the evaluation protocol, not in the VMeanba mechanism itself. Section 2.2 defines S_layer as the validation-set accuracy drop of each candidate layer, Algorithm 1 picks the K layers with the smallest drops, and Appendix D confirms that only the validation set is used. Table 4's 'Accuracy' column is therefore measured on the same images that were used to choose the layers and K. The abstract's 'less than 3% accuracy loss' is consequently a fitted, in-sample statistic rather than an independent prediction. The channel-low-variance observation and the mean/broadcast transform are empirical and not circular by themselves; the speedup measurements are kernel-level and not fitted. The pruning comparison in Table 1 is against an already-pruned baseline, which is a reporting ambiguity rather than a circular step. No load-bearing self-citations or imported uniqueness theorems are present, and the external VMamba/Mamba baselines are not authored by this paper's team. The score of 6 reflects that one of the paper's headline empirical claims reduces, by its own equations and protocol, to a selection procedure over the metric being reported.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on one hand-chosen hyperparameter K, a per-layer selection procedure fitted to the validation set, and the unproven premise that output channels of the scan are near-constant. No new physical or architectural entities are introduced.

free parameters (2)
  • K (number of VMeanba layers) = 4 for headline 1.12x speedup; 8 in pruning experiments; 10 suggested in text
    Chosen by hand after inspecting the accuracy-versus-K tradeoff on the validation set (Figure 4). No separate hyperparameter validation split is used.
  • Layer selection (which K layers are converted) = The K layers with smallest impact score S_layer on the validation set
    The per-layer binary compression decision is effectively fitted to the validation set through Eq. (3), which defines the impact score in terms of the accuracy drop on that same set.
assumptions (3)
  • domain assumption The pretrained VMamba model and its SSM scan equations (Eqs. 4-9) correctly implement the published Mamba and VMamba algorithms.
    The paper builds entirely on the VMamba/Mamba pretrained backbones and their CUDA kernels, and does not re-derive or validate them.
  • ad hoc to paper The output activations of Mamba blocks are approximately constant across the inner channel dimension (Eq. 1).
    This is the key empirical premise of VMeanba. It is asserted from a visual inspection of Figure 3, with no quantitative variance threshold or statistical test.
  • domain assumption The validation set used for layer selection is representative of the test distribution, so accuracy measured on it is a fair proxy for generalization.
    The same validation set is used both to select layers and to report accuracy, which assumes no overfitting to this set despite the explicit optimization of layer choices on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of V"Mean"ba: Visual State Space Models only need 1 hidden dimension." pith.science (2026). https://pith.science/paper/S3TWNTUT

@misc{pith2026241216602,
  author       = {Pith},
  title        = {Pith review of: V"Mean"ba: Visual State Space Models only need 1 hidden dimension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3TWNTUT}},
  note         = {Machine review of arXiv:2412.16602}
}
read the original abstract

Vision transformers dominate image processing tasks due to their superior performance. However, the quadratic complexity of self-attention limits the scalability of these systems and their deployment on resource-constrained devices. State Space Models (SSMs) have emerged as a solution by introducing a linear recurrence mechanism, which reduces the complexity of sequence modeling from quadratic to linear. Recently, SSMs have been extended to high-resolution vision tasks. Nonetheless, the linear recurrence mechanism struggles to fully utilize matrix multiplication units on modern hardware, resulting in a computational bottleneck. We address this issue by introducing \textit{VMeanba}, a training-free compression method that eliminates the channel dimension in SSMs using mean operations. Our key observation is that the output activations of SSM blocks exhibit low variances across channels. Our \textit{VMeanba} leverages this property to optimize computation by averaging activation maps across the channel to reduce the computational overhead without compromising accuracy. Evaluations on image classification and semantic segmentation tasks demonstrate that \textit{VMeanba} achieves up to a 1.12x speedup with less than a 3\% accuracy loss. When combined with 40\% unstructured pruning, the accuracy drop remains under 3\%.

Figures

Figures reproduced from arXiv: 2412.16602 by the authors.

Figure 1
Figure 1. The GPU kernel time of each operation in a VMamba block. The latency is measured using feature maps with an input resolution of 224×224. We rank the kernels by their latency and highlights the top-5 time-consuming kernels on the bar chart. The selective scan operation is one of the major contributors in the VMamba block. To this end, we first analyze the latency break￾down of VMamba [16] and identify the selective s… view at source ↗
Figure 2
Figure 2. Overview of the VMeanba block. VMeanba reduced the channel dimension of the inputs to the associated scan operation by applying a transform T, thereby simplifying the computation. The proposed VMeanba components are highlighted in red, while the original selective scan components are shown in blue and green, with the green block indicating the main area of optimization. 2 Methods 2.1 Distribution Analysis of VMamba … view at source ↗
Figure 3
Figure 3. The figure illustrates the distribution of inner dimension values of [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Our results indicate that the model accuracy remains largely unaffected when an appropriate [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 14 canonical work pages

  1. [1]

    Beltagy, M

    I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  2. [2]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  3. [3]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021

    2021

  4. [4]

    D. Y . Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re. Hungry hungry hippos: Towards language modeling with state space models. In ICLR, 2022

    work page 2022

  5. [5]

    Girshick

    R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015

    2015

  6. [6]

    Girshick, J

    R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014

    2014

  7. [7]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces, May

    work page

  8. [8]

    A. Gu, K. Goel, and C. Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2021

    work page 2021

Show all 31 references
  1. [9]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778, 2016

    2016

  2. [10]

    R. E. Kalman. A new approach to linear filtering and prediction problems. 1960

    1960

  3. [11]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NeurIPS, pages 1106–1114, 2012. 8

    2012

  4. [12]

    K. Li, X. Li, Y . Wang, Y . He, Y . Wang, L. Wang, and Y . Qiao. Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977, 2024

    2024 arXiv

  5. [13]

    H. Lin, G. Han, J. Ma, S. Huang, X. Lin, and S.-F. Chang. Supervised masked knowledge dis- tillation for few-shot transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19649–19659, 2023

    2023

  6. [14]

    Y . Lin, T. Zhang, P. Sun, Z. Li, and S. Zhou. Fq-vit: Post-training quantization for fully quantized vision transformer. arXiv preprint arXiv:2111.13824, 2021

    2021 arXiv

  7. [15]

    X. Liu, H. Peng, N. Zheng, Y . Yang, H. Hu, and Y . Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420–14430, 2023

    2023

  8. [16]

    Y . Liu, Y . Tian, Y . Zhao, H. Yu, L. Xie, Y . Wang, Q. Ye, and Y . Liu. VMamba: Visual State Space Model, Apr. 2024. arXiv:2401.10166 [cs]

    2024 arXiv

  9. [17]

    Z. Liu, H. Hu, Y . Lin, Z. Yao, Z. Xie, Y . Wei, J. Ning, Y . Cao, Z. Zhang, L. Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In CVPR, pages 12009–12019, 2022

    2022

  10. [18]

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021

    2021

  11. [19]

    Z. Liu, Y . Wang, K. Han, W. Zhang, S. Ma, and W. Gao. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103, 2021

    2021

  12. [20]

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV , et al. RWKV: reinventing rnns for the transformer era. InEMNLP, pages 14048–14077, 2023

    2023

  13. [21]

    Redmon, S

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real- time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016

    2016

  14. [22]

    Simonyan and A

    K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015

    2015

  15. [23]

    Y . Teng, Y . Wu, H. Shi, X. Ning, G. Dai, Y . Wang, Z. Li, and X. Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405.14224, 2024

    2024 arXiv

  16. [24]

    Touvron, M

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pages 10347–10357, 2021

    2021

  17. [25]

    S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  18. [26]

    T. Xiao, Y . Liu, B. Zhou, Y . Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418–434, 2018

    2018

  19. [27]

    H. Yang, H. Yin, P. Molchanov, H. Li, and J. Kautz. Nvit: Vision transformer compression and parameter redistribution. 2021

    2021

  20. [28]

    B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017

    2017

  21. [29]

    L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In ICML, 2024

    2024

  22. [30]

    M. Zhu, Y . Tang, and K. Han. Vision transformer pruning.arXiv preprint arXiv:2104.08500, 2021. 9

    2021 arXiv

  23. [2024]

    arXiv:2312.00752 [cs]

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.