{"id":"899f4549-f27a-457f-ac4a-c7ce74e1820d","arxiv_id":"2502.00594","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FastVim reduces Vision Mamba's SSM parallel scan steps from log(h^2) to log(h) by alternately mean-pooling tokens across rows or columns, delivering up to a 72.5% inference speedup at 2048x2048 with roughly unchanged accuracy.","lead":"A new variant of the Vision Mamba architecture called FastVim pools image tokens along one dimension before the selective-scan step, halving the number of parallel scan steps. This makes high-resolution image classification, segmentation, and detection up to 72.5% faster at inference while keeping accuracy close to the baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FastVim's no-degradation claim is confounded: no Vim-T/S baseline with the same post-SSM LayerNorm is provided; Vim-B gained +1.9 from the norm, so pooling's effect on T/S is untested.","rationale":"After reading the full text and supplement, the efficiency analysis (Table 23) is internally consistent: the SSM scan time reduction from 58.20 ms to 0.30 ms at 2048 resolution and the 72.5% overall speedup follow directly from the reduced sequence length after pooling. The reader's information-loss concern is legitimate but is a modeling-premise gap rather than a demonstrated flaw; the paper is explicitly empirical and provides broad benchmark evidence. The more pointed issue is experimental control: FastVim differs from Vim not only by pooling but also by the normalization architecture. Sec. 4.1 shows Vim-B improved from 80.7 to 82.6 with post-SSM LayerNorm, a +1.9 gain. FastVim-B ties the augmented Vim-B, but FastVim-T and FastVim-S are compared to original Vim-T/S without this augmentation. The Tiny comparison is actually a 0.7-point loss (75.4 vs 76.1), already contradicting the conclusion's 'without any performance degradation.' The missing controlled baselines mean the central claim is not yet demonstrated; the paper should remain CONDITIONAL until same-normalization Vim baselines are provided. This aligns with the reader's call for missing baselines, though the reader's stated weakest assumption focuses on information loss rather than this confound, hence partial agreement.","tokens_in":23882,"tokens_out":5949,"duration_ms":60370,"concrete_test":"Train Vim-T and Vim-S from scratch on ImageNet-1k using the exact FastVim normalization scheme (RMSNorm input norm plus post-SSM LayerNorm) and identical training settings (300 epochs, drop path rates, optimizer, augmentation), but with the original Vim SSM scan and no pooling/repeat. Compare the resulting top-1 accuracy to FastVim-T (75.4) and FastVim-S (81.1). If the same-normalization Vim exceeds FastVim by more than 0.5 points, the no-degradation claim is explained by the norm addition rather than by pooling, and the central claim fails. If the accuracies are within noise, the confound is benign and the pooling approach is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FastVim matches Vim 'without any performance degradation' is not established because the comparison is confounded by an architectural change. FastVim adds a post-SSM LayerNorm and uses an RMS-LN normalization combination (Fig. 2, Sec. 4.1, Table 17), whereas the Vim-T and Vim-S baselines in Table 1 are the original numbers without post-SSM LayerNorm. The paper supplies a Vim-B baseline with the added norm (Vim-B w/ LN, 82.6) and FastVim-B ties it, but no such controlled baseline exists for Tiny or Small. This matters because Sec. 4.1 reports that adding post-SSM LayerNorm improved Vim-B from 80.7 to 82.6, a +1.9 gain. If a similar gain applies to Vim-T and Vim-S, the gap between FastVim and a same-normalization Vim would be larger than reported, and the observed parity could be attributable to the norm addition rather than to the pooling strategy being lossless. Furthermore, Table 1 already shows FastVim-T at 75.4 versus Vim-T at 76.1, a 0.7-point drop that contradicts the 'without any performance degradation' wording in the conclusion. Without Vim-T/S trained with the exact FastVim normalization recipe and identical hyperparameters, the claimed no-degradation result is untestable from the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FastVim, a modification of Vision Mamba (Vim) that applies mean pooling across one spatial dimension of the token grid before the SSM scan, alternating row/column pooling across layers, and then repeats the scan output to restore the original resolution. This reduces the number of parallel scan steps from log(h^2) to log(h). The authors report up to 72.5% inference speedup at 2048x2048 resolution, comparable or better accuracy on ImageNet-1k, ADE20K, COCO, and JUMP-CP, and a Mamba-based state of the art of 86.7% with MAE pretraining. They also introduce FastMaskVim and FastChannelVim for masked and per-channel tokenization settings. Code is provided at github.com/insitro/FastVim.","tokens_in":24131,"tokens_out":6546,"duration_ms":63766,"significance":"The complexity reduction is exact and the throughput measurements are detailed and internally consistent: Table 23 shows Vim's SSM scan time growing by roughly 74x from 224 to 2048 resolution, while FastVim's scan time stays nearly flat. The pooling idea is simple, parameter-free, and backed by useful ablations on alternating dimensions, pooling types, and norm combinations. The code release and reproduction details are strengths. However, the headline accuracy-parity claim is not yet cleanly established because the normalization recipe is changed at the same time as the pooling strategy, and the reported Tiny results already show a 0.7-point drop relative to the Vim baseline.","major_comments":[{"comment":"The comparison against Vim-T and Vim-S is confounded by a simultaneous architectural change: FastVim uses an RMS input norm plus a post-SSM LayerNorm (Table 17) and adds the post-SSM LayerNorm, while the Vim-T and Vim-S rows in Table 1 are the original published numbers without that norm. Section 4.1 reports that adding post-SSM LayerNorm improved Vim-B from 80.7 to 82.6, a +1.9 gain. Without Vim-T and Vim-S baselines retrained with the identical normalization recipe and matching hyperparameters (including drop path), the observed parity cannot be attributed to the pooling strategy; the reported numbers are equally consistent with the added norm being responsible for part or all of the difference. The authors should provide controlled Vim-T/S baselines with the same RMS-LN and post-SSM LayerNorm settings before claiming that pooling causes no performance degradation.","section":"Sec. 4.1, Table 1; Sec. 9, Table 17"},{"comment":"The conclusion states that FastVim 'achieves this without any performance degradation compared to the baseline Vim model across multiple tasks,' but Table 1 reports FastVim-T at 75.4 versus Vim-T at 76.1, a 0.7-point drop. Regardless of whether this is within run-to-run noise, the sentence as written is internally inconsistent with the reported numbers. Please qualify the claim (e.g., 'within noise' or 'for Small/Base and downstream tasks') or provide multiple seeds to demonstrate that the Tiny gap is not real.","section":"Conclusion; Table 1"},{"comment":"The MAE experiments depend on an additional, non-architectural scaling factor of 0.25 applied when pooling during fine-tuning and linear probing. Table 15 shows the effect is drastic: linear probing collapses from 60.2% to 0.02% without the scaling factor. This indicates that the constant-divide pooling is not scale-invariant across masking regimes and that the transfer recipe contains a dataset-specific correction. The paper should present this as a limitation of the current pooling scheme rather than as part of a fully parameter-free method, and it should acknowledge that the no-degradation claim for MAE pretraining rests on this correction.","section":"Supplement 8, Tables 14-15"}],"minor_comments":[{"comment":"Algorithm 1 shows pooling along the second spatial dimension with no transpose, while the main text explains that the token grid is transposed every block to alternate pooling dimensions. Please make the algorithm consistent with the described implementation.","section":"Algorithm 1 and Sec. 3.1"},{"comment":"The phrase 'a 324% speedup' should be 'a 3.24x speedup' (or 'a 224% speedup') to avoid ambiguity about whether the total is 324% of the original or an increase of 324%.","section":"Sec. 4.2"},{"comment":"The caption contains a typo: 'btoh' should be 'both'.","section":"Table 2 caption"},{"comment":"The word 'insipired' should be 'inspired'.","section":"Sec. 4.4"},{"comment":"References [51] and [52] are the same paper (Ren et al., 'Autoregressive pretraining with mamba in vision') and should be merged.","section":"References"},{"comment":"The cross-reference for the LayerNorm post-SSM throughput comparison points to 'Fig. 3', which is the loss-stability figure, while the actual comparison appears in Supplement Fig. 10. Please fix the cross-reference.","section":"Sec. 4.2 and Supplement Fig. 10"},{"comment":"The 72.5% speedup is stated without conditions; please specify that it is measured at 2048x2048 resolution, batch size 128, on an H100, as reported in Sec. 4.2.","section":"Abstract and Sec. 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid engineering contribution and the complexity analysis is correct. The main risk is that the accuracy-parity claim outruns the experimental controls; the requested controlled baselines are feasible within the manuscript's scope (retraining Vim-T/S with the same norm recipe), so I recommend major revision rather than rejection. Please also ask the authors to reconcile the Table 1 Tiny result with the conclusion's wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea is real: alternate mean pooling across rows/columns before the SSM scan, then repeat the output back to full resolution within each block. That cuts parallel scan depth from log(h^2) to log(h), and the throughput tables back it up — SSM scan time drops from 58.2 ms to 0.30 ms per block at 2048x2048, and the overall speedup of up to 72.5% on H100 is believable. The ablation showing that alternating the pooling dimension is necessary (81.1 vs 80.0/79.6 for fixed pooling) is a genuine contribution. The MAE extension and per-channel variant are useful additions, and the code is released.\n\nThe paper's central claim is \"no performance degradation compared to baseline Vim.\" That claim is not cleanly established. The stress-test note is correct: FastVim adds a post-SSM LayerNorm, and the Vim-T/S baselines in Table 1 are the original numbers without it. The paper shows Vim-B jumps from 80.7 to 82.6 with the norm, so the norm is worth +1.9 for base. If a similar gain holds for tiny and small, the pooling itself may be costing more than the raw numbers suggest. And Table 1 already shows FastVim-T at 75.4 vs Vim-T 76.1 — a 0.7-point drop that contradicts the \"without any performance degradation\" conclusion. The authors should train Vim-T/S with the same RMS-LN normalization recipe and report those numbers. That is a fixable omission.\n\nOther soft spots: the \"state-of-the-art\" phrase in the abstract is overbroad — the 86.7% MAE result is SOTA only among pure Mamba encoders on ImageNet, and the authors acknowledge ViT does better. No comparison is made against the closest token-reduction baselines (Famba-V, Vim-prune), which the related work cites; a throughput table against them would be the obvious missing experiment. Accuracy differences of 0.1–0.3 points are reported from single runs without variance, so we cannot assess whether the parity is real or noise. These are all fixable.\n\nI do not think the stress-test note invalidates the efficiency mechanism. The complexity reduction is exact and the measured speedups are internally consistent. The paper deserves a serious referee — the method is simple, possibly useful, and the claims are testable. I would want the reviewers to push for the controlled Vim-T/S baseline and a toned-down conclusion, but this is not a desk reject.\n\nRecommendation: send to peer review. It will be a constructive paper worth engaging with.","headline":"A simple, plausible efficiency trick for Vision Mamba with solid throughput measurements, but the 'no degradation' claim is confounded by a normalization change and needs a controlled baseline.","tokens_in":24731,"tokens_out":2561,"would_cite":true,"duration_ms":25006,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FastVim claims that a Vision Mamba encoder can match full-model accuracy while scanning half the tokens, by mean-pooling the token grid along alternating spatial dimensions each layer.","keywords":["vision state space models","Mamba","mean pooling","parallel scan","inference speedup","masked autoencoders","per-channel tokenization","high-resolution imaging"],"falsifier":"Take two 224x224 images that are provably identical after the column-mean and row-mean pooling steps used in FastVim's early blocks (for example, images that differ only in high-frequency variation within each pooled row), and test whether a full Vim model classifies them differently while FastVim produces identical logits.","tokens_in":23632,"feed_emoji":"⚡","tokens_out":6074,"duration_ms":54291,"temperature":0.7,"pith_summary":"FastVim claims that a Vision Mamba (Vim) encoder can match the accuracy of the full model while scanning far fewer tokens, by mean-pooling the 2D token grid along one spatial dimension before each selective state-space scan and alternating the pooled dimension across layers. This cuts the SSM's parallel scan steps from log2($h^{2}$) to log2(h), reducing the per-block SSM scan time at 2048x2048 images from 58.20 ms to 0.30 ms and delivering up to a 72.5% overall inference speedup. The paper reports parity with Vim on ImageNet-1k classification, ADE20K segmentation, COCO detection and instance segmentation, and JUMP-CP cell perturbation prediction, and a new state-of-the-art 86.7% ImageNet-1k accuracy for a Mamba-based encoder when combined with masked autoencoding. If these results hold, high-resolution and long-token vision tasks become far more practical on SSM backbones without an accuracy penalty.","feed_headline":"Mean pooling speeds Vision Mamba 72.5% while keeping accuracy","feed_subtitle":"Alternating row/column pooling halves SSM scan steps and delivers Vim-level accuracy at high resolution.","key_machinery":"The load-bearing mechanism is the pool-scan-repeat roundtrip: a parameter-free mean pooling over one spatial axis of the token grid before the selective SSM scan, a transposition to alternate the pooled axis every layer, and a repetition operation that restores the original token count before the Dxt skip connection. This compresses the sequence seen by the parallel scan from L = h x w tokens to h tokens, halving the number of parallel scan steps, while the residual path and subsequent blocks are what the paper relies on to preserve information that pooling may have discarded.","core_discovery":"The central claim is that the full pairwise token contextualization inside Vim's SSM scan is not necessary at every layer: after a 1D convolution, mean-pooling tokens across columns (or rows) reduces the scan input from $h^{2}$ to h tokens, the SSM processes only the pooled tokens, and the output is repeated back to full resolution before the Dxt skip connection and norm. By transposing the token grid at each block, FastVim alternates which dimension is pooled, so every token still interacts with tokens in other rows and columns across successive layers. The paper argues empirically that this alternation is required, not optional, for accuracy, that the scheme works in Mamba but fails in Vision Transformers, and that the resulting architecture shows no performance degradation relative to the Vim baseline across classification, segmentation, detection, and cell-perturbation tasks, despite contextualizing significantly fewer tokens per scan.","pith_inferences":["Beyond the paper: the success of such sparse, alternating scans suggests Mamba's per-layer contextualization is highly redundant, so even sparser interaction patterns, such as pooling every other layer or fusing tokens across multiple blocks, may preserve accuracy while cutting cost further.","Beyond the paper: the same pool-scan-repeat cycle could be applied along the channel or time axis for video and 3D microscopy data, reducing scan steps by additional log factors; the paper's 2D pooling results already hint this works at smaller patch sizes.","Beyond the paper: the contrast between pooling succeeding in Mamba and failing in ViT points to a structural difference in how recurrent state propagation distributes information across steps, which could be characterized theoretically as a rank or memory property of the SSM hidden state.","A testable extension the paper does not run: ablate the Dxt skip connection in FastVim to measure how much of the preserved accuracy comes from the residual path versus the pooled scan; the authors only moved the decompression after the skip connection, not removed the connection."],"forward_implications":["FastVim cuts the SSM scan time in a block from 58.20 ms to 0.30 ms at 2048x2048 input, and the whole model runs up to 72.5% faster than Vim at that resolution.","The speed gap over Vim widens as resolution increases, and FastVim becomes faster than ViT at 1024x1024 and above while consuming less memory.","FastMaskVim reaches 86.7% ImageNet-1k top-1 accuracy, the current best for a Mamba-based visual encoder, with faster pre-training and fine-tuning than Vim.","FastChannelVim matches ChannelVim accuracy on JUMP-CP while increasing throughput by 62.3% at patch size 8, and improves over ChannelViT by 8.3 points.","Because the pooling is parameter-free, the technique can be dropped into other Mamba-based vision architectures to accelerate their SSM scans as well."],"supporting_citations":[{"why":"The Vision Mamba baseline whose architecture and training settings FastVim modifies and whose accuracy it matches.","marker":"[70]"},{"why":"Defines the selective state space model with input-dependent B, C, Δ that produces the SSM recurrence being accelerated.","marker":"[19]"},{"why":"The parallel scan algorithm that gives the log(L) parallel-step count; FastVim's speedup is a reduction of this step count.","marker":"[57]"},{"why":"Masked Autoencoders: the self-supervised pretraining framework used for FastMaskVim and the source of its fine-tuning recipe.","marker":"[24]"},{"why":"ChannelViT: per-channel tokenization and hierarchical channel sampling that FastChannelVim extends to Mamba.","marker":"[3]"},{"why":"VMamba: supplies the post-SSM LayerNorm stability modification adopted in FastVim-B and used for fair comparisons.","marker":"[39]"},{"why":"JUMP-CP dataset: the microscopy benchmark for the 160-way cell perturbation prediction experiments.","marker":"[8]"}],"fun_headline_variants":["Pooling spatial tokens halves scan steps, speeds Mamba 72.5%","Mean-pool tokens to cut Mamba scan steps, gain 72.5% speed","Alternating pooling speeds Vision Mamba 72.5% with no accuracy loss","Pooling rows and columns halves Mamba steps, boosting speed 72.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's results rest on the empirical premise that alternating mean pooling over rows and columns preserves enough token-interaction information that the residual skip connection and later blocks can recover what a full per-token scan would have provided; if mean pooling discards signal that the skip connections cannot restore, the claimed parity with Vim would fail.","fun_headline_variants_meta":{"raw":{"variants":["Pooling spatial tokens halves scan steps, speeds Mamba 72.5%","Mean-pool tokens to cut Mamba scan steps, gain 72.5% speed","Alternating pooling speeds Vision Mamba 72.5% with no accuracy loss","Pooling rows and columns halves Mamba steps, boosting speed 72.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2947,"prompt_tokens":949,"completion_tokens":1998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1908}},"tokens_in":565,"tokens_out":1998,"duration_ms":17497,"temperature":1.0,"reasoning_tokens":1908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:24:21.571384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two 224x224 images that are provably identical after the column-mean and row-mean pooling steps used in FastVim's early blocks (for example, images that differ only in high-frequency variation within each pooled row), and test whether a full Vim model classifies them differently while FastVim produces identical logits.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Masked Autoencoders: the self-supervised pretraining framework used for FastMaskVim and the source of its fine-tuning recipe."},{"cited_title":"Vmamba: Visual state space model, 2024","cited_arxiv_id":null,"evidence_quote":"VMamba: supplies the post-SSM LayerNorm stability modification adopted in FastVim-B and used for fair comparisons."}],"review_version":1}