REVIEW 2 major objections 3 minor 31 references
Sparsified State-Space Models are Efficient Highway Networks
T0 review · 2 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Hierarchical token pruning of pre-trained state-space models—heavier in upper layers—creates highway-like shortcuts that beat dense Mamba at matched FLOPs and improve long-context perplexity.
desk verdict Solid empirical token pruning for SSMs that beats matched-FLOPs Mamba and improves long-context perplexity, but the influence-score derivation silently approximates an exact counterfactual and the highway evidence is partly self-referential. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the token influence score in Eq. (4), which unrolls the SSM recurrence so that the change in final output caused by deleting token $x_t$ is expressed as $C_T(\prod_{k=t+1}^{T}\bar{A}_k)\bar{B}_t x_t$, aggregated by max pooling over channels. This score turns a global counterfactual question—which tokens matter for the final output—into a per-token quantity computed from local recurrence steps, and it drives the pruning decision at every layer. The second piece is the linear pruning schedule: the fraction of kept tokens decreases steadily with depth, so the network becomes trapezoidal and upper layers act as highways that let early information pass through without dense processing.
What would settle it
Run the true counterfactual: remove a token from a Mamba sequence, recompute the forward pass, and measure the actual change in the final output logits; then compare the ranking of tokens by this true effect with the ranking by Eq. (4). If the two rankings diverge strongly on realistic inputs, or if pruning by the true counterfactual fails to reproduce Simba's gains, the central mechanism is not what the paper claims.
Extended reading notes
Core claim
The paper's discovery is that dense recurrence is itself a bottleneck: upper layers of Mamba encode global, redundant information, and forcing every token through every recurrence step attenuates early-token influence on the final output. Simba removes tokens layer by layer using a score $\Delta y_T(t) = C_T(\prod_{k=t+1}^{T}\bar{A}_k)\bar{B}_t x_t$, which estimates the global effect of token $x_t$ on the final output by accumulating the local recurrence path. Pruning is applied hierarchically with a linear schedule so that upper layers keep only a small fraction of tokens, giving the network a trapezoidal shape. The authors report that this training-free procedure yields better accuracy at matched FLOPs on six NLP benchmarks, better PG-19 perplexity, and—unlike dense Mamba—decreasing perplexity past the 2k context limit, which they attribute to highway-like shortcuts in the upper layers.
Load-bearing premise
The score that decides which tokens to prune assumes that removing a token only deletes that token's own contribution to the final output and leaves all the recurrence matrices unchanged, even though in Mamba those matrices depend on the input tokens.
Editorial extensions
If this is right
- At a fixed FLOPs budget, Simba-2.8b averages 62.5% accuracy on six NLP benchmarks, outperforming Mamba-1.4b (58.8%) and Pythia-2.8b (59.2%) while using computation comparable to the smaller Mamba model.
- On PG-19, Simba's perplexity keeps falling at 4k context while Mamba's rises, meaning the pruned model uses context beyond its training length better than the dense model.
- Moderate pruning can improve same-scale performance: at 130m, 370m, and 790m scale, Simba edges out the original Mamba on average downstream accuracy while using fewer FLOPs.
- Because no fine-tuning is required, the method applies directly to any pre-trained SSM, and a short fine-tuning step improves results further.
- The information-flow analysis shows that upper layers of Simba give more weight to early tokens than dense Mamba does, supporting the highway interpretation.
Reading between the lines
- Editorial inference: the score in Eq. (4) is a leave-one-out approximation; an exact variant that recomputes the input-dependent $\bar{A}_k$ and $\bar{B}_k$ after deletion might either strengthen or weaken the pruning gains.
- Editorial inference: the highway interpretation suggests a direct architectural test—insert explicit residual or skip connections into Mamba's upper layers and compare against Simba's pruning-only highways; if both improve long-context behavior in the same way, the mechanism is about information flow rather than sparsity per se.
- Editorial inference: because pruning decisions are made per input sequence and require no external data, the same score could be applied during autoregressive decoding or chunked streaming generation; the paper sketches chunked prefill but does not evaluate streaming generation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Simba, a training-free hierarchical token-pruning method for pre-trained state-space models (SSMs), applied here to Mamba. Simba removes tokens at every layer following a linear schedule that preserves more tokens in lower layers and fewer in upper layers, so the network becomes trapezoidal. The pruning criterion is a token influence score derived by unrolling the SSM recurrence and computing the contribution of each token to the final output. The authors report that Simba improves the FLOPs-accuracy frontier on six NLP benchmarks, achieves better PG-19 perplexity than dense Mamba at comparable FLOPs, and shows more stable long-context perplexity beyond the 2k training context. They interpret the sparse upper layers as highways that improve information flow from early tokens, and support this with an influence-based visualization across layers. The manuscript also includes ablations against uniform and random pruning, a fine-tuning experiment, and an inference-time study.
Significance. If the empirical claims hold, Simba is a practical, plug-and-play efficiency technique for Mamba-class models: it requires no fine-tuning, is evaluated on standard benchmarks with publicly released code, and its main comparisons are against externally evaluated dense baselines. The paper also contains honest limitations and a useful inference-time analysis. However, the conceptual contribution—that sparsified SSMs are 'efficient highway networks'—rests on an approximate influence score whose exactness is not established, and the information-flow evidence is partly circular because it uses the same score that defines the pruning. The empirical FLOPs-accuracy results are credible and would likely survive a corrected derivation, but the mechanistic interpretation needs to be revised or independently validated.
major comments (2)
- [Section 3.2, Eq. (4)] The derivation of the token influence score treats the transition matrices as fixed when a token is removed. In Mamba, the matrices \bar A_k, \bar B_k, and C_k are input-dependent, so deleting x_t changes more than the single term in Eq. (4). The exact counterfactual difference also contains contributions from earlier tokens whose products of \bar A's are altered by the absence of \bar A_t. The score is therefore an approximation, not the exact 'global impact' claimed in the abstract and Section 3.2. The paper neither states this simplification nor acknowledges it in the Limitations section. Since the pruning criterion and the highway evidence in Section 4.3 both rely on this score, the theoretical framing should be corrected, and the approximation should be validated (e.g., by comparing the score against oracle removal on a subset of data).
- [Section 4.3, Figure 5] The information-flow visualization uses the normalized influence score s(t)/||y_T||_2, which is the same approximate score used to define the pruning decisions. This makes the highway interpretation partly circular: the flattening observed at upper layers may be a mechanical consequence of pruning away low-influence tokens, rather than an independent measurement of how information is routed. To support the claim that upper layers act as highways, the manuscript needs an analysis that does not depend on Eq. (4), such as intervention-based attribution or a gradient-based importance measure.
minor comments (3)
- [Table 5 and Section 4.2] The statement that Simba 'consistently demonstrates decreasing perplexity even with extended contexts' is slightly overstated: Simba-2.8b shows perplexity 2.279 at 3.5k but 2.284 at 4k, so the trend is not strictly monotonic. Please qualify the claim as generally decreasing or note the small uptick at the longest tested length.
- [References] The citation 'Frankle & Carbin, 20189' appears in the text and reference list; the year should be 2019.
- [Figures 3 and 4 and Tables 4 and 5] The caption text says models use 'the same number of FLOPs,' but the actual FLOPs differ somewhat (e.g., Simba-2.8b at 6.07e12 versus Mamba-1.4b at 5.60e12). The comparison is still meaningful, but the wording should be 'comparable FLOPs' or the exact numbers should be cited.
Circularity Check
External FLOPs-accuracy benchmarks are independent, but the highway-mechanism claim reuses the pruning score that defines Simba, and Eq. (4)'s 'global influence' is the single frozen-recurrence term by construction.
-
self definitional
[Section 3.2, Eq. (4)]
"Formally, for a given token sequence (x1,··· , xT ) of length T and the final token output yT of the layer, we estimate the influence of token xt at position t by considering the updated output y(t) T (obtained by removing xt from yT), as follows: ∆yT (t) := yT− y(t) T = ... = CT (∏T k=t+1 ¯Ak) ¯Btxt. (4)"
The paper's own Eq. (3) says that in Mamba '¯At, ¯Bt, and Ct denote the input-dependent matrices created using xt'. Consequently, deleting xt changes the recurrence matrices at later positions and also changes the contributions of all earlier tokens through the missing transition at step t. Eq. (4) instead cancels the two sums while keeping every matrix fixed, so the 'updated output obtained by removing xt' is defined to be the original output minus exactly one term. The claimed global-influence score is therefore that single local term by construction; calling it the effect of removing xt is an assumption built into the definition, not a derived counterfactual.
-
self definitional
[Section 4.3, Figure 5]
"Specifically, we measure the influence of tokens at each position on the final token, using our token importance score in Eq. (4) but normalized to equalize the contribution of each input document, i.e., s(t)/||yT||2. See Appendix A.2 for more details."
Simba is defined by 'removing the tokens with the lowest scores' where the score is s(t) from Eq. (4). Figure 5 then plots the normalized s(t) for the surviving tokens. Because the pruning rule selects tokens with high s(t) and discards low-s(t) tokens, the upper-layer flattening of the influence curve is a direct consequence of the selection criterion: the remaining early tokens are, by construction, the ones the criterion judges most influential. Using the same score both to create the sparsified model and to demonstrate that the sparsified model improves information flow makes the highway evidence a restatement of the method rather than an independent confirmation.
full rationale
The central efficiency claims are externally grounded: Simba is compared on standard NLP benchmarks and PG-19 perplexity against dense Mamba and Pythia at matched FLOPs, with no fine-tuning for those evaluations, so those results are not circular. The HOMER self-citation (Song et al., 2024) appears in related work and in a general remark about context-length limits; it is not load-bearing for Simba's derivation. The circularity is confined to the conceptual highway claim: Eq. (4) presents a single-term deletion under frozen recurrence matrices as the exact influence of removing a token, and Section 4.3 uses that same score to demonstrate the highway effect. The information-flow evidence is therefore partly a restatement of the pruning criterion, not an independent mechanism measurement. Because the benchmark findings retain independent content, the score is moderate rather than high.
Assumptions & free parameters
free parameters (4)
- final-layer token retention ratio =
10% (main experiments), 70% (same-scale comparisons)
- linear pruning schedule slope =
linear reduction to the retention ratio at the final layer
- score aggregation function =
max pooling
- bias exclusion flag =
True
assumptions (4)
- standard math Mamba's state update is linear in the state and input, so the output can be unrolled as a sum over input contributions.
- ad hoc to paper When removing x_t, the transition matrices A_k, B_k, C_k can be treated as unchanged.
- domain assumption Upper-layer tokens are more redundant and can be pruned more aggressively without performance loss.
- domain assumption The importance score relative to the final token is a valid token importance measure for all layers.
Cite this review
Pith. "Pith review of Sparsified State-Space Models are Efficient Highway Networks." pith.science (2026). https://pith.science/paper/7EXD7B7S
@misc{pith2026250520698,
author = {Pith},
title = {Pith review of: Sparsified State-Space Models are Efficient Highway Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/7EXD7B7S}},
note = {Machine review of arXiv:2505.20698}
}
read the original abstract
State-space models (SSMs) offer a promising architecture for sequence modeling, providing an alternative to Transformers by replacing expensive self-attention with linear recurrences. In this paper, we propose a simple yet effective trick to enhance SSMs within given computational budgets by sparsifying them. Our intuition is that tokens in SSMs are highly redundant due to gradual recurrent updates, and dense recurrence operations block the delivery of past information. In particular, we observe that upper layers of SSMs tend to be more redundant as they encode global information, while lower layers encode local information. Motivated by this, we introduce Simba, a hierarchical sparsification method for SSMs based on token pruning. Simba sparsifies upper layers more than lower layers, encouraging the upper layers to behave like highways. To achieve this, we propose a novel token pruning criterion for SSMs, measuring the global impact of tokens on the final output by accumulating local recurrences. We demonstrate that Simba outperforms the baseline model, Mamba, with the same FLOPS in various natural language tasks. Moreover, we illustrate the effect of highways, showing that Simba not only enhances efficiency but also improves the information flow across long sequences. Code is available at https://github.com/woominsong/Simba.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[4]
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555,
-
[5]
We measure the normalized token influence scores(t)/||yT||2 across all documents from the PG-19 test set, truncated at 1000 tokens. For all samples, we gather the influence score into five bins according to the position of the tokens. We report the average influence scores for each bin. A.3 Fine-tuning with Simba We provide the details for our fine-tuning...
work page 2023
-
[7]
The state of sparsity in deep neural networks.arXiv preprint arXiv:1902.09574,
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks.arXiv preprint arXiv:1902.09574,
arXiv 1902
-
[11]
The minipile challenge for data-efficient language models.arXiv preprint arXiv:2304.08442,
Jean Kaddour. The minipile challenge for data-efficient language models.arXiv preprint arXiv:2304.08442,
-
[12]
12 Published in Transactions on Machine Learning Research (03/2025) Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding.arXiv preprint arXiv:2403.06977, 2024a. Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi- dimensi...
arXiv 2025
-
[13]
Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations.arXiv preprint arXiv:2202.07800,
-
[15]
Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention.arXiv preprint arXiv:2209.13802,
Show all 31 references
-
[16]
The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031,
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context.arXiv preprint arXiv:1606.06031,
-
[17]
Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048,
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048,
-
[18]
Mechanistic design and scaling of hybrid architectures
Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures. arXiv preprint arXiv:2403.17844,
-
[19]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox
URL https://arxiv.org/abs/ 1911.05507. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer Assisted Intervention,
1911 arXiv
-
[21]
Highway networks
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387,
-
[23]
Ali Vardasbi, Telmo Pessoa Pires, Robin M Schmidt, and Stephan Peitz
URLhttps://github.com/alxndrTL/mamba.py. Ali Vardasbi, Telmo Pessoa Pires, Robin M Schmidt, and Stephan Peitz. State spaces aren’t enough: Machine translation needs attention.arXiv preprint arXiv:2304.12776,
-
[24]
Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830,
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830,
1905 arXiv
-
[25]
Vision mamba: Efficient visual representation learning with bidirectional state space model.arXiv preprint arXiv:2401.09417,
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model.arXiv preprint arXiv:2401.09417,
-
[26]
To prune, or not to prune: exploring the efficacy of pruning for model compression
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878,
-
[27]
The six benchmarks were chosen according to the experiment setup of (Gu & Dao, 2023)
14 Published in Transactions on Machine Learning Research (03/2025) A Detailed setups A.1 NLP benchmarks Here, we provide the details for downstream experiments in Section 4.1. The six benchmarks were chosen according to the experiment setup of (Gu & Dao, 2023). The metrics ar...
2025
-
[29]
We use AdamW optimizer with a learning rate of 5e-5
with similar data distribution. We use AdamW optimizer with a learning rate of 5e-5. We schedule the learning rate with a linear warmup for 10% of the total training steps and cosine learning rate decay for the remaining steps. We randomly select the pruning ratio between 0% a...
2025
-
[30]
Bold denotes the best results. Model Scale FLOPs Within Context Extrapolation (x1e12) 0.5k 1k 1.5k 2k 2.5k 3k 3.5k 4k Simba (ours) 130m 0.21 3.060 3.006 2.986 2.974 2.961 2.956 2.948 2.942 Pythia 160m 0.60 3.166 3.134 3.128 3.120 3.195 6.333 7.883 7.986 Mamba 130m 0.48 2.943 2...
-
[31]
In this section, we conduct a further analysis on this phenomenon
Model Scale 130m 370m 790m 1.4b 2.8b State Size 24k 32k 48k 64k 80k Mamba 42.56 50.83 55.78 58.82 63.21 Simba 42.75 51.01 55.80 58.21 62.86 Difference +0.19 +0.18 +0.02 -0.60 -0.35 While token pruning is often associated with a decrease in performance compared to dense models,...
2025
-
[2014]
Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457,
-
[2015]
Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,
-
[2017]
Repeat after me: Transformers are better than state space models at copying.arXiv preprint arXiv:2402.01032,
Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying.arXiv preprint arXiv:2402.01032,
-
[2018]
Griffin: Mixing gated linear recurrences with local attention for efficient language models.arXiv preprint arXiv:2402.19427,
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models.arXiv preprint a...
-
[2019]
The Pile: An 800GB dataset of diverse text for language modeling
11 Published in Transactions on Machine Learning Research (03/2025) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling....
2025 arXiv
-
[2020]
Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,
-
[2021]
Llava-prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388,
13 Published in Transactions on Machine Learning Research (03/2025) Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models.arXiv preprint arXiv:2403.15388,
2025
-
[2022]
Jamba: A hybrid transformer-mamba language model
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887,
-
[2023]
The hidden attention of mamba models.arXiv preprint arXiv:2403.01590,
Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models.arXiv preprint arXiv:2403.01590,
-
[2024]
Constitutional ai: Harmlessness from ai feedback
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.