{"id":"17302bd0-e64f-4c0e-81d1-366f7d4d378d","arxiv_id":"2505.20698","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Simba prunes tokens hierarchically in pre-trained SSMs, creating sparse upper layers that act as highways, improving the accuracy-FLOPs trade-off and long-context perplexity.","lead":"This paper introduces Simba, a training-free method that prunes tokens in pre-trained Mamba state-space models, removing more tokens in upper layers to create faster 'highway' paths. On standard language benchmarks, Simba achieves better accuracy than Mamba at similar FLOPs, and it handles long contexts better.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (4) assumes Mamba's input-dependent transition matrices are unchanged when a token is removed, so the 'global influence' score is an approximation; the paper's highway evidence (Fig. 5) uses this same score, making the conceptual claim vulnerable.","rationale":"The paper's central empirical claim—Simba achieves better FLOPs-accuracy curves than dense Mamba—is supported by direct measurements and ablations, and I do not find a fatal flaw in that part. The most load-bearing weakness is the derivation of the token influence score in Eq. (4). The unrolled SSM expression treats \\bar A_k, \\bar B_t, and C_T as fixed coefficients, but in Mamba they are input-dependent. Removing a token does not simply delete one term; it also changes the recurrence path for all preceding tokens, since the transition at the deleted position disappears from the product. The paper does not acknowledge this simplification, yet it uses the score both to select tokens and to visualize 'information flow' (Figure 5). If the score is not the true counterfactual influence, the highway interpretation is unsupported and the novelty claim of a global-influence criterion is overstated. The proposed test—comparing Eq. (4) rankings against exact re-runs—would settle whether the approximation is benign in practice. This matches the reader's weakest_assumption, so I agree with the Conditional verdict and recommend no change.","tokens_in":17296,"tokens_out":16620,"duration_ms":167366,"concrete_test":"On short sequences (e.g., T=64) from PG-19, compute the exact counterfactual: for each token x_t, re-run the pruned sequence through the layer, recomputing all input-dependent \\bar A_k, \\bar B_t, C_T, and measure the actual change in y_T. Compare the ranking of this true influence against Eq. (4)'s score s(t) using Spearman rank correlation. If the correlation is high (>0.9), the approximation is benign for pruning; if it is low, the paper's claim that s(t) measures global impact is false, and Figure 5's highway evidence should be reinterpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3.2, Eq. (4) derives the influence of token x_t on the final output by subtracting the single term C_T (∏_{k=t+1}^T \\bar A_k) \\bar B_t x_t from the unrolled sum. This cancellation assumes the transition matrices \\bar A_k, \\bar B_t, and C_T are unaffected by the removal. In Mamba these matrices are functions of the input tokens. Concretely, deleting x_t removes the state transition at position t, so for any earlier token x_r (r<t) the product of \\bar A's loses the factor \\bar A_t; the contribution of x_r changes, not just the contribution of x_t. The exact counterfactual difference is thus C_T(∏_{k=t+1}^T \\bar A_k)\\bar B_t x_t minus a sum over r<t of C_T[∏_{k=r+1}^T \\bar A_k − ∏_{k=r+1, k≠t}^T \\bar A_k]\\bar B_r x_r. This second term is omitted. The paper uses this approximate score both to prune tokens and as the sole evidence for the highway interpretation (Figure 5), so the title claim that sparsified SSMs 'are efficient highway networks' rests on an unvalidated approximation. The empirical FLOPs-accuracy results may survive because the score can be a useful heuristic, but the theoretical contribution ('measuring the global impact of tokens') and the mechanism evidence are not supported as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Simba, a training-free hierarchical token-pruning method for pre-trained state-space models (SSMs), applied here to Mamba. Simba removes tokens at every layer following a linear schedule that preserves more tokens in lower layers and fewer in upper layers, so the network becomes trapezoidal. The pruning criterion is a token influence score derived by unrolling the SSM recurrence and computing the contribution of each token to the final output. The authors report that Simba improves the FLOPs-accuracy frontier on six NLP benchmarks, achieves better PG-19 perplexity than dense Mamba at comparable FLOPs, and shows more stable long-context perplexity beyond the 2k training context. They interpret the sparse upper layers as highways that improve information flow from early tokens, and support this with an influence-based visualization across layers. The manuscript also includes ablations against uniform and random pruning, a fine-tuning experiment, and an inference-time study.","tokens_in":17614,"tokens_out":4783,"duration_ms":47364,"significance":"If the empirical claims hold, Simba is a practical, plug-and-play efficiency technique for Mamba-class models: it requires no fine-tuning, is evaluated on standard benchmarks with publicly released code, and its main comparisons are against externally evaluated dense baselines. The paper also contains honest limitations and a useful inference-time analysis. However, the conceptual contribution—that sparsified SSMs are 'efficient highway networks'—rests on an approximate influence score whose exactness is not established, and the information-flow evidence is partly circular because it uses the same score that defines the pruning. The empirical FLOPs-accuracy results are credible and would likely survive a corrected derivation, but the mechanistic interpretation needs to be revised or independently validated.","major_comments":[{"comment":"The derivation of the token influence score treats the transition matrices as fixed when a token is removed. In Mamba, the matrices \\bar A_k, \\bar B_k, and C_k are input-dependent, so deleting x_t changes more than the single term in Eq. (4). The exact counterfactual difference also contains contributions from earlier tokens whose products of \\bar A's are altered by the absence of \\bar A_t. The score is therefore an approximation, not the exact 'global impact' claimed in the abstract and Section 3.2. The paper neither states this simplification nor acknowledges it in the Limitations section. Since the pruning criterion and the highway evidence in Section 4.3 both rely on this score, the theoretical framing should be corrected, and the approximation should be validated (e.g., by comparing the score against oracle removal on a subset of data).","section":"Section 3.2, Eq. (4)"},{"comment":"The information-flow visualization uses the normalized influence score s(t)/||y_T||_2, which is the same approximate score used to define the pruning decisions. This makes the highway interpretation partly circular: the flattening observed at upper layers may be a mechanical consequence of pruning away low-influence tokens, rather than an independent measurement of how information is routed. To support the claim that upper layers act as highways, the manuscript needs an analysis that does not depend on Eq. (4), such as intervention-based attribution or a gradient-based importance measure.","section":"Section 4.3, Figure 5"}],"minor_comments":[{"comment":"The statement that Simba 'consistently demonstrates decreasing perplexity even with extended contexts' is slightly overstated: Simba-2.8b shows perplexity 2.279 at 3.5k but 2.284 at 4k, so the trend is not strictly monotonic. Please qualify the claim as generally decreasing or note the small uptick at the longest tested length.","section":"Table 5 and Section 4.2"},{"comment":"The citation 'Frankle & Carbin, 20189' appears in the text and reference list; the year should be 2019.","section":"References"},{"comment":"The caption text says models use 'the same number of FLOPs,' but the actual FLOPs differ somewhat (e.g., Simba-2.8b at 6.07e12 versus Mamba-1.4b at 5.60e12). The comparison is still meaningful, but the wording should be 'comparable FLOPs' or the exact numbers should be cited.","section":"Figures 3 and 4 and Tables 4 and 5"}],"recommendation":"major_revision","confidential_remarks":"My main concern is the gap between the approximate influence score and the exact counterfactual claim in Eq. (4). The empirical results are credible and likely unaffected, but the paper's conceptual contribution—the highway-network interpretation—is not supported as stated. I recommend major revision rather than rejection because the central empirical claims are externally grounded and the theoretical issue can be addressed by reframing the score as a heuristic and by adding an independent validation of the information-flow story."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a genuinely useful empirical paper, and its central claim—training-free, per-input token pruning of pre-trained SSMs with more pruning in upper layers beats dense Mamba at matched FLOPs and improves long-context perplexity—holds up. The pruning score in Eq. (4) is the right SSM analog of Transformer token importance: it sums the contribution of each token to the final output through the recurrent product. The hierarchy observation (upper layers have higher token redundancy) is supported by cosine-similarity plots, and the hierarchical schedule is a sensible use of it. The evaluation is thorough: six NLP benchmarks, several model scales, Pythia and Mamba baselines, FLOPs accounting, wall-clock speedups in the appendix, and code. That is solid, reproducible work.\n\nWhere the paper is softer: Eq. (4) claims to measure the global influence of token x_t by deleting its term from the unrolled sum. That is only exact if removing a token leaves \\bar A_k, \\bar B_k, C_k untouched. In Mamba they are input-dependent, so deleting x_t also changes the transition at position t and therefore the contributions of every earlier token. The stress-test note is right: the exact counterfactual has an extra sum over earlier tokens. The paper never says this is an approximation. So the 'global impact' interpretation is overclaimed, and the derivation in Section 3.2 is a heuristic in disguise. This does not sink the empirical method—the FLOPs-accuracy wins are independent of the derivation—but it does mean the theory section needs a rewrite.\n\nThe highway evidence is also partly circular. Figure 5 shows that the normalized influence score flattens in upper layers for Simba. But that score is exactly what the pruning uses, so the flattening is partially built in. The more convincing evidence for the highway story is the perplexity improvement beyond the 2k training context, which does not rely on Eq. (4). I'd push for an additional analysis using an independent measure, or at least a caveat that the current figure is not a neutral measurement.\n\nThe paper is honest about its limitations (distribution shift, fine-tuning as mitigation). The authors are thinking clearly and the empirical craft is good. My verdict: the empirical contribution is real and citable; the theoretical framing is thinner than stated and needs correction. This paper deserves a serious referee; with a revision I'd ask for the simplification to be stated and for the highway claim to be tested with at least one independent metric.","headline":"Solid empirical token pruning for SSMs that beats matched-FLOPs Mamba and improves long-context perplexity, but the influence-score derivation silently approximates an exact counterfactual and the highway evidence is partly self-referential.","tokens_in":18163,"tokens_out":2895,"would_cite":true,"duration_ms":29399,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hierarchical token pruning of pre-trained state-space models—heavier in upper layers—creates highway-like shortcuts that beat dense Mamba at matched FLOPs and improve long-context perplexity.","keywords":["state-space models","Mamba","token pruning","hierarchical sparsification","highway networks","long-context modeling","FLOPs-accuracy trade-off"],"falsifier":"Run the true counterfactual: remove a token from a Mamba sequence, recompute the forward pass, and measure the actual change in the final output logits; then compare the ranking of tokens by this true effect with the ranking by Eq. (4). If the two rankings diverge strongly on realistic inputs, or if pruning by the true counterfactual fails to reproduce Simba's gains, the central mechanism is not what the paper claims.","tokens_in":17078,"feed_emoji":"🛣️","tokens_out":8229,"duration_ms":69372,"temperature":0.7,"pith_summary":"Simba claims that a pre-trained state-space model like Mamba can be improved by pruning tokens during inference, with more aggressive pruning in upper layers, so that the remaining tokens form a fast 'highway' through the network. The central result is that this training-free sparsification produces models that consistently beat dense Mamba—and often Pythia—at the same FLOPs budget on six natural-language benchmarks, and that the pruned models keep reducing perplexity beyond the 2,000-token context the original model was trained on. If this is right, it means token redundancy in SSMs is not just a cost to be saved but an obstacle to information flow, and that removing redundant recurrences can improve both efficiency and long-context behavior. The argument turns on a token influence score that measures each token's contribution to the final output by unrolling the linear recurrence.","feed_headline":"Training-free token pruning beats dense Mamba at same FLOPs","feed_subtitle":"Hierarchical upper-layer pruning creates highway shortcuts that lift accuracy and long-context perplexity.","key_machinery":"The central object is the token influence score in Eq. (4), which unrolls the SSM recurrence so that the change in final output caused by deleting token $x_t$ is expressed as $C_T(\\prod_{k=t+1}^{T}\\bar{A}_k)\\bar{B}_t x_t$, aggregated by max pooling over channels. This score turns a global counterfactual question—which tokens matter for the final output—into a per-token quantity computed from local recurrence steps, and it drives the pruning decision at every layer. The second piece is the linear pruning schedule: the fraction of kept tokens decreases steadily with depth, so the network becomes trapezoidal and upper layers act as highways that let early information pass through without dense processing.","core_discovery":"The paper's discovery is that dense recurrence is itself a bottleneck: upper layers of Mamba encode global, redundant information, and forcing every token through every recurrence step attenuates early-token influence on the final output. Simba removes tokens layer by layer using a score $\\Delta y_T(t) = C_T(\\prod_{k=t+1}^{T}\\bar{A}_k)\\bar{B}_t x_t$, which estimates the global effect of token $x_t$ on the final output by accumulating the local recurrence path. Pruning is applied hierarchically with a linear schedule so that upper layers keep only a small fraction of tokens, giving the network a trapezoidal shape. The authors report that this training-free procedure yields better accuracy at matched FLOPs on six NLP benchmarks, better PG-19 perplexity, and—unlike dense Mamba—decreasing perplexity past the 2k context limit, which they attribute to highway-like shortcuts in the upper layers.","pith_inferences":["Editorial inference: the score in Eq. (4) is a leave-one-out approximation; an exact variant that recomputes the input-dependent $\\bar{A}_k$ and $\\bar{B}_k$ after deletion might either strengthen or weaken the pruning gains.","Editorial inference: the highway interpretation suggests a direct architectural test—insert explicit residual or skip connections into Mamba's upper layers and compare against Simba's pruning-only highways; if both improve long-context behavior in the same way, the mechanism is about information flow rather than sparsity per se.","Editorial inference: because pruning decisions are made per input sequence and require no external data, the same score could be applied during autoregressive decoding or chunked streaming generation; the paper sketches chunked prefill but does not evaluate streaming generation."],"forward_implications":["At a fixed FLOPs budget, Simba-2.8b averages 62.5% accuracy on six NLP benchmarks, outperforming Mamba-1.4b (58.8%) and Pythia-2.8b (59.2%) while using computation comparable to the smaller Mamba model.","On PG-19, Simba's perplexity keeps falling at 4k context while Mamba's rises, meaning the pruned model uses context beyond its training length better than the dense model.","Moderate pruning can improve same-scale performance: at 130m, 370m, and 790m scale, Simba edges out the original Mamba on average downstream accuracy while using fewer FLOPs.","Because no fine-tuning is required, the method applies directly to any pre-trained SSM, and a short fine-tuning step improves results further.","The information-flow analysis shows that upper layers of Simba give more weight to early tokens than dense Mamba does, supporting the highway interpretation."],"supporting_citations":[{"why":"Defines the Mamba architecture that Simba sparsifies and the equal-FLOPs baseline it must beat.","marker":"Gu & Dao, 2023"},{"why":"Provides the Pythia models used as dense baselines at comparable compute budgets.","marker":"Biderman et al., 2023"},{"why":"HOMER supplies the hierarchical token-pruning idea and the long-context interpretation that Simba adapts to SSMs.","marker":"Song et al., 2024"},{"why":"Power-BERT is the attention-based token pruning criterion that Simba's influence score extends to state-space models.","marker":"Goyal et al., 2020"},{"why":"Token merging motivates the linear pruning schedule that gives Simba its trapezoidal shape.","marker":"Bolya et al., 2023"},{"why":"Provides evidence that Mamba's upper layers attend to global context, supporting the hierarchical redundancy premise.","marker":"Ali et al., 2024"},{"why":"Lottery-ticket-style reasoning that sparse large models can beat small dense models justifies pruning pretrained Mamba rather than training a smaller model.","marker":"Frankle & Carbin, 2018"},{"why":"PG-19 is the dataset used to measure perplexity and long-context extrapolation.","marker":"Rae et al., 2019"}],"fun_headline_variants":["Sparse SSMs become highways: Simba prunes for better flow","Token pruning turns Mamba upper layers into highways","Simba: hierarchical pruning beats dense Mamba at same FLOPs","Prune tokens, not accuracy: Simba outdoes Mamba","Highway effect: sparse SSMs improve long-context perplexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The score that decides which tokens to prune assumes that removing a token only deletes that token's own contribution to the final output and leaves all the recurrence matrices unchanged, even though in Mamba those matrices depend on the input tokens.","fun_headline_variants_meta":{"raw":{"variants":["Sparse SSMs become highways: Simba prunes for better flow","Token pruning turns Mamba upper layers into highways","Simba: hierarchical pruning beats dense Mamba at same FLOPs","Prune tokens, not accuracy: Simba outdoes Mamba","Highway effect: sparse SSMs improve long-context perplexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1680,"prompt_tokens":963,"completion_tokens":717,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":642}},"tokens_in":579,"tokens_out":717,"duration_ms":7470,"temperature":1.0,"reasoning_tokens":642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:48:12.629900+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the true counterfactual: remove a token from a Mamba sequence, recompute the forward pass, and measure the actual change in the final output logits; then compare the ranking of tokens by this true effect with the ranking by Eq. (4). If the two rankings diverge strongly on realistic inputs, or if pruning by the true counterfactual fails to reproduce Simba's gains, the central mechanism is not what the paper claims.","supporting_citations":[],"review_version":1}