Pith. sign in

REVIEW 2 major objections 6 minor 16 references

An Exact Instrument for State Usage in Selective State-Space Models, and the Input-Driven Migration It Reveals

T0 review · 2 major / 6 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Selective state-space models reallocate which modes carry the signal with every input, and that migration is driven by the write map, not the timestep.

desk verdict Exact Gram instrument for diagonal selective SSMs, multi-model Bt-driven mode migration, and honest two-pass headroom—solid measurement paper, not a free pruning win. read the letter →

arxiv 2607.11796 v1 pith:AUDRQ24W submitted 2026-07-13 cs.LG

Raktim Bhattacharya This is my paper
classification cs.LG
keywords selectivestate-spacemodelsMambamodepruningGramtensorinput-drivenmigrationwritemapstateusage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Selective state-space models such as Mamba keep a bank of fixed first-order modes and let a learned selection mechanism decide how strongly each mode is written and read. This paper builds an exact instrument that measures, for any trained layer and any input window, which modes actually carry the output and what error you pay if you drop any subset of them. The instrument shows that the important set of modes is not fixed: it migrates from context to context, so a static keep-set incurs roughly twice the error of a per-input choice at the most affected layers. Freezing each selective signal in turn pins the migration on the input-dependent write map, while the timestep that is usually treated as “selectivity” contributes almost nothing. A two-pass scheduler that simply keeps the modes the instrument measures on each window recovers that loss and, at half the state budget, matches the unpruned model on held-out perplexity from 130M parameters up to a deployed 7B model. The result is a concrete measurement of headroom rather than a deployed saving, because the scheduler still needs a first pass over the tokens it scores.

What carries the argument

The per-(layer, channel, window) Gram tensor built from the exact per-mode output contributions. Because the state matrix is diagonal, the squared output error of any kept set of modes is a quadratic form on that Gram matrix, giving a closed-form, offline error for every pruning budget.

What would settle it

If, on the same models and windows, freezing the write map Bt left the migration gap essentially unchanged while freezing the timestep collapsed it, or if a static energy ranking already matched the two-pass scheduled perplexity at half budget, the central claims on mechanism and headroom would fail.

Watch

Extended reading notes

Core claim

Because the state matrix is diagonal, each channel’s output is an exact sum of per-mode contributions. Accumulating those contributions into a small Gram tensor per layer, channel and window yields the exact squared output error of dropping any subset of modes, offline and at any budget. Using this instrument across the Mamba-1 family, Falcon-Mamba 7B and Mamba-2, the paper shows that trained models re-allocate their state with the input: at the most affected layers a per-input oracle roughly halves the error of any fixed mode set. Frozen-signal counterfactuals attribute the migration primarily to the input-dependent write map Bt; the timestep carries almost none of it. Input-scheduled pruni

Load-bearing premise

That a two-pass oracle which reads the mode mask from the very tokens being scored counts as meaningful realizable headroom, even though every cheaper out-of-window or prefix estimator recovers only a small fraction of the gain.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces an exact instrument for measuring how selective SSMs (Mamba-1/2, Falcon-Mamba) use their diagonal state modes. Because A is diagonal, each channel’s output decomposes into per-mode contributions; a per-(layer, channel, window) Gram tensor then yields the exact squared output error of dropping any mode subset offline (Eq. 4). Validated to ~2e-7 against the reference kernel and to median ~1e-6 against deployed pruning over 4,464 configurations, the instrument is applied across the Mamba-1 family (130M–2.8B), Falcon-Mamba 7B, and Mamba-2. The authors report input-driven mode migration (ρ≈0.44–0.57 at the worst layers), attribute it primarily to the write map Bt via freeze counterfactuals (with energy-matched and Bt-only-live controls), and show that a two-pass, same-window scheduled pruning rule outperforms static, modal-HSV, and LAST rankings and, at half state budget, matches or slightly beats the unpruned model on held-out perplexity—explicitly framed as realizable headroom, not a deployed saving.

Significance. If the results hold, the paper supplies a rare exact, offline measurement tool for selective-SSM state usage and a clear mechanistic account of input-dependent reallocation (Bt-driven, not Δ-driven). That combination is useful both for interpretability and for any future compression work that currently relies on static or activity-based scores. Strengths that raise confidence include: (i) machine-checked numerical gates against the reference implementation and against deployed pruning; (ii) multi-scale, multi-architecture coverage including a deployed 7B model; (iii) Proposition 1 linking the diagonal gap to energy-weighted churn; (iv) explicit same-window vs out-of-window controls and recovery fractions (2–26%); and (v) public checkpoints, fixed seeds, and released code. The compression claim is carefully bounded as oracle headroom rather than a free one-pass saving, which is the right scientific stance.

major comments (2)
  1. §4.3 and Table 2: the end-to-end two-pass protocol needs a precise statement. When masks for all layers are read from a first (unpruned) pass and then applied jointly in a second pass, later layers see shifted inputs, so the first-pass energies are not exactly the energies of the pruned trajectory. Please state whether masks are (a) computed once from the unpruned forward and applied jointly, (b) recomputed layer-by-layer along the pruned trajectory, or (c) something else, and report a short sensitivity check. This is load-bearing for the “matches unpruned at half budget” numbers, which are otherwise the paper’s strongest quantitative claim.
  2. Abstract, Table 2, and §4.3: the striking sub-unpruned margin is same-window only. Table 3 and the half-window control show that out-of-window selection does not beat the unpruned model and recovers only ~18–26% of the oracle gain. The abstract already says “we claim no deployed compute or memory saving,” but the main table caption and the sentence “at half the state budget it matches the unpruned model” still invite over-reading. Please make the oracle / same-window restriction equally prominent in the Table 2 caption and in the first sentence of the Consequence subsection so the central compression claim cannot be mis-cited as a one-pass result.
minor comments (6)
  1. §3.4, Proposition 1: the proof is for the diagonal (energy-only) gap; the text correctly notes full-Gram ρ can slightly exceed 1 (layers 21–22 of 130M). A one-sentence pointer in the proposition statement that the inequality is diagonal-only would prevent misreading.
  2. Figure 1: the migrating-band shading (0.4–0.75) is helpful, but the text also cites 0.6–0.75 for small Mamba-1 models. Align the shaded interval with the prose or note that the shade is a compromise across scales.
  3. §4.1: cross-domain vs within-domain churn ratios (1.15–1.32) are modest and below the pre-registered 1.5× pilot. The paper already treats churn as corroborating; a brief remark that the migration case rests on ρ and the freeze experiments (not on churn alone) would help readers who stop at the churn numbers.
  4. Table 4 / GHOST comparison: the note that the released pipeline can silently no-op under fused kernels is valuable for reproducibility; consider elevating a one-line warning into the Reproducibility paragraph so others do not re-hit the same trap.
  5. Notation: ¯A vs A, and the dual role of Δ (pole and write gain), are introduced cleanly in §3.1 but reappear densely in §4.2. A short reminder that Δ is mode-uniform within a channel would make the “Δ carries almost none” claim easier to parse on a first read.
  6. Limitations already flags untested long contexts; given that migration concentrates in slow modes (Figure 3), a single longer-window (e.g., 4k) spot-check on one small model would strengthen the claim that the phenomenon is not an artifact of L=1024, even if left as a brief appendix note.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: exact Gram identity is a diagonal-A consequence validated against the reference, and migration/mechanism/headroom results are empirical measurements on public checkpoints with oracle columns explicitly labeled.

full rationale

The load-bearing derivation is the per-mode decomposition (Eqs. 3–4): because A_h is diagonal, y_h,t = sum_i m_h,t,i + f_h,t exactly, so the window Gram G_h returns the exact squared output error of any dropped set offline. That identity is definitional of the instrument, not a fitted prediction of an independent quantity; it is cross-checked against the reference implementation (rel. err. ≤2.3e-7) and against deployed pruned-layer error (median rel. dev. ~1e-6 over 4464 configs). Migration gap ρ, churn, participation ratio, and frozen-B_t/C_t/Δ counterfactuals are measurements on released Mamba-1/Falcon-Mamba/Mamba-2 checkpoints, not tautologies forced by the instrument’s definition. Proposition 1 is a short set-energy identity showing the diagonal static–oracle gap equals net energy of churning modes; it does not smuggle the empirical finding. Scheduled columns are repeatedly labeled two-pass same-window oracles that quantify headroom, with out-of-window recovery fractions (2–26%) and half-window controls disclosed; no claim of free one-pass savings is made. References contain no author-overlapping self-citations, uniqueness theorems, or ansatz imports. The paper is self-contained against external benchmarks and public code/checkpoints. Score 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claims rest on the architectural fact that Mamba-1/Falcon state matrices are diagonal (standard domain fact), on the algebraic identity that squared pruning error equals the dropped-mode Gram quadratic form, and on empirical choices of windows, domains, budgets, and freeze-at-corpus-mean counterfactuals. No new physical entities are postulated. Free parameters are experimental design knobs (r, L, domain set), not fitted constants that define the migration result.

free parameters (3)
  • mode budget r (default half-state: r=8 of N=16; Mamba-2 r=64 of 128)
    Chosen operating point for migration gap and perplexity tables; results are also shown at other r but headline claims use half budget.
  • window length L=1024 and domain set (English prose, code, technical text)
    Defines the operating points over which Gram tensors and ρ are accumulated; migration could in principle depend on domain choice and length.
  • corpus-mean freeze values for Bt, Ct, Δ
    Counterfactual mechanism test freezes each selective signal at its empirical corpus mean; the mean is data-dependent though not optimized to produce the result.
assumptions (4)
  • domain assumption State matrix A_h is diagonal and fixed after training, so modes do not interact and channel output is an exact sum of per-mode contributions (Eqs. 1–3).
    Standard Mamba-1 architecture fact; instrument exactness is defined only for this case (Mamba-2 treated separately).
  • standard math Squared output error of dropping a mode set equals the corresponding Gram quadratic form over the window (Eq. 4).
    Algebraic identity from the linear sum of mode outputs; validated numerically against reference and deployed pruning.
  • ad hoc to paper Freezing a selective signal at its corpus mean isolates that signal’s contribution to migration without destroying the layer’s overall energy scale (energy-matched control).
    Mechanism attribution in §4.2 depends on this counterfactual design; authors add energy-matched and Bt-only-live controls to support it.
  • domain assumption Calibration and evaluation windows from the same domain streams, even if document-overlapping, yield conservative (not inflated) static-vs-oracle gaps.
    Stated in Setup §4; residual leakage is argued to help the static baseline.
invented entities (2)
  • Per-(layer, channel, window) Gram tensor G_h for exact mode-subset pruning error independent evidence
    purpose: Offline closed-form measurement of state usage and pruning cost at any budget
    Measurement construct derived from the diagonal decomposition; not a new physical object. Independent handle is the reconstruction and deployed-pruning cross-check.
  • Migration gap ρ(r) = E_diag(r)/E_static(r) independent evidence
    purpose: Scalar summary of excess error of a fixed mode set versus per-window top-r
    Defined measurement, not a postulated mechanism; value is read from data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Exact Instrument for State Usage in Selective State-Space Models, and the Input-Driven Migration It Reveals." pith.science (2026). https://pith.science/paper/AUDRQ24W

@misc{pith2026260711796,
  author       = {Pith},
  title        = {Pith review of: An Exact Instrument for State Usage in Selective State-Space Models, and the Input-Driven Migration It Reveals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUDRQ24W}},
  note         = {Machine review of arXiv:2607.11796}
}
abstract

Selective state-space models such as Mamba route information through a bank of first-order modes whose input coupling is set by a learned selection mechanism. We give an exact instrument for measuring how a trained model uses these modes. Because the state matrix is diagonal, each channel's output decomposes exactly into per-mode contributions, and a per-(layer, channel, window) Gram tensor yields the exact output error of dropping any subset of modes, offline, at any budget. Validated against the reference implementation to a relative error of $2.3\times10^{-7}$ on the Mamba-1 family where it is exact, the instrument predicts a layer's deployed pruning error to a median relative deviation of $5\times10^{-7}$ over $4{,}464$ configurations, its floor set by the reconstruction. Applying the instrument across the Mamba-1 family (130M--2.8B), the deployed 7B Falcon-Mamba, and Mamba-2, we find that trained models re-allocate their state space with the input: which modes carry the signal migrates across contexts, and at the most affected layers a per-input oracle roughly halves the output error of a fixed mode set. Frozen-signal counterfactuals attribute the migration primarily to the input-dependent write map $B_t$; the timestep usually identified with selectivity carries almost none of it. Input-scheduled mode pruning on this measurement outperforms static, Hankel-based, and layer-adaptive rankings at every scale from 130M to the deployed 7B Falcon-Mamba, and at half the state budget it matches the unpruned model. Because the scheduler reads each window's mode usage from a first pass, this demonstrates realizable headroom; we claim no deployed compute or memory saving.

Figures

Figures reproduced from arXiv: 2607.11796 by the authors.

Figure 1
Figure 1. Migration gap ρ(8) = Eoracle/Estatic (left) and effective order (right) versus relative depth, over six models (Mamba-1 130M–2.8B and Falcon-Mamba-7B). Coloured lines are individual models (shared legend); the bold black line is the across-model median with an interquartile ribbon; the shaded region is the migrating band (relative depth 0.4–0.75). Left: ρ < 1 (dashed) means a per-input oracle reconstructs the layer … view at source ↗
Figure 2
Figure 2. Frozen-signal mechanism at Mamba-1 130M: migration gap [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Timescale structure at Mamba-1 130M (blue) and 790M (green): median mode energy (left) and top [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Held-out perplexity versus mode budget r for Mamba-1 130M and 790M. Input-scheduled selection (orange, a two-pass oracle) is below static (blue) at every budget, and at or below the unpruned baseline (dashed) for r ≥ 8; the gap between static and scheduled widens as r …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 linked inside Pith

  1. [1]

    PerfMamba: Per- formance analysis and pruning of selective state space models

    Abdullah Al Asif, Mobina Kashaniyan, Sixing Yu, Juan Pablo Mu ˜noz, and Ali Jannesari. PerfMamba: Per- formance analysis and pruning of selective state space models. InBenchmarking, Measuring, and Optimizing (Bench), volume 16471 ofLecture Notes in Computer Science. Springer, 2026. 10

    2026

  2. [2]

    The hidden attention of Mamba models

    Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of Mamba models. InAnnual Meeting of the Association for Computational Linguistics (ACL), pages 1516–1534, 2025

    2025

  3. [3]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InInternational Conference on Machine Learning (ICML), 2024

    2024

  4. [4]

    Model compression method for S4 with diagonal state space layers using balanced truncation.arXiv preprint arXiv:2402.15993, 2024

    Haruka Ezoe and Kazuhiro Sato. Model compression method for S4 with diagonal state space layers using balanced truncation.arXiv preprint arXiv:2402.15993, 2024

    arXiv 2024

  5. [5]

    Model order reduction of deep structured state-space models: A system-theoretic approach.arXiv preprint arXiv:2403.14833, 2024

    Marco Forgione, Manas Mejari, and Dario Piga. Model order reduction of deep structured state-space models: A system-theoretic approach.arXiv preprint arXiv:2403.14833, 2024

    arXiv 2024

  6. [6]

    All optimal Hankel-norm approximations of linear multivariable systems and theirl ∞-error bounds.International Journal of Control, 39(6):1115–1193, 1984

    Keith Glover. All optimal Hankel-norm approximations of linear multivariable systems and theirl ∞-error bounds.International Journal of Control, 39(6):1115–1193, 1984

    1984

  7. [7]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    arXiv 2023

  8. [8]

    Layer-adaptive state pruning for deep state space models

    Minseon Gwak, Seongrok Moon, Joohwan Ko, and PooGyeon Park. Layer-adaptive state pruning for deep state space models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

Show all 16 references
  1. [9]

    Logan IV , Chandrasekhara Ganesh Jagadeesan, Sameer Singh, Joel R

    Tamanna Hossain, Robert L. Logan IV , Chandrasekhara Ganesh Jagadeesan, Sameer Singh, Joel R. Tetreault, and Alejandro Jaimes. Characterizing Mamba’s selective memory using auto-encoders. InIJCNLP-AACL, 2025

    2025

  2. [10]

    GHOST: Unmasking phantom states in Mamba2 via grouped hidden-state output-aware selection and truncation.arXiv preprint arXiv:2602.11408, 2026

    Michael Menezes and Anastasios Kyrillidis. GHOST: Unmasking phantom states in Mamba2 via grouped hidden-state output-aware selection and truncation.arXiv preprint arXiv:2602.11408, 2026

    2026

  3. [11]

    Bruce C. Moore. Principal component analysis in linear systems: Controllability, observability, and model reduction.IEEE Transactions on Automatic Control, 26(1):17–32, 1981

    1981

  4. [12]

    Balanced truncation of linear time-varying systems.IEEE Transactions on Automatic Control, 49(2):217–229, 2004

    Henrik Sandberg and Anders Rantzer. Balanced truncation of linear time-varying systems.IEEE Transactions on Automatic Control, 49(2):217–229, 2004

    2004

  5. [13]

    Hankel singular value regularization for highly compressible state space models

    Paul Schwerdtner, Jules Berman, and Benjamin Peherstorfer. Hankel singular value regularization for highly compressible state space models. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  6. [14]

    Efficient unstructured pruning of Mamba state-space mod- els for resource-constrained environments

    Ibne Farabi Shihab, Sanjeda Akter, and Anuj Sharma. Efficient unstructured pruning of Mamba state-space mod- els for resource-constrained environments. InEmpirical Methods in Natural Language Processing (EMNLP), 2025

    2025

  7. [15]

    Shokoohi, L

    S. Shokoohi, L. M. Silverman, and P. Van Dooren. Linear time-variable systems: Balancing and model reduction. IEEE Transactions on Automatic Control, 28(8):810–822, 1983

    1983

  8. [16]

    Falcon Mamba: The first competitive attention-free 7B language model.arXiv preprint arXiv:2410.05355, 2024

    Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon Mamba: The first competitive attention-free 7B language model.arXiv preprint arXiv:2410.05355, 2024. 11

    2024 arXiv

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.