REVIEW 3 major objections 5 minor 1 cited by
Quantizing Small-Scale State-Space Models for Edge AI
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Quantization-aware training in recurrent mode lifts a small state-space model from 40% post-training-quantization accuracy back to 96%, with about 6x parameter-memory savings.
desk verdict Solid empirical ablation of S4D quantization with a real QAT recovery result, undercut by a nominal 6x memory claim that omits metadata and state overhead. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the recurrent-mode update $x_{n+1} = \bar{A} x_n + \bar{B} u_n$ of a diagonal state-space layer, where the complex state-transition matrix $\bar{A}$ is trained and then quantized after discretization. The argument's load-bearing mechanism is the unit-circle condition: stability requires all eigenvalues (here the diagonal entries) of $\bar{A}$ to land inside the unit circle, and low-precision quantization can push them outside, so errors grow like $\bar{A}^n$; the paper counters this with per-step clipping of $x$ to $[-50,50]$ and with quantization-aware training. Three parameterizations carry the QAT part: training the discrete matrices directly, training the continuous-time parameterization, and freezing $\bar{A}$ while training the rest, with frozen $\bar{A}$ performing best at the low precisions most relevant to hardware.
What would settle it
Measure the actual deployed memory of the 64-head S4D with the W4A6, $\bar{A}$8, $x$8 scheme on an edge microcontroller or a cycle-accurate memory simulator, counting all scales, offsets, state buffers, and alignment operations; if the measured saving is far below 6x, the headline compression claim fails. Alternatively, run the same PTQ-then-QAT protocol on a second sequence task, such as keyword spotting, and check whether the 8-bit boundary for $\bar{A}$ and $x$ still holds.
Extended reading notes
Core claim
The paper's central claim is that quantization-induced failure in small state-space models is localized to two quantities—the discretized state-transition matrix $\bar{A}$ and the recurrent state $x$—and that this failure can be repaired by quantization-aware training rather than by increasing model width. Because $\bar{A}$ is multiplied into $x$ at every time step, quantization errors in $\bar{A}$ are amplified by powers of the time step, so even one eigenvalue landing outside the unit circle makes the state explode; clipping $x$ to $[-50,50]$ keeps the recurrence bounded. Retraining the quantized model for ten epochs in recurrent mode with a straight-through gradient estimator lifts accuracy from roughly 40% after post-training quantization to 95.99% on the 64-head model, with weights and non-state activations at 4 and 6 bits and $\bar{A}$ and $x$ at 8 bits. This configuration reduces the full-precision model's parameter memory by 84.05%, stated as a 6x compression.
Load-bearing premise
The headline 6x memory saving is counted from the nominal bit-widths of weights and activations and assumes those bit counts dominate the deployed footprint; if the extra bookkeeping values that quantized arithmetic requires (per-head scales, offsets, complex state buffers, and range-alignment overhead) are counted, the true saving could be materially smaller.
Editorial extensions
If this is right
- Post-training quantization alone cannot push the state-transition matrix and state below 8 bits on this task; even wider models fail below 8 bits, so recurrent-mode quantization-aware training is the practical path for small SSMs.
- Quantizing most matrices to 4 bits, non-state activations to 6 bits, and only $\bar{A}$ and $x$ to 8 bits is enough to stay near full-precision accuracy, so memory savings come from the insensitive components rather than from the sensitive ones.
- Freezing $\bar{A}$ during quantization-aware training is the best choice at low bit precision, which means a deployed chip does not need fine-grained updates to the unstable part of the model.
- At 8 bits for $\bar{A}$ and $x$, the heterogeneous scheme attains 95.99% accuracy on the 64-head model while reducing the full-precision parameter memory by 84.05%, i.e., a 6x compression.
Reading between the lines
- The unit-circle diagnosis suggests a targeted training constraint the paper does not explore: forcing the quantized eigenvalues of $\bar{A}$ to lie strictly inside the unit circle could enable sub-8-bit $\bar{A}$, something the paper observes is otherwise unstable.
- Because the failure mode is recurrence-specific, the results likely carry over to other diagonal state-space models run recurrently, but selective state-space models with input-dependent transitions may need a separate analysis.
- The recipe is validated only on sequential MNIST; speech, ECG, or sensor benchmarks would reveal whether the 8-bit boundary and the 6x saving generalize to real edge workloads.
- The paper's memory accounting counts nominal bit-widths, so the true on-device saving depends on overheads such as per-head scales, offsets, and alignment operations that the paper leaves for future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies quantization of small S4D state-space models for edge inference. Using the sequential MNIST benchmark, it first applies post-training quantization (PTQ) in recurrent mode and identifies the state transition matrix A and the recurrent state x as the most sensitive components, requiring at least 8 bits, while B, C, D and non-state activations tolerate 4-6 bits. It then applies quantization-aware training (QAT) with three parameterizations of A (discrete, continuous/original, and frozen) and proposes a heterogeneous scheme (4-bit weights, 6-bit non-state activations, 8-bit A and x) that recovers roughly 96% test accuracy from roughly 40% under PTQ on the 64-head model, with an advertised 6x memory reduction. The paper also provides a mechanistic stability analysis: quantization can move eigenvalues of the discrete transition matrix outside the unit circle, causing errors to grow as powers of the time step, and clipping the state is proposed as a mitigation.
Significance. If the memory figure were accurate, the result would be a useful practical demonstration that recurrent-mode QAT plus a heterogeneous bit-width assignment can make small S4D models edge-viable while retaining near-baseline accuracy. The empirical core is carefully executed: the PTQ ablations use five seeds with error bars, the unit-circle instability explanation is mechanistically plausible, and the benchmark is external, so the reported accuracies are direct measurements rather than quantities that reduce by construction. The main limitations are that the 6x memory-reduction claim rests on an incomplete memory model, the design choices are not protected by a described validation protocol, and the 'without sacrificing performance' wording is stronger than the reported accuracy differences support.
major comments (3)
- [V-A (Table III) and Conclusion] The '6x memory reduction' is computed from the nominal bit widths of weights and activations, but no full memory model is given. For per-head quantization with H=64, each quantized tensor needs at least one scale and zero-point per head; the complex tensors A, B, C require storage for real and imaginary parts even if quantized with a shared scheme; the recurrent state buffers must physically persist during streaming; and the Outlook itself acknowledges 'computational overhead to align the integer ranges'. These terms are omitted from the 'Mem Savings' percentages in Table III. For a 21,514-parameter model, 64 heads times 4 SSM tensors times (scale + zero-point) is already on the order of 1-2 KB against a roughly 13 KB 5-bit payload, so the true deployed footprint reduction may be materially smaller than 6x. Please provide the memory formula used for Table III and, ideally, an actual on-device memory measurement.
- [II and V-A (Fig. 5c, Table III)] The paper does not describe a validation split or a model-selection protocol. The state clipping bound [-50, 50], the gradient clipping bound [-1000, 1000], the 99.999% calibration percentile, and the heterogeneous bit assignment W4A6/A8x8 are all presented as fixed choices, but they appear to be selected after inspecting test-set behavior. If any of these choices were tuned on the sMNIST test set, the reported 95.99 +/- 0.56 accuracy is optimistic. At minimum, report validation accuracy and provide a sensitivity analysis for the state clipping bound and the gradient clipping bound; these are the central free parameters of the proposed method.
- [Abstract and Conclusion; Table III] The claim that the scheme reduces memory by 6x 'without sacrificing performance' is not supported by the reported numbers. The best heterogeneous QAT result for S4D-64 (95.99 +/- 0.56) is 2.8 percentage points below the full-precision baseline (98.79 +/- 0.12), and for S4D-16 the same scheme gives 88.39 +/- 1.40 against a 97.20 +/- 0.25 baseline. The paper reports no significance test; on S4D-64 the error bars do not overlap, so the accuracy difference is likely meaningful. Please either report a statistical comparison of the quantized and full-precision models or temper the claim to 'with a limited accuracy loss'.
minor comments (5)
- [IV, Eq. (2)] Equation (2) uses A both for the continuous-time matrix and inside the discrete denominator: the expression Q(Disc(\bar{A})) = Q(e^{A\Delta}) is notationally unclear and should distinguish the continuous A from the discrete \bar{A} explicitly.
- [II] The text 'qGELUq(x)' appears to be a typographical duplicate; it should read qGELU(x) or a similarly defined quantized GELU once.
- [Conclusion] The Conclusion cites the Loihi 2 implementation [20] when referring to the S4D architecture; the architectural description should cite the original S4D paper [4] instead.
- [Table III] The notation 'A16x16' is ambiguous because the caption says it denotes precisions for A and x, but the table body does not explicitly separate the A bit-width from the x bit-width; please clarify the notation in the table header or caption.
- [Fig. 2b] The legend entries 'Dynamic', 'Static', 'State', and 'Non-state' are difficult to map to the four curves; please label each curve directly or use a clearer legend that distinguishes the quantized tensor (state versus non-state) and the calibration mode.
Circularity Check
No circularity: the paper reports direct empirical measurements and an explicit bit-count arithmetic for memory savings; no prediction reduces to its inputs by construction.
full rationale
The paper does not contain a predictive derivation chain. Its main claims are empirical measurements on the sMNIST benchmark: PTQ sensitivity ablations (Fig. 2, Fig. 3), QAT recovery curves (Fig. 5), and final accuracies in Table III. The 40%-to-96% accuracy recovery is a direct measured comparison between PTQ and QAT runs, not a quantity implied by the paper's own equations. The 6x memory-footprint claim is an explicit arithmetic ratio: the reported savings percentages (76.55%, 84.05%, etc.) equal the reduction obtained by replacing 32-bit elements with the chosen 4-, 6-, or 8-bit elements, as stated in Table III. That is a definitional accounting of nominal bit-widths, not a hidden fit or a self-referential prediction. The paper also discloses in the Outlook that the heterogeneous scheme 'incurs some computational overhead to align the integer ranges,' which is a limitation on the deployed footprint rather than a circular step. There are no load-bearing self-citations: references to S4D, HiPPO, Q-S5, and quantization tooling are external prior work, and the paper's authors do not invoke their own prior results to forbid alternatives or justify the central claim. The skeptical concern about omitted per-head metadata and real/imaginary storage is a correctness or completeness risk about whether the nominal 6x survives on-device, but it does not make any derivation circular. Accordingly, no circularity steps are identified, and the score is 0.
Assumptions & free parameters
free parameters (4)
- State clipping bound =
[-50, +50]
- Calibration percentile =
99.999%
- QAT gradient clipping bound =
[-1000, +1000]
- Heterogeneous bit-width assignment =
4-bit weights, 6-bit activations, 8-bit A and state x
assumptions (4)
- standard math A discrete LTI system is stable if and only if its eigenvalues lie inside the unit circle.
- domain assumption Recurrent-mode inference accurately models how quantization error propagates at the edge.
- domain assumption The straight-through estimator provides usable gradients through the quantization function.
- ad hoc to paper The state-clipping bound of [-50, +50] is a sufficient and general solution to instability.
Cite this review
Pith. "Pith review of Quantizing Small-Scale State-Space Models for Edge AI." pith.science (2026). https://pith.science/paper/HLUGNDXY
@misc{pith2026250612480,
author = {Pith},
title = {Pith review of: Quantizing Small-Scale State-Space Models for Edge AI},
year = {2026},
howpublished = {\url{https://pith.science/paper/HLUGNDXY}},
note = {Machine review of arXiv:2506.12480}
}
read the original abstract
State-space models (SSMs) have recently gained attention in deep learning for their ability to efficiently model long-range dependencies, making them promising candidates for edge-AI applications. In this paper, we analyze the effects of quantization on small-scale SSMs with a focus on reducing memory and computational costs while maintaining task performance. Using the S4D architecture, we first investigate post-training quantization (PTQ) and show that the state matrix A and internal state x are particularly sensitive to quantization. Furthermore, we analyze the impact of different quantization techniques applied to the parameters and activations in the S4D architecture. To address the observed performance drop after Post-training Quantization (PTQ), we apply Quantization-aware Training (QAT), significantly improving performance from 40% (PTQ) to 96% on the sequential MNIST benchmark at 8-bit precision. We further demonstrate the potential of QAT in enabling sub-8-bit precisions and evaluate different parameterization schemes for QAT stability. Additionally, we propose a heterogeneous quantization strategy that assigns different precision levels to model components, reducing the overall memory footprint by a factor of 6x without sacrificing performance. Our results provide actionable insights for deploying quantized SSMs in resource-constrained environments.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
QS4D: Quantization-aware training for efficient hardware deployment of structured state-space sequential models
Quantization-aware training allows S4D sequence models to run at much lower precision, cutting estimated hardware costs by up to two orders of magnitude while keeping accuracy.
Reference graph
Works this paper leans on
-
[1]
Hippo: Recurrent memory with optimal polynomial projections,
A. Gu, T. Dao, S. Ermon, A. Rudra, and C. R ´e, “Hippo: Recurrent memory with optimal polynomial projections,” Advances in neural information processing systems , vol. 33, pp. 1474–1487, 2020
2020
-
[2]
Efficiently modeling long sequences with structured state spaces. arxiv 2021,
A. Gu, K. Goel, and C. R ´e, “Efficiently modeling long sequences with structured state spaces. arxiv 2021,” arXiv preprint arXiv:2111.00396
arXiv 2021
-
[3]
Combining recurrent, convolutional, and continuous-time models with linear state space layers,
A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. R ´e, “Combining recurrent, convolutional, and continuous-time models with linear state space layers,” Advances in neural information processing systems, vol. 34, pp. 572–585, 2021
2021
-
[4]
On the parameterization and initialization of diagonal state space models,
A. Gu, K. Goel, A. Gupta, and C. R ´e, “On the parameterization and initialization of diagonal state space models,” Advances in Neural Information Processing Systems , vol. 35, pp. 35971–35983, 2022
work page 2022
-
[5]
Scalable event-by-event processing of neuromorphic sensory signals with deep state-space models,
M. Sch ¨one, N. M. Sushma, J. Zhuge, C. Mayr, A. Subramoney, and D. Kappel, “Scalable event-by-event processing of neuromorphic sensory signals with deep state-space models,” in 2024 International Conference on Neuromorphic Systems (ICONS) , pp. 124–131, IEEE, 2024
work page 2024
-
[6]
Exploring the capability of mamba in speech applications,
K. Miyazaki, Y . Masuyama, and M. Murata, “Exploring the capability of mamba in speech applications,” arXiv preprint arXiv:2406.16808, 2024
arXiv 2024
-
[7]
Spiking structured state space model for monaural speech enhancement,
Y . Du, X. Liu, and Y . Chua, “Spiking structured state space model for monaural speech enhancement,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 766–770, 2024
work page 2024
-
[8]
Eeg-ssm: Leveraging state-space model for dementia detection,
X.-T. Tran, L. Le, Q. T. Nguyen, T. Do, and C.-T. Lin, “Eeg-ssm: Leveraging state-space model for dementia detection,” arXiv preprint arXiv:2407.17801, 2024
arXiv 2024
Show all 29 references
-
[9]
Harmamba: Effi- cient wearable sensor human activity recognition based on bidirectional selective ssm,
S. Li, T. Zhu, F. Duan, L. Chen, H. Ning, and Y . Wan, “Harmamba: Effi- cient wearable sensor human activity recognition based on bidirectional selective ssm,” arXiv e-prints, pp. arXiv–2403, 2024
2024
-
[10]
Optimis- ing tinyml with quantization and distillation of transformer and mamba models for indoor localisation on edge devices,
T. Suwannaphong, F. Jovan, I. Craddock, and R. McConville, “Optimis- ing tinyml with quantization and distillation of transformer and mamba models for indoor localisation on edge devices,” Scientific Reports , vol. 15, no. 1, p. 10081, 2025
2025
-
[11]
Learning long sequences in spiking neural networks,
M.-I. Stan and O. Rhodes, “Learning long sequences in spiking neural networks,” Scientific Reports, vol. 14, no. 1, p. 21957, 2024
2024
-
[12]
Zero-shot temporal resolution domain adaptation for spiking neural networks,
S. Karilanova, M. Fabre, E. Neftci, and A. ¨Ozc ¸elikkale, “Zero-shot temporal resolution domain adaptation for spiking neural networks,” arXiv preprint arXiv:2411.04760 , 2024
2024
-
[13]
Quantization-guided training for compact tinyml models,
S. Ghamari, K. Ozcan, T. Dinh, A. Melnikov, J. Carvajal, J. Ernst, and S. Chai, “Quantization-guided training for compact tinyml models,” arXiv preprint arXiv:2103.06231 , 2021
2021 arXiv
-
[14]
Quantization and deployment of deep neural networks on microcontrollers,
P.-E. Novac, G. Boukli Hacene, A. Pegatoquet, B. Miramond, and V . Gripon, “Quantization and deployment of deep neural networks on microcontrollers,” Sensors, vol. 21, no. 9, p. 2984, 2021
2021
-
[15]
Quantization and training of neural networks for efficient integer-arithmetic-only inference,
B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2704–2...
2018
-
[16]
Q-s5: Towards quantized state space models,
S. Abreu, J. E. Pedersen, K. M. Heckel, and A. Pierro, “Q-s5: Towards quantized state space models,” arXiv preprint arXiv:2406.09477 , 2024
2024 arXiv
-
[17]
Quamba: A post-training quantization recipe for selective state space models,
H.-Y . Chiang, C.-C. Chang, N. Frumkin, K.-C. Wu, and D. Marculescu, “Quamba: A post-training quantization recipe for selective state space models,” arXiv preprint arXiv:2410.13229 , 2024
2024 arXiv
-
[18]
Qmamba: Post- training quantization for vision state space models,
Y . Li, X. Liu, J. Li, R. Xu, Y . Chen, and Z. Xiong, “Qmamba: Post- training quantization for vision state space models,” arXiv preprint arXiv:2501.13624, 2025
2025 arXiv
-
[19]
Mamba-ptq: Outlier channels in recurrent large language models,
A. Pierro and S. Abreu, “Mamba-ptq: Outlier channels in recurrent large language models,” arXiv preprint arXiv:2407.12397 , 2024
2024 arXiv
-
[20]
A diagonal structured state space model on loihi 2 for efficient streaming sequence processing,
S. M. Meyer, P. Weidel, P. Plank, L. Campos-Macias, S. B. Shrestha, P. Stratmann, and M. Richter, “A diagonal structured state space model on loihi 2 for efficient streaming sequence processing,” arXiv preprint arXiv:2409.15022, 2024
2024 arXiv
-
[21]
Efficient neuromorphic signal processing with loihi 2,
G. Orchard, E. P. Frady, D. B. D. Rubin, S. Sanborn, S. B. Shrestha, F. T. Sommer, and M. Davies, “Efficient neuromorphic signal processing with loihi 2,” in 2021 IEEE Workshop on Signal Processing Systems (SiPS) , pp. 254–259, IEEE, 2021
2021
-
[22]
How to train your hippo: State space models with generalized orthogonal basis projections,
A. Gu, I. Johnson, A. Timalsina, A. Rudra, and C. R ´e, “How to train your hippo: State space models with generalized orthogonal basis projections,” arXiv preprint arXiv:2206.12037 , 2022
2022 arXiv
-
[23]
Estimating or propagating gradients through stochastic neurons for conditional computation,
Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013
2013 arXiv
-
[24]
Simplified state space layers for sequence modeling,
J. T. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” arXiv preprint arXiv:2208.04933 , 2022
2022 arXiv
-
[25]
S7: Selective and simplified state space layers for sequence modeling,
T. Soydan, N. Zubi ´c, N. Messikommer, S. Mishra, and D. Scaramuzza, “S7: Selective and simplified state space layers for sequence modeling,” arXiv preprint arXiv:2410.03464 , 2024
2024 arXiv
-
[26]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752 , 2023
2023 arXiv
-
[27]
Speech commands: A dataset for limited-vocabulary speech recognition,
P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint arXiv:1804.03209 , 2018
2018 arXiv
-
[28]
Tinyml-based classifi- cation in an ecg monitoring embedded system,
E. Kim, J. Kim, J. Park, H. Ko, and Y . Kyung, “Tinyml-based classifi- cation in an ecg monitoring embedded system,” Computers, Materials and Continua, vol. 75, no. 1, pp. 1751–1764, 2023
2023
-
[29]
Long-term stable electromyography classification using canonical correlation analysis,
E. Donati, S. Benatti, E. Ceolini, and G. Indiveri, “Long-term stable electromyography classification using canonical correlation analysis,” in 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 1–4, IEEE, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.