REVIEW 1 major objections 4 minor 21 references
A Deep State-Space Model Compression Method using Upper Bound on Output Error
T0 review · 1 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper claims that the end-to-end error of a compressed deep state-space model is bounded by a depth-weighted sum of per-layer h2 approximation errors, so shrinking shallow layers matters most.
desk verdict End-to-end output-error bound for Deep SSMs is new and worth engaging with, but Theorem 1's proof has a fixable norm mismatch and the abstract's numerical claims need cleanup. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the output-error upper bound of Theorem 1 and Corollary 1, expressed as a weighted sum of layerwise h2_L norms between the original and reduced LQO systems. The weights depend on the Lipschitz constants of layer normalization and on the h2_norms of deeper blocks, and they are monotone in depth: shallower layers carry larger weights. This bound is what converts the intractable goal of minimizing the final output error into a tractable per-layer optimization problem with a provable stationarity guarantee.
What would settle it
Take a single scalar LQO layer with A = 0.99, B = 1, C = 0, M = 1, and constant input u_k = 1 for k = 0,...,L−1; reduce it to Ŝ with  = 0.98. Numerically evaluate the exact quadratic convolution error ∥h2∗(u⊗u − û⊗û)∥_{ℓ∞_L} and compare it with the inequality used in Theorem 1; if the inequality is violated, the literal proof of the central bound is incomplete.
Extended reading notes
Core claim
The central claim is that for a deep state-space model built from LQO blocks, the output error e_ξ between the full model and a reduced model is bounded by a weighted sum of layerwise h2_L error norms: e_ξ ≤ b√(1+b²) Σ_i G̃_i ∥S^(i)−Ŝ^(i)∥_{h2_L} under a bounded-input assumption. The weights G̃_i decrease with depth, so the h2_L errors of shallow layers dominate the bound. The paper then formulates a model-order-reduction problem that minimizes this upper bound, derives gradients for it, and shows numerically that solving it yields compressed models that preserve accuracy and can even work without retraining.
Load-bearing premise
The proof assumes that the convolution of the quadratic kernel with the difference of outer products can be bounded by the diagonal time-sequence norm of that difference; this inequality can fail for legitimate scalar LQO systems, and the paper does not provide the corrected step.
Editorial extensions
If this is right
- If the bound holds, compression algorithms for deep state-space models should allocate more reduced-order capacity to shallow layers, because errors there propagate and amplify through the network.
- The bound provides a theoretical justification for earlier layerwise model-order-reduction methods: minimizing per-layer h2 errors is sufficient to control the overall output error, though the weighting shows that uniform per-layer effort is suboptimal.
- The gradient-based algorithm offers a retraining-free compression route, which is valuable in deployment settings where compute or data for fine-tuning is unavailable.
- The closed-form solution of the Sylvester equations for diagonal A makes the optimization computationally efficient for the diagonal state-space structures used in practice.
Reading between the lines
- The depth-weighting prediction is directly testable by ablations that vary per-layer reduced dimension while keeping total state count fixed; Table I already suggests that allocating more states to shallow layers improves accuracy.
- The proof of Theorem 1 relies on a norm inequality that may fail for near-unstable LQO systems with constant inputs; a corrected version using a two-dimensional tensor norm would likely recover the theorem, and such a fix is worth checking.
- The same output-error bound could be used as a training regularizer to make pretrained models more compressible, not just as a post-hoc reduction criterion.
- Extending the approach to non-quadratic activations (such as the exponential or softplus used in other state-space models) would require replacing the quadratic-output structure with a more general bound, which is the main limitation of the current framework.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies deep state-space models (Deep SSMs) whose internal blocks are discrete-time complex LQO systems. It derives an upper bound (Theorem 1) on the ℓ∞_L error between the final intermediate outputs of a full and a reduced Deep SSM, expressed as a weighted sum of layerwise h2_L approximation errors, and a bounded-input specialization (Corollary 1). The authors then formulate an MOR problem that minimizes this bound, derive gradients for LQO reduced models, propose a stability-enforcing gradient-based algorithm (Algorithm 1), and state a convergence theorem (Theorem 3). Numerical experiments on the LRA IMDb task show that the method can reduce the number of trainable parameters by roughly 80% and can be used without retraining while exhibiting small accuracy loss.
Significance. If the proof of Theorem 1 is repaired, the paper makes a useful system-theoretic contribution: it provides a provable link between layerwise h2-LQO model reduction and end-to-end Deep SSM compression, and it quantifies the intuition that shallow-layer approximation errors have a stronger influence on the final intermediate output. The gradient formulas, the stability mechanism in Algorithm 1, and the convergence theorem make the proposed method concrete and implementable. The experiments are reasonably informative and show a practical advantage over existing MOR-based compression baselines. The manuscript is within the scope of the journal.
major comments (1)
- [§IV, Theorem 1 proof, Eq. (14)] The step ∥h2^{(i)} ∗ (u^{(i)}⊗u^{(i)} − û^{(i)}⊗û^{(i)})∥_{ℓ∞_L} ≤ ∥h2∥_{ℓ2_L} ∥u⊗u−û⊗û∥_{ℓ2_L} is not justified as written. Lemma 1 is proved only for the diagonal time-sequence norm: its proof sums over k, so it bounds ∥(u_k⊗u_k − û_k⊗û_k)_k∥_{ℓ2_L}. The Volterra term in Eq. (2), however, is a full two-index convolution over [0,L−1]^2, and the Young step requires the 2D tensor norm on the outer-product sequence. The two norms can differ by a factor up to √L. For example, for a scalar LQO with A close to 1 and u_k=1, û_k=0, the diagonal norm is √L, the 2D norm is L, and the exact quadratic term scales as L^2, so the displayed chain fails. The theorem is likely repairable: the 2D version of Lemma 1, ∥u⊗u−û⊗û∥_{ℓ2(L×L)} ≤ ∥u−û∥_{ℓ2_L}(∥u∥_{ℓ2_L}+∥û∥_{ℓ2_L}), follows by the same triangle/Cauchy argument, and with the 2D norm the constants in (14) are unchanged. Nevertheless, the manuscript
minor comments (4)
- [Abstract] There are two conflicting versions of the abstract: the first says 'approximately 60%' parameter reduction, while the second says 'roughly 80%' and '4–5% performance drop.' Table I supports the 80% figure. Please align the abstracts.
- [Section I, Notation] The ℓ2_L norm for a two-index kernel h2 is never defined explicitly. Since Lemma 1's ℓ2_L is over the time index only, the two-index norm used for Volterra kernels should be stated explicitly as over [0,L−1]^2.
- [§IV, proof of Theorem 1 after Eq. (15)] The phrase 'solving the recurrence (15) with β_i = β̂_i' is confusing and appears inconsistent with Eq. (14), whose G_i uses ∥u^{(j)}∥ + ∥û^{(j)}∥. The recurrence can be solved without identifying β_i with β̂_i; the sentence should be removed or the identification justified and stated as an assumption in the theorem.
- [§V, Corollary 1 / Eq. (17)] In Corollary 1, b is introduced as a global bound on all layerwise input norms, but the paper does not explain how b is chosen or verified for a given reduced model in the experiments. A brief remark on this would help the reader connect the theoretical bound to the implemented objective.
Circularity Check
No circularity: the output-error bound is a genuine derived inequality; layerwise h2 errors are independently defined optimization targets, not fitted quantities.
full rationale
The paper's central result (Theorem 1 / Corollary 1) is obtained by propagating a single-LQO output-estimate inequality (4) through the Deep SSM using Volterra expansions, the triangle inequality, Young's convolution inequality, Lemma 1, and the Lipschitz bound on LayerNorm. None of these ingredients assumes the conclusion e_xi <= sum G_i ||S_i - S_hat_i||; the h2_L errors are defined independently from the full/reduced LQO parameters, and the input-norm factors make the bound a true upper bound, not a restatement of the optimization objective. Lemma 1 and the gradient formulas are proved in the appendix or follow from external references; the appearance of the same h2 terms in the objective (17) is the intended use of the bound, not a circular definition. Self-citations are present ([14] in the literature review, [20] in the convergence proof of Theorem 3), but they are not load-bearing for the output-error theorem: [14] is a prior compression method used for comparison/initialization, and [20] is cited only to finish a standard descent/stationarity argument together with the external [21]. The manuscript's apparent gap in bounding the quadratic convolution term by a diagonal-sequence norm (Lemma 1) is a proof-rigor/correctness concern, not circular reasoning; the claimed inequality can be repaired with the corresponding 2D tensor norm without changing the structure of the bound.
Assumptions & free parameters
free parameters (2)
- b (input-norm bound) =
not reported
- layerwise reduced dimensions r_i =
[16,16,16,16] and [32,16,12,4]
assumptions (6)
- domain assumption All full-order and reduced LQO systems are asymptotically stable (spectral radius < 1).
- standard math Finite-horizon Young's convolution inequalities for ℓ_p norms hold for the causal Volterra kernels.
- standard math The trace formulas for h2_L norms and the gradients in Lemma 3 are valid as given in [15]-[17].
- domain assumption LayerNorm is Lipschitz with constant bounded by ∥γ1∥∞/√ε, and the maximum over layers ω is finite.
- domain assumption Intermediate layer inputs are uniformly bounded: max_j max(∥u^{(j)}∥,∥û^{(j)}∥) ≤ b.
- standard math Convergence of the Armijo-backtracking gradient method for the semi-algebraic objective follows [20], [21].
Cite this review
Pith. "Pith review of A Deep State-Space Model Compression Method using Upper Bound on Output Error." pith.science (2026). https://pith.science/paper/RBZ6ETTR
@misc{pith2026251014542,
author = {Pith},
title = {Pith review of: A Deep State-Space Model Compression Method using Upper Bound on Output Error},
year = {2026},
howpublished = {\url{https://pith.science/paper/RBZ6ETTR}},
note = {Machine review of arXiv:2510.14542}
}
abstract
We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed in terms of the $h^2$-error norms between the layerwise LQO systems. In particular, we show that reducing the $h^2$ approximation errors of the LQO systems placed in shallow layers is effective in reducing the derived upper bound on the output error. Next, we formulate an optimization problem for the derived upper bound and develop a gradient-based MOR method. In the numerical experiments, using the IMDb task from the LRA benchmark, we demonstrate the effectiveness of the proposed upper-bound-based compression method. In particular, we show that the number of trainable parameters can be reduced by approximately 60\% without retraining while maintaining the performance of the original model.
Figures
Reference graph
Works this paper leans on
-
[11]
Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation,
H. Ezoe and K. Sato, “Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation,”IEEE Access, 2024
2024
-
[14]
Compression Method for Deep Diagonal State Space Model Based onH 2 Optimal Reduction,
H. Sakamoto and K. Sato, “Compression Method for Deep Diagonal State Space Model Based onH 2 Optimal Reduction,”IEEE Control Systems Letters, 2025
2025
-
[1]
Hippo: Recurrent memory with optimal polynomial projections,
A. Gu, T. Dao, S. Ermon, A. Rudra, and C. R ´e, “Hippo: Recurrent memory with optimal polynomial projections,” inAdvances in Neural Information Processing Systems, 2020, pp. 1474–1487
2020
-
[2]
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers,
A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. R ´e, “Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers,” inAdvances in Neural Information Processing Systems, vol. 34, 2021
2021
-
[3]
Efficiently modeling long sequences with structured state spaces,
A. Gu, K. Goel, and C. R ´e, “Efficiently modeling long sequences with structured state spaces,” inInternational Conference on Learning Representations, 2022
2022
-
[4]
Simplified State Space Layers for Sequence Modeling,
J. T. Smith, A. Warrington, and S. Linderman, “Simplified State Space Layers for Sequence Modeling,” inInternational Conference on Learn- ing Representations, 2023
2023
-
[5]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023
arXiv 2023
-
[6]
Long range arena: A benchmark for efficient transformers,
Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, “Long range arena: A benchmark for efficient transformers,” inInternational Conference on Learning Representations, 2021
2021
Show all 21 references
-
[7]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, vol. 30, 2017
2017
-
[8]
State space model for new-generation network alternative to transformers: A survey,
X. Wang, S. Wang, Y. Ding, Y. Li, W. Wu, Y. Rong, W. Kong, J. Huang, S. Li, H. Yanget al., “State space model for new-generation network alternative to transformers: A survey,”arXiv preprint arXiv:2404.09516, 2024
2024 arXiv
-
[9]
State Space Models are Provably Comparable to Transformers in Dynamic Token Selection,
N. Nishikawa and T. Suzuki, “State Space Models are Provably Comparable to Transformers in Dynamic Token Selection,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=QFgbJOYJSE
2025
-
[10]
Model compression and acceleration for deep neural networks: The principles, progress, and challenges,
Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “Model compression and acceleration for deep neural networks: The principles, progress, and challenges,”IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 126– 136, 2018
2018
-
[12]
Model order reduction of deep structured state-space models: A system-theoretic approach,
M. Forgione, M. Mejari, and D. Piga, “Model order reduction of deep structured state-space models: A system-theoretic approach,”arXiv preprint arXiv:2403.14833, 2024
2024 arXiv
-
[13]
Layer-Adaptive State Pruning for Deep State Space Models,
M. Gwak, S. Moon, J. Ko, and P. Park, “Layer-Adaptive State Pruning for Deep State Space Models,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 10 613–10 645
2024
-
[15]
Gramians, energy functionals, and balanced truncation for linear dynamical systems with quadratic outputs,
P. Benner, P. Goyal, and I. P. Duff, “Gramians, energy functionals, and balanced truncation for linear dynamical systems with quadratic outputs,”IEEE Transactions on Automatic Control, vol. 67, no. 2, pp. 886–893, 2021
2021
-
[16]
H 2 optimal model reduction of linear systems with multiple quadratic outputs,
S. Reiter, I. Pontes Duff, I. V. Gosea, and S. Gugercin, “H 2 optimal model reduction of linear systems with multiple quadratic outputs,” arXiv preprint arXiv:2405.05951, 2024
2024 arXiv
-
[17]
Time-limitedH 2-optimal model order reduction of linear systems with quadratic outputs,
U. Zulfiqar, Z.-H. Xiao, Q.-Y. Song, M. M. Uddin, and V. Sreeram, “Time-limitedH 2-optimal model order reduction of linear systems with quadratic outputs,”arXiv preprint arXiv:2408.05965, 2024
2024 arXiv
-
[18]
Solution of the matrix equationAX+ XB=C,
R. H. Bartels and G. W. Stewart, “Solution of the matrix equationAX+ XB=C,”Communications of the ACM, vol. 15, no. 9, pp. 820–826, 1972
1972
-
[19]
Balanced truncation of linear systems with quadratic outputs in limited time and frequency intervals,
Q.-Y. Song, U. Zulfiqar, Z.-H. Xiao, M. M. Uddin, and V. Sreeram, “Balanced truncation of linear systems with quadratic outputs in limited time and frequency intervals,”arXiv preprint arXiv:2402.11445, 2024
2024 arXiv
-
[20]
Data-drivenh 2 model reduction for linear discrete-time systems,
H. Sakamoto and K. Sato, “Data-drivenh 2 model reduction for linear discrete-time systems,”arXiv preprint arXiv:2401.05774, 2025
2025
-
[21]
Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods,
H. Attouch, J. Bolte, and B. F. Svaiter, “Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods,” Mathematical programming, vol. 137, no. 1, pp. 91–129, 2013
2013
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.