Pith. sign in

REVIEW 1 major objections 4 minor 21 references

A Deep State-Space Model Compression Method using Upper Bound on Output Error

T0 review · 1 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Pith's one-line read This paper claims that the end-to-end error of a compressed deep state-space model is bounded by a depth-weighted sum of per-layer h2 approximation errors, so shrinking shallow layers matters most.

desk verdict End-to-end output-error bound for Deep SSMs is new and worth engaging with, but Theorem 1's proof has a fixable norm mismatch and the abstract's numerical claims need cleanup. read the letter →

arxiv 2510.14542 v2 pith:RBZ6ETTR submitted 2025-10-16 eess.SY cs.LGcs.SY

classification eess.SYcs.LGcs.SY MSC 93B1193C5593C10
keywords deepstate-spacemodelsmodelorderreductionlinearquadratic-outputsystemsh2normoutputerrorboundgradient-basedcompressionLongRangeArenaIMDb
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies deep state-space models whose internal blocks are linear quadratic-output (LQO) systems and asks how to compress them without losing the overall model's accuracy. It derives an upper bound on the final output error that separates cleanly into per-layer h2 approximation errors, each multiplied by a weight that is largest for shallow layers. This means that, to keep the whole model accurate, a model-order-reduction method should spend more effort on layers near the input. The authors turn that bound into a gradient-based optimization objective, and on the IMDb sequence-classification task they show the compressed model can shed about 80% of its trainable parameters with only a small accuracy drop, even without retraining.

What carries the argument

The central object is the output-error upper bound of Theorem 1 and Corollary 1, expressed as a weighted sum of layerwise h2_L norms between the original and reduced LQO systems. The weights depend on the Lipschitz constants of layer normalization and on the h2_norms of deeper blocks, and they are monotone in depth: shallower layers carry larger weights. This bound is what converts the intractable goal of minimizing the final output error into a tractable per-layer optimization problem with a provable stationarity guarantee.

What would settle it

Take a single scalar LQO layer with A = 0.99, B = 1, C = 0, M = 1, and constant input u_k = 1 for k = 0,...,L−1; reduce it to Ŝ with  = 0.98. Numerically evaluate the exact quadratic convolution error ∥h2∗(u⊗u − û⊗û)∥_{ℓ∞_L} and compare it with the inequality used in Theorem 1; if the inequality is violated, the literal proof of the central bound is incomplete.

Watch

Extended reading notes

Core claim

The central claim is that for a deep state-space model built from LQO blocks, the output error e_ξ between the full model and a reduced model is bounded by a weighted sum of layerwise h2_L error norms: e_ξ ≤ b√(1+b²) Σ_i G̃_i ∥S^(i)−Ŝ^(i)∥_{h2_L} under a bounded-input assumption. The weights G̃_i decrease with depth, so the h2_L errors of shallow layers dominate the bound. The paper then formulates a model-order-reduction problem that minimizes this upper bound, derives gradients for it, and shows numerically that solving it yields compressed models that preserve accuracy and can even work without retraining.

Load-bearing premise

The proof assumes that the convolution of the quadratic kernel with the difference of outer products can be bounded by the diagonal time-sequence norm of that difference; this inequality can fail for legitimate scalar LQO systems, and the paper does not provide the corrected step.

Editorial extensions

If this is right

  • If the bound holds, compression algorithms for deep state-space models should allocate more reduced-order capacity to shallow layers, because errors there propagate and amplify through the network.
  • The bound provides a theoretical justification for earlier layerwise model-order-reduction methods: minimizing per-layer h2 errors is sufficient to control the overall output error, though the weighting shows that uniform per-layer effort is suboptimal.
  • The gradient-based algorithm offers a retraining-free compression route, which is valuable in deployment settings where compute or data for fine-tuning is unavailable.
  • The closed-form solution of the Sylvester equations for diagonal A makes the optimization computationally efficient for the diagonal state-space structures used in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The depth-weighting prediction is directly testable by ablations that vary per-layer reduced dimension while keeping total state count fixed; Table I already suggests that allocating more states to shallow layers improves accuracy.
  • The proof of Theorem 1 relies on a norm inequality that may fail for near-unstable LQO systems with constant inputs; a corrected version using a two-dimensional tensor norm would likely recover the theorem, and such a fix is worth checking.
  • The same output-error bound could be used as a training regularizer to make pretrained models more compressible, not just as a post-hoc reduction criterion.
  • Extending the approach to non-quadratic activations (such as the exponential or softplus used in other state-space models) would require replacing the quadratic-output structure with a more general bound, which is the main limitation of the current framework.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper studies deep state-space models (Deep SSMs) whose internal blocks are discrete-time complex LQO systems. It derives an upper bound (Theorem 1) on the ℓ∞_L error between the final intermediate outputs of a full and a reduced Deep SSM, expressed as a weighted sum of layerwise h2_L approximation errors, and a bounded-input specialization (Corollary 1). The authors then formulate an MOR problem that minimizes this bound, derive gradients for LQO reduced models, propose a stability-enforcing gradient-based algorithm (Algorithm 1), and state a convergence theorem (Theorem 3). Numerical experiments on the LRA IMDb task show that the method can reduce the number of trainable parameters by roughly 80% and can be used without retraining while exhibiting small accuracy loss.

Significance. If the proof of Theorem 1 is repaired, the paper makes a useful system-theoretic contribution: it provides a provable link between layerwise h2-LQO model reduction and end-to-end Deep SSM compression, and it quantifies the intuition that shallow-layer approximation errors have a stronger influence on the final intermediate output. The gradient formulas, the stability mechanism in Algorithm 1, and the convergence theorem make the proposed method concrete and implementable. The experiments are reasonably informative and show a practical advantage over existing MOR-based compression baselines. The manuscript is within the scope of the journal.

major comments (1)
  1. [§IV, Theorem 1 proof, Eq. (14)] The step ∥h2^{(i)} ∗ (u^{(i)}⊗u^{(i)} − û^{(i)}⊗û^{(i)})∥_{ℓ∞_L} ≤ ∥h2∥_{ℓ2_L} ∥u⊗u−û⊗û∥_{ℓ2_L} is not justified as written. Lemma 1 is proved only for the diagonal time-sequence norm: its proof sums over k, so it bounds ∥(u_k⊗u_k − û_k⊗û_k)_k∥_{ℓ2_L}. The Volterra term in Eq. (2), however, is a full two-index convolution over [0,L−1]^2, and the Young step requires the 2D tensor norm on the outer-product sequence. The two norms can differ by a factor up to √L. For example, for a scalar LQO with A close to 1 and u_k=1, û_k=0, the diagonal norm is √L, the 2D norm is L, and the exact quadratic term scales as L^2, so the displayed chain fails. The theorem is likely repairable: the 2D version of Lemma 1, ∥u⊗u−û⊗û∥_{ℓ2(L×L)} ≤ ∥u−û∥_{ℓ2_L}(∥u∥_{ℓ2_L}+∥û∥_{ℓ2_L}), follows by the same triangle/Cauchy argument, and with the 2D norm the constants in (14) are unchanged. Nevertheless, the manuscript
minor comments (4)
  1. [Abstract] There are two conflicting versions of the abstract: the first says 'approximately 60%' parameter reduction, while the second says 'roughly 80%' and '4–5% performance drop.' Table I supports the 80% figure. Please align the abstracts.
  2. [Section I, Notation] The ℓ2_L norm for a two-index kernel h2 is never defined explicitly. Since Lemma 1's ℓ2_L is over the time index only, the two-index norm used for Volterra kernels should be stated explicitly as over [0,L−1]^2.
  3. [§IV, proof of Theorem 1 after Eq. (15)] The phrase 'solving the recurrence (15) with β_i = β̂_i' is confusing and appears inconsistent with Eq. (14), whose G_i uses ∥u^{(j)}∥ + ∥û^{(j)}∥. The recurrence can be solved without identifying β_i with β̂_i; the sentence should be removed or the identification justified and stated as an assumption in the theorem.
  4. [§V, Corollary 1 / Eq. (17)] In Corollary 1, b is introduced as a global bound on all layerwise input norms, but the paper does not explain how b is chosen or verified for a given reduced model in the experiments. A brief remark on this would help the reader connect the theoretical bound to the implemented objective.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the output-error bound is a genuine derived inequality; layerwise h2 errors are independently defined optimization targets, not fitted quantities.

full rationale

The paper's central result (Theorem 1 / Corollary 1) is obtained by propagating a single-LQO output-estimate inequality (4) through the Deep SSM using Volterra expansions, the triangle inequality, Young's convolution inequality, Lemma 1, and the Lipschitz bound on LayerNorm. None of these ingredients assumes the conclusion e_xi <= sum G_i ||S_i - S_hat_i||; the h2_L errors are defined independently from the full/reduced LQO parameters, and the input-norm factors make the bound a true upper bound, not a restatement of the optimization objective. Lemma 1 and the gradient formulas are proved in the appendix or follow from external references; the appearance of the same h2 terms in the objective (17) is the intended use of the bound, not a circular definition. Self-citations are present ([14] in the literature review, [20] in the convergence proof of Theorem 3), but they are not load-bearing for the output-error theorem: [14] is a prior compression method used for comparison/initialization, and [20] is cited only to finish a standard descent/stationarity argument together with the external [21]. The manuscript's apparent gap in bounding the quadratic convolution term by a diagonal-sequence norm (Lemma 1) is a proof-rigor/correctness concern, not circular reasoning; the claimed inequality can be repaired with the corresponding 2D tensor norm without changing the structure of the bound.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

No new physical entity is introduced; the paper's imports are the LQO h2_L trace/gradient theory (from external works) and the assumption that intermediate activations are bounded. The two free quantities that shape the optimization are the norm bound b and the per-layer reduced state dimensions.

free parameters (2)
  • b (input-norm bound) = not reported
    Corollary 1 and the objective (17) depend on b through \tilde{G}_i; the experiments do not state how b is set or estimated.
  • layerwise reduced dimensions r_i = [16,16,16,16] and [32,16,12,4]
    The compression ratios and the 'shallow layers matter' claim depend on hand-chosen r allocations; the paper selects them to lower objective (17).
assumptions (6)
  • domain assumption All full-order and reduced LQO systems are asymptotically stable (spectral radius < 1).
    Assumed before Theorem 1 and enforced in Algorithm 1 by backtracking; needed for finite-horizon Lyapunov/Sylvester solutions and h2_L norms.
  • standard math Finite-horizon Young's convolution inequalities for ℓ_p norms hold for the causal Volterra kernels.
    Used in Theorem 1 proof to propagate errors layerwise; the finite-horizon tensor form is true, but the paper's diagonal-norm application is not.
  • standard math The trace formulas for h2_L norms and the gradients in Lemma 3 are valid as given in [15]-[17].
    The paper imports the time-limited H2 LQO gradient machinery from Reiter et al. and Zulfiqar et al.; central to the optimization.
  • domain assumption LayerNorm is Lipschitz with constant bounded by ∥γ1∥∞/√ε, and the maximum over layers ω is finite.
    Lemma 2 and the definition of ω in Theorem 1; required to propagate the error recurrence.
  • domain assumption Intermediate layer inputs are uniformly bounded: max_j max(∥u^{(j)}∥,∥û^{(j)}∥) ≤ b.
    Corollary 1 converts the input-dependent bound into the optimization objective (17); b is never specified in the experiments.
  • standard math Convergence of the Armijo-backtracking gradient method for the semi-algebraic objective follows [20], [21].
    Theorem 3's stationary-point guarantee is inherited from prior convergence theory, with an extra assumption that the diagonal poles converge to a stable limit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Deep State-Space Model Compression Method using Upper Bound on Output Error." pith.science (2026). https://pith.science/paper/RBZ6ETTR

@misc{pith2026251014542,
  author       = {Pith},
  title        = {Pith review of: A Deep State-Space Model Compression Method using Upper Bound on Output Error},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RBZ6ETTR}},
  note         = {Machine review of arXiv:2510.14542}
}
abstract

We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed in terms of the $h^2$-error norms between the layerwise LQO systems. In particular, we show that reducing the $h^2$ approximation errors of the LQO systems placed in shallow layers is effective in reducing the derived upper bound on the output error. Next, we formulate an optimization problem for the derived upper bound and develop a gradient-based MOR method. In the numerical experiments, using the IMDb task from the LRA benchmark, we demonstrate the effectiveness of the proposed upper-bound-based compression method. In particular, we show that the number of trainable parameters can be reduced by approximately 60\% without retraining while maintaining the performance of the original model.

Figures

Figures reproduced from arXiv: 2510.14542 by the authors.

Figure 1
Figure 1. Comparison between existing MOR methods and the proposed method [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Structure of the Deep SSM considered in this work. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 6 linked inside Pith

  1. [11]

    Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation,

    H. Ezoe and K. Sato, “Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation,”IEEE Access, 2024

  2. [14]

    Compression Method for Deep Diagonal State Space Model Based onH 2 Optimal Reduction,

    H. Sakamoto and K. Sato, “Compression Method for Deep Diagonal State Space Model Based onH 2 Optimal Reduction,”IEEE Control Systems Letters, 2025

  3. [1]

    Hippo: Recurrent memory with optimal polynomial projections,

    A. Gu, T. Dao, S. Ermon, A. Rudra, and C. R ´e, “Hippo: Recurrent memory with optimal polynomial projections,” inAdvances in Neural Information Processing Systems, 2020, pp. 1474–1487

  4. [2]

    Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers,

    A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. R ´e, “Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers,” inAdvances in Neural Information Processing Systems, vol. 34, 2021

  5. [3]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. R ´e, “Efficiently modeling long sequences with structured state spaces,” inInternational Conference on Learning Representations, 2022

  6. [4]

    Simplified State Space Layers for Sequence Modeling,

    J. T. Smith, A. Warrington, and S. Linderman, “Simplified State Space Layers for Sequence Modeling,” inInternational Conference on Learn- ing Representations, 2023

  7. [5]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,”arXiv preprint arXiv:2312.00752, 2023

  8. [6]

    Long range arena: A benchmark for efficient transformers,

    Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, “Long range arena: A benchmark for efficient transformers,” inInternational Conference on Learning Representations, 2021

Show all 21 references
  1. [7]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, vol. 30, 2017

  2. [8]

    State space model for new-generation network alternative to transformers: A survey,

    X. Wang, S. Wang, Y. Ding, Y. Li, W. Wu, Y. Rong, W. Kong, J. Huang, S. Li, H. Yanget al., “State space model for new-generation network alternative to transformers: A survey,”arXiv preprint arXiv:2404.09516, 2024

  3. [9]

    State Space Models are Provably Comparable to Transformers in Dynamic Token Selection,

    N. Nishikawa and T. Suzuki, “State Space Models are Provably Comparable to Transformers in Dynamic Token Selection,” inThe Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/forum?id=QFgbJOYJSE

  4. [10]

    Model compression and acceleration for deep neural networks: The principles, progress, and challenges,

    Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “Model compression and acceleration for deep neural networks: The principles, progress, and challenges,”IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 126– 136, 2018

  5. [12]

    Model order reduction of deep structured state-space models: A system-theoretic approach,

    M. Forgione, M. Mejari, and D. Piga, “Model order reduction of deep structured state-space models: A system-theoretic approach,”arXiv preprint arXiv:2403.14833, 2024

  6. [13]

    Layer-Adaptive State Pruning for Deep State Space Models,

    M. Gwak, S. Moon, J. Ko, and P. Park, “Layer-Adaptive State Pruning for Deep State Space Models,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 10 613–10 645

  7. [15]

    Gramians, energy functionals, and balanced truncation for linear dynamical systems with quadratic outputs,

    P. Benner, P. Goyal, and I. P. Duff, “Gramians, energy functionals, and balanced truncation for linear dynamical systems with quadratic outputs,”IEEE Transactions on Automatic Control, vol. 67, no. 2, pp. 886–893, 2021

  8. [16]

    H 2 optimal model reduction of linear systems with multiple quadratic outputs,

    S. Reiter, I. Pontes Duff, I. V. Gosea, and S. Gugercin, “H 2 optimal model reduction of linear systems with multiple quadratic outputs,” arXiv preprint arXiv:2405.05951, 2024

  9. [17]

    Time-limitedH 2-optimal model order reduction of linear systems with quadratic outputs,

    U. Zulfiqar, Z.-H. Xiao, Q.-Y. Song, M. M. Uddin, and V. Sreeram, “Time-limitedH 2-optimal model order reduction of linear systems with quadratic outputs,”arXiv preprint arXiv:2408.05965, 2024

  10. [18]

    Solution of the matrix equationAX+ XB=C,

    R. H. Bartels and G. W. Stewart, “Solution of the matrix equationAX+ XB=C,”Communications of the ACM, vol. 15, no. 9, pp. 820–826, 1972

  11. [19]

    Balanced truncation of linear systems with quadratic outputs in limited time and frequency intervals,

    Q.-Y. Song, U. Zulfiqar, Z.-H. Xiao, M. M. Uddin, and V. Sreeram, “Balanced truncation of linear systems with quadratic outputs in limited time and frequency intervals,”arXiv preprint arXiv:2402.11445, 2024

  12. [20]

    Data-drivenh 2 model reduction for linear discrete-time systems,

    H. Sakamoto and K. Sato, “Data-drivenh 2 model reduction for linear discrete-time systems,”arXiv preprint arXiv:2401.05774, 2025

  13. [21]

    Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods,

    H. Attouch, J. Bolte, and B. F. Svaiter, “Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods,” Mathematical programming, vol. 137, no. 1, pp. 91–129, 2013

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.