REVIEW 12 cited by
On the Parameterization and Initialization of Diagonal State Space Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
On the Parameterization and Initialization of Diagonal State Space Models
read the original abstract
State space models (SSM) have recently been shown to be very effective as a deep learning layer as a promising alternative to sequence models such as RNNs, CNNs, or Transformers. The first version to show this potential was the S4 model, which is particularly effective on tasks involving long-range dependencies by using a prescribed state matrix called the HiPPO matrix. While this has an interpretable mathematical mechanism for modeling long dependencies, it introduces a custom representation and algorithm that can be difficult to implement. On the other hand, a recent variant of S4 called DSS showed that restricting the state matrix to be fully diagonal can still preserve the performance of the original model when using a specific initialization based on approximating S4's matrix. This work seeks to systematically understand how to parameterize and initialize such diagonal state space models. While it follows from classical results that almost all SSMs have an equivalent diagonal form, we show that the initialization is critical for performance. We explain why DSS works mathematically, by showing that the diagonal restriction of S4's matrix surprisingly recovers the same kernel in the limit of infinite state dimension. We also systematically describe various design choices in parameterizing and computing diagonal SSMs, and perform a controlled empirical study ablating the effects of these choices. Our final model S4D is a simple diagonal version of S4 whose kernel computation requires just 2 lines of code and performs comparably to S4 in almost all settings, with state-of-the-art results for image, audio, and medical time-series domains, and averaging 85\% on the Long Range Arena benchmark.
Forward citations
Cited by 12 Pith papers
-
Efficiently Modeling Long Sequences with Structured State Spaces
S4 is an efficient state space sequence model that captures long-range dependencies via structured parameterization of the SSM, achieving state-of-the-art results on the Long Range Arena and other benchmarks while bei...
-
Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization
Out-of-distribution extrapolation in neural networks depends on structural commitments in feature and model choices that resolve non-identifiability of data-generating processes from in-distribution data.
-
L2RU: a Structured State Space Model with prescribed L2-bound
L2RU parametrizes SSMs to enforce a prescribed L2-gain bound for guaranteed input-output stability and robustness in all parameter regimes.
-
No Free Lunch in Flow Surrogates under Time-Varying Boundary Conditions: A Two-Regime Study
No single flow-surrogate architecture transfers from a boundary-driven Stokes film to a self-sustained Kármán wake; time treatment decides the winner and pointwise RMSE ranks the wrong models.
-
Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention
A DP-means allocate-on-novelty cache matches full-attention associative recall while storing only distinct items, and a minimal novelty gate recovers the rule end-to-end.
-
Free Parametrization of L_2-Bounded Structured State-Space Controllers for Nonlinear Control with Stability Guarantees
A new free parametrization of L2-bounded LTI systems creates L2RU SSM layers that enforce stability by design, allowing unconstrained nonlinear controller optimization with guarantees via small-gain theorem.
-
Towards Understanding Self-Pretraining for Sequence Classification
Self-pretraining improves Transformer sequence classification by enabling learning of proximity-biased attention from positional encodings that label supervision alone cannot easily acquire from random starts.
-
Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization
Out-of-distribution extrapolation is non-identifiable from in-distribution data alone; the feature map, label map, and model class supply the identifiability bias that determines whether a network succeeds or fails at...
-
Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams
A Dirichlet-process novelty cache plus recurrence matches full-attention coupling at about half the attended tokens on long redundant streams, with inspectable retained slots.
-
Cumsum-Composable Phase Transport for Low-Cost Streaming Keyword Spotting
Finite-window unitary phase-transport layers whose recurrences collapse to cumulative sums match compact CNN baselines on Speech Commands and run with lower latency than a custom scan.
-
Sessa: Selective State Space Attention
Sessa integrates attention within recurrent paths to achieve power-law memory tails and flexible non-decaying selective retrieval, outperforming baselines on long-context tasks.
-
Rivaling Transformers: Multi-Scale Structured State-Space Mixtures for Agentic 6G O-RAN
A 0.70M-parameter multi-scale state-space mixture predicts next-step RSRP on an O-RAN testbed with RMSE 0.29 dB and R2=0.993, running 3-10x faster than the tested Transformers.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.