REVIEW 5 cited by
Diagonal State Spaces are as Effective as Structured State Spaces
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Modeling long range dependencies in sequential data is a fundamental step towards attaining human-level performance in many modalities such as text, vision, audio and video. While attention-based models are a popular and effective choice in modeling short-range interactions, their performance on tasks requiring long range reasoning has been largely inadequate. In an exciting result, Gu et al. (ICLR 2022) proposed the $\textit{Structured State Space}$ (S4) architecture delivering large gains over state-of-the-art models on several long-range tasks across various modalities. The core proposition of S4 is the parameterization of state matrices via a diagonal plus low rank structure, allowing efficient computation. In this work, we show that one can match the performance of S4 even without the low rank correction and thus assuming the state matrices to be diagonal. Our $\textit{Diagonal State Space}$ (DSS) model matches the performance of S4 on Long Range Arena tasks, speech classification on Speech Commands dataset, while being conceptually simpler and straightforward to implement.
Forward citations
Cited by 5 Pith papers
-
Systolic Array-based Accelerator for Structured State-Space Models
A specialized systolic-array accelerator with a reconfigurable processing element and diagonal dataflow claims 2000x inference speedup over GPUs for S4 and Liquid-S4 state-space models.
-
Explicit Context Reasoning with Supervision for Visual Tracking
RSTrack supervises a Mamba-based state reasoning module with true target states, improving visual tracking accuracy on six benchmarks.
-
Tacit Learning with Adaptive Information Selection for Cooperative Multi-Agent Reinforcement Learning
SICA combines selective state-space filtering with attention-based training-time communication and a regeneration module to let MARL agents coordinate without messages at execution time.
-
Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry
A proposed 'Riemann GeoResolver' framework for inverse-distance attention in hyperbolic and spherical geometry, whose central PL inequality rests on an invalid derivative computation.
-
State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking
A small complex-valued recurrent model achieves 100% accuracy on three length-16 state-tracking puzzles, but the claim that state propagation alone is sufficient is not tested against any baseline.
Discussion (0). Continue with ORCID to comment.