REVIEW 3 cited by
Towards a theory of learning dynamics in deep state space models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
State space models (SSMs) have shown remarkable empirical performance on many long sequence modeling tasks, but a theoretical understanding of these models is still lacking. In this work, we study the learning dynamics of linear SSMs to understand how covariance structure in data, latent state size, and initialization affect the evolution of parameters throughout learning with gradient descent. We show that focusing on the learning dynamics in the frequency domain affords analytical solutions under mild assumptions, and we establish a link between one-dimensional SSMs and the dynamics of deep linear feed-forward networks. Finally, we analyze how latent state over-parameterization affects convergence time and describe future work in extending our results to the study of deep SSMs with nonlinear connections. This work is a step toward a theory of learning dynamics in deep state space models.
Forward citations
Cited by 3 Pith papers
-
AutoNeural: Co-Designing Vision-Language Models for NPU Inference
A NPU-native VLM combining a MobileNet-style encoder with a hybrid Transformer-SSM backbone claims 14x lower latency and 7x lower quantization error over ViT-Transformer baselines, though quantized accuracy is not reported.
-
Learning Robust Spectral Dynamics for Temporal Domain Generalization
FreKoo splits evolving model parameters into low-frequency trends and high-frequency noise, extrapolates the trends with a learned Koopman operator, and regularizes the noise to improve temporal domain generalization.
-
Block-Biased Mamba for Long-Range Sequence Processing
B2S6, a block-biased variant of Mamba's S6 unit, restores universal approximation, softens input-dependent forgetting, and beats S4/S4D on Long-Range Arena while keeping Mamba-like language perplexity.
Discussion (0). Continue with ORCID to comment.