Pith. sign in

REVIEW 9 cited by

How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2206.12037 v2 pith:QUDH4H7H submitted 2022-06-24 cs.LG

Albert Gu , Isys Johnson , Aman Timalsina , Atri Rudra , Christopher Ré This is my paper
classification cs.LG
keywords matrixstatehippolongmodelsspaceabilityactually
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Linear time-invariant state space models (SSM) are a classical model from engineering and statistics, that have recently been shown to be very promising in machine learning through the Structured State Space sequence model (S4). A core component of S4 involves initializing the SSM state matrix to a particular matrix called a HiPPO matrix, which was empirically important for S4's ability to handle long sequences. However, the specific matrix that S4 uses was actually derived in previous work for a particular time-varying dynamical system, and the use of this matrix as a time-invariant SSM had no known mathematical interpretation. Consequently, the theoretical mechanism by which S4 models long-range dependencies actually remains unexplained. We derive a more general and intuitive formulation of the HiPPO framework, which provides a simple mathematical interpretation of S4 as a decomposition onto exponentially-warped Legendre polynomials, explaining its ability to capture long dependencies. Our generalization introduces a theoretically rich class of SSMs that also lets us derive more intuitive S4 variants for other bases such as the Fourier basis, and explains other aspects of training S4, such as how to initialize the important timescale parameter. These insights improve S4's performance to 86% on the Long Range Arena benchmark, with 96% on the most difficult Path-X task.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficiently Modeling Long Sequences with Structured State Spaces

    cs.LG 2021-10 unverdicted novelty 8.0 of 10

    S4 is an efficient state space sequence model that captures long-range dependencies via structured parameterization of the SSM, achieving state-of-the-art results on the Long Range Arena and other benchmarks while bei...

  2. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

    cs.LG 2024-01 conditional novelty 7.0 of 10

    Medusa augments LLMs with multiple decoding heads and tree-based attention to predict and verify several tokens in parallel, yielding 2.2-3.6x inference speedup via two fine-tuning regimes.

  3. Continuity Laws for Sequential Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    S4 models exhibit stable time-continuity unlike sensitive S6 models, with task continuity predicting performance and enabling temporal subsampling for better efficiency.

  4. Structured State-Space Regularization for Generation-Friendly Image Tokenization

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    A new regularizer transfers frequency awareness from state-space models into image tokenizers, yielding more compact latents that improve diffusion-model generation quality with little reconstruction penalty.

  5. DeMa: Dual-Path Delay-Aware Mamba for Efficient Multivariate Time Series Analysis

    cs.LG 2026-01 unverdicted novelty 6.0 of 10

    DeMa is a dual-path delay-aware Mamba architecture that decomposes MTS into intra-series temporal and inter-series variate paths to achieve SOTA performance with linear complexity on forecasting, imputation, anomaly d...

  6. Structured State-Space Regularization for Generation-Friendly Image Tokenization

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    Structured state-space regularization induces spectral structure in image tokenizer latent spaces via an SSM-derived objective, improving generative performance with minimal reconstruction loss.

  7. Rivaling Transformers: Multi-Scale Structured State-Space Mixtures for Agentic 6G O-RAN

    cs.NI 2025-10 conditional novelty 5.0 of 10

    A 0.70M-parameter multi-scale state-space mixture predicts next-step RSRP on an O-RAN testbed with RMSE 0.29 dB and R2=0.993, running 3-10x faster than the tested Transformers.

  8. FlexiD-Fuse: Flexible number of inputs multi-modal medical image fusion based on diffusion model

    cs.CV 2025-09 reject novelty 4.0 of 10

    FlexiD-Fuse adapts a denoising diffusion model and an expectation-maximization step to fuse either two or three medical images with one shared network, and reports better scores than fixed-count baselines on standard metrics.

  9. Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State- Space Architectures from S4 to Mamba

    cs.LG 2025-03 unverdicted

    A survey tracing the evolution of state-space models like S4 and Mamba, their efficiency trade-offs, and applications in NLP, vision, and other domains.

Pith tools