Pith. sign in

REVIEW 2 major objections 5 minor 21 references

Delay State Space Models add explicit delayed-state feedback so efficient recurrent sequence models can retrieve long-range context precisely.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 13:14 UTC pith:3DN3K5UR

load-bearing objection Solid methods paper: delayed diagonal SSMs with a real stability split, exact Z-transfer, and contour-shift FFT that actually works on delayed-retrieval tasks; general gains over S4D are real but modest, and LM is missing. the 2 major comments →

arxiv 2607.10244 v1 pith:3DN3K5UR submitted 2026-07-11 cs.LG

DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

Yixiao Qian , Song Chen , Jiaxu Liu , Shengze Cai , Chao Xu This is my paper
classification cs.LG
keywords state space modelsdelay differential equationslong-range memoryFFT trainingsequence modelingdelayed retrievaldiagonal SSMskernel contour shift
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

State space models train in parallel and run in linear time, but they compress the whole past into a fixed-size state, which makes exact long-range retrieval hard. This paper introduces Delay State Space Models: a delay-differential-equation style extension of diagonal SSMs that feeds a buffered past state back into the discrete update. The authors keep the system stable with a simple budget that splits discrete memory between the main branch and the delayed branch, manage continuous delays with a history buffer and linear interpolation, and recover parallel training by evaluating an exact frequency-domain transfer function with a kernel contour shift that controls time-domain aliasing. On synthetic delayed copy, sum, and weighted-sum tasks the delayed models learn the right lag far more reliably than plain diagonal SSMs; on image, speech, medical, and Long Range Arena benchmarks they match or beat that baseline on most metrics. A reader who wants efficient sequence models that still need precise long-horizon recall would see this as a way to add explicit memory without abandoning fast training or inference.

Core claim

The paper claims that diagonal SSM recurrences augmented with explicit delayed-state feedback—under a delay-independent discrete stability condition, with history buffering and continuous-delay interpolation, and trained from the exact Z-domain transfer function via a kernel contour shift—substantially improve targeted delayed-retrieval tasks while outperforming the diagonal SSM baseline on most standard sequence metrics and remaining close on the others, all while retaining efficient FFT training and linear-time recurrent inference.

What carries the argument

Delay State Space Models (DSSMs): the discrete update h_t = Ā h_{t-1} + Ā_d h̃_{t-τ} + B̄ x_t, with mode-wise stability |λ̄| + |μ̄| < 1, linear interpolation of a continuous delay from a fixed history buffer, and FFT training from the closed-form transfer function after a kernel contour shift that damps aliasing then restores the linear kernel.

Load-bearing premise

The argument assumes a fixed history buffer, linear interpolation of a continuous delay, and a conservative stability budget that caps the sum of main and delayed gains are expressive enough to give real long-range memory without being strangled by that budget or by interpolation error.

What would settle it

On the fixed- and variable-delay copy, sum, and weighted-sum tasks, collapse the learned delay to one step or zero the delayed-branch gain; if accuracy stays near the full model instead of collapsing as the paper’s ablations report, the delayed-feedback claim fails. A matched large-scale language-modeling comparison where the delayed model does not improve long-context retrieval would likewise falsify transfer of the mechanism.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Explicit delayed feedback can close much of the gap between pure SSMs and attention on fixed- and variable-delay retrieval without quadratic inference cost.
  • Parallel FFT training remains available for delayed linear recurrences once the transfer function and contour-shift restoration are in place.
  • A distributed bank of channel-wise delays can be preserved under the stability budget rather than collapsing to a single lag.
  • On multimodal and Long Range Arena tasks the same delayed mechanism does not hurt—and often helps—standard accuracy relative to the diagonal baseline.
  • Decoding memory scales with a bounded history horizon rather than full sequence length, unlike full attention caches.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Input-dependent or selective delay allocation could combine this explicit memory with content-aware routing, covering cases where a fixed delay bank is insufficient.
  • The contour-shift route may generalize to other non-Markovian linear systems that lack closed time-domain kernels.
  • Whether a bounded delay buffer remains competitive with hybrid attention–SSM stacks at language-model scale is left open by the paper and is a natural next stress test.
  • Phase initialization near 0 versus π may be a reusable design knob for delayed aggregation versus change detection in other delayed dynamical layers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Delay State Space Models (DSSMs / DS4D), a DDE-inspired extension of diagonal SSMs that augments the discrete recurrence with explicit delayed-state feedback via a history buffer and linear interpolation of a continuous delay. It supplies a delay-independent discrete stability condition (|λ̄|+|μ̄|<1) enforced by a shared budget split (Prop. 1; Eqs. 14–17, 19), an exact Z-domain transfer function for the interpolated recurrence (Prop. 2), and an FFT training algorithm that uses a kernel contour shift to control time-domain aliasing (Alg. 1; App. A.7). Empirically, DS4D substantially improves six synthetic delayed-retrieval tasks relative to S4D/Mamba/LSTM and is competitive with Transformer recipes, while on multimodal and LRA benchmarks it outperforms S4D on most metrics and remains close on the rest (Tables 1–2; Fig. 2).

Significance. If the construction holds, the paper gives a concrete, trainable way to add explicit long-range memory to diagonal SSMs without abandoning parallel FFT training or O(1)-per-step inference. Strengths include careful supporting theory (conjugate symmetry, discrete stability proof by contradiction, exact transfer function, aliasing analysis), matched-parameter synthetic baselines with ablations that isolate the delayed branch (Table 1b), and reported FFT-vs-recurrent numerical agreement at ~1e-5 relative RMSE (Table 7). The contribution is incremental relative to hybrid SSM–attention work, but it is a clean dynamical-systems route to precise delayed retrieval that is of genuine interest to the long-sequence modeling community.

major comments (2)
  1. Sec. 5.3 / Table 2: the central claim that DSSMs transfer beyond delayed retrieval rests on modest, mixed gains over S4D (e.g., SC(BI.) 96.04 vs 96.18; BIDMC-RR 0.262 vs 0.254; Retrieval 91.03 vs 91.09). With only 2–3 seeds and no large-scale language modeling (Sec. 6), it remains unclear whether the delayed branch is generally beneficial or mainly specialized to the synthetic regime of Sec. 5.1. A stronger transfer experiment or a clearer statement of when delay helps would better support the abstract’s broader claim.
  2. Prop. 1 and Eqs. (14)–(17): the delay-independent budget |λ̄|+|μ̄|=|ζ̄|<1 is sufficient but conservative; it forces the main and delayed branches to share a single contraction budget. Combined with a fixed L_max buffer and linear interpolation (Sec. 3.2), this may limit expressivity of the delayed branch on tasks that need both strong local dynamics and large delayed gain. The paper should either quantify the cost of this conservatism (e.g., comparison to a less restrictive stability region) or more carefully bound when the parameterization remains expressive enough for the claimed memory gains.
minor comments (5)
  1. Sec. 6 and App. C: code, training scripts, and exact seeds are not released in the manuscript materials; releasing them would make the FFT accuracy and synthetic ablations fully reproducible.
  2. Fig. 3 / App. C.3: decoding memory is O(L_max) by design; a short comparison to Transformer KV-cache scaling at matched context would make the efficiency trade-off clearer for non-SSM readers.
  3. App. A.4 / Fig. 4: the phase-initialization intuition is helpful but qualitative; a brief quantitative sensitivity of final accuracy to ω≈0 vs π on one synthetic task would strengthen Sec. 3.4.
  4. Notation: the continuous DDE uses A_d while the discrete form uses Ā_d / μ̄; a single consistent naming note early in Sec. 3 would reduce friction.
  5. Table 2 caption: clarify which S4D numbers are re-run vs literature-reported (only sMNIST is re-run) so variance comparisons are unambiguous.

Circularity Check

0 steps flagged

No significant circularity: stability, transfer function, and FFT path are derived; empirical gains are measured on external tasks with ablations.

full rationale

This is a methods paper whose load-bearing chain is (i) a DDE-inspired discrete recurrence with delayed feedback, (ii) a delay-independent stability criterion proved by contradiction on the characteristic equation (Prop. 1 / App. A.2), (iii) a parameterization that enforces that criterion by design (Eqs. 15–17), (iv) an exact Z-domain transfer function for the interpolated recurrence (Prop. 2 / App. A.6), and (v) a kernel contour-shift FFT procedure to control circular aliasing (App. A.7, Alg. 1). None of these steps redefine the target claim as an input: the stability condition is a sufficient criterion proved from the recurrence, not fitted to retrieval accuracy; the transfer function is algebraically equivalent to the stated recurrence, not a post-hoc fit; contour shift is a numerical device with reported FFT-vs-recurrent error (~1e-5 relative RMSE, Table 7). Empirical claims rest on synthetic delayed-retrieval tasks and public benchmarks (sMNIST, SC, BIDMC, LRA) with matched-parameter S4D baselines and ablations that zero the delayed branch or collapse τ (Table 1b). Citations to Gu et al. (S4/S4D) are external background and baselines, not self-citation uniqueness theorems. Hyperparameter choices (delay init, p_n, phase, ε_shift, LR sweeps) are ordinary ML design, not tautological predictions. Score 0; steps empty.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 3 invented entities

The central claim rests on standard linear systems and FFT tools, plus modeling choices that make delayed feedback trainable: diagonal modes, a conservative stability budget, fixed history buffers, linear delay interpolation, and a numerical contour shift. Free parameters are mostly architectural/init/training knobs rather than constants fitted to force a physical law. The main invented entities are the DSSM layer itself and the contour-shift training device; both are operationally defined and tested, not free-floating ontological posits.

free parameters (6)
  • Shared discrete memory budget |ζ̄_n| and branch allocation p_n
    Learned/initialized split of total discrete magnitude between main and delayed branches; p_n init (e.g. 0.1 or task-specific 0.015) strongly affects whether delayed feedback is active.
  • Delay location τ (via q=sigmoid(ν_τ)) and L_max history horizon
    Learnable continuous delay and fixed buffer length bound explicit memory; grid or task-specific inits (row width 28/32) are design choices that shape results.
  • Delayed-phase initialization θ_n ≈ 0 or π
    Hand-chosen bias toward reinforcement vs mismatch sensitivity; paper shows qualitative effect but phase design remains partly heuristic (Sec. 3.4, App. A.4, Sec. 6).
  • Kernel contour-shift target ε_shift (ρ^M)
    Numerical free parameter balancing residual aliasing against ρ^{-t} restoration amplification; experiments use ε_shift=10^{-3}.
  • Step-size Δt and its reduced learning rate
    Softplus-parameterized discretization step with separately reduced LR (often 1e-4); authors note higher sensitivity than base LR.
  • State size N, hidden size H, depth, and matched-parameter retuning
    Architecture knobs adjusted per task to match baselines; not derived from theory.
axioms (6)
  • domain assumption Diagonal A and A_d suffice; modes may be treated as independent scalar DDEs/recurrences.
    Sec. 3.1 follows the diagonal SSM literature (S4D) for FFT efficiency; expressivity of non-diagonal delayed coupling is not explored.
  • standard math Zero-input asymptotic stability is guaranteed by the sufficient condition |λ̄|+|μ̄|<1 for the interpolated delay recurrence.
    Prop. 1 / App. A.2; delay-independent and possibly conservative relative to sharp DDE stability regions.
  • domain assumption Linear interpolation of buffered states adequately approximates continuous-time delayed state on the discrete grid.
    Sec. 3.2; higher-order history reconstruction is not analyzed.
  • standard math LTI delayed recurrence admits an exact rational Z-domain transfer function usable for FFT convolution after contour shift.
    Prop. 2 and App. A.6–A.7; standard Z-transform shift rules plus aliasing identity for circular kernels.
  • standard math Conjugate-pair symmetry of complex parameters yields real outputs for real inputs.
    App. A.1; standard complex-SSM practice.
  • domain assumption S4D-style HiPPO-inspired initialization of shared base dynamics remains a good starting point when a delayed branch is added.
    Sec. 3.4; inherits S4D-Inv/Lin without re-deriving optimality under delay.
invented entities (3)
  • Delay State Space Model (DSSM / DS4D layer) no independent evidence
    purpose: Augment diagonal SSM recurrence with explicit delayed-state feedback and learnable delay while preserving efficient training/inference.
    Core proposed architecture; independent evidence is empirical (synthetic ablations, benchmarks), not external physical measurement.
  • Kernel contour shift for delayed SSM FFT training no independent evidence
    purpose: Suppress time-domain aliasing of long memory echoes so IFFT recovery yields an accurate linear convolution kernel.
    Training device defined by damping ρ and shifted coefficients; validated by FFT-vs-recurrent numerical agreement, not by an external theory theorem beyond standard aliasing analysis.
  • Shared discrete budget split (λ̄, μ̄) from ζ̄ and p no independent evidence
    purpose: Enforce mode-wise stability while allocating memory between instantaneous and delayed branches.
    Parameterization invention specific to making delayed feedback practical under Prop. 1.

pith-pipeline@v1.1.0-grok45 · 25269 in / 4029 out tokens · 37602 ms · 2026-07-14T13:14:42.652325+00:00 · methodology

0 comments
read the original abstract

State Space Models (SSMs) have emerged as a powerful paradigm for efficient long-sequence modeling, offering parallel training and fast linear-time recurrent inference. However, like other recurrent architectures, SSMs must compress an unbounded history into a fixed-size state, which limits context retention and makes precise retrieval over long-range context inherently difficult. To overcome this limitation, we propose Delay State Space Models (DSSMs), a delay differential equation (DDE)-inspired extension of diagonal SSMs that augments discrete SSM recurrences with explicit delayed-state feedback. Making explicit delayed feedback practical requires new stability parameterization, history management, and FFT-training tools. We address these challenges with a practical discretization and parameterization grounded in a simple delay-independent stability condition. To bypass direct time-domain kernel construction, we derive the DSSM transfer function and compute kernels in the frequency domain, using a kernel contour shift to suppress aliasing and recover accurate FFT training. Empirically, DSSMs substantially improve targeted delayed-retrieval tasks while outperforming S4D on most standard sequence metrics and remaining close on the others.

Figures

Figures reproduced from arXiv: 2607.10244 by Chao Xu, Jiaxu Liu, Shengze Cai, Song Chen, Yixiao Qian.

Figure 1
Figure 1. Figure 1: Overview of the discrete DSSM update. The state combines the input term, the main-branch [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Validation trajectories on the six synthetic benchmarks using the best learning rate selected [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Efficiency of DS4D versus S4D under matched-parameter and matched-hidden-size settings. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative effect of delayed-phase initialization under two simple backgrounds. (a) [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: First-layer impulse-response heatmaps for the six synthetic DS4D checkpoints. Rows are [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: First-layer impulse-response heatmaps for the corresponding S4D checkpoints on the six [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 16 linked inside Pith

  1. [1]

    Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927,

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927,

    Pith/arXiv arXiv

  2. [2]

    Hungry hungry hippos: Towards language modeling with state space models.arXiv preprint arXiv:2212.14052,

    Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models.arXiv preprint arXiv:2212.14052,

    Pith/arXiv arXiv

  3. [3]

    Zamba: A compact 7b ssm hybrid model.arXiv preprint arXiv:2405.16712,

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model.arXiv preprint arXiv:2405.16712,

    Pith/arXiv arXiv

  4. [4]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

    Pith/arXiv arXiv

  5. [5]

    Liquid structural state-space models.arXiv preprint arXiv:2209.12951,

    Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models.arXiv preprint arXiv:2209.12951,

    Pith/arXiv arXiv

  6. [6]

    Kakade, and Eran Malach

    Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Trans- formers are better than state space models at copying.arXiv preprint arXiv:2402.01032,

    Pith/arXiv arXiv

  7. [7]

    Li, Berlin Chen, Caitlin Wang, Aviv Bick, J

    Aakash Lahoti, Kevin Y . Li, Berlin Chen, Caitlin Wang, Aviv Bick, J. Zico Kolter, Tri Dao, and Albert Gu. Mamba-3: Improved sequence modeling using state space principles.arXiv preprint arXiv:2603.15569,

    arXiv

  8. [8]

    Jamba: A hybrid transformer- mamba language model.arXiv preprint arXiv:2403.19887,

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model.arXiv preprint arXiv:2403.19887,

    Pith/arXiv arXiv

  9. [9]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling.arXiv preprint arXiv:2406.07522,

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling.arXiv preprint arXiv:2406.07522,

    Pith/arXiv arXiv

  10. [10]

    Exploring the limitations of mamba in copy and cot reasoning

    10 Ruifeng Ren, Zhicong Li, and Yong Liu. Exploring the limitations of mamba in copy and cot reasoning. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 12550–12574,

    2025

  11. [11]

    Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150,

    Noam Shazeer. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150,

    Pith/arXiv arXiv 1911

  12. [12]

    Simplified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933,

    Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling.arXiv preprint arXiv:2208.04933,

    Pith/arXiv arXiv

  13. [13]

    Long range arena: A benchmark for efficient transformers.arXiv preprint arXiv:2011.04006,

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers.arXiv preprint arXiv:2011.04006,

    Pith/arXiv arXiv 2011

  14. [14]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971,

    Pith/arXiv arXiv

  15. [15]

    An empirical study of mamba- based language models.arXiv preprint arXiv:2406.07887,

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba- based language models.arXiv preprint arXiv:2406.07887,

    Pith/arXiv arXiv

  16. [16]

    Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization.arXiv preprint arXiv:2311.14495,

    Shida Wang and Qianxiao Li. Stablessm: Alleviating the curse of memory in state-space models through stable reparameterization.arXiv preprint arXiv:2311.14495,

    Pith/arXiv arXiv

  17. [17]

    State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory.arXiv preprint arXiv:2309.13414,

    Shida Wang and Beichen Xue. State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory.arXiv preprint arXiv:2309.13414,

    Pith/arXiv arXiv

  18. [18]

    Neural delay differential equations.arXiv preprint arXiv:2102.10801,

    Qunxi Zhu, Yao Guo, and Wei Lin. Neural delay differential equations.arXiv preprint arXiv:2102.10801,

    Pith/arXiv arXiv

  19. [19]

    Let ht ∈C and xt ∈R denote the scalar state and input at discrete step t. Withτ= (k+α)∆t, the scalar recurrence is ht = ¯λht−1 + ¯µ[(1−α)ht−k +αh t−k−1] +¯bxt.(50) Applying the Z-transform and using the standard integer-shift rules Z{h t−1}=z −1H(z),Z{h t−k}=z −kH(z),Z{h t−k−1}=z −k−1H(z), gives H(z) = ¯λz−1H(z) + ¯µz−k (1−α) +αz −1 H(z) + ¯bX(z).(51) Sol...

    1997

  20. [20]

    DS4D uses a smaller matched-parameter hidden size than S4D because of its additional delay parameters, while Mamba keeps its default architecture and tunes state size only to best match the total parameter budget. MODEL#LAYERSHIDDENSIZE#HEADSSTATESIZE#PARAMS DELAYCOPY LSTM 2 96 – – 168.4k S4D 2 143 – 64 167.3k Mamba 2 88 – 48 167.8k GPT-2 recipe 2 60 4 – ...

    2048

  21. [21]

    LRA.Long Range Arena (LRA) [Tay et al., 2020] evaluates efficient sequence models on a diverse suite of long-context classification tasks

    The three official targets are respiratory rate (RR), heart rate (HR), and blood oxygen saturation (SpO2). LRA.Long Range Arena (LRA) [Tay et al., 2020] evaluates efficient sequence models on a diverse suite of long-context classification tasks. Following the task summaries used in the S4D/S5 literature, we consider: • ListOps.A length- 2000 hierarchical ...

    2020