{"id":"52eb4375-7c9b-4f9c-9ce6-69391c12f646","arxiv_id":"2508.20441","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A discrete-domain Fourier initialization for diagonal state space models yields uniform spectral coverage and trains from scratch on PathX-256.","lead":"This paper studies why diagonal state space models for long sequences favor local patterns, and proposes a new initialization that places the model's poles directly on the discrete Fourier grid to remove dependence on the discretization step. The new method matches or beats prior initializations on most Long Range Arena tasks and is the first reported from-scratch run on the 65,536-token PathX-256 benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'universal approximator' claim for S4D-DFouT is unsupported: for ξ=0 the kernel is exactly N-periodic, so it spans only circular N-convolutions, not arbitrary length-L kernels; the LRA coverage actually comes from the layer-level NH grid, whose sufficiency is never analyzed.","rationale":"Charitable reading: S4D-DFouT is a plausible discrete-domain initialization, and the PathX-256 from-scratch result is interesting. What would have to be true for the central claim as stated—uniform, alias-free, universal spectral coverage enabling SOTA LRA—is that the discrete Fourier pole grid spans the relevant kernels. The weakest link is the universality claim: at ξ=0 the single-SSM kernel is N-periodic, so it cannot represent arbitrary length-L kernels; the paper's own text restricts exactness to circular N-convolutions before overclaiming. I agree with the reader that this is the weakest assumption. I also checked Proposition 1 and found a concrete off-by-one error in the Gram computation (l=0,...,τ instead of 0,...,τ−1), so the claimed condition number of 1 and one-step convergence are wrong; the delay-spike intuition may still hold, but the proof as written is incorrect. The layer-wise synchronization partially mitigates the off-grid concern for the actual LRA experiments (NH=32768 for PathX-256), which is why I would not reject the paper: the empirical contribution can stand, but the theory needs correction and the claims need qualification. Verdict remains CONDITIONAL; no change from the reader's verdict.","tokens_in":18202,"tokens_out":16975,"duration_ms":165263,"concrete_test":"Run a synthetic delay/copying task with L=4096 and target kernel h[l]=δ_{l,τ}, τ=1000 (not a multiple of N). For a single S4D-DFouT SSM with ξ=0 and N=64,128,256,512, solve the least-squares readout C and compute the ℓ2 error to h. If the error stays large until N approaches L (or until τ becomes a multiple of N), the universal-approximator claim is refuted. This isolates the N-periodic span problem from layer-level synchronization and from the learnable ξ.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2: for ξ=0, λ_n = e^{i2πn/N}, so the kernel K[l]=Re(Σ C_n λ_n^l) is N-periodic in l. The text admits exactness only for circular convolution kernels of length N, then leaps to 'universal approximator.' That leap is invalid: a delay kernel δ_{l,τ} with τ>N is not in the span, and over l=0,...,L−1 with L not a multiple of N the N exponentials are not orthogonal, so the 'complete, non-redundant basis' description applies only to the N-point DFT. The LRA results can be true while this theory is false: layer synchronization (eq. 11) yields H·N=32768 poles for PathX-256, matching L/2, so dense coverage is supplied by the layer, not by single-SSM universality. The paper gives no condition on H,N,L for guaranteed coverage, and its own Section 6 limitation (psCIFAR 65.7%, no improvement) is consistent with the method exploiting periodic local structure rather than universal spectral coverage. Secondary concrete error: Proposition 1's proof sums l=0,...,τ in the DFT orthogonality identity; off-diagonal Gram entries are 1, not 0, so the claimed condition number one and one-step gradient-descent convergence are false.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes diagonal state-space model (SSM) initializations from a frequency-domain perspective. It argues that existing continuous-domain initializations such as S4D-Lin and S4D-Inv entangle decay and frequency through the discretization step Δ, producing non-uniform spectral coverage and sensitivity to the chosen Δ. The authors propose S4D-DFouT, which initializes the discrete-time poles directly on a uniform grid λ_n = exp(-ξn/2 + i2πn/N), optionally with layer-wise phase offsets, and claim this yields alias-free uniform spectral coverage independent of Δ. The paper provides a frequency-response analysis, a motivating delay task, and experiments on the Long Range Arena benchmark, including a from-scratch PathX-256 result at 87.89% accuracy, along with ablations and kernel visualizations suggesting that previously initialized SSMs learn local, row-periodic kernels on image tasks.","tokens_in":18489,"tokens_out":8156,"duration_ms":72427,"significance":"If the empirical results are reproducible, the from-scratch PathX-256 result is a noteworthy advance for diagonal SSMs, and the frequency-based design principle is simple and potentially useful. The experimental section is thorough, with per-task hyperparameters, multiple ablations, and an honest report of a failure on psCIFAR. However, the theoretical claims exceed what is proved: the 'universal approximator' statement in Section 4.2 is false as stated, and Proposition 1's proof contains a concrete algebraic error. These issues do not necessarily invalidate the empirical findings, but they must be corrected before the paper can be accepted.","major_comments":[{"comment":"The claim in Section 4.2 that S4D-DFouT 'enables the system to act as a universal approximator' is not supported by the preceding derivation. For ξ=0, the modes λ_n = e^{i2πn/N} yield basis kernels K_n[l] that are N-periodic in l, so the span of the N modes is confined to N-periodic sequences; a delay kernel δ_{l,τ} with τ>N is not representable. Moreover, for a sequence length L not a multiple of N, these exponentials are not orthogonal over l=0,...,L−1, so the 'complete, non-redundant basis' statement applies only to the N-point DFT/circular convolution. The layer-wise synchronization in Eq. (11) provides NH distinct frequencies, which may be the real source of dense coverage (for PathX-256, NH=32768 and L/2=32768), but no condition on H, N, L is stated under which the fixed grid covers the relevant task frequencies. The claim should be weakened to a statement about uniform coverage at the DFT grid, with the covering condition made explicit.","section":"Section 4.2, Eq. (10)"},{"comment":"Appendix B, Proposition 1 contains an algebraic error. For V_{l,n}=e^{i2πnl/τ} with l=0,...,τ, the off-diagonal entries of the Gram matrix are ∑_{l=0}^{τ} e^{i2π(n−m)l/τ} = 1 for n≠m, not 0. Hence V*V ≠ (τ+1)I, the condition number is not one, and the claimed one-step gradient descent convergence with η=1/(τ+1) does not follow. The proof also assumes continuous poles with no real part (w_n=iπn), whereas S4D-Lin is defined in Eq. (9) with poles −1/2+iπn, so the proposition does not accurately describe S4D-Lin under ZOH discretization. The spike-at-τ conclusion may remain true qualitatively, but the theorem and proof must be corrected or the proposition removed.","section":"Appendix B, Proposition 1"}],"minor_comments":[{"comment":"There are typographical errors in the S4D-DFouT paragraph: 'propsoe' should be 'propose' and 'absis' should be 'basis'. Also, 'S4D-DfouT' appears in Section 5.5 and Figure 8 with inconsistent capitalization.","section":"Section 4.2"},{"comment":"The notation for the kernel index is inconsistent: the proposition uses ℓ while Eq. (5) uses l. Please unify.","section":"Section 3.1 / Proposition 1"},{"comment":"It is unclear whether the experimental results use the full grid of Eq. (10) or the half-plane variant described in Section 4.2; the hyperparameters in Table 5 do not specify. Please clarify this in the experimental setup.","section":"Section 4.2 / Table 5"},{"comment":"The recurrence in Eq. (12) appears to contain a typo: 'h[l + 1] = Λx[l] + Bx[l]' should likely read 'h[l + 1] = Λh[l] + Bx[l]', and 'y[l] = Cx[l]' should likely be 'y[l] = C^T h[l]'.","section":"Appendix A.1, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The reported PathX-256 from-scratch result is striking, and the proposed initialization is simple enough to be of broad interest. The two theoretical issues identified above are the main obstacles; I would support acceptance after a careful revision that corrects the proof and tempers the universal-approximation claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real, useful idea—initializing diagonal SSM poles directly in the discrete domain with uniform DFT spacing and layer-wise phase offsets—and the PathX-256 from-scratch result is worth taking seriously. The paper is not as clean as its own framing suggests: the universal approximator claim is unsupported, and one proof in the appendix has a concrete algebraic error. Neither kills the method, but both need fixing.\n\nWhat's new: the discrete-domain pole placement in Eq. (10) plus the synchronized grid in Eq. (11) genuinely decouples decay from frequency, which prior continuous-time initializations don't do. The analysis in Section 3.2 and Fig. 3 makes the Delta-entanglement problem vivid, and the continuous-copying experiment is a nice minimal demonstration. The local-kernel observation (kernels focus on row-stride 32 for sCIFAR, 128 for PathX) is probably the most insightful part of the paper, and it connects to a real weakness of LRA as a benchmark.\n\nSoft spots, in order of importance. First, Section 4.2's \"universal approximator\" statement overreaches. At ξ=0 the kernel is exactly N-periodic, so the span is circular N-convolutions, not arbitrary length-L kernels. You can't conclude universal approximation for L>N from DFT completeness on N points. The stress-test note is right: what makes LRA work is the layer-level NH grid (32768 poles on PathX-256), not single-SSM universality. The authors' own psCIFAR number (65.7%) is consistent with a method that exploits periodic local structure rather than universal coverage. Remove or carefully qualify that claim.\n\nSecond, Proposition 1's conditioning/gradient-descent part is wrong as written. With V_{l,n}=e^{i2πnl/τ} and l=0..τ, the Gram matrix has off-diagonal entries 1, not 0, so V*V is not (τ+1)I and the condition number is not 1. One-step GD convergence also doesn't follow. The \"spike at τ\" part may survive, but the proof needs a corrected argument or a weaker statement.\n\nThird, reporting hygiene: headline LRA numbers lack error bars, no code is released, and the S4D-Lin PathX-128 result required widening the Delta range (0.0001, 0.1), which is a tuned baseline, not a fixed comparison. Gains over S4D-Inv/S5 on most tasks are 1-2 points; PathX-256 is the real differentiator.\n\nBottom line: core initialization is sound and empirically promising. This deserves peer review, with requests to trim the universality claim, fix Proposition 1, add variance estimates, and release code. I'd bring it to reading group and cite it once the claims are honest.","headline":"A genuinely useful discrete-domain pole initialization for diagonal SSMs, with a clean motivation and a strong PathX-256 result, but the universal-approximator claim and one proof step need fixing before the framing is trustworthy.","tokens_in":19065,"tokens_out":3965,"would_cite":true,"duration_ms":38259,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diagonal state-space models initialized directly in the discrete Fourier domain achieve uniform, alias-free spectral coverage independent of the discretization step, enabling from-scratch training on PathX-256.","keywords":["state space models","spectral bias","initialization","discrete Fourier transform","Long Range Arena","sequence modeling","PathX-256"],"falsifier":"Train S4D-DFouT on a sequence-classification task constructed so the class label depends on the amplitude or phase of a single sinusoid whose frequency is deliberately off the N-point uniform grid (e.g., a repeating period of N+1). If the model still solves it, the universal-coverage claim would outrun the paper's circular-convolution argument; if it fails, the fixed-grid assumption is the load-bearing limitation.","tokens_in":17990,"feed_emoji":"📡","tokens_out":9142,"duration_ms":77284,"temperature":0.7,"pith_summary":"State space models (SSMs) are sequence models that represent long-range dependencies through a convolution kernel parameterized by a linear dynamical system, and diagonal SSMs restrict the state matrix to complex diagonal form. This paper tries to establish that the initialization of that diagonal matrix—not the architecture itself—is what biases these models toward local, short-range solutions, and that the bias can be removed by placing poles directly in the discrete domain. It shows that continuous-time initializations couple decay rate and oscillation frequency through the discretization step $\\Delta$, so the frequency coverage depends on choosing $\\Delta$ correctly, while the proposed S4D-DFouT initialization $\\lambda_n = \\exp(-\\xi n/2 + i 2\\pi n/N)$ covers the spectrum uniformly and alias-free regardless of $\\Delta$. If the claim is right, diagonal SSMs no longer need task-specific timescale tuning, and the paper reports state-of-the-art Long Range Arena results including the first from-scratch training on PathX-256 at 87.89% accuracy.","feed_headline":"Uniform Fourier poles unlock state-space models on PathX-256","feed_subtitle":"First from-scratch training on PathX-256, a task earlier models could only reach after self-pretraining.","key_machinery":"The central object is S4D-DFouT, an initialization for diagonal state-space models that fixes the discrete-time eigenvalues directly as $\\lambda_n = \\exp(-\\xi n/2 + i 2\\pi n/N)$ for $n = 0,\\dots,N-1$, with $\\xi$ a learnable damping factor. A layer of $H$ parallel SSMs is synchronized by assigning each machine a phase offset $\\phi_h = 2\\pi(h-1)/(NH)$, so the union of poles forms a uniform grid of $NH$ frequencies on $[0,2\\pi)$. That grid is what does the work: it decouples decay from frequency selection, removes the discretization step $\\Delta$ from the spectral-coverage question, and eliminates aliasing as long as every mode stays within its Nyquist band. In the $\\xi = 0$ limit the state transition is unitary and the model reduces to a DFT basis, capable of representing any circular convolution kernel of length $N$; a half-plane variant for real inputs keeps only $N/2+1$ poles, halving the state dimension.","core_discovery":"The central discovery is a diagnosis and a cure. In a diagonal SSM, each basis kernel is a damped complex exponential whose discrete frequency is $\\Omega = \\Delta\\omega$; discretizing a continuous pole $\\lambda = -\\alpha + i\\omega$ maps it to $e^{-\\alpha\\Delta + i\\Delta\\omega}$, so $\\Delta$ shifts both decay and resonance. S4D-Lin and S4D-Inv initializations therefore cover different spectral regions depending on $\\Delta$, and when $\\Delta$ misses the task's intrinsic timescale, the learned kernels collapse to a local band—in sCIFAR the effective kernel has peaks only at row-stride offsets and almost no energy beyond. S4D-DFouT instead initializes $\\lambda_n = \\exp(-\\xi n/2 + i 2\\pi n/N)$ in the discrete domain, putting all poles on a uniform grid around the unit circle with a shared learnable damping; this gives complete, non-redundant coverage independent of $\\Delta$ and, in the $\\xi = 0$ limit, reduces the state transition to the DFT. On Long Range Arena the paper reports 87.89% accuracy on PathX-256 from scratch, the first such result without self-pretraining.","pith_inferences":["Editorial inference: the uniform-grid argument suggests an adaptive initialization that places poles at the top-$K$ dominant frequencies estimated from the training data would need fewer modes than a full uniform grid for the same accuracy on local-structure tasks.","Editorial inference: because the universal-approximation claim in the paper holds for $N$-periodic circular kernels rather than arbitrary length-$L$ kernels, a direct comparison on a synthetic task with an off-grid informative frequency would delimit when the initialization helps versus when it is neutral.","Editorial inference: the observed inactivity of most SSM modes under the baseline suggests that the same $H_\\infty$-norm analysis applied to S4D-DFouT after training could yield a pruning criterion, shrinking models with little accuracy loss."],"forward_implications":["S4D-DFouT makes the initialization's frequency coverage independent of the discretization step $\\Delta$, so the same hyperparameter settings transfer across LRA tasks without task-specific $\\Delta$ tuning.","Fourier-initialized SSMs can match or exceed HiPPO-style diagonal baselines when their spectral support is uniform, overturning the earlier view that Fourier initialization underperforms on hard LRA tasks.","Far from requiring long-range receptive fields, LRA successes can be driven by local structure captured by a few spectral modes; in sCIFAR the learned kernels are essentially determined by the first 32 coefficients, the row stride.","The first from-scratch run on PathX-256 (87.89% accuracy) shows diagonal SSMs can scale to 65,536-token sequences without self-pretraining."],"supporting_citations":[{"why":"Defines the HiPPO framework of orthogonal-polynomial projection that continuous-domain diagonal initializations inherit and simplify.","marker":"[8]"},{"why":"Introduces the S4 structured state-space model and the S4-FouT baseline, establishing the architecture family under study.","marker":"[9]"},{"why":"Shows diagonal state spaces can match structured ones, motivating the diagonal setting and the DSS baseline.","marker":"[11]"},{"why":"Defines S4D-Lin, S4D-Inv, and S4D-LegS, the baseline initializations whose discretization sensitivity is the paper's target.","marker":"[12]"},{"why":"Provides the S5 simplified state-space layer used as a competitive LRA baseline.","marker":"[14]"},{"why":"Supplies the Long Range Arena benchmark tasks and metrics on which the empirical claims are made.","marker":"[17]"},{"why":"Previously reported PathX-256 results that required self-pretraining, the direct comparison for the from-scratch claim.","marker":"[28]"},{"why":"Contributes the $H_\\infty$-norm per-mode analysis used to show that most baseline SSM modes remain inactive.","marker":"[16]"}],"fun_headline_variants":["Fourier poles train state-space models on PathX-256 from scratch","New diagonal SSM init breaks PathX-256 record","S4D-DFouT: uniform spectral init for state-space models","Mastering PathX-256 via discrete Fourier initialization","Spectral bias exposed, Fourier fix scales SSMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that the repeating pattern a task depends on has one of the evenly spaced periods the initialization provides; if the key information repeats at any other rate, the fixed grid supplies no mode to detect it.","fun_headline_variants_meta":{"raw":{"variants":["Fourier poles train state-space models on PathX-256 from scratch","New diagonal SSM init breaks PathX-256 record","S4D-DFouT: uniform spectral init for state-space models","Mastering PathX-256 via discrete Fourier initialization","Spectral bias exposed, Fourier fix scales SSMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1274,"prompt_tokens":976,"completion_tokens":298,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":592,"tokens_out":298,"duration_ms":2978,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:45:13.814930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train S4D-DFouT on a sequence-classification task constructed so the class label depends on the amplitude or phase of a single sinusoid whose frequency is deliberately off the N-point uniform grid (e.g., a repeating period of N+1). If the model still solves it, the universal-coverage claim would outrun the paper's circular-convolution argument; if it fails, the fixed-grid assumption is the load-bearing limitation.","supporting_citations":[{"cited_title":"Hippo: Recurrent memory with optimal polynomial projections","cited_arxiv_id":null,"evidence_quote":"Defines the HiPPO framework of orthogonal-polynomial projection that continuous-domain diagonal initializations inherit and simplify."},{"cited_title":"Efficiently modeling long sequences with structured state spaces","cited_arxiv_id":null,"evidence_quote":"Introduces the S4 structured state-space model and the S4-FouT baseline, establishing the architecture family under study."},{"cited_title":"Diagonal state spaces are as effective as structured state spaces","cited_arxiv_id":null,"evidence_quote":"Shows diagonal state spaces can match structured ones, motivating the diagonal setting and the DSS baseline."},{"cited_title":"On the parameterization and initialization of diagonal state space models","cited_arxiv_id":null,"evidence_quote":"Defines S4D-Lin, S4D-Inv, and S4D-LegS, the baseline initializations whose discretization sensitivity is the paper's target."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the S5 simplified state-space layer used as a competitive LRA baseline."},{"cited_title":"Long range arena : A benchmark for efficient transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the Long Range Arena benchmark tasks and metrics on which the empirical claims are made."},{"cited_title":"Never train from scratch: Fair comparison of long-sequence models requires data-driven priors","cited_arxiv_id":null,"evidence_quote":"Previously reported PathX-256 results that required self-pretraining, the direct comparison for the from-scratch claim."},{"cited_title":"Layer-adaptive state pruning for deep state space models","cited_arxiv_id":null,"evidence_quote":"Contributes the $H_\\infty$-norm per-mode analysis used to show that most baseline SSM modes remain inactive."}],"review_version":2}