{"id":"9878874a-1390-4f25-8578-e5d7eb96ab0b","arxiv_id":"2505.17868","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A distillation method that turns spectral-filter sequence models into explicit linear dynamical systems, enabling constant-time per-token generation with a provable (conditional) accuracy bound.","lead":"This paper converts a trained spectral-filter sequence model (STU) into a simple recurrent linear dynamical system, so generating each token takes constant time and memory instead of growing with sequence length. Tests on a 340M-parameter language model show matching accuracy on benchmarks while speeding up generation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's distillation bound is conditioned on h·λmax(M†)=O(1), which is only shown empirically; without an analytic bound the 'provable' claim is not established, and the deployed Algorithm 3 is outside the theorem's scope.","rationale":"The paper has a real and interesting construction: approximating STU spectral filters by a low-dimensional LDS is supported by strong empirical evidence, including near-identical 340M benchmark scores and a clear inference speedup. The issue is not the empirical distillation; it is the mismatch between the adjective 'provable' and the proof. Theorem 1's error bound is linear in λmax, and the only evidence that λmax is benign is experimental (Appendix A.2). Because the theorem is the basis for the abstract's 'first provable method' claim, an unproven numeric condition on a pseudo-inverse is load-bearing. A concrete computation of λmax in the exact experimental regime would settle whether the constant is benign there; if it is, the practical method may still be fine, but the proof would remain conditional until an analytic bound is supplied. The mismatch with Algorithm 3 further weakens the claim that the evaluated system is the proven one, though the theoretical construction itself remains a useful contribution. I would keep the reader's conditional verdict: the theoretical claim needs strengthening, but the empirical work and the constructive reduction are sufficient for conditional acceptance.","tokens_in":20740,"tokens_out":8998,"duration_ms":87199,"concrete_test":"Reproduce Appendix A.2 in the exact headline regime: for k = 24, L = 131072, h = 160 (and h = 80), draw α_i from the distribution used by Algorithm 3 (Fig. 6), construct M row-wise as m_i = Φ1:k µ_L(α_i) (the least-squares spectral coefficients), and compute the largest singular value of the pseudo-inverse. Repeat for L ∈ {4096, 131072, 1048576} and ≥100 draws; report median and worst-case h·λmax. If h·λmax is not O(1) or grows with L, Theorem 1's bound is vacuous in the deployed setting. Optionally, compute the same quantity for the α subset actually selected by Algorithm 3, since that is the matrix used in the experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central guarantee is Theorem 1, whose RHS contains λmax, the largest singular value of M†, where M is the h×k matrix of spectral coefficients produced by Algorithm 2. The proof in Appendix A.3 bounds ||Φ1:k − M†µ|| by λmax·||E||1 and then invokes the spectral-filtering estimate ||E||1 ≤ c·h·e^{−k/log L}; nothing in the argument controls λmax. The paper explicitly says an upper bound is 'not immediate' and appeals to Figs. 4–5. If h·λmax is not O(1) in the regime k = O(log L), h = O(k), the displayed error is vacuous and the claimed independence of state dimension and effective memory is not delivered. The empirical Figs. 4–5 use k = 48 and a tailored α distribution, not the k = 24, L = 131072, h = 160 setting of the language experiments, so they do not directly certify the needed constant there. Additionally, M = ΨΦ1:k^T is a projection of the Vandermonde matrix, so the full-column-rank claim used to define M† is not a direct consequence of Vandermonde nonsingularity; it needs a probability estimate that is not supplied. The practical distillation (Algorithm 3) is a different subset-selection and gradient-refinement procedure, so even a valid Theorem 1 would not literally cover the model evaluated in Table 2.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SpectraLDS, a method for distilling the convolutional filters of a Spectral Transform Unit (STU) into an explicit symmetric linear dynamical system (LDS). The central theoretical result, Theorem 1, states that for h ≥ k randomly sampled geometric filters, Algorithm 2 returns a matrix fM such that the first k spectral filters Φ1:k are approximated by fM times the LDS impulse responses, with error bounded by c·λmax·h·e^{−k/log L}, where λmax is the largest singular value of the pseudo-inverse of the spectral coefficient matrix M. The paper also presents a practical variant (Algorithm 3) based on subset selection and gradient refinement, and evaluates the approach on synthetic LDS identification, filter reconstruction, large-scale language modeling, and inference speed. The main claim is that this distillation preserves predictive accuracy while enabling constant-time per-token inference with provable error bounds independent of the original system's state dimension.","tokens_in":21085,"tokens_out":6400,"duration_ms":64836,"significance":"If the technical obstacles in Theorem 1 were resolved, the contribution would be significant: it would provide a first provable conversion from a spectral-filter representation to a constant-time recurrent form, with approximation error that does not depend on the original LDS state dimension or effective memory. The algebraic transformation in Section 5.2 is elegant and correct, and the empirical work is substantial: the paper reports filter reconstruction errors around 1e-12, detailed synthetic system-identification comparisons, language-modeling benchmarks with near-parity to the original FlashSTU, and careful runtime measurements showing substantial speedups. The paper also makes a credible effort at reproducibility, with detailed hyperparameters and ablations. However, the central 'provable' claim is currently conditional on an unproven spectral-condition bound, and the algorithm actually deployed in the large-scale experiments is not the one analyzed in the theorem. These gaps materially affect the strength of the main contribution as stated.","major_comments":[{"comment":"The error bound in Theorem 1 contains the factor λmax, the largest singular value of the Penrose-Moore pseudo-inverse of the spectral coefficient matrix M. The proof in Appendix A.3 bounds ||Φ1:k − M†µL|| by λmax·||E||_1 and then invokes the spectral filtering estimate ||E||_1 ≤ c·h·e^{−k/log L}. However, nothing in the argument controls λmax. The paper states in §5.1 that an upper bound is 'not immediate' and appeals to the experimental results in Appendix A.2, but those experiments use k = 48 with a tailored α distribution, not the k = 24, h = 160, L = 131072 configuration used in the language modeling experiments of Table 2. Consequently, the condition h·λmax = O(1), which is essential for the displayed bound to be non-vacuous in the relevant regime, is neither proved nor directly certified for the deployed setting. The abstract's claim of a 'provable method' is therefore overstated. The authors should either prove a high-probability bound on λmax for a well-specified sampling distribution, or explicitly restate the theorem as a conditional guarantee and provide a systematic empirical certificate in the exact regime used in the evaluations.","section":"Section 5.1 / Theorem 1 / Appendix A.3"},{"comment":"The proof of Theorem 1 argues that M is full rank by observing that Φ1:k is an orthogonal basis and that µL(α1:h) is a Vandermonde matrix, concluding that 'both matrices are full rank.' This is a non sequitur: M is the h×k matrix whose rows are the spectral coefficient vectors m_i obtained by projecting each geometric filter µL(α_i) onto the spectral basis. The rank of M is a property of the composed linear map, not of either factor individually, and full rank of the Vandermonde matrix µL(α1:h) does not by itself imply that the coefficient matrix M has full column rank. Since the definition of M† and the equality Φ1:k − M†µL = M†E require M to have full column rank, the proof needs a probability estimate showing that i.i.d. samples α_i from the stated distribution yield a full-column-rank M with high probability. Without such an estimate, the 'w.h.p.' claim in Theorem 1 is unsupported.","section":"Appendix A.3, full-rank argument for M"},{"comment":"The large-scale evaluation in Section 6, including Table 2 and Figure 3, uses Algorithm 3, which generates a large set of 1D-LDS/STU pairs, selects a row subset by reconstruction error, expands it greedily, and then refines fM by gradient descent. Theorem 1 analyzes only Algorithm 2, which directly samples α_i and computes the pseudo-inverse of the coefficient matrix M. The paper does not provide any theoretical guarantee that the output of Algorithm 3 approximates the spectral filters; the reported reconstruction error 1.23e-12 in Appendix A.9 is purely empirical. Thus the 'provable distillation' advertised in the abstract does not cover the algorithm that is actually evaluated on language modeling. The authors should either extend the analysis to Algorithm 3 or explicitly scope the theoretical claim to Algorithm 2 and present Algorithm 3 as a heuristic with strong empirical support.","section":"Appendix A.4 / Algorithm 3 vs. Theorem 1"}],"minor_comments":[{"comment":"The norm in the statement of Theorem 1 is unspecified. The proof in Appendix A.3 uses ||E||_1 and ||M^{-1}||_⋆, so the theorem should state the relevant norm (or explain the compatibility between the two sides) to make the bound well-defined.","section":"Section 5.1 / Theorem 1"},{"comment":"There are several typographical errors: 'spectal' in the discussion after Theorem 1, 'symmetrix' in Section 4.1, 'requries' in Section 3, and the ungrammatical phrase 'this distillation procedure this provides' in Section 1.1.","section":"Throughout"},{"comment":"Line 3 constructs the impulse response as (1 − α)[1, α^i, α^{2i}, ..., α^{(L−1)i}], but the index i used in the vector entries is not defined by the algorithm input; it appears to be a typo for α.","section":"Algorithm 1"},{"comment":"The notation for the matrix of impulse responses is inconsistent: Theorem 1 writes µL(α1, . . . , αh), while Algorithm 2 and Appendix A.3 use µL(α1:h). Unify the notation for readability.","section":"Notation"},{"comment":"The column header 'Runtime with K, T= O(L)' is ambiguous, and the formula 'Standard Conv (T K+ T log T + K^2)k' should use parentheses to make the grouping of terms unambiguous.","section":"Table 1"},{"comment":"The phrase 'first provable method' should be qualified in light of the conditional nature of Theorem 1; the comparison with Laughing Hyena in Section 2 is otherwise fair and appropriately drawn.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The main weaknesses are the unproven λmax condition and the mismatch between the algorithm analyzed in Theorem 1 and the algorithm used in the large-scale experiments. These are fixable within the manuscript's scope by reframing the contribution, adding a formal assumption or proof, and clearly separating the provable component from the heuristic component. The self-citation pattern is not problematic here: the imported spectral filtering lemma comes from prior published work with its own proof. The paper fits the scope of cs.LG and the empirical work is otherwise strong, so I do not see grounds for rejection if the authors address the theoretical gaps directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading. The core idea is new and, to my knowledge, not in the cited literature: expressing STU spectral filters as linear combinations of geometric LDS impulse responses and then inverting the spectral coefficient matrix. The algebraic transformation in Section 5.2 is correct, and the reconstruction errors around 1e-12 in Appendix A.8 are real. The large-scale experiment, distilling a 340M FlashSTU into an LDS with state dimension 160 while preserving accuracy across nine benchmarks, is the kind of concrete evidence that makes me take the practical claim seriously. The synthetic system-identification results also favor SpectraLDS clearly. The soft spot is exactly where the reader put it. Theorem 1's bound is c·lambda_max·h·e^{-k/log L}, and lambda_max is the largest singular value of the pseudo-inverse of M. The paper explicitly says an upper bound is \"not immediate\" and appeals to Appendix A.2. That appendix is empirical, uses k=48, and samples alpha from a distribution that heavily oversamples near ±1. The language experiments use k=24, h=160, and a different sample. So the experiment does not directly certify the constant in the regime where the theorem is deployed. Also, the full-rank argument for M is not immediate: M is a projection of a Vandermonde matrix through the spectral basis, so Vandermonde nonsingularity alone does not imply full column rank. That needs a probability estimate that is not supplied. A second gap: the theorem analyzes Algorithm 2, but the experiments use Algorithm 3, which involves subset selection and gradient refinement. That may be fine as a practical matter, but it means the evaluated model is outside the scope of the proven guarantee. The paper should either prove a version of Theorem 1 for Algorithm 3 or explicitly state that the experimental pipeline is heuristic and the theorem is for the idealized procedure. One more small thing: the LDS layers require float64. That is disclosed in Appendix A.1, and it undercuts the constant-time claim a bit, though the wall-clock numbers still show large speedups. If I were editing, I would send this to peer review. The contribution is novel and the empirical work is substantial, but the abstract's \"provable\" will not survive refereeing without a handle on lambda_max. The authors should either prove a bound for a sampling distribution that covers their experiments, or weaken the claim to conditional-on-lambda_max and let the empirical conditioning be explicit.","headline":"A genuinely new and useful STU-to-LDS distillation with strong empirical results, but the headline 'provable' claim rests on an unproven condition on lambda_max, so the paper needs a serious revision before the theory matches the abstract.","tokens_in":777,"tokens_out":827,"would_cite":true,"duration_ms":18202,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A trained Spectral Transform Unit can be provably converted into an explicit linear dynamical system, preserving accuracy while making generation constant-time per token.","keywords":["spectral transform unit","linear dynamical systems","distillation","state-space models","spectral filtering","sequence prediction","constant-time inference","convex optimization"],"falsifier":"Compute the matrix $M$ from Algorithm 2 for $k=24$ and $h=100,200,\\dots,1000$ using the paper's $\\alpha$ distribution and evaluate $h \\lambda_{\\max}$; if this product grows without bound, or if the reconstruction error of $\\Phi_{1:k}$ by $\\tilde{M} \\mu_L(\\alpha_{1:h})$ exceeds the theorem's right-hand side, the guarantee is vacuous and the reported accuracy would rest on Algorithm 3's least-squares fine-tuning rather than Theorem 1.","tokens_in":20530,"feed_emoji":"⚡","tokens_out":9693,"duration_ms":70513,"temperature":0.7,"pith_summary":"Sequence models built on long convolutions, such as the Spectral Transform Unit (STU), capture long-range memory but pay per-token inference costs that grow with context. This paper claims the first provable method to distill a trained STU layer back into an explicit linear dynamical system (LDS), a plain recurrence of the form $x_t = A x_{t-1} + B u_t$, with an accuracy loss bounded by a term that shrinks as the number of spectral filters grows and does not depend on the original system's hidden dimension. If the construction works, a model trained as an STU can be switched at inference time to a recurrence that runs in constant time and constant memory per token while preserving predictive accuracy. The authors implement the distillation in a 340M-parameter language model and report benchmark scores statistically matched to the original STU, with much faster long-sequence generation.","feed_headline":"Spectral filters provably distilled into a fast recurrent LDS","feed_subtitle":"A trained STU layer becomes an explicit linear dynamical system with matching accuracy and per-token cost that no longer grows with context.","key_machinery":"The load-bearing object is the coefficient matrix $M \\in \\mathbb{R}^{h \\times k}$ built by Algorithm 2: row $i$ stores the coefficients that represent the geometric LDS impulse response $\\mu_L(\\alpha_i)$ in the spectral basis $\\Phi_{1:k}$. The matrix $\\tilde{M} := M^{-1}$, taken as the Penrose-Moore pseudo-inverse since $h \\ge k$, maps geometric filters back to spectral filters. The proof uses a lemma from spectral filtering that each geometric response is approximated by the first $k$ spectral filters with error $e^{-k/\\log L}$, together with the fact that the matrix of geometric responses $\\mu_L(\\alpha_1,\\dots,\\alpha_h)$ is a full-rank Vandermonde matrix. The unproved ingredient is control of $\\lambda_{\\max}$, the largest singular value of the pseudo-inverse, which the paper does not bound theoretically but reports empirically to decay as $h$ grows.","core_discovery":"The central claim is that the $k$ spectral filters of an STU can be expressed, up to a controlled error, as a linear combination of impulse responses of $h$ one-dimensional geometric LDS filters. Concretely, Theorem 1 states that for $h \\ge k$, Algorithm 2 returns a matrix $\\tilde{M}$ such that $\\|\\Phi_{1:k} - \\tilde{M} \\mu_L(\\alpha_1,\\dots,\\alpha_h)\\| \\le c \\lambda_{\\max} h e^{-k/\\log L}$, where $\\Phi_{1:k}$ are the spectral filters, $\\mu_L(\\alpha_i) = (1-\\alpha_i)(1, \\alpha_i, \\alpha_i^2, \\dots, \\alpha_i^{L-1})$ are geometric impulse responses, and $\\lambda_{\\max}$ is the largest singular value of the Penrose-Moore pseudo-inverse of the $h \\times k$ coefficient matrix $M$ that expresses each geometric filter in the spectral basis. Because convolving with a geometric filter is exactly the impulse response of a scalar LDS, this identity converts the spectral filters into a diagonal LDS with state dimension $h$, and the alternating (negative-eigenvalue) filters are handled by replacing $A$ with $-A$. The paper argues that the $e^{-k/\\log L}$ spectral approximation error is inherited from spectral filtering theory, so the guarantee is independent of the hidden dimension of the original system.","pith_inferences":["Editorial: the theorem's practical force rests on $h \\lambda_{\\max} = O(1)$, which the paper verifies only empirically in Appendix A.2; turning that observation into a proof for the paper's $\\alpha$ sampling distribution would make the reduction fully parameter-free.","Editorial: the construction is limited to symmetric $A$, i.e. real eigenvalues; replacing the geometric dictionary by complex exponentials would extend the same inversion to oscillatory modes and is the natural next test.","Editorial: because the spectral basis comes from a fixed universal Hankel matrix, the expensive part of the distillation can be computed offline, suggesting that any convolution filter lying near the span of geometric decays could be distilled this way, not only STU filters."],"forward_implications":["For a fixed distilled state dimension ($h \\approx 160$ in the language experiments), generation drops to $O(1)$ time and $O(1)$ memory per token, independent of sequence length, replacing convolution or FFT-based inference.","The approximation error $e^{-k/\\log L}$ means accuracy is provably retained up to an exponentially-in-$k$ spectral error term that does not depend on how large or how marginally stable the original LDS hidden state was.","The same machinery yields an LDS-to-LDS reduction: learn any high-dimensional symmetric LDS through its convex STU parameterization, then distill to a low-dimensional LDS with bounded error, giving a practical state-dimension compression (Section 5.3).","Because training stays in the convex STU representation, the method inherits the STU's training stability for systems with eigenvalues close to 1, and the synthetic experiments show much lower loss and much faster convergence than a gradient-trained LDS baseline."],"supporting_citations":[{"why":"Supplies the spectral filtering method: top-$k$ eigenvectors of the Hankel matrix approximate any geometric response $\\mu(\\alpha)$ with $e^{-k/\\log L}$ error, which is the foundation of the distillation proof.","marker":"[14]"},{"why":"Introduces the Spectral Transform Unit and its convex spectral parameterization, which is the object being distilled.","marker":"[2]"},{"why":"Its Lemma 11.3 is cited in the proof of Theorem 1 for the existence of coefficients $m_j$ approximating each geometric filter in the spectral basis.","marker":"[13]"},{"why":"Establishes the $e^{-k/\\log L}$ length-generalization and spectral approximation bound that Theorem 1 inherits.","marker":"[25]"},{"why":"Defines FlashSTU and the tensor-dot STU variant used in the language-model distillation experiments, supplying the baseline model and its benchmark numbers.","marker":"[22]"},{"why":"Prior convolution-to-LDS distillation that lacks formal guarantees, providing the comparison point for the paper's provable result.","marker":"[26]"},{"why":"Cited to note that for $h \\approx k$ the pseudo-inverse condition number can be exponentially large, motivating the overparameterized regime $h \\gg k$.","marker":"[4]"},{"why":"FutureFill inference acceleration algorithm used as a runtime baseline in the generation benchmarks.","marker":"[1]"}],"fun_headline_variants":["First provable LDS distillation with constant-time inference","Match accuracy, cut inference cost: SpectraLDS","Spectral to recurrent: provable distillation","Convex path to provable LDS distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bound in Theorem 1 only bites if $h \\lambda_{\\max}$ stays $O(1)$, meaning the matrix expressing geometric filters in the spectral basis has a pseudo-inverse with a small largest singular value; the paper proves this nowhere and supports it only with experiments in Appendix A.2.","fun_headline_variants_meta":{"raw":{"variants":["First provable LDS distillation with constant-time inference","Match accuracy, cut inference cost: SpectraLDS","Spectral to recurrent: provable distillation","Convex path to provable LDS distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1620,"prompt_tokens":959,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":600}},"tokens_in":575,"tokens_out":661,"duration_ms":5046,"temperature":1.0,"reasoning_tokens":600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:39:16.287196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the matrix $M$ from Algorithm 2 for $k=24$ and $h=100,200,\\dots,1000$ using the paper's $\\alpha$ distribution and evaluate $h \\lambda_{\\max}$; if this product grows without bound, or if the reconstruction error of $\\Phi_{1:k}$ by $\\tilde{M} \\mu_L(\\alpha_{1:h})$ exceeds the theorem's right-hand side, the guarantee is vacuous and the reported accuracy would rest on Algorithm 3's least-squares fine-tuning rather than Theorem 1.","supporting_citations":[{"cited_title":"Spectral state space models, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces the Spectral Transform Unit and its convex spectral parameterization, which is the object being distilled."},{"cited_title":"Provable length generalization in sequence prediction via spectral filtering, 2024","cited_arxiv_id":null,"evidence_quote":"Establishes the $e^{-k/\\log L}$ length-generalization and spectral approximation bound that Theorem 1 inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FlashSTU and the tensor-dot STU variant used in the language-model distillation experiments, supplying the baseline model and its benchmark numbers."},{"cited_title":"Fu, Hermann Kumbong, Rom N","cited_arxiv_id":null,"evidence_quote":"Prior convolution-to-LDS distillation that lacks formal guarantees, providing the comparison point for the paper's provable result."},{"cited_title":"The condition number of real vandermonde, krylov and positive definite hankel matrices","cited_arxiv_id":null,"evidence_quote":"Cited to note that for $h \\approx k$ the pseudo-inverse condition number can be exponentially large, motivating the overparameterized regime $h \\gg k$."}],"review_version":1}