{"id":"01eb6723-d711-482e-bc10-2350ddc3fade","arxiv_id":"2509.04226","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SSM/Mamba long-range dependency decays exponentially with the time gap by construction; a proposed interaction-based hidden state update can break this decay, but its proven stability covers only a restrictive special case.","lead":"The paper defines long-range memory in sequence models as how much a hidden state changes when a past input is tweaked, and shows that the memory of Mamba-style state-space models fades exponentially with distance while attention is not so constrained. It then proposes an attention-inspired state update for state-space models with a special-case stability proof, but reports no experiments on the new model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4 does not prove stability even under its own special-case assumptions: the tail bound controls only one eigendirection, while other eigenvalues may be 1 and yield an unbounded random walk.","rationale":"The reader's weakest assumption is that Theorem 4's stability guarantee is restricted to a simultaneous-diagonalizability special case and, even then, bounds a product of eigenvalues rather than the full hidden state recursion. My analysis sharpens this into a concrete logical gap: the theorem's conclusion does not follow from its proof even under the theorem's own assumptions, and a counterexample (λ_1=1) satisfies every stated condition while producing an unbounded hidden state component. This directly undermines the abstract's 'prove its stability' and the conclusion's 'will not approach infinity', which are the paper's main theoretical selling points. The authors' future-work section concedes that general stability is unknown and that no benchmark evaluation exists, corroborating that the central contribution is not yet supported. The LRD comparison and the algebraic derivation of the new unrolled form (Theorems 2-3) appear largely correct, but they alone do not justify the paper's positive claims about the new model. Since the reader's REJECT verdict is based on these overclaimed stability and evaluation gaps, my concern reinforces that verdict rather than altering it. I therefore recommend UNCHANGED.","tokens_in":14141,"tokens_out":9015,"duration_ms":86053,"concrete_test":"Instantiate the exact special case of Section 4.2 with H=2: \\bar A = diag(1, 0.5), P=I, λ_H=0.5, γ=0.499 (so λ_H+γ=0.999<1), G and W aligned with the second eigenvector, B=[1,0]^T, x_t i.i.d. N(0,1), Δ=1, h_0=0. Run the recursion (14) for t=1..100000 and record the first component h_t[1]=Σ_{i=1}^t x_i. Its variance grows linearly; max |h_t[1]| grows without bound, and the empirical probability that |h_t| exceeds any fixed M does not vanish. If the authors instead intend λ_i<1, run the same with \\bar A=diag(0.9999,0.5) and show Theorem 4 must still be supplemented by a bound on the sum over i; otherwise the theorem's stated assumptions are insufficient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central positive claim is that the new interaction-based recursion (Eq. 14) 'proves its stability' (abstract) and that hidden states 'will not approach infinity' (Conclusion). The only support is Theorem 4, whose proof bounds the scalar product ∏(λ_H+γx_i^2) in a single eigendirection under simultaneous diagonalizability (Eq. 17–18). Even granting all special-case assumptions (scalar inputs, Δ=1, G,W aligned to a common eigenvector), the theorem's tail bound does not control the full hidden state. In Eq. (18), each eigen-direction has its own factor: λ_1^t, ..., ∏(λ_H+γx_i^2). The paper only restricts λ_i ≤ 1 ('not exceed 1'), so λ_1=1 is allowed. If λ_1=1, the first component of h_t accumulates Σ B_1 x_i, an unbounded random walk, contradicting 'will not approach infinity'. The proof also drops the P,P^T factors and never bounds the sum over inputs; a union bound over suffixes of the product is absent. The theorem's condition λ_H+γ<1 only tames the last eigendirection. The paper's own Future Work admits general stability is unknown and evaluation is missing. Thus the headline stability guarantee is not established, and in a permitted parameter regime it is false.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a derivative-based definition of long-range dependency (LRD), LR D(t+k,t)=dh_{t+k}/dx_t, and uses it to compare SSM/Mamba and transformer models. It claims SSM/Mamba LRD decays exponentially with time gap while transformer LRD is not constrained to exponential decay, proposes a new state update with an 'interaction' term (Eqs. 13-14), gives unrolled forms (Theorems 2-3), provides a simulation of LRD norms (Fig. 1), and attempts to prove stability of the new recursion under Gaussian inputs in a special case (Theorem 4). The conclusion states the new model 'addressed the fundamental limitation' of SSM/Mamba and that stability guarantees hidden states will not approach infinity.","tokens_in":14433,"tokens_out":8793,"duration_ms":93260,"significance":"If the results were correct, the paper would contribute a formal framework for comparing sequence-model memory and a new linear-time architecture with attention-like flexibility. The LRD definition is clear and the unrolling algebra in Theorem 2 is correct. The proposed interaction term is a reasonable idea worth exploring. However, the paper's central stability theorem does not establish hidden-state stability even under its own assumptions, the transformer LRD formula contains a missing term, the exponential-decay bound in Corollary 1 requires an unstated normality condition, and the claimed practical advantage is supported neither by theory nor by any downstream evaluation. The theoretical contribution is therefore not yet at the level claimed by the abstract and conclusion.","major_comments":[{"comment":"The bound ||exp(τ A)||_2 ≤ exp(λ_1 τ) is not valid for an arbitrary matrix A. It holds only when A is normal or, more generally, when the eigenbasis is well conditioned. SSM/Mamba transition matrices, including HiPPO-initialized A, are not guaranteed to be normal. Thus the per-time-step exponential decay claim as stated is incorrect; at most an asymptotic spectral-radius statement can be made. The proof needs an explicit normality or diagonalizability assumption, or a different norm.","section":"Section 3.1, Corollary 1, Eq. (11)"},{"comment":"Theorem 4 bounds only the product ∏_{i=1}^t (λ_H + γ x_i^2), a single diagonal factor of the transition product. It does not bound the full hidden state. Under the paper's own assumption λ_i ≤ 1, λ_1 = 1 is allowed. In that case the first component in Eq. (18) is h_{0,1} + Σ_{i=1}^t (P^T B)_1 x_i (plus similar terms), which, for generic B, is a random walk with unbounded variance. The theorem therefore does not prove that hidden states 'will not approach infinity'; indeed the stated conclusion is false in an allowed parameter regime. A valid stability proof must require λ_i < 1 for all i and must control all eigen-directions and the sum over inputs, not just one product.","section":"Section 4.2, Theorem 4 and Eq. (18)"},{"comment":"The stability theorem is proved only under the assumption that G x_t x_t^T W^T is simultaneously diagonalizable with \\bar A, which forces G and W to be scalings of the same eigenvector of \\bar A. This is a very restrictive structural condition that no trained or randomly initialized model will generally satisfy. The abstract's phrase 'prove its stability under a standard Gaussian distribution of the input data' omits this condition entirely. Section 5 itself concedes that general stability is 'unknown'. The paper should either restrict the headline claim explicitly to the special case or provide a stability analysis that does not depend on eigenvector alignment.","section":"Section 4.2, simultaneous diagonalizability assumption and abstract"},{"comment":"The derivative formula for transformer LRD is missing a term. Since h_{t+k} = Σ_{i=1}^{t+k} w(i,t+k) W_V x_i, the derivative with respect to x_t includes Σ_{i=1}^{t+k} (∂w(i,t+k)/∂x_t) W_V x_i plus the direct term w(t,t+k)W_V. The theorem's summation stops at t+k−1, omitting the i=t+k term (∂w(t+k,t+k)/∂x_t) W_V x_{t+k}. This is a concrete error in a central formula; it should be corrected even though the qualitative flexibility claim may survive.","section":"Section 3.2, Theorem 1, Eq. (12)"},{"comment":"The introduction states that the proposed new model 'outperforms SSM', and the conclusion says the model 'addressed the fundamental limitation' of Mamba. However, the only supporting evidence is a single simulated LRD-norm curve in Figure 1 with no parameter details or error bars, and no task-based experiments. Section 5 explicitly acknowledges that 'evaluation using existing benchmarking datasets is needed'. The unsupported performance claim should be removed or substantiated with experiments.","section":"Introduction, Section 4.1, and Section 5"}],"minor_comments":[{"comment":"The notation 'λ_1, the largest eigenvalue of A' is ambiguous for non-symmetric A with complex eigenvalues; the relevant quantity is the largest real part. The phrase 'non-positive if the SSM is stable' should also be clarified for discrete vs. continuous stability.","section":"Section 3.1"},{"comment":"The theorem is stated before the scalar-input assumption is introduced, but the derivative 2 x_t G W^T in Eq. (16) is only valid for scalar x_t. For vector inputs the derivative of G x_t x_t^T W^T is a third-order tensor. The scalar assumption should be stated before Theorem 3 or the theorem should be restricted accordingly.","section":"Section 4, Theorem 3"},{"comment":"The simulation in Figure 1 needs details: matrix dimensions, distributions for A, B, G, W, the value of Δ, the random seed, and error bars over multiple trials. As presented it is a single-instance illustration, not an empirical validation.","section":"Figure 1"},{"comment":"The lower-tail argument for log(c+γX^2) contains a somewhat informal comparison involving the chi-square CDF and constants; it would be clearer to state the exact sub-exponential parameters and to avoid reusing 'c' for both a constant and the eigenvalue λ_H.","section":"Appendix, Lemma 1 proof"},{"comment":"There are several typographical errors ('fundemantal', 'srong', 'LRDtransformerpt+k,tq') and inconsistent use of Δ_t vs. Δ_t=1. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper is an early-version theoretical claim with a central stability theorem that is false under the assumptions it explicitly permits. The transformer LRD formula also has a missing term, and the promised practical advantage is unaudited by any experiments. Even as a theory-first submission, the main claim would need substantial new analysis (full-state tail bounds, strict eigenvalue assumptions, and a corrected transformer formula) before it could be considered publishable. I would not encourage a quick resubmission without those repairs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Phil — quick read of 2509.04226. The paper does three things: define LRD as the norm of dh_{t+k}/dx_t, show that SSM/Mamba LRD decays exponentially, and propose an interaction-modulated SSM (Eq. 14) intended to escape that decay. The comparison results are correct but not new: linear RNNs/SSMs having exponential memory is exactly what [36] establishes, and attention's data-dependent weights are standard. The unrolled formulas and LRD derivatives check out. The interaction update is a genuinely nice idea — a rank-1 perturbation of the transition matrix, in the spirit of attention's value-times-key interaction.\n\nThe soft spot is the advertised stability guarantee. Theorem 4 bounds the product ∏(λ_H+γx_i²) in one eigendirection, under the simultaneous-diagonalizability assumption. Even granting that special case, the proof only controls the last diagonal entry of Eq. (18). Other eigenvalues are allowed to be 1; with λ_1=1, the first component of h_t accumulates Σ B_1 x_i, an unbounded random walk. The conclusion that hidden states 'will not approach infinity' doesn't follow. This is not a minor gap — it's the central positive claim. The paper's own future-work section admits general stability is open, but even the special case isn't fully proven. The empirical part is also weak: one unseeded simulation, no training, no benchmark, so 'outperforms SSM' in the intro is unsupported.\n\nOn balance: the paper is a useful, correct-if-superficial derivation of known facts plus an interesting but unvalidated model. It is not publishable as is. But it's not incoherent, and the failure is instructive — a referee could tell the authors exactly how to fix it: prove a true bound on the full state or drop the stability claim and provide experiments. I would send it out, not desk-reject, precisely because the flaw is subtle and fixable. For your own work, I wouldn't cite it yet; the idea might be worth building on if they sort out the stability.\n\nAll the best.","headline":"The LRD comparison restates known linear-RNN behavior and the new interaction-based SSM is a plausible idea, but Theorem 4 does not prove stability and the abstract/conclusion overstate it.","tokens_in":14980,"tokens_out":4176,"would_cite":false,"duration_ms":40646,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"State-space models provably forget the past exponentially; transformers need not—and a new interaction term lets SSMs do the same.","keywords":["long-range dependency","state-space models","Mamba","transformers","attention","exponential decay","hidden-state stability","Gaussian tail bounds"],"falsifier":"Train or randomly instantiate a Mamba and compute the Jacobian norms $\\left\\| \\frac{d h_{t+k}}{d x_t} \\right\\|$ over many random input sequences; if any sequence shows a non-exponential decay, the universal bound in Corollary 1 fails in practice. Then run the proposed update with $G$ and $W$ randomly drawn, not aligned with any eigenvector of $\\bar A$, streaming standard-Gaussian scalar inputs with $\\lambda_H + \\gamma < 1$; if $\\| h_t \\|$ diverges, the stability certificate does not cover the generic configuration.","tokens_in":13919,"feed_emoji":"📉","tokens_out":11331,"duration_ms":105198,"temperature":0.7,"texified_at":"2026-08-05T20:25:02.770107+00:00","pith_summary":"The paper sets out to explain, from theory, why state-space models (SSMs) such as Mamba and transformers differ in long-range dependency. Using a definition of long-range dependency as the derivative of a later hidden state with respect to an earlier input, it proves that SSM/Mamba sensitivity decays exponentially with the time gap, because each step multiplies by a transition matrix whose eigenvalues are at most one. A single attention layer, by contrast, is not forced into exponential decay, since attention weights can grow or reorganize with distance. To combine SSM efficiency with transformer-like flexibility, the paper adds an input-dependent interaction term to the SSM recursion, shows empirically that the new recursion removes the exponential decay, and proves a probabilistic stability bound under Gaussian inputs for a special eigenvector-aligned case.","texify_model":"deepseek-v4-flash","texify_usage":{"total_tokens":10525,"prompt_tokens":843,"completion_tokens":9682,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":843,"completion_tokens_details":{"reasoning_tokens":8850}},"feed_headline":"Mamba memory decays exponentially; transformer memory is flexible","feed_subtitle":"A derivative-based measure shows why state-space models struggle with long memory—plus an interaction term to fix it.","key_machinery":"The paper's operative object is the sensitivity matrix $\\mathrm{LRD}(t+k,t) = \\frac{d h_{t+k}}{d x_t}$ (Eq. 8), a discrete analog of the RNN memory function. The exponential-decay proof uses the identity that Mamba's transition matrices are matrix exponentials $\\exp(\\Delta_i A)$, so a time-ordered product collapses to $\\exp((\\sum \\Delta_i) A)$, whose spectral norm is controlled by the largest eigenvalue $\\lambda_1$ of $A$. The proposed architecture centers on the rank-one input-dependent update $\\bar A_t + G x_t x_t^T W^T$ inserted into the SSM recursion; the stability analysis in Theorem 4 reduces to bounding products of scalar factors $\\lambda_H + \\gamma x_i^2$ for i.i.d. standard Gaussian $x_i$, using sub-exponential tail bounds on sums of $\\log(\\lambda_H + \\gamma x_i^2)$.","core_discovery":"The central claim is that long-range dependency, formalized as the norm of $\\mathrm{LRD}(t+k,t) = \\frac{d h_{t+k}}{d x_t}$, decays exponentially with $k$ for SSM/Mamba models but is not constrained to do so for transformers. Corollary 1 proves the decay: with $\\lambda_1$ the largest eigenvalue of $A$, $\\| \\mathrm{LRD}(t+k+1,t) \\|_2 \\leq \\exp(\\lambda_1 \\Delta_{t+k+1}) \\| \\mathrm{LRD}(t+k,t) \\|_2$, so the memory of a past input is suppressed at a rate set by the transition matrix. For transformers, Theorem 1 expresses LRD as a sum of an attention-weight derivative term and the direct weight $w(t,t+k) W_V$, which can increase with distance when interactions strengthen, so no exponential cap applies. The paper's proposed recursion $h_t = (\\bar A_t + G x_t x_t^T W^T)$ h_{","pith_inferences":["The exponential-decay bound is proved for the single linear recurrence; stacked SSMs with nonlinear layers could in principle combine layers to extend memory, so the bound should be read as a property of one transition, not of a full deep network.","Because the stability certificate requires G and W to be scalings of the same eigenvector of \\bar A, a practical design would be to enforce or softly regularize that alignment during training, or to prove a bound for random G,W; neither is in the paper.","The derivative-based measure captures local sensitivity, not information storage; on tasks where recall means retrieving a specific stored token, a mutual-information or retrieval-accuracy measure might rank Mamba and transformers differently than the LRD norm does."],"forward_implications":["A single SSM/Mamba layer has a provable ceiling on how far back an input can influence hidden states: every additional time step multiplies the influence by at most exp(λ_1 Δ), so long-range tasks cannot be solved by simply lengthening the memory of one layer.","Transformers can in principle express LRD that stays flat or grows with distance, so their advantage on long-range tasks is not just empirical; it is a structural difference in how attention weights can reorganize.","Hybrid transformer-SSM architectures are a natural consequence of the theory: SSM supplies fast, stable medium-range memory, and attention supplies unconstrained long-range interactions.","For the proposed interaction update, stability is not automatic; the paper's condition λ_H+γ<1 together with the Gaussian tail bound gives a high-probability guarantee that hidden states do not diverge, but only in the special case analyzed."],"supporting_citations":[{"why":"Supplies the attention formula (softmax(QK^T)V) whose weights define transformer LRD in Theorem 1.","marker":"[3]"},{"why":"Supplies Mamba's selective state-space recursion with input-dependent Δ_t, the model class whose exponential decay is proved in Corollary 1 and which the new update extends.","marker":"[22]"},{"why":"Supplies the RNN memory-function concept that the paper's derivative-based LRD definition generalizes.","marker":"[36]"},{"why":"Supplies the sub-exponential random-variable characterization used to prove the stability tail bound (Theorem 4).","marker":"[40]"},{"why":"Supplies the sub-exponential tail bound applied to the sums log(λ_H+γx_i^2) in the proof of Theorem 4.","marker":"[41]"}],"fun_headline_variants":["Proof: SSM memory decays exponentially, attention doesn't","Mamba's long memory is exponentially capped; attention is free","Why transformers can hold long memory but SSMs can't","New derivative measure reveals SSM memory fades fast","Exponential decay in Mamba vs flexible attention memory"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper's comparison assumes long-range dependency is best measured by how much a later hidden state changes when an earlier input is nudged; if memory should instead be measured by stored information or output accuracy, the exponential-decay result and the flexibility comparison do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Proof: SSM memory decays exponentially, attention doesn't","Mamba's long memory is exponentially capped; attention is free","Why transformers can hold long memory but SSMs can't","New derivative measure reveals SSM memory fades fast","Exponential decay in Mamba vs flexible attention memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1167,"prompt_tokens":785,"completion_tokens":382,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":302}},"tokens_in":529,"tokens_out":382,"duration_ms":4177,"temperature":1.0,"reasoning_tokens":302,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:16:06.842947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or randomly instantiate a Mamba and compute the Jacobian norms $\\left\\| \\frac{d h_{t+k}}{d x_t} \\right\\|$ over many random input sequences; if any sequence shows a non-exponential decay, the universal bound in Corollary 1 fails in practice. Then run the proposed update with $G$ and $W$ randomly drawn, not aligned with any eigenvector of $\\bar A$, streaming standard-Gaussian scalar inputs with $\\lambda_H + \\gamma < 1$; if $\\| h_t \\|$ diverges, the stability certificate does not cover the generic configuration.","supporting_citations":[{"cited_title":"State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory","cited_arxiv_id":null,"evidence_quote":"Supplies the RNN memory-function concept that the paper's derivative-based LRD definition generalizes."},{"cited_title":"Wainwright","cited_arxiv_id":null,"evidence_quote":"Supplies the sub-exponential tail bound applied to the sums log(λ_H+γx_i^2) in the proof of Theorem 4."}],"review_version":1}