{"id":"9f73ed30-a75c-439d-ad11-a3dee869246c","arxiv_id":"2505.08516","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AGF replaces softmax attention with a learned singular-value-domain graph filter that runs in O(n d^2), and reports moderate accuracy improvements on UEA and LRA benchmarks.","lead":"A new linear-complexity attention mechanism, AGF, learns a graph filter from data by generating singular vectors and values instead of computing softmax attention. It reports accuracy gains over several linear transformers on time series and long-sequence benchmarks, but its theoretical justification has gaps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's proof assumes singular values of the softmax attention matrix obey λ1=1 and |λi|<1 (Appendix E, Eq. 17), but these are undirected spectral facts; row-stochastic directed attention matrices can have largest singular value >1, so the low/high-pass claim is unsupported.","rationale":"The reader correctly identified the theoretical scaffolding as the weak point. I focus on Theorem 2 because it is the load-bearing bridge between 'self-attention is low-pass' and 'AGF learns high frequencies.' The proof in Appendix E explicitly imports undirected spectral facts (Von Luxburg) for singular values of a directed row-stochastic matrix; a 2x2 counterexample shows the bound is false. This invalidates both the low-pass and high-pass conclusions as stated and breaks the claimed connection to SVD-based GSP. The empirical gains in Tables 1-2 are still plausible as an empirical architecture, and the runtime complexity O(nd^2) is straightforward, so I would not reject outright; the verdict stays conditional pending a corrected theorem (e.g., stated for the generated sigmoid-bounded Σ, not for the attention matrix) or a trimmed claim. Eq. (11) is a less central but concrete bug that should also be fixed. No code is provided, so the empirical results cannot be independently checked; that supports conditions rather than acceptance.","tokens_in":20526,"tokens_out":6734,"duration_ms":67757,"concrete_test":"Take Z with i.i.d. standard normal entries, M=softmax(Z) row-wise, and compute singular values of M for n=10 and n=100; check whether max singular value exceeds 1. Also compute singular values of the 2x2 counterexample above. If any exceed 1, the premise of Theorem 2 fails. Additionally, print the shapes in Eq. (11): if V is n×d, V V^T - I_d raises a size mismatch or silently broadcasts incorrectly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix E's proof of Theorem 2 uses Von Luxburg's undirected spectral results to assert λ1=1 and |λ_i|<1 for the singular values of the row-stochastic attention matrix \\bar A. This is false for directed row-stochastic matrices: e.g., [[0.5,0.5],[0.9,0.1]] has singular values ≈1.089 and ≈0.368. Softmax attention matrices are generally non-symmetric row-stochastic, so their SVD need not satisfy the premise. The paper itself (Sec. 3.2) requires ||\\bar A||≤1 for the Maskey et al. SVD framework, a condition not guaranteed and not verified. Consequently, Theorem 2 does not establish that sign-changing θ coefficients produce high-pass behavior, and the central motivation that AGF 'incorporates both low and high-frequency information' lacks theoretical support. A separate well-posedness issue appears in Eq. (11): with V(X)^⊺ ∈ R^{d×n}, V(X) ∈ R^{n×d}, so V V^T is n×n, but it is compared to I_d; the orthogonality penalty is dimensionally invalid as written (should be V^T V or the shapes corrected).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Attentive Graph Filter (AGF), a linear-complexity self-attention layer that abandons the explicit softmax attention map and instead generates, via neural networks, matrices U(X), Sigma(X), and V(X) intended to play the roles of singular vectors and singular values. The singular values are modulated by a learnable Jacobi-polynomial graph filter, and an orthogonality regularizer is added to encourage U and V to behave like SVD factors. The authors motivate the method by two theoretical claims: that softmax self-attention is a low-pass graph filter (Theorem 1), and that a polynomial filter with sign-changing coefficients can pass high-frequency information in the singular-value domain (Theorem 2). The empirical sections report results on UEA time-series classification, Long Range Arena, ImageNet classification with DeiT-small, ablation studies, runtime/memory comparisons, and an analysis of over-smoothing in deep transformers.","tokens_in":20862,"tokens_out":4667,"duration_ms":49218,"significance":"If the method and its motivation were sound, AGF would be a useful drop-in linear attention layer, and the paper would contribute a new GSP-based design space for efficient transformers. The empirical evaluation is a genuine strength: it covers multiple benchmarks (UEA, LRA, ImageNet), includes ablations on the polynomial type, activation function, filter variant, hyperparameters, and report runtime/memory and Wilcoxon signed-rank tests. The paper states a reproducibility appendix, although no code is provided. However, the two theoretical pillars of the paper have concrete errors, and one equation is dimensionally invalid; the central 'low/high-frequency' motivation is therefore not established as written. The empirical results may still stand, but the supporting theory needs substantial correction or reframing.","major_comments":[{"comment":"Theorem 1 states that softmax self-attention acts as a low-pass filter, but the statement and proof concern lim_{t->infinity} M^t x, i.e., the repeated application of the row-stochastic matrix M. In a Transformer layer, the graph filter H = \\bar A is applied once to the value signal (Eqs. (1) and (3)), not infinitely many times. The asymptotic dominance of the Perron eigenvalue under repeated application does not imply that a single application attenuates high-frequency components relative to low-frequency components; for a single application the HFC/LFC ratio can behave differently. This theorem is the stated motivation for the entire method, so the claim that vanilla self-attention suppresses high-frequency information is not supported.","section":"Section 3.1, Theorem 1 (Appendix D)"},{"comment":"The proof of Theorem 2 assumes that the singular values of the attention matrix satisfy \\lambda_1 = 1 and |\\lambda_i| < 1 for i >= 2, citing Von Luxburg's undirected spectral-clustering results. These facts hold for eigenvalues of a row-stochastic matrix (spectral radius 1, all eigenvalues in the unit disk), but they do not hold for singular values of a non-symmetric row-stochastic matrix. For example, the row-stochastic matrix [[0.5, 0.5], [0.9, 0.1]] has largest singular value approximately 1.09. Softmax attention matrices are generally non-symmetric, and Section 3.2 itself requires ||\\bar A|| <= 1 for the Maskey et al. SVD framework, a condition that is neither guaranteed for softmax attention nor verified in the paper. Consequently, the low-pass and high-pass conclusions in Theorem 2, including the ratio |(1+\\alpha)/(1+\\alpha \\lambda_i)| > 1 in Eq. (21), are not justified for the singular values of a directed attention matrix. The paper's central claim that AGF 'incorporates both low and high-frequency information' lacks theoretical support.","section":"Section 3.6, Theorem 2 (Appendix E, Eq. (17))"},{"comment":"The orthogonality regularizer is dimensionally invalid. With V(X) in R^{n x d} as defined in Eq. (10), the product V(X) V(X)^T is an n x n matrix, but Eq. (11) compares it to I in R^{d x d}. The analogous term should be V(X)^T V(X) to yield a d x d matrix, or the shapes must be corrected consistently. As written, the regularizer does not enforce the intended row-orthogonality of V and breaks the SVD interpretation that U, Sigma, and V are factors of a single attention matrix.","section":"Section 3.4, Eq. (11)"},{"comment":"The reported hyperparameter search ranges do not cover the hyperparameters listed as best. Appendix H.1 states that K is grid-searched over {2, 3, 4, 5} for UEA, but Table 10 reports best values K=6 (EthanolConcentration), K=6 (Heartbeat), K=9 (FaceDetection, SelfRegulationSCP1), and K=10 (UWaveGestureLibrary). Similarly, Appendix H.2 states K in {2, 3, 4, 5} for LRA, but Table 11 reports K=7 for Image. This inconsistency makes it impossible to reproduce the reported results from the stated protocol and needs to be corrected or explained.","section":"Appendix H.1/H.2 vs Tables 10/11"}],"minor_comments":[{"comment":"The theorem uses both N and n for the ambient dimension ('For all x in R^N' vs. M in R^{n x n}), and the ratio ||HFC[M^t(x)]||_2 / ||LFC[M^t(x)]||_2 is written with the limit placed slightly ambiguously; the definitions of LFC and HFC should be restated in the appendix to match the statement.","section":"Theorem 1 statement"},{"comment":"The text says the polynomial-type ablation results are shown in Table 18, but Table 18 displays the activation-function ablation; the polynomial-type results appear in Table 19. The cross-reference is wrong.","section":"Appendix L.2"},{"comment":"There are repeated typos in the proof, including 'hgih-pass' instead of 'high-pass' and 'grah filter' instead of 'graph filter'; these should be corrected.","section":"Appendix E"},{"comment":"The manuscript does not describe how the filter response in Figure 3(a) is computed from the DeiT and DeiT+AGF models; a precise definition (which matrix or operator is analyzed, and over which inputs) is needed for the over-smoothing claim to be verifiable.","section":"Figure 3"},{"comment":"The reproducibility statement in Appendix A promises dataset descriptions and best hyperparameters, but no code release or seed information is mentioned; providing code or at least detailed training configurations would strengthen the empirical contribution.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical evaluation is extensive and the AGF method may well be a useful linear attention layer, but the theoretical framing is currently the paper's main weakness. I would not reject solely on the theory, since the empirical comparisons are independent; however, the authors should either repair Theorem 2 and Eq. (11) or explicitly demote the low/high-frequency claims to conjectures supported only by experiments. The hyperparameter mismatch in Appendix H is a concrete reproducibility error that must be fixed. The manuscript is likely salvageable as a major revision, with the theoretical sections substantially rewritten or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2505.08516. First, the empirical claim looks real: AGF, a learnable-SVD linear attention with a Jacobi polynomial filter, gets 60.1 on LRA (vs 59.0 for Nyströmformer) and 75.1 on UEA, with consistent margins across datasets and sensible ablations. Second, the theory in the paper does not hold as written: Theorem 1 is about the limit of repeated application M^t, whereas self-attention applies the filter once per layer; Theorem 2's proof imports undirected spectral facts (λ1=1, |λi|<1) that do not apply to singular values of directed row-stochastic matrices; and Eq. (11) compares V(X)V(X)^T, which is n×n under their own definitions, to the d×d identity. These are not cosmetic. The paper's central motivation — that AGF leverages high-frequency information because sign-changing coefficients yield high-pass behavior — depends on Theorem 2, and the proof as given does not establish it.\n\nWhat is genuinely new: the specific combination of parameterized SVD with a learnable polynomial filter in the singular value domain. Singularformer uses parameterized SVD but no graph filter; the authors' prior NeurIPS paper has the graph-filter view of attention but not the singular-value parameterization. The empirical work is the real contribution: benchmark coverage includes UEA, LRA, and ImageNet, with runtime and memory numbers, ablations on polynomial order, Jacobi parameters, and activation, plus a Wilcoxon signed-rank test against the UEA baselines. That is more than most linear-attention submissions do.\n\nThe soft spots, in order of severity. First, the theoretical scaffold needs repair or removal. Theorem 1 can be fixed by stating it as a property of the attention operator's stationary behavior, but as written it does not prove that one application of attention is low-pass. Theorem 2's proof applies Von Luxburg's undirected spectral facts to singular values of a directed matrix; a simple counterexample — [[0.5,0.5],[0.9,0.1]] has largest singular value about 1.089 — shows the premise fails. The paper itself requires ||Ā||≤1 to invoke Maskey et al., and that condition is never verified. Second, Eq. (11) has a dimension mismatch: with V(X)^T in R^{d×n}, V(X) in R^{n×d}, V V^T is n×n, but the identity is I_d; the regularization should be on V^T V or the shapes corrected. This is minor to fix but needs stating. Third, no code is released, which matters for a method whose value is mainly empirical. The paper acknowledges the state-space model comparison gap in its conclusion; that is a fair limitation but not the main issue.\n\nWho is this for? People working on efficient attention who want a concrete alternative to Nyströmformer or Primalformer that reads as a drop-in replacement. The paper deserves a serious referee — the empirical claims are extensive and reproducible in principle — but it should come back with the theory either fixed or explicitly downgraded to intuition. I would accept it as a conditional, not reject it.","headline":"A plausible linear-attention variant with honest benchmark gains, but the theoretical story has two load-bearing errors and needs a rewrite.","tokens_in":21366,"tokens_out":2249,"would_cite":false,"duration_ms":21085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-attention is a low-pass graph filter; AGF replaces it with a learned singular-value-domain filter that passes high frequencies and runs in linear time.","keywords":["attentive graph filter","linear transformer","singular value domain","graph signal processing","low-pass filter","Jacobi polynomial","long range arena","time series classification"],"falsifier":"A direct frequency-response measurement would settle the central claim: take a trained AGF layer, feed value signals that are pure sinusoids at known frequencies, and compare output amplitudes; the paper's story predicts that learned $\\theta_k$ produce bands that amplify high frequencies. The paper's own $\\mathbf{H}_{UV^\\top}$ ablation, which fixes $\\Sigma$ to the identity, already drops UEA average accuracy from 75.1 to 71.7, showing that the filter term itself carries the effect.","tokens_in":20333,"feed_emoji":"","tokens_out":8155,"duration_ms":70772,"temperature":0.7,"pith_summary":"Self-attention in a Transformer can be read as a graph filter: multiplying value vectors by a softmax-normalized attention matrix is a first-order, low-pass operation that smooths tokens and suppresses high-frequency information. The paper proposes AGF (Attentive Graph Filter), a linear-complexity attention layer that instead learns the filtering directly in the singular value domain: small networks generate singular vectors and singular values from the input, the singular values are shaped by a Jacobi-polynomial filter with learnable coefficients, and the result multiplies the value matrix with cost $\\mathcal{O}(nd^2)$ in sequence length $n$ and hidden dimension $d$. The paper claims this filter can pass both low and high frequencies, and reports state-of-the-art average accuracy of 75.1 on UEA time-series classification and 60.1 on the Long Range Arena benchmark, beating the vanilla Transformer and prior linear methods. If correct, AGF is a drop-in replacement for softmax attention that avoids quadratic cost without giving up frequency information.","feed_headline":"Singular-value attention filter beats linear baselines on LRA","feed_subtitle":"Learned matrix factors filter attention in the singular value domain, keeping high-frequency detail at linear cost.","key_machinery":"The object doing the work is the learned singular-value-domain factorization $\\mathbf{H}(\\mathbf{X}) = \\mathbf{U}(\\mathbf{X}) \\Sigma(\\mathbf{X}) \\mathbf{V}(\\mathbf{X})^\\top$, where $\\mathbf{U}$ and $\\mathbf{V}^\\top$ are softmax-normalized linear projections of the input and $\\Sigma$ is a per-token singular-value matrix filtered through a truncated Jacobi polynomial expansion. The Jacobi basis is an orthogonal polynomial family on $[-1,1]$, so the learnable coefficients $\\theta_k$ give a stable filter instead of an ill-conditioned monomial expansion. The argument uses this factorization to avoid ever forming the $n \\times n$ attention map: the multiplication with the value matrix is reordered to contract the $d \\times d$ part first, which is what yields $\\mathcal{O}(nd^2)$ complexity and makes the layer a practical linear-attention replacement.","core_discovery":"On the paper's own terms, the discovery is that a linear transformer does not have to approximate the softmax attention map; it can instead learn a graph filter. The paper proves that the standard softmax attention matrix is a low-pass filter (Theorem 1) because repeated multiplication by the row-stochastic matrix drives signals to the dominant eigenvector. It then constructs AGF, which parameterizes $\\mathbf{U}(\\mathbf{X}) = \\mathrm{softmax}(\\mathbf{X}\\mathbf{W}_U)$, singular values $\\Sigma(\\mathbf{X}) = \\sum_{k=0}^K \\theta_k T_k(\\mathrm{diag}(\\sigma(\\mathbf{X}\\mathbf{W}_\\Sigma)))$ with $T_k$ a Jacobi polynomial, and $\\mathbf{V}(\\mathbf{X})^\\top = \\mathrm{softmax}((\\mathbf{X}\\mathbf{W}_V)^\\top)$, and applies the filter as $\\mathbf{H} = \\mathbf{U} \\Sigma \\mathbf{V}^\\top$ to the value matrix. Because the coefficients $\\theta_k$ can be negative, the filter can emphasize high-frequency components, and because the singular values differ per token, each token gets its own frequency response. Empirically AGF reports average accuracy 75.1 on the ten UEA datasets versus 71.9 for the vanilla Transformer, 60.1 on LRA versus 58.8, marginal ImageNet gains when plugged into DeiT-small, and reduced over-smoothing: token cosine similarity stays near 0.5 instead of rising to about 0.9 in a 12-layer vision transformer.","pith_inferences":["A natural testable extension is to apply AGF to tasks where sharp local features matter, such as anomaly detection, boundary detection, or punctuation-heavy text, where preserving high frequencies should matter more than in the benchmarks reported.","The complexity claim hides a $d^2$ factor: AGF is most attractive for long sequences with moderate hidden dimension, and for very high-dimensional embeddings the $d \\times d$ contraction could dominate; the paper does not report this crossover.","Because the paper's SVD interpretation is only softly enforced, a direct diagnostic would be to measure $\\|\\mathbf{U}^\\top\\mathbf{U}-\\mathbf{I}\\|$ and $\\|\\mathbf{V}\\mathbf{V}^\\top-\\mathbf{I}\\|$ on trained models; if the learned factors are far from orthogonal, the benefits may come from the parameterization rather than the spectral story.","The graph-filter framing suggests variants not explored here: other orthogonal bases or explicit band-pass coefficient designs, or applying AGF inside state-space sequence models, which the paper names as future work."],"forward_implications":["AGF can be dropped into existing Transformer layers in place of softmax attention with the same input and output shapes, changing only the attention computation and training objective.","Because it learns coefficients that can go negative, AGF need not behave as a low-pass filter; it can preserve high-frequency components of value vectors, which the paper argues is why it outperforms linear attention methods that merely approximate softmax attention.","The token-specific singular values mean different tokens can filter their own frequency content, an ability the paper ties to better handling of complex dynamics in hidden representations.","In deep Transformers, the paper reports that AGF mitigates over-smoothing: token embeddings remain more distinguishable across layers than with vanilla attention."],"supporting_citations":[{"why":"Supplies the directed-graph GSP-through-SVD formalism, including the $\\alpha$-power of a symmetrically normalized adjacency, that AGF builds on.","marker":"[Maskey et al., 2023]"},{"why":"Source of the theorem that non-negative or alternating filter coefficients produce low-pass or high-pass behavior, which motivates learning $\\theta_k$.","marker":"[Chien et al., 2021]"},{"why":"Provides the spectral facts $\\lambda_1=1$ and $|\\lambda_i|<1$ used in the low-pass proof of the graph filter.","marker":"[Von Luxburg, 2007]"},{"why":"Defines the Jacobi orthogonal polynomial basis used to stabilize the learnable graph filter.","marker":"[Askey and Wilson, 1985]"},{"why":"Defines the Long Range Arena benchmark on which AGF reports its long-sequence accuracy.","marker":"[Tay et al., 2020]"},{"why":"Provides the UEA multivariate time-series classification archive used for the main accuracy comparison.","marker":"[Bagnall et al., 2018]"},{"why":"Establishes the interpretation of self-attention as a normalized adjacency matrix and connects it to over-smoothing.","marker":"[Shi et al., 2022]"}],"fun_headline_variants":["Singular-value graph filter lets linear attention keep high-frequency info","Linear attention as graph filter: tune singular values per token","Singular-value attention filter: high-frequency at linear cost","Graph-filter attention: learnable singular values for each token","Singular-value domain gives linear attention a frequency dial"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learned matrices $\\mathbf{U}$, $\\Sigma$, and $\\mathbf{V}$ behave as a genuine singular-value decomposition with $\\Sigma$ diagonal and $\\mathbf{U}$, $\\mathbf{V}$ orthogonal, so that the proven low- and high-pass behavior of spectral graph filters transfers to AGF; the orthogonality is only softly regularized, and the regularizer as written mismatches dimensions.","fun_headline_variants_meta":{"raw":{"variants":["Singular-value graph filter lets linear attention keep high-frequency info","Linear attention as graph filter: tune singular values per token","Singular-value attention filter: high-frequency at linear cost","Graph-filter attention: learnable singular values for each token","Singular-value domain gives linear attention a frequency dial"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001163,"raw_usage":{"total_tokens":4894,"prompt_tokens":1101,"completion_tokens":3793,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":717,"completion_tokens_details":{"reasoning_tokens":3712}},"tokens_in":717,"tokens_out":3793,"duration_ms":24929,"temperature":1.0,"reasoning_tokens":3712,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:52:52.135045+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct frequency-response measurement would settle the central claim: take a trained AGF layer, feed value signals that are pure sinusoids at known frequencies, and compare output amplitudes; the paper's story predicts that learned $\\theta_k$ produce bands that amplify high frequencies. The paper's own $\\mathbf{H}_{UV^\\top}$ ablation, which fixes $\\Sigma$ to the identity, already drops UEA average accuracy from 75.1 to 71.7, showing that the filter term itself carries the effect.","supporting_citations":[{"cited_title":"A fractional graph laplacian approach to oversmoothing","cited_arxiv_id":null,"evidence_quote":"Supplies the directed-graph GSP-through-SVD formalism, including the $\\alpha$-power of a symmetrically normalized adjacency, that AGF builds on."},{"cited_title":"Adaptive universal generalized PageR- ank graph neural network","cited_arxiv_id":null,"evidence_quote":"Source of the theorem that non-negative or alternating filter coefficients produce low-pass or high-pass behavior, which motivates learning $\\theta_k$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Jacobi orthogonal polynomial basis used to stabilize the learnable graph filter."},{"cited_title":"Revisiting over-smoothing in bert from the perspective of graph","cited_arxiv_id":null,"evidence_quote":"Establishes the interpretation of self-attention as a normalized adjacency matrix and connects it to over-smoothing."}],"review_version":1}