{"id":"a3425b7f-678b-4dd6-93e3-a491a82652eb","arxiv_id":"2607.07478","paper_version":1,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"FFT-based spectral preprocessing of Q/K projections reduces character-level TinyShakespeare validation loss by up to 79% over standard dot-product attention, with gains attributed to phase-coherent global sequence mixing.","lead":"The paper applies FFT-based spectral filtering to transformer query-key projections before computing attention scores, reporting large validation-loss reductions on character-level TinyShakespeare. A generalist might read it to learn whether frequency-domain preprocessing of attention is a useful inductive bias for sequence models.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Fourier-QK's claimed non-leakage (§4.4, Appendix A Eq. 8) is mathematically false: Eq. 8 is algebraically identical to the irfft reconstruction (Eq. 6) that the paper itself identifies as leaking. The Gaussian spectral filter produces a bilateral kernel κ(−τ)=κ(τ), coupling every position to future","rationale":"The reader correctly identified leakage as the central concern and noted the shuffled-gap diagnostic may be insufficient. I go further: the non-leakage claim is not merely untested — it is mathematically false. Eq. 8 in Appendix A is algebraically identical to Eq. 6, which the paper itself identifies as leaking. The Gaussian spectral filter w(ω,f*) is real and symmetric, so its impulse response satisfies κ(−τ) = κ(τ), making the operation bilateral by construction. The paper's own leakage hierarchy table labels Fourier-QK as having 'Mild (bilateral FFT)' leakage, contradicting the text claim of non-leakage in the same section.\n\nThis does not change the verdict from CONDITIONAL because the reader already conditioned acceptance on a causal implementation achieving comparable gains or a formal proof of non-leakage. My analysis shows the second condition (formal proof of non-leakage) is unmeetable — the claim is provably wrong. But the first condition (causal implementation achieving comparable gains) remains valid and is now more urgent. If a causal Fourier-QK works, the result is solid regardless of bilateral leakage. If it fails (as all causal Morlet variants did), the gain is leakage-dependent.\n\nThe paper has genuine merits: the phase-coherence analysis (§5, Theiler surrogates) is well-executed, the orthogonal/non-orthogonal baselines are appropriate controls, and the EMD cross-validation (§4.5) provides independent support for the multi-scale ordering. The transparency about implementation bugs and the MultiFourier leakage is commendable. But the mathematical error in the non-leakage claim undermines the paper's central defense and means the magnitude of even the single-frequency result is suspect.\n\nThe reader's confidence of UNKNOWN is appropriate. The novelty rating of 6.0 seems fair — the idea of spectral preprocessing of Q/K is interesting but the execution is compromised by the leakage issue.\n\nOne additional note: the paper's claim that 'the buggy MultiFourier remains the valid primary result because the bug suppresses leakage' (§4.6) is not defensible. An implementation bug that happens to reduce leakage does not make the result valid — it makes the result dependent on an artifact. The corrected MultiFourier (val=0.019, gap=0.38) is clearly leakage-dominated, and the buggy version's better performance is attributable to the bug attenuating the very leakage the paper identifies, not to genuine learning.","tokens_in":16342,"tokens_out":5282,"duration_ms":213948,"concrete_test":"Implement a strictly causal version of single-frequency Fourier-QK where q̃(i) is computed using only the causal prefix q(0..i). Concretely: at each position i, compute the windowed DFT q̂_i(ω) = Σ_{t=0}^{i} q(t)·window(i−t)·exp(−i2πωt/T) using a one-sided window (e.g., exponential decay or Hann), apply the same Gaussian spectral filter w(ω,f*), and reconstruct. If this causal variant achieves val within ~0.15 nats of the bilateral Fourier-QK-Init4 result (mean val=0.236), the gain is genuine and leakage is not load-bearing. If it fails to improve over BASE-DOT (like the causal Morlet filters in §4.3), the bilateral operation — and hence leakage — is essential to the gain, and the claimed improvement is an artifact. As a trivial analytical check: compute κ(τ) = IRFFT[w(ω,f*)](τ) for the learned f* and verify κ(−τ) = κ(τ); this directly confirms the kernel is bilateral.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central defense against leakage rests on the claim that Fourier-QK 'does not leak' (§4.4, Appendix A). The argument is that Eq. 8 — q̃(i) = Σ_ω q̂(ω)·w(ω,f*)·exp(i2πωi/T) — applies the Gaussian weight 'in the frequency domain before any position-indexed reconstruction,' producing 'a single complex number per position' rather than a 'reconstruction that explicitly couples neighbouring positions through a symmetric kernel.'\n\nThis distinction is false. Expanding Eq. 8:\n  q̃(i) = Σ_ω [Σ_t q(t)·exp(−i2πωt/T)] · w(ω,f*) · exp(i2πωi/T)\n        = Σ_t q(t) · [Σ_ω w(ω,f*)·exp(i2πω(i−t)/T)]\n        = Σ_t q(t) · κ(i−t)\n\nwhere κ(τ) = IRFFT[w(ω,f*)](τ). Since w(ω,f*) is real-valued and symmetric (Eq. 5: a Gaussian in |ω|), κ(−τ) = κ(τ). This is exactly the bilateral coupling the paper identifies as leaking in Eq. 6-7. The 'single complex number per position' IS a position-indexed reconstruction through a symmetric kernel.\n\nThe paper's own leakage hierarchy (§4.4 table) labels Fourier-QK as 'Mild (bilateral FFT)' leakage with gap=+4.0, directly contradicting the text claim that it 'does not leak.' The gap decrease from BASE-DOT (+5.78) to Fourier-QK (+4.32) is consistent with partial leakage: the model obtains some information from non-causal spectral mixing, reducing its dependence on genuine temporal order while retaining enough order-dependence to produce a non-trivial gap. The shuffled-gap diagnostic cannot distinguish 'genuine learning with no leakage' from 'genuine learning plus mild leakage.'\n\nThis matters because the single-frequency Fourier-QK result (mean val=0.236) is the paper's most defensible claim, and its magnitude may be inflated by bilateral leakage. The causal filter experiments (§4.3, all failing) are consistent with the bilateral operation being essential — but this is equally consistent with 'leakage is essential' as with 'global mixing is essential.'","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper proposes FourierQK, a method that applies FFT-based spectral preprocessing to learned query-key (Q/K) projections in transformer attention. On character-level TinyShakespeare, the author reports substantial improvements over standard dot-product attention, with a single learned frequency at paragraph scale achieving val=0.608 and a multi-frequency variant achieving val=0.309. The paper includes ablations showing that random orthogonal/non-orthogonal projections yield no gain, that causal time-domain filters underperform, and that phase coherence (rather than amplitude spectra) drives the improvement. A shuffled-validation diagnostic is used throughout to argue against positional leakage. The paper is part of a seven-paper series by the same author on spectral methods in attention.","tokens_in":16615,"tokens_out":1901,"duration_ms":395220,"significance":"The paper provides a clear architectural distinction from FNet by preserving the Q/K score structure and applying spectral filters only to Q/K representations. The phase randomization and Theiler-style surrogate analyses are well-designed and provide genuine mechanistic insight into why spectral preprocessing helps. The code is publicly available, supporting reproducibility. The systematic reporting of negative results (causal filters failing at character scale, implementation bug discovery and correction) demonstrates scientific transparency. The multi-seed robustness table for the corrected Init4 result (mean val=0.236, std=0.019) is a credible confirmation of the single-frequency result.","major_comments":[{"comment":"§4.4 and Appendix A, Eq. (8): The paper claims that Fourier-QK 'does not leak' because the Gaussian weight is applied 'in the frequency domain before any position-indexed reconstruction,' producing 'a single complex number per position.' However, Eq. (8) is algebraically identical to the irfft reconstruction in Eq. (6) that the paper itself identifies as leaking. Expanding Eq. (8): q̃(i) = Σ_ω [Σ_t q(t)·exp(−i2πωt/T)] · w(ω,f*) · exp(i2πωi/T) = Σ_t q(t) · [Σ_ω w(ω,f*)·exp(i2πω(i−t)/T)] = Σ_t q(t)·κ(i−t), where κ(τ) = IRFFT[w(ω,f*)](τ). Since w(ω,f*) is real-valued and symmetric (Eq. 5), κ(−τ)=κ(τ). This is exactly the bilateral coupling the paper identifies as leakage in Eq. (6)-(7). The 'single complex number per position' IS a position-indexed reconstruction through a symmetric kernel. The paper's own leakage hierarchy table in §4.4 labels Fourier-QK as 'Mild (bilateral FFT)' with gap=","section":null},{"comment":"§4.6, Table 6: The headline MultiFourier-QK result (val=0.309, Δ=+1.166) is from a single seed under the buggy double-scaling regime. The multi-seed table with corrected scaling reveals that MultiFourier (fixed) achieves mean val=0.019 with gap=+0.38, which the paper itself identifies as 'leakage-dominated' (comparable to Hilbert-OrthoQK val=0.018). The paper states: 'The buggy MultiFourier (mean=0.289) remains the valid primary result of this paper because the double scaling attenuated filter sharpness, limiting effective spectral coverage and leakage.' This is not a scientifically valid justification for a primary result: the gain in the headline number is attributable to an implementation bug that incidentally limited leakage, not to a genuine architectural improvement. The paper's central claim that MultiFourier-QK achieves a 79% reduction over standard attention is not supported by,","section":null},{"comment":"§4.4: The shuffled-validation diagnostic (gap = val_shuffled − val_ordered) is used as the primary test for positional leakage. The paper claims that a large gap 'provides evidence against positional leakage.' However, this diagnostic tests whether the model depends on token order, not whether it exploits non-causal spectral information. A model could exploit future-token information encoded in the bilateral FFT reconstruction while still depending on token order for other aspects of its prediction, producing a non-trivial gap. The gap decreases monotonically with spectral coverage (Init4 gap=+4.0, MultiFourier gap=+0.38, Hilbert gap=+0.06), which is consistent with partial leakage: the model obtains some information from non-causal spectral mixing, reducing its dependence on genuine temporal order while retaining enough order-dependence to produce a non-zero gap. The diagnostic cannot '","section":null},{"comment":"§4.6, Table 6: The multi-seed table caption states 'seeds = {42, 3,653,403,231, 2,735,729,615}' but the table header reads 'Seed 42, Seed 123, Seed 456.' These are inconsistent. If the actual seeds used are the large integers listed in the caption, the table headers are mislabeled; if the seeds are 42/123/456, the caption text is incorrect. This must be resolved for the reproducibility claim to hold.","section":null}],"minor_comments":[{"comment":"§4.1: The paper reports SA-v3 results (SA-K2-v3 val=0.632) alongside Fourier-QK results, but the relationship between Scalogram Attention and FourierQK is not clearly explained. It is unclear whether SA is a variant of FourierQK or a separate architecture. A brief clarifying sentence would help.","section":null},{"comment":"Table 1: The 'Gap' column for causal filters (Gaussian-QK, MexHat-QK, Morlet-causal) shows '—' (missing). These values should be reported for consistency, especially since the gap diagnostic is central to the paper's leakage argument.","section":null},{"comment":"§4.2, Table 3: Fourier-QK is listed with 'Bin ≈27' but §4.1 states the learned frequency converges to 'bin ≈ 3.5–4.1' after the weight-decay correction. It appears Table 3 reports the pre-correction value. This should be clarified or updated.","section":null},{"comment":"§4.5: The EMD validation references 'the companion adaptive-decomposition paper [Huang et al., 1998]' but Huang et al. is the original EMD method paper, not a companion paper by the author. This citation appears to be an error.","section":null},{"comment":"§5, filter comparison table: The table lists 'Gauss-Collapse val=0.608' which is identical to Fourier-QK-Init4 val=0.608. It is unclear whether these are the same experiment under different names or a coincidence. Clarification is needed.","section":null},{"comment":"References: Five of the seven series papers (Zeris 2026a-f) are cited as 'arXiv preprint, 2026 (submitted concurrently)' or 'in preparation' without arXiv IDs. These are load-bearing citations for the paper's framing. At minimum, the concurrently submitted companion papers should have stable identifiers.","section":null},{"comment":"§3: The experimental setup is described as 'Identical to Papers 1–4' with a brief parameter list. For a self-contained submission, the key hyperparameters (learning rate, batch size, optimizer parameters) should be stated explicitly rather than requiring readers to consult companion papers.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper's framing is unusual in that the headline result (MultiFourier val=0.309) is explicitly acknowledged to arise from an implementation bug that incidentally limited leakage, while the corrected version (val=0.019) is leakage-dominated. The author deserves credit for transparently reporting this, but the paper as written still presents the buggy result as the primary finding. The single-frequency Init4 result (mean val=0.236, std=0.019) appears to be the most defensible contribution, but even this result is complicated by the leakage concern raised in Major Comment 1 — the bilateral FFT kernel in Fourier-QK is algebraically identical to the irfft reconstruction the paper identifies as leaking. The author may not have recognized this algebraic equivalence. If the author can show that the single-frequency Fourier-QK result survives a genuinely causal implementation (e.g., causal time-domain convolution at the learned frequency), that would be a clean, publishable result. As it stands, the paper's central claims are not adequately supported. The heavy reliance on self-citation (five companion papers, most without stable identifiers) is also a concern for a journal submission."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and substantive review. The referee raises four major points: (1) an algebraic argument that Eq. (8) is equivalent to the irfft reconstruction the paper identifies as leaking; (2) a challenge to the scientific validity of retaining the buggy MultiFourier result as the primary headline; (3) a concern that the shuffled-validation diagnostic cannot distinguish genuine causal learning from partial spectral leakage; and (4) an inconsistency between seed labels in the caption and table header of Table 6. We agree with points (1), (2), and (4) and will revise accordingly. On point (3) we partially agree: the diagnostic is weaker than we claimed, and we will revise our language, but we argue it retains diagnostic value when interpreted alongside the leakage hierarchy rather than as a standalone test.","responses":[{"response":"The referee's algebra is correct. Expanding Eq. (8) as written, q̃(i) = Σ_ω q̂(ω)·w(ω,f*)·exp(i2πωi/T) = Σ_t q(t)·κ(i−t), where κ(τ) = IRFFT[w(ω,f*)](τ). Since w(ω,f*) is real-valued and symmetric, κ(−τ) = κ(τ), and this is indeed the same bilateral coupling identified as leakage in Eqs. (6)–(7). We cannot defend the claim that Fourier-QK 'does not leak' or that it 'avoids irfft leakage entirely.' The distinction we attempted to draw — that the Gaussian weight is applied 'in the frequency domain before any position-indexed reconstruction' — does not change the algebra: the result is still a bilateral convolution with a symmetric kernel. We will revise the manuscript to retract the non-leakage claim for Fourier-QK. Specifically: (a) §4.4's statement that 'Fourier-QK avoids the irfft leakage entirely' will be removed and replaced with an acknowledgment that Fourier-QK uses the same bilateral FFT reconstruction and therefore has the same structural non-causality. (b) Appendix A's 'Why Fourier-QK does not leak' subsection will be rewritten as 'Leakage severity of Fourier-QK,' acknowledging that the bilateral kernel applies but that the narrow Gaussian selector (σ²=4 bins, ~3.9% spectral coverage) produces a kernel with much weaker off-diagonal coupling than wideband filters, which is consistent with the mild gap reduction (+4.0 vs +5.78 for BASE-DOT) rather than the catastrophic gap collapse seen in Hilbert-OrthoQK (+0.06). (c) The leakage hierarchy table will be relabeled to reflect that Fourier-QK exhibits 'mild bilateral leakage' rather than 'no leakage.' We note that this revision does not affect the Init4 multi-seed result (mean val=0.236, gap=+4.0), which is the paper's robustly confirmed finding; it affects only the framing of why the gap is reduced relative to BASE.","revision_made":"yes","referee_comment":"Eq. (8) is algebraically identical to the irfft reconstruction in Eq. (6) that the paper identifies as leaking. The 'single complex number per position' IS a position-indexed reconstruction through a symmetric kernel. The paper's own leakage hierarchy table labels Fourier-QK as 'Mild (bilateral FFT)'."},{"response":"We agree. The referee is correct that retaining a result from a known implementation bug as the 'valid primary result' is not scientifically justified. The corrected MultiFourier result (mean val=0.019, gap=+0.38) is leakage-dominated, as the paper itself acknowledges, and the buggy result's lower loss is attributable to incidental leakage attenuation from the double scaling, not to a genuine architectural improvement. We will revise the manuscript as follows: (a) The abstract will remove the claim of '79% reduction over standard dot-product attention' and the MultiFourier val=0.309 headline number. The abstract will instead report the robustly confirmed single-frequency result: Init4 mean val=0.236 (±0.0019), Δ=+1.240 over BASE-DOT, a 84% reduction. (b) §4.6 will be restructured to present the corrected MultiFourier result as a negative finding: multi-frequency bilateral spectral attention with K=4 near-dyadic filters covers ~15% of the spectrum and is leakage-dominated, establishing that leakage severity scales with spectral coverage. The buggy result will be reported only as a historical note explaining why the initial headline was wrong. (c) The multi-scale frequency ordering finding (near-geometric ratios across seeds) will be retained as it is consistent across both scaling regimes and all three seeds, but will be framed as a finding about what the optimizer discovers, not as evidence of genuine improvement. We acknowledge that the paper's central claim must be downgraded from the multi-frequency result to the single-frequency result, which is the only result that is both multi-seed confirmed and has a gap consistent with genuine learning.","revision_made":"yes","referee_comment":"The headline MultiFourier-QK result (val=0.309) is from a single seed under a buggy double-scaling regime. The corrected multi-seed result (mean val=0.019) is leakage-dominated. The paper's justification for retaining the buggy result as primary is not scientifically valid."},{"response":"We partially agree. The referee correctly identifies that the shuffled-gap diagnostic tests order-dependence, not non-causality, and that a model exploiting bilateral FFT leakage could retain non-trivial order-dependence. The monotonic decrease of the gap with spectral coverage (Init4: +4.0, MultiFourier: +0.38, Hilbert: +0.06) is indeed consistent with the partial-leakage interpretation the referee describes. We will revise the manuscript to acknowledge this limitation explicitly: the shuffled gap is not a sufficient test for leakage on its own. However, we argue it retains diagnostic value when interpreted alongside the leakage hierarchy and the spectral-coverage analysis. The key observation is that the gap does not merely decrease — it collapses to near-zero (Hilbert: +0.06) only when the bilateral FFT covers the full spectrum (100%), while Init4 at ~3.9% coverage retains a gap of +4.0, which is 69% of the BASE-DOT gap (+5.78). This is more consistent with mild bilateral coupling (which the referee's Comment 1 shows is present) than with the catastrophic leakage seen at high coverage. We cannot rule out that some fraction of the Init4 gain comes from non-causal spectral information, and we will state this clearly. The Theiler-style surrogate analysis (§5.1), run on the causal implementation, provides a complementary line of evidence that phase coherence drives the gain; we will note that this experiment is the strongest evidence for a genuine mechanism but acknowledge it does not fully resolve the bilateral-FFT leakage concern for the bilateral results in Table 1. We will revise all instances of 'provides evidence against positional leakage' to 'is consistent with genuine sequence learning but does not rule out partial spectral leakage,' and add a paragraph in §4.4-","revision_made":"no","referee_comment":"The shuffled-validation diagnostic tests whether the model depends on token order, not whether it exploits non-causal spectral information. A model could exploit future-token information while still depending on token order, producing a non-trivial gap. The gap decreases monotonically with spectral coverage, consistent with partial leakage."},{"response":"This is a genuine error. The actual seeds used are the large integers listed in the caption: {42, 3653403231, 2735729615}, drawn uniformly from [0, 2^32) via NumPy's default random generator with meta-seed 0. The table headers 'Seed 123' and 'Seed 456' are mislabeled — these were placeholder labels from an earlier draft that were not updated when the actual seeds were drawn. We will correct the table headers to 'Seed 42,' 'Seed 3653403231,' and 'Seed 2735729615' (or abbreviated forms with the full values in a footnote) to match the caption. The experimental results are unaffected; only the labels were wrong.","revision_made":"yes","referee_comment":"Table 6 caption states seeds = {42, 3,653,403,231, 2,735,729,615} but the table header reads 'Seed 42, Seed 123, Seed 456.' These are inconsistent."}],"tokens_in":16418,"tokens_out":1848,"duration_ms":274855,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The stress-test note is right, and it's the thing that matters most. The paper claims Fourier-QK 'does not leak' because the Gaussian spectral weight is applied 'in the frequency domain before position-indexed reconstruction' (Appendix A, Eq. 8). But Eq. 8 is algebraically identical to the irfft reconstruction the paper itself identifies as leaking (Eq. 6-7). Expanding: q̃(i) = Σ_ω [Σ_t q(t)·exp(−i2πωt/T)]·w(ω,f*)·exp(i2πωi/T) = Σ_t q(t)·κ(i−t), where κ(τ) = IRFFT[w](τ). Since w is real and symmetric (Gaussian in |ω|), κ(−τ) = κ(τ). This is the same bilateral coupling. The paper's own leakage hierarchy table labels Fourier-QK as 'Mild (bilateral FFT)' with gap=+4.0, directly contradicting the text claim that it 'does not leak.' The shuffled-gap diagnostic cannot distinguish 'genuine learning with no leakage' from 'genuine learning plus mild bilateral leakage' — a model can exploit non-causal spectral information while still depending on token order, producing a non-zero gap. The gap decrease from BASE-DOT (+5.78) to Fourier-QK (+4.32) is consistent with partial leakage, not its absence. This is not a minor concern; it is load-bearing for the central claim. The single-frequency result (mean val=0.236 across 3 seeds) is the most defensible finding, but its magnitude may be inflated by bilateral leakage, and the paper provides no causal implementation that achieves comparable gains to rule this out. The headline MultiFourier-QK result (val=0.309) is from a single seed, and the multi-seed table reveals the corrected-scaling version is val=0.019 with gap=0.38 — leakage-dominated. The paper's argument that the buggy version 'remains the valid primary result' because the bug suppresses leakage is not a defense; it is an admission that the strongest advertised result depends on an implementation accident to avoid a known confound. What is genuinely new and well-executed: applying FFT-based spectral filtering specifically to Q/K projections while preserving the attention score matrix is architecturally distinct from FNet. The Theiler surrogate analysis (Table 7) is a legitimate methodological contribution — the 64/36 phase-vs-amplitude decomposition on the causal implementation is clean and provides real mechanistic insight. The non-spectral controls (random orthogonal and non-orthogonal projections producing null results) are a well-designed ablation. The EMD validation of the multi-scale ordering is independent and reasonable. The paper is honest about its negative results (causal filters all fail), which is commendable. But the causal filter failures are equally consistent with 'leakage is essential' as with 'global mixing is essential' — the paper does not acknowledge this alternative. Experiments are limited to a single small dataset at character level with a single seed for most comparisons. The self-citation pattern (five companion papers) is heavy but the core architecture is self-contained. This paper is for researchers interested in spectral methods in attention. The surrogate analysis and ablation design have value. But the central empirical claim is not trustworthy as presented: the leakage defense is false, and the headline result depends on a bug. A serious referee should be assigned to verify whether the single-frequency result survives a genuinely causal implementation and whether the gains persist beyond TinyShakespeare. Without that, the contribution is a well-designed negative result plus a mechanistic analysis on a confounded positive result.","headline":"The stress-test concern about leakage is correct and lands hard. The paper's central defense against non-causal information leakage is mathematically false, and the headline result depends on an implementation bug to suppress that leakage.","tokens_in":17349,"tokens_out":870,"would_cite":false,"duration_ms":135812,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"FFT-based spectral filtering of Q/K projections cuts attention loss by 79%","keywords":["transformer attention","spectral preprocessing","Fourier transform","query-key projections","phase coherence","frequency-domain filtering","character-level language modeling","multi-scale decomposition"],"falsifier":"If phase-randomized Q/K representations (preserving amplitude spectra, destroying phase) perform no worse than standard attention, the core mechanism claim fails. The paper reports they perform worse than baseline in all conditions.","tokens_in":16241,"feed_emoji":"🌊","tokens_out":1433,"duration_ms":119952,"temperature":0.7,"pith_summary":"This paper claims that applying a Fourier transform to the learned query and key projections in a transformer—filtering them in the frequency domain before computing attention scores—substantially improves language modeling on character-level text. The central object is the spectral filter: a Gaussian weighting over DFT frequency bins, applied to Q and K before the dot-product score. The author reports that even a fixed random spectral filter improves over standard attention, that a single learned frequency at paragraph scale improves further, and that four learned frequencies spanning paragraph-to-word scales achieve a 79% reduction in validation loss over standard dot-product attention. The mechanism, the author argues, is phase-coherent temporal structure: the gain disappears when phases are randomized, and it is specific to frequency-domain mixing rather than any generic linear transformation of Q/K vectors. The paper also documents a structural limitation: the bilateral FFT couples every position to future tokens, making it non-causal, and causal time-domain alternatives fail at character scale due to limited receptive field.","feed_headline":"FFT filtering of query-key projections cuts attention loss by 79%","feed_subtitle":"Frequency-domain preprocessing of Q/K vectors exploits phase-coherent temporal structure at paragraph scale that standard dot-product misses","key_machinery":"FourierQK applies a soft Gaussian selector over DFT frequency bins to the query and key projections, then reconstructs the filtered representations before computing attention scores. The filter can be fixed (random or at a chosen frequency) or learned (one or more dominant frequencies per attention head). The bilateral FFT provides global sequence mixing that no causal local filter with fewer than T taps can replicate.","core_discovery":"The paper's central finding is that frequency-domain preprocessing of query-key projections carries genuine signal for attention, and that this signal lives in the phase structure of the Q/K representations at paragraph scale (roughly 50-70 tokens per cycle), not in amplitude spectra. A single fixed frequency at paragraph scale nearly matches a learned frequency, suggesting a robust optimum. When extended to four learned frequencies, the model independently converges to a near-geometric multi-scale ordering (periods of approximately 49, 27, 10, and 6 tokens), which an independent Empirical Mode Decomposition analysis also recovers. The author identifies a precise architectural boundary: only","pith_inferences":["The shuffled-gap diagnostic tests whether the model depends on token order, but a model could exploit non-causal spectral information from future tokens while still depending on token order overall—producing a non-zero gap without being leakage-free. The paper's claim that Fourier-QK 'does not leak' rests on this diagnostic being complete, which is questionable given that the corrected MultiFourie","If the bilateral FFT is structurally non-causal and the strongest results (MultiFourier-QK, val=0.309) come from the buggy-scaling regime that accidentally limited leakage, then the practical utility of this approach for causal language modeling remains an open question rather than a settled result.","The paragraph-scale optimum (period ~60 tokens) may be specific to dramatic text structure in Shakespeare; whether the same scale or a different one emerges in other text domains is testable but untested here.","A natural extension would test whether RoPE (which contributes no additive frequency components to Q/K) preserves the spectral gain better than learned additive positional encoding, as the paper itself suggests."],"forward_implications":["If spectral preprocessing of Q/K genuinely captures phase-coherent discourse structure, then attention mechanisms that operate purely in the time domain may be missing a class of signal that is naturally available in the frequency domain.","The convergence to a near-geometric multi-scale ordering, if it generalizes beyond TinyShakespeare, suggests that transformer representations contain latent hierarchical temporal structure that standard dot-product attention does not exploit.","The failure of causal time-domain filters at character scale, combined with the success of bilateral FFT, points to a trade-off between causality and global spectral access that may require architectural innovation to resolve.","The phase-randomization result implies that amplitude-based spectral features (which frequencies are present) are uninformative or harmful for attention, while phase relationships (where those frequencies align in the sequence) are the active signal."],"fun_headline_variants":["Spectral filtering of query-key projections cuts attention loss 79%","FFT preprocessing of Q/K vectors exploits phase structure dot-product misses","Four learned frequencies in Q/K projections converge to multi-scale ordering","Frequency-domain Q/K preprocessing reduces character-level attention loss 79%","Bilateral spectral mixing of query-key projections cuts loss by 79%"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper's claim that Fourier-QK avoids future-token leakage rests on a shuffled-validation diagnostic: if the model performs much worse on shuffled tokens, it must be using genuine temporal order. But a model could exploit non-causal spectral information from future tokens while still depending on token order overall, producing a non-zero gap without being leakage-free. The diagnostic may not detect all forms of non-causal information use.","fun_headline_variants_meta":{"raw":{"variants":["Spectral filtering of query-key projections cuts attention loss 79%","FFT preprocessing of Q/K vectors exploits phase structure dot-product misses","Four learned frequencies in Q/K projections converge to multi-scale ordering","Frequency-domain Q/K preprocessing reduces character-level attention loss 79%","Bilateral spectral mixing of query-key projections cuts loss by 79%","Spectral Q/K preprocessing at paragraph scale improves attention 79%","FFT filtering of Q/K projections preserves attention structure, cuts loss 79%","Frequency-domain Q/K preprocessing beats random projection approaches","Spectral preprocessing of query-key vectors recovers paragraph-scale phase signal","Four-frequency Q/K filter converges to geometric multi-scale ordering"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":1607,"prompt_tokens":644,"completion_tokens":963,"prompt_tokens_details":null},"tokens_in":644,"tokens_out":963,"duration_ms":47688,"temperature":1.0,"reasoning_tokens":959,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T09:34:47.935972+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If phase-randomized Q/K representations (preserving amplitude spectra, destroying phase) perform no worse than standard attention, the core mechanism claim fails. The paper reports they perform worse than baseline in all conditions.","supporting_citations":[],"review_version":1}