Pith. sign in

REVIEW 4 major objections 7 minor 18 references

FFT-based spectral filtering of Q/K projections cuts attention loss by 79%

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 09:34 UTC pith:5I5PMB6M

load-bearing objection The stress-test concern about leakage is correct and lands hard. The paper's central defense against non-causal information leakage is mathematically false, and the headline result depends on an implementation bug to suppress that leakage. the 4 major comments →

arxiv 2607.07478 v1 pith:5I5PMB6M submitted 2026-07-08 cs.LG cs.CLeess.SP

FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

Athanasios Zeris This is my paper
classification cs.LG cs.CLeess.SP
keywords transformer attentionspectral preprocessingFourier transformquery-key projectionsphase coherencefrequency-domain filteringcharacter-level language modelingmulti-scale decomposition
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that applying a Fourier transform to the learned query and key projections in a transformer—filtering them in the frequency domain before computing attention scores—substantially improves language modeling on character-level text. The central object is the spectral filter: a Gaussian weighting over DFT frequency bins, applied to Q and K before the dot-product score. The author reports that even a fixed random spectral filter improves over standard attention, that a single learned frequency at paragraph scale improves further, and that four learned frequencies spanning paragraph-to-word scales achieve a 79% reduction in validation loss over standard dot-product attention. The mechanism, the author argues, is phase-coherent temporal structure: the gain disappears when phases are randomized, and it is specific to frequency-domain mixing rather than any generic linear transformation of Q/K vectors. The paper also documents a structural limitation: the bilateral FFT couples every position to future tokens, making it non-causal, and causal time-domain alternatives fail at character scale due to limited receptive field.

Core claim

The paper's central finding is that frequency-domain preprocessing of query-key projections carries genuine signal for attention, and that this signal lives in the phase structure of the Q/K representations at paragraph scale (roughly 50-70 tokens per cycle), not in amplitude spectra. A single fixed frequency at paragraph scale nearly matches a learned frequency, suggesting a robust optimum. When extended to four learned frequencies, the model independently converges to a near-geometric multi-scale ordering (periods of approximately 49, 27, 10, and 6 tokens), which an independent Empirical Mode Decomposition analysis also recovers. The author identifies a precise architectural boundary: only

What carries the argument

FourierQK applies a soft Gaussian selector over DFT frequency bins to the query and key projections, then reconstructs the filtered representations before computing attention scores. The filter can be fixed (random or at a chosen frequency) or learned (one or more dominant frequencies per attention head). The bilateral FFT provides global sequence mixing that no causal local filter with fewer than T taps can replicate.

Load-bearing premise

The paper's claim that Fourier-QK avoids future-token leakage rests on a shuffled-validation diagnostic: if the model performs much worse on shuffled tokens, it must be using genuine temporal order. But a model could exploit non-causal spectral information from future tokens while still depending on token order overall, producing a non-zero gap without being leakage-free. The diagnostic may not detect all forms of non-causal information use.

What would settle it

If phase-randomized Q/K representations (preserving amplitude spectra, destroying phase) perform no worse than standard attention, the core mechanism claim fails. The paper reports they perform worse than baseline in all conditions.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If spectral preprocessing of Q/K genuinely captures phase-coherent discourse structure, then attention mechanisms that operate purely in the time domain may be missing a class of signal that is naturally available in the frequency domain.
  • The convergence to a near-geometric multi-scale ordering, if it generalizes beyond TinyShakespeare, suggests that transformer representations contain latent hierarchical temporal structure that standard dot-product attention does not exploit.
  • The failure of causal time-domain filters at character scale, combined with the success of bilateral FFT, points to a trade-off between causality and global spectral access that may require architectural innovation to resolve.
  • The phase-randomization result implies that amplitude-based spectral features (which frequencies are present) are uninformative or harmful for attention, while phase relationships (where those frequencies align in the sequence) are the active signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The shuffled-gap diagnostic tests whether the model depends on token order, but a model could exploit non-causal spectral information from future tokens while still depending on token order overall—producing a non-zero gap without being leakage-free. The paper's claim that Fourier-QK 'does not leak' rests on this diagnostic being complete, which is questionable given that the corrected MultiFourie
  • If the bilateral FFT is structurally non-causal and the strongest results (MultiFourier-QK, val=0.309) come from the buggy-scaling regime that accidentally limited leakage, then the practical utility of this approach for causal language modeling remains an open question rather than a settled result.
  • The paragraph-scale optimum (period ~60 tokens) may be specific to dramatic text structure in Shakespeare; whether the same scale or a different one emerges in other text domains is testable but untested here.
  • A natural extension would test whether RoPE (which contributes no additive frequency components to Q/K) preserves the spectral gain better than learned additive positional encoding, as the paper itself suggests.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes FourierQK, a method that applies FFT-based spectral preprocessing to learned query-key (Q/K) projections in transformer attention. On character-level TinyShakespeare, the author reports substantial improvements over standard dot-product attention, with a single learned frequency at paragraph scale achieving val=0.608 and a multi-frequency variant achieving val=0.309. The paper includes ablations showing that random orthogonal/non-orthogonal projections yield no gain, that causal time-domain filters underperform, and that phase coherence (rather than amplitude spectra) drives the improvement. A shuffled-validation diagnostic is used throughout to argue against positional leakage. The paper is part of a seven-paper series by the same author on spectral methods in attention.

Significance. The paper provides a clear architectural distinction from FNet by preserving the Q/K score structure and applying spectral filters only to Q/K representations. The phase randomization and Theiler-style surrogate analyses are well-designed and provide genuine mechanistic insight into why spectral preprocessing helps. The code is publicly available, supporting reproducibility. The systematic reporting of negative results (causal filters failing at character scale, implementation bug discovery and correction) demonstrates scientific transparency. The multi-seed robustness table for the corrected Init4 result (mean val=0.236, std=0.019) is a credible confirmation of the single-frequency result.

major comments (4)
  1. §4.4 and Appendix A, Eq. (8): The paper claims that Fourier-QK 'does not leak' because the Gaussian weight is applied 'in the frequency domain before any position-indexed reconstruction,' producing 'a single complex number per position.' However, Eq. (8) is algebraically identical to the irfft reconstruction in Eq. (6) that the paper itself identifies as leaking. Expanding Eq. (8): q̃(i) = Σ_ω [Σ_t q(t)·exp(−i2πωt/T)] · w(ω,f*) · exp(i2πωi/T) = Σ_t q(t) · [Σ_ω w(ω,f*)·exp(i2πω(i−t)/T)] = Σ_t q(t)·κ(i−t), where κ(τ) = IRFFT[w(ω,f*)](τ). Since w(ω,f*) is real-valued and symmetric (Eq. 5), κ(−τ)=κ(τ). This is exactly the bilateral coupling the paper identifies as leakage in Eq. (6)-(7). The 'single complex number per position' IS a position-indexed reconstruction through a symmetric kernel. The paper's own leakage hierarchy table in §4.4 labels Fourier-QK as 'Mild (bilateral FFT)' with gap=
  2. §4.6, Table 6: The headline MultiFourier-QK result (val=0.309, Δ=+1.166) is from a single seed under the buggy double-scaling regime. The multi-seed table with corrected scaling reveals that MultiFourier (fixed) achieves mean val=0.019 with gap=+0.38, which the paper itself identifies as 'leakage-dominated' (comparable to Hilbert-OrthoQK val=0.018). The paper states: 'The buggy MultiFourier (mean=0.289) remains the valid primary result of this paper because the double scaling attenuated filter sharpness, limiting effective spectral coverage and leakage.' This is not a scientifically valid justification for a primary result: the gain in the headline number is attributable to an implementation bug that incidentally limited leakage, not to a genuine architectural improvement. The paper's central claim that MultiFourier-QK achieves a 79% reduction over standard attention is not supported by,
  3. §4.4: The shuffled-validation diagnostic (gap = val_shuffled − val_ordered) is used as the primary test for positional leakage. The paper claims that a large gap 'provides evidence against positional leakage.' However, this diagnostic tests whether the model depends on token order, not whether it exploits non-causal spectral information. A model could exploit future-token information encoded in the bilateral FFT reconstruction while still depending on token order for other aspects of its prediction, producing a non-trivial gap. The gap decreases monotonically with spectral coverage (Init4 gap=+4.0, MultiFourier gap=+0.38, Hilbert gap=+0.06), which is consistent with partial leakage: the model obtains some information from non-causal spectral mixing, reducing its dependence on genuine temporal order while retaining enough order-dependence to produce a non-zero gap. The diagnostic cannot '
  4. §4.6, Table 6: The multi-seed table caption states 'seeds = {42, 3,653,403,231, 2,735,729,615}' but the table header reads 'Seed 42, Seed 123, Seed 456.' These are inconsistent. If the actual seeds used are the large integers listed in the caption, the table headers are mislabeled; if the seeds are 42/123/456, the caption text is incorrect. This must be resolved for the reproducibility claim to hold.
minor comments (7)
  1. §4.1: The paper reports SA-v3 results (SA-K2-v3 val=0.632) alongside Fourier-QK results, but the relationship between Scalogram Attention and FourierQK is not clearly explained. It is unclear whether SA is a variant of FourierQK or a separate architecture. A brief clarifying sentence would help.
  2. Table 1: The 'Gap' column for causal filters (Gaussian-QK, MexHat-QK, Morlet-causal) shows '—' (missing). These values should be reported for consistency, especially since the gap diagnostic is central to the paper's leakage argument.
  3. §4.2, Table 3: Fourier-QK is listed with 'Bin ≈27' but §4.1 states the learned frequency converges to 'bin ≈ 3.5–4.1' after the weight-decay correction. It appears Table 3 reports the pre-correction value. This should be clarified or updated.
  4. §4.5: The EMD validation references 'the companion adaptive-decomposition paper [Huang et al., 1998]' but Huang et al. is the original EMD method paper, not a companion paper by the author. This citation appears to be an error.
  5. §5, filter comparison table: The table lists 'Gauss-Collapse val=0.608' which is identical to Fourier-QK-Init4 val=0.608. It is unclear whether these are the same experiment under different names or a coincidence. Clarification is needed.
  6. References: Five of the seven series papers (Zeris 2026a-f) are cited as 'arXiv preprint, 2026 (submitted concurrently)' or 'in preparation' without arXiv IDs. These are load-bearing citations for the paper's framing. At minimum, the concurrently submitted companion papers should have stable identifiers.
  7. §3: The experimental setup is described as 'Identical to Papers 1–4' with a brief parameter list. For a self-contained submission, the key hyperparameters (learning rate, batch size, optimizer parameters) should be stated explicitly rather than requiring readers to consult companion papers.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and substantive review. The referee raises four major points: (1) an algebraic argument that Eq. (8) is equivalent to the irfft reconstruction the paper identifies as leaking; (2) a challenge to the scientific validity of retaining the buggy MultiFourier result as the primary headline; (3) a concern that the shuffled-validation diagnostic cannot distinguish genuine causal learning from partial spectral leakage; and (4) an inconsistency between seed labels in the caption and table header of Table 6. We agree with points (1), (2), and (4) and will revise accordingly. On point (3) we partially agree: the diagnostic is weaker than we claimed, and we will revise our language, but we argue it retains diagnostic value when interpreted alongside the leakage hierarchy rather than as a standalone test.

read point-by-point responses

  1. Referee: Eq. (8) is algebraically identical to the irfft reconstruction in Eq. (6) that the paper identifies as leaking. The 'single complex number per position' IS a position-indexed reconstruction through a symmetric kernel. The paper's own leakage hierarchy table labels Fourier-QK as 'Mild (bilateral FFT)'.

    Authors: The referee's algebra is correct. Expanding Eq. (8) as written, q̃(i) = Σ_ω q̂(ω)·w(ω,f*)·exp(i2πωi/T) = Σ_t q(t)·κ(i−t), where κ(τ) = IRFFT[w(ω,f*)](τ). Since w(ω,f*) is real-valued and symmetric, κ(−τ) = κ(τ), and this is indeed the same bilateral coupling identified as leakage in Eqs. (6)–(7). We cannot defend the claim that Fourier-QK 'does not leak' or that it 'avoids irfft leakage entirely.' The distinction we attempted to draw — that the Gaussian weight is applied 'in the frequency domain before any position-indexed reconstruction' — does not change the algebra: the result is still a bilateral convolution with a symmetric kernel. We will revise the manuscript to retract the non-leakage claim for Fourier-QK. Specifically: (a) §4.4's statement that 'Fourier-QK avoids the irfft leakage entirely' will be removed and replaced with an acknowledgment that Fourier-QK uses the same bilateral FFT reconstruction and therefore has the same structural non-causality. (b) Appendix A's 'Why Fourier-QK does not leak' subsection will be rewritten as 'Leakage severity of Fourier-QK,' acknowledging that the bilateral kernel applies but that the narrow Gaussian selector (σ²=4 bins, ~3.9% spectral coverage) produces a kernel with much weaker off-diagonal coupling than wideband filters, which is consistent with the mild gap reduction (+4.0 vs +5.78 for BASE-DOT) rather than the catastrophic gap collapse seen in Hilbert-OrthoQK (+0.06). (c) The leakage hierarchy table will be relabeled to reflect that Fourier-QK exhibits 'mild bilateral leakage' rather than 'no leakage.' We note that this revision does not affect the Init4 multi-seed result (mean val=0.236, gap=+4.0), which is the paper's robustly confirmed finding; it affects only the framing of why the gap is reduced relative to BASE. revision: yes

  2. Referee: The headline MultiFourier-QK result (val=0.309) is from a single seed under a buggy double-scaling regime. The corrected multi-seed result (mean val=0.019) is leakage-dominated. The paper's justification for retaining the buggy result as primary is not scientifically valid.

    Authors: We agree. The referee is correct that retaining a result from a known implementation bug as the 'valid primary result' is not scientifically justified. The corrected MultiFourier result (mean val=0.019, gap=+0.38) is leakage-dominated, as the paper itself acknowledges, and the buggy result's lower loss is attributable to incidental leakage attenuation from the double scaling, not to a genuine architectural improvement. We will revise the manuscript as follows: (a) The abstract will remove the claim of '79% reduction over standard dot-product attention' and the MultiFourier val=0.309 headline number. The abstract will instead report the robustly confirmed single-frequency result: Init4 mean val=0.236 (±0.0019), Δ=+1.240 over BASE-DOT, a 84% reduction. (b) §4.6 will be restructured to present the corrected MultiFourier result as a negative finding: multi-frequency bilateral spectral attention with K=4 near-dyadic filters covers ~15% of the spectrum and is leakage-dominated, establishing that leakage severity scales with spectral coverage. The buggy result will be reported only as a historical note explaining why the initial headline was wrong. (c) The multi-scale frequency ordering finding (near-geometric ratios across seeds) will be retained as it is consistent across both scaling regimes and all three seeds, but will be framed as a finding about what the optimizer discovers, not as evidence of genuine improvement. We acknowledge that the paper's central claim must be downgraded from the multi-frequency result to the single-frequency result, which is the only result that is both multi-seed confirmed and has a gap consistent with genuine learning. revision: yes

  3. Referee: The shuffled-validation diagnostic tests whether the model depends on token order, not whether it exploits non-causal spectral information. A model could exploit future-token information while still depending on token order, producing a non-trivial gap. The gap decreases monotonically with spectral coverage, consistent with partial leakage.

    Authors: We partially agree. The referee correctly identifies that the shuffled-gap diagnostic tests order-dependence, not non-causality, and that a model exploiting bilateral FFT leakage could retain non-trivial order-dependence. The monotonic decrease of the gap with spectral coverage (Init4: +4.0, MultiFourier: +0.38, Hilbert: +0.06) is indeed consistent with the partial-leakage interpretation the referee describes. We will revise the manuscript to acknowledge this limitation explicitly: the shuffled gap is not a sufficient test for leakage on its own. However, we argue it retains diagnostic value when interpreted alongside the leakage hierarchy and the spectral-coverage analysis. The key observation is that the gap does not merely decrease — it collapses to near-zero (Hilbert: +0.06) only when the bilateral FFT covers the full spectrum (100%), while Init4 at ~3.9% coverage retains a gap of +4.0, which is 69% of the BASE-DOT gap (+5.78). This is more consistent with mild bilateral coupling (which the referee's Comment 1 shows is present) than with the catastrophic leakage seen at high coverage. We cannot rule out that some fraction of the Init4 gain comes from non-causal spectral information, and we will state this clearly. The Theiler-style surrogate analysis (§5.1), run on the causal implementation, provides a complementary line of evidence that phase coherence drives the gain; we will note that this experiment is the strongest evidence for a genuine mechanism but acknowledge it does not fully resolve the bilateral-FFT leakage concern for the bilateral results in Table 1. We will revise all instances of 'provides evidence against positional leakage' to 'is consistent with genuine sequence learning but does not rule out partial spectral leakage,' and add a paragraph in §4.4- revision: no

  4. Referee: Table 6 caption states seeds = {42, 3,653,403,231, 2,735,729,615} but the table header reads 'Seed 42, Seed 123, Seed 456.' These are inconsistent.

    Authors: This is a genuine error. The actual seeds used are the large integers listed in the caption: {42, 3653403231, 2735729615}, drawn uniformly from [0, 2^32) via NumPy's default random generator with meta-seed 0. The table headers 'Seed 123' and 'Seed 456' are mislabeled — these were placeholder labels from an earlier draft that were not updated when the actual seeds were drawn. We will correct the table headers to 'Seed 42,' 'Seed 3653403231,' and 'Seed 2735729615' (or abbreviated forms with the full values in a footnote) to match the caption. The experimental results are unaffected; only the labels were wrong. revision: yes

Circularity Check

0 steps flagged

No significant circularity; self-citations are contextual, not load-bearing. The non-leakage claim contains a mathematical error (Eq. 8 ≡ Eq. 6 by algebra) but this is a correctness defect, not circularity.

full rationale

The paper's central empirical claim — that FFT-based spectral preprocessing of Q/K projections improves attention — is grounded in self-contained experiments: the architecture is defined by Eqs. 2–5, baselines use standard dot-product attention (BASE-DOT), and controls (Random-Orth, Random-Proj) are run within the same framework. The five companion papers by the same author (Zeris 2026a–f) are cited for background context and architectural framing (e.g., the bilateral/causal boundary), but none are load-bearing for the main experimental results. The EMD validation (§4.5) uses an independent method (Huang et al. 1998), and the Theiler surrogate analysis (§5.1) uses a standard external method (Theiler et al. 1992). The non-leakage claim (§4.4, Appendix A Eq. 8) does contain a significant mathematical error: Eq. 8 expands algebraically to Σ_t q(t)·κ(i−t) where κ(τ) = IRFFT[w(ω,f*)](τ), which is identical to the leaking reconstruction of Eq. 6 — the paper's own table even labels Fourier-QK as 'Mild (bilateral FFT)' leakage, contradicting the text claim that it 'does not leak.' However, this is a correctness defect (false mathematical distinction), not circularity: the conclusion is not embedded in the premises by construction, and the shuffled-gap diagnostic, while imperfect, is an independent empirical test rather than a tautological one. The paper transparently acknowledges the diagnostic's limitations ('cannot rule out all forms of spectral artifact'). Score 2 reflects the presence of non-load-bearing self-citations and the fact that the central results have independent empirical content.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new particles, forces, dimensions, or postulated entities. The spectral filter and frequency selector are architectural components, not invented physical entities. The free parameters are standard learnable model parameters (frequencies, widths) and experimental design choices (K, kernel size). The axioms are domain assumptions about the informativeness of spectral structure and the adequacy of the leakage diagnostic, plus the representativeness of the benchmark. The most load-bearing axiom is the sufficiency of the shuffled-gap diagnostic for ruling out leakage, since the bilateral FFT is structurally non-causal.

free parameters (4)
  • Learned dominant frequency f* per head = bin ~3.7-4.7 (period ~55-69 tokens) for single-frequency; bins 4.6, 14.5, 31.0, 51.2 for multi-frequency
    Learned via gradient descent on cross-entropy loss; the paper shows convergence to paragraph scale.
  • Gaussian selector width sigma^2 = 4 bins (fixed)
    Chosen by convenience, not learned; the paper acknowledges this as a limitation and proposes learnable width as future work.
  • Number of frequencies K = 1 or 4
    Set by the experimenter; the paper tests K=1 and K=4 but does not sweep K systematically.
  • Filter kernel size K_taps (causal variants) = 32 or 128
    Set by the experimenter for causal convolution variants.
axioms (4)
  • domain assumption The DFT of Q/K projections along the position axis captures meaningful spectral structure related to linguistic hierarchy.
    Section 2: the entire approach assumes that frequency-domain representation of Q/K sequences contains useful information beyond what single-token dot products capture.
  • ad hoc to paper The shuffled-validation diagnostic (gap = val_shuffled - val_ordered) is a sufficient test for positional leakage.
    Section 3.1: all leakage claims rest on this diagnostic. A model exploiting non-causal spectral information while still depending on token order would produce a non-zero gap.
  • domain assumption TinyShakespeare at character level with T=256 is a representative testbed for attention mechanism evaluation.
    Section 3: all experiments use this single dataset and tokenisation. The paper acknowledges scale limitations but the generalisability of conclusions depends on this assumption.
  • domain assumption The BASE-DOT baseline (val=1.474) is a well-tuned reference point.
    All Delta values are computed against this baseline. If the baseline is undertrained or misconfigured, the reported gains are inflated.

pith-pipeline@v1.1.0-glm · 19156 in / 2952 out tokens · 369432 ms · 2026-07-09T09:34:47.935972+00:00 · methodology

0 comments
read the original abstract

FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (Delta=+0.443); a single learned frequency at paragraph scale achieves val=0.608 (Delta=+0.867); and four learned frequencies spanning paragraph to word scale achieve val=0.309 (Delta=+1.166), a 79% reduction over standard dot-product attention. The single-frequency result is confirmed across three random seeds (mean val=0.236, std=0.019). The four frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens/cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales. The gain is specific to spectral preprocessing: random orthogonal and non-orthogonal projections of Q/K produce no measurable improvement, suggesting the benefit comes from global frequency-domain mixing rather than metric distortion. All results are verified by a shuffled-validation diagnostic against positional leakage. Causal filters (Gaussian, Mexican Hat, Morlet) do not improve over standard attention at character-level tokenisation: the bilateral FFT kernel is structurally non-causal, coupling every position to future tokens. This defines an architectural boundary between bilateral spectral attention (this paper) and genuinely causal spectral attention at word-scale tokenisation (companion paper MorletQK). This work is architecturally distinct from FNet (Lee-Thorp et al., 2021), which replaces attention with Fourier mixing of token embeddings. Here, spectral preprocessing applies only to Q/K projections while the full attention score structure is preserved.

Figures

Figures reproduced from arXiv: 2607.07478 by Athanasios Zeris.

Figure 1
Figure 1. Figure 1: Summary of FOURIERQK results (corrected scaling throughout). Top left: learning curves for key models, contrasting buggy double-scaling (dash-dot) with corrected single 1/ √ hs scaling (solid); Init4 fixed improves from val = 0.636 to 0.216. Top centre: the phase × positional￾encoding matrix (§5), showing phase randomisation is harmful under all three PE conditions. Top right: multi-seed robustness (3 seed… view at source ↗
Figure 2
Figure 2. Figure 2: Theiler-style surrogate analysis of a frozen, trained causal SpectralQK. [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 8 internal anchors

  1. [1]

    Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention

    Zeris, A. E nergy- G ated A ttention: S pectral S alience as an I nductive B ias for T ransformer A ttention. arXiv preprint arXiv:2605.21842, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

    Zeris, A. E nergy- G ated A ttention and W avelet P ositional E ncoding: C omplementary I nductive B iases for T ransformer A ttention. arXiv preprint arXiv:2605.26355, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding

    Zeris, A. B eyond S inusoids: A M orlet W avelet F ramework for T ransformer P ositional E ncoding. arXiv preprint arXiv:2606.01258, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  4. [4]

    Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram

    Zeris, A. M ultiscale POD of T ransformer A ttention F ields: S cale- S elective A nalysis via M orlet S calogram. arXiv preprint arXiv:2606.06573, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  5. [5]

    FourierQK : F ilter S hape, A dmissibility, and the L eakage-- C overage L aw

    Zeris, A. FourierQK : F ilter S hape, A dmissibility, and the L eakage-- C overage L aw. arXiv preprint, 2026 (submitted concurrently; arXiv ID to be added in v2)

    work page 2026

  6. [6]

    MorletQK : C ausal W avelet Q uery-- K ey A ttention at W ord S cale

    Zeris, A. MorletQK : C ausal W avelet Q uery-- K ey A ttention at W ord S cale. arXiv preprint, 2026 (in preparation; arXiv ID to be added in v2)

    work page 2026

  7. [7]

    Theiler, J., Eubank, S., Longtin, A., Galdrikian, B., and Farmer, J. D. Testing for nonlinearity in time series: the method of surrogate data. Physica D, 58(1-4):77--94, 1992

    work page 1992

  8. [8]

    Wavelet transforms and their applications to turbulence

    Farge, M. Wavelet transforms and their applications to turbulence. Annual Review of Fluid Mechanics, 24(1):395--458, 1992

    work page 1992

  9. [9]

    Efficiently Modeling Long Sequences with Structured State Spaces

    Gu, A., Goel, K., and R\'e, C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  10. [10]

    E., Shen, Z., Long, S

    Huang, N. E., Shen, Z., Long, S. R., Wu, M. C., Shih, H. H., Zheng, Q., Yen, N.-C., Tung, C. C., and Liu, H. H. The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proceedings of the Royal Society of London A, 454(1971):903--995, 1998

    work page 1971

  11. [11]

    Wavelet GPT: Wavelet Inspired Large Language Models

    Verma, P. WaveletGPT : Wavelet Inspired Large Language Models. arXiv preprint arXiv:2409.12924, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    FNet: Mixing Tokens with Fourier Transforms

    Lee-Thorp, J., Ainslie, J., Eckstein, I., and Ontanon, S. FNet : M ixing T okens with F ourier T ransforms. arXiv preprint arXiv:2105.03824, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  13. [13]

    A Wavelet Tour of Signal Processing

    Mallat, S. A Wavelet Tour of Signal Processing. Academic Press, 2nd edition, 1999

    work page 1999

  14. [14]

    RoFormer : Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. RoFormer : Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    work page 2024

  15. [15]

    Language through a prism: A spectral approach for multiscale language representations

    Tamkin, A., Jurafsky, D., and Goodman, N. Language through a prism: A spectral approach for multiscale language representations. In NeurIPS, volume 33, 2020

    work page 2020

  16. [16]

    and Compo, G

    Torrence, C. and Compo, G. P. A practical guide to wavelet analysis. Bulletin of the American Meteorological Society, 79(1):61--78, 1998

    work page 1998

  17. [17]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. In NeurIPS, volume 30, 2017

    work page 2017

  18. [18]

    Towards Signal Processing In Large Language Models

    Verma, P. and Pilanci, M. Towards signal processing in large language models. arXiv preprint arXiv:2406.10254, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024