{"id":"d7e30e0b-5ddd-4038-9faa-59c821aa0f29","arxiv_id":"2505.00033","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A spectral dictionary language model with learned sinusoidal atoms and a Gaussian mixture prior reports competitive validation perplexity at lower cost, but lacks test-set and code support.","lead":"This paper proposes a language model that replaces the standard attention mechanism with a learned set of frequency-based atoms and per-token mixing weights, claiming linear-time scaling. It reports validation perplexities on WikiText-2 and Penn Treebank that are close to transformer baselines while using far fewer parameters, but no code or test-set results are provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central accuracy claim rests on an unvalidated capacity premise: a single global sinusoidal dictionary (Eq. 2) with coefficients from one causal Conv1D (Eqs.","rationale":"The reader's weakest_assumption identifies the expressivity of the shallow spectral representation as the load-bearing premise; I agree. The strength of this concern is that the central claim—O(KL) with competitive perplexity—requires both the complexity bound (which is straightforward) and the accuracy bound (which is entirely unsupported). The architecture described by Eqs. (2) and (5) is a low-rank, nearly linear reconstruction; no evidence in the paper demonstrates that such a space can represent the information needed for next-token prediction at the reported level. The absence of test-set numbers, error bars, code, and the internal inconsistency of the GMM prior loss (Eq. 7/13 vs. Section 3.6) further weaken the empirical case. The paper does deserve credit for a clear complexity analysis, a stated ablation, and an explicit acknowledgment in Section 5 that independent GMM sampling may limit generation coherence; these show good-faith framing. However, the central accuracy claim remains unverified. The proposed concrete test—a faithful reproduction with multi-seed validation and test perplexity—would settle whether the capacity premise holds. Since the reader already issued CONDITIONAL with similar requirements, my stress-test does not change the verdict.","tokens_in":8102,"tokens_out":14118,"duration_ms":148490,"concrete_test":"Implement the architecture exactly as specified: D=512, K=256, L=128, a single causal Conv1D over token embeddings, global sinusoidal dictionary per Eq. (2), reconstruction per Eq. (5), pointer-generator head, and loss hyperparameters (alpha,beta,gamma) = (1.0, 0.5, 0.1) with delta = 0. Train on WikiText-2 for 10 epochs with early stopping, over three seeds, and report both validation and test perplexity. If the mean validation perplexity exceeds ~36, or the test perplexity deviates from validation by more than 3 points, the capacity premise fails and the central claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a K=256 global sinusoidal dictionary, mixed by coefficients from a single causal Conv1D, yields competitive validation perplexity (31.2 on WikiText-2) at O(KL) cost. The load-bearing condition is expressivity: equations (2) and (5) define, at each position t, a reconstruction confined to the K-dimensional span of the dictionary atoms; with K=256 and D=512 this is a lossy bottleneck, and the entire predictive path before the pointer-generator head is one conv layer plus a linear mixing operation. No capacity analysis, scaling study, layer-depth study, or error analysis is provided to show this hypothesis space can carry the syntactic and semantic information required for near-GPT-2 perplexity. The evaluation is also too thin: Table 1 reports only single-run validation perplexity, with no test-set numbers despite Section 4.3 promising both, and no code or generated samples. An internal inconsistency compounds the problem: the objective in Eq. (13) includes delta * L_prior during training, yet Section 3.6 states the GMM is fitted only after convergence, so the prior term as written cannot influence training; delta is never specified. If the capacity premise is false, the reported numbers are not reproducible, and the O(KL) efficiency claim—though arithmetically sound—becomes moot.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Spectral Dictionary Generative Model (SDGM), which replaces self-attention with a learned global dictionary of K parameterized sinusoidal atoms and per-token mixing coefficients produced by a causal 1D convolutional encoder. Embeddings are reconstructed as linear combinations of the dictionary atoms, and training combines time-domain MSE, STFT magnitude loss, a language-modeling loss, and (nominally) a GMM-prior term. Experiments on WikiText-2 and PTB report validation perplexities of 31.2 and 57.1 with 22.8M parameters, 2100 tokens/s, and 6.5GB memory, which the paper claims are competitive with Transformer-XL and GPT-2 while operating at O(KL) complexity.","tokens_in":8421,"tokens_out":4684,"duration_ms":49684,"significance":"If substantiated, the central claim would be a useful efficiency-oriented alternative to self-attention: the spectral-dictionary formulation is simple, the O(KL) asymptotics are attractive for long sequences, and the dual-domain reconstruction objective is a plausible regularizer. The paper also makes an explicit falsifiable performance claim against established baselines. However, the manuscript currently lacks the evidence needed to establish the central claim: the experimental configuration contradicts the stated K << L assumption, the training objective is internally inconsistent, no test-set numbers or error bars are provided, no code is released, and there is no capacity analysis showing that the low-rank spectral bottleneck can support competitive language modeling. The contribution is therefore an interesting but unvalidated architectural proposal.","major_comments":[{"comment":"The complexity claim 'By choosing K << L, SDGM achieves O(KL) time and memory complexity...' (Section 1) is contradicted by the experimental configuration K=256, L=128, D=512 in Section 4.2, where K=2L. At this operating point the decoder's O(B*K*L*D) cost exceeds the O(B*L^2*D) cost of the attention matrix for the same length, so the reported latency and memory advantages are not explained by the stated asymptotics. Please evaluate in a regime where K << L holds, or report empirical FLOPs and memory measurements at the actual configuration.","section":"Section 1 vs. Section 4.2"},{"comment":"The training objective is not well defined. Equation (10) defines LNLL as an autoregressive next-token negative log-likelihood, but Eqs. (11) and (13) label the same term a 'Masked LM Loss' LMLM(X̂,X); the weight δ multiplying Lprior is never specified; and Section 3.6 states that the GMM is fitted only after convergence, so the δLprior term in Eq. (13) cannot influence training as written. Please state the objective actually optimized and either remove Lprior from the training loss or explain how a fitted-after-training prior is incorporated during optimization.","section":"Section 3.5, Eqs. (7)-(13)"},{"comment":"The load-bearing expressivity premise is not established. With K=256 and D=512, each reconstructed embedding is restricted to a 256-dimensional subspace selected by a single causal Conv1D layer, and the paper provides no capacity analysis, scaling study, layer-depth study, or error analysis to show that this bottleneck can carry the semantic and syntactic information needed for near-GPT-2 perplexity. Without such evidence, the reported competitive perplexities cannot be interpreted.","section":"Sections 3.2-3.4"},{"comment":"The empirical evidence is insufficient to support the headline claims. Table 1 reports only single-run validation perplexities, although Section 4.3 promises test-set perplexity; there are no error bars or multiple seeds, no generated samples despite the claims of 'generation quality' in the abstract and discussion, no code release, and the baselines are described only as 'retrained' without architecture, hyperparameter, or tokenizer details. Please add test-set perplexities, variance across seeds, sample outputs, code, and complete baseline configurations.","section":"Section 4.3 and Table 1"}],"minor_comments":[{"comment":"The loss presentation is redundant and inconsistent: Eq. (7) lists all four terms, Eq. (11) duplicates the composite loss with altered notation, and Eq. (13) repeats it again with still different notation. Please consolidate into a single, unambiguous objective.","section":"Section 3.5"},{"comment":"The notation t is overloaded: it denotes both the index t ∈ {1,...,L} in Eq. (2) and the normalized vector t = [1/L,...,1] in Eq. (3). Using a separate symbol such as τ for normalized time would clarify the definition.","section":"Eq. (3)"},{"comment":"The discussion contains an unresolved reference 'Equation (??)'; please update it to the correct equation number.","section":"Section 5, Discussion"},{"comment":"The statement that SDGM 'closely matches' GPT-2 is not supported by the PTB numbers (57.1 vs. 55.3), where the gap is non-negligible; please qualify the claim or add significance testing.","section":"Section 4.4"},{"comment":"The assertion of 'fluent, autoregressive text' in the generation section is not accompanied by any quantitative evaluation or example generations; either add samples or soften the claim.","section":"Section 3.7"}],"recommendation":"major_revision","confidential_remarks":"The paper cites two self-authored works, [11] (in review) and [12] (arXiv preprint), in the related-work discussion. These are not load-bearing for the technical claims, but the editor may wish to be aware that one is unpublished and there is no external validation. The larger concern is reproducibility: no code, no test-set numbers, and no seed information for the single-run results, so the reported efficiency gains cannot currently be verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: SDGM is a real alternative to attention, not just another kernel approximation. The combination of a learned parameterized Fourier dictionary, per-token coefficients from a causal conv, and a dual-domain reconstruction loss is not in the cited literature. The O(KL) complexity claim is arithmetically correct, and the architecture is simple enough to implement. The paper also deserves credit for the ablation showing the STFT loss helps perplexity, and for explicitly noting the limitation that GMM sampling at each step ignores temporal dependence.\n\nNow the problems. The evaluation is not enough to support the headline 'competitive perplexity.' Table 1 reports single-run validation PPL only, despite Section 4.3 promising test sets. There is no code, no error bars, no generated text. The baselines are described as retrained, but with no training budget or context length numbers, a 117M GPT-2 scoring 29.5 on WikiText-2 is hard to interpret—especially when SDGM uses L=128 while GPT-2 typically uses 1024. The capacity premise is plausible but unproven: one conv layer producing K=256 coefficients that mix 512-dimensional sinusoids is a narrow hypothesis space, and no depth or scaling study is offered.\n\nMore serious than the thin evaluation is an internal contradiction. Eq. (10) defines a standard autoregressive NLL, then Eq. (11) calls it a masked LM loss. That is a typo-level problem. The load-bearing issue is Eq. (7)/(13), which include delta * L_prior with delta unspecified, while Section 3.6 says the GMM is fitted only after convergence. So the term as written cannot be part of training. The paper needs to either drop it from the objective or explain how it is computed during training. As it stands, the training objective is not well-defined.\n\nMy take: the idea is worth a serious look, but the current manuscript is not there. I would send it to peer review with an explicit request for major revision: code release, test-set perplexity with error bars, generated samples, and a corrected training objective. If the numbers reproduce, this is a useful addition to the linear-attention toolbox; if they don't, the paper at least outlines a path.","headline":"A new attention-free architecture with a clear efficiency claim, but the training objective is internally inconsistent and the validation-only evaluation is too thin to accept.","tokens_in":8887,"tokens_out":4337,"would_cite":false,"duration_ms":35188,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned Fourier dictionary can replace self-attention in language modeling.","keywords":["spectral dictionary learning","self-attention replacement","Fourier atoms","language modeling","linear complexity","STFT reconstruction loss","Gaussian mixture prior","interpretable embeddings"],"falsifier":"Hold $K$ fixed at 256 and raise the sequence length from 128 to 1024 on a long-context corpus; if validation perplexity degrades far more steeply than a transformer baseline and reconstruction cosine similarity falls, the fixed global dictionary is not position-general. A second check would be to ablate the STFT loss and look at where reconstruction error concentrates: if errors cluster at high-frequency embedding dimensions, the frequency loss may be masking a capacity gap in the time-domain reconstruction.","tokens_in":1577,"feed_emoji":"🎛️","tokens_out":2484,"duration_ms":92891,"temperature":0.7,"pith_summary":"The paper introduces the Spectral Dictionary Generative Model (SDGM), which replaces the self-attention matrix with a global set of $K$ learnable sinusoidal atoms and per-token mixing coefficients produced by a lightweight causal convolution. Training reconstructs the original token embeddings in both the time domain and the frequency domain (via Short-Time Fourier Transform magnitude matching), together with a standard language-modeling objective. The central claim is that this spectral representation carries enough of the information attention normally supplies to reach perplexities close to transformer baselines on WikiText-2 and Penn Treebank, while cutting complexity from $O(L^2)$ to $O(KL)$. If true, this gives a concrete, low-cost route toward longer contexts and smaller memory footprints, and the learned atoms are explicit frequencies and phases that can be inspected directly.","feed_headline":"Fourier dictionary model matches transformers at 80% fewer parameters","feed_subtitle":"SDGM reaches 31.2 WikiText-2 perplexity with linear-time mixing and 6.5 GB memory.","key_machinery":"The carrying object is the learned spectral dictionary $\\mathbf{S}\\in\\mathbb{R}^{K\\times L\\times D}$, whose atoms are explicit sinusoids with per-dimension amplitude, frequency, and phase, paired with a causal Conv1D encoder that outputs per-token coefficient vectors. The reconstruction $\\hat{X}=\\mathrm{einsum}('blk,kld\\rightarrow bld', C, S)$ substitutes a $K\\times L$ mixing matrix for the $L\\times L$ attention matrix, producing $O(KL)$ time and memory per sequence when $K\\ll L$. Training minimizes a weighted sum of time-domain MSE, STFT magnitude matching, negative log-likelihood, and a GMM prior term; after training the GMM is fit to the flattened coefficient vectors and later sampled to generate text.","core_discovery":"The central discovery is that a single global, time-varying Fourier dictionary, shared across all sequences, can take over the token-mixing role of attention in an autoregressive language model. Each atom is parameterized as $S_{k,t,d}=a_{k,d}\\sin(2\\pi f_{k,d}\\,t/L+\\phi_{k,d})$, and the reconstructed embedding is $\\hat{X}_{b,t,d}=\\sum_{k=1}^{K} C_{b,t,k}S_{k,t,d}$, with coefficient vectors $C_{b,t,:}$ computed by a causal 1D convolution over the embeddings. With $K=256$, $L=128$, and embedding dimension 512, the model reports validation perplexity 31.2 on WikiText-2 and 57.1 on Penn Treebank, close to the Transformer-XL and GPT-2 Small baselines, while using 22.8 million parameters and 6.5 GB of memory. The paper further argues that the frequency-domain STFT loss is load-bearing for this result: removing it raises perplexity from 31.2 to 33.5 and lowers reconstruction cosine similarity from 0.92 to 0.88.","pith_inferences":["One implication the paper leaves implicit is that the fixed dictionary size $K$ is untested against sequence length; holding $K$ at 256 while lengthening $L$ would directly reveal whether the global sinusoids are position-general or tuned to the training length.","A natural extension is to make the coefficient prior autoregressive rather than fitting a single aggregate GMM; this would isolate whether the generation bottleneck is the dictionary representation or the independent sampling scheme.","The interpretability claim could be tested empirically by correlating learned frequencies $f_{k,d}$ with token categories or syntactic boundaries, which would give a concrete linguistic reading to the atom parameters."],"forward_implications":["If the reported numbers hold, self-attention is not required for competitive language modeling at this scale: a learned harmonic dictionary plus per-token coefficients can absorb the mixing role.","The $O(KL)$ complexity makes the architecture a candidate for long-context processing, where full attention's quadratic cost in $L$ becomes prohibitive.","The 80% parameter reduction relative to GPT-2 Small and the lower memory footprint support deployment on memory-limited hardware, assuming the result transfers to larger data.","The STFT magnitude loss appears to improve both perplexity and embedding fidelity, suggesting that spectral supervision is a useful training signal rather than a decorative regularizer.","The GMM prior provides a structured sampling route for generation, although the paper notes that independent sampling of coefficients may limit long-range coherence."],"supporting_citations":[{"why":"Supplies the transformer attention mechanism and the quadratic $O(L^2)$ cost that SDGM sets out to replace.","marker":"[18]"},{"why":"Establishes the spectral-mixing baseline whose fixed Fourier transform SDGM upgrades to a learned dictionary.","marker":"[14]"},{"why":"Provides the Transformer-XL baseline whose validation perplexities SDGM reports closely matching.","marker":"[5]"},{"why":"Provides the GPT-2 Small baseline that SDGM approaches in perplexity while reducing parameters by 80%.","marker":"[17]"},{"why":"Provides the Linformer linear-attention baseline that SDGM outperforms on perplexity while using fewer parameters.","marker":"[19]"},{"why":"Supplies the dictionary-learning formulation that the paper adapts from sparse coding to continuous Fourier atoms.","marker":"[1]"}],"fun_headline_variants":["Linear-time spectral dictionary model rivals GPT-2 on perplexity","Spectral dictionary model hits 31.2 perplexity with linear-time mixing","Attention out, atoms in: linear-time language model near-parity","Fourier dictionary replaces attention at 80% fewer parameters","From attention to atoms: linear-time LLM with 80% smaller memory"],"cache_read_input_tokens":11008,"weakest_assumption_plain":"The load-bearing premise is that one shared set of $K$ sinusoids evaluated at position $t/L$, combined with coefficients from a single causal 1D convolution, can represent token embeddings well enough to support next-token prediction that is competitive with attention.","fun_headline_variants_meta":{"raw":{"variants":["Linear-time spectral dictionary model rivals GPT-2 on perplexity","Spectral dictionary model hits 31.2 perplexity with linear-time mixing","Attention out, atoms in: linear-time language model near-parity","Fourier dictionary replaces attention at 80% fewer parameters","From attention to atoms: linear-time LLM with 80% smaller memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000751,"raw_usage":{"total_tokens":3338,"prompt_tokens":932,"completion_tokens":2406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2313}},"tokens_in":548,"tokens_out":2406,"duration_ms":17223,"temperature":1.0,"reasoning_tokens":2313,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:21:29.675660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold $K$ fixed at 256 and raise the sequence length from 128 to 1024 on a long-context corpus; if validation perplexity degrades far more steeply than a transformer baseline and reconstruction cosine similarity falls, the fixed global dictionary is not position-general. A second check would be to ablate the STFT loss and look at where reconstruction error concentrates: if errors cluster at high-frequency embedding dimensions, the frequency loss may be masking a capacity gap in the time-domain reconstruction.","supporting_citations":[{"cited_title":"Gomez, Łukasz Kaiser, and Illia Polosukhin","cited_arxiv_id":null,"evidence_quote":"Supplies the transformer attention mechanism and the quadratic $O(L^2)$ cost that SDGM sets out to replace."},{"cited_title":"Le, and Ruslan Salakhut- dinov","cited_arxiv_id":null,"evidence_quote":"Provides the Transformer-XL baseline whose validation perplexities SDGM reports closely matching."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"Provides the GPT-2 Small baseline that SDGM approaches in perplexity while reducing parameters by 80%."},{"cited_title":"K-svd: An algorithm for designing overcomplete dictionaries for sparse representation","cited_arxiv_id":null,"evidence_quote":"Supplies the dictionary-learning formulation that the paper adapts from sparse coding to continuous Fourier atoms."}],"review_version":1}