{"id":"e7511bd1-54a0-4cdc-8676-e1f721622f2b","arxiv_id":"2505.02266","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A deterministic Fourier expansion of token IDs plus a lightweight shared MLP matches learned embeddings on STS-B at a fraction of the parameter count in small transformers.","lead":"This paper replaces a transformer's learned word-embedding table with a fixed set of sine and cosine waves computed from each word's ID, followed by a small shared neural network. If it holds up, this could shrink model memory while keeping accuracy, but the evidence so far comes only from tiny proof-of-concept models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The untested premise that BPE ID order encodes usable structure is load-bearing; a token-permutation ablation would settle whether Fourier features, not just the shared MLP, drive the result.","rationale":"My reading aligns with the reader's weakest_assumption: the method's novelty and scaling rationale depend on the claim that frequency-ordered BPE IDs carry structure the Fourier basis can exploit. This premise is explicitly stated in Section 2, explicitly flagged as a hypothesis, and never directly tested. The near-collision analysis in Section 2 and the limitation in Section 4 make the premise even more load-bearing, because the Fourier features for adjacent IDs are provably very close at the tested vocabulary size and become closer as V grows. The MLP must therefore perform a high-dynamic-range separation task, and whether it can do so at scale is unknown. I also considered the training-time confound from the custom CUDA kernel and the uncontrolled Table 2 comparison against official BERT reports; these are genuine issues for secondary claims, but they do not threaten the core result as directly as the untested ordering premise. The paper is a proof-of-concept, and the current evidence supports only a conditional acceptance: the headline result is plausible but needs the permutation ablation to confirm the proposed mechanism. Since the reader's verdict is already CONDITIONAL and flags the same assumption, my assessment does not change the recommended verdict.","tokens_in":6999,"tokens_out":5908,"duration_ms":81674,"concrete_test":"Train the 2-layer, d=256 Fourier transformer exactly as in Table 1, but with one change: before normalization, apply a fixed random permutation π to token IDs, so x = 2π(p)/(V-1) - 1. Use the same hyperparameters, data, contrastive loss, and CUDA kernel, and report zero-shot STS-B Spearman over multiple seeds. If the score stays within about 1 point of the 76.38 reported for the unpermuted condition, the BPE-ordering premise is not needed and the Fourier basis is incidental to the result. If the score drops materially (say by more than 3 points), the frequency-order structure is genuinely exploited. A second arm replacing the Fourier basis with fixed random orthogonal features, keeping the same MLP and residual, would further separate the choice of basis from the effect of ID ordering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 5 rests on the Section 2 hypothesis that frequency-ordered BPE token IDs carry statistically exploitable structure. The Fourier basis is a fixed smooth map p -> T(p); if nearby IDs are not semantically related, T(p) is essentially a fixed deterministic projection of a scalar label, and the shared MLP is left to learn an arbitrary scalar-to-vector map. That task is plausible for 30k tokens but should degrade at larger vocabularies. The paper's own near-collision estimate quantifies the risk: adjacent normalized IDs differ by Δx = 2/(V-1), and the largest Fourier component difference for d=256, V=30,522 is bounded by about (d/2)πΔx ≈ 0.026. Neighboring base features are therefore almost identical, and the MLP must amplify tiny input differences; whether this remains feasible for hundreds of thousands of tokens is untested. Crucially, no experiment isolates whether the ordering matters: if token IDs are randomly permuted before normalization, the set of Fourier features is unchanged but any correlation between frequency order and semantics is destroyed. Without this control, the small gaps in Table 1 (e.g., 76.38 vs 77.34 for 2-layer d=256 models) could be explained by the MLP and residual connection performing the actual work, with the Fourier basis being incidental. The paper flags the near-collision issue in Section 4 but defers the needed validation to future work, leaving the mechanism behind the headline result unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the learned embedding lookup table in transformers with a deterministic Fourier expansion of normalized token IDs, followed by a shared lightweight MLP and a residual connection. The method is motivated by the hypothesis that frequency-ordered BPE token IDs carry statistically exploitable structure. The authors train standard transformers and their Fourier-embedding variant on SNLI and MNLI using a contrastive loss, evaluate zero-shot on STS-B, and report that the Fourier models achieve competitive performance with substantially fewer parameters and shorter training times. They also report fine-tuned PETE models in Table 2 with parameter counts as low as 58k, compared against official tiny-BERT baselines. The paper includes code and pre-trained weights.","tokens_in":7267,"tokens_out":6655,"duration_ms":79487,"significance":"If the core mechanism holds, the approach is relevant to memory-constrained and edge deployments, since embedding tables dominate parameter counts in large-vocabulary models. The paper has clear strengths: it ships reproducible code and weights, the zero-shot STS-B evaluation is a genuine out-of-distribution transfer after contrastive training on SNLI/MNLI, the Fourier basis is fixed and not tuned to the evaluation metric, and Table 1 provides a within-paper matched comparison under a shared training regime. The reported parameter reductions (e.g., 1.1m vs 8.9m parameters at comparable accuracy) are potentially valuable. However, the evidence as presented does not yet establish the central mechanism: the paper's own hypothesis about BPE ID ordering is untested, the reported gaps in Table 1 are within the range of run-to-run noise, and the universal approximation argument contains a mathematical gap. With additional controls and statistical rigor, the contribution could be solid; in its current form the claims outrun the evidence.","major_comments":[{"comment":"The central claim of competitiveness rests on differences of 1 to 3 Spearman points (e.g., 76.38 vs 77.34 for the 2-layer d=256 models, and 74.93 vs 77.01 for the 1-layer d=256 models). No error bars, confidence intervals, or repeated seeds are reported for any entry. In contrastive sentence-embedding training, seed-to-seed variance of this magnitude is typical, so the gaps may be within noise. I request repeated runs with at least three seeds and reporting of mean and standard deviation for the main Table 1 comparisons.","section":"Section 3, Table 1"},{"comment":"The paper's motivating hypothesis is that frequency-ordered BPE token IDs contain exploitable structure, and Section 2 explicitly states that the effectiveness of the fixed basis 'relies entirely' on this empirical hypothesis. However, no experiment isolates this factor. A token-permutation ablation—randomly permuting token IDs before normalization while keeping the Fourier features and MLP identical—would directly test whether the ordering matters. Without this control, the small gaps in Table 1 could be explained by the MLP and residual connection doing the actual work, with the specific Fourier ordering being incidental. Section 4 defers near-collision validation to future work, but the permutation test is a simple, load-bearing control that should be included now.","section":"Section 2 and Section 4"},{"comment":"The universal approximation argument invokes phi^{-1}(z), where phi(x)=T(p), but the Fourier map T is not injective on [-1,1]: at x=-1 and x=1, all sine terms vanish and all cosine terms coincide, so T(-1)=T(1) for every embedding dimension. Consequently phi^{-1} is undefined, and the final embedding E(p)=MLP(T(p))+T(p) provably cannot distinguish token IDs 0 and V-1. This is a concrete mathematical defect in the theoretical justification. The authors should either handle the endpoints separately (e.g., by excluding one endpoint or adding a small symmetry-breaking term) or weaken the universal approximation claim.","section":"Section 2, T_i(p) definition and universal approximation argument"},{"comment":"The fine-tuned comparison in Table 2 uses official reported baselines (BERT-Tiny, BERT-Mini, TinyBERT, etc.) without any control for task-specific heads, training data, optimizer settings, or fine-tuning protocol. The statement in Section 5 that 'in configurations with equal overall parameters, PETE even outperforms the traditional transformer' is not supported by this uncontrolled comparison. A controlled fine-tuning experiment under identical settings is needed before such a claim can be made.","section":"Section 3, Table 2"}],"minor_comments":[{"comment":"Clarify whether the index i in the Fourier basis definition is 0-based or 1-based, and state how odd d_model values are handled; the current definition is ambiguous for odd dimensions.","section":"Section 2"},{"comment":"The claim that 'applying dropout to these normalized continuous mappings disrupts the smooth progression of token IDs and degrades performance' is asserted without any supporting experiment. Either add a dropout ablation or present this as a qualitative observation rather than an empirical result.","section":"Section 2 and Section 3"},{"comment":"The text says the baseline and Fourier models differ 'only in the embedding layer,' but the baseline uses dropout 0.1 while the Fourier model omits dropout. This makes the Table 1 comparison confounded; the experimental setup should be stated precisely, or a dropout-matched baseline should be included.","section":"Section 3"},{"comment":"Training times are reported in minutes and hours without specifying the hardware configuration, whether times include data loading and evaluation, or how the custom CUDA kernel affects the comparison. Please provide these details.","section":"Section 3, Table 1"},{"comment":"References [1] (Bertschinger et al.) and [19] (Williams and Beer) are not cited in the body text, and references [17] and [21] have incomplete author lists ('Wang and Others'). The bibliography should be tidied.","section":"References"},{"comment":"The claim that PETE outperforms traditional transformers when total parameters are equal is not demonstrated by Table 1 (which compares different parameter counts) or Table 2 (which is uncontrolled). Either add a controlled equal-parameter experiment or remove this sentence.","section":"Section 5"},{"comment":"The abstract calls the embedding scheme 'parameter-free,' but the method includes a learnable MLP. Consider phrasing such as 'embedding-table-free' or 'deterministic base mapping' to avoid overstating the parameter reduction.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This is an honest proof-of-concept with clearly stated limitations and useful negative-space framing (e.g., the near-collision discussion in Section 4). The main blockers are the missing permutation ablation, the absence of repeated-seed statistics, and the mathematical issue with T(-1)=T(1). If the authors add the permutation control and rerun the main experiments with seeds, the contribution could become publishable; the current version, however, overclaims relative to its evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The one genuinely new thing is the empirical demonstration that a fixed Fourier expansion of normalized token IDs plus a shared MLP can get within a point or two of learned embeddings on zero-shot STS-B, at a fraction of the parameters. That result in Table 1 is plausible as far as it goes, and the authors are refreshingly honest about the toy scale and resource limits. They also release code, which makes this reproducible rather than just suggestive.\n\nThe soft spots are real but not fatal. First, there are no error bars or repeated seeds, so the 1–3 point gaps could easily be noise. Second, the dropout claim is asserted without an experiment — they just say dropout 'degrades performance' and move on. Third, Table 2 compares against official reported baselines, not models trained in the same controlled setup, so those numbers are soft. Fourth, the 'trains faster' claim is confounded by the custom CUDA kernel, which fuses normalization and Fourier expansion; that's a systems optimization, not a property of the method.\n\nThe biggest gap, though, is the load-bearing assumption from Section 2: that frequency-ordered BPE token IDs carry structure the Fourier basis can exploit. The stress-test note puts it well — a token-permutation ablation would settle whether the ordering matters, and without it, the shared MLP plus residual connection could be doing all the work, with the Fourier basis essentially incidental. The paper does flag the near-collision problem in Section 4, but it defers the validation to future work, which leaves the central mechanism unsupported.\n\nOn citation pattern: the embedding-compression literature is cited, but they miss the Fourier feature line of work (sinusoidal positional encodings, random Fourier features). That's not a fatal flaw, but it weakens the novelty claim — this is a direct transfer of a known technique, not a new construction.\n\nMy verdict: conditional. The idea could be interesting at scale, and the proof-of-concept is worth taking seriously. But the paper needs variance estimates, controlled baselines, and that permutation ablation before the central claim is solid. Who is this for? Someone working on parameter-efficient architectures or efficient serving might find it a useful starting point. I'd send it to peer review — a workshop or short paper track would be appropriate — because it's reproducible and the question is worthwhile, but I'd expect substantial revision.","headline":"A small, honest proof-of-concept that deterministic Fourier token embeddings can rival learned embeddings at toy scale; the mechanism is untested and the evidence is thin, but it deserves a referee.","tokens_in":7812,"tokens_out":1564,"would_cite":true,"duration_ms":21148,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a deterministic Fourier expansion of token IDs, followed by a small shared MLP, can replace the learned embedding table of a transformer without sacrificing task accuracy, while using a fraction of the parameters…","keywords":["parameter-efficient transformers","Fourier embeddings","token embeddings","byte-pair encoding","natural language inference","contrastive learning","sentence similarity","embedding compression"],"falsifier":"Shuffle the token IDs randomly before training, holding the vocabulary, model, and data fixed; if a Fourier-embedding transformer trained on shuffled IDs performs about the same as the unshuffled version, the ordering assumption is not doing the work, and if performance drops sharply, the frequency ordering is load-bearing.","tokens_in":6730,"feed_emoji":"📉","tokens_out":6343,"duration_ms":77352,"temperature":0.7,"pith_summary":"Transformers normally store a learned vector for every token, and that lookup table is usually the largest parameter block in the model. This paper proposes generating each token's embedding deterministically from its token ID, using a fixed Fourier expansion of a normalized ID followed by a small shared MLP, and presents evidence that this substitution matches a standard learned-embedding transformer on sentence-pair tasks while using a fraction of the parameters and less training time. The experiments train on SNLI and MNLI with a contrastive loss and evaluate zero-shot on STS-B; the smallest fine-tuned Fourier models outperform several officially reported tiny BERT variants. If the claim holds, embedding memory no longer needs to scale with vocabulary size, which would matter most for large language models.","feed_headline":"Fourier embeddings match learned tables at 1/8 the parameters","feed_subtitle":"A deterministic expansion of token IDs plus a small MLP replaces the embedding lookup and trains faster on NLI tasks.","key_machinery":"The central object is the Fourier expansion of normalized token IDs: each token ID $p$ is normalized to $x = 2p/(V-1) - 1$, and component $i$ of the base embedding is $T_i(p) = \\sin((\\lfloor i/2 \\rfloor + 1)\\pi x)$ for even $i$ and $\\cos((\\lfloor i/2 \\rfloor + 1)\\pi x)$ for odd $i$, with final embedding $E(p) = \\mathrm{MLP}(T(p)) + T(p)$. Low-order terms capture broad trends across the ID order, high-order terms separate nearby IDs, and the shared MLP is the learnable residual that separates tokens with near-colliding base representations. This turns the embedding layer from a stored $V \\times d$ matrix into a small learned function of a scalar.","core_discovery":"The paper's central claim is that a parameter-free, deterministic Fourier basis can serve as the token-embedding layer of a transformer, as long as a lightweight shared MLP and a residual connection are added to refine the fixed features. Concretely, each token ID $p$ is mapped to $x \\in [-1,1]$, its Fourier expansion up to the embedding dimension forms $T(p)$, and the final embedding is $E(p) = \\mathrm{MLP}(T(p)) + T(p)$. On the configurations tested, transformers using Fourier embeddings reached STS-B scores within roughly a point of standard transformers while using about one-eighth to one-quarter of the parameters, and training was faster; the paper's largest PETE model after fine-tuning reached a Spearman correlation of 81.9, above the official BERT-Tiny and BERT-Mini reports. The authors read this as support for the hypothesis that the frequency ordering of byte-pair-encoding token IDs carries weak but exploitable structure, which the fixed basis captures and the MLP corrects.","pith_inferences":["If the ordering assumption holds at scale, this design could make embedding layers essentially free in web-scale models, where vocabularies of hundreds of thousands of tokens dominate memory.","The smooth ID-to-vector map may give interpretability: probing how the final embedding evolves with token ID could expose the model's internal frequency-to-semantics organization.","A direct stress test is vocabulary randomization on larger models; if performance collapses, frequency ordering is load-bearing, and if it does not, the MLP alone is doing the work."],"forward_implications":["Embedding parameter count becomes independent of vocabulary size; adding tokens to the vocabulary no longer adds parameters to the embedding layer.","Because the embedding function is fixed and smooth, models can be trained without dropout, and custom kernels that fuse normalization and Fourier expansion cut training time.","Freed parameter budget can be moved from the embedding table into attention depth or width, which is where the paper observes the best returns.","At equal total parameter counts, the Fourier-embedding transformers matched or exceeded the traditional transformer on STS-B, suggesting learned tables waste capacity at small scale."],"supporting_citations":[{"why":"Supplies the BERT tokenizer whose frequency-ordered vocabulary defines the token IDs used for normalization.","marker":"[4]"},{"why":"Provides SNLI, one of the two natural language inference training sets.","marker":"[2]"},{"why":"Provides MNLI, the second natural language inference training set.","marker":"[18]"},{"why":"Inspires the contrastive training objective used to pull matching sentence pairs together.","marker":"[8]"},{"why":"Supplies the InfoNCE-style contrastive loss formulation used for training.","marker":"[16]"},{"why":"Provides the BPE subword-tokenization rationale that frequency-ordered IDs carry entropy-relevant statistical information.","marker":"[10]"}],"fun_headline_variants":["Fourier transform removes the embedding table","Embeddings from sine waves, not lookups","No embedding table: use Fourier features","Deterministic embeddings replace trained tables","Sine-wave embeddings save 8x parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the premise that the frequency-based ordering of BPE token IDs correlates with useful embedding structure; the paper itself warns that for very large vocabularies the normalized ID space becomes densely packed, so distinct tokens' base Fourier embeddings nearly collide and the MLP must separate a dense field of near-identical inputs, which has not been demonstrated at scale.","fun_headline_variants_meta":{"raw":{"variants":["Fourier transform removes the embedding table","Embeddings from sine waves, not lookups","No embedding table: use Fourier features","Deterministic embeddings replace trained tables","Sine-wave embeddings save 8x parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000842,"raw_usage":{"total_tokens":3640,"prompt_tokens":889,"completion_tokens":2751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":2686}},"tokens_in":505,"tokens_out":2751,"duration_ms":27526,"temperature":1.0,"reasoning_tokens":2686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:56:51.486444+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the token IDs randomly before training, holding the vocabulary, model, and data fixed; if a Fourier-embedding transformer trained on shuffled IDs performs about the same as the unshuffled version, the ordering assumption is not doing the work, and if performance drops sharply, the frequency ordering is load-bearing.","supporting_citations":[{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understan ding","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT tokenizer whose frequency-ordered vocabulary defines the token IDs used for normalization."},{"cited_title":"A large an- notated corpus for learning natural language inference","cited_arxiv_id":null,"evidence_quote":"Provides SNLI, one of the two natural language inference training sets."},{"cited_title":"A br oad-coverage challenge cor- pus for sentence understanding through inference","cited_arxiv_id":null,"evidence_quote":"Provides MNLI, the second natural language inference training set."},{"cited_title":"Represe ntation learning with contrastive predictive coding","cited_arxiv_id":null,"evidence_quote":"Supplies the InfoNCE-style contrastive loss formulation used for training."},{"cited_title":"Neur al machine translation of rare words with subword units","cited_arxiv_id":null,"evidence_quote":"Provides the BPE subword-tokenization rationale that frequency-ordered IDs carry entropy-relevant statistical information."}],"review_version":1}