{"id":"d8caa39c-2ea3-45cd-8b0e-b40115190b4d","arxiv_id":"2506.06398","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Wavelet-based positional encodings are claimed to improve how transformers extrapolate to longer sequences, with a toy experiment supporting the claim but with weak theory.","lead":"Positional encodings tell transformers where each token sits in a sequence. This paper proposes new encodings built from wavelets and Legendre polynomials, claims a theory of accuracy and extrapolation, and reports that wavelet encodings beat sinusoidal ones on a toy task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wavelet extrapolation bound in §6.1.4 contradicts compact support: claimed exponential decay is actually an abrupt drop to zero, so the theoretical basis for wavelet extrapolation advantage collapses.","rationale":"The reader's weakest assumption identifies the same load-bearing flaw: the exponential decay assumption in §6.1.4 is incompatible with compactly supported Daubechies-4 wavelets. This is not a mere technical gap; it breaks the theoretical bridge from the encoding construction to the claimed extrapolation advantage. The empirical results, even if reproducible, would not compensate for the loss of the paper's central theoretical contribution, because the headline claim explicitly asserts a theoretical framework that explains and predicts the superiority of orthogonal-transform encodings. With that framework invalidated, the paper does not establish its central claim; the verdict should remain REJECT, i.e., unchanged from the reader's verdict.","tokens_in":14918,"tokens_out":2293,"duration_ms":29535,"concrete_test":"Implement the exact wavelet encoding from §6.1.1 with Daubechies-4, J=5, and Nmax=50. For positions pos = 51, 52, …, 200, compute ∥PEwavelet(pos) − PEwavelet(Nmax)∥ (both raw and normalized as in §6.1.3). If the difference does not follow the exponential decay predicted by §6.1.4 but instead becomes zero or constant once pos exceeds the support of all basis functions, the bound is false and the theoretical extrapolation claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that wavelet encodings extrapolate better than sinusoidal relies on the bound in §6.1.4, which asserts |ψ_{j,k}(pos)| ≤ C 2^{-j/2} exp(−β(pos − Nmax)/2^{-j}) for pos > Nmax. This is internally inconsistent with the Daubechies-4 wavelets used in §7, which are compactly supported: for any finite-support basis function, ψ_{j,k}(pos) is exactly zero once pos exceeds the support limit, not exponentially decaying. Thus the difference ∥PEwavelet(pos) − PEwavelet(Nmax)∥ does not decay smoothly as claimed; it stays constant after an abrupt drop (and, if all coefficients vanish, the normalized encoding is 0/0, undefined). The paper provides no argument that a finite wavelet basis defined on [0,Nmax] yields the claimed exponential tail outside that interval. Since this bound is the only theoretical justification for the wavelet extrapolation advantage, its failure removes the theoretical support for the paper's headline claim. The empirical result in Table 5 may still hold, but it is unexplained by the theory and is based on an unreproducible toy experiment (no code, seeds, or error bars).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a theoretical framework for analyzing positional encodings in transformers, with claims about expressiveness, Rademacher-complexity generalization bounds, and extrapolation to longer sequences. It introduces two new encoding schemes based on wavelets and Legendre polynomials, and reports a lightweight NumPy experiment on a synthetic running-sum task which the authors interpret as showing that wavelet encodings outperform sinusoidal and ALiBi encodings when extrapolating to sequences of length 100 and 200. The central theoretical claim is that orthogonal-transform encodings, especially wavelet-based ones, provide better extrapolation than sinusoidal encodings.","tokens_in":15316,"tokens_out":6823,"duration_ms":66883,"significance":"The topic is timely and important, and the paper is clearly organized. The idea of using wavelet and Legendre bases as positional encodings is interesting and the synthetic running-sum task is a sensible testbed for extrapolation. If the theoretical results were correct, the paper would offer useful guidance for choosing positional encodings in long-context transformers. However, the main theoretical justification for the wavelet extrapolation advantage is invalid: it relies on an exponential-decay assumption that contradicts the compact support of the Daubechies wavelets used in the experiments. In addition, the generalization bounds in Section 4 are mostly asserted rather than derived, and the ALiBi extrapolation bound in Section 5 is essentially a restatement of a Lipschitz assumption. The experimental evidence is a single run with no error bars, seeds, or code, which is too weak to support the headline empirical claim.","major_comments":[{"comment":"This bound is asserted for Daubechies-4 wavelets, which are compactly supported. For any compactly supported wavelet, ψ_{j,k}(pos) is exactly zero outside a finite interval, not a smooth exponential tail. The inequality is trivially true where the wavelet is zero, but it does not imply that ‖PE_wavelet(pos) − PE_wavelet(Nmax)‖₂ decays as O(exp(-β(pos−Nmax))); the actual difference is step-like, becoming constant after an abrupt drop and ultimately zero once all basis functions have exited their support. For the normalized encoding ĥPE_wavelet(pos) = PE_wavelet(pos)/‖PE_wavelet(pos)‖₂, positions beyond all supports yield a zero vector and the normalized encoding is undefined. Since this extrapolation bound is the only theoretical argument for the wavelet advantage claimed in Section 7, the paper's central theoretical claim is unsupported.","section":"Section 6.1.4, displayed bound |ψ_{j,k}(pos)| ≤ C 2^{-j/2} exp(-β(pos-Nmax)/2^{-j})"},{"comment":"The bounds R_m(F) = O(L_trans B/√m) are introduced through a 'standard covering-number argument', but no covering-number calculation, no chain rule for Rademacher complexity, and no dependence on depth, width, or number of parameters is given. The section does not state a theorem; it simply writes order-of-magnitude expressions. As a result, the comparison in Table 2 reduces to comparing input norm bounds, and the claimed contribution on how PEs affect generalization is not established beyond a generic Lipschitz argument.","section":"Section 4.2, Rademacher complexity bounds"},{"comment":"The bound |A(µ, −αd) − A(µ, −αNmax)| ≤ L_A α(d − Nmax) is an immediate consequence of assuming that A is Lipschitz in its second argument; it does not explain why the learned attention function of a transformer should have this Lipschitz property, nor does it connect this local statement about attention weights to an end-to-end guarantee on sequence-level extrapolation error. The 'unified extrapolation framework' is therefore a restatement of the assumption rather than a theoretical derivation, and it does not distinguish ALiBi from other Lipschitz biases.","section":"Section 5.3, Eq. (9)"},{"comment":"The paper claims that ALiBi yields universal approximation for arbitrary sequence length, marked 'Yes' in the 'Universal (Any N)' column of Table 1, but no construction or proof is provided. This claim is in tension with Section 3.4, which acknowledges that purely relative encodings may fail when absolute position information is required. Because ALiBi's bias depends only on |i−j|, it is not clear how it can represent absolute positions; without a proof, the claim is unsupported and appears inconsistent with the paper's own discussion of relative encodings.","section":"Section 3.5 and Table 1"},{"comment":"The experimental support for the headline claim consists of a single run on a toy running-sum task with no error bars, no seeds, no code, and no ablations. The encoder is a minimal 2-layer transformer without layer normalization or dropout, and the implementation details of the wavelet basis selection (e.g., which 64 of the available coefficients are kept) are not fully specified. Consequently, the conclusions in Section 7.3 that wavelet encodings 'retain best performance' and that results 'align with theoretical extrapolation bounds' are not empirically justified.","section":"Section 7, Table 5"}],"minor_comments":[{"comment":"The text defines δ_pos as 'the Dirac delta at position pos' and then writes ⟨δ_pos, ϕ⟩ = ϕ(pos); for integer positions this should be phrased as a Kronecker delta or as an evaluation functional on a continuous domain, since the Dirac delta is a distribution.","section":"Section 6.1.1"},{"comment":"In the Taylor expansion, the approximation e^{-2γΔpos/Nmax} ≈ 1 − 2γΔpos/Nmax requires Δpos ≪ Nmax/(2γ), but this condition is not stated before the bound is used to claim an extrapolation range up to O(Nmax/d^{2.5}_{model}).","section":"Section 6.2.4"},{"comment":"The statement that 'wavelet basis functions extend beyond any finite interval (though they decay)' is imprecise for the Daubechies wavelets used later, which are compactly supported; this wording contributes to the confusion about the extrapolation analysis.","section":"Section 5.4, first paragraph"},{"comment":"The paper uses 'ALiBi' as a positional encoding in the abstract and Section 2, but ALiBi is a bias applied to attention logits rather than an additive position embedding; the distinction should be made explicit earlier, especially because the generalization-bound comparison in Table 2 treats it as an input-norm effect.","section":"Section 2.4 and elsewhere"}],"recommendation":"reject","confidential_remarks":"The paper is not ready for publication. The primary theoretical contribution rests on a false assumption about the decay of compactly supported wavelets, and the generalization analysis is too informal to be load-bearing. The experiment is a single run without error bars or code, so it cannot compensate for the theoretical problems. I would not recommend major revision unless the authors replace the extrapolation analysis with a correct treatment and substantially strengthen the empirical evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes wavelet- and Legendre-based positional encodings and argues, both theoretically and with a toy experiment, that they extrapolate better than sinusoidal PEs. The idea is genuinely interesting and the toy results are suggestive: wavelet beats ALiBi slightly at 2x and 4x length on a running-sum task. If the extrapolation advantage holds in real settings, it would be a useful addition to the PE toolbox.\n\nThe paper also does a reasonable job of collecting known expressiveness and generalization results into a single framework. The universal approximation claims and the Rademacher scaling arguments are not new, but the presentation is organized and readable.\n\nThe soft spot is where it matters. Section 6.1.4 claims that wavelet basis functions decay exponentially beyond Nmax, giving a bounded difference between PE(pos) and PE(Nmax). But the Daubechies-4 wavelets used in the experiments are compactly supported: beyond their support they are exactly zero, not exponentially decaying. The actual behavior is an abrupt cutoff, and for positions beyond the largest support edge the encoding vector becomes zero. The claimed smooth exponential bound is the theoretical justification for the wavelet extrapolation advantage, so this flaw removes the paper's main theoretical support. The empirical result may still be real, but it is currently unexplained.\n\nOther issues are less severe but real. Section 4's Rademacher bounds are asserted with scaling arguments rather than derived; they are plausible but not proven. The experiments are a single tiny task with no code, seeds, or error bars, and the wavelet vs ALiBi differences are small enough to be noise. A few references are misattributed or irrelevant, which suggests the literature review was not careful.\n\nThe right audience for this paper is a workshop on position encodings, not a top venue as is. The core question matters, and the proposed encodings deserve investigation, but the theory needs a correct extrapolation analysis (possibly based on the actual support structure rather than a claimed exponential tail), the generalization bounds need real derivations, and the experiments need to be reproducible and include variance. I would send it to peer review because the idea is worth refereeing, but I would expect a major revision or a reject. A serious referee could help the author identify the wavelet issue and refocus the paper.","headline":"Interesting wavelet and Legendre PE proposals, but the main theoretical bound for wavelet extrapolation is internally inconsistent and the toy experiments are too weak to carry the claim.","tokens_in":15668,"tokens_out":3366,"would_cite":false,"duration_ms":34764,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","42C40","41A10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that wavelet-based positional encodings let transformers extrapolate to longer sequences better than sinusoidal encodings, with a unified theory of expressiveness, generalization, and extrapolation.","keywords":["positional encodings","transformers","extrapolation","wavelet transforms","Rademacher complexity","ALiBi","Legendre polynomials","expressiveness"],"falsifier":"Compute the exact normalized Daubechies-4 wavelet encodings used in the paper for positions $N_{\\max}+1$ through $N_{\\max}+100$ and measure $\\|PE_{\\text{wavelet}}(pos) - PE_{\\text{wavelet}}(N_{\\max})\\|_2$. If the difference is exactly zero, or normalization becomes undefined, once $pos$ leaves the support of every basis function, then the claimed exponential-decay extrapolation bound has no object to apply to; if a correctly normalized encoding still satisfies the bound, the premise is confirmed.","tokens_in":14692,"feed_emoji":"🧭","tokens_out":8364,"duration_ms":74987,"temperature":0.7,"pith_summary":"This paper tries to put positional encodings in transformers on a common theoretical footing, treating expressiveness as function approximation, generalization as Rademacher complexity, and extrapolation as behavior past the training length. Its central claim is that encodings built from orthogonal transforms, especially wavelets, can beat the standard sinusoidal encoding for both generalization and extrapolation. If true, this matters because transformer users could choose position schemes by predicted length behavior instead of by tradition. The supporting experiments are deliberately small, but they align with the theory on a running-sum task.","feed_headline":"Wavelet encodings beat sinusoidals on longer inputs","feed_subtitle":"Theory ties expressiveness, generalization, and extrapolation to the choice of position encoding; toy tests back it.","key_machinery":"The central object is an encoding vector formed by evaluating an orthonormal wavelet basis, specifically Daubechies-4 scaling and wavelet functions at integer positions, at scales up to $\\lfloor \\log_2(N_{\\max}) \\rfloor$. The mechanism doing the work is multi-resolution locality: fine-scale wavelets have small supports and disappear for positions beyond $N_{\\max}$, while coarse scales continue to carry position information, so the normalized encoding changes little past the training length. The argument also relies on Rademacher complexity, a standard capacity measure, to bound generalization, and on a Lipschitz-continuity model of the attention function to bound ALiBi's extrapolation error.","core_discovery":"The paper claims that positional encodings are not neutral: they determine whether a transformer can keep working when the input is longer than anything seen in training. It unifies sinusoidal, learned, relative, ALiBi, wavelet, and Legendre encodings under one framework, with expressiveness defined by universal approximation, generalization bounded by Rademacher complexity, and extrapolation measured by how the encoding vector changes for positions beyond $N_{\\max}$. The specific new claim is that wavelet encodings preserve coarse positional information after fine-scale components vanish, so their vector remains close to the encoding at $N_{\\max}$; on the synthetic running-sum task, the wavelet encoding achieves lower MSE than sinusoidal at two and four times the training length, and slightly lower than ALiBi at both. The paper also derives that ALiBi's linear bias gives graceful but linearly growing extrapolation error, and that tanh-saturated Legendre encodings collapse toward a constant vector for very long positions.","pith_inferences":["Not proven by the paper, but suggested: the wavelet advantage would need a proof not based on exponential decay, since Daubechies-4 wavelets have compact support; one route is to treat the exact zero tail as a truncation and show a continuation rule preserves coarse position.","A testable extension the paper does not run: combine wavelet position vectors with an ALiBi-style bias and compare on long-context language or vision tasks rather than the synthetic running-sum task.","If the multi-scale mechanism is what drives extrapolation, other bounded high-frequency feature maps, such as truncated Fourier or random features, should show similar length robustness, a prediction the paper leaves implicit."],"forward_implications":["If the theory holds, practitioners who need to test on sequences longer than training should prefer wavelet or ALiBi encodings over sinusoidal or learned absolute encodings.","Normalized wavelet encodings should not hurt interpolation accuracy, because the Rademacher bound shows the same input-norm capacity as sinusoidal encodings.","Legendre encodings with the tanh map are safe only for modest extrapolation; beyond roughly $N_{\\max} + O(N_{\\max}/d_{\\text{model}}^{2.5})$ they collapse toward a constant vector.","ALiBi's extrapolation error grows linearly with the distance beyond $N_{\\max}$, so the slope should be chosen with the expected test length in mind."],"supporting_citations":[{"why":"Introduces the transformer architecture and the sinusoidal positional encoding that serves as the main baseline.","marker":"Vaswani et al., 2017"},{"why":"Introduces ALiBi, the bias-based method whose extrapolation behavior the paper formalizes and extends.","marker":"Press et al., 2021"},{"why":"Supplies relative position embeddings, one of the encoding families compared in the expressiveness analysis.","marker":"Shaw et al., 2018"},{"why":"Establishes universal approximation for transformers, which the expressiveness analysis leans on.","marker":"Yun et al., 2019a"},{"why":"Provides the Rademacher-complexity generalization bounds used in Section 4.","marker":"Bartlett and Mendelson, 2002"},{"why":"Provides the covering-number argument linking Lipschitz constants to generalization bounds.","marker":"Neyshabur et al., 2018"},{"why":"Is the informal blog proposal of wavelet- and polynomial-based encodings that the paper formalizes and tests.","marker":"Li, 2021"}],"fun_headline_variants":["Wavelets keep transformers working on longer inputs","Position encodings decide extrapolation: wavelets win","Theory shows wavelet positions beat sinusoidals on long sequences","Why wavelets beat sinusoidals for long-input transformers","Wavelets extend transformer reach beyond training length"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is the exponential-decay bound in Section 6.1.4 for wavelet values beyond the training length; the Daubechies-4 wavelets used in the experiments are exactly zero outside their support, so the bound cannot be true for them.","fun_headline_variants_meta":{"raw":{"variants":["Wavelets keep transformers working on longer inputs","Position encodings decide extrapolation: wavelets win","Theory shows wavelet positions beat sinusoidals on long sequences","Why wavelets beat sinusoidals for long-input transformers","Wavelets extend transformer reach beyond training length"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2837,"prompt_tokens":916,"completion_tokens":1921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1846}},"tokens_in":532,"tokens_out":1921,"duration_ms":12074,"temperature":1.0,"reasoning_tokens":1846,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:39.128947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the exact normalized Daubechies-4 wavelet encodings used in the paper for positions $N_{\\max}+1$ through $N_{\\max}+100$ and measure $\\|PE_{\\text{wavelet}}(pos) - PE_{\\text{wavelet}}(N_{\\max})\\|_2$. If the difference is exactly zero, or normalization becomes undefined, once $pos$ leaves the support of every basis function, then the claimed exponential-decay extrapolation bound has no object to apply to; if a correctly normalized encoding still satisfies the bound, the premise is confirmed.","supporting_citations":[{"cited_title":"and Mendelson, S","cited_arxiv_id":null,"evidence_quote":"Provides the Rademacher-complexity generalization bounds used in Section 4."}],"review_version":1}