Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

CoPE: A Lightweight Complex Positional Encoding

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CoPE stores position in the imaginary part of a complex embedding, and its first-layer phase-aware attention removes the long-term decay that standard encodings impose.

desk verdict CoPE's central no-decay claim rests on an unjustified proportionality and the experiments are too thin to support 'superior performance'—the idea is plausible but the current proof and evidence do not hold up. read the letter →

arxiv 2508.18308 v1 pith:WJT3Y47X submitted 2025-08-23 cs.LG cs.AI

Avinash Amballa This is my paper
classification cs.LGcs.AI
keywords complexpositionalencodingphase-awareattentionlong-termdecaylineartransformerGLUEbenchmarkrotarypositionembeddingsequencemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoPE claims that a transformer can carry positions in the imaginary part of its input embeddings and keep semantic content in the real part, avoiding the interference of additive positional encodings. The paper introduces a first-layer phase-aware attention that computes complex query-key overlaps, then reduces them to real scores through magnitude, phase, real, or hybrid variants; deeper layers stay standard. It proves that the positional term in these scores is oscillatory rather than exponentially decaying, and that the same mechanism can be rewritten as four real inner products, making it compatible with linear attention. On three GLUE tasks with a small model trained from scratch, the best CoPE variant matches or beats RoPE, sinusoidal, and learned embeddings while doing complex operations in only one layer instead of every layer.

What carries the argument

The complex input embedding E_complex = E_vocab + iγ sin(ω·pos), which stores content and position in orthogonal real and imaginary axes. The load-bearing mechanism is phase-aware first-layer attention: complex-valued query and key projections, a Hermitian inner product, and reduction of the complex scores to real values via one of five scoring rules (magnitude, phase cosine, real part, hybrid, hybrid-norm). The no-decay claim rides on the identity sin(ωp)sin(ωq) = ½[cos(ω(p−q))−cos(ω(p+q))], whose relative-distance term is oscillatory. The linear-attention compatibility claim rides on doubling real and imaginary features and decomposing the complex kernel into four real inner products that

What would settle it

Train the smallest CoPE model on SST-2, freeze it, and for a fixed query token plot the average attention weight assigned to the same token placed at every relative distance from 1 to 256. If the curve falls systematically with distance, for example exponentially, the claimed absence of long-term decay is false. A cheaper calculation-level check is to compute Q_proj·sin(ωp)·K_proj·sin(ωq) with the trained matrices and compare it with sin(ωp)sin(ωq); large deviations break Eq. (8), on which the no-decay conclusion rests.

Watch

Extended reading notes

Core claim

The paper's central object is the complex embedding z(x,p) = e_x + iγ sin(ωp), with token semantics in the real part and position in the imaginary part. In the first attention layer, queries and keys are obtained by complex-linear projections, and the attention score is the Hermitian inner product Q·K*. The positional part of this score expands to γ²[Q sin(ωp)·K sin(ωq)], which the paper treats as proportional to sin(ωp)sin(ωq) and rewrites as ½[cos(ω(p−q))−cos(ω(p+q))]. Because the relative term cos(ω(p−q)) has no multiplicative decay factor, the paper concludes CoPE does not impose long-term decay. It further shows that by lifting complex queries and keys to doubled real features, the atte

Load-bearing premise

The no-decay proof assumes that the learned projection matrices scale the sinusoidal position vectors like numbers, so the positional part of an attention score is just a constant times sin(query position) times sin(key position); if the projections mix position with content, the decay-free conclusion does not follow.

Editorial extensions

If this is right

  • A transformer built with CoPE can attend to distant positions without the built-in penalty that RoPE and similar encodings impose, so retrieval from arbitrary context positions is not discounted by distance.
  • CoPE can be plugged into linear-attention models: the same first layer gives relative position signal while retaining O(N) complexity, not the O(N²) of full attention.
  • Replacing RoPE's per-layer rotations with a single complex first layer cuts the positional-transform cost by roughly a factor of L, the number of layers.
  • The real/imaginary split gives a clean separation of content and position, so downstream layers receive already-position-aware representations without additional positional bias.
  • The variant list (magnitude, phase, real, hybrid, hybrid-norm) gives model builders a dial for how much phase difference should control attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the no-decay claim would be to train the smallest CoPE model, freeze it, and plot average attention weight versus relative distance for a fixed query token; if that curve falls systematically with distance, the claimed absence of long-term decay does not hold in the final attention distribution even if the raw score term is oscillatory.
  • Because the imaginary part uses sinusoidal encoding, CoPE may inherit sinusoidal extrapolation failures beyond the training length; the paper lists this as a limitation, but the phase-aware first layer could behave differently and is worth testing head-to-head against ALiBi.
  • The phase and magnitude variants win different tasks, so the practical value may lie in the architectural separation of content and position rather than in any single scoring rule; comparing CoPE on long-document retrieval would clarify which variant to use.
  • CoPE's complex first layer could be combined with existing linear-attention long-context models without per-layer changes, a testable route to long-context efficiency that the paper does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CoPE, a complex-valued positional encoding that separates token content (real part) from sinusoidal position information (imaginary part). A phase-aware attention mechanism is applied only in the first transformer layer, with five variants for reducing complex attention scores to real scores. The authors claim CoPE does not exhibit long-term decay, encodes both relative and absolute positions, is compatible with linear attention, and achieves superior performance with lower computational cost than RoPE, Sinusoidal, and Learned positional encodings. Experiments are reported on three GLUE tasks (SST-2, MRPC, QNLI) using a small 6-layer transformer trained from scratch.

Significance. If the central claims were established, CoPE would be a genuinely useful contribution: it is a simple first-layer adapter that avoids the long-term decay bias and is compatible with linear attention, while the linear-attention derivation in Section 3.3 is explicit and appears internally consistent. However, the proof of the no-decay property rests on an unjustified proportionality, and the experimental evidence consists of single runs with best-of-five variant selection and no significance testing. The empirical superiority and computational advantage claims are therefore not supported. The idea is interesting and clearly presented, but the load-bearing theoretical and empirical support is missing. No code or reproducibility artifacts are provided.

major comments (4)
  1. [Section 3.3, Eq. (8)] The no-decay theorem rests on the assertion Qproj sin(ωp) · Kproj sin(ωq) ∝ sin(ωp) sin(ωq). This is neither proved nor generally true. Qproj and Kproj are arbitrary learned complex matrices, and sin(ωp) in Eq. (1) is a scalar; a matrix applied to a scalar is undefined unless sin(ωp) is implicitly broadcast to a vector, and even then a generic learned matrix will mix the positional sinusoid with content dimensions. The proportionality holds only under strong constraints (e.g., scalar or diagonal projections, or projections that preserve the sinusoidal subspace), and no such constraint is stated or enforced in Section 3.2. Since Eqs. (9)-(10) and the claimed absence of multiplicative distance decay follow only from Eq. (8), the central theoretical contribution is unsupported. This is an internal gap, not a disagreement with the long-term-decay literature.
  2. [Section 3.3, Property 2] The claim that phase-aware attention encodes both relative and absolute positions relies on the same Eq. (8) proportionality. In the actual computation, Acomplex includes content-position cross terms (Eqs. (5)-(6)) that are not oscillatory functions of p−q alone. Unless Qproj and Kproj are constrained, the phase of Acomplex contains content-dependent contributions, so the phase difference is not purely positional. The relative-position interpretation is therefore not justified by the given derivation.
  3. [Section 4, Table 1] The empirical claim of superior performance is not supported. Each configuration appears to be a single run with no seeds, no error bars, and no significance tests, and the 'best' CoPE variant is selected per dataset from five variants, inflating the chance of favorable comparisons. The reported gaps are small (e.g., SST-2: CoPE phase 82.57 ties sinusoidal; MRPC: 81.71 vs. 81.55 for learned; QNLI: CoPE magnitude 61.63 is second-best behind 63.87 sinusoidal). Under these conditions, the results do not establish superiority over RoPE, sinusoidal, or learned encodings. The authors' own Section 5 limitation statement, noting the evaluation is limited to small from-scratch models, further undermines the broad claims in the abstract and conclusion.
  4. [Section 3.4] The claim that CoPE is 'L times faster than RoPE' is an order-of-magnitude argument that omits constant factors and implementation costs. RoPE's rotation is a relatively cheap operation applied to queries/keys in every layer, whereas CoPE replaces the first layer's attention with complex projections and phase-aware score computation; no wall-clock measurements are provided. The asymptotic comparison O(L·N·H·T·dk) vs. O(N·H·T·dk) ignores that the first complex layer is more expensive than a standard attention layer and that the remaining layers still run standard attention. Thus 'less computational complexity' in the abstract is not established.
minor comments (5)
  1. [Throughout] There are numerous typographical errors: 'transfomer', 'propogate', 'postional', 'Sof tmax', 'isclosely', 'i.e,'. These should be corrected.
  2. [Section 3.2, variants 4-5] The formulas for Hybrid and Hybrid-norm are missing explicit division by sqrt(dk) and parentheses are ambiguous. For example, 'Areal = (Amagnitude + α · cos(Aphase))√dk' should presumably be '... /√dk'. Please clarify.
  3. [Section 3.3, Eq. (17)] Equation (17) is an empty line; remove or number the subsequent equations correctly.
  4. [Section 4] The experimental setup does not report random seeds, number of runs, or standard deviations. The choice of α=0.2 and γ=1 is not justified, and no sensitivity analysis is provided.
  5. [Figures 1-3] The figure legends are hard to read in the text (e.g., 'Complex Valued phase Complex Valued magnitude Complex Valued hybrid norm ROPE'). Please ensure the captions and legends are self-contained and legible.

Circularity Check

1 steps flagged · score 2.0 of 10

The no-decay theorem restates the sinusoidal definition; otherwise the paper is self-contained and externally benchmarked.

  1. self definitional [Section 3.3, Eq. (1) and Eqs. (8)-(10)]
    "Reformulating our definition of the complex positional embedding for token x at position p as z(x, p) = ex + i γ sin(ωp), ... Qproj sin(ωp) · Kproj sin(ωq) ∝ sin(ωp) sin(ωq) ... Thus, the positional contribution ... ∝ cos (ω(p − q)) − cos (ω(p + q)) ... has no multiplicative decay factor such as e−α|p−q|."

    The no-long-term-decay conclusion is not an emergent consequence of the complex architecture; it is a restatement of the chosen sinusoidal imaginary component. Because the positional signal is defined as a non-decaying sine (Eq. (1)), the trigonometric identity sin(ωp)sin(ωq) = (cos(ω(p−q)) − cos(ω(p+q)))/2 used in Eqs. (8)-(10) guarantees an oscillatory relative-position term by construction. The claimed 'property' is therefore equivalent to the definition of the encoding rather than an independent derived prediction. Separately, Eq. (8)'s proportionality is asserted rather than proven for the learned Qproj/Kproj, but that is a correctness gap, not circularity.

full rationale

The paper's empirical contribution is evaluated against external baselines (RoPE, Sinusoidal, Learned) on standard GLUE tasks, so the performance claims are not circular. No parameter is fitted to a subset of data and then renamed as a prediction; no self-citation chain is load-bearing; and no uniqueness theorem is imported from the authors' prior work. The only circularity-adjacent element is the no-long-term-decay proof in Section 3.3: the result follows directly from defining the imaginary positional component as a sine wave, so the theorem is essentially the chosen encoding's functional form. This is a modest self-definitional burden, not a high one. The limitations section honestly notes that extrapolation and large-scale pretraining/finetuning are not yet evaluated, which further supports a low score. Overall the central derivation has independent trigonometric content, and the empirical comparisons are external, so the appropriate score is 2.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

CoPE is an architectural design; it introduces no new physical entity, mediator, or conserved quantity. The proposed mechanism has no independent falsifiable handle outside the model itself, so nothing meets the invented-entity criteria for external evidence.

free parameters (3)
  • alpha = 0.2
    Phase coefficient in Hybrid and Hybrid-norm attention; manually set, controls balance between magnitude and phase similarity.
  • gamma = 1
    Scales the imaginary positional component in Eq (1); set by hand without reported tuning.
  • omega = not reported (assumed standard sinusoidal base)
    Base angular frequency for imaginary sinusoidal position; determines periodicity and decay properties; not learned from data and not reported.
assumptions (5)
  • standard math sin(omega p) sin(omega q) = 1/2[cos(omega(p-q)) - cos(omega(p+q))]
    Used in Eq (9), Section 3.3, a standard trigonometric identity.
  • domain assumption Positional component is a scalar gamma sin(omega p) added to the imaginary part of every token embedding
    Eq (1); dimensionally problematic because token embeddings are vectors, and no vector positional form is specified.
  • ad hoc to paper Qproj sin(omega p) dot Kproj sin(omega q) is proportional to sin(omega p) sin(omega q)
    Eq (8); assumes learned complex projection matrices act as scalar multipliers on the sinusoidal position signal, which is not generally true.
  • domain assumption Complex-valued attention scores mapped via magnitude, phase, real, or hybrid variants are legitimate softmax logits
    Section 3.2 introduces five mapping variants with no theoretical or empirical comparison of their validity.
  • domain assumption Linear attention with doubled real features and real denominator preserves O(N) complexity and model quality
    Section 3.3; the derivation is algebraic, but no experiments verify that the complex kernel keeps task performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoPE: A Lightweight Complex Positional Encoding." pith.science (2026). https://pith.science/paper/WJT3Y47X

@misc{pith2026250818308,
  author       = {Pith},
  title        = {Pith review of: CoPE: A Lightweight Complex Positional Encoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJT3Y47X}},
  note         = {Machine review of arXiv:2508.18308}
}
read the original abstract

Recent studies have demonstrated the effectiveness of position encoding in transformer architectures. By incorporating positional information, this approach provides essential guidance for modeling dependencies between elements across different sequence positions. We introduce CoPE (a lightweight Complex Positional Encoding), a novel architecture that leverages complex-valued encoding to encode both content and positional information. Our approach replaces traditional positional encodings with complex embeddings where the real part captures semantic content and the imaginary part encodes positional information. We introduce phase-aware attention in the first layer of the transformer model to capture position-dependent patterns, followed by standard attention layers for higher-levels. We show that CoPE doesn't exhibit long term decay and is compatible with linear attention. Experimental evaluation on the GLUE benchmark suggest that our approach achieves superior performance with less computational complexity, compared to RoPE, Sinusoidal and Learned positional encodings.

Figures

Figures reproduced from arXiv: 2508.18308 by the authors.

Figure 1
Figure 1. Training loss vs. epochs on SST2 with CoPE vs RoPE. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Training loss vs. epochs on MRPC with CoPE vs RoPE. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Training loss vs. epochs on QNLI with CoPE vs RoPE. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

  2. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 reject novelty 6.0 of 10

    AFA re-derives self-attention as a precision-weighted robust state estimator for a linear SDE, recovering rotary encodings in a limit, but the abstract's language-modeling results are absent from the text.

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [4]

    William B

    URL https://arxiv.org/abs/1810.04805. William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Interna- tional Joint Conference on Natural Language Processing,

    arXiv

  2. [6]

    URL http://dx.doi.org/10.1109/ ICASSP49357.2023.10095349

    doi: 10.1109/icassp49357.2023.10095349. URL http://dx.doi.org/10.1109/ ICASSP49357.2023.10095349. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning,

    arXiv 2023

  3. [11]

    Alec Radford and Karthik Narasimhan

    URL https://arxiv.org/abs/2108.12409. Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training

    arXiv

  4. [13]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    URL https://arxiv.org/abs/2104.09864. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need,

    arXiv

  5. [14]

    Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen

    URL https://arxiv.org/abs/1706.03762. Benyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. On position embeddings in bert. In International conference on learning representations, 2020a. Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex e...

    arXiv 1912

  6. [2005]

    org/CorpusID:16639476

    URL https://api.semanticscholar. org/CorpusID:16639476. Florian Eilers and Xiaoyi Jiang. Building blocks for a complex-valued transformer architecture. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , page 1–5. IEEE, June

    work page 2023

  7. [2017]

    Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Ning Ding, Youbang Sun, Biqing Qi, Yuchen Fan, Xuekai Zhu, and Bowen Zhou

    URL https://arxiv.org/abs/1705.03122. Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Ning Ding, Youbang Sun, Biqing Qi, Yuchen Fan, Xuekai Zhu, and Bowen Zhou. Fourier position embedding: Enhancing attention’s periodic extension for length generalization,

    arXiv

  8. [2018]

    10 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D

    URL https://arxiv.org/abs/1803.02155. 10 Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing ,

    arXiv

Show all 14 references
  1. [2019]

    URL https://arxiv.org/abs/1901. 02860. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding,

    1901

  2. [2020]

    ChiYan Lee, Hideyuki Hasegawa, and Shangce Gao

    URL https://arxiv.org/abs/ 1909.11942. ChiYan Lee, Hideyuki Hasegawa, and Shangce Gao. Complex-valued neural networks: A comprehensive survey. IEEE/CAA Journal of Automatica Sinica , 9(8):1406–1426,

    1909 arXiv

  3. [2021]

    Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, and Wei Liu

    URL https://arxiv.org/abs/2101.12249. Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, and Wei Liu. Hope: A novel positional encoding without long-term decay for enhanced context awareness and extrapolation,

    arXiv

  4. [2022]

    Yang Leng, Qingfeng Lin, Long-Yin Yung, Jingreng Lei, Yang Li, and Yik-Chung Wu

    doi: 10.1109/JAS.2022.105743. Yang Leng, Qingfeng Lin, Long-Yin Yung, Jingreng Lei, Yang Li, and Yik-Chung Wu. Unveiling the power of complex-valued transformers in wireless communications,

    2022

  5. [2023]

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V

    URL https://arxiv.org/abs/2102.10882. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V . Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context,

    arXiv

  6. [2025]

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franccois Fleuret

    URL https://arxiv.org/abs/2412.17739. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franccois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning ,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.