Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Rotating values with the same position-dependent rotations as keys turns RoPE attention into an offset-aware attentive convolution, and this parameter-free change consistently improves long-context behavior in trained language models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:50 UTC pith:NCUD7RVA

load-bearing objection Correct theory and clean setup, but the empirical claim of consistent gains rests on single training runs with no variance; needs multi-seed evidence before the headline holds. the 3 major comments →

arxiv 2606.11275 v2 pith:NCUD7RVA submitted 2026-06-09 cs.LG cs.AI

RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

Alejandro Garc\'ia-Castellanos , Maurice Weiler , Erik J Bekkers This is my paper
classification cs.LG cs.AI
keywords Rotary Position EmbeddingsAttentive ConvolutionRelative position encodingValue pathwayOV circuitLong-context generalizationMatrix mixerBlock-Toeplitz
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the value pathway of rotary position embedding (RoPE) attention is incomplete: RoPE makes which tokens are attended to depend on relative offset, but how a selected token is transformed does not. The proposed fix, RoVE, rotates each value by the same position-dependent rotation already applied to keys, then rotates the aggregate back into the query's frame. This replaces the fixed value projection W_V with an offset-indexed kernel ψ_δ = R_δ W_V, turning the attention layer into an attentive convolution and giving its mixer a block-Toeplitz structure. The paper reports consistent gains over RoPE at 124M and 354M parameter scales on few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval, with the largest gains on tasks that demand long-range aggregation. If right, this closes a known asymmetry in transformer circuits and unifies several independently rediscovered operations in vision, robotics, and language modeling.

Core claim

The paper's central claim is that applying the rotary position embedding to values as well as to keys and queries yields a strictly richer relative-position mechanism at no parameter cost. Concretely, RoVE computes the output as a sum over attended positions of A_ij R_(j-i) W_V x_j, with A_ij the usual RoPE attention weights, so the effective value kernel depends on the offset δ=j-i. In matrix-mixer form this converts the attention operator from a single Kronecker product A⊗W_V into a sum of Kronecker products over offset diagonals, giving a block-Toeplitz structure—the signature of an attentive convolution. The paper also frames the operation as a frame change: rotate each value into a shar

What carries the argument

The central object is the offset-indexed value kernel ψ_δ = R_δ W_V, where R_δ is the same block-diagonal rotation family used by RoPE (each 2×2 block rotates by δ ω_m for geometrically spaced frequencies ω_m) and W_V is the learned value projection. Substituting this family for the constant W_V endows the OV circuit with relative-position sensitivity. The paper analyzes the effect through three equivalent lenses: as an attentive convolution (content-dependent scalar gates A_ij times a fixed offset-dependent kernel), as a matrix mixer whose block diagonal structure becomes block-Toeplitz (a sum of Kronecker products over offset diagonals), and as a frame-change operation (rotate values into

Load-bearing premise

The empirical claim of consistent gains rests on single training runs per configuration with no reported variance for Core accuracy and perplexity, so the reported differences could reflect seed noise rather than a real effect.

What would settle it

Train the 354M model with at least five random seeds in each condition under identical hyperparameters; if the RoVE-vs-RoPE Core accuracy gap of 0.0192 and the long-context perplexity gaps do not consistently reproduce, the empirical conclusion fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • RoVE introduces no new parameters and is compatible with efficient attention kernels, since the rotations act on values before and after the attention call.
  • RoPE attention becomes a special case of attentive convolution, recovering the position-blind value map when ψ_δ is constant; conversely, RoVE makes the OV circuit offset-sensitive without changing the QK scores.
  • RoVE and YaRN are complementary: frequency interpolation of the shared rotation family keeps the gap between the two methods intact while improving both.
  • At 124M and 354M scales, RoVE improves few-shot in-context accuracy, in-context perplexity, out-of-distribution perplexity up to 16× context length, and long-context retrieval NLL, with the largest gains on tasks requiring long-range information aggregation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the structural account is right, RoVE should most help tasks that are essentially offset-indexed copy or associative recall; probing the effective OV kernel at different offsets on synthetic recall tasks would test this directly.
  • The paper leaves open whether the OV pathway's rotation frequencies need a different interpolation schedule than the QK pathway; tuning frequency scaling for values separately is a natural extension.
  • The block-Toeplitz view suggests a direct bridge to sub-quadratic models: because RoVE's mixer is a content-gated Toeplitz operator, one could seek linear-time approximations that preserve the offset-dependent kernel structure.
  • Circuit-level interpretability would change visibly: the effective OV circuit is no longer a single matrix per head but a family of offset-rotated matrices, so attention-head analysis should condition on relative offset.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes RoVE, a parameter-free modification of RoPE attention that rotates value vectors by the same position-dependent rotations applied to keys, and rotates the aggregated output back. Equation (3) rewrites the resulting operation as an attentive convolution with offset-indexed kernel ψ_δ = R_δ W_V, replacing the position-blind OV circuit of standard RoPE with a relative-position-dependent value pathway. The authors provide a structural analysis in terms of matrix mixers and transformer circuits, claiming that RoVE turns the attention mixer from Kronecker structure into block-Toeplitz structure. They further claim unification with prior independent work in vision, robotics, and LLM architectures, and report empirical gains at 124M and 354M scale over RoPE baselines on DCLM-Core ICL accuracy, out-of-distribution perplexity, and RULER long-context retrieval.

Significance. If the empirical claims hold, RoVE is a simple, inexpensive, and theoretically clean modification that improves long-context behavior by making the value pathway relative-position-sensitive. The theoretical contribution is sound: Eq. (3) is a correct algebraic identity, the block-Toeplitz description in Section 3 and Appendix E follows by construction, and the framework properly situates RoVE within attentive convolutions and the transformer-circuits literature. The paper's strength is its clear structural account and the fact that RoVE introduces no additional parameters and remains FlashAttention-compatible. However, the empirical validation, which is the load-bearing evidence for the headline claim of consistent gains, currently rests on single training runs per configuration, with no variance estimates for the Core ICL and perplexity comparisons. The paper is therefore a solid theoretical contribution whose central empirical conclusion is not yet fully supported.

major comments (3)
  1. [Section 4, Tables 1 and 3; Appendix A.2] The central empirical claim — that RoVE consistently improves over RoPE — is based on single training runs. Core ICL accuracy and perplexity are reported as point values with no standard deviation, confidence interval, or seed count. For example, the 354M Core improvement from 0.1664 to 0.1856, and the 124M improvement from 0.1375 to 0.1416, could easily be within typical run-to-run variation for few-shot ICL benchmarks. The same applies to the in-context perplexity differences (e.g., 17.68 vs 17.52 at 512 tokens for 354M). Without repeated seeds or error bars, the 'consistent gains' claim in Appendix A.2 is unsubstantiated. This should be addressed by reporting mean and variance over multiple seeds, or at minimum by demonstrating statistical significance with paired or unpaired tests.
  2. [Tables 2 and 4; Appendix A.1] The standard deviations reported in Tables 2 and 4 are over the 500 RULER evaluation samples, not over training runs. They quantify evaluation noise for a fixed model, not the sensitivity of the comparison to random initialization and training stochasticity. Since the manuscript claims consistent improvements across scales and tasks, the relevant uncertainty is seed-to-seed variation. The paper should report results from multiple independent training runs (e.g., at least 3 seeds) for the main comparisons, or explicitly justify why the reported differences are expected to exceed training-seed noise. As it stands, the empirical evidence for RoVE's advantage is incomplete.
  3. [Appendix D, Limitations] The paper honestly notes that it does not fully identify the mechanism behind the improved OOD perplexity and offers only a working hypothesis. This is not a flaw in itself, but it means the conclusion that 'relative-position-aware values stand as a robust structural bias for LLMs' goes beyond what the current evidence demonstrates. The proposed frequency-band analysis is a reasonable test, but it is not conducted here. I would encourage the authors to add such an analysis or temper the conclusion accordingly.
minor comments (4)
  1. [Abstract and Section 1] There are several typographical and spacing issues: 'Ro VE' appears with a spurious space in the author header, 'Y aRN' in Section 2, and 'RoVEattention' in Section 1. The paper would benefit from a careful proofread.
  2. [Appendix A, Tables 5 and 6] Tables 5 and 6 are restatements of Tables 1 and 2. If the purpose is cross-scale comparison, consider presenting the small and medium results side by side rather than duplicating the same tables, to reduce redundancy.
  3. [Section 2, Eq. (2)] The notation A(X)_ij is used for the attention weight, but later in Eq. (3) and the text the same symbol is used as a matrix. This is standard, but a brief sentence clarifying that A(X)_ij denotes the (i,j)-th entry of the attention matrix would improve readability.
  4. [Section 3, 'Matrix mixer lens'] The phrase 'block-Toeplitz structure' is qualified with 'up to the content-dependent modulation by A_ij,' which is correct. Because A_ij depends on the input, the mixer is not a fixed block-Toeplitz operator. The authors should make this qualification more prominent to avoid overstating the structural result.

Circularity Check

0 steps flagged

No significant circularity: RoVE's attentive-convolution form is a direct algebraic identity from the paper's own definitions, and the empirical/unification claims rely on external benchmarks and independent prior work.

full rationale

The derivation chain is self-contained. Eq. (2) defines RoPE attention; Eq. (3) defines RoVE and rewrites it as sum_j A_ij R_{j-i} W_V x_j, which is exactly Eq. (1) with kernel psi_delta = R_delta W_V. This is a substitution from the paper's own definitions, not a fitted parameter or an imported conclusion, and the block-Toeplitz/circuits analysis in Section 3 and Appendix E follows from that identity and the definition of matrix mixers. The unified-formulation claim explicitly credits independent prior works (Miyato et al., Wu et al., Li et al., Klee et al., DeepSeek-AI) and does not rename a single known result as new. The only self-citation is Romero et al. (2020) for the attentive-convolution definition, on which Bekkers is a co-author; that definition is restated in Eq. (1), and the paper's result does not depend on the citation for validity, so it is not load-bearing. Appendix D's stated limitation (the mechanism behind OOD perplexity gains is not fully identified) is a completeness concern, not a circularity. The absence of seed-level variance in Tables 1 and 3 is a legitimate statistical-support concern, but it is not circularity: no evaluation result is used to fit a parameter that is then reported as a prediction. No circular step is therefore present.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No free parameters are introduced by RoVE; the theoretical result is a direct algebraic reparametrization. The only external dependencies are standard definitions of RoPE and matrix mixers. No new entities are postulated.

axioms (3)
  • standard math RoPE rotation matrices R_t are orthogonal block-diagonal rotations as defined in Su et al. (2024)
    Used throughout Section 3; no proof needed as standard definition.
  • domain assumption The matrix-mixer framework (Hwang et al., 2024) and attentive-convolution definition (Romero et al., 2020; Fuchs et al., 2020) apply to self-attention layers with content-dependent gates
    Eq. (1) and (2) assume the standard formulation of attention as a mixer; if one does not accept this framework, the block-Toeplitz interpretation is less natural.
  • domain assumption Attention logits A_ij are unaffected by the value rotation, so RoVE leaves the QK circuit unchanged
    Definition 1 constructs RoVE post-hoc by rotating values; this preserves the softmax weights but assumes the model's trained representation still benefits from the reparametrization.

pith-pipeline@v1.3.0-alltime-deepseek · 13949 in / 10214 out tokens · 105674 ms · 2026-08-02T11:50:33.656527+00:00 · methodology

0 comments
read the original abstract

Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind: the message sent by a value token is the same regardless of its distance from the query. We propose RoVE, a parameter-free modification that makes values position-sensitive by rotating them simultaneously with keys, and show that it turns RoPE attention into attentive convolution. This new perspective unifies several independent formulations of the same operation across computer vision, robotics, and modern LLM architectures. Trained 124M and 354M GPT-2 models show consistent empirical gains over RoPE on few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval, with the clearest improvements on tasks that require long-range aggregation.

Figures

Figures reproduced from arXiv: 2606.11275 by Alejandro Garc\'ia-Castellanos, Erik J Bekkers, Maurice Weiler.

Figure 1
Figure 1. Figure 1: Matrix-mixer view of RoPE and RoVE. RoPE factorises into (a) position-sensitive attention weights and (b) a constant shared value projection WV across all offsets (c). (d) RoVE replaces WV with the offset-indexed family ψδ = RδWV , (e) producing a block-Toeplitz mixer whose diagonals rotate systematically with relative offset, the signature of an attentive convolution. standalone module in standard (non-sh… view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of the circuits of the one-layer attention-only transformer for [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere

    math.DS 2026-07 accept novelty 6.0

    Normalized query/key-only RoPE attention on the sphere has reversible consensus kernels with exact Bessel-aliasing spectra, explicit regional contraction rates from a sharp softmax floor, and RoPE-selected twisted equ...

Reference graph

Works this paper leans on

26 extracted references · 13 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Y Zou, Atri Rudra, and Christopher R´ e. Zoology: Measuring and improving recall in efficient language models. InInternational conference on learning representations, volume 2024, pages 15664–15730,

    2024

  2. [9]

    Anand Gopalakrishnan, Robert Csord´ as, J¨ urgen Schmidhuber, and Michael C Mozer

    URLhttps://proceedings.neurips.cc/paper_files/paper/2020/hash/ 15231a7ce4ba789d13b722cc5c955834-Abstract.html. Anand Gopalakrishnan, Robert Csord´ as, J¨ urgen Schmidhuber, and Michael C Mozer. De- coupling the” what” and” where” with polar coordinate positional embeddings.arXiv preprint arXiv:2509.10534,

    Pith/arXiv arXiv 2020

  3. [11]

    arXiv:2407.09941 [cs]

    URLhttp://arxiv.org/abs/ 2407.09941. arXiv:2407.09941 [cs]. David Klee, Boce Hu, Andrew Cole, Heng Tian, Dian Wang, Robert Platt, and Robin Walters. RA VEN: End-to-end equivariant robot learning with RGB cameras. InThe Fourteenth International Conference on Learning Representations,

    Pith/arXiv arXiv

  4. [12]

    Functional interpolation for relative positions improves long context transformers

    Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Za- heer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. InInternational Conference on Learning Representations, volume 2024, pages 11303–11328, 2024b. Peter Lippmann, Gerr...

    2024

  5. [13]

    co/datasets/HuggingFaceFW/fineweb-edu

    URLhttps://huggingface. co/datasets/HuggingFaceFW/fineweb-edu. Takeru Miyato, Bernhard Jaeger, Max Welling, and Andreas Geiger. Gta: A geometry- aware attention mechanism for multi-view transformers. InInternational Conference on Learning Representations, volume 2024, pages 8172–8208,

    2024

  6. [14]

    GitHub repository. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, and Jiaming et al. Kong. Rwkv: Reinventing rnns for the transformer era.arXiv:2305.13048,

    Pith/arXiv arXiv

  7. [15]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. InInternational Conference on Learning Representations, volume 2024, pages 31932–31951,

    2024

  8. [16]

    arXiv:2302.10866 [cs]

    URLhttp://arxiv.org/abs/2302.10866. arXiv:2302.10866 [cs]. Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409,

    Pith/arXiv arXiv

  9. [17]

    arXiv:2002.03830 [cs]

    URLhttp://arxiv.org/abs/ 2002.03830. arXiv:2002.03830 [cs]. 8 Ro VE: Rotary V alue Embeddings Attention Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    Pith/arXiv arXiv 2002

  10. [18]

    Mrrope: Mixed-radix rotary position embedding.arXiv preprint arXiv:2601.22181,

    Qingyuan Tian, Wenhong Zhu, Xiaoran Liu, Xiaofeng Wang, and Rui Wang. Mrrope: Mixed-radix rotary position embedding.arXiv preprint arXiv:2601.22181,

    arXiv

  11. [19]

    arXiv:2601.15275 [cs]

    URLhttp: //arxiv.org/abs/2601.15275. arXiv:2601.15275 [cs]. Chuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong, Jiankai Sun, Jingyao Li, Minbin Huang, Xiaozhe Ren, Michael Ng, Xin Jiang, et al. Dape v2: Process attention score as feature map for length extrapolation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (...

    Pith/arXiv arXiv

  12. [20]

    Full Experimental Results A.1

    9 Garc´ıa-Castellanos, Weiler & Bekkers Appendix A. Full Experimental Results A.1. Setup Models:We train two GPT-2-style transformers in the nanoGPT framework (nanoGPT, 2022). Thesmallmodel (≈124M parameters) has 12 layers, 12 attention heads, and em- bedding dimension 768; themediummodel (≈354M parameters) has 24 layers, 16 heads, and embedding dimension

    2022

  13. [21]

    Theonlyarchitectural difference between theRoPEandRoVEconditions is the value pathway; all other architectural and training hyperparameters are held fixed. T raining:Both models are trained for one epoch on FineWebEdu-10B (≈10B tokens of educational web text tokenised with the GPT-2 tiktoken encoder) (Lozhkov et al., 2024), with a sequence length of 1024 ...

    2024

  14. [22]

    positional interpolation at inference time without any fine-tuning, where the frequency modulation is applied to all rotation matrices, covering both the QK- and OV-circuits. •RULER long-context retrieval.We evaluate on four RULER synthetic tasks (Hsieh et al., 2024), namely Common Word Extraction (CWE), multi-key Needle-in-a-Haystack (NIAH), Question Ans...

    2024

  15. [24]

    formalises this through a polar decomposition of (4), identifying a content-dependent phase cross-term that entangles positional and semantic information, which is replaced by pure-magnitude representations to yield a score that factors into a content product and a positional cosine, improving perplexity and length generalisation. These works restructure ...

    2019

  16. [25]

    However, the operation requires materialising the fulln×nscore tensor before softmax, ruling outFlashAttention

    takes a complementary approach, applying a narrow convolution kernel across heads over the pre-softmax score tensor (on top of a standard ad- ditive positional bias), and shows that this convolution component alone provably suffices for associative recall even when the bias is zeroed out. However, the operation requires materialising the fulln×nscore tens...

    2023

  17. [26]

    direct path

    takes this frequency-dependent logic to its principled con- clusion by treating each dimension according to its wavelengthλ m = 2π/ω m relative to the training contextL. Dimensions withλ m ≪Lcomplete many full rotations within the training window; their angles are robustly periodic and can therefore be safelyextrapolated (left unscaled) at inference time....

    2024

  18. [2017]

    what” and “where

    add a fixed or learned vectorp i to each token embedding before projection, yielding scores Aape ij = (WQ(xi +p i))⊤(WK(xj +p j)), which depend on the absolute indicesiandjseparately, making extrapolation to unseen lengths fragile. •Additive relative encodings(ARPE; Raffel et al. 2020; Press et al. 2021; Chi et al. 2022; Li et al. 2024b) replace the absol...

    2020

  19. [2019]

    On the relationship between self-attention and convolutional layers.arXiv preprint arXiv:1911.03584,

    Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers.arXiv preprint arXiv:1911.03584,

    Pith/arXiv arXiv 1911

  20. [2020]

    Extending con- text window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595,

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending con- text window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595,

    Pith/arXiv arXiv

  21. [2021]

    Daniel Y

    https://transformer- circuits.pub/2021/framework/index.html. Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher R´ e. Hungry Hungry Hippos: Towards language modeling with state space models. In International Conference on Learning Representations, 2023a. Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michae...

    Pith/arXiv arXiv 2021

  22. [2022]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509,

    Pith/arXiv arXiv 1904

  23. [2023]

    5 Garc´ıa-Castellanos, Weiler & Bekkers Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse...

    1901

  24. [2024]

    Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205,

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veliˇ ckovi´ c. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205,

    Pith/arXiv arXiv

  25. [2025]

    Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654,

    Pith/arXiv arXiv

  26. [2026]

    Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753,

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens.arXiv preprint arXiv:2402.13753,

    Pith/arXiv arXiv