Pith. sign in

REVIEW 4 major objections 7 minor 50 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

RoPE Frequencies Match Data's Dependency Width, Paper Shows

2026-07-09 02:32 UTC pith:VX6DA5LT

load-bearing objection Clean scaling law θ* ≍ 1/W linking RoPE frequency to data dependency width; the gap between geometric optimality and learned behavior is the main soft spot. the 4 major comments →

arxiv 2607.07678 v1 pith:VX6DA5LT submitted 2026-07-08 cs.LG

How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

Xinyi Wu , Siyuan Liu , Ali Jadbabaie This is my paper
classification cs.LG
keywords Rotary Position EmbeddingsRoPEposition interpolationlength generalizationtransformerpositional encodingfrequency selectionself-similarity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the RoPE (Rotary Position Embedding) frequencies a trained transformer learns to use are determined by the relative-distance structure of its training data. The central object is a data-induced positional dependency kernel K(r), which describes how strongly a prediction at one position depends on information at relative distance r. The characteristic width W of this kernel sets the optimal RoPE frequency via a field-resolution tradeoff: a frequency θ provides unambiguous positional contrast only within its field π/θ, and among frequencies whose field covers the dependency width, the highest (sharpest) one is optimal. This yields the frequency-matching principle θ⋆ = π/W. The paper validates this on synthetic block-structured data (where the fitted constant c ≈ 3.02 is close to the theoretical π) and on iGSM math problems (where broader reasoning chains shift usage toward lower frequencies). The same tradeoff is then applied to position interpolation (PI), the technique of rescaling RoPE frequencies by θ → θ/α to extend a model from context length L to αL. PI expands each frequency's field by α but coarsens its resolution by the same factor. The paper proves that PI preserves frequency utility if and only if the test-time dependency structure is a self-similar dilation of the training-time structure (Theorem 2). Under this self-similarity, the optimal training-scale frequency maps exactly to the optimal longer-context frequency. The paper shows natural language exhibits approximate self-similarity across tokenization scales, explaining why PI works for language models, while arithmetic tasks lack this structure and PI degrades accuracy even when perplexity appears acceptable.

Core claim

The paper's central result is the frequency-matching principle (Theorem 1): for a positional dependency kernel of width W, the optimal admissible RoPE frequency is θ⋆ = π/W, where admissibility requires the frequency's unambiguous field π/θ to cover W. This is proven by showing that the average positional contrast U(θ;K) = ∫(1−cos(θr)) dP_K(r) is strictly increasing on the admissible set (0, π/W], so the maximum is attained at the boundary. The second key result (Theorem 2) shows that under self-similarity — defined as the test-time dependency measure being the pushforward of the training-time measure under dilation by α — position interpolation exactly preserves frequency utility: U(θ/α, αW

What carries the argument

Positional dependency kernel K(r); field-resolution tradeoff (field = π/θ, resolution = arccos(1−τ)/θ); admissible frequency set A_W = {θ : θW ≤ π}; self-similarity condition P_{αW} = (S_α)_# P_W; RoPE frequency energy spectrum E_m

Load-bearing premise

The theory assumes that the usefulness of a RoPE frequency can be captured by a purely geometric measure of positional contrast — how well the phase separates distances — while ignoring the content-dependent projection coefficients and softmax normalization in actual attention. The gap between this idealized single-frequency utility and the multi-frequency learned attention mechanism is not bridged by a learning theorem showing that training actually converges to thepredicted

What would settle it

Train transformers on datasets with controlled dependency widths W and measure whether the learned RoPE frequency spectrum concentrates at θ ≈ π/W. If the spectrum systematically deviates from the 1/W scaling — for instance, if it depends on sequence length, vocabulary size, or model width rather than W alone — the frequency-matching principle is incomplete.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the frequency-matching principle is correct, one could diagnose a model's learned positional inductive bias by measuring the dependency width of its training data and checking whether the learned RoPE spectrum matches the predicted 1/W scaling.
  • The self-similarity condition for PI gives a concrete diagnostic: before applying PI to a new domain, measure whether that domain's dependency profiles are approximately dilation-invariant across scales. If not, PI should not be expected to help.
  • The result that perplexity can overstate long-context ability on non-self-similar tasks (arithmetic) suggests that long-context benchmarks should include tasks requiring precise positional resolution, not just next-token likelihood.
  • The mixture-of-scales view of natural language (KNL as a mixture over widths W) predicts that models trained on data with richer multi-scale dependencies should exhibit broader RoPE frequency bands, which is testable by comparing spectra across domains with different dependency profiles.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The theory could be extended to predict optimal per-layer or per-head frequency allocation: different heads or layers might specialize to different dependency widths present in the data, producing a mixture of optimal frequencies rather than a single one.
  • The self-similarity framework suggests a quantitative metric for how much PI will help on a given dataset: measure the deviation from exact self-similarity (e.g., the L1 distance between normalized dependency profiles at different scales) and correlate with PI performance degradation.
  • If the frequency-matching principle holds during training dynamics, one could monitor the learned RoPE spectrum during training to infer the effective dependency width the model has discovered, providing a data-independent probe of what the model has learned about positional structure.
  • The framework could inform RoPE frequency grid design: instead of the standard geometric spacing θ_m = base^{-2(m-1)/d}, one could space frequencies to match the expected distribution of dependency widths in the target domain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper proposes a data-centered theory of RoPE frequency selection in transformers. The central theoretical result (Theorem 1) shows that for a data-induced positional dependency kernel of width W, the optimal admissible RoPE frequency—defined as the one maximizing average positional contrast U(θ;K)—scales as θ⋆ = π/W. The authors then connect this to position interpolation (PI) for length generalization, showing via Theorem 2 that PI preserves frequency utility if and only if the test-time dependency structure is a self-similar dilation of the training-time structure. Empirical validation includes: (1) a controlled block-structured synthetic task where the effective frequency follows an inverse scaling law with block size (c=3.02 vs. π≈3.14), (2) iGSM experiments showing broader dependencies shift frequency usage lower, and (3) demonstrations that PI helps on self-similar natural language but fails on non-self-similar arithmetic tasks.

Significance. The paper addresses a well-motivated question: why do trained models use RoPE frequencies non-uniformly? The parameter-free prediction θ⋆ = π/W (Theorem 1) is a clean, falsifiable result, and the block-structured experiment (Section 5.1, Figure 2C) provides a direct test with c=3.02 close to π. The connection between PI and self-similarity (Theorem 2, Corollary 1) yields a principled explanation for when PI should help, supported by the arithmetic counterexample (Figure 4C) and the natural-language mutual information analysis (Figure 4B). The perplexity-vs-accuracy analysis for arithmetic (Section 5.2, Tables 1–2) is a useful practical finding. The framework is internally consistent and the predictions are testable, which are strengths.

major comments (4)
  1. §3.3, Theorem 1 and the link to learned weights: The utility function U(θ;K) = ∫(1−cos(θr)) dPK(r) is purely geometric—it depends only on the kernel K and frequency θ, with no dependence on model weights, loss function, or learning dynamics. The paper's central empirical claim is that learned frequency energy E_m = E[a²_{ij,m} + b²_{ij,m}] concentrates near θ⋆. However, nothing in the theory establishes that minimizing cross-entropy loss drives E_m toward the frequency that maximizes U. The coefficients a_{ij,θ}, b_{ij,θ} are content-dependent learned projections; the model could in principle achieve low loss using a frequency with low U by compensating with large weight norms or by relying on content-based attention. This gap between the idealized single-frequency utility and the actual learned multi-frequency attention mechanism is the load-bearing assumption. The paper should either (
  2. §5.1, Figure 2C: The block-structured experiment provides the strongest quantitative evidence, but the fit c=3.02 vs. π≈3.14 is based on only 7 data points (B ∈ {32,...,2048}) with no error bars, on a 2-layer single-head model, on a task where positional retrieval is the only solution strategy (no content cues compete). The lack of error bars or confidence intervals makes it impossible to assess whether the agreement with π is statistically meaningful or coincidental. Adding bootstrap confidence intervals on c, or reporting results across multiple random seeds, would substantially strengthen this central claim.
  3. §5.1, Figure 3B: The iGSM experiment provides only qualitative support—Figure 3B shows that log θ_eff decreases as ops increase, but there is no quantitative fit to the predicted 1/W scaling (unlike the block experiment). The dependency width W is not directly measured or controlled here; it is only qualitatively argued to be broader for larger ops. A quantitative comparison between the measured W (from the MI profiles in Figure 3A) and the predicted θ⋆ = π/W would make this a genuine test rather than a qualitative trend.
  4. §4.2, Theorem 2 and Definition 4: The self-similarity condition P_{αW} = (S_α)_# P_W is a strong structural assumption. The natural-language evidence (Figure 4B) uses tokenization granularity as a proxy for positional dilation, but it is unclear whether changing BPE vocabulary size truly corresponds to the dilation map S_α(r) = αr in the sense of Definition 4. The paper should discuss whether this proxy is a valid approximation of the formal self-similarity condition, or whether it is merely suggestive.
minor comments (7)
  1. §2: The definition of θ_eff = exp(Σ_m ω_m log(θ_m)) is introduced in the block experiment paragraph but the notation ω_m is only defined later. Consider defining it at first use.
  2. Figure 2C: The y-axis label 'log-mean effective frequency' could be clarified—is it log(θ_eff) or θ_eff on a log scale? The text uses both formulations in different places.
  3. §3.2: The admissible set A_W is defined as {θ > 0 : θW ≤ π}, but the field F(θ) = π/θ is defined as the half-period. It would help to explicitly note that admissibility requires F(θ) ≥ W, i.e., the field covers the dependency width, to make the connection immediate.
  4. Appendix D.1: Training for only 200 iterations seems very short. Were the models converged? A note on convergence checking would be helpful.
  5. Figure 4A: The figure is quite dense and the text in the panels is small. Consider enlarging or splitting into subfigures for readability.
  6. §5.2: The arithmetic experiment uses Llama-2-7B with context length 4096, but Table 1 reports results for context windows up to 16384. It would be useful to clarify whether the 8192 and 16384 results use PI with α=2 and α=4 respectively.
  7. References: Several citations are to non-peer-reviewed sources (Reddit posts [6,10], GitHub pull requests [5]). While these are relevant to the RoPE scaling literature, the authors should verify accuracy of these citations.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major points: (1) the gap between the geometric utility U(θ;K) and the claim that cross-entropy minimization drives learned frequency energy E_m toward θ⋆; (2) the absence of error bars or multiple-seed confidence intervals for the c=3.02 fit in Figure 2C; (3) the lack of a quantitative θ⋆=π/W fit in the iGSM experiment (Figure 3B); and (4) whether BPE vocabulary variation is a valid proxy for the formal self-similarity condition in Definition 4. We agree that points (1), (2), and (3) identify genuine gaps between the theory and the empirical evidence, and we will revise the manuscript to address them. For point (4), we agree the discussion should be expanded but argue the proxy is appropriate as a first-order approximation. Details follow.

read point-by-point responses

  1. Referee: §3.3, Theorem 1 and the link to learned weights: U(θ;K) is purely geometric with no dependence on model weights, loss, or learning dynamics. Nothing establishes that minimizing cross-entropy drives E_m toward the frequency maximizing U. The model could achieve low loss using a low-U frequency by compensating with large weight norms or content-based attention. This gap is the load-bearing assumption.

    Authors: The referee correctly identifies the central gap between Theorem 1 and the empirical claims. The theorem proves that θ⋆=π/W maximizes the average positional contrast U(θ;K), which is a geometric quantity. It does not prove that gradient descent on cross-entropy loss drives the learned energy E_m toward θ⋆. We agree this is a load-bearing assumption that the current manuscript does not adequately justify. In the revision, we will add a new subsection (Section 3.4) that explicitly states this as a linking assumption and provides the following supporting argument. The key insight is that in the block-structured task (Section 5.1), positional retrieval is the only solution strategy: the observed tokens are latent value plus i.i.d. Gaussian noise, and the target is a token at a fixed offset. Content-based attention cannot solve this task because the content at the target offset is uncorrelated with the query content across different inputs. The model must use positional information to identify the correct offset. In this setting, the attention score contribution from frequency θ_m is s_{ij}^{(m)} = a_{ij,m}cos(θ_m r) + b_{ij,m}sin(θ_m r), and the model must allocate energy to frequencies that provide unambiguous positional contrast over the dependency width W. A frequency with low U provides poor positional separation over the relevant distances, so compensating with large weight norms would amplify noise rather than signal. This is why the controlled experiment is designed to eliminate content cues: it isolates the positional mechanism that Theorem 1 analyzes. We acknowledge that for natural language, where content-based attention can partially substitute for positional information, the linking assumption is less tightly controlled. We will state this limitation explicitly. revision: yes

  2. Referee: §5.1, Figure 2C: The fit c=3.02 vs. π≈3.14 is based on only 7 data points with no error bars, on a 2-layer single-head model, on a task where positional retrieval is the only solution strategy. Lack of error bars or confidence intervals makes it impossible to assess whether agreement with π is statistically meaningful.

    Authors: The referee is correct that the absence of error bars or confidence intervals weakens the quantitative claim. We will address this in the revision by reporting results across at least 5 random seeds for each block size B, computing bootstrap confidence intervals on the fitted constant c, and including these in an updated Figure 2C. We will also report the standard error of the fit. We note that the current experiment was designed as a controlled proof-of-concept: the 2-layer single-head architecture and the single-solution-strategy task were chosen precisely to isolate the positional mechanism. However, the referee's point that statistical significance cannot be assessed without error bars is well taken, and we will provide the necessary statistical evidence. revision: yes

  3. Referee: §5.1, Figure 3B: The iGSM experiment provides only qualitative support—log θ_eff decreases as ops increase, but there is no quantitative fit to 1/W scaling. The dependency width W is not directly measured or controlled; it is only qualitatively argued to be broader for larger ops. A quantitative comparison between measured W and predicted θ⋆=π/W would make this a genuine test.

    Authors: The referee correctly notes that the iGSM experiment provides only qualitative support. We agree that a quantitative comparison between the measured dependency width W and the predicted θ⋆=π/W would substantially strengthen the evidence. In the revision, we will add this quantitative analysis. Specifically, we will measure W from the mutual-information CDFs in Figure 3A by computing the ρ-field width W_ρ (Definition 2 in Section 3.1) for a fixed ρ (e.g., ρ=0.9) for each iGSM variant. We will then compare the measured effective frequency θ_eff against the predicted π/W_ρ for each of the three iGSM configurations. This will transform Figure 3B from a qualitative trend into a quantitative test of the scaling law. We acknowledge that the iGSM dependency profiles are not compactly supported, so the comparison will use the ρ-field width rather than the exact W from Theorem 1, and we will discuss this distinction explicitly. revision: yes

  4. Referee: §4.2, Theorem 2 and Definition 4: The self-similarity condition is a strong structural assumption. The natural-language evidence (Figure 4B) uses tokenization granularity as a proxy for positional dilation, but it is unclear whether changing BPE vocabulary size truly corresponds to the dilation map S_α(r)=αr in the sense of Definition 4. The paper should discuss whether this proxy is a valid approximation or merely suggestive.

    Authors: We agree that the relationship between BPE vocabulary variation and the formal self-similarity condition deserves more discussion. In the revision, we will add a paragraph in Section 5.2 that addresses this point. The argument is as follows. Changing the BPE vocabulary size changes the number of tokens per unit of text. If the underlying linguistic dependency structure (measured in characters or words) is approximately scale-invariant, as suggested by prior work on fractal patterns in language (Alabdulmohsin et al., 2024), then the dependency profile measured in token distances should approximately dilate when the tokenization granularity changes. Specifically, if a vocabulary change maps k characters to one token instead of αk characters to one token, then a dependency at character-distance d appears at token-distance d/k in the finer tokenization and d/(αk) in the coarser one, corresponding to a dilation by factor α. The mutual-information profiles in Figure 4B show approximate stability under this transformation, which is consistent with—but does not formally prove—self-similarity in the sense of Definition 4. We will explicitly state that the BPE experiment provides suggestive but not definitive evidence, and that a rigorous verification of self-similarity would require controlled data where the dilation factor is known exactly, which is an interesting direction for future work. revision: partial

Circularity Check

0 steps flagged

No significant circularity found; theorems are self-contained and the empirical fit tests an independent prediction.

full rationale

The paper's central theoretical result (Theorem 1) is a self-contained mathematical derivation: it defines a geometric utility U(θ;K) = ∫(1−cos(θr)) dPK(r), defines the admissible set AW = {θ : θW ≤ π}, and proves U is strictly increasing on this set via differentiation under the integral sign (Appendix B). The conclusion θ⋆ = π/W follows directly from the monotonicity proof, with no circular dependence on the conclusion. Theorem 2 (PI preserves utility under self-similarity) is likewise a straightforward change-of-variables proof (Appendix C). The empirical validation in Section 5.1 fits θeff = c/B to block-structured data and obtains c = 3.02, which is then compared to the theoretical prediction π ≈ 3.14 — this is an independent test of a parameter-free prediction, not a fit renamed as a prediction. The self-citations ([42], [43]) by the same authors concern attention masks and position bias, and are cited for background context rather than as load-bearing premises for the frequency-matching theorem. The notable gap between the geometric utility U and actual learning dynamics (i.e., whether gradient descent maximizes U) is a correctness/completeness concern, not a circularity: the paper does not assume its conclusion in its derivation. Score 1 reflects the minor, non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 2 invented entities

The axiom ledger reveals one ad-hoc modeling assumption (the form of U(θ;K)) that is load-bearing but not derived from first principles of attention. The invented entities (K(r), W) are formalizations with empirical handles.

free parameters (1)
  • c (fit constant in block data experiment) = 3.02
    Fitted to the empirical effective frequency vs block size data in Figure 2C. The theory predicts c=π, and the fit yields 3.02, serving as validation rather than a free parameter of the theory itself.
axioms (3)
  • ad hoc to paper The utility of a RoPE frequency θ against a dependency kernel K is given by the average positional contrast U(θ;K) = ∫(1−cos(θr))dPK(r).
    Section 3.3. This is the core modeling assumption. It equates 'usefulness' with a purely geometric measure of phase separation, abstracting away the learned content-dependent projection coefficients and softmax dynamics.
  • domain assumption The field of a frequency θ is defined as F(θ) = π/θ, the half-period over which positional contrast is monotone.
    Section 3.2. This is a standard signal-processing definition applied to the RoPE phase, used to define the admissibility constraint.
  • domain assumption Natural language dependency structure is approximately self-similar across positional scales.
    Section 5.2. Supported empirically by mutual information profiles across tokenization scales (Figure 4B) and cited prior work [1]. This is the condition under which PI is predicted to work.
invented entities (2)
  • Positional dependency kernel K(r) independent evidence
    purpose: To formalize the data-side property that controls RoPE frequency selection.
    While the kernel itself is a formalization introduced by the paper, it is grounded in measurable quantities (autocorrelation, mutual information). The paper provides empirical measurements of K(r) for synthetic and text data.
  • Field width W(K) independent evidence
    purpose: To quantify the largest relevant relative distance in the data.
    Defined as the support of K(r) or the ρ-field width. Measurable from data.

pith-pipeline@v1.1.0-glm · 41539 in / 2395 out tokens · 539957 ms · 2026-07-09T02:32:00.986868+00:00 · methodology

0 comments
read the original abstract

Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what determines this frequency usage and propose a data-centered explanation: RoPE frequencies are selected to match the relative-distance structure of the training data. Viewing each frequency as a positional lens, we formalize a field-resolution tradeoff and show that, for a data-induced dependency profile of width $W$, the optimal frequency scales as $1/W$. This frequency-matching principle explains controlled observations on synthetic and text-based data, and suggests that the mid-low frequency bands observed in language models arise from the multi-scale dependency structure of natural language. We further connect frequency selection to position-interpolation-based length generalization: scaling frequencies down expands the effective field while reducing resolution. This helps when longer-context dependencies are approximate dilations of those seen during training, but can fail when relevant dependencies do not scale with context length. Empirically, we show that natural language exhibits approximate self-similarity across positional scales, explaining why test-time frequency scaling can support long-context generalization. Overall, our results identify a data-driven mechanism behind emergent RoPE frequency usage and show that long-context generalization depends on two forms of scale matching: between learned frequencies and training-time dependencies, and between frequency scaling and how those dependencies extend to longer contexts.

Figures

Figures reproduced from arXiv: 2607.07678 by Ali Jadbabaie, Siyuan Liu, Xinyi Wu.

Figure 1
Figure 1. Figure 1: Frequency energy (averaged over heads) of Qwen-2.5-1.5B model tested on alpaca and gsm8k datasets. The frequency usage concentrates on low frequencies after training and remains stable across prompts. Latent value (piecewise constant) Data structure: block-structured latent sequence with noise +1 -1 Observed tokens (Latent + N(0,1)) +2 -2 0 -4 +4 Block index Position 1 2 3 4 1 2 … B B+1 B+2 … 2B 2B+1 … 3B … view at source ↗
Figure 2
Figure 2. Figure 2: Block-structured data reveals frequency matching between RoPE usage and data dependency width. (A) Synthetic block-drift data. Each sequence consists of latent blocks of length B, where the latent value is piecewise constant and sampled uniformly from {+1, −1}; observed tokens are obtained by adding independent unit Gaussian noise. The model is trained to predict the token at a fixed offset t − ∆ from the … view at source ↗
Figure 3
Figure 3. Figure 3: Data dependency width predicts RoPE frequency usage. (A) Mutual-information CDFs over relative positions show that increasing the number of operations broadens the dependency width: iGSM-2 concentrates information at shorter distances, while iGSM-5 and iGSM-10 shifts dependency mass to longer distances. (B) Head-level log-mean effective frequencies for 12-layer transformer models trained separately on iGSM… view at source ↗
Figure 4
Figure 4. Figure 4: PI is effective when dependencies scale with context length, and can fail when they do not. (A) At training length n = 8, attending to the middle pair and attending to a fixed offset are indistinguishable. At test length n = 16, No PI preserves the fixed-offset pattern, whereas PI with α = 2 dilates the attention pattern toward the middle. Hence PI helps for middle-pair retrieval but hurts fixed-offset ret… view at source ↗
Figure 5
Figure 5. Figure 5: An example from the iGSM dataset Training details All models were trained on a NVIDIA L40 GPU. We pretrained 12-layer GPT￾style Transformer models using the nanochat [18] framework. In each experiment, we followed the default data scaling configuration provided by the framework: a batch size of 524,288 tokens is used for 2,205 steps, corresponding to approximately 1.2B tokens in total. The context length w… view at source ↗
Figure 6
Figure 6. Figure 6: Frequency energy (averaged over heads) of Llama-2-7B model tested on alpaca and gsm8k datasets. The frequency usage concentrates on low frequencies after training and remains stable across prompts. F.1 RoPE frequency spectra are stable across model families We repeat the frequency-energy analysis from Section 2 on Llama-2-7B [38]. Specifically, we compute Em on Alpaca [35] and GSM8K [8]; the results are sh… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 50 canonical work pages · 9 internal anchors

  1. [1]

    Alabdulmohsin, Vinh Quoc Tran, and Mostafa Dehghani

    Ibrahim M. Alabdulmohsin, Vinh Quoc Tran, and Mostafa Dehghani. Fractal patterns may illuminate the success of next-token prediction. InNeurIPS, 2024

    work page 2024

  2. [2]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. InICLR, 2015

    work page 2015

  3. [3]

    Round and round we go! what makes rotary positional encodings useful? InICLR, 2025

    Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Velivckovic. Round and round we go! what makes rotary positional encodings useful? InICLR, 2025

    work page 2025

  4. [4]

    Relational inductive biases, deep learning, and graph networks

    Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flo- res Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charlie Nash, Victoria Langston, Chris Dyer, Nicolas...

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  5. [5]

    by Parts

    bloc97. Add NTK-Aware Interpolation “by Parts” Correction. https://github.com/ jquesnelle/scaled-rope/pull/1, 2023. GitHub pull request

    work page 2023

  6. [6]

    NTK-Aware Scaled RoPE Allows LLaMA Models to Have Extended (8k+) Context Size Without Any Fine-Tuning and Minimal Perplexity Degradation

    bloc97. NTK-Aware Scaled RoPE Allows LLaMA Models to Have Extended (8k+) Context Size Without Any Fine-Tuning and Minimal Perplexity Degradation. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_ rope_allows_llama_models_to_have/, 2023. Reddit post

    work page 2023

  7. [7]

    Extending Context Window of Large Language Models via Positional Interpolation

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation.arXiv preprint arXiv:2306.15595, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  9. [9]

    Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training

    Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Celine Lin, Jan Kautz, and Pavlo Molchanov. Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. arXiv preprint, 2025. URLhttps://arxiv.org/abs/2504.13161

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    Dynamically Scaled RoPE Further Increases Performance of Long Context LLaMA with Zero Fine-Tuning

    emozilla. Dynamically Scaled RoPE Further Increases Performance of Long Context LLaMA with Zero Fine-Tuning. https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/ dynamically_scaled_rope_further_increases/, 2023. Reddit post

    work page 2023

  11. [11]

    What is Wrong with Perplexity for Long-context Language Modeling?

    Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Rethinking invariance in in-context learning

    Lizhe Fang, Khashayar Gatmiry Yifei Wang, Lei Fang, and Yisen Wang. Rethinking invariance in in-context learning. InICLR, 2025

    work page 2025

  13. [13]

    When attention sink emerges in language models: An empirical view

    Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. InICLR, 2025

    work page 2025

  14. [14]

    Serial position effects of large language models

    Xiaobo Guo and Soroush V osoughi. Serial position effects of large language models. InACL, 2025

    work page 2025

  15. [15]

    Large language models are zero-shot rankers for recommender systems

    Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In ECIR, 2024. 10

    work page 2024

  16. [16]

    Fourier position embedding: Enhancing attention’s periodic extension for length generalization

    Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Ning Ding, Youbang Sun, Biqing Qi, Yuchen Fan, Xuekai Zhu, and Bowen Zhou. Fourier position embedding: Enhancing attention’s periodic extension for length generalization. InICML, 2025

    work page 2025

  17. [17]

    Massive values in self-attention modules are the key to contextual knowledge understanding

    Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. Massive values in self-attention modules are the key to contextual knowledge understanding. InICML, 2025

    work page 2025

  18. [18]

    nanochat: The best chatgpt that $100 can buy, 2025

    Andrej Karpathy. nanochat: The best chatgpt that $100 can buy, 2025. URL https://github. com/karpathy/nanochat

    work page 2025

  19. [19]

    The impact of positional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023

    work page 2023

  20. [20]

    Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. Structured attention networks. InICLR, 2017

    work page 2017

  21. [21]

    Mutual information functions of natural language texts

    Wentian Li. Mutual information functions of natural language texts. 1989

    work page 1989

  22. [22]

    Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 2024

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 2024

    work page 2024

  23. [23]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InICLR, 2019

    work page 2019

  24. [24]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2022

    work page 2022

  25. [25]

    Base of rope bounds context length, 2024

    Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length, 2024

    work page 2024

  26. [26]

    Note on the bias of information estimates.Information theory in psychology: Problems and methods, 1955

    George Miller. Note on the bias of information estimates.Information theory in psychology: Problems and methods, 1955

    work page 1955

  27. [27]

    Rethinking the role of demonstrations: What makes in-context learning work? InEMNLP, 2022

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? InEMNLP, 2022

    work page 2022

  28. [28]

    Mitchell

    Tom M. Mitchell. The need for biases in learning generalizations. InReadings in Machine Learning. Morgan Kauffman, 1980

    work page 1980

  29. [29]

    Frequency bands in roPE: Base frequency and context length shape the interpolation–extrapolation trade-off

    Yui Oka, Itsumi Saito, Kyosuke Nishida, and Kuniko Saito. Frequency bands in roPE: Base frequency and context length shape the interpolation–extrapolation trade-off. InICLR, 2026

    work page 2026

  30. [30]

    In-context Learning and Induction Heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova Dassarma, Thomas Henighan, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, John Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom B. Brown, Jack Clark, Jared Kaplan, Sam McCan...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  31. [31]

    Yarn: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. InICLR, 2024

    work page 2024

  32. [32]

    The mechanistic basis of data dependence and abrupt learning in an in-context classification task

    Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. InICLR, 2024

    work page 2024

  33. [33]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. 2023

    work page 2023

  34. [34]

    Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T

    Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. InNeurIPS, 2020. 11

    work page 2020

  35. [35]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    work page 2023

  36. [36]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm.github.io/blog/qwen2.5/

    work page 2024

  37. [37]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arxiv 2023.arXiv preprint arXiv:2302.13971, 10, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  38. [38]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  39. [39]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InNeurIPS, 2017

    work page 2017

  40. [40]

    Kakade, Hao Peng, and Heng Ji

    Ziqi Wang, Hanlin Zhang, Xiner Li, Kuan-Hao Huang, Chi Han, Shuiwang Ji, Sham M. Kakade, Hao Peng, and Heng Ji. Eliminating position bias of language models: A mechanistic approach. InICLR, 2025

    work page 2025

  41. [41]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- ar...

    work page 2020

  42. [42]

    On the role of attention masks and layernorm in transformers

    Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. InNeurIPS, 2024

    work page 2024

  43. [43]

    On the emergence of position bias in transformers

    Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the emergence of position bias in transformers. InICML, 2025

    work page 2025

  44. [44]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InICLR, 2024

    work page 2024

  45. [45]

    Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process

    Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process.ArXiv e-prints, abs/2407.20311, July 2024. Full version available athttp://arxiv.org/abs/2407.20311

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    Reddi, and Sanjiv Kumar

    Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. O(n) connections are expressive enough: Universal approximability of sparse transformers. InNeurIPS, 2020

    work page 2020

  47. [47]

    Found in the middle: How language models use long contexts better via plug-and-play positional encoding, 2024

    Zhenyu (Allen) Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. Found in the middle: How language models use long contexts better via plug-and-play positional encoding, 2024

    work page 2024

  48. [48]

    Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh

    Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. InICML, 2021

    work page 2021

  49. [49]

    Xing, Haotong Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InNeurIPS, 2023. 12 A Related Work RoPE variants and analyses of RoPERoPE encodes relative position by rotating query...

    work page 2023

  50. [50]

    [43] show the multi-layer effects of masks and positional encodings [43]

    observe that causal masking and RoPE can introduce position dependence; and Wu et al. [43] show the multi-layer effects of masks and positional encodings [43]. Several works further study how masks and positional encodings affect length generalization and positional processing. Yun et al.[46] analyze the approximation power of transformers under different...

    work page 2048