Pith. sign in

REVIEW 4 major objections 6 minor 72 references

Disentangling the Expressivity of RoPE

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Component-periodic RoPE transformers recognize exactly LTL[P,MOD], the past temporal logic with modular predicates; conventional non-repeating RoPE gives only a bounded, precision-limited look-back.

desk verdict Theorem 3.1 is a real, exact characterization of periodic RoPE; the non-periodic side is a bounded simulation plus experiments, not the formal separation the framing suggests. read the letter →

arxiv 2608.11909 v1 pith:AMBGUHTO submitted 2026-08-12 cs.LG cs.FL

Selim Jerad , Anej Svete , Jiaoda Li , Ryan Cotterell This is my paper · ORCID
classification cs.LGcs.FL MSC 68Q4568Q1903B4468T07
keywords rotarypositionembeddingsRoPEexpressivitylineartemporallogicmodularpredicatestransformerfinite-precisionarithmeticlengthgeneralizationpositionalencodings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to settle what rotary position embeddings (RoPE) actually give a transformer, and it does so by separating two regimes that are usually conflated. When every rotary component repeats after a fixed number of positions (a component-periodic schedule, realized by a lookup table indexed by position modulo the period), a fully uniform finite-precision soft-attention transformer recognizes exactly the languages definable in past temporal logic with modular predicates, $\mathrm{LTL}[P,\mathrm{MOD}]$: for inputs of any length it can decide properties such as 'the string has even length' or 'the input is a repetition of $ab$'. Conventional RoPE, whose rotation frequencies are irrational and never repeat, cannot achieve this all-length modular behavior; its relative scoring yields only a bounded, precision-dependent look-back by fixed offsets. That distinction matters because it predicts when RoPE helps (fetching nearby context) and when it subtly harms models that must aggregate information from distant, position-independent sources, and the paper's controlled experiments on small formal languages confirm both sides of the prediction.

What carries the argument

Three mechanisms carry the argument. The component-periodic schedule: each two-dimensional rotary block takes finitely many realized values and cycles with period $m_d$, implemented not by forming the unbounded product $\theta_d \cdot i$ but by a lookup table indexed by $i \bmod m_d$, so the realized map repeats by construction and every matrix entry becomes a finite disjunction of modular predicates. The BOS-anchor head: a rotary pair compares the query's rotated phase at position $i$ against the fixed identity phase of the beginning-of-string key, while an unrotated pair supplies a constant non-BOS baseline chosen so that the winner leads by at least the finite-precision margin $f_{\mathrm{large}}$; stable softmax turns that comparison into the exact truth value of $\mathrm{MOD}^r_m(i)$, and the anchor is what makes a pairwise relative score computable from unary position information — the same behavior observed as attention sinks. The certified range $N_{\max}^{(k)}(g,C,F)$: for an irrational angle $g$, a $k$-offset head scores position $i-k$ by $C\cos(g(i-k-j))$, and $N_{\max}$ is the largest length up to which that target outscores every other key by at least $f_{\mathrm{large}}$; beyond it the offset head can fail, which is why the $Y^k$ simulation is bounded. Unrotated subspaces carry the NoPE-style $\mathrm{LTL}[P]$ computation, and the pieces compose by structural induction.

What would settle it

Evaluate, in actual floating-point arithmetic, the realized score gap of an irrational-frequency offset head at increasing lengths: the paper predicts a finite certified length $N_{\max}$ beyond which the target position $i-k$ no longer beats every competing offset by the margin $f_{\mathrm{large}}$, whereas a periodic lookup-table schedule keeps the gap constant. A concrete search: train or hand-construct a conventional-RoPE transformer on $(aa)^*$ and scan base values beyond the tested range $10^{-12}$ through $10^{12}$ for perfect length generalization through 500; the paper's separation claim predicts that none will be found and that only the modular construction reaches it.

Watch

Extended reading notes

Core claim

The paper's central claim is a paired characterization. On the periodic side, Theorem 3.1 states that $\mathrm{SMAT}[\mathrm{RoPE}_P] = \mathrm{LTL}[P,\mathrm{MOD}] = \mathrm{PFO}_2[<,\mathrm{MOD}]$: component-periodic RoPE transformers know exactly the languages of past temporal logic with modular predicates, equivalently the past fragment of two-variable first-order logic with unary modular predicates. Theorem 3.3 adds algebraic and automata-theoretic faces of the same class — left-deterministic modular polynomials, syntactic monoids in $\mathcal{QR}$, and automata whose $k$-blowup is partially ordered — and Theorem 3.2 shows periodic absolute sinusoidal encodings satisfy the same equivalence. On the non-periodic side, Proposition 4.1 records that irrational rotations never repeat, and Corollary 4.1 shows that conventional RoPE can only $L$-simulate the fixed-offset operators $Y^k$ up to a certified length $N_{\max}(g,C,F)$; within that range it simulates $\mathrm{LTL}[P,Y]$ formulas, but beyond it, finite-precision rounding lets competing offsets tie or beat the intended target. The experiments match this split: constructed periodic schedules reach perfect accuracy through length 500 on $(ab)^*$ and $(aa)^*$, while no tested conventional base value — seven bases spanning twenty-four orders of magnitude — generalizes on either language.

Load-bearing premise

The load-bearing premise is that the finite-precision value set can be tailored to the task, representing the rotation-table entries, the margin constants, and their arithmetic exactly; if real hardware cannot represent values such as $2\pi/m$ exactly, the exact equivalence holds only inside the abstract model and the practical case rests on the experiments.

Editorial extensions

If this is right

  • Practical periodic schedules — the cyclic-lookup and rounded-wavelength RoPE variants used in deployed models — inherit the exact characterization, so they can recognize modular-position regularities such as $(ab)^*$ and $(aa)^*$ uniformly over all input lengths.
  • Changing the base of conventional RoPE cannot buy modular generalization: with irrational rotation angles the score gap that isolates an offset is finite-precision-limited, so the look-back mechanism is certified only up to a length $N_{\max}$ that depends on the angle, the scale, and the precision set.
  • Conventional RoPE is a double-edged local bias: at six of seven tested bases it improves on NoPE for the look-back language $\Sigma^*a$, but at every base it lowers the mean longest perfect length on the position-invariant languages $a\Sigma^*$ and $\Sigma^*ab\Sigma^*$.
  • The decisive factor is periodicity of the realized position-to-embedding map rather than the relative-versus-absolute distinction, because periodic absolute sinusoidal encodings satisfy the same equivalence as periodic RoPE.
  • The class $\mathrm{LTL}[P,\mathrm{MOD}]$ stays incomparable with full past-temporal logic: it cannot recognize PARITY, and over alphabets with more than two symbols it cannot recognize the locally testable language $\Sigma^*ab\Sigma^*$, which needs the adjacent-position operator $Y$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the BOS-anchor construction suggests a diagnostic for attention-sink behavior — if a RoPE model's sink token partly computes modular-position predicates, ablating the sink should degrade periodic regularities such as alternation more than it degrades local copying; the paper does not run this experiment.
  • Editorial inference: the certified-range formula gives a quantitative precision prediction — moving from a higher- to a lower-precision regime should measurably shorten the length at which fixed-offset heads fail, a test that standard perplexity metrics would not surface.
  • Editorial inference: the two characterizations imply a hybrid-schedule design rule — reserve some rotary dimensions for periodic modular counting, some for unrotated global aggregation, and optionally some for non-periodic local offsets — which the paper only conjectures for partial-RoPE variants.
  • Editorial inference: on this account, long-context failures of RoPE models are not merely missing capability; the rotation geometry actively biases against position-invariant access to distant content, which may explain reported gains from dropping positional embeddings in long-context setups.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper studies the expressivity of rotary position embeddings (RoPE) in the fully uniform, finite-precision soft-attention transformer model (SMAT). The central result, Theorem 3.1, states that component-periodic RoPE (RoPEP) transformers recognize exactly the languages definable in past temporal logic with modular predicates: SMAT[RoPEP] = LTL[P,MOD] = PFO2[<,MOD]. The upper bound (Lemma 3.1) simulates the finite-precision transformer using modular predicates, while the lower bound (Lemma 3.2) gives a constructive, BOS-anchored attention mechanism that computes each modular atom. Theorem 3.2 extends the equality to periodic sinusoidal positional encodings, and Theorem 3.3 provides additional algebraic and automata characterizations (left-deterministic modular polynomials, QR syntactic monoids, PODFA^k). For conventional, non-periodic RoPE, Corollary 4.1 proves a bounded-length L-simulation of the operators Y^k and LTL[P,Y] up to a certified length N_max(g,C,F). The experiments train transformers on (aa)*, (ab)*, aΣ*, Σ*a, and Σ*abΣ*: constructed periodic schedules reach perfect length generalization (N*=500) on the modular languages, conventional RoPE and SiPE do not at any tested base, and RoPENP improves over NoPE on an LTL[Y] task while underperforming on LTL[P] tasks. The paper is explicit that the periodic lower bound uses a formula-tailored finite-precision value set and that the locality result of Section 4 is bounded.

Significance. Within the stated abstract model, Theorem 3.1 is a clean, constructive, and non-circular characterization: the schedule is derived from the target formula, no data are fitted, and the proofs are checkable by hand. The BOS-anchor construction is a genuinely insightful mechanism and connects the expressivity picture to the empirical attention-sink literature. Theorem 3.3 usefully extends the Dartois-Paperman and Li-Cotterell toolkit to the modular past fragment, and Proposition 3.1's reduction of binary modular predicates to unary ones is elegant. The paper is also unusually candid about scope: Section C.3 acknowledges that the finite-precision set F is tailored to the formula, and the Limitations paragraph states that the RoPENP result is bounded and not an all-length characterization. The main shortfall is that the framing of the RoPEP/RoPENP distinction, advertised in the abstract and Section 5 as a qualitative dichotomy, is stronger than what the theorems establish: no upper bound on SMAT[RoPENP] is given, so whether a non-periodic schedule can recognize (aa)* or (ab)* at all lengths is open, and only the experiments address that question.

major comments (4)
  1. [Abstract; §4.1–4.3; §5 (Limitations)] The paper asserts a dichotomy between RoPEP and RoPENP (abstract: 'Conventional RoPE is different ... rather than an all-length modular characterization'; §5: 'The conventional RoPENP behaves qualitatively differently') that the theorems do not establish. Corollary 4.1 is a positive bounded-simulation result only; there is no upper bound on SMAT[RoPENP], and the paper's own Limitations paragraph concedes that 'The locality result of §4 is bounded and is not an all-length characterization of RoPENP.' In particular, nothing rules out a fixed-precision RoPENP transformer with an irrational schedule and unrotated subspaces recognizing (aa)* or (ab)* at all lengths; because the realized rotation sequence takes values in a finite set, it is eventually periodic, and the paper does not analyze whether eventual periodicity of the readout forbids the modular languages. Since (aa)* and (ab)* are the paper's canonical separation examples (§3.4), this gap is load-bearing for the narrative. I ask the authors to state explicitly in the abstract and Section 5 that the RoPENP upper bound is open, to either prove an eventual-periodicity upper bound or phrase the dichotomy as a conjecture with a clear falsification criterion, and to present the experiments as the only current evidence for the separation. The stress-test concern about this point is valid; it is partially mitigated by the stated Limitations, but the abstract does not carry the caveat.
  2. [§E.3; §3.5] The headline empirical claim (RoPEP accuracy 1.00 and N*=500 on both (aa)* and (ab)*) is not reproducible from the manuscript. Section E.3 specifies only that 'The component-periodic scheduler is explicitly written out in the code implementation,' but no code, no data, and no textual description of the schedule (periods m_d, angles, which dimensions are rotated or unrotated) are provided. Since the periodic schedule is the independent variable that separates RoPEP from RoPENP in the experiments, the authors should either include the schedule in the appendix or release the code and data; without this, the experimental separation cannot be verified or compared against.
  3. [Prop. 4.1] The proof of Proposition 4.1 is incomplete as written. The cited result (Walters 1982, Thm 1.8) concerns density of the ideal irrational rotation i ↦ Rot_{g,i}, but the proposition concerns the realized map i ↦ round_F(Rot_{g,i}), and the inference from density to non-periodicity of the rounded map is not immediate and is not argued. A correct proof must show that for any m ≥ 1 with gm mod 2π ≠ 0, the set of points at which the rounding outcome changes under the nonzero rotation has nonempty interior (it is an open set of positive measure, since the rounding cells are finitely many arcs), and that the dense orbit necessarily intersects it. The claim is true, but the step needs to be spelled out rather than left to the citation.
  4. [Prop. 3.2 (proof)] The formula in the proof of Proposition 3.2 is off by one. For the word w_1⋯w_m, the conjunct MOD^j_m(i) ⇒ π_{w_j}(i) over 0 ≤ j < m references the undefined symbol w_0, and because LTL formulas are evaluated at the readout position N+1 (Eq. 11), the length condition should be MOD^1_m at the readout (equivalent to N ≡ 0 mod m), not MOD^0_m. The claim itself is true; the displayed formula should be corrected, for example by using MOD^j_m(i) ⇒ π_{w_j}(i) for 1 ≤ j < m, MOD^0_m(i) ⇒ π_{w_m}(i), and the readout condition MOD^1_m.
minor comments (6)
  1. [Table 2; §3.5] Please clarify what the 'Max accuracy' and N* columns are computed over. As defined in §3.5 (N* is the largest N with 100% accuracy at every tested length through N), the row '(ab)* NoPE: Acc 0.96, N* 96' would require perfect accuracy at every length 41–96 while the maximum accuracy over test lengths is 0.96; if 'maximum accuracy' instead means the best mean accuracy over the run grid, or if N* includes training lengths, please state this explicitly in the caption and the text.
  2. [Fig. 2; Table 4] Figures 2 and 3 mix best-run reference lines with mean ± 1 std markers. This is acceptable, but since Table 4 reports best-run RoPENP N* values of at most 98 on (ab)* and 65 on (aa)*, the text in §3.5 could cite those maxima directly to support the claim that 'no conventional base generalizes perfectly,' making the argument less dependent on the asymmetric best-run versus mean presentation.
  3. [Abstract; §C.3] The abstract's closing phrase 'bringing theoretical expressivity characterizations closer to models used in practice' sits in tension with the Section C.3 caveat that Theorem 3.1 holds for a formula-tailored finite-precision set F and 'does not assert that standard floating-point hardware represents 2π/m ... exactly.' A sentence acknowledging the abstract-model caveat in the abstract or introduction would align the framing with the proof.
  4. [§4.2; §D] The realized-score notation bS^{g,C,k,F}_{i,j} appears in Eq. (10) before the hat convention for realized quantities is introduced in Appendix D; moving one sentence of explanation into the main text would help the reader.
  5. [Def. 2.1; §4.1] Since the realized rotation map takes values in the finite set F^{D×D}, it is necessarily eventually periodic; a remark noting that 'non-periodic' (no exact period m for all i) is compatible with eventual periodicity would prevent a natural confusion when reading Section 4.1.
  6. [Title page] The running title on the first page reads 'Disentangling the Expressivity ofRoPE' with a missing space before 'RoPE'; this should be fixed in the camera-ready version.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Theorem 3.1 is a genuine construction, and the cited NoPE-to-LTL[P] result is independent support.

full rationale

The central claim SMAT[RoPE_P] = LTL[P,MOD] is not circular. The upper bound (Lemma 3.1) uses Prop. 2.1, whose Eq. (18) expresses each realized matrix entry as a finite disjunction of MOD^a_M(i); this is a genuine reduction of periodic rotation to modular predicates, not a renaming. The lower bound (Lemma 3.2) is a constructive completeness argument: it builds a period-m lookup table T_m(s)=Rot_{2*pi/m,s} and an attention head comparing the rotated query phase against the BOS key to compute MOD^r_m(i). That the construction realizes periodic rotations by design is exactly what a lower-bound construction should do; it does not smuggle in the target class. No parameter is fitted from data and then renamed a prediction. The self-citations to Li & Cotterell (2025) for the NoPE result LTL[P]=PFO2[<] and for the structural-induction simulation are load-bearing, but that cited theorem is independently stated with assumptions (fully uniform finite-precision soft attention, no positional encodings) that do not include RoPE or modular predicates, so it counts as real evidence rather than circularity. The paper also candidly brackets its own limits: Section C.3 states that the value set F is tailored to the target formula and 'does not assert that standard floating-point hardware represents 2*pi/m... exactly,' and Section 5 states that 'the locality result of Section 4 is bounded and is not an all-length characterization of RoPE_NP.' These are acknowledged scope and abstraction limitations, not concealed inputs. The absence of an upper bound for SMAT[RoPE_NP] is a completeness gap in the practical-separation narrative, but it is explicitly flagged by the authors and is not a circular derivation.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or architectural entities. Its constructions introduce hand-chosen schedules, a tailored precision set, and a score scale, which are free parameters in the abstract model. The theoretical claims rest on the standard results listed as axioms, several of which come from the authors' prior framework.

free parameters (4)
  • Tailored finite-precision set F = chosen per formula; contains roots of unity, margins, f_large
    Lemma 3.2 and Section C.3 assume F includes exactly the values used in the modular construction; the equality holds for such F, not necessarily for 32-bit floats.
  • Component periods m_d of the custom schedule = equals the modulus of each target modular predicate
    The lower bound defines one period-m rotary pair per MOD^r_m atom; this is a hand-chosen schedule permitted by Section 2.3.
  • Score scale C = C >= 2 f_large / delta_m
    Chosen in Lemma 3.2 to guarantee the stable-softmax margin of f_large; not fitted to data.
  • RoPE base beta (experiments) = 10^-12 to 10^12 in powers of 10^4
    Control variable swept in Appendix E.4; not fit, but the empirical conclusions about conventional RoPE depend on this sweep.
assumptions (6)
  • domain assumption Soft-attention transformers without positional encoding (SMAT) recognize exactly LTL[P] (Li and Cotterell 2025).
    Used as a black box in Lemmas 3.1 and 3.2 for the non-rotary part of the simulation; the current paper does not re-derive it.
  • standard math PFO2[<] corresponds to R-trivial monoids and PODFAs (Brzozowski and Ellen 1980; Li and Cotterell 2025).
    Basis for Lemma B.1 and the algebraic characterizations in Theorem 3.3.
  • standard math Augmenting a class characterized by a local variety V with modular predicates yields QV (Dartois and Paperman 2015), and R is local (Almeida 2025).
    Used in Lemma B.2 to prove PFO2[<,MOD] = QR; if either cited result is wrong, Theorem 3.3(i) and (ii) would not follow.
  • standard math An irrational rotation generates a non-periodic symbolic sequence under finite rounding (Walters 1982, Thm 1.8).
    Prop 4.1 uses this to conclude conventional RoPE (with a 1-radian component) is non-periodic; the separation of the two accounts relies on this.
  • standard math LTL[S] = FO[<] = star-free languages (Kamp 1968; McNaughton and Papert 1971; Gabbay et al. 1980).
    Used in Section 3.4 and Prop B.2 to establish incomparability between LTL[P,MOD] and LTL[S].
  • domain assumption The fixed-precision model with stable softmax in Eq. (2), where exp(-f_large) rounds to 0, permits exact hard selection when the winning score leads by f_large.
    The modular and Y^k constructions rely on the f_large margin for exact attention; this is part of the framework from Li and Cotterell 2025.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangling the Expressivity of RoPE." pith.science (2026). https://pith.science/paper/AMBGUHTO

@misc{pith2026260811909,
  author       = {Pith},
  title        = {Pith review of: Disentangling the Expressivity of RoPE},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AMBGUHTO}},
  note         = {Machine review of arXiv:2608.11909}
}
read the original abstract

Two accounts recur in explanations of the success of rotary position embeddings (RoPE). Expressivity studies associate periodic position information with modular predicates, whereas mechanistic and long-context studies emphasize positional anchors and local offsets. We formalize both accounts for fully uniform, finite-precision soft-attention transformers. We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates. Conventional RoPE is different: The rotations it computes never repeat. This yields a precision-dependent bounded simulation of fixed-offset look-back operators, rather than an all-length modular characterization. Controlled experiments match this separation: Constructed periodic schedules length-generalize on modular languages, while conventional RoPE behaves more like a bounded locality bias and can impair tasks requiring position-invariant access to distant context. Altogether, our findings shed light on RoPE transformers, bringing theoretical expressivity characterizations closer to models used in practice.

Figures

Figures reproduced from arXiv: 2608.11909 by the authors.

Figure 1
Figure 1. The expressivity landscape of fully uniform finite-precision transformers. The [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Mean longest perfect length N∗ for conventional non-periodic encodings on the modular languages. Markers are means over the run grid and whiskers are ±1 standard deviation. The two reference lines are best-run values for NoPE and RoPEP on these tasks. Neither non-periodic encoding generalizes perfectly at any tested base. RoPEP generalizes perfectly on LTL[P, MOD]. The period-targeting RoPEP model achieves accuracy … view at source ↗
Figure 3
Figure 3. Mean longest perfect length N∗ . Whiskers are ±1 standard deviation over the run grid and the grey band is the corresponding NoPE interval. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: PODFA for Σ ∗ aΣ ∗ over Σ = {a, b}. The partial order q0⪯q1 holds since q1 is reachable from q0 but not vice versa. To link automata with modular predicates, we now introduce the k-automaton of some given automaton. In first-order logic with modular predicates, strings…
Figure 5
Figure 5. Figure 5: Minimal automata for (ab) ∗ and its corresponding 2-automaton. A.5 The Syntactic Monoid The syntactic monoid is a tool that partitions Σ ∗ into classes of strings, where each class contains strings that all contain the same syntactic information with respect to a given…
Figure 6
Figure 6. Figure 6: Minimal DFA for Σ ∗ abΣ ∗ over Σ = {a, b, c} A1 = A can be seen to not be partially ordered. Consider some integer k > 1 and the k-automaton Ak of A. Let wa ∈ Σ k be a string of k times the symbol a, and wc ∈ Σ k a string 4Note that over the two-letter alphabet Σ = {a,…
Figure 7
Figure 7. Figure 7: Minimal DFA for PARITY over Σ = {0, 1} A1 = A can be seen to not be partially ordered. Consider some integer k > 1 and the k-automaton Ak of A. Let w be a string of length k with an odd number of 1s. We have that δ k (q0, w) = q1 and δ k (q1, w) = q0. Therefore, there …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 25 canonical work pages

  1. [1]

    arXiv , author =:2407.21783 , primaryclass =

    The Llama 3 Herd of Models , url =. arXiv , author =:2407.21783 , primaryclass =

    arXiv

  2. [2]

    Meyer and C

    Albert R. Meyer and C. Thompson , journal =. Remarks on algebraic decomposition of automata , url =

    work page

  3. [3]

    arXiv , author =:2411.07107 , primaryclass =

    Training Neural Networks as Recognizers of Formal Languages , url =. arXiv , author =:2411.07107 , primaryclass =

    arXiv

  4. [4]

    Pseudovarieties of semigroups , url =

    Almeida, Jorge , doi =. Pseudovarieties of semigroups , url =. Asian-European Journal of Mathematics , month = may, publisher =

    work page

  5. [5]

    arXiv , author =:2305.19466 , primaryclass =

    The Impact of Positional Encoding on Length Generalization in Transformers , url =. arXiv , author =:2305.19466 , primaryclass =

    arXiv

  6. [6]

    arXiv , author =:2505.09388 , primaryclass =

    Qwen3 Technical Report , url =. arXiv , author =:2505.09388 , primaryclass =

    arXiv

  7. [7]

    arXiv , author =:2510.00368 , primaryclass =

    The Transformer Cookbook , url =. arXiv , author =:2510.00368 , primaryclass =

    arXiv

  8. [8]

    Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers , url =

    Andy Yang and David Chiang , booktitle =. Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers , url =

    work page

Show all 72 references
  1. [9]

    arXiv , author =:2310.13897 , primaryclass =

    Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages , url =. arXiv , author =:2310.13897 , primaryclass =

    arXiv

  2. [10]

    arXiv , author =:2506.16055 , primaryclass =

    Knee-Deep in C-RASP: A Transformer Depth Hierarchy , url =. arXiv , author =:2506.16055 , primaryclass =

  3. [11]

    Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't , url =

    Anej Svete and William Merrill and Ryan Cotterell and Ashish Sabharwal , booktitle =. Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't , url =

  4. [12]

    arXiv , author =:1706.03762 , primaryclass =

    Attention Is All You Need , url =. arXiv , author =:1706.03762 , primaryclass =

    arXiv

  5. [13]

    arXiv , author =:2308.12950 , primaryclass =

    Code Llama: Open Foundation Models for Code , url =. arXiv , author =:2308.12950 , primaryclass =

    arXiv

  6. [14]

    arXiv , author =:2306.00946 , primaryclass =

    Exposing Attention Glitches with Flip-Flop Language Modeling , url =. arXiv , author =:2306.00946 , primaryclass =

    arXiv

  7. [15]

    Ash and Surbhi Goel and Akshay Krishnamurthy and Cyril Zhang , booktitle =

    Bingbin Liu and Jordan T. Ash and Surbhi Goel and Akshay Krishnamurthy and Cyril Zhang , booktitle =. Transformers Learn Shortcuts to Automata , url =

  8. [16]

    Buss, Samuel , booktitle =. The. doi:10.1145/28395.28409 , isbn =

  9. [17]

    Circuit Complexity Bounds for

    Chen, Bo and Li, Xiaoyu and Liang, Yingyu and Long, Jiangxuan and Shi, Zhenmei and Song, Zhao and Zhang, Jiahao , booktitle =. Circuit Complexity Bounds for. doi:10.18653/v1/2025.emnlp-main.561 , editor =

    2025 doi

  10. [18]

    doi:10.4230/LIPIcs.STACS.2013.329 , editor =

    Dartois, Luc and Paperman, Charles , booktitle =. doi:10.4230/LIPIcs.STACS.2013.329 , editor =

    2013 doi

  11. [19]

    Regular languages in NC1 , url =

    David A. Regular languages in NC1 , url =. Journal of Computer and System Sciences , number =. doi:https://doi.org/10.1016/0022-0000(92)90014-A , issn =

    doi

  12. [20]

    Transformers in Uniform

    David Chiang , issn =. Transformers in Uniform. Transactions on Machine Learning Research , note =

  13. [21]

    arXiv , author =:2202.12172 , primaryclass =

    Overcoming a Theoretical Limitation of Self-Attention , url =. arXiv , author =:2202.12172 , primaryclass =

    arXiv

  14. [22]

    arXiv , author =:2301.10743 , primaryclass =

    Tighter Bounds on the Expressivity of Transformer Encoders , url =. arXiv , author =:2301.10743 , primaryclass =

    arXiv

  15. [23]

    arXiv , author =:2512.02556 , primaryclass =

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , url =. arXiv , author =:2512.02556 , primaryclass =

    arXiv

  16. [24]

    arXiv , author =:2606.19348 , primaryclass =

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , url =. arXiv , author =:2606.19348 , primaryclass =

  17. [25]

    arXiv , author =:2410.06205 , primaryclass =

    Round and Round We Go! What makes Rotary Positional Encodings useful? , url =. arXiv , author =:2410.06205 , primaryclass =

    arXiv

  18. [26]

    arXiv , author =:2504.02732 , primaryclass =

    Why do LLMs attend to the first token? , url =. arXiv , author =:2504.02732 , primaryclass =

    arXiv

  19. [27]

    On the temporal analysis of fairness , url =

    Gabbay, Dov and Pnueli, Amir and Shelah, Saharon and Stavi, Jonathan , booktitle =. On the temporal analysis of fairness , url =. doi:10.1145/567446.567462 , isbn =

  20. [28]

    arXiv , author =:2403.08295 , primaryclass =

    Gemma: Open Models Based on Gemini Research and Technology , url =. arXiv , author =:2403.08295 , primaryclass =

    arXiv

  21. [29]

    arXiv , author =:2207.02098 , primaryclass =

    Neural Networks and the Chomsky Hierarchy , url =. arXiv , author =:2207.02098 , primaryclass =

    arXiv

  22. [30]

    arXiv , author =:2309.17453 , primaryclass =

    Efficient Streaming Language Models with Attention Sinks , url =. arXiv , author =:2309.17453 , primaryclass =

    arXiv

  23. [31]

    Venkateswaran , doi =

    H. Venkateswaran , doi =. Properties that characterize LOGCFL , url =. Journal of Computer and System Sciences , number =

  24. [32]

    Theoretical Limitations of Self-Attention in Neural Sequence Models , url =

    Hahn, Michael , doi =. Theoretical Limitations of Self-Attention in Neural Sequence Models , url =. Transactions of the Association for Computational Linguistics , pages =

  25. [33]

    Why are Sensitive Functions Hard for Transformers? , url =

    Hahn, Michael and Rofin, Mark , booktitle =. Why are Sensitive Functions Hard for Transformers? , url =. doi:10.18653/v1/2024.acl-long.800 , editor =

    2024 doi

  26. [34]

    Tense Logic and the Theory of Linear Order , url =

    Hans Kamp , school =. Tense Logic and the Theory of Linear Order , url =

  27. [35]

    Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity , url =

    Hao, Yiding and Angluin, Dana and Frank, Robert , doi =. Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity , url =. Transactions of the Association for Computational Linguistics , pages =

  28. [36]

    Languages of

    Janusz Antoni Brzozowski and Faith Ellen , doi =. Languages of. Journal of Computer and System Sciences , number =

  29. [37]

    Unique Hard Attention: A Tale of Two Sides , url =

    Jerad, Selim and Svete, Anej and Li, Jiaoda and Cotterell, Ryan , booktitle =. Unique Hard Attention: A Tale of Two Sides , url =. doi:10.18653/v1/2025.acl-short.76 , editor =

    2025 doi

  30. [38]

    arXiv , author =:2104.09864 , primaryclass =

    RoFormer: Enhanced Transformer with Rotary Position Embedding , url =. arXiv , author =:2104.09864 , primaryclass =

    arXiv

  31. [39]

    arXiv , author =:2605.00768 , primaryclass =

    Characterizing the Expressivity of Local Attention in Transformers , url =. arXiv , author =:2605.00768 , primaryclass =

    arXiv

  32. [40]

    arXiv , author =:2505.23623 , primaryclass =

    Characterizing the Expressivity of Fixed-Precision Transformer Language Models , url =. arXiv , author =:2505.23623 , primaryclass =

  33. [41]

    arXiv , author =:2309.16609 , primaryclass =

    Qwen Technical Report , url =. arXiv , author =:2309.16609 , primaryclass =

    arXiv

  34. [42]

    arXiv , author =:1401.6576 , primaryclass =

    Adding modular predicates to first-order fragments , url =. arXiv , author =:1401.6576 , primaryclass =

    arXiv

  35. [43]

    Counter-Free Automata , url =

    McNaughton, Robert and Papert, Seymour , isbn =. Counter-Free Automata , url =

  36. [44]

    arXiv , author =:2602.07599 , primaryclass =

    Rational Transductors , url =. arXiv , author =:2602.07599 , primaryclass =

  37. [45]

    The Parallelism Tradeoff: Limitations of Log-Precision Transformers , url =

    Merrill, William and Sabharwal, Ashish , doi =. The Parallelism Tradeoff: Limitations of Log-Precision Transformers , url =. Transactions of the Association for Computational Linguistics , pages =

  38. [46]

    Merrill, William and Sabharwal, Ashish and Smith, Noah A. , doi =. Saturated Transformers are Constant-Depth Threshold Circuits , url =. Transactions of the Association for Computational Linguistics , pages =

  39. [47]

    arXiv , author =:2603.11611 , primaryclass =

    Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE , url =. arXiv , author =:2603.11611 , primaryclass =

  40. [48]

    On the Relation between Position Information and Sentence Length in Neural Machine Translation , url =

    Neishi, Masato and Yoshinaga, Naoki , booktitle =. On the Relation between Position Information and Sentence Length in Neural Machine Translation , url =. doi:10.18653/v1/K19-1031 , editor =

    doi

  41. [49]

    Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation , url =

    Ofir Press and Noah Smith and Mike Lewis , booktitle =. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation , url =

  42. [50]

    The temporal logic of programs , url =

    Pnueli, Amir , booktitle =. The temporal logic of programs , url =. doi:10.1109/SFCS.1977.32 , keywords =

    1977 doi

  43. [51]

    arXiv , author =:2509.14233 , primaryclass =

    Apertus: Democratizing Open and Compliant LLMs for Global Language Environments , url =. arXiv , author =:2509.14233 , primaryclass =

  44. [52]

    Analysis of Positional Encodings for Neural Machine Translation , url =

    Rosendahl, Jan and Tran, Viet Anh Khoa and Wang, Weiyue and Ney, Hermann , booktitle =. Analysis of Positional Encodings for Neural Machine Translation , url =

  45. [53]

    arXiv , author =:2304.07687 , primaryclass =

    MLRegTest: A Benchmark for the Machine Learning of Regular Languages , url =. arXiv , author =:2304.07687 , primaryclass =

    arXiv

  46. [54]

    arXiv , author =:2009.11264 , primaryclass =

    On the Ability and Limitations of Transformers to Recognize Formal Languages , url =. arXiv , author =:2009.11264 , primaryclass =

    2009 arXiv

  47. [55]

    arXiv , author =:2605.27980 , primaryclass =

    Periodic RoPE for Infinite Context LLMs , url =. arXiv , author =:2605.27980 , primaryclass =

    arXiv

  48. [56]

    What Formal Languages Can Transformers Express? A Survey , url =

    Strobl, Lena and Merrill, William and Weiss, Gail and Chiang, David and Angluin, Dana , doi =. What Formal Languages Can Transformers Express? A Survey , url =. Transactions of the Association for Computational Linguistics , pages =

  49. [57]

    Can Transformers Learn n -gram Language Models? , url =

    Svete, Anej and Borenstein, Nadav and Zhou, Mike and Augenstein, Isabelle and Cotterell, Ryan , booktitle =. Can Transformers Learn n -gram Language Models? , url =. doi:10.18653/v1/2024.emnlp-main.550 , editor =

    2024 doi

  50. [58]

    Transformers Can Represent n -gram Language Models , url =

    Svete, Anej and Cotterell, Ryan , booktitle =. Transformers Can Represent n -gram Language Models , url =. doi:10.18653/v1/2024.naacl-long.381 , editor =

    2024 doi

  51. [59]

    arXiv , author =:2512.13961 , primaryclass =

    Olmo 3 , url =. arXiv , author =:2512.13961 , primaryclass =

    arXiv

  52. [60]

    arXiv , author =:1010.3455 , primaryclass =

    On the representation theory of finite J-trivial monoids , url =. arXiv , author =:1010.3455 , primaryclass =

    arXiv

  53. [61]

    , isbn =

    Walters, P. , isbn =. An Introduction to Ergodic Theory , url =

  54. [62]

    Resonance

    Wang, Suyuchen and Kobyzev, Ivan and Lu, Peng and Rezagholizadeh, Mehdi and Liu, Bang , booktitle =. Resonance. doi:10.18653/v1/2024.findings-acl.32 , editor =

    2024 doi

  55. [63]

    arXiv , author =:2210.02671 , primaryclass =

    A Logic for Expressing Log-Precision Transformers , url =. arXiv , author =:2210.02671 , primaryclass =

    arXiv

  56. [64]

    arXiv , author =:2603.03612 , primaryclass =

    Why Are Linear RNNs More Parallelizable? , url =. arXiv , author =:2603.03612 , primaryclass =

    arXiv

  57. [65]

    arXiv , author =:2604.03444 , primaryclass =

    Olmo Hybrid: From Theory to Practice and Back , url =. arXiv , author =:2604.03444 , primaryclass =

    arXiv

  58. [66]

    arXiv , author =:2310.05209 , primaryclass =

    Scaling Laws of RoPE-based Extrapolation , url =. arXiv , author =:2310.05209 , primaryclass =

    arXiv

  59. [67]

    arXiv , author =:2412.18040 , primaryclass =

    Theoretical Constraints on the Expressive Power of. arXiv , author =:2412.18040 , primaryclass =

    arXiv

  60. [68]

    arXiv , author =:2405.14591 , primaryclass =

    Base of RoPE Bounds Context Length , url =. arXiv , author =:2405.14591 , primaryclass =

    arXiv

  61. [69]

    Rope to Nope and Back Again: A New Hybrid Attention Strategy , url =

    Yang, Bowen and Venkitesh, Bharat and Talupuru, Dwaraknath Gnaneshwar and Lin, Hangyu and Cairuz, David and Blunsom, Phil and Locatelli, Acyr , booktitle =. Rope to Nope and Back Again: A New Hybrid Attention Strategy , url =

  62. [70]

    Locally testable languages , url =

    Yechezkel Zalcstein , doi =. Locally testable languages , url =. Journal of Computer and System Sciences , number =

  63. [71]

    arXiv , author =:2512.12167 , primaryclass =

    Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings , url =. arXiv , author =:2512.12167 , primaryclass =

  64. [72]

    arXiv , author =:2605.15514 , primaryclass =

    RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably , url =. arXiv , author =:2605.15514 , primaryclass =

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.