Pith.

Physics and Society · arXiv:2611.13146 v1 · submitted 9 Nov 2026 · 44 references

Everything in its right place: attention is a radio head

Lluís Arola-Fernández, Aissam Khadraoui, Alex Arenas

The paper says

“Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order.”

more of the abstract

We show that attention can instead emerge from the synchronization of a receiver with its context tokens. Each past token is an oscillator whose state carries its content and whose frequency drifts with its age. The present token selects them by content, aligning an antenna with its query, and by position, setting the frequency of a receiver with its dial. Solving Kuramoto's problem for this receiver yields an attention layer in closed form with two phases: above a critical coupling set by the context, the receiver locks and its weights normalize; below it, it stays silent and its weights sum to less than one. Trained networks use both phases to select, abstain and count, fail where the theory predicts and keep reading far beyond the training length. Each part of attention finds its place in one device, a radio head: it tunes in to what to hear and when, and stays silent when nothing relevant is on air.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 11 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.

A genuinely new closed-form derivation of attention from synchronization, sound in its stated regime, but the headline extrapolation and content-reading claims are delivered by slow heads outside that regime — a scoping gap worth pushing in revision. This line is wrong

4 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 44 references say.

The paper says, at §S1 C, §S2, Fig. 2(c), Fig. 3(h)

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.67):

“In narrow or slow heads it becomes the mean field of the dynamics rather than its exact state.”

Pith says: Objection 1 of 4 · major

The central approximation (averaging beats) is valid only for resolved receivers where epsilon >= Kr (b89, b228). The paper's most striking downstream claim—extrapolation in context length (Fig. 3h, b119)—is carried by the slowest heads (epsilon ~ 0.01, b118), which are unresolved (Kr/epsilon >> 1).

more of objection 1

The paper acknowledges that for such heads the closed form is 'the mean field of the dynamics rather than its exact state' (b126) and reports 10–20% deviation (b92). The theoretical explanation for extrapolation (convergent sum Sigma G^2/Delta^2, S1 H) is derived within the averaged framework whose validity condition the slow heads violate. While the empirical demonstration of extrapolation is valuable, the theoretical guarantee covering the most visible downstream claim is not established for the heads that deliver it. The authors should either (a) provide a separate argument for why the mean-field approximation suffices for the extrapolation prediction, or (b) explicitly state that the extrapolation mechanism is a plausible interpretation supported by the mean-field form but not rigorously proven for the unresolved regime.

The paper says, at §S1 C, §S2, Fig. 2(c)

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The error scaling Kr/(m epsilon) is derived and tested for resolved receivers (b91). However, the paper also claims that the closed form holds within 10–20% for unresolved receivers (b92). The source of this error bound is not derived; it is an empirical observation from the specific trained contexts tested.

more of objection 2

The paper should clarify whether this 10–20% bound is expected to hold generally (e.g., for any context and any head parameters) or is a property of the particular trained models. If the latter, the scope of the theoretical claims should be narrowed accordingly.

The paper says, at §S4, Table S2

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The training protocol uses a single pass over TinyStories (one epoch) and a fixed learning rate schedule. The comparison with softmax and PG is performed under this protocol. While the paper checks robustness to protocol variations (S5 C), the absolute performance numbers (perplexity ~7–9) are far from state-of-the-art on TinyStories.

more of objection 3

This is not a flaw per se, but it limits the strength of the claim that the radio head 'matches' softmax accuracy. The claim should be qualified as holding for small transformers under a specific training protocol, not as a general equivalence.

The paper says, at §S1 H, Fig. 3(d) inset

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The paper derives a scaling relation n ~ ln(Kr/epsilon)/ln(m) for the directivity (S1 H) and reports a linear correlation between n and ln(1/epsilon) in trained networks (Fig. 3d inset, r=0.97). This is presented as a consistency check. However, the derivation assumes that the head resolves about Kr/epsilon tokens in its band and that each unrelated token contributes squared gain m^{-n}.

more of objection 4

The empirical correlation is based on only 8 data points (2 seeds, 4 heads each). The authors should report the uncertainty on the correlation and ideally test on more seeds or a wider range of model sizes to support the claimed scaling law.

7 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 7

The abstract states 'attention can instead emerge from the synchronization of a receiver with its context tokens.' This is a strong claim; consider adding a qualifier such as 'in a model where each token is an oscillator' to clarify the scope.

The paper says, at Eq. (6)

Pith says: minor objection 2 of 7

The definition of w_ij in Eq. (6) uses Kr_i + i Delta_ij in the denominator. It would be helpful to explicitly note that r_i is the order parameter defined in Eq. (4) and that the denominator is complex, so w_ij is a complex amplitude whose squared modulus gives the attention weight.

show 5 more

The paper says, at §S1 F

Pith says: minor objection 3 of 7

The linewidth D is introduced to regularize the transition for training. The paper states D = 10^{-3} K. It would be useful to comment on the sensitivity of the results to the value of D and whether the same value works across all tasks and model sizes.

The paper says, at Fig. 2(a)

Pith says: minor objection 4 of 7

The figure shows both resolved and unresolved receivers. The unresolved receivers show systematic deviations from the root of Eq. (10). The caption should explicitly note that the closed form is not exact for unresolved receivers and that the deviation is expected.

The paper says, at §S5 A, Table of perplexities

Pith says: minor objection 5 of 7

The table reports perplexity and accuracy for softmax, radio head, and PG at various context lengths. The radio head's perplexity at length 1024 is 9.42, which is higher than at length 128 (6.99). The paper attributes this to the convergent sum, but the increase from 6.99 to 9.42 is non-negligible. A brief comment on the source of this degradation would be helpful.

The paper says, at References

Pith says: minor objection 6 of 7

Reference [S7] is a bachelor thesis. While this is acceptable, the authors should ensure that the thesis is publicly accessible or provide a link if possible.

The paper says, at General

Pith says: minor objection 7 of 7

The paper uses the term 'radio head' throughout. While evocative, this term is not standard in the literature. The authors should define it clearly at first use and consider whether a more descriptive term (e.g., 'synchronization attention layer') would aid readability.

What would settle it

On the trained language radio head (d_model = 256, four heads, Fig 3h), take the slowest head (smallest learned ε, deepest lock). At context lengths 128, 256, 512, and 1024, directly integrate Eq. (S5) (fourth-order Runge–Kutta, renormalizing each step, as in S2) for that head's receivers using the learned G_ij, Δ_ij and contents, without the beat-averaging approximation, and compare the demodulated per-token amplitudes (Eq. S3 readout) against the closed-form weights of Eq. (6) as the layer computes them. If the relative deviation grows with context length — from the ~10–20% measured at length 128 toward uncontrolled values at 1024 — then the closed form and its Σ Δ^{-2} convergence argument do not cover the extrapolation claim's own regime, and the beyond-training-length accuracy is carried by dynamics the theory does not control.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 11 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees: says it holds up only with conditions 1 min 13 s · 22,861 tokens read, 10,621 written The closed form is exact only for resolved receivers (ε ≥ K_c), yet the paper's signature extrapolation and content-reading claims live in the unresolved slow-head regime where the one approximation is uncontrolled; the theoretical explanation of extrapolation is not established for the heads…
  3. Editor wrote the verdict above 29 s · 24,012 tokens read, 4,977 written
  4. Referee major revision: 4 major objections, 7 minor 31 s · 23,470 tokens read, 1,818 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Short version: this paper is worth a serious referee, and I'd take it. It derives attention in closed form from a synchronization model — position as frequency, silence below a critical coupling — and the derivation is self-contained, with one stated approximation and error scaling verified numerically. The real caveat is a scoping gap: the headline extrapolation and content-reading results are carried by slow heads outside the regime where the closed form is exact, and the theory's own mechanism for those results is not established there.

What's new: Pasqualetti and Guo's sync-based attention had no natural frequencies, hence no position and no silence. The radio head adds age-dependent frequency drift, so a token's age is a detuning, and a threshold below which the receiver abstains. Eq. (6) is derived, not fitted: one self-consistency condition yields both phases, and the only approximation — averaging beats — is exact for orthogonal contents, with error proportional to Kr/(m epsilon), verified in Fig. 2. That is a checkable, reproducible result, and the paper ships the code. The training work is honest: wide heads approach softmax, induction accuracy collapses at the locking-range edge as predicted, and counting uses the silent phase. The authors disclose the bachelor-thesis provenance and state their own limitations. Good.

Now the soft spot, and I think the stress-test note is right about it. The validity condition is epsilon > Kr (resolved receivers; b89, b221, b228). The most distinctive trained behaviors — reading by content alone, reading far beyond the training length — are credited to the slowest heads (b118, b119), which are unresolved: epsilon around 0.01 with K/Kc of 10-20 puts several tens of tokens inside the band. The paper itself says slow heads follow the closed form only within 10-20%, as a mean field rather than the exact state (b92, b126), yet the extrapolation explanation via the convergent sum of G2/Delta2 is applied exactly there. So the empirical demonstrations stand, the mechanism is plausible, but the theory does not actually cover the heads that deliver the signature claims. I'd ask the authors to separate how much of each result lives in the exact regime versus the mean-field one, or to prove the qualitative statements (convergent threshold, decaying tails) hold beyond the averaging condition.

Minor, and acknowledged: everything is small-scale, one pass over TinyStories; scaling is open.

For you: if you care about physical accounts of attention, oscillator hardware, or the attention-as-synchronization line, read it. It deserves a real referee. My own verdict: accept with revision, with the regime question as the main ask.

The strongest objection

The derivation rests on one approximation — averaging the beats between token modes — valid when the beats (at multiples of ε) are faster than the receiver band Kr, i.e., ε > Kr, the 'resolved' condition stated in S1 C and S2 (b89, b221, b228). Figure 2a–c validates the closed form for resolved receivers, with error ∝ Kr/(mε). But the trained networks' most distinctive behaviors are carried by the slowest heads: b118 says the slowest head 'locks deepest and reads by content alone', and S5 reports median K/K_c of 10–20 for it. For a slow head (ε ≈ 0.01, trained values spread down to 0.01), Kr/ε runs to tens of tokens, so ε < Kr and the head is unresolved: beats between tokens inside the band oscillate slower than the band and do not average out. The paper labels exactly this situation as 'the mean field of the dynamics rather than its exact state' (b126) and reports 10–20% deviation for such heads (b92). The headline extrapolation claim ('keeps reading far beyond the training length', b119, Fig 3h) is explained via the closed form's convergent sum Σ G²/Δ² (S1 H, S5), but that explanation is phrased in the averaged, mean-field framework whose validity condition the slow heads violate. Thus the theoretical guarantee covering the most visible downstream claim is not established for the heads that deliver it; the extrapolation is demonstrated empirically by trained networks, with a plausible but uncontrolled mechanism behind it. This is a scoping gap, not an inconsistency: the derivation is sound in its stated resolved, wide-head regime, and the paper explicitly acknowledges slow heads are mean-field. What is unresolved is that the signature claims (extrapolation, resonance-based position, content reading) are validated empirically in the regime where the theory's own validity condition fails, without separating how much of each result lives in the exact versus the mean-field regime.

The reader's weakest_assumption (beat averaging, error O(Kr/mε), narrow/slow-head limitation) identifies the same root: the one approximation in the derivation. My stress-test sharpens it to a scoping gap rather than a correctness failure. The central derivation — the closed form, the threshold K_c = (Σ G²/Δ²)^{-1/2}, the normalization in power, and the silent phase Σ a_j = K²/K_c² — is internally consistent, machine-checkable in its algebra, and empirically validated for resolved receivers (Fig 2a–c), where the error scaling Kr/(mε) is confirmed. The silent-state stability argument (matrix determinant lemma, S1 D) is clean. The paper's own statements support the gap: b126 concedes narrow or slow heads make the closed form a mean field, b92 bounds the slow-head deviation to 10–20%, and S5 notes the slowest head locks deepest (median K/K_c 10–20) while ε spreads down to 0.01, so Kr/ε ≫ 1 for precisely those heads. Yet the abstract's most visible claim — 'keep reading far beyond the training length' — and the S1 H explanation of that behavior (Lorentzian tail, convergent Σ Δ^{-2}) are presented in the mean-field framework whose validity condition (ε > Kr) those heads fail. The trained-network extrapolation result (Fig 3h) is real empirical evidence, but it does not by itself verify that the closed-form theory predicts it; the paper also concedes (S5) the language test concatenates unrelated stories, so age-decaying weights in any form suffice there, further weakening the attribution of the extrapolation to the theory's resonance mechanism. Because the derivation and the resolved-regime results are solid, I do not reject; but the headline extrapolation and phase-behavior claims should be explicitly scoped to the resolved regime, or verified directly by dynamics integration for the slow heads, before the theory is credited with them. Hence CONDITIONAL: accept the derivation and the resolved-receiver validation, condition the acceptance on separating the exact from the mean-field regime in the extrapolation and content-reading claims. This agrees with the reader's identification of the approximation as weakest, but goes beyond it by observing that the paper's own signature claims are concentrated in the regime where that approximation is uncontrolled.

The referee's report

If the central claim holds, the paper provides a principled physical derivation of attention from synchronization, unifying selection, normalization, and positional encoding under a single self-consistency condition. This is a significant conceptual advance over existing engineering-oriented approaches. The derivation is self-contained and the closed form is tested against numerical integration. The paper makes falsifiable predictions (e.g., the scaling of error with 1/m, the phase boundary for induction failure) and demonstrates them in trained networks. The work also opens a potential path to physical implementations of attention using oscillators. The strengths include: a parameter-free closed form derived from first principles, explicit error scaling, and empirical validation on small transformers showing that the layer matches softmax accuracy in wide heads and extrapolates beyond the training length.

Pith's reading of the paper (a machine's summary, not the authors')

This paper claims that the three designed ingredients of softmax attention — the exponential selection, the normalizing division, and the added positional code — can instead fall out of one physical picture. In that picture every past token is an oscillator whose phase carries its content and whose frequency drifts with age, so position becomes a frequency and the present token is a receiver with a dial. Solving the receiver's self-consistency — the Kuramoto problem, generalized by Lohe to vectors — yields attention weights in closed form, $w_{ij}=KG_{ij}/(Kr+i\Delta_{ij})$, together with a critical coupling $K_c$ that separates a locked phase, in which the weights normalize to sum one, from a silent phase, in which they sum to less than one. The authors test this in small trained transformers and report that wide heads match softmax accuracy, and that the same phase boundaries predict where the networks abstain, count, and fail.

What carries the argument. The load-bearing object is the Lohe equation, Kuramoto's synchronization model on the sphere: $\dot z_i=-i\alpha_i z_i+K(h_i-r_i z_i)$ with $|z_i|=1$. The unit-norm constraint is where everything is decided — it is what forces normalization above threshold, what creates the critical coupling, and what makes the silent phase possible, since a linear receiver would neither normalize nor switch off. The closed-form amplitudes $w_{ij}=KG_{ij}/(Kr+i\Delta_{ij})$ are the responses of damped oscillators driven at each token's detuning, with a common friction $Kr$; the self-consistency condition $\sum_j K^2G_j^2/((Kr)^2+\Delta_j^2)=1$ fixes that friction and has a root only for $K>K_c$. The single approximation of the derivation is the rotating-wave averaging of beats between tokens, exact for orthogonal contents, and its error $Kr/m\varepsilon$ means the head dimension plays the role that population size plays in Kuramoto's theory.

Core claim. The paper's central claim is that attention is the stationary state of a listener. The present token $i$ is a receiver $z_i$, a unit vector in $\mathbb{C}^m$ following Lohe's equation, pulled with strength $K$ toward the field of its context; each earlier token $j$ is an oscillator carrying its value $v_j$ and turning at a frequency set by its age, so the detuning $\Delta_{ij}=\alpha_i-\varepsilon(i-j)$ encodes position. A query sets the antenna gain $G_{ij}=|\langle q_i|v_j\rangle|^n$, and a dial sets $\alpha_i$, so content selects what to hear and position selects when. With one approximation — averaging out beats between tokens, exact for orthogonal contents and of order $Kr/m\varepsilon$ otherwise — the locked receiver is a superposition of token modes with weights $w_{ij}=KG_{ij}/(Kr+i\Delta_{ij})$, and Kuramoto's self-consistency closes the equations: above $K_c=(\sum_j G_j^2/\Delta_j^2)^{-1/2}$ the receiver locks and $\sum_j|w_{ij}|^2=1$ (normalization), while below it the receiver stays silent, the weights sum to $K^2/K_c^2<1$, and the missing norm sits in an unread remainder. Normalization, abstention, and counting are therefore three readings of one synchronization transition, and position is a Lorentzian resonance rather than an added code.

The premise it rests on. The derivation assumes that beats between tokens average out to zero, which is exact only for orthogonal token contents; for narrow or slow heads the error $Kr/m\varepsilon$ grows, and once $\varepsilon<K_c$ the authors themselves state that the closed form becomes a mean-field approximation of the dynamics rather than its exact state.

If the paper is right (Pith's inferences, not the authors')

Selection, normalization, and position cease to be separate design choices: all three follow from one self-consistency condition, and the layer contains no exponential, no key feature dimension, and no added positional code.

Abstention becomes automatic: a query that matches nothing keeps its receiver silent, the weights sum to $K^2/K_c^2<1$, and no attention-sink token is needed to absorb the excess.

Counting falls out of the silent phase: below threshold the total weight is a sum over tokens, so copies of a query add linearly until the receiver saturates at one, while softmax, always normalized to one, can only report a length-dependent fraction.

Extrapolation beyond the trained length follows from resonance: a token is heard only inside its locking range, and because the Lorentzian tail decays as the inverse square of age, older tokens add a bounded background rather than a growing one.

The phase diagram is predictive: at the trained length, wide heads match the accuracy of softmax with rotary position on the language task, and the measured failure points coincide with the theoretically predicted phase boundaries.

My extension: because position is encoded as an age rather than an absolute index, the layer should be invariant to left-shifting a context and robust to repeated spans of text — a distinguishable property from the extrapolation the paper measures, and directly testable by a shift-perturbation experiment.

My extension: the $Kr/m\varepsilon$ error scaling implies a testable law for standard transformers — next-token accuracy should degrade predictably as head dimension shrinks, and the accuracy gap versus softmax should close as $m\to\infty$; the reported curves point this way but the paper does not claim it.

My extension: whether large pretrained models actually use the silent phase is left open by the paper; a practical diagnostic would measure the fraction of near-silent heads on natural text and check whether that fraction correlates with calibrated abstention behavior.

My extension: the construction suggests a physical implementation in which coupled spin-torque, electronic, or optical oscillators perform one attention read in a few oscillation periods; the paper points at this hardware route but does not demonstrate it.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order. We show that attention can instead emerge from the synchronization of a receiver with its context tokens. Each past token is an oscillator whose state carries its content and whose frequency drifts with its age. The present token selects them by content, aligning an antenna with its query, and by position, setting the frequency of a receiver with its dial. Solving Kuramoto's problem for this receiver yields an attention layer in closed form with two phases: above a critical coupling set by the context, the receiver locks and its weights normalize; below it, it stays silent and its weights sum to less than one. Trained networks use both phases to select, abstain and count, fail where the theory predicts and keep reading far beyond the training length. Each part of attention finds its place in one device, a radio head: it tunes in to what to hear and when, and stays silent when nothing relevant is on air.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-11

The authors' main claim, as the classifier picked it out 0.71

“We show that attention can instead emerge from the synchronization of a receiver with its context tokens.”

How the main claim is stated: asserted 0.93

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.73 · human participants 0.02 · controlled comparison 0.05 · randomized assignment 0.01 · states quantitative result 0.27 · compares with prior results 0.29 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order. background 1.00 0.10 0.04 asserted 1.00
We show that attention can instead emerge from the synchronization of a receiver with its context tokens. contribution 0.98 0.87 0.69 asserted 0.93
Each past token is an oscillator whose state carries its content and whose frequency drifts with its age. method 0.84 0.41 0.06 asserted 1.00
The present token selects them by content, aligning an antenna with its query, and by position, setting the frequency of a receiver with its dial. method 0.72 0.37 0.16 asserted 1.00
Solving Kuramoto's problem for this receiver yields an attention layer in closed form with two phases: above a critical coupling set by the context, the receiver locks and its weights normalize; below it, it stays silent and its weights sum to less than one. result 0.89 0.86 0.26 asserted 1.00
Trained networks use both phases to select, abstain and count, fail where the theory predicts and keep reading far beyond the training length. result 0.97 0.85 0.18 asserted 1.00
Each part of attention finds its place in one device, a radio head: it tunes in to what to hear and when, and stays silent when nothing relevant is on air. interpretation 1.00 0.45 0.62 asserted 1.00

What kind of paper the classifier reports this is: theoretical result 1.00 · evidence from proof or derivation 0.95

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-11: 1 request, 5,600 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. FIG. 1. The radio head. Each token is a station, numbered
Figure 2 of the paper
Figure 2. FIG. 2. The receiver and the layer against
Figure 3 of the paper
Figure 3. FIG. 3. The receiver in trained networks (radio head; synthetic tasks: two layers, one head,

Figures are the authors', reproduced from the paper.

References (44)
  1. [1] E. C. Cherry, Some experiments on the recognition of speech, with one and with two ears, J. Acoust. Soc. Am. 25, 975 (1953) on Pith
  2. [2] N. Ding and J. Z. Simon, Emergence of neural encoding of auditory objects while listening to competing speakers, Proc. Natl. Acad. Sci. USA109, 11854 (2012) on Pith
  3. [3] E. M. Zion Golumbicet al., Mechanisms underlying selec- tive neuronal tracking of attended speech at a “cocktail party”, Neuron77, 980 (2013) on Pith
  4. [4] W. Singer and C. M. Gray, Visual feature integration and the temporal correlation hypothesis, Annu. Rev. Neu- rosci.18, 555 (1995) on Pith
  5. [5] P. Fries, Rhythms for cognition: communication through coherence, Neuron88, 220 (2015) on Pith
  6. [6] A. Vaswaniet al., Attention is all you need, NeurIPS (2017) on Pith
  7. [7] J. Suet al., RoFormer: enhanced transformer with ro- tary position embedding, Neurocomputing568, 127063 (2024) on Pith
  8. [8] G. Xiaoet al., Efficient streaming language models with attention sinks, ICLR (2024) on Pith
  9. [9] Z. Qiuet al., Gated attention for large language models: non-linearity, sparsity, and attention-sink-free, NeurIPS (2025) on Pith
  10. [10] P. Veliˇ ckovi´ cet al., Softmax is not enough (for sharp size generalisation), ICML (2025) on Pith
  11. [11] O. Press, N. A. Smith and M. Lewis, Train short, test 6 long: attention with linear biases enables input length extrapolation, ICLR (2022) on Pith
  12. [12] B. Penget al., YaRN: efficient context window extension of large language models, ICLR (2024) on Pith
  13. [13] J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. USA79, 2554 (1982) on Pith
  14. [14] D. H. Ackley, G. E. Hinton and T. J. Sejnowski, A learn- ing algorithm for Boltzmann machines, Cogn. Sci.9, 147 (1985) on Pith
  15. [15] J. Sohl-Dicksteinet al., Deep unsupervised learning using nonequilibrium thermodynamics, ICML (2015) on Pith
  16. [16] J. Kaplanet al., Scaling laws for neural language models, arXiv:2001.08361 (2020) on PithPith's reviewarXiv
  17. [17] J. Hoffmannet al., Training compute-optimal large lan- guage models, arXiv:2203.15556 (2022) on PithPith's reviewarXiv
  18. [18] L. Ziyin and M. Ueda, Zeroth, first, and second-order phase transitions in deep neural networks, Phys. Rev. Res.5, 043243 (2023) on Pith
  19. [19] N. Rubin, I. Seroussi and Z. Ringel, Grokking as a first order phase transition in two layer networks, ICLR (2024) on Pith
  20. [20] H. Cui, F. Behrens, F. Krzakala and L. Zdeborov´ a, A phase transition between positional and semantic learn- ing in a solvable model of dot-product attention, NeurIPS (2024) on Pith
show 24 more
  1. [21] L. Arola-Fern´ andez and L. Lacasa, Effective theory of col- lective deep learning, Phys. Rev. Res.6, L042040 (2024) on Pith
  2. [22] L. Arola-Fern´ andez, Intuition emerges in Maximum Cal- iber models at criticality, arXiv:2508.06477 (2025) on PitharXiv
  3. [23] D. Krotov and J. J. Hopfield, Dense associative memory for pattern recognition, NeurIPS (2016) on Pith
  4. [24] H. Ramsaueret al., Hopfield networks is all you need, ICLR (2021) on Pith
  5. [25] A. Vivet and A. Arenas, Emergent self-attention from astrocyte-gated associative memory dynamics, Phys. Rev. Res.8, 023276 (2026) on Pith
  6. [26] Y. Kuramoto,Chemical Oscillations, Waves, and Turbu- lence(Springer, 1984) on Pith
  7. [27] S. H. Strogatz, From Kuramoto to Crawford, Physica D 143, 1 (2000) on Pith
  8. [28] A. Arenaset al., Synchronization in complex networks, Phys. Rep.469, 93 (2008) on Pith
  9. [29] M. Romeraet al., Vowel recognition with four coupled spin-torque nano-oscillators, Nature563, 230 (2018) on Pith
  10. [30] A. Todri-Sanialet al., Computing with oscillators from theoretical underpinnings to applications and demonstra- tors, npj Unconv. Comput.1, 14 (2024) on Pith
  11. [31] T. Miyatoet al., Artificial Kuramoto oscillatory neurons, ICLR (2025) on Pith
  12. [32] J. Nunley, Kuramoto attention: synchronizing self- attention on the torus, arXiv:2606.11585 (2026); At- tention as frustrated synchronization, arXiv:2606.18694 (2026) on PithPith's reviewarXiv
  13. [33] J. Liuet al., Krause synchronization transformers, ICML (2026), arXiv:2602.11534 on PithPith's reviewarXiv
  14. [34] B. Geshkovskiet al., A mathematical perspective on transformers, Bull. Am. Math. Soc.62, 427 (2025) on Pith
  15. [35] H. Hays, Selective synchronization attention, arXiv:2602.14445 (2026) on PitharXiv
  16. [36] F. Pasqualetti and T. Guo, Attention by synchronization in coupled oscillator networks, Trans. Mach. Learn. Res. (2026) on Pith
  17. [37] M. Born and E. Wolf,Principles of Optics, 7th ed. (Cam- bridge University Press, Cambridge, 1999) on Pith
  18. [38] M. A. Lohe, Non-abelian Kuramoto models and synchro- nization, J. Phys. A42, 395101 (2009) on Pith
  19. [39] J. H. Scofield, Frequency-domain description of a lock-in amplifier, Am. J. Phys.62, 129 (1994) on Pith
  20. [40] See the Supplemental Material below for the derivations, the comparison with other attention layers, the imple- mentation, the methods and further checks on Pith
  21. [41] R. Adler, A study of locking phenomena in oscillators, Proc. IRE34, 351 (1946) on Pith
  22. [43] S. Aroraet al., Zoology: measuring and improving recall in efficient language models, ICLR (2024) on Pith
  23. [44] C. Olssonet al., In-context learning and induction heads, Transformer Circuits Thread (2022). 7 Supplemental Material for “Everything in its right place: attention is a radio head” Llu ´ ıs Arola-Fern´ andez, Aissam Khadraoui and Alex Arenas This document collects what the let... on Pith
  24. [512] Induction [S14]: the token that followed the previous occurrence of the query, trained at length 32 and tested up to 1024. synthetic tasks language (TinyStories) layers, heads 2, 1 2, 1 to 8 dmodel, feed-forward width 64, 128 64 to 1024, 2d model dropout 0 0.1 vocabulary 51 (r... on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment