Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

Burc Gokden

REVIEW 1 major objections 4 minor 51 references

Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

T0 review · 1 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Attention built on a learned, input-generated metric operator contains standard attention exactly, and at inference collapses to a cached tensor.

desk verdict A carefully scoped theory-plus-audit paper: the formal results are clean and partly Lean-checked, the collapse is honestly measured on one checkpoint, and the real gap is between the title's generality and a narrow empirical base. read the letter →

arxiv 2608.10288 v1 pith:MGV5DG23 submitted 2026-08-10 cs.LG cs.CL

Burc Gokden This is my paper · ORCID

classification cs.LGcs.CL MSC 68T0715B4805C5082C27

keywords powerlawgraphattentionscaleddot-productlearnedbilinearoperatorinputinvarianceinferencecollapserotarypositionembeddingscommutantself-organizedcriticality

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper gives a full analytical account of Power Law Graph Attention (PLGA), the attention mechanism inside the PLDR-LLM language model family, in which the fixed bilinear form of scaled dot-product attention (SDPA) is replaced by a learned operator generated from the input itself. The central theoretical claim is that SDPA is exactly the identity-operator point of this larger family, and that if a trained model's learned operators are exactly input-invariant, then the entire deep nonlinear subnetwork that produces them can be deleted at inference and replaced by a single cached constant tensor with no change to the model's outputs. The paper further reports that on a released checkpoint this invariance is measured at floating-point resolution, so the cached model is empirically indistinguishable from the full recomputed model on the tested workloads. The broader ambition is a family of models in which the attention geometry itself is learned, regularized, measured, and cached, rather than fixed a priori.

What carries the argument

The central object is the learned bilinear score operator G_LM, a d_k-by-d_k matrix per head produced through a chain: the rotated query Gram D_Q feeds a deep row-wise residual network (the metric learner), whose output generator A is transformed by an elementwise power law with learned exponents and a learned affine superposition into G_LM; attention scores are then q^T G_LM k / $\sqrt$(d_k). A second load-bearing object is the shared row map $\varphi$ of the metric learner: because phi acts identically on every row and every head of a layer, its collapse to a locally constant map forces the generator A to be rank-one and identical across heads, which is exactly the condition measured on the released checkpoint. The proof of collapse rests on the exact input-invariance hypothesis, with propagation of small perturbation errors bounded by explicit constants.

What would settle it

Run the same Appendix D audit on a released near-critical checkpoint at a different scale (e.g. the 41B-token 110M model with a different prompt distribution, or any larger released variant): if the recomputed G_LM differs from the cached value by more than the decoding margin on any step, or if the composite Jacobian of the row map phi exceeds 1 at any visited row, then the collapse is an artifact of the specific audited checkpoint rather than a trained property of the family.

Watch

Extended reading notes

Core claim

The central discovery is an inference-collapse theorem: if every deductive output of PLGA is exactly input-invariant, taking values G*(l,i), then the inference map of the full architecture coincides with the map of a generalized SDPA in which the learned subnetwork (3.3)-(3.6) is replaced by the constant operators G*(l,i). At the identity operator G = I this recovers ordinary scaled dot-product attention exactly. Empirically, on the released near-critical checkpoint, the recomputed G_LM equals the cached value bitwise at every compared decoding step, the greedy token choices agree at every step, and the logit deviations lie roughly two orders of magnitude below the smallest realized decision margin. The same architecture at a different checkpoint shows substantial historical-row movement (74.5% of compared entries), so the collapse is a trained property, not an architectural identity.

Load-bearing premise

The load-bearing empirical premise is that the invariance measured in Appendix D on one released 110M checkpoint with eight fixed prompts is representative of PLDR-LLM inference more broadly; the causal mechanism additionally assumes, as an unproved hypothesis, that the trained row map $\varphi$ is contractive on the visited tube, so if either the measured invariance fails to generalize or the contraction hypothesis is false on a different checkpoint, the title-level claim of empirical collapse loses its support.

Editorial extensions

If this is right

If the invariance theorem holds for a trained model, the entire PLGA subnetwork can be excised at inference, replaced by cached operators G*(l,i), without changing any output; this is what makes the G-cache exact relative to the prompt-frozen KV-cache, and empirically bitwise faithful to full recomputation on the audited checkpoint.
SDPA is not a rival architecture but the identity-operator point of the PLGA family, so any comparative statement between the two is a statement about a family and its base point, not about separate designs.
Under nonresonance of the standard rotary frequencies, a constant operator G* leaves scores dependent only on relative position exactly when G* lies in the rotation commutant; for a generic learned G* the trained operator carries absolute-position information, with d_k^2 - d_k off-commutant dimensions per head beyond what a pre-RoPE-linearly-reparameterized SDPA head can express.
The blockwise training objective used by the released pipelines contains target-exposed auxiliary summands for historical rows; the paper shows they are not next-token log probabilities as a protocol unless historical-row prefix consistency holds, which the global Gram does not supply in general and which is measured to fail on a released checkpoint.
The three-stage mechanism (rotary twirl, concentration, row-map contraction) provides a quantitative, ingredient-tagged explanation of how input invariance can arise during training, with each stage's measurable diagnostics computed directly on the released checkpoint rather than assumed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A plausible testable extension: on checkpoints at intermediate training steps, the measured historical-row movement and the order parameter should decay together with the composite Jacobian of the metric learner's row map, which would locate the collapse along the training trajectory rather than only at the end state.
The commutant residual measured in Appendix D suggests a causal experiment not performed in the paper: project a trained G* onto the RoPE commutant and measure the behavioral change in decoding; if the off-commutant energy is largely unused, the measured absolute-position sensitivity is occupancy without function, which would refine the significance of Corollary 7.1.
If invariance is a trained attractor property rather than an architectural identity, then the same collapse phenomenon, or its absence, should be reproducible in non-SOC training regimes at larger scale, which would determine whether the criticality framework of Section 6 is essential to the effect or merely correlated with it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague —

Read the PLGA paper. The formal work is genuine, the audit is a model of how to report checkpoint-level measurements, and the paper's own claims are more carefully scoped than its title. The reader's CONDITIONAL verdict is right; the stress-test worry about narrow coverage is fair as far as it goes, but it doesn't break the paper on its own terms, because the invariance is explicitly a measured hypothesis, not a theorem.

What's actually new: the commutant characterization of relative-position invariance under RoPE (Prop. 4.1) is clean, with a correctly stated nonresonance hypothesis. The inference-collapse theorem (Thm. 5.4(ii)) is honestly sufficiency-only, and the SDPA-at-G=I containment is exactly what it claims. The epsilon-LayerNorm analysis (Lem. 5.11) is careful, and measuring the scale error directly rather than trusting the proxy is a nice touch. Selected proof cores are machine-checked in Lean with an axiom audit, and the Appendix C coverage table is unusually explicit about what is and isn't kernel-checked.

Appendix D earns real credit. Singular-value spectra instead of float-zero determinants, trained-versus-initialized controls, bitwise cached-versus-recomputed comparison on a pinned checkpoint, and — most refreshingly — an explicit statement that its own perturbation budget fails by ~45 orders of magnitude to certify decoding. That is honest quantification of a mechanism, not a disguised certificate.

Soft spots, in proportion: the empirical base is one checkpoint, eight prompts, four of them for greedy fidelity with heavily repetitive continuations (duplicate 4-gram fractions 0.49–0.80). The contrast checkpoint showing 74.5% historical-row movement confirms collapse is a property of one training run, not the architecture. The Stage-3 contraction mechanism is sampled, not certified — but it's not load-bearing; the measured collapse stands even if the proposed origin is wrong. The SOC phase labeling is acknowledged circular, which mitigates but doesn't erase the title's framing debt.

Who gets value: attention-theory people for Prop. 4.1 and Thm. 5.4; anyone running model audits can learn from Appendix D; the epistemic-labeling scheme is worth stealing.

Recommendation: send it out. The right referee will push on generality of the empirical premise and on what's consolidation versus new, but the formal core is sound and the empirical reporting is a cut above the usual standard.

Referee Report

1 major / 4 minor

Summary. The paper presents an analytical treatment of Power Law Graph Attention (PLGA) and the PLDR-LLM architecture. It proves several structural properties: PLGA contains scaled dot-product attention exactly at the identity operator (Theorem 5.4(i)); the generator A is strictly entrywise positive with Perron-Frobenius structure (Theorem 3.8); a rank-one singularity condition holds exactly (Proposition 3.9); the DAG regularizer has the NOTEARS walk-counting form with an explicit positivity obstruction (Theorem 4.4, Remark 4.5); and a commutant characterization for rotary embeddings identifies which operators preserve relative-position dependence (Proposition 4.1). The central result is Theorem 5.4(ii): if every deductive output is exactly input-invariant, then the inference map coincides with a generalized SDPA with constant learned bilinear forms. The empirical companion is a numerical audit on a released 110M checkpoint (Appendix D), reporting bitwise invariance of the operator under greedy decoding and agreement between cached and recomputed inference, alongside a contrast checkpoint showing historical-row movement. The paper also proposes a conditional three-stage mechanism for the invariance (rotary twirl, concentration, row-map contraction) and frames self-organized criticality as a phenomenological hypothesis. Claims are explicitly labeled as theorems, conditional theorems, measurements, or conjectures, with selected proof cores machine-checked in Lean 4.

Significance. The paper is exceptionally careful in labeling the epistemic status of each claim, and it provides an unusually detailed reproducibility package: pinned checkpoints and commits, Lean 4 formalization of elementary proof cores, and a numerical audit with raw arrays and hashes. The exact containment of SDPA in the PLGA family and the conditional inference-collapse theorem are clean conceptual contributions that would be of interest to the community working on attention variants and efficient inference. The empirical audit is honest and self-critical, explicitly reporting where bounds fail to close margins and which mechanisms are not certified. The main limitation is the narrowness of the empirical basis for the title-level 'empirical collapse at inference' claim: a single checkpoint, a small number of prompts, and greedy decoding only. If the title claim is narrowed accordingly, the paper's contribution is solid and publishable.

major comments (1)

[Title/Abstract; Appendix D.7, D.9, D.10] The title's unqualified claim 'empirical collapse at inference' is stronger than the evidence reported in the manuscript. Exact invariance is measured on a single released 110M checkpoint (PLDR-LLM-v51-SOC-110M-5) under greedy decoding of 48 tokens on four prompts, with 960 per-layer bitwise comparisons of the recomputed versus cached GLM (Appendix D.7). The stochastic-continuation order-parameter measurements in Appendix D.9 do not compare cached versus recomputed sampled decisions. Appendix D.10 shows that a contrast checkpoint of the same architecture (PLDR-LLM-v51-SOC-110M-1) exhibits 74.5% historical-row movement, demonstrating that collapse is not an architectural identity. Because the measured invariance is the empirical premise that activates Theorem 5.4(ii), this narrow tube is a load-bearing limitation of the central advertised claim. I recommend either explicitly restricting the title and abstract to 'empirical collapse at inference on the released checkpoint under greedy decoding' or substantially widening the audit to include more checkpoints, longer and more diverse prompts, and cached-versus-recomputed comparisons under stochastic sampling.

minor comments (4)

[Section 4.5, Remark 4.9] The symbol Ψ is used in equation (4.15) and in the surrounding text ('the deductive chain Ψ') without being defined; please define it explicitly, ideally before its first use in Remark 4.9.
[Section 1, Related Work] The genealogical paragraph comparing PLGA to PaTH attention is extremely long and difficult to parse; a structured summary or table would improve readability without losing the careful distinctions being drawn.
[Appendix D.7 and Corollary 5.13] The audit reports the margin check as '2B = 7.2e-5 < 6.0e-3', but Corollary 5.13 gives both a 2B criterion and a √2B criterion depending on whether the bound is in ℓ∞ or ℓ2; please state explicitly which norm the measured B refers to and which criterion is being applied.
[Abstract] The phrase 'the assembled proxy misses the decoding margin' is cryptic without reading Section 5.4 and Appendix D.7; adding a brief explanatory clause (for example, that the worst-case bound is far too large to certify bit-identical decoding, while measured deviations are far below the margin) would make the abstract more accessible.

Circularity Check

1 steps flagged · score 4.0 of 10

SOC phase labels are defined through the order parameter they are then used to explain; the collapse theorem and measured invariance are independent.

self definitional [Definition 6.5 and Section 6.3, after Definition 6.5]
"near-critical if its order parameter (5.1) satisfies m(θ)≈ 0 (empirically ≲ 10−2, typically ≤10−5) while text generation is non-degenerate; sub-critical if m(θ)=O(1) or larger... The near/sub-critical labels are substantially defined through the order parameter and generation quality they are then used to explain, so an independent phase criterion fixed in advance is needed to break the circularity."

The phase labels 'near-critical' and 'sub-critical' are constructed in Definition 6.5 from thresholds of the order parameter m(θ), which is itself the measured fluctuation of the deductive outputs. The paper then uses 'proximity of m(θ) to zero' as evidence that the phases are separated and as part of the criticality-based explanation of why the deductive outputs become invariant (Hypothesis 5.14(3)). Thus the classification into the phases being explained is fixed by the same quantity that is presented as the explanatory order parameter; the separation is true by labeling convention rather than by independent empirical discovery. The paper explicitly concedes this loop ('independent phase criterion fixed in advance is needed to break the circularity').

full rationale

The main derivation chain is not circular. Theorem 5.4(i) is an algebraic substitution at GLM = I and is machine-checked. Theorem 5.4(ii) is an explicitly conditional statement: the hypothesis is exact input-invariance of the deductive outputs, and the conclusion is the definitional replacement of the subnetwork by its constant values; the theorem does not fabricate the invariance. The empirical collapse claim is grounded in Appendix D's independent re-measurement on a pinned public checkpoint, including bitwise equality of recomputed versus cached GLM at every compared step and agreement with decoding margins; the SDPA-containment and commutant results are proved in place or against classical literature, with the Lean formalization providing independent support. The one genuine circular step is the SOC phase vocabulary of Definition 6.5/Section 6.3: near-critical and sub-critical are defined by thresholds of m(θ), and that same m(θ) is then cited as the order parameter separating the phases and as evidence for the criticality-based selection mechanism. The paper itself flags this as a labeling circularity requiring an independent phase criterion. Because this loop sits in the phenomenological/interpretive layer and is not load-bearing for the title-level collapse theorem or the measured invariance, the score is 4 rather than higher.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The central theorems are elementary and mostly self-contained. The main unproved inputs are the empirically measured contraction of the trained row map and the representativeness of the audited checkpoint. No new physical entity is introduced; the architecture's learned tensors are trained parameters, not ad hoc free parameters. The paper's SOC phase labeling contains acknowledged circularity, but the collapse theorem does not depend on it.

free parameters (2)

Learned metric-learner and power-law tensors (Phi_res, W, b_W, P, a, b_a) = Trained on 8B-41B tokens; values in released checkpoint, not tabulated in paper
These are the architecture's learned parameters, not ad hoc free parameters introduced for a derivation. The central theorems are conditional on exact invariance of the trained tensors; the measured values support the empirical claim. No fitted constant is used to manufacture a prediction.
Training control parameters (eta_max, T_w) = Not specified in the excerpt; schedules in cited prior work
These define near-critical versus sub-critical phases empirically. They are externally selected control parameters, not fitted to make the collapse theorem work, but they are load-bearing for the claimed phase separation.

assumptions (7)

standard math Perron-Frobenius theorem for positive matrices
Used in Theorem 3.8 to assert a simple positive Perron root of entrywise positive A_LM.
standard math Measurable multiplicative Cauchy equation has only power-law solutions
Used in Prop. 6.1 to derive scale covariance forces power laws; cited to Aczel.
standard math Matrix Bernstein inequality
Used in Prop. 5.10 for concentration of the normalized density operator; the paper states the idealization explicitly.
domain assumption Nonresonance of RoPE frequencies theta_j = Theta^{-2(j-1)/dk}
Proposition 4.1 requires distinct non-real eigenvalues; the paper proves the standard Theta=1e4 frequencies satisfy it.
domain assumption Independence and common second moment of query vectors (stationarity idealization)
Proposition 5.10 is conditional on this idealization; Appendix D measures that it is far from holding, with first-half and second-half aggregates differing by roughly 33% median. This weakens Stage 2 as an explanation.
ad hoc to paper Contraction hypothesis for the trained row map phi on the visited tube
Proposition 5.12's Stage 3 is conditional on the trained row map being contractive; Appendix D measures composite Jacobians, not a tube-uniform certificate. This is a load-bearing premise for the mechanism section, explicitly labeled a hypothesis.
domain assumption Ideal masked softmax semantics and floating-point underflow of masked entries
Proposition 3.12 and the causal mask statements are proved for the ideal masked softmax; transfer to implementations relies on underflow, quantified in Remark 3.13.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference." pith.science (2026). https://pith.science/paper/MGV5DG23

@misc{pith2026260810288,
  author       = {Pith},
  title        = {Pith review of: Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGV5DG23}},
  note         = {Machine review of arXiv:2608.10288}
}

abstract

The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator $G_{LM}$, built from a positive tensor $A_{LM}$ by elementwise power laws. The architecture is fully specified, verified against pinned reference releases; claims are labeled theorem, conditional theorem, measurement, or conjecture. Unconditionally: PLGA contains SDPA exactly at $G_{LM}=I$; $A_{LM}$ and $A_P$ are strictly entrywise positive, with Perron-Frobenius structure on $A_{LM}$; the DAG regularizer has the NOTEARS walk-counting form and positivity obstructs exact acyclicity; and, under nonresonance (satisfied by standard rotary frequencies), a commutant criterion identifies which operators preserve relative-position dependence. An inference-collapse theorem: exact input invariance of deductive outputs collapses inference to generalized SDPA with a constant operator. Measured invariance: relative fluctuations of $10^{-6}$ and below; perturbation bounds quantify but do not certify cached inference; the assembled proxy misses the decoding margin. A conditional three-stage mechanism (rotary twirl, concentration, row-map contraction) is measured on a released checkpoint. Blockwise training and scoring under the global Gram are stated with explicit target exposure; on tested samples, block and sequential scoring select identical answers and agree on the published TruthfulQA probability-mass metric within $5\times 10^{-5}$ per item. Self-organized criticality enters as a phenomenological framework with an intrinsic order parameter; open claims become falsifiable conjectures. Selected proof cores are machine-checked in Lean 4.

Figures

Figures reproduced from arXiv: 2608.10288 by the authors.

**Figure 1.** Information flow in one PLGA head. Solid path: row-local operations (row i uses rows ≤ i through the masked softmax). Dashed path: the global density operator; the sequence axis is contracted before any mask acts, so the learned operator GLM depends on every supplied row. Under the online contract (4.8) all supplied rows are known context; historical rows of a longer call are recomputed under the enlarged context (R… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 35 canonical work pages

[1]

János Aczél,Lectures on functional equations and their applications, Academic Press, New York, 1966

work page 1966
[2]

Hinton,Layer normalization, 2016, arXiv:1607.06450

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton,Layer normalization, 2016, arXiv:1607.06450

arXiv 2016
[3]

1, 364–374

Per Bak, Chao Tang, and Kurt Wiesenfeld,Self-organized criticality, Physical Review A38(1988), no. 1, 364–374

work page 1988
[4]

Beggs and Dietmar Plenz,Neuronal avalanches in neocortical circuits, Journal of Neuroscience23 (2003), no

John M. Beggs and Dietmar Plenz,Neuronal avalanches in neocortical circuits, Journal of Neuroscience23 (2003), no. 35, 11167–11177

work page 2003
[5]

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier,Language modeling with gated convolutional networks, Proceedings of the 34th International Conference on Machine Learning (ICML), 2017, pp. 933–941

work page 2017
[6]

Muñoz, Alessandro Vespignani, and Stefano Zapperi,Paths to self-organized criticality, Brazilian Journal of Physics30(2000), no

Ronald Dickman, Miguel A. Muñoz, Alessandro Vespignani, and Stefano Zapperi,Paths to self-organized criticality, Brazilian Journal of Physics30(2000), no. 1, 27–41

work page 2000
[7]

2793–2803

Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas,Attention is not all you need: pure attention loses rank doubly exponentially with depth, Proceedings of the 38th International Conference on Machine Learning (ICML), 2021, pp. 2793–2803

work page 2021
[8]

Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou,XCiT: cross-covariance image transformers, Advances in Neural Information Processing Systems 34, 2021, arXiv:2106.09681

arXiv 2021

Show all 51 references

[9]

Bolin Gao and Lacra Pavel,On the properties of the softmax function with application in game theory and reinforcement learning, 2017, arXiv:1704.00805

2017 arXiv
[10]

Burc Gokden,CoulGAT: An experiment on interpretability of graph attention networks, 2019, arXiv:1912.08409

2019 arXiv
[11]

,Power law graph transformer for machine translation and representation learning, 2021, arXiv:2107.02039

2021 arXiv
[12]

,PLDR-LLM: Large language model from power law decoder representations, 2024, arXiv:2410.16703

2024 arXiv
[13]

,PLDR-LLMs learn a generalizable tensor operator that can replace its own deep neural net at inference, 2025, arXiv:2502.13502

2025 arXiv
[14]

,PLDR-LLMs reason at self-organized criticality, 2026, arXiv:2603.23539

2026
[15]

Nigel Goldenfeld,Lectures on phase transitions and the renormalization group, Addison-Wesley, 1992

1992
[16]

Le,Hypernetworks, 2016, arXiv:1609.09106

David Ha, Andrew Dai, and Quoc V. Le,Hypernetworks, 2016, arXiv:1609.09106

2016 arXiv
[17]

Janina Hesse and Thilo Gross,Self-organized criticality as a fundamental property of neural systems, Frontiers in Systems Neuroscience8(2014), 166

2014
[18]

Horn and Charles R

Roger A. Horn and Charles R. Johnson,Matrix analysis, 2nd ed., Cambridge University Press, 2013

2013
[19]

Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber,Going beyond linear transform- ers with recurrent fast weight programmers, Advances in Neural Information Processing Systems 34, 2021, arXiv:2106.06295

2021 arXiv
[20]

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret,Transformers are RNNs: fast autoregressive transformers with linear attention, Proceedings of the 37th International Conference on Machine Learning, 2020, arXiv:2006.16236

2020 arXiv
[21]

Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang,Bilinear attention networks, Advances in Neural Information Processing Systems 31, 2018, arXiv:1805.07932

2018 arXiv
[22]

Dong Le, Thong Nguyen, Cong-Duy Nguyen, and Anh Tuan Luu,Don’t read everything: a curvature-conditioned query for linear attention, 2026, arXiv:2606.01294

2026 arXiv
[23]

Stephanie Lin, Jacob Hilton, and Owain Evans,TruthfulQA: Measuring how models mimic human falsehoods, 2021, arXiv:2109.07958; published at ACL 2022

2021 arXiv
[24]

2, 41–74

Dimitrije Marković and Claudius Gros,Power laws and self-organized criticality in theory and nature, Physics Reports536(2014), no. 2, 41–74

2014
[25]

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher,Pointer sentinel mixture models, 2016, arXiv:1609.07843; the WikiText corpora

2016 arXiv
[26]

Misner, Kip S

Charles W. Misner, Kip S. Thorne, and John A. Wheeler,Gravitation, W. H. Freeman, San Francisco, 1973, Ch. 21.12

1973
[27]

Mark E. J. Newman,Power laws, Pareto distributions and Zipf’s law, Contemporary Physics46(2005), no. 5, 323–351

2005
[28]

Moseley, Akshay Chaudhari, and Curtis Langlotz, LieRE: Lie rotational positional encodings, 2024, arXiv:2406.10322

Sophie Ostmeier, Brian Axelrod, Maya Varma, Michael E. Moseley, Akshay Chaudhari, and Curtis Langlotz, LieRE: Lie rotational positional encodings, 2024, arXiv:2406.10322

2024 arXiv
[29]

Bosheng Qin, Juncheng Li, Siliang Tang, and Yueting Zhuang,DBA: Efficient transformer with dynamic bilinear low-rank attention, 2022, arXiv:2211.16368

2022 arXiv
[30]

Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré,Sinkformers: Transformers with doubly stochastic attention, 2021, arXiv:2110.11773

Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré,Sinkformers: Transformers with doubly stochastic attention, 2021, arXiv:2110.11773. POWER LA W GRAPH ATTENTION 61

2021 arXiv
[31]

Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber,Linear transformers are secretly fast weight programmers, Proceedings of the 38th International Conference on Machine Learning, 2021, arXiv:2102.11174

2021 arXiv
[32]

Simon Schug, Seijin Kobayashi, Yassir Akram, João Sacramento, and Razvan Pascanu,Attention as a hypernetwork, International Conference on Learning Representations (ICLR), 2025, arXiv:2406.05816

2025 arXiv
[33]

Eugene Seneta,Non-negative matrices and Markov chains, 2nd ed., Springer, 2006

2006
[34]

Noam Shazeer,GLU variants improve transformer, 2020, arXiv:2002.05202

2020 arXiv
[35]

Noam Shazeer, Zhenzhong Lan, Youlong Cheng, Nan Ding, and Le Hou,Talking-heads attention, 2020, arXiv:2003.02436

2020 arXiv
[36]

Didier Sornette and Guy Ouillon,Dragon-kings: Mechanisms, statistical methods and empirical evidence, European Physical Journal Special Topics205(2012), 1–26

2012
[37]

Eugene Stanley,Scaling, universality, and renormalization: Three pillars of modern critical phenomena, Reviews of Modern Physics71(1999), no

H. Eugene Stanley,Scaling, universality, and renormalization: Three pillars of modern critical phenomena, Reviews of Modern Physics71(1999), no. 2, S358–S366

1999
[38]

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu,RoFormer: Enhanced transformer with rotary position embedding, 2021, arXiv:2104.09864

2021 arXiv
[39]

Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng,Synthesizer: Rethinking self-attention in transformer models, 2020, arXiv:2005.00743

2020 arXiv
[40]

Nguyen,Functional equivalence in attention: a comprehensive study with applications to linear mode connectivity, 2026, arXiv:2606.17830

Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh, Tan Lai Ngoc, and Tan M. Nguyen,Functional equivalence in attention: a comprehensive study with applications to linear mode connectivity, 2026, arXiv:2606.17830

2026 arXiv
[41]

Tropp,An introduction to matrix concentration inequalities, Foundations and Trends in Machine Learning8(2015), no

Joel A. Tropp,An introduction to matrix concentration inequalities, Foundations and Trends in Machine Learning8(2015), no. 1–2, 1–230

2015
[42]

Ramaswamy,Faster query-key learning sharpens attention in self-attention models, Proceedings of the International Conference on Machine Learning (ICML), 2026, arXiv:2608.06776

Rahul Vashisht and Harish G. Ramaswamy,Faster query-key learning sharpens attention in self-attention models, Proceedings of the International Conference on Machine Learning (ICML), 2026, arXiv:2608.06776

2026 arXiv
[43]

Gomez, Łukasz Kaiser, and Illia Polosukhin,Attention is all you need, Advances in Neural Information Processing Systems 30 (NIPS), 2017, pp

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin,Attention is all you need, Advances in Neural Information Processing Systems 30 (NIPS), 2017, pp. 6000–6010

2017
[44]

Hong Wang and Kelly Wang,Complete characterization of gauge symmetries in transformer architectures, NeurIPS Workshop on Symmetry and Geometry in Neural Representations (NeurReps), 2025,https:// openreview.net/forum?id=KrkbYbK0cH

2025
[45]

Wilson and John Kogut,The renormalization group and theϵ expansion, Physics Reports12 (1974), no

Kenneth G. Wilson and John Kogut,The renormalization group and theϵ expansion, Physics Reports12 (1974), no. 2, 75–199

1974
[46]

Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda, and Yoon Kim,PaTH attention: position encoding via accumulating Householder transformations, Advances in Neural Information Processing Systems (NeurIPS), 2025, arXiv:2505.16381

2025
[47]

Yehan Yang, Junyuan Shang, Yang Li, Guanqun Zhao, Shuohuan Wang, and Dianhai Yu,Autonomy-of-heads: data-free sparse attention from frozen query-key geometry, 2026, arXiv:2608.06849

2026 arXiv
[48]

Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guanghao Li, Shuting Dong, Huaisong Zhang, and Chun Yuan,ComRoPE: Scalable and robust rotary position embedding parameterized by trainable commuting angle matrices, 2025, arXiv:2506.03737

2025 arXiv
[49]

I, Proceedings of the Physico-Mathematical Society of Japan17(1935), 48–57

Hideki Yukawa,On the interaction of elementary particles. I, Proceedings of the Physico-Mathematical Society of Japan17(1935), 48–57

1935
[50]

Xing,DAGs with NO TEARS: Continuous optimization for structure learning, Advances in Neural Information Processing Systems 31 (NeurIPS), 2018, pp

Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing,DAGs with NO TEARS: Continuous optimization for structure learning, Advances in Neural Information Processing Systems 31 (NeurIPS), 2018, pp. 9492–9503

2018
[51]

Zipf,Human behavior and the principle of least effort, Addison-Wesley, 1949

George K. Zipf,Human behavior and the principle of least effort, Addison-Wesley, 1949

1949

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

János Aczél,Lectures on functional equations and their applications, Academic Press, New York, 1966

work page 1966

[2] [2]

Hinton,Layer normalization, 2016, arXiv:1607.06450

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton,Layer normalization, 2016, arXiv:1607.06450

arXiv 2016

[3] [3]

1, 364–374

Per Bak, Chao Tang, and Kurt Wiesenfeld,Self-organized criticality, Physical Review A38(1988), no. 1, 364–374

work page 1988

[4] [4]

Beggs and Dietmar Plenz,Neuronal avalanches in neocortical circuits, Journal of Neuroscience23 (2003), no

John M. Beggs and Dietmar Plenz,Neuronal avalanches in neocortical circuits, Journal of Neuroscience23 (2003), no. 35, 11167–11177

work page 2003

[5] [5]

Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier,Language modeling with gated convolutional networks, Proceedings of the 34th International Conference on Machine Learning (ICML), 2017, pp. 933–941

work page 2017

[6] [6]

Muñoz, Alessandro Vespignani, and Stefano Zapperi,Paths to self-organized criticality, Brazilian Journal of Physics30(2000), no

Ronald Dickman, Miguel A. Muñoz, Alessandro Vespignani, and Stefano Zapperi,Paths to self-organized criticality, Brazilian Journal of Physics30(2000), no. 1, 27–41

work page 2000

[7] [7]

2793–2803

Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas,Attention is not all you need: pure attention loses rank doubly exponentially with depth, Proceedings of the 38th International Conference on Machine Learning (ICML), 2021, pp. 2793–2803

work page 2021

[8] [8]

Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou,XCiT: cross-covariance image transformers, Advances in Neural Information Processing Systems 34, 2021, arXiv:2106.09681

arXiv 2021