Transformer models are gauge invariant: A mathematical connection between AI and particle physics

Leo van Nierop

REVIEW 4 major objections 5 minor 6 references

Transformer models are gauge invariant: A mathematical connection between AI and particle physics

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper demonstrates that transformer stacks possess an exact gauge symmetry: continuous families of weight matrices produce identical outputs, so a redundant set of parameters can be removed without changing the model's function.

desk verdict A real observation about attention symmetry, overgeneralized to default transformers via a derivation that silently drops LayerNorm scale and shift. read the letter →

arxiv 2412.14543 v1 pith:OWPD4ST2 submitted 2024-12-19 cs.LG hep-th

Leo van Nierop This is my paper · ORCID

classification cs.LGhep-th MSC 68T0781T13

keywords gaugeinvariancetransformerarchitecturelayernormalizationattentionmechanismparameterredundancylosslandscapeflatdirectionsfixing

topics Yang-Mills Mass Gap

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the transformer architecture has a symmetry of the same kind as gauge invariance in particle physics: for every transformer stack there is a continuous family of weight matrices and biases that all compute exactly the same input–output function. If true, every trained transformer contains exact flat directions in its loss landscape — directions of parameter change that never alter predictions — and a gauge choice can remove the corresponding parameters without touching representational power. The symmetry has two parts: a rotation symmetry of the embedding space left over after layer normalization, and a per-head rotation or rescaling of the attention keys and queries. The paper counts the redundant parameters, gives concrete reductions for GPT-2 and LLaMA, and shows how the ordinary transformer arises as a gauge-fixed, discretized version of a gauge field theory. A sympathetic reader would care because precise parameter removal and the transfer of gauge-theory tools, such as topological constraints, would cut training and inference cost directly.

What carries the argument

The load-bearing object is the embedding normalization constraint: layer normalization maps token embeddings onto $S^{d_e-2}$, the unit sphere inside the hyperplane perpendicular to the all-ones vector, which makes $\mathrm{SO}(d_e-1)$, the subgroup of rotations fixing the all-ones vector, the natural symmetry of the embedding space. The attention mechanism supplies the second mechanism: because attention weights depend on the dot product $Q^T K$, a simultaneous $\mathrm{GL}(d_h)$ transformation on keys with its inverse transpose on queries leaves the attention matrix unchanged. The derivation chains these constraints through the value matrices, linear layers, feed-forward network, and skip connections to obtain the transformation rules and the redundancy count, with the skip connections being the obstacle that initially ties the embedding rotation across all layers.

What would settle it

Take a trained transformer and a fixed token string, compute the output logits, apply a nontrivial gauge transformation (for example a rotation $g$ acting on the first normalized embeddings and the corresponding inverse rotations on keys, queries, values, linear layers, and the final embedding readout according to the paper's rules), and compare the logits again; any difference beyond machine precision would falsify the claimed exact invariance. A cheaper check is to vary only the gauge parameters and compare the rank of the output Jacobian against the predicted $2 n_t n_h d_h^2 + \tfrac12(d_e-1)(d_e-2)$ flat directions.

Watch

Extended reading notes

Core claim

The paper's central claim is that a stack of transformers is invariant under a specific continuous group of transformations, so that a whole family of weight settings describes the same model function. The embedding-space symmetry is $\mathrm{SO}(d_e-1)$: after layer normalization, embeddings lie on the unit sphere in the hyperplane perpendicular to the all-ones vector, i.e. on $S^{d_e-2}$, and any rotation preserving the all-ones vector leaves the normalized embeddings equivalent. Accompanying it is a per-head symmetry under $\mathrm{GL}(d_h)$, where keys and queries are multiplied by inverse transposes of the same head rotation so that the attention matrix is unchanged. The derivation fixes the transformation of every weight matrix — value matrices, the linear output layer, the feed-forward network, and skip connections — and finds that skip connections force the same embedding rotation to be used everywhere unless extra gauge rotations are inserted into the skip connections, which makes the standard architecture a gauge-fixed discretized $\mathrm{SO}(d_e-1)$ gauge theory. The total number of redundant dimensions is $2 n_t n_h d_h^2 + \tfrac12(d_e-1)(d_e-2)$, and the paper presents this as the default transformer having partially but not fully removed the gauge freedom.

Load-bearing premise

The proof assumes that a layer-normalized embedding transforms under the gauge rotation exactly as a plain vector would, which requires the normalization's learned per-dimension scale and shift to rotate along with it; in a standard transformer those per-dimension parameters are not rotated, so the claimed exact invariance may fail precisely at that step.

Editorial extensions

If this is right

Every trained transformer, regardless of data or initialization, has exact flat valleys in its loss landscape of dimension $2 n_t n_h d_h^2 + \tfrac12(d_e-1)(d_e-2)$ and no change in model output along them.
Choosing a gauge, for example setting redundant rotations to the identity, removes those parameters from the model and cuts both training compute and inference operations with zero change to the model output.
The standard transformer stack is a gauge-fixed, discretized representation of an $\mathrm{SO}(d_e-1)$ gauge theory, and inserting extra rotation matrices into the skip connections enlarges the symmetry so each transformer layer has its own gauge freedom.
Gauge-theory tools such as topological classification of configurations become applicable to transformer weights, suggesting that seed initialization may land in a topological class that gradient descent cannot deform into the optimal solution.
For GPT-2, GPT-2-XL, and LLaMA the counted redundancy is roughly 1.3%, 0.7%, and 0.3% of parameters respectively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The derivation silently assumes that layer normalization's learned per-dimension scale and shift rotate along with the embeddings; an immediate testable extension is to rotate $\gamma$ and $\beta$ as well, or to remove them, and verify numerically whether the exact invariance then holds.
Even if ordinary transformers violate exact invariance at the LayerNorm step, the symmetry suggests designing a modified normalization that is exactly invariant, which could serve as an architectural regularizer or as a post-training compression step orthogonal to pruning and quantization.
The redundancy count yields a direct numerical test: the Hessian of the loss at a trained model should have near-zero eigenvalues along the predicted flat directions, and the numerical rank of output changes under gauge moves should match the predicted count.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Leo,

The useful part of this paper is the explicit count: for a transformer block where LayerNorm is just mean-variance normalization and gamma/beta are absent, the attention matrix is invariant under GL(dh) transformations of Q and K, and the stack has a redundancy of 2*nt*nh*dh^2 + (de-1)(de-2)/2 parameters. That count is arithmetically fine, and the flat-valley interpretation is legitimate. If you strip away the physics packaging, there is a real, if modest, parameter-reduction observation here.

The problem is the word 'default'. The architecture as defined in Section 4.2 includes LayerNorm with learned scale and shift, but the gauge transformation list in equations (6)-(13) never transforms gamma or beta. For standard LayerNorm, y = gamma * (x - mean)/std + beta, and a rotation g does not commute with the componentwise multiplication by gamma. No transformation of gamma and beta can fix that for generic rotations, because g D_gamma is not diagonal. So equation (7) is false for the architectures the paper claims to cover. This isn't a minor omission; it breaks the central claim in Section 6.

Two smaller issues: the ReLU analysis rules out all positive diagonal scalings, but relu(lambda y) = lambda relu(y) for lambda >= 0, so those are valid gauge directions. And g(3)=g(0) in (10) is asserted without resolving the index structure of the value path. Also, the paper cites only the nGPT hypersphere paper and the usual LLM references; the QK invariance under invertible linear maps is already in the ML symmetry literature, and that prior work is not engaged.

All that said, the paper is clearly written and the author is honestly trying to pin down a real symmetry. It just overclaims. I wouldn't cite this as evidence that standard transformers have gauge freedom, but the stripped-down version of the redundancy count could be salvageable.

If it lands on your desk, I'd send it to review with an eye toward scoping the claim to architectures with rotation-equivariant normalization (or no scale/shift). It's also a good reading-group exercise in checking whether symmetries really close.

Best.

Referee Report

4 major / 5 minor

Summary. The paper claims that a standard transformer stack possesses an exact continuous reparameterization symmetry: under specified transformations of the embedding, Q, K, V, L, W, and W-hat matrices (Eqs. 6-13), the model function is unchanged for all inputs. The authors derive invariance conditions in Eqs. (14)-(16), count the redundant parameters in Eq. (17), report redundancy percentages for GPT-2, GPT-2-XL, and LLaMA in Table 1, and interpret the architecture as a gauge-fixed discretized SO(de-1) gauge theory. The central claim is that every default transformer has exactly flat directions in parameter space that can be removed without loss of representational power.

Significance. If the proof were correct, the result would be a notable theoretical insight: it would give a principled parameter-reduction scheme for transformers, connect them to gauge theories, and predict exact flat valleys in the loss landscape. The paper is ambitious and clearly written, and the arithmetical redundancy counts in Table 1 are consistent with formula (17). However, the technical derivation fails at several load-bearing points: the affine LayerNorm scale and shift parameters are absent from the gauge transformation, the value-matrix condition contradicts the invariance requirement, and the ReLU homogeneity argument is incorrect. There is no numerical verification of the claimed invariance on any trained model. Because the main theorem is unsupported for the standard transformer architecture described in the paper, the manuscript cannot be accepted in its present form.

major comments (4)

[Section 4.2.1 and Eqs. (6)-(7)] The architecture explicitly includes LayerNorm scale and shift, but the gauge transformations in Eqs. (6)-(13) contain no transformation for the per-dimension parameters gamma and beta. If y = gamma * z + beta is the post-LayerNorm output with z the strictly normalized vector, then equivariance under g in SO(de-1) would require some gamma', beta' such that gamma' * (g z) + beta' = g(gamma * z + beta) for all z. Because the right-hand side contains g diag(gamma) z, which is not diagonal for a generic rotation, no such gamma', beta' can realize the symmetry. Consequently Eq. (7), which asserts Ebar_0 transforms as g Ebar_0, is false for standard learned LayerNorm. If Ebar is instead intended to exclude the affine part, then the scale and shift are never used in the derivation and the formula does not apply to the models in Table 1. Either way, the invariance of Eq. (2), the conditions (14)-(16), the redundancy count (17), and the conclusion in Section 6 are unsupported for the default transformer architecture.
[Section 4.4, Eq. (10) and following] The paper states 'From applying the values matrix similarly we also have g(3) = g(0).' This is inconsistent with the invariance requirement. With V' = h(3) V g(3) and Ebar' = g(0) Ebar, the term V Ebar transforms as V' Ebar' = h(3) V g(3) g(0) Ebar. For this to equal h(3) V Ebar for all Ebar, one needs g(3) g(0) = I, hence g(3) = (g(0))^{-1}. Since g(0) is a general element of SO(de-1), g(0)^{-1} is not equal to g(0) in general. The equality stated in the paper is therefore incorrect and the subsequent combined conditions inherit this error.
[Section 4.4, Eqs. (12)-(13)] The sentence 'the nonlinearity in the feed forward network forces f(0) = f(1) = 1' is not justified. ReLU is positively homogeneous: ReLU(c x) = c ReLU(x) for any c > 0, and similarly for positive diagonal matrices. Thus any positive scalar (or positive diagonal) f(0) with f(1) = (f(0))^{-1} leaves the feed-forward block invariant. The conclusion f(0)=f(1)=1 only follows under an additional restriction that is never stated or argued. As a result, the derivation of Eq. (17) underestimates the available symmetry and the redundancy count is incomplete.
[Section 4.4, Eq. (14)] As printed, Eq. (14) does not follow from the index conventions in Eqs. (2), (8), and (9). For the attention bilinear form Ebar^T Q K Ebar with the transformations as written, the conditions for invariance involve different placements of the transpose on g(1) and g(2) than those stated, or the notation in Eqs. (8)-(9) must be changed. The paper should either correct the transpose placement or explicitly define the matrix orientation of Q and K so that Eq. (14) can be verified. This is not merely a typographical point, since Eq. (14) is the first condition used to fix the gauge group.

minor comments (5)

[Notation, Eqs. (2) and (9)] The index order for Q is inconsistent: Eq. (2) writes Q_{mu A}, while Eq. (9) writes Q_{A mu}. The conventions for Q, K, and V should be fixed throughout so that matrix products in Eqs. (2)-(5) are well defined.
[Table 1] Table 1 reports redundancy percentages for GPT-2, GPT-2-XL, and LLaMA but does not give the values of de, dh, nh, and nt used in Eq. (17), nor the source of the stated parameter counts. Please provide these details so the arithmetic can be checked.
[Eq. (22)] Equation (22) is an empty numbered line; it should be removed or renumbered.
[References and metadata] The references contain several typographical issues ('er al.' instead of 'et al.', 'Timoth´ ee' instead of 'Timothée', missing spaces in URLs), and the header contains placeholder metadata ('JMLR TBD', 'Submitted 12/25', 'Published 12/29'). These should be corrected in a revised submission.
[Empirical verification] The paper would be substantially strengthened by a small numerical experiment that applies the claimed gauge transformation to a trained transformer and reports the maximum output difference across inputs; no such verification is currently included.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the invariance is proved directly from stated transformation rules; no fitted parameter is relabeled as a prediction and no load-bearing self-citation is used.

full rationale

The paper's central claim is a mathematical invariance statement, not an empirical prediction. The proof in Section 4.4 explicitly specifies candidate groups G=SO(de-1), H=GL(dh), and F=GL(df), lists transformation rules (6)-(13), and solves the resulting constraints (14)-(16). The redundancy count in equation (17) is the dimension of that explicitly constructed group action; it is a direct consequence of the proof, not a fitted parameter renamed as a prediction. The paper contains no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. The main caveat is a correctness/rigor point outside the circularity definition: the LayerNorm scale and shift parameters are listed in Section 4.2 but are absent from the transformation rules, so the equivariance step (7) is unproved for standard learned gamma and beta. That omission would undermine the theorem for default transformer architectures, but it is not a case where the conclusion is equivalent to the input by construction. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The derivation introduces no fitted constants, but it relies on several unstated modeling assumptions about LayerNorm, softmax, and ReLU. The most consequential is that per-dimension LayerNorm scale and shift either transform under the gauge group or are trivial, which does not hold for default transformers.

assumptions (4)

domain assumption LayerNorm scale and shift parameters either transform under the same gauge rotation or are absent.
Section 4.2.1 lists layer normalization with scale and shift, but the gauge transformations in equations (6) through (13) do not include gamma and beta. For the normalized embeddings to rotate by g, per-dimension affine parameters must be invariant or co-rotated, which is not established.
domain assumption The row normalization in attention (softmax) and the mask are invariant under the proposed transformations of Q, K, and embeddings.
Equation (2) uses Rownorm of a Masked attention score; the derivation assumes the attention matrix is exactly invariant, which holds only if softmax is applied after cancellations and the mask is embedding-independent.
ad hoc to paper The nonlinearity in the feed-forward network is assumed to force f0 = f1 = 1.
Section 4.4 states the nonlinearity forces f0 = f1 = 1, but ReLU admits positive diagonal scaling symmetries of the form relu(Dz) = D relu(z) for positive diagonal D, so the general GL(df) analysis is incomplete.
domain assumption The initial embedding and final unembedding can be adjusted by group elements at the boundaries.
Section 4.3 allows applying group elements just after token-to-vector and before vector-to-probability, but standard transformers do not have such adjustable boundary maps explicitly parameterized, so the invariance of the end-to-end model is conditional.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Transformer models are gauge invariant: A mathematical connection between AI and particle physics." pith.science (2026). https://pith.science/paper/OWPD4ST2

@misc{pith2026241214543,
  author       = {Pith},
  title        = {Pith review of: Transformer models are gauge invariant: A mathematical connection between AI and particle physics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWPD4ST2}},
  note         = {Machine review of arXiv:2412.14543}
}

read the original abstract

In particle physics, the fundamental forces are subject to symmetries called gauge invariance. It is a redundancy in the mathematical description of any physical system. In this article I will demonstrate that the transformer architecture exhibits the same properties, and show that the default representation of transformers has partially, but not fully removed the gauge invariance.

Figures

Figures reproduced from arXiv: 2412.14543 by the authors.

**Figure 2.** Example of an unhelpful flat direction. That being said, there is a natural group that I will pursue in this work. This may not be the full symmetry group of transformer models, but it is the symmetry that I have uncovered with certainty thus far. For an embedding space of dimension de, the layer normalization restricts the tokens to live on intersection of the unit sphere in Rde and the plane through the origin per… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

6 extracted references · 5 linked inside Pith

[2]

Gemini Team, Rohan Anil, and et al

URL https://arxiv.org/ abs/2303.08774. Gemini Team, Rohan Anil, and et al. Gemini: A family of highly capable multimodal models,

arXiv
[3]

URL https://arxiv.org/abs/2312.11805

arXiv
[5]

URL https://arxiv.org/abs/1706.03762. C. N. Yang and R. L. Mills. Conservation of isotopic spin and isotopic gauge invariance. Phys. Rev., 96:191–195, Oct

arXiv
[1954]

URL https://link

doi: 10.1103/PhysRev.96.191. URL https://link. aps.org/doi/10.1103/PhysRev.96.191. 10

doi:10.1103/physrev.96.191
[2023]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

URL https://arxiv.org/abs/2302.13971. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need,

arXiv
[2024]

org/abs/2410.01131

URLhttps://arxiv. org/abs/2410.01131. OpenAI, Josh Achiam, and er al. Gpt-4 technical report,

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [2]

Gemini Team, Rohan Anil, and et al

URL https://arxiv.org/ abs/2303.08774. Gemini Team, Rohan Anil, and et al. Gemini: A family of highly capable multimodal models,

arXiv

[2] [3]

URL https://arxiv.org/abs/2312.11805

arXiv

[3] [5]

URL https://arxiv.org/abs/1706.03762. C. N. Yang and R. L. Mills. Conservation of isotopic spin and isotopic gauge invariance. Phys. Rev., 96:191–195, Oct

arXiv

[4] [1954]

URL https://link

doi: 10.1103/PhysRev.96.191. URL https://link. aps.org/doi/10.1103/PhysRev.96.191. 10

doi:10.1103/physrev.96.191

[5] [2023]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

URL https://arxiv.org/abs/2302.13971. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need,

arXiv

[6] [2024]

org/abs/2410.01131

URLhttps://arxiv. org/abs/2410.01131. OpenAI, Josh Achiam, and er al. Gpt-4 technical report,

arXiv