Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read All LayerNorm layers can be removed from every GPT-2 model for a small loss increase, so they are not load-bearing for language modeling.

desk verdict The LN-removal scaling is real and useful, but the 'small loss increase' claim holds mainly on a post-hoc filtered Pile, and the abstract overreaches beyond the paper's own behavioral findings. read the letter →

arxiv 2507.02559 v1 pith:2YE4ZXKG submitted 2025-07-03 cs.LG

Luca Baroni , Galvin Khara , Joachim Schaeffer , Marat Subkhankulov , Stefan Heimersheim This is my paper
classification cs.LG
keywords LayerNormremovalGPT-2mechanisticinterpretabilitydirectlogitattributionpatchingconfidenceneuronsFakeLNfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LayerNorm, the nonlinear normalizer used throughout transformer language models, is dispensable at inference time. It does this by replacing every LayerNorm in GPT-2 Small, Medium, Large, and XL with a linear 'FakeLN' operation, then fine-tuning the model one LayerNorm at a time. The resulting models stay within roughly 0.01 to 0.1 cross-entropy loss of strong baselines on filtered text, with GPT-2 XL within 0.03, and the authors conclude that LayerNorm cannot play a substantial role in language modeling. A consequence that matters for interpretability is that direct logit attribution becomes exact once the final nonlinearity is gone. The paper also reports that attribution patching does not improve, that first-token norms stop being anomalous, and that confidence-regulating neurons lose their effect.

What carries the argument

The central object is the FakeLN block, which replaces LayerNorm's per-position division by the vector's standard deviation with division by a fixed scalar, the average standard deviation estimated over a batch, followed by the original learned gain and bias: $\text{FakeLN}(x) = (x - \mu)/\sigma_{\text{avg}} \odot \gamma + \beta$. Because this is a linear operation, it can be folded into neighboring layers, leaving no inference-time nonlinearity. The removal procedure sequentially swaps each LayerNorm for FakeLN while fine-tuning, and an auxiliary loss penalizes deviations of per-position standard deviations from a target, which stabilizes the process. This machinery carries the argument because it converts a nonlinear normalization into a linear one without retraining the model from scratch, and it is what makes the interpretability improvements possible.

What would settle it

Run the exact sequential LN-removal protocol on a modern 7B-parameter model trained with current recipes and evaluate on unfiltered held-out data; if the loss gap to the baseline exceeds roughly 0.1 cross-entropy or training becomes unstable, the conclusion that LayerNorm is dispensable for language modeling would fail outside the GPT-2 family.

Watch

Extended reading notes

Core claim

The central claim is that all LayerNorm layers can be removed from every GPT-2 model with only a small increase in validation loss (for example +0.03 cross-entropy loss for GPT-2 XL), and that this loss persists even with extended fine-tuning, indicating LayerNorm provides a small but real benefit that cannot be tuned away. The removal is done by replacing each LayerNorm with a linear transformation initialized close to the original normalization, then fine-tuning the model while removing one LayerNorm at a time, from the MLP norms through the query/key and value norms and finally the final norm. On the interpretability side, the paper shows that direct logit attribution, which previously deviated from the true direct effect by roughly 50%, becomes exactly correct in LN-free models. Attribution patching accuracy, however, does not improve, suggesting that its errors come from nonlinearities other than LayerNorm. The paper also finds that the same 'confidence neurons' identified in the original models persist in LN-free models but have no measurable effect, confirming that LayerNorm's nonlinearity was their enabling mechanism.

Load-bearing premise

The claim that removing LayerNorm costs only a small loss increase rests on evaluating with The Pile-filtered, which drops sequences containing tokens absent from OpenWebText; on unfiltered The Pile, the largest LN-free model has a mean cross-entropy loss of 130.22 driven by just three samples, so the small-loss conclusion depends on the curated evaluation distribution.

Editorial extensions

If this is right

  • The released LN-free GPT-2 models can serve as proxies for the original models in mechanistic interpretability studies, with the caveat that internal similarity is not exact.
  • Direct logit attribution becomes an exact measure of each component's direct effect on logits, eliminating the need for linearization approximations that carry roughly 50% error in the original models.
  • The failure of attribution patching to improve in LN-free models implies that its inaccuracies are caused by nonlinearities other than LayerNorm, redirecting future work on that method.
  • First-position tokens lose their outsized norm in LN-free models, and attention sink rates drop, suggesting LayerNorm is a major driver of the first-token norm anomaly.
  • The persistence of structurally identical confidence neurons with no functional effect confirms that these neurons regulate confidence by exploiting the final LayerNorm's scaling.
  • The finding that fine-tuning data for LN removal grows sublinearly with model size suggests the method may scale to larger models, though the paper only demonstrates this up to 1.5 billion parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the exact direct effect available in LN-free models could be used to validate attribution-based circuit discovery methods, since ground-truth component effects become computable without approximation.
  • A testable extension would be applying the same sequential removal protocol to modern decoder-only models such as Llama or Pythia; if the small-loss result does not transfer, the claim that LayerNorm is dispensable in language modeling would be limited to GPT-2-style training and scale.
  • The catastrophic overconfidence on rare formatting tokens in unfiltered text suggests LN-free models may be poorly calibrated on out-of-distribution inputs, and a simple calibration regularizer or temperature correction could be tested as a mitigation.
  • The dramatic reduction in first-token norm inflation raises the question whether attention-sink phenomena in other models are likewise LayerNorm-driven, which could be checked by measuring sink rates before and after LN removal in a second model family.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a fine-tuning protocol that replaces every LayerNorm (LN) block in GPT-2 Small, Medium, Large, and XL with a linear 'FakeLN' using a fixed average standard deviation, removing LN blocks sequentially while fine-tuning on OpenWebText with an auxiliary norm-consistency loss. The authors report that the resulting LN-free models achieve cross-entropy losses within roughly 0.01–0.1 of the original or vanilla fine-tuned baselines on OpenWebText and on a custom filtered subset of The Pile, and they use these models to study interpretability: direct logit attribution becomes exactly equal to the direct effect, attribution patching accuracy does not improve, confidence neurons in the final MLP are disabled, and first-token norm/attention-sink behavior changes. The paper concludes that LN cannot play a substantial role in language modeling and releases an LN-free GPT-2 model suite.

Significance. If the central claim held as stated, the work would be valuable for two communities: it challenges the assumption that normalization layers are indispensable in pretrained transformers, and it provides concrete interpretability benefits, since DLA becomes exact and the released models can serve as linearized proxies for mechanistic analysis. The empirical protocol is well documented, the model suite is released on Hugging Face, and the paper includes a thoughtful control (vanilla fine-tuning) that separates fine-tuning effects from LN removal. The attribution-patching null result and the confidence-neuron ablation study are useful, falsifiable findings. However, the strongest claim in the abstract and conclusion is calibrated to a post-hoc filtered evaluation distribution and is in tension with the paper's own behavioral findings, so the headline contribution needs substantial reframing.

major comments (4)
  1. [Abstract; Section 4, Table 1; Appendix C] Abstract; Section 4, Table 1; Appendix C
  2. [Section 5.4; Section 5.3; Section 6.1] Section 5.4; Section 5.3; Section 6.1
  3. [Section 4; Appendix B] Section 4; Appendix B
  4. [Section 5.1, Eqs. (4)–(6)] Section 5.1, Eqs. (4)–(6)
minor comments (5)
  1. [Section 5.4] Section 5.4
  2. [Table 1; Appendix C] Table 1; Appendix C
  3. [Section 5.2] Section 5.2
  4. [Section 5.3] Section 5.3
  5. [Section 3, Eq. (3)] Section 3, Eq. (3)

Circularity Check

1 steps flagged · score 1.0 of 10

Main LN-removal claim is externally benchmarked and non-circular; the exact-DLA result is true by construction.

  1. self definitional [Section 5.1, Eqs. (4)–(5), with FakeLN defined in Eq. (2)]
    "In contrast, the LN-free fine-tuned model achieves a perfect 0.00% [0.00%, 0.00%] NMAE, empirically confirming that removing the non-linearity introduced by LN eliminates the discrepancy between DLA and direct ablation methods."

    In the LN-free suite, every LN is replaced by FakeLN, an affine operation with a fixed scalar σavg (Eq. 2). Under an affine final norm, DE(c) = LN(r)·WU − LN(r−c)·WU equals LN(c)·WU exactly, and DLA(c) with the cached constant scale is the same quantity. Hence NMAE = 0.00% is guaranteed by the construction of the model, not by an empirical test. This is a self-definitional result: the 'exact direct effect' claim is a restatement of linearity. It is not load-bearing for the main LN-removal loss claim, which is benchmarked externally.

full rationale

The paper's central claim — that all LN layers can be removed from GPT-2 models with only a small validation-loss increase — is an empirical result measured against external baselines (original GPT-2 and vanilla fine-tuned models) on OpenWebText and The Pile-filtered. The FakeLN replacement uses σavg estimated from the model's own activations, but this constant is not fitted to the final loss or to the validation outcome; it is a data-derived scale used during fine-tuning. The comparison to original and vanilla-finetuned models provides an independent benchmark, so the main result is not derived from its own inputs. The interpretability analyses are also mostly empirical: attribution patching shows no improvement, and the confidence-neuron ablation results are measured, not assumed. The one by-construction item is the DLA result: once the final LN is replaced by a linear FakeLN, the direct effect and direct logit attribution are algebraically identical, so the reported 0.00% NMAE is a mathematical consequence rather than an empirical discovery. This does not undermine the main loss-based conclusion, which stands on external evaluation. The Pile-filtered evaluation is a data-curation concern rather than circularity: the paper discloses the filtering criterion and also reports the unfiltered The Pile value, so the limitation is visible; it affects the generality of the '+0.03' claim but does not make the derivation circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result is empirical: LN layers are replaced by a linear FakeLN module, and the model is fine-tuned. The only data-derived quantities are the per-layer scaling constants and the auxiliary-loss hyperparameters, which are disclosed. The load-bearing assumption is that the filtered evaluation sets capture 'small loss increase' while unfiltered data shows dramatic overconfidence, and that GPT-2 results generalize to all transformers.

free parameters (3)
  • σavg per LN block = Not reported; frozen at removal time
    FakeLN replaces the nonlinear norm with a scalar σavg estimated from activations; it is a data-derived constant that shapes the linear replacement and is specific to each model and layer.
  • Auxiliary loss weight λ = 0.1 (Small/Medium), 0.03 (Large), 0.01 (XL)
    Hyperparameter controlling the auxiliary standard-deviation-consistency loss; chosen per model size to stabilize fine-tuning (Section 3, Table 3).
  • LN removal schedule (start/gap steps) = e.g., Small: MLP start 20 gap 2; XL: V start 434 gap 6
    Sequence and spacing of layer removals; required to avoid instabilities and irrecoverable loss spikes (Appendix B).
assumptions (4)
  • domain assumption Fine-tuning on OpenWebText is a suitable proxy for the model's training distribution and for evaluating language modeling capability.
    The paper uses OpenWebText for fine-tuning and validation, treating it as representative of general language modeling performance.
  • ad hoc to paper The filtered Pile evaluation set, which removes sequences with tokens absent from OpenWebText, is the appropriate benchmark for the central small-loss claim.
    The filter was introduced after observing extreme losses on unfiltered The Pile; the main conclusion depends on this curated set (Appendix C).
  • domain assumption Loss on validation sets (OWT, The Pile-filtered) is a sufficient measure of whether LN is needed for language modeling.
    The paper equates language modeling quality with cross-entropy loss on these sets, without examining other capabilities like long-range coherence or robustness.
  • domain assumption Results on the GPT-2 family generalize to other transformer architectures.
    The title and abstract make a general claim about transformers, while the experiments only cover GPT-2 variants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability." pith.science (2026). https://pith.science/paper/2YE4ZXKG

@misc{pith2026250702559,
  author       = {Pith},
  title        = {Pith review of: Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YE4ZXKG}},
  note         = {Machine review of arXiv:2507.02559}
}
read the original abstract

Layer-wise normalization (LN) is an essential component of virtually all transformer-based large language models. While its effects on training stability are well documented, its role at inference time is poorly understood. Additionally, LN layers hinder mechanistic interpretability by introducing additional nonlinearities and increasing the interconnectedness of individual model components. Here, we show that all LN layers can be removed from every GPT-2 model with only a small increase in validation loss (e.g. +0.03 cross-entropy loss for GPT-2 XL). Thus, LN cannot play a substantial role in language modeling. We find that the amount of fine-tuning data needed for LN removal grows sublinearly with model parameters, suggesting scaling to larger models is feasible. We release a suite of LN-free GPT-2 models on Hugging Face. Furthermore, we test interpretability techniques on LN-free models. Direct logit attribution now gives the exact direct effect of individual components, while the accuracy of attribution patching does not significantly improve. We also confirm that GPT-2's "confidence neurons" are inactive in the LN-free models. Our work clarifies the role of LN layers in language modeling, showing that GPT-2-class models can function without LN layers. We hope that our LN-free analogs of the GPT-2 family of models will enable more precise interpretability research and improve our understanding of language models.

Figures

Figures reproduced from arXiv: 2507.02559 by the authors.

Figure 1
Figure 1. Main training loss curves for all GPT-2 variants during LN removal. Original GPT-2 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Activation patching and attribution patching applied on the residual stream at different [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. L2 norm growth for first position tokens (left) versus other positions tokens (right) for [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Absolute change in cross-entropy (CE) when ablating top-3 confidence neurons in GPT-2 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Identification of confidence neurons in GPT-2 Small (top) and GPT-2 Medium (bottom) [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: SVD of the unembedding matrix for GPT-2 Small (left) and GPT-2 Medium (right) across [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Change in CE loss upon mean ablation of top-3 confidence neurons for GPT-2 Small (left) [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Cumulative effect of ablating the top three confidence neurons in GPT-2 Medium. Left: [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Verifiable Transformers: Solver-Checkable Circuit Explanations

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    Presents a solver-verifiable framework for Transformer circuits, with exhaustive checks on small symbolic tasks and surrogate methods for larger models.

  2. Discovering Interpretable Algorithms by Decompiling Transformers to RASP

    cs.LG 2026-02 conditional novelty 7.0 of 10

    Trained length-generalizing transformers on small algorithmic and formal-language tasks can be automatically decompiled into short, interpretable RASP-style programs, while non-generalizing models generally cannot.

Reference graph

Works this paper leans on

40 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    2025

  2. [2]

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016

    2016

  3. [3]

    Why do LLMs attend to the first token? arXiv preprint arXiv:2504.02732, 2025

    Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do LLMs attend to the first token? arXiv preprint arXiv:2504.02732, 2025

    arXiv 2025

  4. [4]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  5. [5]

    a nni, Avery Griffin, J \

    Lucius Bushnaq, Stefan Heimersheim, Nicholas Goldowsky-Dill, Dan Braun, Jake Mendel, Kaarel H \"a nni, Avery Griffin, J \"o rn St \"o hler, Magdalena Wache, and Marius Hobbhahn. The local interaction basis: Identifying computationally-relevant and sparsely interacting features in neural networks. arXiv preprint arXiv:2405.10928, 2024

    arXiv 2024

  6. [6]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

    2021

  7. [7]

    Jacobian sparse autoencoders: Sparsify computations, not just activations

    Lucy Farnik, Tim Lawson, Conor Houghton, and Laurence Aitchison. Jacobian sparse autoencoders: Sparsify computations, not just activations. arXiv preprint arXiv:2502.18147, 2025

    arXiv 2025

  8. [8]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    arXiv 2020

Show all 40 references
  1. [9]

    Finding alignments between interpretable causal variables and distributed neural representations

    Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pages 160--187. PMLR, 2024

    2024

  2. [10]

    Gemini Team , Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Li...

    2024 arXiv

  3. [11]

    Openwebtext corpus

    Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  4. [12]

    Geometric interpretation of layer normalization and a comparative analysis with rmsnorm, 2025

    Akshat Gupta, Atahan Ozdemir, and Gopala Anumanchipalli. Geometric interpretation of layer normalization and a comparative analysis with rmsnorm, 2025

    2025

  5. [13]

    Universal neurons in gpt2 language models

    Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. Transactions on Machine Learning Research, 2024

    2024

  6. [14]

    You can remove gpt2's LayerNorm by fine-tuning

    Stefan Heimersheim. You can remove gpt2's LayerNorm by fine-tuning. In NeurIPS Workshop: Interpretable AI: Past, Present and Future, 2024

    2024

  7. [15]

    How to use and interpret activation patching

    Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255, 2024

    2024 arXiv

  8. [16]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine ...

    2015

  9. [17]

    Visit: Visualizing and interpreting the semantic information flow of transformers

    Shahar Katz and Yonatan Belinkov. Visit: Visualizing and interpreting the semantic information flow of transformers. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14094--14113, 2023

    2023

  10. [18]

    Sparse autoencoders work on attention layer outputs, Jan 2024

    Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Sparse autoencoders work on attention layer outputs, Jan 2024. URL https://www.alignmentforum.org/posts/DtdzGwFh9dCfsekZZ/sparse-autoencoders-work-on-attention-layer-outputs

    2024

  11. [19]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  12. [20]

    Sparse feature circuits: Discovering and editing interpretable causal graphs in language models

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024

    2024 arXiv

  13. [21]

    Copy Suppression: Comprehensively Understanding an Attention Head

    Callum McDougall , Arthur Conmy , Cody Rushing , Thomas McGrath , and Neel Nanda . Copy Suppression: Comprehensively Understanding an Attention Head . arXiv e-prints, art. arXiv:2310.04625, October 2023. doi:10.48550/arXiv.2310.04625

    2023 doi arXiv

  14. [22]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT . Advances in Neural Information Processing Systems, 36, 2022. arXiv:2202.05262

    2022 arXiv

  15. [23]

    Tinymodel: A tinystories lm with saes and transcoders, 2024

    Noa Nabeshima. Tinymodel: A tinystories lm with saes and transcoders, 2024. URL https://github.com/noanabeshima/tinymodel. Accessed: 2025-05-15

    2024

  16. [24]

    Attribution patching: Activation patching at industrial scale, Mar 2023 a

    Neel Nanda. Attribution patching: Activation patching at industrial scale, Mar 2023 a . URL https://www.alignmentforum.org/posts/gtLLBhzQTG6nKTeCZ/attribution-patching-activation-patching-at-industrial-scale

    2023

  17. [25]

    Exploratory analysis demo (transformerlens)

    Neel Nanda. Exploratory analysis demo (transformerlens). https://colab.research.google.com/github/neelnanda-io/TransformerLens/blob/main/demos/Exploratory_Analysis_Demo.ipynb, 2023 b

    2023

  18. [26]

    Transformerlens

    Neel Nanda and Joseph Bloom. Transformerlens. https://github.com/TransformerLensOrg/TransformerLens, 2022

    2022

  19. [27]

    interpreting gpt: the logit lens, Aug 2020

    nostalgebraist. interpreting gpt: the logit lens, Aug 2020. URL https://www.alignmentforum.org/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  20. [28]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024

  21. [29]

    Direct and indirect effects

    Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373--392. 2022

    2022

  22. [30]

    Confidence regulation neurons in language models

    Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. arXiv preprint arXiv:2406.16254, 2024

    2024 arXiv

  23. [31]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  24. [32]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  25. [33]

    Understanding the failure of batch normalization for transformers in nlp

    Jiaxi Wang, Ji Wu, and Lei Huang. Understanding the failure of batch normalization for transformers in nlp. Advances in Neural Information Processing Systems, 35: 0 37617--37630, 2022 a

    2022

  26. [34]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022 b

    2022 arXiv

  27. [35]

    Re-examining layernorm

    Eric Winsor. Re-examining layernorm. Alignment Forum, 2022. URL https://www.alignmentforum.org/posts/jfG6vdJZCwTQmG7kb/re-examining-layernorm

    2022

  28. [36]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. Advances in Neural Information Processing Systems, 37, 2024. arXiv:2309.17453

    2024 arXiv

  29. [37]

    Interpreting the repeated token phenomenon in large language models

    Itay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, and Yossi Gandelsman. Interpreting the repeated token phenomenon in large language models. arXiv preprint arXiv:2503.08908, 2025

    2025 arXiv

  30. [38]

    Root mean square layer normalization, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019

    2019

  31. [39]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042, 2023

    2023 arXiv

  32. [40]

    Transformers without normalization, 2025

    Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization, 2025

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.