Understanding Gated Neurons in Transformers from Their Input-Output Functionality

Hinrich Sch\"utze; Sebastian Gerstner

REVIEW 3 major objections 6 minor 31 references

Understanding Gated Neurons in Transformers from Their Input-Output Functionality

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Across 12 language models, transformer neurons show consistent depth-wise roles: early layers enrich, later layers deplete.

desk verdict Solid weight-geometry study with a likely-real layer trend; the enrichment/depletion labels are not yet backed by activation statistics, so treat the headline as descriptive. read the letter →

arxiv 2505.17936 v1 pith:DBJRVLWB submitted 2025-05-23 cs.LG cs.CL

Sebastian Gerstner , Hinrich Schütze This is my paper

classification cs.LGcs.CL

keywords transformerinterpretabilitygatedactivationfunctionsSwiGLUneuronsneuroninput-outputanalysisenrichmentanddepletioncosinesimilarityofweightsresidualstreamfactualrecall

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a gated neuron's input–output functionality in a transformer—what concept it detects versus what concept it writes—is readable from the geometry of its three weight vectors alone. Using the cosine similarities between the linear input, gate, and output weights, the authors define six neuron classes (enrichment, depletion, conditional enrichment, conditional depletion, proportional change, and orthogonal output) and apply them to all MLP neurons in 12 language models. They find the same depth-wise trend everywhere: neurons that amplify a detected direction dominate early-middle layers, while neurons that suppress detected directions become more common in the final layers. A sympathetic reader would care because this offers a cheap, exhaustive, parameter-only lens on internal computation that complements activation-based methods, and because it ties the layer trend to a known account of factual recall—first enrich a concept representation, then sharpen it for next-token prediction.

What carries the argument

The carrying machinery is the three weight vectors of a gated neuron—$w_{\rm in}$, $w_{\rm gate}$, $w_{\rm out}$—and the three cosine similarities among them, reduced to two meaningful ones by the symmetry that flipping both $w_{\rm in}$ and $w_{\rm out}$ preserves behavior. The taxonomy places neurons by $\cos(w_{\rm in}, w_{\rm out})$ (positive means enrichment, negative means depletion, near zero means orthogonal output) and $|\cos(w_{\rm gate}, w_{\rm out})|$ (high means unconditional, low means conditional), with 0.5 as the threshold. The named device 'double checking' is the geometric fact that two orthogonal reading vectors can both have high similarity to a third concept direction, which shrinks the activation region to a quadrant-like intersection and enables more precise concept detection. This weight-geometry machine is what generates the layer-by-layer statistics and the six input–output classes.

What would settle it

Take a random sample of neurons the weights label as enrichment or depletion, run the model on a large corpus, and measure whether each neuron's positive activation actually adds or removes the detected direction from the residual stream, by comparing the residual stream projection along $w_{\rm in}$ before and after the MLP. If the weight-based labels fail to predict the sign of the change for most sampled neurons, the layer-trend claim would not correspond to functional behavior.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that gated neurons in language models can be classified by the relation between what they read and what they write, computed as $\cos(w_{\rm in}, w_{\rm out})$ and $|\cos(w_{\rm gate}, w_{\rm out})|$; this yields a complete taxonomy with six input–output classes rather than a sparse list of special neurons. Across all 12 models, the median $\cos(w_{\rm in}, w_{\rm out})$ starts positive, peaks in early-middle layers, and turns negative near the end, meaning early-middle layers are dominated by enrichment and conditional enrichment while late layers shift toward depletion. The authors further report that in Llama more than 80% of input manipulators are conditional enrichment neurons, and that gate and input weights are mostly near-orthogonal while still checking the same concept—a phenomenon they call double checking. They interpret enrichment as representation enrichment, the first step of factual recall, and depletion as residual sharpening, and they present case studies showing that negative Swish values can invert a neuron's behavior, a mechanism previous work had not exhibited.

Load-bearing premise

The load-bearing premise is that a neuron's functional class can be read from its raw weight vectors alone, ignoring bias terms and layer norms, without checking whether these cosine geometries predict which neurons actually activate and what they do on real inputs.

Editorial extensions

If this is right

Because the classification uses only weights, it is exhaustive and fast: every neuron of an MLP gets an input–output class, whereas output-based functional-role schemes cover only a few percent of neurons.
The consistent early-enrichment/late-depletion trend across 12 models supports the stages-of-inference view of transformer depth: early-middle layers perform feature engineering or representation enrichment, and final layers perform residual sharpening.
The dominance of conditional enrichment—more than 80% of input manipulators in Llama—together with the near-orthogonality of $w_{\rm gate}$ and $w_{\rm in}$, implies that most neurons detect concepts with two independent checks rather than a single direction.
Negative Swish values are functional: the case studies show depletion and proportional-change neurons relying on weakly negative gate activations to invert what they write, so treating Swish as essentially ReLU is not a safe simplification for gated models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A concrete validation the authors did not run: for a sample of neurons, compare the weight-based class with an activation-based measurement of whether positive activation increases or decreases the detected direction in the residual stream; agreement on most neurons would convert the geometric trend into a functional one.
If the trend holds functionally, then model editing or factual-recall interventions should preferentially target enrichment neurons in early-middle layers and depletion neurons in the final layers, because those are the neurons that add or remove concept directions at the right time.
The same read-versus-write geometry could classify attention heads, which also read from and write to the residual stream, extending the taxonomy beyond MLP neurons; the paper does not do this.
The near-orthogonality of $w_{\rm gate}$ and $w_{\rm in}$ may reflect a robustness strategy under superposition: two independent checks for one concept reduce false-positive activation, a design principle that could generalize to other architectures.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid descriptive study of the weight geometry of gated neurons, and the cross-model layer trend in cos(win,wout) is likely real. But the paper's functional vocabulary—'enrichment' and 'depletion'—is a step beyond what the weight-only analysis supports. The stress-test note is right: the actual update is Swish(wgate·x)·(win·x)·wout, so a 'conditional enrichment' neuron whose gate preactivation is usually negative will subtract the detected direction, not add it. The paper knows Swish can be negative (Section 3.2) and even relies on it in the depletion case study, but the taxonomy collapses the gate sign using |cos(wgate,wout)|, and the layer counts don't check activation statistics. So the headline claim should be read as a statement about weight geometry, not verified input-output functionality.

What is genuinely new: a complete six-class taxonomy for gated activations, the dual notions of input manipulation and double checking, and an exhaustive layer-resolved classification across 12 SwiGLU/GeGLU models. The observation that the median cosine flips from positive to negative across depth, consistently across models, is a clean addition to the interpretability toolkit, and it connects naturally—though speculatively—to Geva et al.'s enrichment stage and Lad et al.'s stages of inference. Credit where due: they are careful to say the interpretations are possible, not definitive; they include a contingency table against Gurnee et al.'s functional roles; they release code (though the text says 'this Github URL' without actually giving the URL—minor but sloppy); and the appendix is a model of transparency, with many case studies.

Soft spots, in order: (1) The enrichment/depletion labels lack activation-based validation. This is the main weakness and it is load-bearing for the abstract's functional claims. (2) tau=0.5 is arbitrary; they admit it. (3) Bias and LayerNorm are ignored; that is fine for OLMo but not for Llama/Gemma. (4) The explanatory sections are speculation, which they acknowledge.

Overall, the descriptive contribution is solid enough for a serious referee. The right fix is either to rename the classes as weight-geometry classes or to add a validation experiment that tracks actual neuron contributions on real data. I'd send it to peer review, but I would push for that validation. I would cite it for the empirical cosine trend, with the caveat.

Referee Report

3 major / 6 minor

Summary. The paper proposes a weight-based taxonomy of gated MLP neurons in transformers. For each neuron with weights (win, wgate, wout), it computes the cosines cos(win,wout), |cos(wgate,wout)|, and |cos(wgate,win)|, and defines six IO classes: enrichment, depletion, conditional enrichment, conditional depletion, proportional change, and orthogonal output. The method is applied to 12 open-weight LLMs. The central empirical finding is that the median cos(win,wout) is positive in early-middle layers and negative in later layers across all 12 models, which the authors interpret as enrichment neurons dominating early-middle layers and depletion neurons dominating later layers. The paper also introduces the concept of 'double checking' and provides six case studies from OLMo-7B combining weight-based and activation-based analyses, plus an extensive appendix with per-model figures.

Significance. If the main claim holds, the paper contributes a cheap, exhaustive, and scalable tool for studying a relatively underexplored axis of neuron behavior: the interaction between what a neuron reads and what it writes. Strengths include the breadth of the empirical survey (12 models), the clearly presented weight-geometry plots, the open-source software, and the case studies that connect the taxonomy to functional roles from prior work. The main caveat is that the headline layer trend is currently a statement about weight geometry; whether it is a statement about input-output functionality on real data is not yet demonstrated. The paper's own Limitations section acknowledges that weight similarity is not one-to-one with semantic behavior, which is exactly the point that needs quantitative support.

major comments (3)

[Section 4.3 / Table 1 / Section 5.1] The enrichment/depletion labels are assigned from cos(win,wout) and |cos(wgate,wout)| alone, but the residual-stream update is Swish(wgate·xnorm)·(win·xnorm)·wout. When Swish(wgate·xnorm) is negative, a neuron with cos(win,wout)>0 subtracts rather than adds the detected direction and therefore behaves as a depletion neuron in that context. The paper itself emphasizes that Swish is negative on weakly negative gate inputs (Section 3.2) and relies on this mechanism in the depletion case study (Section 6.2). No activation statistics are reported to show that neurons labeled 'conditional enrichment' have mostly positive Swish values on real inputs. Consequently, the central claim that 'enrichment neurons dominate early-middle layers' is currently a claim about weight geometry, not about verified input-output functionality. I request an aggregate activation-based check: for each class and layer, measure the fraction of neurons whose sign(Swish(wgate·xnorm)) is positive on a representative corpus, and re-report the layer trend using the sign-corrected direction of the update.
[Section 4.4 / Figure 3] The discrete class counts that support statements such as 'the majority of these input manipulators (more than 80% in Llama) belong to just one class: conditional enrichment' depend on the arbitrary threshold tau=0.5. No sensitivity analysis is provided. Please vary tau (e.g., 0.3 and 0.7) and show that the class-composition trends and the early-to-late enrichment-to-depletion shift are stable, or report a continuous analogue, for example the joint density of cos(win,wout) and |cos(wgate,wout)| across layers, which Figure 5 only partially provides. Without this, the discrete-class statements are not robust to the choice of cutoff.
[Section 3.1 / Limitations] The classification ignores bias terms and LayerNorm parameters, and all layer-level claims are computed purely from weights. Since the functional interpretation depends on the distribution of xnorm, and LayerNorm changes the norm and mean of xmid, the authors should either justify that these omissions do not affect the conclusions or add a validation on real activations. The paper's own Limitations section states that 'mathematical similarities of weights are insightful, but they should not be taken as one-to-one representations of semantic similarity.' This is precisely the concern that needs a quantitative answer; otherwise the abstract's phrase 'input-output functionality' overstates what has been measured.

minor comments (6)

[Section 4.3] There is a LaTeX error in the line 'We first focus on on textbfenrichment and depletion'; it should read 'textbf{enrichment}'.
[Section 4.4] The sentence 'such a neuron may write a concept different from but semantically related to the one it detects (say, Ireland -> Dublin) and thus be be similar to an enrichment neuron' contains a duplicated 'be'.
[Figure 3] The stacked bar charts would benefit from a y-axis label (count or proportion); currently the numbers are printed but the axis is unlabeled, which makes the figure harder to scan.
[Appendix F.2] The kurtosis threshold of 230.9736 is reported to four decimal places with no explanation of how it was derived; please state the exact criterion used.
[Abstract / Section 7.4] The abstract says enrichment neurons are 'largely responsible for enriching concept representations, one of the first steps of factual recall,' while Section 7.4 presents this as a hypothesis and the conclusion states that ablation experiments are future work. Consider softening the abstract to 'consistent with' to match the strength of the evidence.
[Section 7.2] The term 'double checking' is used in the Introduction and in Section 6.2 before it is formally defined in Section 7.2; consider defining it at first use.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the layer trend is a direct empirical statistic of weight cosines, not a quantity derived from a fitted parameter or a self-citation.

full rationale

The paper's central claim—enrichment neurons dominate early-middle layers while later layers tend toward depletion—is an observed statistical pattern over the cosine similarities of pretrained model weights. The taxonomy is defined directly from these cosines (Section 4.3, Table 1), and the layer trend is then read off from the resulting histograms (Figure 3, Appendix I). There is no fitted parameter that is later renamed as a prediction: the threshold tau=0.5 is set a priori as a 'relatively permissive cutoff' (Section 4.4), and the authors do not tune it to reproduce the layer trend. No self-citation is load-bearing; the references to prior work (e.g., Elhage et al., Gurnee et al., Geva et al.) are background and not used to force any conclusion. The concern that negative Swish values could make nominally 'enrichment' neurons behave as depletion neurons on real inputs is a validity threat about whether weight geometry predicts activation-gated behavior, but it is not a circularity: the paper's classification is explicitly parameter-based, and the layer claim is about the distribution of these weight-defined classes, not about a quantity that was derived from the claim itself. The explanatory sections (Sections 7.3–7.5) are explicitly framed as hypotheses and future ablation tests, not as derivations from the data. Therefore no step in the paper's derivation chain reduces to its own inputs by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The main analysis is weight-based and does not fit any parameters to a target result; the only hand-chosen constant for the central claim is tau=0.5. The secondary thresholds in Appendix F affect only the functional-role comparison. The semantic interpretation relies on the linear representation hypothesis, and the explanatory claims about stages of inference are hypotheses the authors explicitly defer testing. No new physical entities are introduced; 'double checking' is a proposed mechanism without independent validation.

free parameters (4)

classification threshold tau = 0.5
Used in Section 4.4 to separate cosine values near 0 from |cos| >= 0.5 for assigning IO classes. Described as 'relatively permissive'. Class counts and the enrichment-dominance figures depend on this cutoff.
kurtosis threshold for prediction/suppression = 230.9736 (excess kurtosis)
Set in Appendix F.2 so prediction/suppression classes are disjoint from partition neurons. Affects the contingency table in Appendix F.3, not the main layer trend.
variance threshold for partition neurons = 0.0007
Set in Appendix F.2 to define partition neurons. A secondary threshold for the functional-role comparison, not for the main enrichment/depletion finding.
attention (de)activation cutoff = sqrt(2)/2 ≈ 0.7071
Used in Appendix F.2 to identify attention (de)activation neurons. Secondary to the central claim.

assumptions (4)

domain assumption Residual stream states can be interpreted as intermediate predictions and directions correspond to concepts (linear representation hypothesis).
Used in Section 4.1 and 7.4 to give semantic meaning to enrichment and depletion. Not necessary for the classification itself, but the paper's explanatory claims rely on it.
domain assumption Biases and layer norm parameters can be ignored when characterizing neuron IO functionality.
Section 3.1 states they are ignored for simplicity. The entire weight-only analysis assumes these do not change the qualitative cosine geometry.
standard math Swish negative values can produce meaningful negative activations that flip a neuron's enrichment/depletion behavior.
Section 3.2 discusses negative Swish; the case studies depend on this. This is a mathematical property of Swish, but its functional role is an assumption.
standard math Symmetry: flipping signs of win and wout preserves IO behavior; therefore the sign of cos(wgate,wout) is irrelevant.
Section 4.2; exact property of the activation function, used to define classes.

invented entities (2)

Input manipulation (neuron class)
purpose: A category of neurons whose output weight direction is aligned (or anti-aligned) with one of their reading directions, as opposed to orthogonal output neurons.
Introduced in Section 4 as a weight-geometry classification. The paper provides illustrative case studies but no external, falsifiable predictions that distinguish this category from other possible taxonomies.
Double checking (circuit mechanism)
purpose: Proposed mechanism where wgate and win are near-orthogonal yet both check for the same concept, narrowing the activation region for more precise detection.
Section 7.2 shows a geometric example and case studies, but no ablation or controlled experiment is provided. The authors list ablation as future work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Understanding Gated Neurons in Transformers from Their Input-Output Functionality." pith.science (2026). https://pith.science/paper/DBJRVLWB

@misc{pith2026250517936,
  author       = {Pith},
  title        = {Pith review of: Understanding Gated Neurons in Transformers from Their Input-Output Functionality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBJRVLWB}},
  note         = {Machine review of arXiv:2505.17936}
}

read the original abstract

Interpretability researchers have attempted to understand MLP neurons of language models based on both the contexts in which they activate and their output weight vectors. They have paid little attention to a complementary aspect: the interactions between input and output. For example, when neurons detect a direction in the input, they might add much the same direction to the residual stream ("enrichment neurons") or reduce its presence ("depletion neurons"). We address this aspect by examining the cosine similarity between input and output weights of a neuron. We apply our method to 12 models and find that enrichment neurons dominate in early-middle layers whereas later layers tend more towards depletion. To explain this finding, we argue that enrichment neurons are largely responsible for enriching concept representations, one of the first steps of factual recall. Our input-output perspective is a complement to activation-dependent analyses and to approaches that treat input and output separately.

Figures

Figures reproduced from arXiv: 2505.17936 by the authors.

**Figure 1.** Median of cos(win, wout) by layer (x-axis) for 12 models. For all models, the value is positive in the beginning and negative in the end, indicating that earlymiddle layers “enrich” the residual stream whereas later layers tend more towards depletion. WU to produce next-token logits. The information contained in the residual stream is represented as a highdimensional vector (of dimension dmodel). Individual model… view at source ↗

**Figure 2.** We define six input-output functionality [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Distribution of neurons by layer and category. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 5.** Figure 5: Fine-grained analysis of neuron IO behavior [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Visualization of the SwiGLU activation function for a single neuron. Boxes represent vectors, ellipses [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Distribution of neurons by layer and category for a range of models [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: Continuation of Figure [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

**Figure 9.** Figure 9: Boxplots for the distribution of weight cosine similarities in each layer. For [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

**Figure 10.** Figure 10: Continuation of Figure [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 18.** Figure 18: Llama-3.2-3B 27 [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 5 canonical work pages

[1]

01.AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, ...

arXiv 2025
[2]

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. https://arxiv.org/pdf/2303.08112 Eliciting latent predictions from transformers with the tuned lens

arXiv 2023
[3]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. https://transformer-circuits.pub/2022/toy_model/index.html Toy models of superposition

2022
[4]

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2...

work page 2021
[5]

Amit Elhelo and Mor Geva. 2024. https://arxiv.org/abs/2412.11965 Inferring functionality of attention heads from their parameters . Preprint, arXiv:2412.11965

arXiv 2024
[6]

Team Gemma. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma

doi:10.34740/kaggle/m/3301 2024
[7]

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.751 Dissecting recall of factual associations in auto-regressive language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216--12235, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.emnlp-main.751 2023
[8]

Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30--45, Abu Dhabi, United Arab Emirates. Association f...

doi:10.18653/v1/2022.emnlp-main.3 2022

Show all 31 references

[9]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446 Transformer feed-forward layers are key-value memories . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484--5495, ...

2021 doi
[10]

Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tusha...

2024
[11]

Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. 2024. https://arxiv.org/pdf/2401.12181 Universal neurons in gpt2 language models

2024 arXiv
[12]

Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. https://arxiv.org/pdf/2305.01610 Finding neurons in a haystack: Case studies with sparse probing

2023 arXiv
[13]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 arXiv
[14]

Vedang Lad, Wes Gurnee, and Max Tegmark. 2024. https://arxiv.org/abs/2406.19384 The remarkable robustness of llms: Stages of inference? Preprint, arXiv:2406.19384

2024 arXiv
[15]

Joseph Miller and Clement Neo. 2023. https://www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2 We found an neuron in gpt-2

2023
[16]

Beren Millidge and Sid Black. 2022. https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight The singular value decompositions of transformer weight matrices are highly interpretable

2022
[17]

Morcos, David G.T

Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. 2018. https://arxiv.org/pdf/1803.06959.pdf On the importance of single directions for generalization

2018 arXiv
[18]

Neel Nanda and Joseph Bloom. 2022. Transformerlens. https://github.com/TransformerLensOrg/TransformerLens

2022
[19]

Jingcheng Niu, Andrew Liu, Zining Zu, and Gerald Penn. 2024. https://arxiv.org/pdf/2405.02421 What does the knowledge neuron thesis have to do with knowledge?

2024 arXiv
[20]

nostalgebraist. 2020. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens Interpreting gpt: The logit lens

2020
[21]

Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. https://proceedings.mlr.press/v235/park24c.html The linear representation hypothesis and the geometry of large language models . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceeding...

2024
[22]

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

2019
[23]

Cody Rushing and Neel Nanda. 2024. https://proceedings.mlr.press/v235/rushing24a.html Explorations of self-repair in language models . In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 42836--...

2024
[24]

Noam Shazeer. 2020. https://arxiv.org/pdf/2002.05202 Glu variants improve transformer

2020 arXiv
[25]

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennigh...

2024
[26]

Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. 2024. https://arxiv.org/abs/2406.16254 Confidence regulation neurons in language models

2024 arXiv
[27]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aur \'e lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/pdf/2302....

2023 arXiv
[28]

Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://api.semanticscholar.org/CorpusID:13756489 Attention is all you need . In Neural Information Processing Systems

2017
[29]

Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2024. https://doi.org/10.18653/v1/2024.findings-acl.75 Neurons in large language models: Dead, n-gram, positional . In Findings of the Association for Computational Linguistics: ACL 2024, pages 1288--1301, Bangkok, Tha...

2024 doi
[30]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

2024 arXiv
[31]

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

01.AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, ...

arXiv 2025

[2] [2]

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. https://arxiv.org/pdf/2303.08112 Eliciting latent predictions from transformers with the tuned lens

arXiv 2023

[3] [3]

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. https://transformer-circuits.pub/2022/toy_model/index.html Toy models of superposition

2022

[4] [4]

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2...

work page 2021

[5] [5]

Amit Elhelo and Mor Geva. 2024. https://arxiv.org/abs/2412.11965 Inferring functionality of attention heads from their parameters . Preprint, arXiv:2412.11965

arXiv 2024

[6] [6]

Team Gemma. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma

doi:10.34740/kaggle/m/3301 2024

[7] [7]

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.751 Dissecting recall of factual associations in auto-regressive language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216--12235, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.emnlp-main.751 2023

[8] [8]

Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30--45, Abu Dhabi, United Arab Emirates. Association f...

doi:10.18653/v1/2022.emnlp-main.3 2022