Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

Inferring Functionality of Attention Heads from their Parameters

T0 review · 2 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that an attention head's token-level operations can be read off its weights alone, and that these readings match real model behavior.

desk verdict A useful parameter-only head scanner with solid but partly over-tuned validation; worth refereeing, with a request for sensitivity analysis. read the letter →

arxiv 2412.11965 v2 pith:CKLMIZ6R submitted 2024-12-16 cs.CL

Amit Elhelo , Mor Geva This is my paper · ORCID
classification cs.CL
keywords attentionheadsmechanisticinterpretabilityvocabularyspaceprojectionstoken-to-tokenrelationslargelanguagemodelsparameter-onlyanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Attention heads are small building blocks inside large language models, and this paper claims their jobs can be inferred directly from the head's parameters, without running the model or training anything. The authors build MAPS, which reads the head's writing weights as a matrix of scores between every pair of tokens in the vocabulary, then checks which of twenty predefined token-to-token relations (country to capital, word to past tense, English to French, and so on) the head ranks highly. They show these static scores correlate with the tokens the head actually promotes during inference, and that deleting the heads flagged for a relation degrades tasks needing that relation far more than deleting random heads. The same vocabulary-pair matrix lets them find a head's most salient operations automatically, producing plausible human-checked descriptions for most middle- and upper-layer heads in two models. If the claim holds, attention-head analysis becomes cheap and exhaustive: a full map of a model's functions without inference or circuit hunting.

What carries the argument

The load-bearing object is the vocabulary-pair matrix $M = E(W_{VO})U$, where $E$ is the embedding matrix, $W_{VO}$ the head's output-value weights, and $U$ the unembedding matrix; each entry scores how strongly the head maps source token $s$ to target token $t$. The companion quantity is the relation score $\varphi_R(M)$, the fraction of a relation's token pairs whose target lands in the top-$k$ of the source row, with a suppression variant computed on $-M$. These convert functionality inference into submatrix reading, and pair with the saliency score $\sigma_t = \|e_t W_{VO}\| / \|e_t\|$ to find which tokens the head most affects.

What would settle it

Take a head MAPS labels as implementing country-to-capital and run prompts that place the country name in different syntactic positions while recording the attention weight the head puts on the country token; if the head no longer promotes the capital whenever that attention weight is low, the operation lives in the attention pattern as much as in the writing weights, and the parameter-only reading is incomplete.

Watch

Extended reading notes

Core claim

The central discovery is that the writing part of an attention head, folded into the matrix $M = E(W_{VO})U$, is a readable token-to-token mapping. For a predefined relation $R$ built from pairs $(s,t)$, the head's implementation strength is $\varphi_R(M)$, the fraction of source tokens for which the target is in the top-$k$ promoted tokens of that row. The paper argues this parameter-only score is not a projection artifact: it correlates 0.36--0.95 with the head's observed outputs across 20 relations and six LLMs, and causally removing heads with high scores drops task accuracy by at least 32% relative to random ablations while leaving control tasks less affected. The matrix also yields salient operations by scoring how much the head changes each token's embedding norm, and an LLM-based pipeline turns those salient mappings into natural-language descriptions that human annotators judge plausible in the large majority of pattern-present cases.

Load-bearing premise

The load-bearing premise is that a head's function is visible in the linear token-to-token map made from its writing weights alone, so attention patterns, biases, layer norms, and later layers can be ignored when naming what the head does.

Editorial extensions

If this is right

  • Given any operation expressible as token pairs, MAPS produces a complete head-by-head map of where the model implements it, for every layer and head, in a single pass over parameters.
  • Heads flagged as implementing a relation are causally load-bearing for tasks that need it: ablating them drops accuracy much more than ablating random heads.
  • The same reading method recovers previously known name-mover and capital heads and finds dozens of additional same-function heads that circuit analyses missed.
  • Function encoding shows regularities across models: similar relations get similar top scores, grouped-query heads in Llama-3.1 share functions, and smaller models pack more relations per head.
  • Most attention heads in middle and upper layers admit an automatic natural-language description of their salient operation, as judged against human annotations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If parameter-only inference is as faithful as the correlations suggest, it becomes a practical tool for auditing large models: mapping every head's operations without the cost of corpus-scale inference.
  • Because the framework ignores the query-key attention matrix, an obvious extension is to condition inferred functions on attention: the same $M$ could be paired with $W_{QK}$ to predict when a head switches operation by context.
  • The saliency-guided pipeline could serve as an unsupervised circuit-discovery step, generating candidate functions for heads before any task-specific probing.
  • The relation datasets are limited to single-token pairs; extending to multi-token and cross-lingual relations is the natural stress test of the framework's expressiveness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces MAPS, a framework that infers the functionality of attention heads directly from their parameters by interpreting the head's output projection as a vocabulary-to-vocabulary interaction matrix M = E(W_VO)U. For a predefined relation R, MAPS computes a static relation score phi_R(M) as the fraction of source-target pairs for which the target appears among the top-k tokens in the corresponding row of M (Eq. 2). For open-ended analysis, it proposes a saliency score (Eq. 3) to identify the tokens on which a head operates most prominently, then uses an LLM to describe the resulting salient mappings. The paper validates the static scores in two ways: correlations with dynamic head outputs during inference (Eq. 4) and causal ablation experiments in which removing the heads identified by MAPS degrades task accuracy more than removing random heads. It also reports multi-token generalization, comparisons with previously identified heads, and a human study of the automatic descriptions. The experiments span 6 LLMs and 20 relations across four categories, and the authors release code and computed mappings.

Significance. If the central claim is robust, MAPS would be a useful and efficient tool for mechanistic interpretability: it avoids training or inference and scales vocabulary-projection analysis from specific circuits to whole models. The paper's strengths are its breadth (6 models, 20 relations), the combination of correlative and causal validation, the multi-token generalization check, the comparison with previously known heads, and the release of code and data. The human study for the salient-operation pipeline is also a concrete step beyond anecdotal evaluation. However, the main significance depends on whether the reported results are stable properties of head parameters or artifacts of the particular choices of M, k, and tau, which the current manuscript does not yet establish.

major comments (2)
  1. [§4.1, §A.2] The relation score in Eq. (2) is evaluated under a protocol that varies the definition of M and the scoring hyperparameters across models and relations: the first MLP is applied to E for most models but explicitly not for Llama-3.1 70B because 'better results' were observed (§A.2); k is set to 1 or 3 for copying relations and 10 or 25 for other relations (§4.1); and the threshold tau=15% is fixed after comparison with randomly initialized heads (§A.3). Because the same selected protocol is then used in the correlative and causal validations, the reported correlations of 0.71–0.95 and the causal drops do not yet establish that the inference is a stable property of head parameters; they may reflect the selected configuration. Please add a sensitivity analysis that varies k, tau, and the M variant (with and without first MLP, and comparing E(W_VO)U with E(W_VO)E^T) under a fixed protocol, and report how the correlations and the head rankings change.
  2. [§4.2, Eq. (4)] The non-contextualized dynamic score is not a fully independent test of the static score. Both the static and dynamic computations use the same relation pair dataset D_R, the same top-k operation over vocabulary projections, and closely related matrices: the static matrix is E(W_VO)U, while the dynamic score projects a self-attended head output through U. High correlation in this condition is therefore partly measuring internal consistency of the projection rather than independent behavioral evidence. The contextualized condition, the multi-token experiment (§4.3), and the causal ablation provide more independent support and should be presented as the primary validation, or the authors should add a comparison in which the dynamic score is computed with a different projection that includes layer norms and bias terms.
minor comments (4)
  1. [Abstract and Introduction] There is a typo in the Introduction: 'Predefined relationsignify groups of mappings' should be 'Predefined relations signify groups of mappings'.
  2. [Tables 6–7 and Figures 5, 9–12] The word 'Lingustic' appears in several figure labels and table headers; it should be 'Linguistic'.
  3. [Table 2] The main-text causal table omits standard deviations, which are deferred to the appendix; at least ranges or a note that full variance is in §B.2 would help the reader judge the stability of the random-head baseline.
  4. [§A.3] The random-baseline experiment for tau is reported only qualitatively (e.g., 5 random matrices for GPT-2 xl); a statistical comparison of the random and trained classification rates would strengthen the justification for the threshold choice.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; MAPS is validated against external dynamic head outputs and causal ablations, with only minor self-citation that is not load-bearing.

full rationale

The paper's central claim is that the static relation score phi_R(M) in Eq. 2, computed from M = E(WVO)U, reflects the head's actual behavior. The correlative experiment in Section 4.2 compares this static score to a dynamic score phi*_R(h) in Eq. 4, which is computed from real head outputs projected by U. These are not identical by construction: the dynamic score uses the head's output on prompted inputs (with attention optionally restricted to self), which depends on the actual residual stream and the WV bias term, whereas the static score uses the parameter-only matrix M, optionally with the first MLP applied to E. The two scores share the same top-k operator, the same relation-pair dataset, and the same unembedding projection, so some correlation is expected, but the dynamic quantity is empirically measured from model inference rather than derived from the static formula. The causal experiment in Section 4.2 is genuinely external: heads are selected by the static score and then ablated, and the resulting accuracy drops on held-out task prompts are compared against random-head ablations and control tasks. The comparisons to previously known heads (Wang et al., 2023; Merullo et al., 2024a) provide additional external anchoring. The main self-citations (Geva et al., 2021, 2022; Dar et al., 2023) justify the vocabulary-space formulation, but this premise is not the sole support for the paper's conclusions because of the independent dynamic and causal validations. The selection of k, tau, and whether to apply the first MLP to E is a methodological-robustness concern and could mean the reported correlations are optimistic, but no equation in the paper reduces a prediction to its fitted input, so this does not constitute circularity under the stated criteria.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are introduced. The main external assumptions are the linear logit-lens approximation and the restriction to single-token mappings. Free parameters k, tau, salient counts, and the per-model MLP choice are tuned on data and affect every reported relation score.

free parameters (4)
  • top-k window k = 1 or 10 for GPT-2/Pythia/Phi-2; 3 or 25 for Llama-3.1 depending on relation
    Chosen by the authors after inspecting tokenizer behavior; affects every relation score phi_R and thus all head classifications.
  • classification threshold tau = 0.15
    Selected after 'experimenting with different thresholds and comparing against randomly initialized heads' (Section 4.1, A.3); used to label heads as implementing a relation.
  • salient-token count k and mapping count n = k=30, n=5
    Hand-chosen in Section 5.1 for the automatic GPT-4o pipeline; affects which mappings are described and thus the inferred salient operations.
  • first-MLP application decision = Applied for all models except Llama-3.1 70B
    The authors report 'observed better results' without the first MLP for Llama-3.1 70B (Appendix A.2), making this a per-model post-hoc choice that changes the M matrix.
assumptions (5)
  • domain assumption The linear projection to vocabulary space, M = E(W_VO)U, faithfully approximates how the head's output affects the final logits.
    Adopted from Elhage et al. (2021) and Dar et al. (2023) in Section 2, and acknowledged in Limitations to ignore biases and nonlinearities.
  • domain assumption Attention-head functionality can be expressed as single-token to single-token mappings in the vocabulary space.
    Section 3 and Limitations state that idioms, positional features, and operations requiring multi-token context are outside the framework's expressivity.
  • domain assumption If the target token t is among the top-k tokens in row m_s of M, the head implements the relation from s to t.
    This is the definition of relation score phi_R in Eq. 2; it equates promoting a token in the projected matrix with implementing an operation.
  • ad hoc to paper The saliency score sigma_t = ||e_t W_VO|| / ||e_t|| identifies the tokens on which the head's operation is most prominent.
    Proposed in Eq. 3 without independent validation; used to choose which token rows the GPT-4o pipeline examines.
  • domain assumption GPT-4o pattern descriptions correspond to observable patterns in the salient mappings.
    The human study (Section 5.2, E.2) partially supports this, but 87% of the failure cases are hallucinations where GPT-4o described a pattern humans could not see.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inferring Functionality of Attention Heads from their Parameters." pith.science (2026). https://pith.science/paper/CKLMIZ6R

@misc{pith2026241211965,
  author       = {Pith},
  title        = {Pith review of: Inferring Functionality of Attention Heads from their Parameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKLMIZ6R}},
  note         = {Machine review of arXiv:2412.11965}
}
read the original abstract

Attention heads are one of the building blocks of large language models (LLMs). Prior work on investigating their operation mostly focused on analyzing their behavior during inference for specific circuits or tasks. In this work, we seek a comprehensive mapping of the operations they implement in a model. We propose MAPS (Mapping Attention head ParameterS), an efficient framework that infers the functionality of attention heads from their parameters, without any model training or inference. We showcase the utility of MAPS for answering two types of questions: (a) given a predefined operation, mapping how strongly heads across the model implement it, and (b) given an attention head, inferring its salient functionality. Evaluating MAPS on 20 operations across 6 popular LLMs shows its estimations correlate with the head's outputs during inference and are causally linked to the model's predictions. Moreover, its mappings reveal attention heads of certain operations that were overlooked in previous studies, and valuable insights on function universality and architecture biases in LLMs. Next, we present an automatic pipeline and analysis that leverage MAPS to characterize the salient operations of a given head. Our pipeline produces plausible operation descriptions for most heads, as assessed by human judgment, while revealing diverse operations.

Figures

Figures reproduced from arXiv: 2412.11965 by the authors.

Figure 1
Figure 1. Illustration of MAPS, a framework for infer￾ring the functionality of attention heads in LLMs from their parameters. MAPS casts the head as a matrix M which assigns a score for every pair of tokens in the model’s vocabulary. Then, it considers groups of token pairs (sub-matrices in M) to measure how strongly the head implements a given operation (A) and to inspect the head’s salient operations (B). may overlook some… view at source ↗
Figure 3
Figure 3. Relation scores for all heads of Llama-3.1 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 2
Figure 2. Functionality mapping by MAPS for 20 rela￾tions of 4 categories — algorithmic, knowledge, linguis￾tic, translation — across all attention heads in GPT-2 xl, Phi-2, Pythia 12B, Llama-3.1 70B. A head is marked as a specific category if it implements at least one relation from this category. tative relations in multiple models showing two interesting trends. First, despite architecture and training data differences, mo… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Portion of heads where GPT-4o identified a [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Functionality mapping by MAPS. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Comparison between relation heads discovered by MAPS and heads classified in prior works. [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: First part of human annotation instructions. [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: Second part of human annotation instructions. [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 10
Figure 10. Figure 10: Quality of GPT-4o interpretation (§E) - Hu￾man annotation distribution for Question 2. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 12
Figure 12. Figure 12: Input skewness versus output space size for [PITH_FULL_IMAGE:figures/full_fig_p033_12.png]
Figure 11
Figure 11. Figure 11: Quality of GPT-4o interpretation (§E) - Hu￾man annotation distribution for Question 3. 0 20 40 layer 0.0 0.5 1.0 1.5 2.0 Input Skewness GPT2 xl 0 15 30 layer 0.0 0.5 1.0 1.5 2.0 Input Skewness Pythia 6.9b 0.0 0.1 0.2 0.3 0.4 Output Space Size Specific head Global head…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

    cs.CV 2025-11 conditional novelty 7.0 of 10

    Supervising an MLLM's intrinsic self-attention with patch-level GUI labels, aggregated via a learnable anchor token and hidden-state-selected query tokens, reaches state-of-the-art 3B-scale GUI grounding accuracy with...

  2. Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Over 40 checkpoints of OLMo-7B, attention heads and FFNs shift from general-purpose to specialized roles for factual recall, with location-based facts learned earlier and more stably than name-based facts.

  3. Understanding Gated Neurons in Transformers from Their Input-Output Functionality

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Across 12 language models, neurons in early-middle layers tend to add the direction they detect (enrichment), while later layers tend to reduce it (depletion), based on input-output weight cosine similarity.

Reference graph

Works this paper leans on

51 extracted references · 11 canonical work pages · cited by 3 Pith papers

  1. [1]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.298 GQA : Training generalized multi-query transformer models from multi-head checkpoints . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895--4901, Si...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  2. [2]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717

    arXiv 2024

  3. [3]

    Amos Azaria and Tom Mitchell. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.68 The internal state of an LLM knows when it`s lying . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967--976, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.68 2023

  4. [4]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. https://proceedings.mlr.press/v202/biderman23a.html Pythia: A suite for analyzing large language models across trai...

    work page 2023

  5. [5]

    Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037

    arXiv 2022

  6. [6]

    Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Vi \'e gas, and Martin Wattenberg. 2021. https://arxiv.org/abs/2104.07143 An interpretability illusion for bert . ArXiv preprint, abs/2104.07143

    arXiv 2021

  7. [7]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/W19-4828 What does BERT look at? an analysis of BERT ' s attention . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276--286, Florence, Italy. Association for Computational Linguistics

    doi:10.18653/v1/w19-4828 2019

  8. [8]

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. https://doi.org/10.18653/v1/2023.acl-long.893 Analyzing transformers in embedding space . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16124--16170, Toronto, Canada. Association for Computational Linguistics

    doi:10.18653/v1/2023.acl-long.893 2023

Show all 51 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . ArXiv preprint, abs/2407.21783

    2024 arXiv

  2. [10]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12

    2021

  3. [11]

    Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-juss \`a . 2024. https://arxiv.org/abs/2405.00208 A primer on the inner workings of transformer-based language models . ArXiv preprint, abs/2405.00208

    2024 arXiv

  4. [12]

    Leo Gao, Tom Dupr \'e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. https://arxiv.org/abs/2406.04093 Scaling and evaluating sparse autoencoders . ArXiv preprint, abs/2406.04093

    2024 arXiv

  5. [13]

    Trujillo

    Jorge Garc \' a - Carrasco, Alejandro Mat \' e , and Juan C. Trujillo. 2024. https://proceedings.mlr.press/v238/garcia-carrasco24a.html How does GPT-2 predict acronyms? extracting and understanding a circuit via mechanistic interpretability . In International Conference on Art...

    2024

  6. [14]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natural L...

    2022 doi

  7. [15]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446 Transformer feed-forward layers are key-value memories . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484--5495, ...

    2021 doi

  8. [16]

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. https://openreview.net/forum?id=5uwBzcn885 Patchscopes: A unifying framework for inspecting hidden representations of language models . In Forty-first International Conference on Machine Learning

    2024

  9. [17]

    Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. 2024. https://openreview.net/forum?id=kvcbV8KQsi Successor heads: Recurring, interpretable attention heads in the wild . In The Twelfth International Conference on Learning Representations

    2024

  10. [18]

    Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. 2025. Enhancing automated interpretability with output-centric feature descriptions. arXiv preprint arXiv:2501.08319

    2025 arXiv

  11. [19]

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. https://openreview.net/forum?id=w7LU2s14kE Linearity of relation decoding in transformer language models . In The Twelfth International Conference...

    2024

  12. [20]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . ArXiv preprint, abs/2410.21276

    2024 arXiv

  13. [21]

    Mojan Javaheripi and Sébastien Bubeck. 2023. https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ Phi-2: The surprising power of small language models

    2023

  14. [22]

    Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.142 Backward lens: Projecting language model gradients into the vocabulary space . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin...

    2024 doi

  15. [23]

    Geonhee Kim, Marco Valentino, and Andr \'e Freitas. 2024. https://arxiv.org/abs/2408.08590 A mechanistic interpretation of syllogistic reasoning in auto-regressive language models . ArXiv preprint, abs/2408.08590

    2024 arXiv

  16. [24]

    Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. 2024. https://openreview.net/forum?id=fewUBDwjji Interpreting attention layer outputs with sparse autoencoders . In ICML 2024 Workshop on Mechanistic Interpretability

    2024

  17. [25]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://openreview.net/forum?id=VD-AYtP0dve Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations

    2023

  18. [26]

    Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. 2015. https://proceedings.mlr.press/v44/li15convergent.html Convergent learning: Do different neural networks learn the same representations? In Proceedings of the 1st International Workshop on Feature Extra...

    2015

  19. [27]

    Edward Loper and Steven Bird. 2002. https://doi.org/10.3115/1118108.1118117 NLTK : The natural language toolkit . In Proceedings of the ACL -02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics , pages 63--70, ...

    2002

  20. [28]

    Callum Stuart McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.22 Copy suppression: Comprehensively understanding a motif in language model attention heads . In Proceedings of the 7th BlackboxNLP Worksh...

    2024 doi

  21. [29]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html Locating and editing factual associations in GPT . In Advances in Neural Information Processing System...

    2022

  22. [30]

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024 a . https://openreview.net/forum?id=fpoAYV6Wsk Circuit component reuse across tasks in transformer language models . In The Twelfth International Conference on Learning Representations

    2024

  23. [31]

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2024 b . https://openreview.net/forum?id=LUsx0chTsL Talking heads: Understanding inter-layer communication in transformer language models . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  24. [32]

    Beren Millidge and Sid Black. 2022. https://www.alignmentforum.org/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight The singular value decompositions of transformer weight matrices are highly interpretable

    2022

  25. [33]

    Neel Nanda and Joseph Bloom. 2022. Transformerlens. https://github.com/TransformerLensOrg/TransformerLens

    2022

  26. [34]

    nostalgebraist. 2020. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens Interpreting gpt: the logit lens

    2020

  27. [35]

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. https://arxiv.org/abs/2209.11895 In-context learning and induction heads . ArXiv preprint, abs/2209.11895

    2022 arXiv

  28. [36]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  29. [37]

    Schwarte

    Patrick Schober, Christa Boer, and Lothar A. Schwarte. 2018. https://api.semanticscholar.org/CorpusID:13354506 Correlation coefficients: Appropriate use and interpretation . Anesthesia & Analgesia, 126:1763–1768

    2018

  30. [38]

    Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. 2024. https://openreview.net/forum?id=3Ds5vNudIE LLM circuit analyses are consistent across training and scale . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  31. [39]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neural I...

    2017

  32. [40]

    Jesse Vig and Yonatan Belinkov. 2019. https://doi.org/10.18653/v1/W19-4808 Analyzing the structure of attention in a transformer language model . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63--76, Florence, It...

    2019 doi

  33. [41]

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. https://doi.org/10.18653/v1/P19-1580 Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned . In Proceedings of the 57th Annual Meeting of the Associatio...

    2019 doi

  34. [42]

    Denny Vrande c i\' c and Markus Kr\" o tzsch. 2014. https://doi.org/10.1145/2629489 Wikidata: a free collaborative knowledgebase . Commun. ACM, 57(10):78–85

    2014 doi

  35. [43]

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. https://openreview.net/pdf?id=NpsVSN6o4ul Interpretability in the wild: a circuit for indirect object identification in GPT-2 small . In The Eleventh International Conference on Lear...

    2023

  36. [44]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  37. [45]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. https://openreview.net/forum?id=NG7sS51zVF Efficient streaming language models with attention sinks . In The Twelfth International Conference on Learning Representations

    2024

  38. [46]

    Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2024. https://aclanthology.org/2024.lrec-main.840 Jump to conclusions: Short-cutting transformers with linear transformations . In Proceedings of the 2024 Joint International Conference on Computational Linguistic...

    2024

  39. [47]

    Lei Yu, Meng Cao, Jackie CK Cheung, and Yue Dong. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.466 Mechanistic understanding and mitigation of language model non-factual hallucinations . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7...

    2024 doi

  40. [48]

    Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Bo Tang, Feiyu Xiong, and Zhiyu Li. 2024. https://arxiv.org/abs/2409.03752 Attention heads of large language models: A survey . ArXiv preprint, abs/2409.03752

    2024 arXiv

  41. [49]

    Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, Kun Wang, Yang Liu, Junfeng Fang, and Yongbin Li. 2024. https://arxiv.org/abs/2410.13708 On the role of attention heads in large language model safety . ArXiv preprint, abs/2410.13708

    2024 arXiv

  42. [50]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.