Pith. sign in

REVIEW 3 major objections 6 minor 48 references

Reversed Attention: On The Gradient Descent Of Attention Layers In GPT

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the softmax derivative in GPT attention, called Reversed Attention, is an implicit attention map that reveals how gradient descent would edit attention, and that its norms rank head importance while averaged maps can…

desk verdict RA is a real and useful object with two new applications, but the batched formula in Eq. 17 does not compute what it claims and the empirical support needs robustness work before the central claims can be trusted. read the letter →

arxiv 2412.17019 v1 pith:2ZAVAUDE submitted 2024-12-22 cs.CL

Shahar Katz , Lior Wolf This is my paper · ORCID
classification cs.CL
keywords reversedattentiongradientsbackwardpassinterpretabilitycausalmediationpatchingGPTsoftmaxderivative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the softmax derivative computed during the backward pass of GPT attention is not just an engineering artifact but an implicit attention-like matrix, which they call Reversed Attention (RA). RA shows, for a chosen loss target, which token pairs a head would amplify or suppress if gradient descent updated its weights. The paper argues that this makes RA a cheap interpretability signal: RA norms rank attention heads by importance nearly as well as causal mediation, and injecting averaged RA maps into the forward pass ('attention patching') nudges a frozen model toward task behavior without any weight change. A sympathetic reading is that RA offers a single-backward-pass window into how GPTs would edit their own attention, with applications to localization and intervention.

What carries the argument

The central object is the Reversed Attention (RA) matrix $R$, the derivative of the softmax attention scores with respect to their logits, assembled from per-token VJPs. Concretely, $R = A \odot (\tilde{E}^\top - \mathrm{diag}(A\tilde{E}^\top))^\top \sqrt{h/d}$ with $\tilde{E} = \Delta \hat{W}_o^\top V^\top$, where $A$ is the forward attention, $\Delta$ holds the output-projection VJPs, $\hat{W}_o$ is the output projection, and $V$ holds the value vectors. Because $A$ is lower-triangular (causal masking), $R$ is lower-triangular too, and $R$ plays the role of an attention matrix in the backward pass: it multiplies forward keys $K$ to form query VJPs and transposed rows multiply forward queries $Q$ to form key VJPs. The paper's argument is that this matrix is what gradient descent would edit if it took a step: positive entries push the corresponding query and key together under a negative learning rate, negative entries push them apart.

What would settle it

Run attention patching on a test set whose prompt lengths or templates differ from the 25 training examples: if the accuracy gains vanish (or degrade to the no-patch baseline), the premise that averaged RA maps are task-level editing instructions fails. Alternatively, compare the signed entries of $R$ for a single head against the actual change in forward attention after one genuine gradient step on that head; if positive RA entries do not correspond to increased attention under a negative learning rate, RA is not what gradient descent would do.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the derivative of the softmax operation inside each GPT attention head, $R = A \odot (\tilde{E}^\top - \mathrm{diag}(A\tilde{E}^\top))^\top \sqrt{h/d}$, is a triangular, attention-like matrix whose rows are the VJPs that gradient descent would use to update the query and key projections. Positive entries of $R$ mark query–key pairs whose attention score GD would increase (for a negative learning rate) and negative entries mark pairs it would decrease. The paper demonstrates two consequences: the per-head norm of $R$ ranks the importance of attention heads for a given prediction, competitive with causal mediation in perturbation tests; and the average of $R$ over 25 same-format examples, injected additively into the forward attention with a negative learning rate, improves task accuracy on several ICL and fact-relation tasks without touching weights. The authors present RA as what the backward pass is 'really doing' when a GPT is asked to move its prediction toward a target token.

Load-bearing premise

RA maps averaged over 25 labeled examples of one fixed template (same length and format) are assumed to generalize to held-out examples of that task; the paper tests this only on simple, templated ICL and fact-relation tasks, without variance estimates or heterogeneous inputs.

Editorial extensions

If this is right

  • RA norm per head provides an ordering of attention-head importance that performs competitively with causal mediation in perturbation tests, at the cost of one forward and one backward pass instead of one forward pass per head.
  • Averaging RA maps over a small set of same-length, same-format examples and adding them into the forward attention of a frozen model improves accuracy on several ICL tasks, sometimes matching few-shot prompting without showing any examples.
  • RA maps are sparse and target-specific: changing the editing target changes which token pair a head amplifies, suggesting the backward pass localizes task-relevant information per inference.
  • Because close-to-zero RA scores yield near-zero VJPs and gradients, focusing on high-RA-normal heads captures most of the gradient-descent update.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implicit extension is that RA could serve as a general-purpose attribution method for any differentiable attention-based model, not just decoder-only GPT; the derivation only relies on the structure of softmax attention, so encoder-decoder or bidirectional attention would produce analogous matrices.
  • RA patching's reliance on fixed-length templates is a constraint the paper acknowledges; a natural extension is to align RA maps across variable-length inputs (e.g., by position or by attention heads' token roles), which would let the method apply to open-ended generation.
  • A direct test the paper does not run is whether the signed pattern of $R$ predicts the actual change in forward attention after one real gradient step on a head; if the two signs systematically disagree, RA would be an interpretability heuristic rather than an exact editing plan.
  • Because RA is computed from a loss that selects a target token, it is essentially a gradient-based saliency map for the attention sublayer; combining RA norms with gradient×input-style scaling could yield even sharper head rankings than the plain norm.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the backward pass of attention layers in GPT models. It derives the vector-Jacobian products for the attention weight matrices and identifies the derivative of the attention softmax, denoted R and termed "Reversed Attention" (RA), as an implicit attention map. The authors argue that RA encodes how gradient descent would edit attention, and they use RA norms to rank attention heads in a perturbation test and to perform "attention patching," where averaged RA maps are injected into forward attention to steer predictions without weight updates. Experiments are conducted on GPT2-xl, OPT, GPT-j, and Llama2-7B across in-context learning and fact-relation tasks.

Significance. If the central derivation is corrected, the paper offers a cheap, single-backward-pass signal for attention-head localization and a no-weight-update intervention mechanism, which are attractive properties for interpretability research. The scalar softmax-derivative derivation (Eqs. 12-15 and 18-19) is correct, and the perturbation test provides an external benchmark that does not rely on the RA definition itself. The paper also compares across multiple models and tasks, which strengthens the empirical picture. However, the batched definition of R in Eq. 17 is mathematically inconsistent with the scalar formula, so the paper as written does not actually define the object it claims to study. The relationship between attention patching and a single gradient descent step on attention logits also deserves explicit recognition, since it affects how novel and how explanatory the intervention is.

major comments (3)
  1. [Section 4, Eq. (17)] Equation (17) is not equivalent to the row-wise formula in Eq. (15), and the statement "hence R_j = r_j" is false. In Eq. (17), the term diag(A tilde E^T) subtracts the scalar c_j = sum_k A_{j,k} tilde E_{j,k} only from the diagonal entry of row j after the transposition, yielding row j of R as A_j ⊙ tilde E_j - c_j A_{j,j} e_j. The correct softmax derivative requires subtracting c_j from every entry of the row, i.e., A_j ⊙ (tilde E_j - c_j 1). To make this concrete, for A = [[0.5,0.5],[0,1]] and tilde E = [[a,b],[c,d]], Eq. (15) gives row 1 = [0.25(a-b), -0.25(a-b)], while Eq. (17) gives [0.25(a-b), 0.5b]. This is a load-bearing error: any reader implementing RA from the paper's formal definition will compute a different object from the softmax derivative. The authors must correct the batched formula (e.g., by using a row-wise subtraction of the vector of row sums of A ⊙ tilde E) or explicitly state that all experiments use the scalar/autograd form.
  2. [Section 5.3] The attention-patching method injects a multiple of the RA map into the forward attention scores. Since R is the derivative of the loss with respect to the attention logits (up to the scaling factor), adding λ R with negative λ is exactly a gradient descent step on the attention logits. The paper should state this equivalence and discuss its consequences. In particular, the success of attention patching should be framed as evidence that a first-order approximation of a loss-minimizing edit can steer the model, rather than as independent evidence that RA reflects the model's "desired attention" in a broader sense. This does not invalidate the results, but it is central to the interpretation and novelty of contribution (v).
  3. [Sections 5.2-5.3 and Appendices B-C] The empirical validations assume that RA maps averaged over 25 examples from a single task template are stable and transferable to held-out examples. The paper does not report variance across example subsets or across random seeds, and the patching results depend on a fixed learning rate of -30 without a sensitivity analysis. Since the head-ranking and patching claims rest on this transferability, the authors should provide error bars or multiple-split results for the AUC values in Tables 1-12 and a learning-rate sensitivity study for the patching results in Table 13.
minor comments (6)
  1. [Section 4, Eqs. (8) and (10)] The outer-product order in the update equation for W_o is inconsistent with the surrounding notation: Eq. (9) writes δ_o × x_o^T, but the derivation of Eq. (11) requires x_o^T × δ_o. Please correct the notation.
  2. [Section 4, text after Eq. (16)] The sentence "In Equation 16 we defined R" is inaccurate; Eq. (16) defines tilde E, while R is defined in Eq. (17).
  3. [Tables 1 and 2] The method name is misspelled as "Casual Mediation" in several table headers; it should be "Causal Mediation."
  4. [Section 5.3 and Appendix C] Using the term "learning rate" for the scalar that scales the injected RA map is potentially confusing, since no parameter update is performed; consider renaming it "injection strength" or "patching coefficient."
  5. [Section 3 and Limitations] The paper describes the analysis as "assumption-free full GPT models," but it omits LayerNorm and, in later experiments, uses models with RoPE. The Limitations section acknowledges some of this, but the earlier claim should be softened to avoid overstatement.
  6. [Appendix B] The perturbation test uses AUC as the summary statistic, but the tables do not report standard deviations or significance tests. Given that some differences between methods appear small, this makes it hard to judge whether RA's advantage over CM is reliable.

Circularity Check

1 steps flagged · score 4.0 of 10

Attention patching is a near-tautological gradient-step validation, but the core head-ranking claim rests on an independent perturbation benchmark.

  1. fitted input called prediction [Section 5.3 (Attention patching) and Appendix C (Attention Patching), building on Eq. 15/17 in Section 4.]
    "In Equation 16 we defined R as the softmax derivative. ... For each test example and for each attention head, we add (inject) the RA map to the forward pass attention map, using a learning rate as a scaling factor. ... If we inject forward pass maps, a successful learning rate is a positive number. For reversed attention, a negative learning rate would improve the model."

    RA is defined as the VJP of the loss through the attention softmax, i.e., the gradient of the loss w.r.t. the pre-softmax attention logits. Patching averages RA over 25 training examples and injects it into the forward attention with a negative learning rate, which is at least an approximate gradient-descent step on the same loss that was used to define RA. The observed improvement ('a negative learning rate would improve the model') is therefore the expected consequence of moving along the loss gradient, not an independent confirmation that RA is an interpretable 'desired attention' map. The held-out transfer is not guaranteed, so the reduction is partial, and the perturbation benchmark in Section 5.2 provides independent support for the head-ranking claim.

full rationale

The mathematical derivation of RA from the softmax VJP is a standard chain-rule computation and is not circular by itself. The head-ranking validation in Section 5.2 uses an external perturbation benchmark (AUC of progressively unmasking heads) and does not reduce to the definition of RA, so the central interpretability claim has independent content. The one partially circular element is attention patching in Section 5.3/Appendix C: RA is defined as the loss derivative, and injecting it with a negative learning rate is close to taking a gradient step on that same loss, making the patching success a near-tautological validation rather than independent evidence. Because the injection is described as acting on the forward attention map (not exactly the logits) and the transfer to held-out examples is empirical, I score this as partial circularity rather than full reduction. No load-bearing self-citation was found; the Katz et al. (2024) reference in Related Work is contextual. The Eq. 17 batched-formula discrepancy is a mathematical correctness issue, not a circularity, and is not scored here.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The mathematical contribution consists of restating standard backpropagation through attention and naming the softmax Jacobian as RA. The only hand-set scalars are the patching learning rates and the number of averaging examples; the masking and loss choices are standard domain assumptions.

free parameters (3)
  • RA attention patching learning rate = -30
    Chosen by hand and applied uniformly to all models and tasks; scales the injected RA map. The paper notes different rates change performance and that they deliberately did not tune per task.
  • FA attention patching learning rate = 1
    Baseline scaling for forward-attention patching, set by hand.
  • Number of examples for averaging RA maps = 25
    Used in both perturbation and patching experiments to compute average RA norms and maps; chosen by the authors, not justified by convergence analysis.
assumptions (4)
  • standard math Backpropagation and the Jacobian of the softmax function
    Section 4 derives VJPs and gradients using the chain rule and the standard identity for the derivative of softmax.
  • domain assumption Causal, lower-triangular attention mask
    Equation 3 restricts attention to previous tokens; the triangularity and causality of RA follow from this mask.
  • domain assumption LayerNorm omitted from the mathematical walk-through
    Section 3 states LayerNorm is omitted because it contributes little to gradients; this is an assertion, not derived, though experiments use autograd on full models.
  • domain assumption Next-token cross-entropy loss with a chosen target token
    RA is computed by backpropagating a specific target answer through the loss; the VJP at the attention output, delta_o, is determined by this loss choice.
invented entities (1)
  • Reversed Attention (RA) matrix R independent evidence
    purpose: Interpretability and intervention: RA norms rank attention heads by importance, and averaged RA maps are injected into the forward pass to alter predictions (attention patching).
    R is explicitly defined as the softmax derivative and can be computed from any GPT backward pass; the paper provides falsifiable applications (ranking, patching) that other researchers can reproduce, though the current evidence is proof-of-concept.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reversed Attention: On The Gradient Descent Of Attention Layers In GPT." pith.science (2026). https://pith.science/paper/2ZAVAUDE

@misc{pith2026241217019,
  author       = {Pith},
  title        = {Pith review of: Reversed Attention: On The Gradient Descent Of Attention Layers In GPT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZAVAUDE}},
  note         = {Machine review of arXiv:2412.17019}
}
read the original abstract

The success of Transformer-based Language Models (LMs) stems from their attention mechanism. While this mechanism has been extensively studied in explainability research, particularly through the attention values obtained during the forward pass of LMs, the backward pass of attention has been largely overlooked. In this work, we study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as "Reversed Attention". We examine the properties of Reversed Attention and demonstrate its ability to elucidate the models' behavior and edit dynamics. In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model's weights, using a novel method called "attention patching". In addition to enhancing the comprehension of how LM configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass.

Figures

Figures reproduced from arXiv: 2412.17019 by the authors.

Figure 1
Figure 1. In this paper we examine the attention maps [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The norms of the attention maps per head and per layer. (b) Forward and (c) Reversed Attention of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. RA model editing dynamics. (a) The query matrix Wˆ q will be updated with a VJP directed towards the forward pass key of “tomato”, while the key matrix Wˆ k will be updated with a VJP directed towards the query from the token “of”. (b) The latent space of the queries and keys. The circles represent a forward pass query and a key. If their Reversed Attention score is a relatively low negative number, the directions t… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Attention patching using Reversed Attention (RA): first we collect the RA maps of the model without [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The forward and Reversed Attention (RA) maps of an attention head from GPT2-small (layer 11, head index 2), given the editing target “cherry” with the prompt “Cherry tomato is a type of”. The pattern pre￾sented by the RA map attempts to increase the forward pass attent…
Figure 6
Figure 6. Figure 6: The forward and Reversed Attention maps of GPT2-xl, given the editing target “Paris” and the prompt “I [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Attention maps for the prompt “I like Italy and France. I visited the city of” (head 8, layer 30). While the [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The forward and Reversed Attention (RA) of OPT-350m, given the editing target “Paris” and the prompt [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Attention maps for the prompt “I like Italy and France. I visited the city of”. The head with the highest [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: GPT2-xl casual mediation and Reversed Attention maps for the ICL capitalize task with 5-shots. \\Q: <question2> \A: <answer2> \\Q: <question> \A:”. The natural language questions we utilized are from (Hernandez et al., 2024). This dataset com￾prises tasks, each with a…
Figure 11
Figure 11. Figure 11: Perturbation test results visualized for Llama2-7B on the capitalize ICL task. With no shots, when the [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Forward and Reversed attention patching results are presented as a function of the learning rate, a [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 18 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marco Ancona, Enea Ceolini, Cengiz \"O ztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for deep neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30-May 3, 2018, Conference Track Proceedings. OpenReview. net

    work page 2018

  4. [4]

    Oren Barkan, Edan Hauon, Avi Caciularu, Ori Katz, Itzik Malkiel, Omri Armstrong, and Noam Koenigstein. 2021. Grad-sam: Explaining transformers via gradient self-attention maps. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2882--2887

    work page 2021

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  6. [6]

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2023. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models

    work page 2023

  7. [7]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    arXiv 2022

  8. [8]

    N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. 2021. https://transformer-circuits.pub/2021/framework/index.html A mathematical framework for transformer circuits

    work page 2021

Show all 48 references
  1. [9]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30--45

    2022

  2. [10]

    Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.95 Knowledge is a region in weight space for fine-tuned language models . In Findings of the Association for Computational Linguistics: E...

    2023 doi

  3. [11]

    Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, pages 8342--83...

    2020

  4. [12]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations

    2021

  5. [13]

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. Linearity of relation decoding in transformer language models. In Proceedings of the 2024 International Conference on Learning Representations

    2024

  6. [14]

    Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543--3556

    2019

  7. [15]

    Shahar Katz and Yonatan Belinkov. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.939 VISIT : Visualizing and interpreting the semantic information flow of transformers . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14094--14113, Singap...

    2023 doi

  8. [16]

    Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.142 Backward lens: Projecting language model gradients into the vocabulary space . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin...

    2024 doi

  9. [17]

    Guillaume Lample, Alexis Conneau, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv \'e J \'e gou. 2018. Word translation without parallel data. In International Conference on Learning Representations

    2018

  10. [18]

    Y Le Cun. 1988. A theoretical framework for backpropagation. In Proceedings of the 1988 Connectionist Models Summer School

    1988

  11. [19]

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

    2023

  12. [20]

    Dougal Maclaurin. 2016. Modeling, inference and optimization with composable differentiable procedures. Ph.D. thesis

    2016

  13. [21]

    Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. 2023. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576

    2023 arXiv

  14. [22]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT . Advances in Neural Information Processing Systems, 36

    2022

  15. [23]

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. International Conference on Learning Representations

    2023

  16. [24]

    Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, et al. 2024. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. ar...

    2024

  17. [25]

    Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Distinguishing antonyms and synonyms in a pattern-based neural network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Asso...

    2017

  18. [26]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  19. [27]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training

    2018

  20. [28]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog

    2019

  21. [29]

    Sofia Serrano and Noah A Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931--2951

    2019

  22. [30]

    K Simonyan, A Vedaldi, and A Zisserman. 2014. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR

    2014

  23. [31]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063

    2024

  24. [32]

    Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. 2023. Transformers as support vector machines. arXiv preprint arXiv:2308.16898

    2023 arXiv

  25. [33]

    Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du. 2023. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. arXiv preprint arXiv:2305.16380

    2023 arXiv

  26. [34]

    Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. 2023. Function vectors in large language models. arXiv preprint arXiv:2310.15213

    2023 arXiv

  27. [35]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  29. [37]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33:12388--12401

    2020

  30. [38]

    Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. 2023. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.04827

    2023 arXiv

  31. [39]

    Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model . https://github.com/kingoflolz/mesh-transformer-jax

    2021

  32. [40]

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations

    2023

  33. [41]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  34. [42]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations

    2023

  35. [43]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4800

    2019

  36. [44]

    Fred Zhang and Neel Nanda. 2023. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042

    2023 arXiv

  37. [45]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  38. [46]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  39. [47]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  40. [48]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.