Pith. sign in

REVIEW 5 major objections 5 minor 81 references

From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A transformer concept such as 'safety' or 'reasoning' can be localized to 3-10 attention heads and controlled by one scalar.

desk verdict Useful, honest pipeline paper whose main results are undercut by missing control modules and grid-searched scalars, but worth refereeing for revision. read the letter →

arxiv 2506.17052 v1 pith:PIOYV767 submitted 2025-06-20 cs.LG cs.AIcs.CL

Jingtong Su , Julia Kempe , Karen Ullrich This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords attentionheadattributionconceptvectorssparseautoencodersinterventiontransformerinterpretabilityjailbreakingvisiontransformerscosinesimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an arbitrary concept in a transformer—'French', 'safety', 'reasoning', or an ImageNet label—can be localized to a small set of attention heads by averaging the cosine similarity between a vector representation of the concept and each head's additive contribution to the residual stream. Across all concepts and models tested, the selected module is sparse, containing 3-10 heads, and scaling exactly those heads' contributions by a single scalar weakens or amplifies the concept. The authors report that suppressing the 'safety' module raises attack success on HarmBench by 72.7 points, amplifying the 'reasoning' module improves GSM8K accuracy by about 1.6 points, and scaling down a label module in a vision transformer drives target-label recognition to 0%. If this picture is right, concept-level control over large transformers reduces to knowing which handful of attention heads to rescale, and much of the machinery of activation steering and prompting becomes an indirect way of doing that.

What carries the argument

The load-bearing object is the residual-stream decomposition of a transformer layer into per-head additive contributions $a_{l,h}(p)$, so each head can be assigned a score by the average cosine similarity between $a_{l,h}(p)$ and a concept vector $v_c$ (SAMD, Eq.~3). The top-$K$ heads form the attention module, and Scalar Attention Module Intervention (SAMI) rescales exactly those heads' contributions by one scalar $s$ at the attention output projection (Eq.~4). Because the scalar acts as a coefficient on existing weights, the whole intervention costs no retraining, no prompt edit, and (at inference) only a tiny change to output matrices.

What would settle it

Take the same safety and reasoning modules but select K heads with the lowest cosine similarity instead of the highest, apply SAMI with the same scalar, and compare HarmBench attack success and GSM8K accuracy; if low-similarity heads produce comparable effects, the cosine criterion is not doing the selection work. A stronger version is to replace SAMD with causal patching per head and check whether the heads whose ablation most changes concept output coincide with the top-K cosine heads.

Watch

Extended reading notes

Core claim

The paper's central discovery is that attention-head modules found by top-K cosine similarity between head contributions and a concept vector are sparse and causally controllable: 3-10 heads encode each concept, and multiplying only those heads' output magnitudes by a scalar s produces concept-level behavior changes in both language and vision transformers. The same module locations reappear before and after post-training, supporting the view that concept knowledge is already present in the base model; safety modules concentrate in middle layers, output-language modules in later layers, and image-label modules in final layers. The intervention consistently beats or matches vector-steering baselines while touching about 0.1% of model weights, and it works without prompt changes or optimization.

Load-bearing premise

The load-bearing premise is that higher cosine similarity between a head's contribution and a concept vector reliably identifies the heads that encode that concept; the paper asserts this link from earlier representation work rather than validating it against random heads or causal attribution.

Editorial extensions

If this is right

  • Concept localization becomes a one-forward-pass, training-free operation: average cosine similarities against a concept vector and take the top-K heads.
  • Only about 0.1% of weights—10 heads for safety, 5 for reasoning—need to be rescaled to move benchmark behavior: +72.7 points on HarmBench attack success and about +1.6 points on GSM8K.
  • The same scalar intervention carries over to vision: rescaling 3 heads removes a target ImageNet label's recognizability to 0% while leaving other labels mostly intact.
  • Module stability before and after post-training implies that aligning a model does not rewrite where concepts live; it changes what the model does with them.
  • Because SAMI is equivalent to rescaling output-projection weights, it composes cheaply with existing inference pipelines and can be applied per concept without accumulating model copies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If cosine ranking is genuine, attention heads act like an overcomplete linear basis for concepts; a testable follow-up is whether modules for different concepts overlap, and whether scaling one module measurably moves another.
  • The same pipeline could probe concepts the paper does not include, such as social bias, hallucination tendency, or persona, by supplying the corresponding contrastive dataset; whether 3-10 heads still suffice is open.
  • The paper's own caveat that its modules may be overcomplete or incomplete suggests a minimality test: iteratively drop the lowest-similarity head from the module and check how much intervention strength must grow to keep the same behavioral effect.
  • Because SAMI's scalar is set by grid search, a natural extension is to make the module scale a learned parameter during fine-tuning, freezing the discovered heads and optimizing s directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces Scalable Attention Module Discovery (SAMD), a method that represents a concept as a vector and scores each attention head by the average cosine similarity between that head's output contribution and the concept vector over a positive dataset; the top-K heads form the 'concept module.' It then proposes Scalar Attention Module Intervention (SAMI), which multiplies the output magnitude of the selected heads by a scalar s to diminish or amplify the concept. Experiments cover SAE-derived concepts on Gemma-2-2B(-IT) (including a French-language module evaluated on FQuAD), a reasoning module evaluated on GSM8K with Llama-3.1-8B-Instruct and Gemma-7B-Base, a safety module evaluated on HarmBench with three aligned LLMs, and label-recognition modules in a ViT-B/32 evaluated on ImageNet. The paper claims that only 3-10 attention heads encode these diverse concepts, that modules are stable across post-training, and that SAMI can jailbreak (+72.7% ASR), improve GSM8K (+1.6%), and suppress ViT target-label accuracy to 0%.

Significance. If the central claim is correct, SAMD would be a valuable, low-cost attribution tool: it requires a single forward pass per input, operates on arbitrary concepts across LLMs and ViTs, leverages public SAE features, and produces sparse modules that are amenable to a simple scalar intervention. The paper's strengths include its cross-domain scope, the use of open libraries (TransformerLens, ViT-Prisma), and the concrete qualitative demonstrations that negative intervention on the discovered 'French' module switches output language and that positive intervention on a 'dog' module causes repetition of dog-related content. However, the load-bearing quantitative claims are currently undermined by the absence of control interventions and by grid-search-based scalar selection on the evaluation benchmarks, so the causal interpretation of the modules is not yet established. The paper also overstates the HarmBench comparison for one model and internally concedes in Appendix E that causality and stability are not analyzed.

major comments (5)
  1. [Sections 4.1-4.4, Eq. (3)-(4)] The paper never compares the top-K cosine-selected modules with control modules (e.g., random sets of K heads or bottom-K heads) under the same SAMI protocol. Because SAMI changes the magnitude of K attention-head contributions by a scalar, a generic perturbation on any K heads may produce large behavioral changes; without such controls, the headline results (HarmBench 71.1% ASR, GSM8K +1.6%, FQuAD 85.35% to 1.66%, ViT target-label 0%) do not establish that Eq. (3) selects concept-encoding heads. Please add random-head and bottom-K baselines with the identical scalar search and report the comparison.
  2. [Sections 4.2-4.3, Tables 1-2] The intervention scalars for the central quantitative results (s=1.4/1.2 for GSM8K and s=-1.7/-0.7/-0.8 for HarmBench) are selected by grid search on the same benchmark on which the results are reported, with no held-out split or explicit selection rule. The reported improvements are therefore maxima over the searched grid rather than out-of-sample predictions. Please either fix the scalars on a validation set and report test performance, or present the full sweep curves together with the selection protocol.
  3. [Section 4.3, Table 2] The text asserts that the safety module intervention is 'more powerful than the vector-based ORTHO intervention,' but for Qwen7B the safety module reaches 78.0% ASR versus 79.2% for ORTHO. The claim as stated is contradicted by the table; please qualify the comparison (e.g., by model) or correct the text.
  4. [Section 4.2, Table 1] The GSM8K gains (+0.83 and +2.35 percentage points for the two models) are reported without variance estimates, multiple evaluation seeds, or any significance assessment, and the baseline appears to be a single run. Given that the scalar is chosen by grid search, the reader cannot distinguish a true amplification effect from selection noise. Please report standard errors over evaluation seeds or an equivalent perturbation analysis.
  5. [Appendix E] The limitations section explicitly states that 'we do not emphasize causality' and that the discovered module 'could be either overcomplete or incomplete,' yet the abstract and Section 4.4 describe SAMI as 'facilitating jailbreaking' and 'disabling' target-label recognition, and the conclusion states that SAMI 'allows direct control over model prediction and generation.' These causal claims should be reconciled with the stated limitation, either by softening the language or by adding the causal controls requested above.
minor comments (5)
  1. [Section 4.1] The threshold for the positive dataset Dp ('activation above 80% of the maximum strength') is stated without a sensitivity analysis; please indicate whether the discovered modules and the FQuAD result are stable to reasonable variations of this threshold.
  2. [Figure 26] The x-axis label 'Intervention coefficient' is ambiguous; the sweep procedure is described in words, but the figure should state whether larger values correspond to stronger negative or positive steering and give the units or scale of the coefficient.
  3. [Figure 9] The caption says '4 labels with the lowest/highest generalization error after intervention' but the four panels are not identified; please name the labels and state which two are lowest and which two are highest.
  4. [Table 2] The evaluation protocol for the DR baseline and the Gemma7B row is not fully specified (e.g., whether the same HarmBench harness and system-prompt settings are used for all rows); please add a footnote describing the exact evaluation setup.
  5. [Abstract and Section 3.1] The term 'concept-agnostic' is used although the method requires a positive dataset Dp and a chosen concept-vector construction; consider 'concept-general' to avoid overstatement.

Circularity Check

3 steps flagged · score 6.0 of 10

Headline benchmark gains are in-sample grid-search fits: the scalar intervention strengths for HarmBench, GSM8K, and ImageNet are selected on the same benchmark whose result is reported, so the quantitative claims carry a fitting burden; module discovery itself is not self-derivational.

  1. fitted input called prediction [Section 4.3 (Safety Module), scalar choices after Eq. (4); reported in Table 2.]
    "The safety module is comprised of K = 10 attention heads for all models (see Appendix C), and for jailbreaking we intervene with s =−1.7,−0.7,−0.8 respectively, found via a grid search. ... DEFENDER ... SAFETYMODULE(OURS) ... LLAMA-2 7B ... 71.1"

    The headline HarmBench result is an in-sample fit, not a prediction. The scalar s is grid-searched on the same HarmBench 'standard behaviors' set whose attack success rate is then reported, so the 71.1, 78.0, and 84.3 numbers are selected operating points of the search. The abstract's '+72.7%' jailbreaking gain is therefore the value at a strength optimized on the evaluation set rather than a test of whether the topK safety heads, at any fixed strength, cause the reported effect. The paper reports no held-out strength or pre-registered value for s, so the fitted scalar is load-bearing for the quantitative claim.

  2. fitted input called prediction [Section 4.2 (Reasoning Module), Table 1.]
    "We apply SAMI to LLAMA -3.1-8B-I NSTRUCT and GEMMA -7B-B ASE, and amplify the 'reasoning module' by s = 1.4 and 1.2 respectively, determined via a grid search. ... Table 1: GSM8K reasoning benchmark results: baselines and with our positive intervention. ... LLAMA 3.1-8B-I NST 84.61 85.44"

    The reported GSM8K improvement is selected by grid-searching s on the same GSM8K benchmark whose accuracy is then reported. The '+1.6%' figure is thus the outcome of an optimization over the intervention strength on the evaluation metric itself, not an out-of-sample prediction at a fixed intervention. While the module was selected independently via cosine similarity, the strength that produces the headline number is fitted to the target benchmark, which makes the reported improvement a fit under the fitted-input-called-prediction pattern.

1 more flagged steps
  1. fitted input called prediction [Section 4.4 (Recognition modules in vision transformers), Figure 9.]
    "We vary the intervention strength −1≤ s≤ 1. The results of this experiment are summarized in Figure 9. We observe that the model rapidly loses its ability to recognize the target label."

    The 'suppress the target label to 0%' result is the extremal point of a continuous sweep over s on the ImageNet validation set used for reporting. The paper does not fix s in advance or evaluate a distinct held-out set for the reported value, so the 0% target-error outcome is an in-sample operating point of the swept curve rather than a prediction about the discovered heads. Without sweeping a random-head or bottom-K control module under the same protocol, the topK selection is not independently tested by this experiment.

full rationale

SAMD's module selection (Eq. 3) is not circular: the topK heads are chosen by cosine similarity to a concept vector, and the intervention (Eq. 4) is a distinct operation whose behavioral effects are not logically entailed by the selection rule. The French FQuAD result uses a fixed s = -1 and is therefore a more genuine test, and the module-location observations for SAE concepts and ViTs are independent of any fitted scalar. The circularity that does exist is confined to the headline quantitative claims in Sections 4.2-4.4: in each case the intervention scalar s is grid-searched on the very benchmark whose improved or suppressed number is reported (GSM8K, HarmBench, ImageNet), so those numbers are selected fits, not predictions. No control module (random or bottom-K heads under the same s protocol) is reported, and the paper itself acknowledges in Appendix E that causality and stability are not analyzed, which is consistent with the conclusion that the intervention results are correlational. These fitted scalars are load-bearing for the abstract's '+72.7%', '+1.6%', and 'to 0%' claims, giving partial circularity under the fitted-input-called-prediction pattern; the central discovery procedure and the fixed-s FQuAD result retain independent content, so the overall score is 6 rather than higher.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on four main assumptions: the residual stream decomposition, the validity of concept vectors from SAE or difference-in-means, the cosine similarity as a relevance metric, and the assumption that top-K heads can be independently scaled. No new entities are introduced. The main free parameters are the module size K, the intervention scalar s (grid-searched for benchmark results), the activation threshold, and dataset sizes.

free parameters (4)
  • K (module size) = 5 (SAE concepts), 10 (safety), 3 (ViT), 5 (reasoning)
    Number of top cosine-similarity heads selected; chosen per experiment by visual inspection of the sorted similarity plots (Appendix C) rather than by a fixed rule.
  • Intervention scalar s = -1 (French/SAE), -1.7/-0.7/-0.8 (safety), 1.4/1.2 (reasoning), swept [-1,1] (ViT)
    Strength of module scaling; grid-searched for the benchmark results, so the headline numbers are fitted to the evaluation data.
  • SAE prompt activation threshold = 80% of max activation
    Threshold used to build the positive dataset Dp for SAE concepts (Section 4.1), chosen by hand.
  • Dp sizes = ~100 prompts (SAE), 100 GSM8K prompts, 200 ImageNet images
    Dataset sizes used to estimate the concept vector; no sensitivity analysis provided.
assumptions (4)
  • standard math Residual stream decomposition: r_l = r_{l-1} + sum_h a_{l,h} + m_l
    Used as the backbone of the method (Eq. 1), taken from Elhage et al. [18]; accepted in mechanistic interpretability.
  • domain assumption Concept vectors vc can be obtained by averaging activations, SAE decoder vectors, or difference-in-means
    Section 2.2; the three constructions are treated as valid concept abstractions, but their quality is not evaluated independently.
  • domain assumption Cosine similarity between head contribution and concept vector implies semantic relevance
    Section 3.1, Eq. (3); the core selection hypothesis, borrowed from representation-similarity work (Templeton et al. [20]) and asserted without validation.
  • domain assumption The top-K heads form an independent, interventionally effective module
    Sections 3.1-3.2; assumes the chosen heads can be scaled independently without disrupting other functions, which is tested only implicitly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers." pith.science (2026). https://pith.science/paper/PIOYV767

@misc{pith2026250617052,
  author       = {Pith},
  title        = {Pith review of: From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PIOYV767}},
  note         = {Machine review of arXiv:2506.17052}
}
read the original abstract

Transformers have achieved state-of-the-art performance across language and vision tasks. This success drives the imperative to interpret their internal mechanisms with the dual goals of enhancing performance and improving behavioral control. Attribution methods help advance interpretability by assigning model outputs associated with a target concept to specific model components. Current attribution research primarily studies multi-layer perceptron neurons and addresses relatively simple concepts such as factual associations (e.g., Paris is located in France). This focus tends to overlook the impact of the attention mechanism and lacks a unified approach for analyzing more complex concepts. To fill these gaps, we introduce Scalable Attention Module Discovery (SAMD), a concept-agnostic method for mapping arbitrary, complex concepts to specific attention heads of general transformer models. We accomplish this by representing each concept as a vector, calculating its cosine similarity with each attention head, and selecting the TopK-scoring heads to construct the concept-associated attention module. We then propose Scalar Attention Module Intervention (SAMI), a simple strategy to diminish or amplify the effects of a concept by adjusting the attention module using only a single scalar parameter. Empirically, we demonstrate SAMD on concepts of varying complexity, and visualize the locations of their corresponding modules. Our results demonstrate that module locations remain stable before and after LLM post-training, and confirm prior work on the mechanics of LLM multilingualism. Through SAMI, we facilitate jailbreaking on HarmBench (+72.7%) by diminishing "safety" and improve performance on the GSM8K benchmark (+1.6%) by amplifying "reasoning". Lastly, we highlight the domain-agnostic nature of our approach by suppressing the image classification accuracy of vision transformers on ImageNet.

Figures

Figures reproduced from arXiv: 2506.17052 by the authors.

Figure 1
Figure 1. A summary of our proposed pipeline. Left: Top-Left: The residual stream viewpoint of a transformer layer (Sec 2.1). Each attention head adds its linear contribution to the current representation. Stacking these contributions together for all H attention heads in a layer across L layers, we obtain a H × L matrix. Top-Right: The vector abstraction of an arbitrary concept in representation space (Sec 2.2). Bottom: Our … view at source ↗
Figure 2
Figure 2. “French” module: We determine the “French” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B-IT. The black bounding boxes mark the 5 most important heads, and hence define the module. Visibly, the identified heads are located in layers 15-26 matching previous findings that multilingual LLMs “think” in English first and translate into a target language later [33, 34]. An intri… view at source ↗
Figure 3
Figure 3. “San Francisco” intervention: When the “San Francisco” module is amplified the LLM tends to repeat the proper noun, when diminished the LLM can not recall the concept or even the related California geography, suggesting the corresponding SAE feature might not be monosemantic. Next, we apply SAMI to either diminish (negative intervention by factor s = −1) or amplify (positive intervention by factor s = 104 ) the modu… view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: “Dog” intervention: A negative intervention leads to the LLM to avoid mentioning dogs, while a positive one incites it to repeat the dog concept. Prompt: HELLO! YELL WITH ME! Generation (no intervention): YEEEAAAH! ... Generation (negative intervention): Yelling is a p…
Figure 5
Figure 5. Figure 5: “Yelling” intervention: Negative intervention on the “yelling” module not just leads the model to avoid paralanguage but also to advise the user to refrain from yelling. We evaluate the effectiveness of our intervention on the GSM8K benchmark [17]. As shown in [PITH_F…
Figure 6
Figure 6. Figure 6: “French” intervention: The “French” module can be used to control the response language: negative intervention compels the model to reply in English even though it was prompted in French. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 …
Figure 7
Figure 7. Figure 7: Safety module in LLAMA-2-CHAT-7B: - marked by black bounding boxes - is primarily located in the middle layers of the transformer. It is comprised of only 10 attention heads. finding, we see that positive intervention leads to repetition, as illustrated in the promptin…
Figure 8
Figure 8. Figure 8: Safety intervention: Similar to previous results, amplifying the safety module leads to repetition of the concept. Interestingly, the word safety was not directly part of the prompt dataset we used to determine the concept vector. Negative intervention on the other han…
Figure 9
Figure 9. Figure 9: SAMI results on VIT-B/32. Left: 4 labels with the lowest/highest generalization error after intervention. Right: Average effect across all labels of ImageNet. With our discovered modules, under negative intervention, we disable the recognizability on the attacked targe…
Figure 10
Figure 10. Figure 10: “Dog” module: We determine the “Dog” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B-IT. The black bounding boxes mark the 5 most important heads, and hence define the module. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2…
Figure 11
Figure 11. Figure 11: “YELLING” module: We determine the “YELLING” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B-IT. The black bounding boxes mark the 5 most important heads, and hence define the module. B.2 Modules discovered using SAE features…
Figure 12
Figure 12. Figure 12: “San Francisco” module: We determine the “San Francisco” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B-IT. The black bounding boxes mark the 5 most important heads, and hence define the module. 18 [PITH_FULL_IMAGE:figures/…
Figure 13
Figure 13. Figure 13: “Dog” module: We determine the “Dog” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B. The black bounding boxes mark the 5 most important heads, and hence define the module. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2…
Figure 14
Figure 14. Figure 14: “YELLING” module: We determine the “YELLING” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B. The black bounding boxes mark the 5 most important heads, and hence define the module. B.3 Chain-of-Thought-Reasoning Module In [P…
Figure 15
Figure 15. Figure 15: “San Francisco” module: We determine the “San Francisco” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B. The black bounding boxes mark the 5 most important heads, and hence define the module. 19 [PITH_FULL_IMAGE:figures/ful…
Figure 16
Figure 16. Figure 16: “French” module: We determine the “French” concept vector vc, and compare its cosine similarity to every attention head in GEMMA-2-2B. The black bounding boxes mark the 5 most important heads, and hence define the module. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1…
Figure 17
Figure 17. Figure 17: CoT module: We determine the CoT concept vector vc, and compare its cosine similarity to every attention head in LLAMA3.1-8B-INSTRUCT. The black bounding boxes mark the 5 most important heads, and hence define the module. B.5 ViT module In [PITH_FULL_IMAGE:figures/fu…
Figure 18
Figure 18. Figure 18: CoT module: We determine the CoT concept vector vc, and compare its cosine similarity to every attention head in GEMMA-7B-BASE. The black bounding boxes mark the 5 most important heads, and hence define the module. 10) usually lead to selecting the most significant he…
Figure 19
Figure 19. Figure 19: Safety module: We determine the safety concept vector vc, and compare its cosine similarity to every attention head in QWEN-7B-CHAT. The black bounding boxes mark the 5 most important heads, and hence define the module. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 …
Figure 20
Figure 20. Figure 20: Safety module: We determine the safety concept vector vc, and compare its cosine similarity to every attention head in GEMMA-7B-IT. The black bounding boxes mark the 5 most important heads, and hence define the module. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: “Tabby cat” module: The module for “tabby cat” recognition in a vision transformer unsurprisingly sits at the very end (last layers) of the transformer. More surprisingly, our VIT-B/32 model relies on only 3 attention heads for the task. 0 50 100 150 200 Attention Hea…
Figure 22
Figure 22. Figure 22: AVG cosine similarity plot of the SAE concepts. Red dots indicate the chosen attention [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 23
Figure 23. Figure 23: AVG cosine similarity plot of the “safety” concept. Red dots indicate the chosen attention [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]
Figure 24
Figure 24. Figure 24: AVG cosine similarity plot of the “reasoning” concept. Red dots indicate the chosen [PITH_FULL_IMAGE:figures/full_fig_p024_24.png]
Figure 25
Figure 25. Figure 25: AVG cosine similarity plot of ViT. Red dots indicate the chosen attention heads (the most [PITH_FULL_IMAGE:figures/full_fig_p025_25.png]
Figure 26
Figure 26. Figure 26: SAE FR-EN ratio (%) over different intervention strengths. We start from the recom [PITH_FULL_IMAGE:figures/full_fig_p025_26.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 44 canonical work pages

  1. [1]

    Interpretable machine learning–a brief history, state-of-the-art and challenges

    Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. Interpretable machine learning–a brief history, state-of-the-art and challenges. In Joint European conference on machine learning and knowledge discovery in databases, pages 417–431. Springer, 2020. 1

    2020

  2. [2]

    Explainable ai: A review of machine learning interpretability methods

    Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy, 23(1):18, 2020. 1

    work page 2020

  3. [3]

    Deep inside convolutional networks: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034,

    arXiv

  4. [4]

    Understanding neural networks through deep visualization

    Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579, 2015

    arXiv 2015

  5. [5]

    Visualizing deep neural network decisions: Prediction difference analysis

    Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In International Conference on Learning Representations, 2017

    work page 2017

  6. [6]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017

    2017

  7. [7]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017. 1

    work page 2017

  8. [8]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. 1

    work page 2017

Show all 81 references
  1. [9]

    Rethink- ing interpretability in the era of large language models

    Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethink- ing interpretability in the era of large language models. arXiv preprint arXiv:2402.01761, 2024. 1

    2024 arXiv

  2. [10]

    Transformer feed-forward layers are key-value memories

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. 2

    2021

  3. [11]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . Association for Computational Li...

    2022

  4. [12]

    The emergence of number and syntax units in lstm language models

    Yair Lakretz, Germ ´an Kruszewski, Th ´eo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units in lstm language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational ...

    2019

  5. [13]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems , 35:17359–17372,

  6. [14]

    Mass- editing memory in a transformer

    Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022. 2, 4, 17

    2022 arXiv

  7. [15]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023. 2, 6 11

    2023

  8. [16]

    HarmBench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. InProceedings of the 41st I...

    2024

  9. [17]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  10. [18]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

  11. [19]

    Efficient estimation of word representations in vector space, 2013

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013. URL https://arxiv.org/abs/1301.3781. 3

    2013 arXiv

  12. [20]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  13. [21]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J´anos Kram´ar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147,

    arXiv

  14. [22]

    Linear representations of sentiment in large language models

    Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154, 2023. 4

    2023 arXiv

  15. [23]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023

    2023 arXiv

  16. [24]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023

    2023 arXiv

  17. [25]

    Improving activation steering in language models with mean-centring

    Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813, 2023

    2023 arXiv

  18. [26]

    Refusal in language models is mediated by a single direction

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717, 2024. 4, 7, 8, 9

    2024 arXiv

  19. [27]

    interpreting gpt: the logit lens

    nostalgebraist. interpreting gpt: the logit lens. LessWrong, 2020. URL https://www. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens . 4

    2020

  20. [28]

    Investigating gender bias in language models using causal mediation analysis

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388–12401, 2020. 4

    2020

  21. [29]

    Localizing model behavior with path patching

    Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969, 2023

    2023 arXiv

  22. [30]

    Inducing causal structure for interpretable neural networks

    Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In International Conference on Machine Learning, pages 7324–7338. PMLR, 2022. 4 12

    2022

  23. [31]

    Transformerlens

    Neel Nanda and Joseph Bloom. Transformerlens. https://github.com/ TransformerLensOrg/TransformerLens, 2022. 5

    2022

  24. [32]

    Vit prisma: A mechanistic interpretability library for vision transformers

    Sonia Joseph. Vit prisma: A mechanistic interpretability library for vision transformers. https://github.com/soniajoseph/vit-prisma, 2023. 5

    2023

  25. [33]

    How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815, 2024

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815, 2024. 5, 6

    2024 arXiv

  26. [34]

    Do llamas work in english? on the latent language of multilingual transformers

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do llamas work in english? on the latent language of multilingual transformers. arXiv preprint arXiv:2402.10588,

    arXiv

  27. [35]

    Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders

    Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20526, 2024. 5, 17

    arXiv

  28. [36]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr ´e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. 5, 6, 17

    2024 arXiv

  29. [37]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. 5, 7

    2024 arXiv

  30. [38]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguye...

    2023

  31. [39]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. 6

    2022

  32. [40]

    A comprehensive survey on test-time adaptation under distribution shifts

    Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. International Journal of Computer Vision, 133(1):31–64, 2025. 6

    2025

  33. [41]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Lang...

    2019

  34. [42]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 7

    2021 arXiv

  35. [43]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 7

    2021 arXiv

  36. [44]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openrev...

    2023

  37. [45]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 7 13

    2023

  38. [46]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  39. [47]

    Interpretability dreams

    Christopher Olah. Interpretability dreams. Transformer Circuits Thread , 2023. https://transformer-circuits.pub/2023/interpretability-dreams/index.html. 7

    2023

  40. [48]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 7

    2023 arXiv

  41. [49]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 7

    2023 arXiv

  42. [50]

    Universal and transferable adversarial attacks on aligned language models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023. 8, 9

    2023 arXiv

  43. [51]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 8

    2010 arXiv

  44. [52]

    Vision transformers need registers

    Timoth´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=2dnO3LLiJ1. 9

    2024

  45. [53]

    Visualizing and understanding convolutional networks

    Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer, 2014. 9

    2014

  46. [54]

    Network dissec- tion: Quantifying interpretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541–6549, 2017. 17

    2017

  47. [55]

    Interpreting deep visual representa- tions via network dissection

    Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representa- tions via network dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9):2131–2145, 2018

    2018

  48. [56]

    Interpretable basis decomposition for visual explanation

    Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 119–134, 2018

    2018

  49. [57]

    Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020

    David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020. 17

    2020

  50. [58]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. 17

    2023

  51. [59]

    Does localization inform editing? surprising differences in causality-based localization vs

    Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36:17643–17668, 2023

    2023

  52. [60]

    Editing common sense in transformers

    Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense in transformers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8214–8232, 2023

    2023

  53. [61]

    Massive editing for large language models via meta learning

    Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=L6L1CJQ2PE. 14

    2024

  54. [62]

    Pmet: Precise model editing in a transformer

    Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 18564–18572, 2024

    2024

  55. [63]

    Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons

    Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, page...

    2024

  56. [64]

    Are neural nets modu- lar? inspecting functional modularity through differentiable weight masks

    R´obert Csord ´as, Sjoerd van Steenkiste, and J ¨urgen Schmidhuber. Are neural nets modu- lar? inspecting functional modularity through differentiable weight masks. arXiv preprint arXiv:2010.02066, 2020. 17

    2010 arXiv

  57. [65]

    Sparse interventions in language models with differentiable masking

    Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. arXiv preprint arXiv:2112.06837, 2021

    2021 arXiv

  58. [66]

    Knowledge neurons in pretrained transformers

    Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.08696, 2021

    2021 arXiv

  59. [67]

    Finding and editing multi- modal neurons in pre-trained transformers

    Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. Finding and editing multi- modal neurons in pre-trained transformers. In Findings of the Association for Computational Linguistics ACL 2024, pages 1012–1037, 2024

    2024

  60. [68]

    Towards neuron attributions in multi-modal large language models

    Junfeng Fang, Zac Bi, Ruipeng Wang, Houcheng Jiang, Yuan Gao, Kun Wang, An Zhang, Jie Shi, Xiang Wang, and Tat-Seng Chua. Towards neuron attributions in multi-modal large language models. Advances in Neural Information Processing Systems, 37:122867–122890,

  61. [69]

    Retrieval head mecha- nistically explains long-context factuality

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mecha- nistically explains long-context factuality. arXiv preprint arXiv:2404.15574, 2024. 17

    2024 arXiv

  62. [70]

    Successor heads: Recurring, interpretable attention heads in the wild

    Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, interpretable attention heads in the wild. arXiv preprint arXiv:2312.09230, 2023. 17

    2023 arXiv

  63. [71]

    Sparse autoen- coders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600,

    arXiv

  64. [72]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. 17

    2020 doi

  65. [73]

    Mechanistic interpretability for ai safety–a review

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety–a review. arXiv preprint arXiv:2404.14082, 2024. 17

    2024 arXiv

  66. [74]

    Open problems in mechanistic interpretability

    Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496, 2025. 17

    2025 arXiv

  67. [75]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  68. [76]

    Interpretability in the wild: a circuit for indirect object identification in GPT-2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openrev...

    2023

  69. [77]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=9XFSbDPmdW. 17

    2023

  70. [78]

    Iteration head: A mechanistic study of chain-of-thought

    Vivien Cabannes, Charles Arnal, Wassim Bouaziz, Alice Yang, Francois Charton, and Ju- lia Kempe. Iteration head: A mechanistic study of chain-of-thought. arXiv preprint arXiv:2406.02128, 2024. 17

    2024 arXiv

  71. [79]

    the Golden Gate Bridge

    Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri `a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability.Advances in Neural Information Processing Systems, 36:16318–16352, 2023. 17 16 A Related Work In this sectio...

    2023

  72. [81]

    Tabby cat

    usually lead to selecting the most significant heads. The results on SAE concepts (Figure 22), safety (Figure 23), reasoning (Figure 24) and ViT concepts (Figure 25) are provided. D FQuad SAE sweep result In Figure 26, we provide the FR-EN ratio under sparse autoencoder vector...

  73. [2021]

    https://transformer-circuits.pub/2021/framework/index.html. 3

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.