Pith. sign in

REVIEW 3 major objections 6 minor 47 references

Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that language models fail at balanced parentheses not because sound mechanisms are absent, but because faulty components' noisy votes overshadow reliable ones, and that amplifying the reliable components fixes the errors.

desk verdict Solid empirical study with a working intervention, but the central 'overshadowing' mechanism is not nailed down; the GPT-2 Small four-paren failure is the unresolved exception. read the letter →

arxiv 2507.00322 v2 pith:DBD6XTPG submitted 2025-06-30 cs.CL cs.AIcs.SE

Daking Rai , Samuel Miller , Kevin Moran , Ziyu Yao This is my paper
classification cs.CLcs.AIcs.SE
keywords balancedparentheseslanguagemodelinterpretabilitylogitlensattentionheadsfeed-forwardneuronsactivationsteeringsyntacticcodegenerationRASteer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks why language models that excel at complex coding still miscount closing parentheses. Using the logit lens (projecting each component's activation into the vocabulary space), it looks inside seven models (124M to 7B parameters) and finds that attention heads and feed-forward neurons act like a crowd of independent voters: a few reliably vote for the correct parenthesis token across many input shapes, while most are noisy and push wrong tokens. The paper's central claim is that LMs do not fail because sound mechanisms are missing; they fail because faulty mechanisms' noisy votes overwhelm the sound ones. On that diagnosis it builds RASteer, which ranks components by reliability and scales up the reliable ones, raising accuracy on the hardest subtasks from 0% to near 100% for several models without hurting HumanEval performance.

What carries the argument

The load-bearing object is the logit-lens vote of a single component: project the component's activation (an attention head's output, or an FF neuron's value times its activation strength) through the unembedding matrix into vocabulary space, and treat that vector as the component's contribution to the final logit. RASteer then ranks components by generalizability across the four subtasks and by recall, precision, or F1 of the tokens they promote, and scales the activations of the top-$k$ components by a multiplier $\alpha \in [1.1, 2.0]$ before adding them to the residual stream. The additive residual-stream motif is what makes both the diagnosis and the intervention meaningful.

What would settle it

Take a prompt where the model errs, compute every component's logit-lens contribution to the four candidate closing-parenthesis tokens, sum those contributions, and check whether that sum, with the sound components scaled as RASteer does, reproduces the full model's final logits. If the summed contributions cannot predict the model's errors, or if zeroing out the top-ranked faulty components fails to flip a wrong prediction to the correct token, the overshadowing account is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a transformer LM's final prediction on the balanced-parentheses task is an additive vote of many components, and error is an interference effect, not an absence effect. Each attention head and each qualifying FF neuron is read through the logit lens, so every component's activation is projected into vocabulary space and treated as its ballot among candidate parenthesis tokens. A small set of components are sound: they give the correct token the top logit across the one-, two-, three-, and four-closing-parenthesis subtasks, and one CodeLlama-7b attention head (layer 30, head 0) reaches nearly 100% accuracy across all subtasks, beating the full model's 96%. Most components, however, are noisy and non-selective, promoting correct and distractor tokens together; errors appear exactly when the aggregate noise of these faulty components swamps the reliable vote. The paper concludes that the model already contains the right mechanisms, and that amplifying them (RASteer) should fix the errors, which it reports doing.

Load-bearing premise

The argument stands or falls on the additive motif: the final logit is the simple sum of each component's independent vote, so ranking components by their individual logit-lens projections and scaling them reveals the true cause of errors; if components interact or the model suppresses noise non-additively, both the diagnosis and the steering gains could be artifacts.

Editorial extensions

If this is right

  • Because the sound mechanisms are already inside the model, accuracy on hard syntactic subtasks can be improved at inference time by scaling up a handful of reliable attention heads, with no retraining or finetuning.
  • A very small set of heads (often the top 5 to 10) suffices to move a model from 0% to roughly 100% on the three- and four-paren subtasks, and larger models need fewer boosts.
  • Promoting components that are causally important but do not directly shape the final logit can destabilize generation: adding five such circuit-discovered heads to GPT-2 Medium dropped accuracy from about 100% to 4%, so only output-proximate reliable components should be steered.
  • Targeted steering of the top-20 heads preserves and sometimes improves general code generation on HumanEval (Llama2-7b improved by 5.49%), while steering more than about 20 heads starts to degrade it.
  • The same ranking-and-steering recipe transfers to an arithmetic reasoning task, with reported gains up to 20.25% on Pythia-6.9b multiplication.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the additive-vote account is right, faulty components are actively harmful rather than merely unhelpful, so suppressing or ablating the lowest-ranked noisy components instead of only boosting reliable ones should give comparable or better gains, and that experiment would directly test the overshadowing mechanism.
  • Beyond the paper: the same rank-by-reliability and amplify recipe could be applied to other position-sensitive syntactic tasks such as indentation, JSON bracket matching, or XML tag closure, where the paper's analysis predicts the same pattern of a few reliable heads buried under many noisy voters.
  • Beyond the paper: the additive assumption is the natural point to probe, since if component contributions interact multiplicatively or the model suppresses noise through dedicated mechanisms, the reported steering gains could come from directly boosting tokens that the selected heads already favor rather than from restoring an overshadowed sound mechanism.
  • Beyond the paper: the finding that a single head can outperform the full model suggests the output is diluted by averaging in many weak voters, so an untested practical corollary is that test-time ensembling of just the sound heads might yield cheap accuracy gains on other code-completion tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper studies why transformer LMs make balanced-parentheses errors on a synthetic code-completion task. It decomposes the task into one- to four-closing-paren sub-tasks across seven models (124M–7B) and uses logit-lens projections of attention heads and FF neurons to characterize each component as an independent predictor. The main empirical findings are: (i) components vary widely in per-sub-task accuracy and generalization; (ii) most components promote correct and incorrect tokens simultaneously (noisy promotion, low selectivity); (iii) the authors hypothesize that errors occur because faulty, noisy components overshadow sound ones; and (iv) their RASTEER method ranks components by generalizability and promotion F1 on training data, then scales top-k component activations, improving three- and four-paren accuracy dramatically on held-out test sets, with modest gains on arithmetic and no major loss on HumanEval.

Significance. If the interpretation is correct, the paper offers a practical and economical intervention, and the head- and neuron-level analysis across seven models is a useful reference. Strengths include public code and data, held-out evaluation, a circuit baseline, and an honest discussion of limitations. However, the causal mechanism is inferred from correlations and a single steering intervention, the additive assumption is explicitly acknowledged in Section 7, and one model (GPT-2 Small, four-paren) is a clear anomaly that the paper itself leaves unexplained. These issues do not invalidate the empirical gains, but they do weaken the paper's central claim as currently stated.

major comments (3)
  1. [Section 5.4.2, Figure 4d, Table 4, Figure 6] The GPT-2 Small four-paren failure is a load-bearing counterexample to the paper's central hypothesis: the model has an accurate four-paren head (Table 4, L7H6), top-60 F1-score distributions similar to GPT-2 Medium (Figure 6), yet RASTEER-attention steering leaves accuracy at 0% (Figure 4d). The paper labels this an 'unexplored factor' but does not report the minimal diagnostic of whether the steered activation actually raised the correct-token logit at the final layer. Without that diagnostic, the additive-overshadowing account in Section 1 cannot explain the anomaly, and the universal claim that errors are caused by faulty mechanisms overshadowing sound ones is not supported for this model. Please add the final-layer logit comparison and test the three failure modes the paper itself implies: measurement at the wrong position/unembedding layer, downstream cancellation or suppression, and non-additive interaction.
  2. [Section 7 and Section 4.1] The core mechanistic narrative depends on the additive motif acknowledged in Section 7, but the improvement from RASTEER does not distinguish the overshadowing explanation from the simpler explanation that the intervention directly boosts tokens that selected components already favor. Because components are ranked by promotion F1 on training prompts (Section 4.1) and then amplified by a scalar multiplier, the held-out gains show generalization of a learned ranking but do not establish that faulty components actively suppressed correct logits before steering. A concrete test would be to compare RASTEER against a control that amplifies the same number of random or equally accurate heads, or to ablate the ranking metric and check whether post-steering logit changes match the predicted additive sums.
  3. [Section 5.4.1, Figure 5] The minimal steering experiment shows that for Pythia-6.9b, steering only the heads shared between RASTEER and the circuit baseline gives no improvement, while steering either method's disjoint set reaches roughly 100% accuracy. This demonstrates redundancy, but it also shows that the specific heads identified by RASTEER are not individually necessary. The paper should therefore temper statements that these heads implement 'the' sound mechanisms; at present the evidence supports the existence of multiple redundant paths, not a unique overshadowed mechanism.
minor comments (6)
  1. [Section 2.2 and Table 3] The text states 'All models had 100% accuracy for the one and two-paren task,' but Table 3 reports CodeLlama-7b at 99% on one-paren; please reconcile the text and table.
  2. [Section 5.4.1 and Figure 5] The phrase '0.2 overlap percentage' is confusing because 0.2 is used as a fraction (one shared head out of five); please express the quantity consistently as either a percentage (20%) or a fraction.
  3. [Throughout] The acronym appears with inconsistent capitalization (RASTEER, RASteer, and 'RASteer' in the abstract); please unify the spelling.
  4. [Appendix C, Equation (4)] Equation (4) divides by P_clean(r') and P_patched(r), which may be zero for some token/model combinations; please add a smoothing term or describe how zero probabilities are handled.
  5. [Section 5.3 and Abstract] The HumanEval result for CodeLlama-7b is a decline from 30.48% to 29.87%; the abstract's claim of 'without impairing general coding ability' is defensible but should be qualified as 'no substantial degradation' given this small decrease.
  6. [Section 3.3, Algorithm 2] The promotion threshold tau is fixed at 0.5; since the qualitative claim about the absence of high-precision components is threshold-sensitive, reporting precision and recall across a range of tau values would strengthen the claim.

Circularity Check

2 steps flagged · score 4.0 of 10

RASteer's accuracy gains are partly manufactured by its component-selection criterion, and Section 7 uses those gains to reinforce the very additive/overshadowing assumption the method presupposes.

  1. fitted input called prediction [Section 4.1 (RASTEER ranking and steering), Algorithms 1-2, Section 5.2]
    "Given a sorted list of LM components, we perform LM steering to increase the impact of the top-k components on the final prediction. Specifically, for each selected component c, we scale its activation h_c by a multiplier α∈[1.1,2.0] before adding it to the residual stream."

    The ranking that feeds the steering is built from Algorithm 2's promotion label: a component is ranked highly because its logit-lens projection already gives the correct token a logit at least τ·max(l_c) on training prompts. Scaling that activation adds α·l_c to the correct-token logit under the paper's additive assumption, so the post-steering accuracy gain is partly a mechanical consequence of the selection criterion rather than an independent confirmation that faulty mechanisms were the cause. The held-out test set saves the result from being a pure tautology, but the reported 0%→~100% improvements are not evidence that 'overshadowing' specifically is the error mechanism, since any correct-token-promoting component would behave the same way when amplified.

  2. other [Section 7 (Limitations and Conclusion)]
    "our approach assumes that LMs follow a simple additive motif, where the final logit is formed by simply adding the contributions from individual components. While this assumption is also supported by several prior findings [10, 14] and further reinforced by our results with RASTEER"

    The paper uses RASTEER's success to reinforce the additive/overshadowing assumption, but RASTEER's operation presupposes that assumption: it selects components by their measured correct-token vote and then adds scaled versions of those votes into the residual stream. Observing that accuracy rises after adding correct-token votes therefore does not independently verify the additive motif or the claim that faulty components 'overshadow' sound ones; it restates the selection criterion in performance terms. The circularity is partial because the component analyses in Section 3 and the held-out evaluations are independent empirical content, but the evidential loop closes where the method's success is cited as support for the mechanism that the method assumes.

full rationale

RASteer's evaluation uses held-out test labels, so the reported accuracy gains are not fitted to the test set; that keeps this from being a fully circular paper. The component analysis in Section 3 (accuracy distributions, generalizable heads in Tables 1 and 4, precision-recall scatter plots) is an independent empirical measurement, and the circuit baseline provides a non-trivial comparison. The partial circularity is in the validation loop: components are ranked by their measured tendency to promote the correct token on training prompts, then RASTEER amplifies those same votes, and the resulting improvement is presented as reinforcing the 'faulty mechanisms overshadow sound ones' hypothesis. That improvement is partly built into the method, since under the paper's stated additive assumption scaling a correct-token-promoting component mechanically raises the correct token's logit. The paper's own GPT-2 Small four-paren exception (Section 5.4.2) is not a circular step but is a genuine limitation: the paper concedes 'an unexplored factor' and does not report the diagnostic check—whether the steered activations actually raised the correct-token logit at the final layer—that would be the minimal condition for its own hypothesis. I found no load-bearing self-citation: the authors' prior works are cited for technique or as examples of syntactic failures, not as the source of the central claim. Overall score 4: one 'prediction' (the RASteer improvement) is partially constructed from its selection inputs, but the central claim retains independent empirical content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No fundamentally new physical entities are postulated; 'sound' and 'faulty' are labels for existing attention heads and neurons. The central claims rest on the additive logit-lens assumption, training-set transfer, and synthetic task definitions, each listed above.

free parameters (5)
  • promotion threshold tau = 0.5
    Algorithm 2 labels a token as promoted only if its logit is at least tau times the max logit; the value is fixed by the authors, not derived.
  • generalization accuracy threshold = 0.7
    Components with accuracy >= 0.7 on more than one sub-task are called generalizable or sound in Section 3.2; the cutoff is arbitrary.
  • steering multiplier alpha = 1.1 to 2.0, tuned per model on dev
    Section 5.1 selects the optimal alpha on the dev set for each model; the final value is a fitted hyperparameter.
  • number of steered components k (top-k) = varies (e.g., 5 to 60 heads)
    The paper sweeps k and reports performance curves; the recommended k for HumanEval (20 heads) is selected after observing degradation beyond it.
  • ranking metric = F1 (recall also used for arithmetic)
    Section 4.1 says the ranking metric is chosen based on the experimental results among recall, precision, and F1, making it a selected hyperparameter.
assumptions (5)
  • domain assumption Final logits are an additive sum of independent component contributions, so scaling one component does not change other components' effects.
    Invoked throughout Sections 3 and 4 via logit lens and activation scaling; the authors explicitly flag it in Section 7 as potentially overlooking non-additive suppression mechanisms.
  • domain assumption Logit-lens projection of a component activation with the unembedding matrix reveals which tokens that component promotes.
    Used in Algorithms 1 and 2 to label components as sound or faulty; follows prior work [29,16] but is not independently validated on these models.
  • domain assumption Component reliability measured on the 350-example training set transfers to the held-out test set and to HumanEval.
    RASteer ranks on training prompts and is evaluated on test prompts and HumanEval; no analysis of ranking stability across data splits is provided.
  • domain assumption The four synthetic prompt templates faithfully represent the balanced parentheses problem LMs face in real code.
    The paper states in Section 7 that synthetic data may have exaggerated the effectiveness of the method.
  • domain assumption FF neurons whose input-independent projection does not contain a parenthesis token in the top-50 or bottom-50 logits cannot affect the target prediction.
    Static pre-filtering in Section 3.1 may exclude neurons that participate through interactions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones." pith.science (2026). https://pith.science/paper/DBD6XTPG

@misc{pith2026250700322,
  author       = {Pith},
  title        = {Pith review of: Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBD6XTPG}},
  note         = {Machine review of arXiv:2507.00322}
}
abstract

Despite remarkable advances in coding capabilities, language models (LMs) still struggle with simple syntactic tasks such as generating balanced parentheses. In this study, we investigate the underlying mechanisms behind the persistence of these errors across LMs of varying sizes (124M-7B) to both understand and mitigate the errors. Our study reveals that LMs rely on a number of components (attention heads and FF neurons) that independently make their own predictions. While some components reliably promote correct answers across a generalized range of inputs (i.e., implementing "sound mechanisms''), others are less reliable and introduce noise by promoting incorrect tokens (i.e., implementing "faulty mechanisms''). Errors occur when the faulty mechanisms overshadow the sound ones and dominantly affect the predictions. Motivated by this insight, we introduce RASteer, a steering method to systematically identify and increase the contribution of reliable components for improving model performance. RASteer substantially improves performance on balanced parentheses tasks, boosting accuracy of some models from $0$% to around $100$% without impairing the models' general coding ability. We further demonstrate its broader applicability in arithmetic reasoning tasks, achieving performance gains of up to around $20$%.

Figures

Figures reproduced from arXiv: 2507.00322 by the authors.

Figure 1
Figure 1. Accuracy distributions of attention heads across sub-tasks for CodeLlama-7b. Attention [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Average logit values and coefficients of FF neuron L19N11 of CodeLlama when the input [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Scatter plots of average precision vs. recall across all sub-tasks for attention heads (left [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance of RASTEER (steering either attention heads or FF neurons) and the circuit baseline on the three-paren (top) and the four-paren (bottom) sub-tasks. When zero heads or neurons are steered, it shows each model’s raw performance without steering. Results of RA…
Figure 5
Figure 5. Figure 5: Percentage overlap of the top-k RASTEER attention heads, where k=5 for GPT-2 XL, CodeLlama-7b, and Pythia-6.9b and k=10 for GPT-2 Medium and Llama2-7b, with the top-60 attention heads from the respective circuit baseline, for models where the steering of a small number…
Figure 6
Figure 6. Figure 6: F1-score distributions of the top-60 RASTEER attention heads sorted by sub-task F1-Score for GPT-2 Small and GPT-2 Medium. attention heads and similar performance on the four-paren sub-task before applying RASTEER. We examine the sub-task level F1-scores of the top-60 …
Figure 7
Figure 7. Figure 7: The plots illustrate how attention head accuracy varies across sub-tasks across six models. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Scatter plots of average precision vs. recall across all sub-tasks for attention heads ( [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Performance of the model after steering both attention heads and FF neurons using [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Comparison of RASTEER performance using recall-based (top) and precision-based (bottom) ranking metrics when steering attention heads on the three-paren and four-paren sub-tasks. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 16 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024. 11

    arXiv 2024

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Refusal in language models is mediated by a single direction.arXiv preprint arXiv:2406.11717, 2024

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction.arXiv preprint arXiv:2406.11717, 2024

    arXiv 2024

  4. [4]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  5. [5]

    Mechanistic interpretability for AI safety - a re- view.Transactions on Machine Learning Research, 2024

    Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety - a re- view.Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //openreview.net/forum?id=ePUVetPKu6. Survey Certification, Expert Certification

    work page 2024

  6. [6]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  7. [7]

    Towards monosemanticity: Decomposing language models with dictionary learning.Transformer Circuits Thread, 2023

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and ...

    2023

  8. [8]

    Causal scrubbing: A method for rigorously testing interpretability hypotheses

    Lawrence Chan, Adria Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: A method for rigorously testing interpretability hypotheses. InAI Alignment Forum, volume 2, 2022

    work page 2022

Show all 47 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  2. [10]

    Summing up the facts: Additive mechanisms behind factual recall in llms.arXiv preprint arXiv:2402.07321, 2024

    Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms.arXiv preprint arXiv:2402.07321, 2024

    2024 arXiv

  3. [11]

    Analyzing transformers in embedding space.arXiv preprint arXiv:2209.02535, 2022

    Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space.arXiv preprint arXiv:2209.02535, 2022

    2022 arXiv

  4. [12]

    What’s wrong with your code generated by large language models? an extensive study.arXiv preprint arXiv:2407.06153, 2024

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, et al. What’s wrong with your code generated by large language models? an extensive study.arXiv preprint arXiv:2407.06153, 2024

    2024

  5. [13]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    2024

  6. [14]

    A mathematical framework for transformer circuits.Transformer Circuits Thread,

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, 12 Kamal Ndousse, D...

  7. [15]

    Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis.arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  8. [16]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.arXiv preprint arXiv:2203.14680, 2022

    Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.arXiv preprint arXiv:2203.14680, 2022

    2022 arXiv

  9. [17]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.Advances in Neural Information Processing Systems, 36:76033–76060, 2023

    Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.Advances in Neural Information Processing Systems, 36:76033–76060, 2023

    2023

  11. [19]

    Have faith in faithfulness: Going be- yond circuit overlap when finding model mechanisms

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going be- yond circuit overlap when finding model mechanisms. InICML 2024 Workshop on Mechanistic Interpretability, 2024. URLhttps://openreview.net/forum?id=grXgesr5dT

    2024

  12. [20]

    How to use and interpret activation patching.arXiv preprint arXiv:2404.15255, 2024

    Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching.arXiv preprint arXiv:2404.15255, 2024

    2024 arXiv

  13. [21]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.CoRR, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.CoRR, 2024

    2024

  14. [22]

    Inference- time intervention: Eliciting truthful answers from a language model.Advances in Neural Information Processing Systems, 36:41451–41530, 2023

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model.Advances in Neural Information Processing Systems, 36:41451–41530, 2023

    2023

  15. [23]

    Starcoder: may the source be with you!Transactions on Machine Learning Research, 2023

    Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, M...

    2023

  16. [24]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  17. [25]

    In-context vectors: making in context learning more effective and controllable through latent space steering

    Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: making in context learning more effective and controllable through latent space steering. InProceedings of the 41st International Conference on Machine Learning, pages 32287–32307, 2024

    2024

  18. [26]

    Mechanistic understanding of language models in syntactic code completion.arXiv preprint arXiv:2502.18499, 2025

    Samuel Miller, Daking Rai, and Ziyu Yao. Mechanistic understanding of language models in syntactic code completion.arXiv preprint arXiv:2502.18499, 2025. 13

    2025 arXiv

  19. [27]

    Code- gen2: Lessons for training llms on programming and natural languages.arXiv preprint arXiv:2305.02309, 2023

    Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Code- gen2: Lessons for training llms on programming and natural languages.arXiv preprint arXiv:2305.02309, 2023

    2023 arXiv

  20. [28]

    Arithmetic without algo- rithms: Language models solve math with a bag of heuristics

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. Arithmetic without algo- rithms: Language models solve math with a bag of heuristics. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=O9YTt26r2P

    2025

  21. [29]

    Interpreting gpt: the logit lens.AI Alignment Forum, 2020

    nostalgebraist. Interpreting gpt: the logit lens.AI Alignment Forum, 2020. https://www. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  22. [30]

    Zoom in: An introduction to circuits.Distill, 5(3):e00024–001, 2020

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 5(3):e00024–001, 2020

    2020

  23. [31]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  24. [32]

    An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of llms

    Daking Rai and Ziyu Yao. An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7174–7193, 2024

    2024

  25. [33]

    A practical review of mech- anistic interpretability for transformer-based language models.arXiv preprint arXiv:2407.02646, 2024

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mech- anistic interpretability for transformer-based language models.arXiv preprint arXiv:2407.02646, 2024

    2024

  26. [34]

    Red-teaming language models via activation engineering.Accessed: October, 13:2023, 2023

    Nina Rimsky. Red-teaming language models via activation engineering.Accessed: October, 13:2023, 2023

    2023

  27. [35]

    Steering llama 2 via contrastive activation addition

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational...

    2024 doi

  28. [36]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  29. [37]

    Attribution patching outperforms automated circuit discovery

    Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407–416, 2024

    2024

  30. [38]

    Linear representations of sentiment in large language models.CoRR, 2023

    Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models.CoRR, 2023

    2023

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  32. [40]

    Activation addition: Steering language models without optimization.CoRR, 2023

    Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization.CoRR, 2023

    2023

  33. [41]

    Extending activation steering to broad skills and multiple behaviours.arXiv preprint arXiv:2403.05767, 2024

    Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours.arXiv preprint arXiv:2403.05767, 2024

    2024 arXiv

  34. [42]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017. 14

    2017

  35. [43]

    Interpretability in the wild: a circuit for indirect object identification in GPT-2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https: //openrevi...

    2023

  36. [44]

    Where do large language models fail when generating code?arXiv preprint arXiv:2406.08731, 2024

    Zhijie Wang, Zijie Zhou, Da Song, Yuheng Huang, Shengmai Chen, Lei Ma, and Tianyi Zhang. Where do large language models fail when generating code?arXiv preprint arXiv:2406.08731, 2024

    2024 arXiv

  37. [45]

    Reft: Representation finetuning for language models.Advances in Neural Information Processing Systems, 37:63908–63962, 2024

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models.Advances in Neural Information Processing Systems, 37:63908–63962, 2024

    2024

  38. [46]

    #print the string 160\nprint(str(160

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency.CoRR, 2023. 15 A Model Performance on Balanced Parentheses Ta...

    2023

  39. [2021]

    https://transformer-circuits.pub/2021/framework/index.html

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.