Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Large language models solve mental arithmetic with almost all input-specific computation at the final token, needing only two middle layers of information transfer from the other tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:44 UTC pith:IYOOFC23

load-bearing objection Solid empirical paper with a genuinely useful transfer-layer result, but the 'waiting' claim rests on a counterfactual ablation that hasn't been validated. the 5 major comments →

arxiv 2509.09650 v1 pith:IYOOFC23 submitted 2025-09-11 cs.CL

All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens

Siddarth Mamidanna , Daking Rai , Ziyu Yao , Yilun Zhou This is my paper
classification cs.CL
keywords mental mathmechanistic interpretabilityContext-Aware Mean AblationAttention-Based PeekingAll-for-One subgrapharithmetic in LLMsinformation transfercausal ablation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that LLMs do not need the full transformer when doing mental math. On expressions like A+B+C, it isolates a minimal 'All-for-One' subgraph: the first fifteen layers perform task-general processing without letting tokens exchange input-specific values, the next two layers pass the operand information to the last token, and all later layers let the last token compute the answer alone. This subgraph keeps near-full accuracy on many arithmetic tasks and across multiple model families, and removing the transfer attention in layer 15 sharply degrades performance. The authors conclude that arithmetic is non-compositional across token positions and that most of the model's depth is spent computing rather than communicating.

Core claim

The central claim is that for 'mental math' prompts—direct arithmetic answered by next-token prediction without chain of thought—the model's computation has a three-stage structure. In stage one, the first Lwait layers (up to 15 in Llama-3-8B) perform only task-general processing; no input-specific information flows between tokens. In stage two, just two middle layers (layer indices 15 and 16) transfer each operand's value to the final token. In stage three, all remaining layers are run with only self-attention, and the last token alone computes the answer. The paper reports 'no performance degradation' from this pruning and calls the subgraph sufficient and necessary: faithfulness stays hig

What carries the argument

Context-Aware Mean Ablation (CAMA) replaces a token's residual-stream activation at the end of the waiting stage with its conditional expectation given that token's value, averaged over the task distribution: \tilde{x}_t^{(Lwait)} = E_{x'\sim P(x|x_t)}[m(x',t,Lwait)]. This is meant to erase what the token learned from other input tokens while keeping task-general background computation. Attention-Based Peeking (ABP) modifies attention masks so that each query can attend only to chosen keys; the paper uses it to force non-last tokens to self-attend and to restrict the last token to full-peeking for exactly Ltransfer layers before it too only self-attends. The combination defines the AF1 subgr

Load-bearing premise

The whole waiting-stage result rests on CAMA being a genuine counterfactual for 'the token received no input-specific information from other tokens'; if the conditional average merely injects useful mean statistics into the residual stream, the long waiting period is an ablation artifact rather than a property of the model.

What would settle it

Take an AF1-ablated Llama-3-8B on A+B+C and train a linear probe on the last token's residual vector at layer 15 (before transfer) to predict the value of an earlier operand A. If probe accuracy is significantly above chance on held-out contexts, input-specific information about other tokens is leaking through CAMA, and the 15-layer waiting period is not actually input-free.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the AF1 subgraph is faithful, Llama-3-8B does not compute A+B+C by composing partial results across token positions; all input-specific arithmetic happens in the last token's residual stream.
  • Layer 15 attention from the last token to the operands is necessary: removing it in the otherwise full model degrades accuracy on every arithmetic task tested.
  • Two transfer layers are sufficient to carry all operand information, implying that the other roughly thirty layers are mostly local computation rather than inter-token communication.
  • The same AF1 subgraph transfers across Llama-3 and Llama-3.1 and across multiple two- and three-operand operations, including some verbal phrasings, suggesting a common arithmetic mechanism rather than a task-specific trick.
  • For Pythia and GPT-J, the same methods find AF1-like subgraphs with shorter waits and longer transfers, indicating the phenomenon is not exclusive to one model family.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: if the long waiting period is a product of next-token training pressure, then models fine-tuned to produce intermediate reasoning tokens (e.g., chain-of-thought or code) should show a shorter Lwait under CAMA/ABP; this can be checked directly.
  • The AF1 boundary can serve as a tokenizer diagnostic: models that split numbers into digit tokens (the paper names Qwen and Gemma as such) should require a longer or differently positioned transfer stage, since per-digit CAMA would need to coordinate across tokens.
  • The authors' waiting-stage claim is only as strong as CAMA's counterfactual validity; a sharper test would compare CAMA to a direct attention-mask intervention that blocks all early cross-token attention while preserving MLP computation, to see whether layer 14 is the true boundary or an artifact of averaging.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes two intervention techniques for transformer interpretability: Context-Aware Mean Ablation (CAMA), which replaces a token's hidden state at a given layer with its conditional expectation over the task distribution, and Attention-Based Peeking (ABP), which prunes attention masks to selected key positions. Using these, the authors identify an 'All-for-One' (AF1) subgraph for mental arithmetic in Llama-3-8B and Llama-3.1-8B, with a long waiting period (up to Lwait=15 layers), a short transfer period (Ltransfer=2 layers, specifically layers 15 and 16), and final computation only at the last token. They report high faithfulness across eight arithmetic tasks, similar albeit softer patterns in Pythia and GPT-J, head-level analyses of the transfer layers, and logit-lens evidence that the subgraph reproduces the full model's internal computation. The paper claims the AF1 subgraph is both sufficient and necessary for high performance and transfers across models.

Significance. If the central claim holds, the paper provides a striking and non-compositional picture of arithmetic in LLMs: operand-specific computation is deferred almost entirely to the last token, with a very short information-transfer window. This would be a useful contribution to mechanistic interpretability and would introduce two generally applicable ablation tools (CAMA and ABP). The paper's strengths include multiple converging lines of evidence: grid faithfulness, single-layer attention deletion, iterative head pruning, logit-lens consistency, and cross-model/cross-format trends. The authors are also candid about limitations (Sec. 4.6 and Limitations). However, the long-waiting conclusion depends on CAMA being a faithful counterfactual, a premise that is not independently validated, and the abstract's necessity and cross-model-transfer claims go beyond what the experiments strictly show. With additional validation and more careful framing, the paper could make a substantial contribution.

major comments (5)
  1. [§3.3, Eq. (1); §4.2; §4.8] The long-waiting conclusion (Fig. 2 top, Lwait=15 boundary) rests entirely on CAMA being the correct counterfactual for 'this token received no input-specific information from other tokens.' Eq. (1) replaces x_t^(Lwait) with a conditional expectation over the task distribution. A mean of hidden states is in general not a state the model ever produces, and downstream layers can respond to averages differently than to any actual state. Section 4.8 shows only that DEC/RTMA/SPAW/IFP fail; it does not establish that CAMA equals the true counterfactual. Please provide a validation: for example, compare CAMA states to states produced by an independent no-cross-token-computation forward pass (not based on mean replacement), measure distributional distance to natural activations, probe CAMA states for residual cross-token information, and report the number of CAMA samples and sensitivity to that
  2. [§4.2, Fig. 3] The AF1 configuration is selected from the same grid that is later used to report sufficiency: the authors grid-search Lwait and Ltransfer over the faithfulness metric, then take the best region as the discovered subgraph and derive 'necessary conditions' from the same data. This is partly self-fulfilling: the sufficiency result is constructed by the search. The independent necessity evidence in §4.4 supports the transfer-layer part, but the exact boundaries Lwait≤15 and Lwait+Ltransfer≥17 are not independently validated. Please report the configuration-selection procedure, account for multiple comparisons, and, if possible, validate the selected subgraph on a held-out set of tasks or with a fixed threshold.
  3. [Tables 2, 3, 6; Figs. 2, 3, 4] All faithfulness numbers are point estimates with no sample sizes, confidence intervals, or error bars. Claims such as 'we observe no performance degradation' (§4.2) and 'performance stays high as long as Ltransfer ≥ 2' are quantitative boundary claims and need uncertainty quantification. This matters especially for low raw-accuracy tasks (e.g., A−B+C for Llama-3-8B, raw accuracy 0.296), where faithfulness is estimated from few correct examples, and for the Pythia/GPT-J grids where the boundary is described as soft in §4.7.
  4. [Abstract vs. §4.7] The abstract claims that the AF1 subgraph 'transfers across different models,' but §4.7 does not test the AF1 llama subgraph on Pythia and GPT-J; instead, it searches for new AF1 configurations with different Lwait and Ltransfer values and reports lower faithfulness (Table 6). What transfers is the discovery methodology, not the subgraph itself. Either reword the abstract and discussion, or run the exact cross-model transfer experiment.
  5. [Abstract; §4.4] The claim that the AF1 subgraph is 'necessary for high model performance' is stronger than what is shown. Section 4.4 demonstrates that last-token attention to other tokens in layers 15–16 is necessary in the full model, and Fig. 3 gives necessary conditions for the modified subgraph family; neither establishes that the full three-stage AF1 subgraph (including CAMA waiting) is necessary for the original model. Please restrict the necessity claim to the transfer layers, or provide an experiment that tests necessity of the entire subgraph.
minor comments (6)
  1. [§4.2] The text says 'with 14 CAMA layers and 2 information transfer layers' immediately after stating Lwait=15. The number of CAMA layers should be 15 (layers 0–14) or the off-by-one should be explained.
  2. [§4.2] Typo: 'for the remaining remaining ABP layers' should read 'for the remaining ABP layers.'
  3. [§3.4; Fig. 2 caption] The descriptions say the last token 'only attends to itself' in the final stage, but K_t always contains the BOS token. Please clarify throughout that self-peeking still includes BOS attention; otherwise the notation K_t={t} conflicts with K_t←K_t∪{1}.
  4. [§4.4, Table 4] The head-removal procedure is described only qualitatively. Please specify the stopping criterion, the number of prompts used to evaluate each head, and whether the reported 'least important' removals are averaged over multiple random seeds or a single run.
  5. [Fig. 6 caption] Typo: 'Llama-8-8B' should be 'Llama-3-8B.'
  6. [Related Work] Typo in citation to Csordás et al.: 'tendancies' should be 'tendencies.'

Circularity Check

0 steps flagged

No significant circularity: the AF1 subgraph is found by direct intervention and grid search, not by fitting a parameter to the quantity later called a prediction; the necessity evidence is independent of CAMA, and the recited self-citations are background rather than load-bearing.

full rationale

I walked the paper's derivation chain looking for steps where a 'prediction' or 'first-principles result' is equivalent to its inputs by construction. CAMA (Sec. 3.3, Eq. 1) is an intervention: it replaces a hidden state with a conditional expectation over the task distribution. By construction that state does not depend on other tokens, but whether the resulting subgraph retains accuracy is an empirical fact, not a tautology. The AF1 configuration (L_wait=15, L_transfer=2) is obtained by searching the (L_wait, L_transfer) grid and then reporting the found high-faithfulness region; this is a discovery/search procedure, not a fitted parameter later relabeled as a prediction. The necessity claim for the transfer layers is supported independently by Sec. 4.4, where removing last-token attention to other tokens in layers 15/16 of the full, unablated model causes large drops. No load-bearing self-citation appears: the authors' prior work (Rai et al. 2024; Rai and Yao 2024) is cited only as related work, and the central results are tested against model outputs rather than justified by those citations. The legitimate concern that CAMA's conditional mean may not be a faithful counterfactual for 'the token received no input-specific information' is a validity or counterfactual-fidelity limitation, not a circularity of the paper's equations; Sec. 4.8 compares several alternative ablations, which further shows CAMA's behavior is not assumed into the conclusion. The self-identified limitations (tokenizer dependence, task scope) are also not circular. I find no step in which an output equation reduces to an input equation by definition, and no fitted quantity is presented as an independent prediction. Therefore the paper receives a circularity score of 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper introduces no physical entities. Its free parameters are the empirical choices defining the AF1 subgraph (Lwait, Ltransfer, pruning thresholds), and its assumptions are the counterfactual validity of CAMA, single-token number tokenization, the faithfulness metric, and the benign nature of ABP mask changes.

free parameters (4)
  • Lwait (number of CAMA waiting layers) = 15 for Llama-3-8B AF1
    Selected by grid search to be the maximal waiting length that preserves faithfulness; it defines the claim that waiting extends to layer 14.
  • Ltransfer (number of information transfer layers) = 2 for Llama-3-8B AF1
    Smallest value from the grid that keeps faithfulness high after removal of later full-peeking; central to the two-layer transfer finding.
  • Head-pruning accuracy threshold = 95% preserved accuracy
    Criterion used to decide when to stop removing heads in Sec 4.4; influences which heads are called important.
  • CAMA / attention visualization sample count = not reported (attention plots use 100 prompts)
    Number of samples used to estimate the conditional expectation and average attention patterns is unspecified in the main text, affecting precision of all reported faithfulness values.
axioms (5)
  • domain assumption CAMA conditional expectation is a faithful counterfactual for no cross-token interaction
    Eq. 1 defines the replacement; the entire waiting-stage conclusion depends on this assumption, and Sec. 4.8 only shows alternatives fail without proving CAMA equals the true counterfactual.
  • domain assumption Each operand is a single token
    Stated limitation; required for ABP and CAMA to isolate operand values; excludes Qwen/Gemma-style digit-split tokenizers.
  • domain assumption Faithfulness on full-model-correct prompts is the right evaluation
    Used throughout (Eq. 2) to measure the subgraph; conditions on correct full-model outputs, which makes strong claims about high performance only relative to the correct subset.
  • domain assumption Attention mask modification (ABP) has no side effects beyond restricting keys
    Sec. 3.4 assumes replacing scores with -inf and renormalizing preserves all other computational behavior; attention sink phenomenon forces an exception for the BOS token, showing the intervention is not entirely benign.
  • standard math Logit lens projection measures intermediate computation
    Sec. 4.5 uses unembedding projection as evidence about when the answer emerges; this is a common interpretability assumption, not proven in-paper.

pith-pipeline@v1.3.0-alltime-deepseek · 16716 in / 14015 out tokens · 142941 ms · 2026-08-04T18:44:29.547644+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) demonstrate proficiency across numerous computational tasks, yet their inner workings remain unclear. In theory, the combination of causal self-attention and multilayer perceptron layers allows every token to access and compute information based on all preceding tokens. In practice, to what extent are such operations present? In this paper, on mental math tasks (i.e., direct math calculation via next-token prediction without explicit reasoning), we investigate this question in three steps: inhibiting input-specific token computations in the initial layers, restricting the routes of information transfer across token positions in the next few layers, and forcing all computation to happen at the last token in the remaining layers. With two proposed techniques, Context-Aware Mean Ablation (CAMA) and Attention-Based Peeking (ABP), we identify an All-for-One subgraph (AF1) with high accuracy on a wide variety of mental math tasks, where meaningful computation occurs very late (in terms of layer depth) and only at the last token, which receives information of other tokens in few specific middle layers. Experiments on a variety of models and arithmetic expressions show that this subgraph is sufficient and necessary for high model performance, transfers across different models, and works on a variety of input styles. Ablations on different CAMA and ABP alternatives reveal their unique advantages over other methods, which may be of independent interest.

Figures

Figures reproduced from arXiv: 2509.09650 by Daking Rai, Siddarth Mamidanna, Yilun Zhou, Ziyu Yao.

Figure 1
Figure 1. Figure 1: The full AF1 subgraph consists of three stages. First, input-specific computation is suppressed where the input embeddings skip the first Lwait layers with context-aware mean ablation (CAMA). Then, the result￾ing activations x (Lwait) pass through Ltransfer layers of attention-based peeking (ABP) where the only cross￾token attentions are those from the last token to preced￾ing ones. Last, for the remaining… view at source ↗
Figure 2
Figure 2. Figure 2: Top: Llama-3-8B performance after making [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Faithfulness of the full Llama-3-8B but with [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Logit lens top-3 accuracy of for each attention [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Attention patterns for the three key operand [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: plots the faithfulness for different AF1 configurations on A + B for Pythia (top) and GPT￾J (bottom). There are several notable differences from that for Llama in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: AF1 performance across all models on two- and three-operand tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Attention pattern for A+B-C, at attention [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Logit lens top-3 accuracy of for each atten [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SSA: Improving Performance With a Better Scoring Function

    cs.CL 2025-08 unverdicted novelty 5.0

    Replacing Softmax with Scaled Signed Averaging in transformer attention improves generalization under distribution shifts for in-context learning and boosts results on NLP benchmarks.

Reference graph

Works this paper leans on

42 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Leonard Bereska and Stratis Gavves. 2024. https://openreview.net/forum?id=ePUVetPKu6 Mechanistic interpretability for AI safety - a review . Transactions on Machine Learning Research. Survey Certification, Expert Certification

    2024

  2. [2]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397--2430. PMLR

    2023

  3. [3]

    Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. 2024. Hopping too late: Exploring the limitations of large language models on multi-hop queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14113--14130

    2024

  4. [4]

    Nicola Cancedda. 2024. Spectral filters, dark signals, and attention sinks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4792--4808

    2024

  5. [5]

    Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldwosky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. 2022. Causal scrubbing, a method for rigorously testing interpretability hypotheses. AI Alignment Forum. https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing

    2022

  6. [6]

    Bilal Chughtai, Alan Cooney, and Neel Nanda. 2024. Summing up the facts: Additive mechanisms behind factual recall in llms. arXiv preprint arXiv:2402.07321

    Pith/arXiv arXiv 2024

  7. [7]

    Manning, and Christopher Potts

    Róbert Csordás, Christopher D. Manning, and Christopher Potts. 2025. https://arxiv.org/abs/2505.13898v2 Do language models use their depth efficiently? Preprint, arXiv:2505.13898v2

    arXiv 2025

  8. [8]

    Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. 2024. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. Trans. Mach. Learn. Res

    2024

  9. [9]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021. A mathematical framework for transformer circuits. Transformer C...

    2021

  10. [10]

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216--12235

    2023

  11. [11]

    Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. 2023. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969

    Pith/arXiv arXiv 2023

  12. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  13. [13]

    Tal Haklay, Hadas Orgad, David Bau, Aaron Mueller, and Yonatan Belinkov. 2025. https://arxiv.org/abs/2502.04577 Position-aware automatic circuit discovery . Preprint, arXiv:2502.04577

    Pith/arXiv arXiv 2025

  14. [14]

    Roee Hendel, Mor Geva, and Amir Globerson. 2023. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318--9333

    2023

  15. [15]

    Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2023. Linearity of relation decoding in transformer language models. CoRR

    2023

  16. [16]

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156--5165. PMLR

    2020

  17. [17]

    Maximilian Li and Lucas Janson. 2024. Optimal ablation for interpretability. Advances in Neural Information Processing Systems, 37:109233--109282

    2024

  18. [18]

    Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024. https://doi.org/10.18653/v1/2024.findings-acl.576 Understanding and patching compositional reasoning in LLM s . In Findings of the Association for Computational Linguistics: ACL 2024, pages 9668--9688, Bangkok, Thailand. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-acl.576 2024

  19. [19]

    Davide Maltoni and Matteo Ferrara. 2024. Arithmetic with language models: From memorization to computation. Neural Networks, 179:106550

    2024

  20. [20]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359--17372

    2022

  21. [21]

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217

    Pith/arXiv arXiv 2023

  22. [22]

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2024. Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272

    Pith/arXiv arXiv 2024

  23. [23]

    nostalgebraist. 2020. Interpreting gpt: the logit lens. AI Alignment Forum. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  24. [24]

    Daking Rai and Ziyu Yao. 2024. An investigation of neuron activation as a unified lens to explain chain-of-thought eliciting arithmetic reasoning of llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7174--7193

    2024

  25. [25]

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646

    arXiv 2024

  26. [26]

    Jie Ren, Qipeng Guo, Hang Yan, Dongrui Liu, Quanshi Zhang, Xipeng Qiu, and Dahua Lin. 2024. Identifying semantic induction heads to understand in-context learning. In Findings of the Association for Computational Linguistics ACL 2024, pages 6916--6932

    2024

  27. [27]

    Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, Andr \'e Bauer, Kyle Chard, and Ian Foster. 2023. https://doi.org/10.18653/v1/2023.blackboxnlp-1.26 Memory injections: Correcting multi-hop reasoning failures during inference in transformer-based language models . In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and I...

    work page doi:10.18653/v1/2023.blackboxnlp-1.26 2023

  28. [28]

    Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7035--7052

    2023

  29. [29]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    Pith/arXiv arXiv 2024

  30. [30]

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv:1905.05950

    Pith/arXiv arXiv 2019

  31. [31]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  32. [32]

    Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model . https://github.com/kingoflolz/mesh-transformer-jax

    2021

  33. [33]

    Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. 2024. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. In ICML 2024 Workshop on Mechanistic Interpretability

    2024

  34. [34]

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593

    Pith/arXiv arXiv 2022

  35. [35]

    Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. 2023. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in neural information processing systems, 36:78205--78226

    2023

  36. [36]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453

    Pith/arXiv arXiv 2023

  37. [37]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  38. [38]

    Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. https://doi.org/10.18653/v1/2024.acl-long.550 Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210--10229, Bangkok, Thailand. Associatio...

    doi:10.18653/v1/2024.acl-long.550 2024

  39. [39]

    Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-Ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. 2024. Interpreting and improving large language models in arithmetic calculation. In International Conference on Machine Learning, pages 59932--59950. PMLR

    2024

  40. [40]

    Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. 2024. Pre-trained large language models use fourier features to compute addition. arXiv preprint arXiv:2406.03445

    Pith/arXiv arXiv 2024

  41. [41]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  42. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...