REVIEW 3 major objections 3 minor 1 cited by
Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability
T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The IOI circuit in GPT-2 small generalizes to DoubleIO and TripleIO prompts by reusing all 26 nodes and 84–92% of its edges, adding only input paths, with the knockout artifact S2 Hacking explaining its overperformance.
desk verdict Useful but partly circular: the S2 Hacking finding is real, and the circuit-reuse claim is shaped by the discovery procedure. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the circuit-discovery and evaluation procedure of path patching with mean ablation, which identifies causally relevant attention heads and edges. Applied to the variants, the procedure reveals that the only structural change is the addition of edges from the IO2 (and IO3) tokens to the Duplicate and Previous Token heads, while all 26 nodes stay the same. The secondary mechanism is S2 Hacking, defined as the bias introduced by the knockout procedure when S2 is the only input token with surviving paths to the Duplicate and Induction heads; it carries the explanation for why the base circuit outperforms the full model without any of its heads changing function.
What would settle it
Run the same path-patching discovery for DoubleIO initialized from scratch, with no heads carried over from the base IOI circuit. If the resulting circuit contains far fewer of the base circuit's 26 nodes, or if restoring all ablated paths (removing the S2-only knockout) eliminates the circuit's logit-difference advantage over the full model, then the reuse claim and the S2 Hacking explanation would both fail.
Extended reading notes
Core claim
The central claim is that the IOI circuit in GPT-2 small generalizes to prompt variants where the original algorithm should fail, and it does so by reusing all of its components and mechanisms while adding only input edges. Concretely, the DoubleIO circuit shares 100% of nodes and 91.66% of edges with the base IOI circuit, and the TripleIO circuit shares 100% of nodes and 84.61% of edges, with the extra edges connecting duplicated IO tokens to the Duplicate and Previous Token heads. The authors also report S2 Hacking, a mechanism in which the circuit's mean-ablation evaluation leaves S2 as the only input path to the Duplicate and Induction heads, biasing S-Inhibition heads toward the subject token and letting Name Mover heads return an IO token; this explains why the base circuit outperforms the full model on the variants even though the full model is the object being explained. They further report that all head types, Name Mover, S-Inhibition, Duplicate, Previous Token, and Induction, retain their original functions, and they identify head 2.2 as a 'first come, first serve' decision point that chooses which duplicated name to suppress based on order of appearance.
Load-bearing premise
The load-bearing premise is that starting the discovery process from the base IOI circuit and restoring only its ablated paths yields the true DoubleIO and TripleIO circuits; if a search that could add completely new heads would find much lower overlap, the reuse result is an artifact of initialization.
Editorial extensions
If this is right
- Circuit explanations are not single-format: the same 26-node subgraph explains IOI across base, DoubleIO, and TripleIO prompts, so the circuit is a valid explanation for a family of related tasks rather than one template.
- The adaptation is quantitative and structured: DoubleIO adds exactly 10 edges and TripleIO adds 20, all from duplicated IO tokens, so the model generalizes by extending existing duplicate-handling machinery rather than recruiting new heads.
- Faithfulness scores above 1 in circuit evaluations flag ablation artifacts: the base circuit's 1.285 and 2.586 faithfulness on DoubleIO and TripleIO trace directly to S2 Hacking, not to better task-solving by the circuit.
- The order in which names appear determines which duplicate gets suppressed, with head 2.2 acting as the decision point; this order dependence is a concrete behavioral prediction of the discovered DoubleIO circuit.
- The authors' 'strong generalization' scenario from Figure 1 is realized: perfect node reuse and high edge reuse, which they state is the first demonstration of circuit generalization through circuit reuse.
Reading between the lines
- Because the authors start their search from the base IOI circuit and only restore paths that were originally ablated, the 100% node overlap is partly guaranteed by construction; an unconstrained search that could add new heads might report lower reuse, so the reuse claim should be read as a lower bound on the circuit's flexibility.
- S2 Hacking is an evaluation artifact, not a model behavior: it implies that any circuit-evaluation protocol that mean-ablates paths can create superhuman circuit performance, so future circuit studies should check whether faithfulness greater than 1 is produced by the ablation itself.
- The same path-patching discovery could be run on other well-studied circuits, such as Greater-Than or Arithmetic; if those also show near-total node reuse on prompt variants, then circuit reuse may be a general property of small transformers, and if not, IOI may be an unusually flexible circuit.
- A sharper test of the decision-point claim would vary the position of the duplicated IO token while holding the rest of the sentence fixed; head 2.2's attention should track the first-appearing name in every such permutation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether the indirect object identification (IOI) circuit in GPT-2 small, originally discovered by Wang et al. (2023), generalizes to two prompt variants (DoubleIO and TripleIO) that violate the assumptions of the IOI algorithm. The authors report that the base IOI circuit maintains high logit differences on these variants, explain this through a mechanism they call S2 Hacking, and then discover new circuits for the variants that reuse 100% of the nodes and 91.66%/84.61% of the edges of the base circuit. They interpret this as strong generalization through circuit reuse and claim it is the first demonstration of such reuse.
Significance. If the reuse claim were established, it would be a valuable contribution to mechanistic interpretability, as it would show that circuits can adapt to new prompt formats with only minimal structural additions. The paper is also transparent about the S2 Hacking artifact, which is a useful cautionary example of how mean-ablation evaluation can create spurious circuit behavior. However, the central generalization claim is not currently supported because the discovery procedure is constrained to start from the base circuit, and the apparent success of the base circuit on the variants is largely an artifact of the knockout procedure. The paper's genuine contributions are the S2 Hacking analysis and the demonstration that the base circuit can serve as a productive seed for finding variant circuits, but these do not establish the stronger claim of generalization through reuse.
major comments (3)
- [§5.1, Table 2] The discovery procedure starts with the base IOI circuit and only restores paths that were originally ablated, so the reported 100% node overlap and 91.66%/84.61% edge overlap are lower bounds imposed by the search space, not quantities discovered by an unconstrained search. The paper states in §5.1: 'we start with the base IOI circuit and restore some of these paths from other input tokens that were originally ablated out.' Consequently, the reuse result is built into the initialization, and the claim that the model 'reuses all components' is not supported as a discovery.
- [§4, Table 1] The observation that the base IOI circuit 'performs well' on the DoubleIO and TripleIO variants is driven by S2 Hacking, which the paper itself describes as 'a byproduct of the knockout procedure for evaluating the circuit and not actually how the full model solves the task' (§4). Therefore the high circuit logit differences in Table 1 (2.722 and 3.174) do not constitute evidence of model generalization; they are artifacts of the mean-ablation evaluation. This undermines the abstract's claim that 'the circuit generalizes surprisingly well,' because the circuit's performance is not faithful to the full model, as the faithfulness values of 1.285 and 2.586 also indicate.
- [§5.2] The claim that this is 'the first demonstration of circuit generalization through circuit reuse' is not supported, because the method does not allow an alternative circuit to be discovered. Since the search is seeded with the base circuit and only adds edges from other input tokens to the existing Duplicate and Previous Token heads, the outcome that all 26 nodes and most edges are reused is a direct consequence of the search space. To substantiate the generalization claim, the authors would need to perform unconstrained circuit discovery for the variants (e.g., running path patching over all heads and all input tokens) and show that the resulting circuit overlaps with the base circuit to the reported degree. As written, the paper demonstrates that the base circuit provides a useful seed for finding variant circuits, not that an unconstrained search would rediscover it.
minor comments (3)
- [Figure 2] The caption of Figure 2 is somewhat unclear: it says 'Deviation from model' but the figure compares the circuit and the full model, and the caption does not explain what the color scale represents or how the deviation was averaged over token positions.
- [Table 2] The paper reports edge overlap percentages but does not report confidence intervals or significance tests for the overlap or faithfulness values, even though the same analysis is based on a sample of prompts; adding such intervals would help assess variability.
- [Appendix C] The prompt templates in Appendix C use [B] and [A] to denote the subject and indirect object, while the main text uses S and IO; this inconsistency may confuse readers trying to map templates to the analysis.
Circularity Check
The circuit-reuse claim is partly built into the discovery procedure: starting from the base IOI circuit guarantees 100% node overlap and fixes edge-overlap percentages, and the 'surprising' base-circuit performance is conceded to be an artifact of mean-ablation.
-
self definitional
[Section 5.1, 'Adding Paths from Input Tokens']
"For the variants, we start with the base IOI circuit and restore some of these paths from other input tokens that were originally ablated out, and see if any of them have a causal effect on the output of the model."
The DoubleIO and TripleIO circuits are defined as the base IOI circuit plus candidate restored edges, so every candidate contains all 26 base nodes and 110 base edges by initialization. The reported 100% node overlap is therefore guaranteed, and the edge-overlap values 91.66% and 84.61% in Table 2 are exactly 110/(110+10) and 110/(110+20), the arithmetic consequence of adding 10 or 20 edges to the fixed base graph. The claim that the variant circuits 'reuse all of the heads and paths from the base IOI circuit' (Section 5.2) therefore restates the construction rather than reporting an unconstrained circuit discovery.
-
other
[Section 4, 'S2 Hacking: Performance without Faithfulness']
"Note that this phenomenon only occurs in the base IOI circuit, as it is a byproduct of the knockout procedure for evaluating the circuit and not actually how the full model solves the task."
The paper's evidence that the base IOI circuit 'generalizes surprisingly well' on DoubleIO and TripleIO comes from mean-ablating all non-S2 input paths, which forces the circuit to attend to S2 and return the IO token. Because the paper concedes this is not how the full model solves the task, the high variant logit differences are properties of the evaluation procedure, not of the model's algorithm. This artifact is then called a mechanism ('S2 Hacking') and used as the seed for finding the variant circuits, so the generalization evidence loops back to the ablation procedure rather than to independently observed model behavior.
full rationale
The strongest overlap claims are circular: since Section 5.1 starts the variant-circuit search from the base IOI circuit and only restores ablated edges, 100% node overlap is true by construction and the edge-overlap percentages are simply base-edge count divided by base-plus-added-edge count. Section 4 explicitly concedes that the base circuit's high performance on the variants is a byproduct of mean-ablation knockout rather than the full model's computation, so that behavioral evidence cannot support the reuse claim. The paper does retain some independent empirical content: it measures direct causal effects for Name Mover and Inhibition heads (Section 5.2), and it empirically selects which input-token paths improve faithfulness (Section 5.1). The Wang et al. (2023) citations are external prior work rather than a self-citation chain. Because the central reuse result is substantially determined by the discovery procedure but not entirely devoid of independent measurements, the appropriate score is 6.
Assumptions & free parameters
free parameters (1)
- Causal effect significance threshold =
not reported
assumptions (4)
- domain assumption The base IOI circuit from Wang et al. (2023) correctly describes GPT-2 small's behavior on the original IOI prompt format.
- domain assumption Mean ablation (replacing non-circuit activations with means) is a valid way to evaluate a circuit's causal contribution.
- domain assumption Direct causal effect scores from path patching identify the heads and paths that are causally relevant for a task.
- domain assumption The generated datasets of 200 prompts per variant are representative of the variant task.
invented entities (1)
-
S2 Hacking
Cite this review
Pith. "Pith review of Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability." pith.science (2026). https://pith.science/paper/NZG3GDNR
@misc{pith2026241116105,
author = {Pith},
title = {Pith review of: Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability},
year = {2026},
howpublished = {\url{https://pith.science/paper/NZG3GDNR}},
note = {Machine review of arXiv:2411.16105}
}
read the original abstract
Mechanistic interpretability aims to understand the inner workings of large neural networks by identifying circuits, or minimal subgraphs within the model that implement algorithms responsible for performing specific tasks. These circuits are typically discovered and analyzed using a narrowly defined prompt format. However, given the abilities of large language models (LLMs) to generalize across various prompt formats for the same task, it remains unclear how well these circuits generalize. For instance, it is unclear whether the models generalization results from reusing the same circuit components, the components behaving differently, or the use of entirely different components. In this paper, we investigate the generality of the indirect object identification (IOI) circuit in GPT-2 small, which is well-studied and believed to implement a simple, interpretable algorithm. We evaluate its performance on prompt variants that challenge the assumptions of this algorithm. Our findings reveal that the circuit generalizes surprisingly well, reusing all of its components and mechanisms while only adding additional input edges. Notably, the circuit generalizes even to prompt variants where the original algorithm should fail; we discover a mechanism that explains this which we term S2 Hacking. Our findings indicate that circuits within LLMs may be more flexible and general than previously recognized, underscoring the importance of studying circuit generalization to better understand the broader capabilities of these models.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 1 Pith paper
-
From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits
GPT-2 small performs syllogisms through truth-copying attention heads and a suppression-plus-MLP pathway that can output a negated truth value.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
A mathematical framework for transformer circuits
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...
2021
-
[3]
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023. URL https://arxiv.org/abs/2305.00586
arXiv 2023
-
[4]
Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=TZ0CCGDcuT
2024
-
[5]
Zoom in: An introduction to circuits
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in
-
[6]
Fine-tuning enhances existing mechanisms: A case study on entity tracking, 2024
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking, 2024
2024
-
[7]
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://api.semanticscholar.org/CorpusID:258865170
work page 2023
-
[8]
Investigating gender bias in language models using causal mediation analysis
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33: 0 12388--12401, 2020
2020
Show all 12 references
-
[9]
Interpretability in the wild: a circuit for indirect object identification in GPT -2 small
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT -2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevi...
2023
-
[10]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[11]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[12]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.