Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability

T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read The IOI circuit in GPT-2 small generalizes to DoubleIO and TripleIO prompts by reusing all 26 nodes and 84–92% of its edges, adding only input paths, with the knockout artifact S2 Hacking explaining its overperformance.

desk verdict Useful but partly circular: the S2 Hacking finding is real, and the circuit-reuse claim is shaped by the discovery procedure. read the letter →

arxiv 2411.16105 v2 pith:NZG3GDNR submitted 2024-11-25 cs.LG cs.AIcs.CL

classification cs.LGcs.AIcs.CL
keywords mechanisticinterpretabilitycircuitgeneralizationindirectobjectidentificationGPT-2smallS2Hackingpathpatchingreuselogitdifference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a circuit discovered on one prompt format still explains a model's behavior when the prompt format changes. Using GPT-2 small's indirect object identification (IOI) circuit as a test case, the authors construct two variants, DoubleIO and TripleIO, that should break the circuit's hypothesized 'suppress the duplicated name' algorithm. They report that the base circuit still solves both variants, that a knockout artifact they call S2 Hacking explains its overperformance, and that newly discovered circuits for the variants reuse 100% of the base circuit's nodes and 91.66% and 84.61% of its edges, adding only input edges from the extra duplicate tokens. If correct, this would be the first demonstration of circuit generalization through circuit reuse: the same subgraph, with small structured additions, explains a family of related tasks. This matters because mechanistic interpretability explanations are usually validated on one prompt format, and this result suggests those explanations can carry over to broader behavior.

What carries the argument

The load-bearing machinery is the circuit-discovery and evaluation procedure of path patching with mean ablation, which identifies causally relevant attention heads and edges. Applied to the variants, the procedure reveals that the only structural change is the addition of edges from the IO2 (and IO3) tokens to the Duplicate and Previous Token heads, while all 26 nodes stay the same. The secondary mechanism is S2 Hacking, defined as the bias introduced by the knockout procedure when S2 is the only input token with surviving paths to the Duplicate and Induction heads; it carries the explanation for why the base circuit outperforms the full model without any of its heads changing function.

What would settle it

Run the same path-patching discovery for DoubleIO initialized from scratch, with no heads carried over from the base IOI circuit. If the resulting circuit contains far fewer of the base circuit's 26 nodes, or if restoring all ablated paths (removing the S2-only knockout) eliminates the circuit's logit-difference advantage over the full model, then the reuse claim and the S2 Hacking explanation would both fail.

Watch

Extended reading notes

Core claim

The central claim is that the IOI circuit in GPT-2 small generalizes to prompt variants where the original algorithm should fail, and it does so by reusing all of its components and mechanisms while adding only input edges. Concretely, the DoubleIO circuit shares 100% of nodes and 91.66% of edges with the base IOI circuit, and the TripleIO circuit shares 100% of nodes and 84.61% of edges, with the extra edges connecting duplicated IO tokens to the Duplicate and Previous Token heads. The authors also report S2 Hacking, a mechanism in which the circuit's mean-ablation evaluation leaves S2 as the only input path to the Duplicate and Induction heads, biasing S-Inhibition heads toward the subject token and letting Name Mover heads return an IO token; this explains why the base circuit outperforms the full model on the variants even though the full model is the object being explained. They further report that all head types, Name Mover, S-Inhibition, Duplicate, Previous Token, and Induction, retain their original functions, and they identify head 2.2 as a 'first come, first serve' decision point that chooses which duplicated name to suppress based on order of appearance.

Load-bearing premise

The load-bearing premise is that starting the discovery process from the base IOI circuit and restoring only its ablated paths yields the true DoubleIO and TripleIO circuits; if a search that could add completely new heads would find much lower overlap, the reuse result is an artifact of initialization.

Editorial extensions

If this is right

  • Circuit explanations are not single-format: the same 26-node subgraph explains IOI across base, DoubleIO, and TripleIO prompts, so the circuit is a valid explanation for a family of related tasks rather than one template.
  • The adaptation is quantitative and structured: DoubleIO adds exactly 10 edges and TripleIO adds 20, all from duplicated IO tokens, so the model generalizes by extending existing duplicate-handling machinery rather than recruiting new heads.
  • Faithfulness scores above 1 in circuit evaluations flag ablation artifacts: the base circuit's 1.285 and 2.586 faithfulness on DoubleIO and TripleIO trace directly to S2 Hacking, not to better task-solving by the circuit.
  • The order in which names appear determines which duplicate gets suppressed, with head 2.2 acting as the decision point; this order dependence is a concrete behavioral prediction of the discovered DoubleIO circuit.
  • The authors' 'strong generalization' scenario from Figure 1 is realized: perfect node reuse and high edge reuse, which they state is the first demonstration of circuit generalization through circuit reuse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the authors start their search from the base IOI circuit and only restore paths that were originally ablated, the 100% node overlap is partly guaranteed by construction; an unconstrained search that could add new heads might report lower reuse, so the reuse claim should be read as a lower bound on the circuit's flexibility.
  • S2 Hacking is an evaluation artifact, not a model behavior: it implies that any circuit-evaluation protocol that mean-ablates paths can create superhuman circuit performance, so future circuit studies should check whether faithfulness greater than 1 is produced by the ablation itself.
  • The same path-patching discovery could be run on other well-studied circuits, such as Greater-Than or Arithmetic; if those also show near-total node reuse on prompt variants, then circuit reuse may be a general property of small transformers, and if not, IOI may be an unusually flexible circuit.
  • A sharper test of the decision-point claim would vary the position of the duplicated IO token while holding the rest of the sentence fixed; head 2.2's attention should track the first-appearing name in every such permutation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper investigates whether the indirect object identification (IOI) circuit in GPT-2 small, originally discovered by Wang et al. (2023), generalizes to two prompt variants (DoubleIO and TripleIO) that violate the assumptions of the IOI algorithm. The authors report that the base IOI circuit maintains high logit differences on these variants, explain this through a mechanism they call S2 Hacking, and then discover new circuits for the variants that reuse 100% of the nodes and 91.66%/84.61% of the edges of the base circuit. They interpret this as strong generalization through circuit reuse and claim it is the first demonstration of such reuse.

Significance. If the reuse claim were established, it would be a valuable contribution to mechanistic interpretability, as it would show that circuits can adapt to new prompt formats with only minimal structural additions. The paper is also transparent about the S2 Hacking artifact, which is a useful cautionary example of how mean-ablation evaluation can create spurious circuit behavior. However, the central generalization claim is not currently supported because the discovery procedure is constrained to start from the base circuit, and the apparent success of the base circuit on the variants is largely an artifact of the knockout procedure. The paper's genuine contributions are the S2 Hacking analysis and the demonstration that the base circuit can serve as a productive seed for finding variant circuits, but these do not establish the stronger claim of generalization through reuse.

major comments (3)
  1. [§5.1, Table 2] The discovery procedure starts with the base IOI circuit and only restores paths that were originally ablated, so the reported 100% node overlap and 91.66%/84.61% edge overlap are lower bounds imposed by the search space, not quantities discovered by an unconstrained search. The paper states in §5.1: 'we start with the base IOI circuit and restore some of these paths from other input tokens that were originally ablated out.' Consequently, the reuse result is built into the initialization, and the claim that the model 'reuses all components' is not supported as a discovery.
  2. [§4, Table 1] The observation that the base IOI circuit 'performs well' on the DoubleIO and TripleIO variants is driven by S2 Hacking, which the paper itself describes as 'a byproduct of the knockout procedure for evaluating the circuit and not actually how the full model solves the task' (§4). Therefore the high circuit logit differences in Table 1 (2.722 and 3.174) do not constitute evidence of model generalization; they are artifacts of the mean-ablation evaluation. This undermines the abstract's claim that 'the circuit generalizes surprisingly well,' because the circuit's performance is not faithful to the full model, as the faithfulness values of 1.285 and 2.586 also indicate.
  3. [§5.2] The claim that this is 'the first demonstration of circuit generalization through circuit reuse' is not supported, because the method does not allow an alternative circuit to be discovered. Since the search is seeded with the base circuit and only adds edges from other input tokens to the existing Duplicate and Previous Token heads, the outcome that all 26 nodes and most edges are reused is a direct consequence of the search space. To substantiate the generalization claim, the authors would need to perform unconstrained circuit discovery for the variants (e.g., running path patching over all heads and all input tokens) and show that the resulting circuit overlaps with the base circuit to the reported degree. As written, the paper demonstrates that the base circuit provides a useful seed for finding variant circuits, not that an unconstrained search would rediscover it.
minor comments (3)
  1. [Figure 2] The caption of Figure 2 is somewhat unclear: it says 'Deviation from model' but the figure compares the circuit and the full model, and the caption does not explain what the color scale represents or how the deviation was averaged over token positions.
  2. [Table 2] The paper reports edge overlap percentages but does not report confidence intervals or significance tests for the overlap or faithfulness values, even though the same analysis is based on a sample of prompts; adding such intervals would help assess variability.
  3. [Appendix C] The prompt templates in Appendix C use [B] and [A] to denote the subject and indirect object, while the main text uses S and IO; this inconsistency may confuse readers trying to map templates to the analysis.

Circularity Check

2 steps flagged · score 6.0 of 10

The circuit-reuse claim is partly built into the discovery procedure: starting from the base IOI circuit guarantees 100% node overlap and fixes edge-overlap percentages, and the 'surprising' base-circuit performance is conceded to be an artifact of mean-ablation.

  1. self definitional [Section 5.1, 'Adding Paths from Input Tokens']
    "For the variants, we start with the base IOI circuit and restore some of these paths from other input tokens that were originally ablated out, and see if any of them have a causal effect on the output of the model."

    The DoubleIO and TripleIO circuits are defined as the base IOI circuit plus candidate restored edges, so every candidate contains all 26 base nodes and 110 base edges by initialization. The reported 100% node overlap is therefore guaranteed, and the edge-overlap values 91.66% and 84.61% in Table 2 are exactly 110/(110+10) and 110/(110+20), the arithmetic consequence of adding 10 or 20 edges to the fixed base graph. The claim that the variant circuits 'reuse all of the heads and paths from the base IOI circuit' (Section 5.2) therefore restates the construction rather than reporting an unconstrained circuit discovery.

  2. other [Section 4, 'S2 Hacking: Performance without Faithfulness']
    "Note that this phenomenon only occurs in the base IOI circuit, as it is a byproduct of the knockout procedure for evaluating the circuit and not actually how the full model solves the task."

    The paper's evidence that the base IOI circuit 'generalizes surprisingly well' on DoubleIO and TripleIO comes from mean-ablating all non-S2 input paths, which forces the circuit to attend to S2 and return the IO token. Because the paper concedes this is not how the full model solves the task, the high variant logit differences are properties of the evaluation procedure, not of the model's algorithm. This artifact is then called a mechanism ('S2 Hacking') and used as the seed for finding the variant circuits, so the generalization evidence loops back to the ablation procedure rather than to independently observed model behavior.

full rationale

The strongest overlap claims are circular: since Section 5.1 starts the variant-circuit search from the base IOI circuit and only restores ablated edges, 100% node overlap is true by construction and the edge-overlap percentages are simply base-edge count divided by base-plus-added-edge count. Section 4 explicitly concedes that the base circuit's high performance on the variants is a byproduct of mean-ablation knockout rather than the full model's computation, so that behavioral evidence cannot support the reuse claim. The paper does retain some independent empirical content: it measures direct causal effects for Name Mover and Inhibition heads (Section 5.2), and it empirically selects which input-token paths improve faithfulness (Section 5.1). The Wang et al. (2023) citations are external prior work rather than a self-citation chain. Because the central reuse result is substantially determined by the discovery procedure but not entirely devoid of independent measurements, the appropriate score is 6.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claims rest on the correctness of the prior IOI circuit, on the mean-ablation evaluation method (which the paper shows can produce artifacts), and on a circuit discovery procedure that seeds the search with the base circuit. These are substantial assumptions that shape the quantitative results.

free parameters (1)
  • Causal effect significance threshold = not reported
    The circuit discovery procedure includes paths whose direct causal effect exceeds an implicit significance threshold; the threshold is not specified, so the reported edge additions depend on a hand-chosen criterion.
assumptions (4)
  • domain assumption The base IOI circuit from Wang et al. (2023) correctly describes GPT-2 small's behavior on the original IOI prompt format.
    The entire generalization study assumes this circuit is a faithful explanation for the base task; if it is not, the overlap and S2 Hacking analyses inherit that error.
  • domain assumption Mean ablation (replacing non-circuit activations with means) is a valid way to evaluate a circuit's causal contribution.
    The paper's circuit performance numbers and the S2 Hacking discovery both depend on this. The paper itself demonstrates that this procedure can introduce behaviors absent from the full model, so the validity of the assumption is questionable.
  • domain assumption Direct causal effect scores from path patching identify the heads and paths that are causally relevant for a task.
    The new circuits for DoubleIO and TripleIO are discovered by comparing direct causal effects; the paper does not validate this against an independent ground-truth mechanism.
  • domain assumption The generated datasets of 200 prompts per variant are representative of the variant task.
    All quantitative claims use these datasets; no diversity statistics or out-of-distribution checks are given.
invented entities (1)
  • S2 Hacking
    purpose: Explains why the ablated base IOI circuit achieves high logit difference on DoubleIO and TripleIO variants: only the S2 token has non-ablated paths into duplicate/induction/inhibition heads, so the circuit always suppresses subject tokens and returns an IO token.
    The paper itself calls S2 Hacking a 'byproduct of the knockout procedure' (Section 4), so it is not an independently verified mechanism in the full model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability." pith.science (2026). https://pith.science/paper/NZG3GDNR

@misc{pith2026241116105,
  author       = {Pith},
  title        = {Pith review of: Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NZG3GDNR}},
  note         = {Machine review of arXiv:2411.16105}
}
read the original abstract

Mechanistic interpretability aims to understand the inner workings of large neural networks by identifying circuits, or minimal subgraphs within the model that implement algorithms responsible for performing specific tasks. These circuits are typically discovered and analyzed using a narrowly defined prompt format. However, given the abilities of large language models (LLMs) to generalize across various prompt formats for the same task, it remains unclear how well these circuits generalize. For instance, it is unclear whether the models generalization results from reusing the same circuit components, the components behaving differently, or the use of entirely different components. In this paper, we investigate the generality of the indirect object identification (IOI) circuit in GPT-2 small, which is well-studied and believed to implement a simple, interpretable algorithm. We evaluate its performance on prompt variants that challenge the assumptions of this algorithm. Our findings reveal that the circuit generalizes surprisingly well, reusing all of its components and mechanisms while only adding additional input edges. Notably, the circuit generalizes even to prompt variants where the original algorithm should fail; we discover a mechanism that explains this which we term S2 Hacking. Our findings indicate that circuits within LLMs may be more flexible and general than previously recognized, underscoring the importance of studying circuit generalization to better understand the broader capabilities of these models.

Figures

Figures reproduced from arXiv: 2411.16105 by the authors.

Figure 1
Figure 1. Left: Different scenarios for the degree to which a circuit could change as the task format changes. Right: The IOI algorithm (top) and the result of applying that algorithm to the DoubleIO prompt variant (bottom) where the subject and indirect object tokens are both duplicated. a general and reliable explanation for how the model performs the task [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Deviation in attention scores from base IOI inputs to DoubleIO (left) and TripleIO (right) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. S2 Hacking in S-Inhibition head 8.6. Left: Attention pattern at the END position for a DoubleIO prompt. Placing all attention on the S2 token would lead to near-perfect accuracy on the task. Head 8.6 splits attention between IO2 and S2 in the full model, but in the base IOI circuit it focuses primarily on S2. Right: Knockout procedure for evaluating circuits, where paths that are not part of the circuit (marked in b… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Left: Confidence ratios for model and base IOI circuit. S2 Hacking can be seen in heads 8.6, 5.5, 5.9, and 3.0, where confidence ratio is close to 1 for the model but greater than 1 for the circuit. Right: Functional faithfulness scores for the S and IO tokens. The out…
Figure 5
Figure 5. Figure 5: Logit difference and normalized faithfulness for DoubleIO (left) and TripleIO (right) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Direct causal effect of all heads in the model on the values of Inhibition heads from the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Left: Circuit discovered for DoubleIO. All nodes and edges from the base IOI circuit are reused, with additional edges to the IO Duplicate Handling Sub-circuit. Bottom Right: Duplicate Handling Sub-circuit, which appears twice in the circuit to deal with the two duplic…
Figure 8
Figure 8. Figure 8: Left: Performance of the DoubleIO circuit and full model based on the order of appearance of S and IO in the prompt. Both perform better when IO appears first. Right: Average attention scores for head 2.2, which places much more attention on the first name that appears…
Figure 9
Figure 9. Figure 9: Attention scores of Layer 4 Head 11 indicate that the Previous Token Head attends equally [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Comparison of head 2.2 attention patterns for different input orders. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Difference between the absolute direct effects between the S2 and IO2 edges of the same [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: We see that for head 4.11, both Subject and IO edges have high effect, but for 2.2 only [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Change in direct effect from subject to IO token for Induction and Duplicate Token heads. [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Indirect Object Identification to Syllogisms: Exploring Binary Mechanisms in Transformer Circuits

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GPT-2 small performs syllogisms through truth-copying attention heads and a suppression-plus-MLP pathway that can output a negated truth value.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

  3. [3]

    How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023

    Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023. URL https://arxiv.org/abs/2305.00586

  4. [4]

    Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=TZ0CCGDcuT

  5. [5]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

  6. [6]

    Fine-tuning enhances existing mechanisms: A case study on entity tracking, 2024

    Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking, 2024

  7. [7]

    A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis

    Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://api.semanticscholar.org/CorpusID:258865170

  8. [8]

    Investigating gender bias in language models using causal mediation analysis

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33: 0 12388--12401, 2020

Show all 12 references
  1. [9]

    Interpretability in the wild: a circuit for indirect object identification in GPT -2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT -2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevi...

  2. [10]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  3. [11]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  4. [12]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.