{"id":"5e4b3c95-d8da-4309-a9e5-e98ca2a6f052","arxiv_id":"2411.16105","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-2 small's IOI circuit adapts to new prompt formats mostly by reusing components, but the base circuit's apparent success is partly an evaluation artifact called S2 Hacking.","lead":"This paper tests whether GPT-2 small's indirect object identification circuit still works when prompts are changed so that the original algorithm should fail. It finds the circuit reuses its components but often succeeds through an artifact of how circuits are evaluated, which the authors call S2 Hacking.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The circuit-reuse result is partly built into the discovery procedure: starting from the base IOI circuit and only restoring ablated edges guarantees 100% node overlap and a high edge-overlap floor, so unconstrained discovery is needed to support the generalization claim.","rationale":"The reader's weakest assumption identifies exactly this issue. The paper is transparent about the seeding and about S2 Hacking, which is a point in its favor, but the headline generalization claim needs the unconstrained check. S2 Hacking itself is a useful, falsifiable mechanistic finding. The proposed test is computationally feasible on GPT-2 small. Until then, the appropriate verdict is conditional, not reject: the paper's own constrained experiments still show that the base circuit is a good scaffold, and the S2 Hacking mechanism is independent of the reuse claim.","tokens_in":13654,"tokens_out":6444,"duration_ms":59447,"concrete_test":"Run an unconstrained circuit discovery for DoubleIO and TripleIO, e.g., ACDC or full path patching over all heads and all input-token paths, initialized from an empty circuit rather than from the base IOI circuit, with the same faithfulness and edge-overlap metrics. If the unconstrained circuits still share 100% of nodes and roughly 92%/85% of edges with the base IOI circuit, the reuse claim survives; if overlap drops substantially, the reported reuse is an artifact of seeding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 states the discovery procedure explicitly: 'For the variants, we start with the base IOI circuit and restore some of these paths from other input tokens that were originally ablated out, and see if any of them have a causal effect on the output of the model.' The search space is therefore the base circuit plus candidate edges from other input tokens to the existing Duplicate and Previous Token heads. All 26 nodes and the 110 base edges are included by construction, so the reported 100% node overlap and 91.66%/84.61% edge overlap in Table 2 are floors imposed by the initialization, not quantities discovered by an unconstrained search. This matters because the central claim is that the model generalizes by reusing the IOI circuit. Section 4 also concedes that the base circuit's high variant performance comes from S2 Hacking, 'a byproduct of the knockout procedure for evaluating the circuit and not actually how the full model solves the task,' so that part of the evidence cannot support the reuse claim either. As written, the paper demonstrates that the IOI circuit can serve as a useful seed for finding variant circuits, not that an unconstrained search would rediscover it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether the indirect object identification (IOI) circuit in GPT-2 small, originally discovered by Wang et al. (2023), generalizes to two prompt variants (DoubleIO and TripleIO) that violate the assumptions of the IOI algorithm. The authors report that the base IOI circuit maintains high logit differences on these variants, explain this through a mechanism they call S2 Hacking, and then discover new circuits for the variants that reuse 100% of the nodes and 91.66%/84.61% of the edges of the base circuit. They interpret this as strong generalization through circuit reuse and claim it is the first demonstration of such reuse.","tokens_in":13880,"tokens_out":2941,"duration_ms":27133,"significance":"If the reuse claim were established, it would be a valuable contribution to mechanistic interpretability, as it would show that circuits can adapt to new prompt formats with only minimal structural additions. The paper is also transparent about the S2 Hacking artifact, which is a useful cautionary example of how mean-ablation evaluation can create spurious circuit behavior. However, the central generalization claim is not currently supported because the discovery procedure is constrained to start from the base circuit, and the apparent success of the base circuit on the variants is largely an artifact of the knockout procedure. The paper's genuine contributions are the S2 Hacking analysis and the demonstration that the base circuit can serve as a productive seed for finding variant circuits, but these do not establish the stronger claim of generalization through reuse.","major_comments":[{"comment":"The discovery procedure starts with the base IOI circuit and only restores paths that were originally ablated, so the reported 100% node overlap and 91.66%/84.61% edge overlap are lower bounds imposed by the search space, not quantities discovered by an unconstrained search. The paper states in §5.1: 'we start with the base IOI circuit and restore some of these paths from other input tokens that were originally ablated out.' Consequently, the reuse result is built into the initialization, and the claim that the model 'reuses all components' is not supported as a discovery.","section":"§5.1, Table 2"},{"comment":"The observation that the base IOI circuit 'performs well' on the DoubleIO and TripleIO variants is driven by S2 Hacking, which the paper itself describes as 'a byproduct of the knockout procedure for evaluating the circuit and not actually how the full model solves the task' (§4). Therefore the high circuit logit differences in Table 1 (2.722 and 3.174) do not constitute evidence of model generalization; they are artifacts of the mean-ablation evaluation. This undermines the abstract's claim that 'the circuit generalizes surprisingly well,' because the circuit's performance is not faithful to the full model, as the faithfulness values of 1.285 and 2.586 also indicate.","section":"§4, Table 1"},{"comment":"The claim that this is 'the first demonstration of circuit generalization through circuit reuse' is not supported, because the method does not allow an alternative circuit to be discovered. Since the search is seeded with the base circuit and only adds edges from other input tokens to the existing Duplicate and Previous Token heads, the outcome that all 26 nodes and most edges are reused is a direct consequence of the search space. To substantiate the generalization claim, the authors would need to perform unconstrained circuit discovery for the variants (e.g., running path patching over all heads and all input tokens) and show that the resulting circuit overlaps with the base circuit to the reported degree. As written, the paper demonstrates that the base circuit provides a useful seed for finding variant circuits, not that an unconstrained search would rediscover it.","section":"§5.2"}],"minor_comments":[{"comment":"The caption of Figure 2 is somewhat unclear: it says 'Deviation from model' but the figure compares the circuit and the full model, and the caption does not explain what the color scale represents or how the deviation was averaged over token positions.","section":"Figure 2"},{"comment":"The paper reports edge overlap percentages but does not report confidence intervals or significance tests for the overlap or faithfulness values, even though the same analysis is based on a sample of prompts; adding such intervals would help assess variability.","section":"Table 2"},{"comment":"The prompt templates in Appendix C use [B] and [A] to denote the subject and indirect object, while the main text uses S and IO; this inconsistency may confuse readers trying to map templates to the analysis.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper's framing overclaims novelty given the constrained search. The S2 Hacking finding is interesting and the authors are honest about its artifact nature, but the 'circuit generalization' claim needs either a much more careful wording or an unconstrained discovery experiment. The paper may be better positioned as a cautionary study about circuit evaluation artifacts and a proof-of-concept for seed-based circuit search, rather than as the first demonstration of circuit reuse."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful paper, but the headline claim is partly built into the method. The S2 Hacking result is real and worth knowing; the 'circuit reuse' result is less convincing than it looks.\n\nWhat's new: they test the IOI circuit on DoubleIO and TripleIO, find that the base circuit outperforms the full model, trace this to mean ablation concentrating information at S2 (S2 Hacking), and then discover variant circuits by restoring edges from IO2/IO3. They also identify a decision point in head 2.2. The paper is transparent about these steps—it explicitly labels S2 Hacking an artifact—and the quantification of overlap in Table 2 is useful.\n\nThe soft spot: the discovery procedure in Section 5.1 starts from the base IOI circuit and only restores previously ablated paths. That guarantees 100% node overlap and sets a high floor on edge overlap. So the 'first demonstration of circuit generalization through circuit reuse' is overstated; it demonstrates that the base circuit is a good seed, not that an unconstrained search would rediscover it. And because the base circuit's high variant performance comes from S2 Hacking, the earlier 'generalizes surprisingly well' observation doesn't support the reuse claim either. The paper itself admits both of these points, which is good, but the abstract and conclusion still push the stronger interpretation.\n\nA second, lesser issue: head-level 'functionality' is inferred from attention patterns only. Attention is a proxy; they don't show that the outputs of these heads have the same causal effect. The faithfulness numbers are moderate (0.77–0.79), not strong evidence of crisp reuse.\n\nNet: the paper has a solid cautionary story and a replicable methodology. The main claim needs unconstrained discovery, or at least a much more careful framing. I'd send it to peer review with a request for that revision—the current version would probably be rejected if the editor only checks the abstract, but it's worth refereeing because the S2 Hacking mechanism is a real contribution and the overlap analysis is useful.\n\nReading group: yes. I'd cite it for S2 Hacking and for the seed-initialization caveat.","headline":"Useful but partly circular: the S2 Hacking finding is real, and the circuit-reuse claim is shaped by the discovery procedure.","tokens_in":14407,"tokens_out":2196,"would_cite":true,"duration_ms":20248,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The IOI circuit in GPT-2 small generalizes to DoubleIO and TripleIO prompts by reusing all 26 nodes and 84–92% of its edges, adding only input paths, with the knockout artifact S2 Hacking explaining its overperformance.","keywords":["mechanistic interpretability","circuit generalization","indirect object identification","GPT-2 small","S2 Hacking","path patching","circuit reuse","logit difference"],"falsifier":"Run the same path-patching discovery for DoubleIO initialized from scratch, with no heads carried over from the base IOI circuit. If the resulting circuit contains far fewer of the base circuit's 26 nodes, or if restoring all ablated paths (removing the S2-only knockout) eliminates the circuit's logit-difference advantage over the full model, then the reuse claim and the S2 Hacking explanation would both fail.","tokens_in":13462,"feed_emoji":"🧠","tokens_out":11186,"duration_ms":86854,"temperature":0.7,"pith_summary":"The paper asks whether a circuit discovered on one prompt format still explains a model's behavior when the prompt format changes. Using GPT-2 small's indirect object identification (IOI) circuit as a test case, the authors construct two variants, DoubleIO and TripleIO, that should break the circuit's hypothesized 'suppress the duplicated name' algorithm. They report that the base circuit still solves both variants, that a knockout artifact they call S2 Hacking explains its overperformance, and that newly discovered circuits for the variants reuse 100% of the base circuit's nodes and 91.66% and 84.61% of its edges, adding only input edges from the extra duplicate tokens. If correct, this would be the first demonstration of circuit generalization through circuit reuse: the same subgraph, with small structured additions, explains a family of related tasks. This matters because mechanistic interpretability explanations are usually validated on one prompt format, and this result suggests those explanations can carry over to broader behavior.","feed_headline":"IOI circuit reuses every node on harder prompts","feed_subtitle":"DoubleIO and TripleIO circuits keep all 26 nodes and 84–92% of edges, adding only input paths.","key_machinery":"The load-bearing machinery is the circuit-discovery and evaluation procedure of path patching with mean ablation, which identifies causally relevant attention heads and edges. Applied to the variants, the procedure reveals that the only structural change is the addition of edges from the IO2 (and IO3) tokens to the Duplicate and Previous Token heads, while all 26 nodes stay the same. The secondary mechanism is S2 Hacking, defined as the bias introduced by the knockout procedure when S2 is the only input token with surviving paths to the Duplicate and Induction heads; it carries the explanation for why the base circuit outperforms the full model without any of its heads changing function.","core_discovery":"The central claim is that the IOI circuit in GPT-2 small generalizes to prompt variants where the original algorithm should fail, and it does so by reusing all of its components and mechanisms while adding only input edges. Concretely, the DoubleIO circuit shares 100% of nodes and 91.66% of edges with the base IOI circuit, and the TripleIO circuit shares 100% of nodes and 84.61% of edges, with the extra edges connecting duplicated IO tokens to the Duplicate and Previous Token heads. The authors also report S2 Hacking, a mechanism in which the circuit's mean-ablation evaluation leaves S2 as the only input path to the Duplicate and Induction heads, biasing S-Inhibition heads toward the subject token and letting Name Mover heads return an IO token; this explains why the base circuit outperforms the full model on the variants even though the full model is the object being explained. They further report that all head types, Name Mover, S-Inhibition, Duplicate, Previous Token, and Induction, retain their original functions, and they identify head 2.2 as a 'first come, first serve' decision point that chooses which duplicated name to suppress based on order of appearance.","pith_inferences":["Because the authors start their search from the base IOI circuit and only restore paths that were originally ablated, the 100% node overlap is partly guaranteed by construction; an unconstrained search that could add new heads might report lower reuse, so the reuse claim should be read as a lower bound on the circuit's flexibility.","S2 Hacking is an evaluation artifact, not a model behavior: it implies that any circuit-evaluation protocol that mean-ablates paths can create superhuman circuit performance, so future circuit studies should check whether faithfulness greater than 1 is produced by the ablation itself.","The same path-patching discovery could be run on other well-studied circuits, such as Greater-Than or Arithmetic; if those also show near-total node reuse on prompt variants, then circuit reuse may be a general property of small transformers, and if not, IOI may be an unusually flexible circuit.","A sharper test of the decision-point claim would vary the position of the duplicated IO token while holding the rest of the sentence fixed; head 2.2's attention should track the first-appearing name in every such permutation."],"forward_implications":["Circuit explanations are not single-format: the same 26-node subgraph explains IOI across base, DoubleIO, and TripleIO prompts, so the circuit is a valid explanation for a family of related tasks rather than one template.","The adaptation is quantitative and structured: DoubleIO adds exactly 10 edges and TripleIO adds 20, all from duplicated IO tokens, so the model generalizes by extending existing duplicate-handling machinery rather than recruiting new heads.","Faithfulness scores above 1 in circuit evaluations flag ablation artifacts: the base circuit's 1.285 and 2.586 faithfulness on DoubleIO and TripleIO trace directly to S2 Hacking, not to better task-solving by the circuit.","The order in which names appear determines which duplicate gets suppressed, with head 2.2 acting as the decision point; this order dependence is a concrete behavioral prediction of the discovered DoubleIO circuit.","The authors' 'strong generalization' scenario from Figure 1 is realized: perfect node reuse and high edge reuse, which they state is the first demonstration of circuit generalization through circuit reuse."],"supporting_citations":[{"why":"Supplies the base IOI circuit, the head-type taxonomy, the path-patching and mean-ablation methodology, and the prompt-generation strategy that this paper extends to DoubleIO and TripleIO.","marker":"Wang et al. (2023)"},{"why":"Provides the mathematical setup of transformers as composed of attention heads and residual paths, which defines how circuits are identified and composed.","marker":"Elhage et al. (2021)"},{"why":"Establishes the notion of circuits as minimal subgraphs faithfully representing model behavior, the object this paper evaluates for generalization.","marker":"Olah et al. (2020)"},{"why":"The Greater-Than circuit study whose prompt-variant evaluation and overlap comparison this paper builds on as prior evidence of circuit generalization.","marker":"Hanna et al. (2023)"},{"why":"The Arithmetic circuit study that similarly evaluates a discovered circuit on prompt variants and reports qualitative overlap.","marker":"Stolfo et al. (2023)"},{"why":"Introduces faithfulness and circuit-overlap metrics used here to quantify how well the discovered variant circuits match the full model.","marker":"Hanna et al. (2024)"}],"fun_headline_variants":["S2 Hacking lets IOI circuit beat its own algorithm","IOI circuit reuses all nodes and adds only input edges","Even when its algorithm fails, IOI circuit reuses all components","All head types keep roles: IOI circuit generalizes without new nodes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that starting the discovery process from the base IOI circuit and restoring only its ablated paths yields the true DoubleIO and TripleIO circuits; if a search that could add completely new heads would find much lower overlap, the reuse result is an artifact of initialization.","fun_headline_variants_meta":{"raw":{"variants":["S2 Hacking lets IOI circuit beat its own algorithm","IOI circuit reuses all nodes and adds only input edges","Even when its algorithm fails, IOI circuit reuses all components","All head types keep roles: IOI circuit generalizes without new nodes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3867,"prompt_tokens":1029,"completion_tokens":2838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":2773}},"tokens_in":645,"tokens_out":2838,"duration_ms":19956,"temperature":1.0,"reasoning_tokens":2773,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:32:11.373892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same path-patching discovery for DoubleIO initialized from scratch, with no heads carried over from the base IOI circuit. If the resulting circuit contains far fewer of the base circuit's 26 nodes, or if restoring all ablated paths (removing the S2-only knockout) eliminates the circuit's logit-difference advantage over the full model, then the reuse claim and the S2 Hacking explanation would both fail.","supporting_citations":[{"cited_title":"A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis","cited_arxiv_id":null,"evidence_quote":"The Arithmetic circuit study that similarly evaluates a discovered circuit on prompt variants and reports qualitative overlap."}],"review_version":1}