Pith. sign in

REVIEW 2 major objections 4 minor 41 references

Circuit extraction does not determine which mechanism a model uses: the same checkpoint pair can look circuit-sharing at the attention-head level and mechanism-different at the exact-edge level, so the reported comparison level decides the

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:57 UTC pith:FTPPNCZL

load-bearing objection Careful empirical paper on a known thesis, but its headline 'exact edges at chance' number is built on a derived bundle-ID grouping that is itself a comparison choice; raw node-level Jaccard is never reported, so the strongest quantitative claim is not yet supported. the 2 major comments →

arxiv 2607.18921 v1 pith:FTPPNCZL submitted 2026-07-21 cs.LG cs.AI

Circuit Claims Depend on What Is Extracted and How It Is Compared

Yang Sheng , Jie Fu This is my paper
classification cs.LG cs.AI
keywords circuit extractionmechanistic interpretabilityattribution ambiguityJaccard overlapattention headsweight sparsitycompositional generalizationLean theorem proving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Circuit extraction tries to isolate the small set of model components whose presence preserves a behavior, but this paper argues that the extraction itself does not single out the mechanism: what you report and how you compare it decide the claim. On a synthetic Lean proof-step prediction benchmark with nine fixed proof rules randomized in surface form, an 8-layer transformer's dense and 75%-weight-sparse checkpoints overlap strongly in the set of selected attention heads (Jaccard@10 of 0.666 on solved prompts) yet nearly not at all in exact component-to-component edges (0.163 on solved prompts, and 0.142 on near-misses, statistically indistinguishable from a random baseline). Coarser summaries are stable where exact edges are not: splitting each head's query and key support into separate nodes barely changes graph sizes, and all three graph objects preserve the ranking of RL initialization conditions across tested loss budgets. The authors conclude that a circuit claim is well defined only when the graph, pruning threshold, query/key representation, and comparison level are stated, and they offer a reporting practice to make those choices explicit.

Core claim

The paper's central discovery is that the same two checkpoints can be made to look as if they share a circuit, or not, purely by choosing the granularity at which the extracted graphs are compared. On the pre-specified dense versus 75% weight-sparse RL comparison, grouping top routing edges by attention head yields Jaccard@10 of 0.666 on the success split and 0.554 on the near-miss split, far above random top-k baselines of 0.199 and 0.185. Exact structural edge lists, by contrast, yield only 0.163 and 0.142; the near-miss edge overlap is at chance (p=0.106 against a 0.117 random baseline), while head-level overlap is clearly above chance. At the same time, object-level node-fraction summari

What carries the argument

The load-bearing choice is the comparison level. The same extracted graph can be summarized either as an exact component-to-component edge list, formed by mapping edges to structural bundle IDs by layer, source/target membership category, and node family, or as a routing-head set, which groups top routing edges by attention-head identity before computing Jaccard@10. Both are measured against random top-k baselines and bootstrap intervals over 16 matched task-object entries per split. The benchmark—nine synthetic Lean tactic-prediction tasks with fixed proof rules and randomized names, distractors, and surface forms—is what lets differences between extracted circuits be attributed to extracti

Load-bearing premise

The claim that exact edges are not reproducible while attention-head sets are assumes that this synthetic setup—one 8-layer architecture, learned-mask extraction with post-ablation exact-match selection, and one edge-bundling convention—represents circuit extraction generally.

What would settle it

Re-run the dense-versus-75%-weight-sparse comparison with a different edge-bundling rule (for example, grouping edges by layer and attention head only, ignoring node family) or with a different pruning threshold; if exact-edge Jaccard@10 rises clearly above the random baseline while routing-head overlap stays high, the paper's central contrast between reproducible head sets and irreproducible edge lists would reverse.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Whether two checkpoints 'share a circuit' is not a settled fact about behavior; it is a statement about the comparison level, and the same pair can support opposite answers.
  • Exact edge lists are not a dependable way to compare circuits across checkpoints in this setting: they hover near random overlap, while attention-head sets are substantially more reproducible.
  • Object-level summaries—circuit size and the ranking of conditions by size—are robust to whether query and key support are merged or split, and to the tested loss-threshold range.
  • In the RL comparisons, the largest compositional-task accuracy gains are accompanied by the largest fraction of circuit nodes outside the matched atomic-task circuits, as an observed association rather than a causal effect.
  • Circuit-extraction studies should state which graph they report, how it was pruned, how attention heads are represented, and at what level the comparison is made, or the claim itself is under-specified.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the coarse-versus-fine stability hierarchy holds more broadly, head-level or functional-path summaries should become the default for cross-checkpoint claims, and edge-level mismatches should be read as reporting artifacts rather than mechanistic differences.
  • The reporting practice transfers naturally to other interpretability pipelines, such as activation patching or feature circuits: before publishing a circuit claim, one can audit whether changing the comparison granularity would reverse it.
  • A testable extension is to run the same protocol on larger or heterogeneous architectures and check whether the exact-edge overlap stays near random while head-set overlap stays high; the paper's own cross-seed supplement already shows head-level above structural at every tested anchor.
  • The bridge-fraction association suggests a cheap screening signal for whether RL will recruit structure beyond atomic circuits, but the paper's evidence rests on two curriculum cells, so a larger condition grid would be needed to turn that association into a quantitative rule.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper argues that circuit-extraction claims are underdetermined by behavior preservation alone. In a synthetic Lean tactic-prediction benchmark with fixed proof rules and randomized surface form, the authors train dense and weight-sparse (25–90%) 8-layer transformers, extract three graph objects (core circuit, extended support graph, fixed-loss pruning graph) under coupled and factorized Q/K representations, and compare overlaps across dense/75%-sparse RL checkpoints. Finding 1: routing-head-set Jaccard@10 is high (0.666/0.554) while the reported 'exact-edge' structural Jaccard@10 is low (0.163/0.142, with near-miss not above a random baseline). Finding 2: Q/K factorization and loss-budget sweeps preserve RL-condition orderings by graph size. Finding 3: curriculum-initialized RL has the largest compositional gains and the largest core-circuit bridge fraction. The paper concludes by proposing a reporting practice that requires stating the extracted graph, extraction rule, pruning threshold, Q/K representation, and comparison level. Source code is released.

Significance. If the quantitative core holds, this is a valuable, carefully scoped empirical contribution: it demonstrates in a controlled setting that descriptive granularity can reverse whether two checkpoints appear to share a mechanism, and it provides bootstrap confidence intervals, Monte-Carlo random baselines, matched/mismatched/random negative controls, and explicit boundary tests. The proposed reporting practice is sensible and directly actionable for the circuit-extraction community. The main caveat is that the 'exact-edge' metric is not raw edge-to-edge identity as claimed in the abstract; the headline contrast therefore requires verification before the central claim can be accepted as stated.

major comments (2)
  1. [Section 3.4 / Table 11] The metric called 'exact-edge' or 'structural' Jaccard@10 is not a comparison of exact component-to-component edges. A structural bundle ID groups edges by layer, source membership category, target membership category, and source/target node family; membership category is a derived, checkpoint-dependent label (shared-with-atomic, reused-atomic, compositional-only). The same literal edge (same source and target component IDs at the same layer) can therefore receive different bundle IDs in the dense and 75%-sparse checkpoints because the atomic circuits differ. Since Finding 1 and the abstract contrast reproducible head sets with non-reproducible exact edge lists, the low structural Jaccard may be an artifact of the bundle-ID convention rather than of the edge structure itself. Please report the raw edge Jaccard (source node ID → target node ID), with and without top-10 truncation, for the
  2. [Section 4.2 / Table 15] The claim that the RL-condition orderings are 'preserved' rests on observed 3/3 agreement, but the bootstrap flip probability reaches 0.232 for the extensively expanded fixed-loss pruning graph (curriculum vs. composition-only). This means the ordering conclusion is not strongly supported in that cell; with only 15 sparsity-seed units, a failure to detect a flip is weak evidence of preservation. The paper should either provide a more powerful test or explicitly qualify the stability claim for this cell. The same caveat applies to the loss-budget sweep in Table 16, where no uncertainty is reported for the 3/3 agreements.
minor comments (4)
  1. [Abstract / Table 1] The phrase 'exact component-to-component edge overlap' is used in the abstract and Table 1, but Section 3.4 defines the metric over bundle IDs rather than raw edges. Terminology should be aligned once the raw edge metric is reported.
  2. [Table 18] In matched-sparsity cross-seed comparisons, structural Jaccard@10 is substantially higher (0.363–0.549) than in the dense-versus-75%-sparse comparison. The text should state explicitly that the 'exact edges do not reproduce' result is specific to cross-sparsity comparisons, not a global property of the two checkpoints.
  3. [Appendix H] The boundary tests show near-zero held-out transfer (0.4–6.7%), and Appendix J appropriately concedes the graph evidence is local. These are strengths in scoping, but the main text could point readers to Appendix J earlier to avoid over-generalization.
  4. [Figure 4 / Figure 11] The representative cases are selected by proximity to the pool mean, which is fine, but the caption should note that the figures are illustrative and that the aggregate evidence in Tables 11–16 is the basis for the quantitative claims.

Circularity Check

0 steps flagged

No significant circularity; the study is empirical, self-contained, and explicitly scopes its claims.

full rationale

This is an empirical paper, not a derivation: the reported overlaps, orderings, and bridge fractions are measured from fixed extraction pipelines with stated selection rules. The tau-sweep (Table 16) holds the fitted pruning ranking fixed while varying the loss budget, so ordering stability is a genuine sensitivity check rather than a fitted input relabeled as a prediction. There are no self-citations by the authors and no imported uniqueness theorem. The one metric-construction caveat is that 'structural' exact-edge Jaccard is computed on bundle IDs that include checkpoint-dependent membership categories (Section 3.4), so the low exact-edge overlap is not literally raw source/target edge identity and the paper never reports component-level edge Jaccard; however, this is a validity/operationalization limitation, not a case where a claimed result is equivalent by construction to its inputs. The paper itself flags the small n=16 graph-evidence scope in Appendix J and disclaims causal status for the bridge-fraction association (Section 4.3), and Appendix H documents that held-out shifts collapse. Consequently, no load-bearing circular step is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests less on fitted constants than on disclosed analysis choices: which graph object is reported, the tau budget, the bundle-ID grouping that defines exact edges, and the comparison granularity. All are specified in the text; several are swept (tau 0.04-0.20, Table 16), but the headline numbers remain conditional on the dense-vs-75%-sparse anchor and on exact-match accuracy as the behavioral criterion.

free parameters (5)
  • Fixed-loss pruning budget tau = 0.12 (main); swept 0.04-0.20
    Hand-chosen post-ablation loss budget that defines the smallest selected pruning graph; the size-ordering claim is robust across the sweep (Table 16), but every node-fraction and bridge-fraction number is conditional on it.
  • Near-miss scoring weights = 0.5*token-acc + 0.3*first-tactic + 0.2*output-acc
    Hand-specified weights define the near-miss split that supplies the headline near-miss overlap numbers (p=0.106 result); a different scoring rule would yield a different split.
  • Top-k for Jaccard@10 = k = 10
    All overlap statistics truncate to the top 10 scored entries; both the observed values and the random-baseline magnitudes depend on this choice.
  • Mask lambda_l0 and binarization threshold = lambda_l0 = 0.05; threshold 0.5; pruning k-penalty 3e-5
    Extraction hyperparameters that shape how many nodes and edges survive into the core and support graphs, and hence every size, fraction, and bridge-fraction statistic.
  • Graph-extraction anchor = dense vs 75% weight-sparse
    The pre-specified comparison point for all detailed graph analyses; Figure 2 shows 75% sparsity is where expanded-distribution SFT accuracy is lowest, and the headline overlap numbers attach to this disclosed anchor.
axioms (5)
  • domain assumption Residual-stream node vocabulary (reads/writes, attention Q/K/V/write, MLP read/neuron/write) captures the model's mechanism
    Section 3.3 adopts the Elhage et al. 2021 / Gao et al. 2025 vocabulary; if the true computation lives outside this vocabulary, every extracted object is incomplete by construction.
  • domain assumption Exact-match tactic accuracy after ablation operationalizes 'behavior preserved'
    Core and support graphs are selected by best post-ablation exact-match accuracy (Table 5); a different behavioral metric (e.g., loss or token-level accuracy) would select different graphs.
  • domain assumption Fixed proof-state generation rules make the atomic/compositional decomposition the 'true' task structure
    Section 3.1: all causal attribution to extraction choices assumes the generator, not the model, fixes the true task decomposition; boundary tests show format cues still leak (Appendix H).
  • domain assumption GRPO with composition-only prompts is a valid RL refinement
    Section 3.2: all RL conditions share this recipe; the RL gains and the bridge-fraction association are conditional on this training choice.
  • standard math Jaccard, bootstrap, and Monte-Carlo baselines are unbiased for comparing circuits
    Section 3.4: standard statistics; validity depends on the candidate-set and bundle definitions being the right sample space, which the paper controls and discloses.

pith-pipeline@v1.3.0-alltime-deepseek · 23569 in / 25056 out tokens · 224547 ms · 2026-08-01T13:57:49.275874+00:00 · methodology

0 comments
read the original abstract

Circuit extraction identifies a small set of model components whose presence preserves a target behavior under ablation, and the resulting circuit is often read as the mechanism behind that behavior. We argue that this reading is under-determined: preserving behavior does not single out one circuit, because the claim it supports depends on which circuit is reported and how two circuits are compared. We make this concrete in a synthetic Lean tactic-prediction benchmark -- predicting the next step of a proof -- where fixed proof rules with randomized surface form let differences between extracted circuits be attributed to these choices rather than to the task. Across dense and weight-sparse checkpoints (most weights constrained to zero) of the same transformer, evaluated on atomic (single-rule) and compositional (multi-rule) proofs, we vary which extracted object is reported (a compact prediction-preserving circuit, a broader graph that also keeps surrounding read, write, and routing structure, or the smallest subgraph meeting a post-ablation loss threshold), and whether each attention head's query and key are represented jointly or separately. Exact component-to-component edge overlap is low and sensitive to these choices, at times dropping to a random baseline, while two coarser summaries stay stable: the set of selected attention heads, and the circuit-size ranking of conditions that differ in which supervised checkpoint initializes reinforcement learning (RL). The largest accuracy gains from RL on compositional proofs come with the most structure beyond the atomic circuits. A circuit-level claim is therefore well defined only once one states which circuit is reported, the pruning threshold used to extract it, and the level at which circuits are compared. We distill these requirements into a reporting practice for circuit-extraction studies.

Figures

Figures reproduced from arXiv: 2607.18921 by Jie Fu, Yang Sheng.

Figure 1
Figure 1. Figure 1: Overview of the three findings. A: at the primary dense-versus-75%-sparse comparison point, the set of selected attention heads overlaps much more across the dense and 75% weight-sparse RL checkpoints than the exact component-to￾component edge lists do; the head-level overlap is also far above a random top-k baseline, while the exact-edge overlap is not consistently above it ( [PITH_FULL_IMAGE:figures/ful… view at source ↗
Figure 2
Figure 2. Figure 2: Supervised accuracy across sparsity levels. A: the baseline training distribution across dense and four weight-sparse regimes; the highest-accuracy checkpoint and the checkpoint used for detailed graph extraction need not coincide. B: at the same sparsity points, expanded training distributions reduce supervised compositional accuracy, especially at the 75% weight-sparse graph-extraction checkpoint. 3.2 Mo… view at source ↗
Figure 3
Figure 3. Figure 3: ; Tables 11, 12, 18 Dense vs. 75% weight-sparse RL comparison; near-miss exact-edge overlap is not detectably above random (p = 0.106). Object-level summaries preserve RL ordering At τ = 0.12, curriculum coupled-vs-factorized node-fraction gaps are at most 0.024 (fixed-loss pruning graph, extensively expanded family) across the three objects; all pairwise RL-condition orderings agree (3/3) across objects, … view at source ↗
Figure 4
Figure 4. Figure 4: Representative 75% weight-sparse RL case for AND-OR. Arrows denote the direction of information flow. The recurring pattern is a coarse read-to-write path: a late-position read → a compact query/key selector (the attention pattern that decides what to attend to) → a small group of value-vector components carrying signed contributions → the attention write back into the residual stream → downstream MLP comp… view at source ↗
Figure 5
Figure 5. Figure 5: Representative extraction under coupled (merged) versus factorized (separate) query/key support at the main loss budget on an RL checkpoint at 75% weight sparsity. This schematic illustrates the object-level pattern at one cell; the aggregate selected-node fractions and RL-condition orderings in Tables 13–15 confirm that the orderings are preserved at this pruning threshold [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figure 6
Figure 6. Figure 6: Compositional gains align with core-circuit bridge fraction. A: curriculum-initialized RL gains concentrate on compositional tasks. B: bridge fraction is the share of compositional-task core-circuit nodes outside matched atomic-task circuits. C: query/key gaps remain small and preserve RL-condition orderings (Tables 13, 16); expanded-grid endpoint accuracies are reported in [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figure 7
Figure 7. Figure 7: Expanded-grid RL endpoint accuracies for the moderately and extensively expanded distributions. Bars compare the upstream supervised checkpoint with the RL-refined endpoint across sparsity levels; annotations report the corresponding RL–SFT accuracy changes [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Exact-edge supplement beyond the main dense and 75% weight-sparse comparison point. Routing￾head-set Jaccard@10 remains higher than structural Jaccard@10 across both cross-sparsity and cross-seed comparisons; the cross-seed panel provides the strongest additional support. Semantic overlap exactly matches structural overlap throughout this supplement. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Boundary tests and controls for the narrow-family regimes at the dense and 75% weight-sparse comparison points. A: the strongest baseline-family behavior does not carry over to the evaluated held￾out surface/depth shifts. B: matched atomic unions preserve the most target predictions after ablation, and mismatched atomic unions remain well above random controls. The matched-vs-mismatched gap indicates that … view at source ↗
Figure 10
Figure 10. Figure 10: Graph-level counterpart to [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Paired dense/weight-sparse routing support for the representative AND-OR success case discussed in Section 4.1. Both the dense and 75% weight-sparse RL endpoints preserve the same coarse early-read → middle-selector → late-attention-write organization, but the supporting components differ: the dense endpoint places more selected support on MLP-side components, whereas the weight-sparse endpoint places mor… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 3 canonical work pages

  1. [1]

    Finding transformer circuits with edge pruning

    Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017-0587

    doi:10.52202/079017-0587 2024

  2. [2]

    Metaxas, Zhaoran Wang, Yutao Yue, and Gerald Penn

    Xi Chen, Mingyu Jin, Jingcheng Niu, Yutong Yin, Jinman Zhao, Bangwei Guo, Dimitris N. Metaxas, Zhaoran Wang, Yutao Yue, and Gerald Penn. All circuits lead to rome: Rethinking functional anisotropy in circuit and sheaf discovery for LLMs . arXiv preprint arXiv:2605.12671, 2026. URL https://arxiv.org/abs/2605.12671

    Pith/arXiv arXiv 2026

  3. [3]

    Transformers as soft reasoners over language

    Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 3882--3890, 2020. doi:10.24963/ijcai.2020/537

    doi:10.24963/ijcai.2020/537 2020

  4. [4]

    Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso

    Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, volume 36, 2023

    2023

  5. [5]

    Sparse interventions in language models with differentiable masking

    Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. Sparse interventions in language models with differentiable masking. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 16--27, 2022. doi:10.18653/v1/2022.blackboxnlp-1.2

    work page doi:10.18653/v1/2022.blackboxnlp-1.2 2022

  6. [6]

    Sparse attention post-training for mechanistic interpretability

    Florent Draye, Anson Lei, Hsiao-Ru Pan, Ingmar Posner, and Bernhard Sch \"o lkopf. Sparse attention post-training for mechanistic interpretability. arXiv preprint arXiv:2512.05865, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Transcoders find interpretable LLM feature circuits

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In Advances in Neural Information Processing Systems, volume 37, pages 24375--24410, 2024. doi:10.52202/079017-0768

    doi:10.52202/079017-0768 2024

  8. [8]

    Bootstrap methods: another look at the jackknife

    Bradley Efron. Bootstrap methods: another look at the jackknife. The Annals of Statistics, 7 0 (1): 0 1--26, 1979. doi:10.1214/aos/1176344552

    arXiv 1979

  9. [9]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

    2021

  10. [10]

    Weight-sparse transformers have interpretable circuits

    Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V Govande, Bowen Baker, and Dan Mossing. Weight-sparse transformers have interpretable circuits. arXiv preprint arXiv:2511.13653, 2025

    arXiv 2025

  11. [11]

    Causal abstraction: A theoretical foundation for mechanistic interpretability

    Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26 0 (83): 0 1--64, 2025

    2025

  12. [12]

    Localizing model behavior with path patching

    Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    InterpBench : Semi-synthetic transformers for evaluating mechanistic interpretability techniques

    Rohan Gupta, Iv \'a n Arcuschin, Thomas Kwa, and Adri \`a Garriga-Alonso. InterpBench : Semi-synthetic transformers for evaluating mechanistic interpretability techniques. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017-2950

    work page doi:10.52202/079017-2950 2024

  14. [14]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre...

    Pith/arXiv arXiv 2022

  15. [15]

    Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67: 0 757--795, 2020

    Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67: 0 757--795, 2020. doi:10.1613/jair.1.11674

    doi:10.1613/jair.1.11674 2020

  16. [16]

    \'E tude comparative de la distribution florale dans une portion des alpes et des jura

    Paul Jaccard. \'E tude comparative de la distribution florale dans une portion des alpes et des jura. Bulletin de la Soci \'e t \'e Vaudoise des Sciences Naturelles , 37: 0 547--579, 1901

    1901

  17. [17]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361

    Pith/arXiv arXiv 2001

  18. [18]

    Measuring compositional generalization: A comprehensive method on realistic data

    Daniel Keysers, Nathanael Sch \"a rli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data. In International Conference on Learning Rep...

    2020

  19. [19]

    Cogs: A compositional generalization challenge based on semantic interpretation

    Najoung Kim and Tal Linzen. Cogs: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087--9105, 2020. doi:10.18653/v1/2020.emnlp-main.731

    doi:10.18653/v1/2020.emnlp-main.731 2020

  20. [20]

    AtP* : An efficient and scalable method for localizing LLM behaviour to components

    J \'a nos Kram \'a r, Tom Lieberum, Rohin Shah, and Neel Nanda. AtP* : An efficient and scalable method for localizing LLM behaviour to components. arXiv preprint arXiv:2403.00745, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    The disagreement problem in explainable machine learning: A practitioner's perspective

    Satyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arXiv:2202.01602, 2022. URL https://arxiv.org/abs/2202.01602

    Pith/arXiv arXiv 2022

  22. [22]

    Lake and Marco Baroni

    Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2873--2882, 2018

    2018

  23. [23]

    Tracr: Compiled transformers as a laboratory for interpretability

    David Lindner, J \'a nos Kram \'a r, Sebastian Farquhar, Matthew Rahtz, Tom McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. In Advances in Neural Information Processing Systems, 2023

    2023

  24. [24]

    Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller

    Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In International Conference on Learning Representations, 2025

    2025

  25. [25]

    Everything, everywhere, all at once: Is mechanistic interpretability identifiable? arXiv preprint arXiv:2502.20914, 2025 a

    Maxime M \'e loux, Silviu Maniu, Fran c ois Portet, and Maxime Peyrard. Everything, everywhere, all at once: Is mechanistic interpretability identifiable? arXiv preprint arXiv:2502.20914, 2025 a . URL https://arxiv.org/abs/2502.20914

    Pith/arXiv arXiv 2025

  26. [26]

    Mechanistic interpretability as statistical estimation: A variance analysis

    Maxime M \'e loux, Fran c ois Portet, and Maxime Peyrard. Mechanistic interpretability as statistical estimation: A variance analysis. arXiv preprint arXiv:2510.00845, 2025 b . URL https://arxiv.org/abs/2510.00845

    Pith/arXiv arXiv 2025

  27. [27]

    Transformer circuit evaluation metrics are not robust

    Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=zSf8PJyQb2

    2024

  28. [28]

    Circuit compositions: Exploring modular structures in transformer-based language models

    Philipp Mondorf, Sondre Wold, and Barbara Plank. Circuit compositions: Exploring modular structures in transformer-based language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14934--14955, 2025. doi:10.18653/v1/2025.acl-long.727

    doi:10.18653/v1/2025.acl-long.727 2025

  29. [29]

    MIB : A mechanistic interpretability benchmark

    Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iv \'a n Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, Martin Tutek, Amir Zur, David Bau, and Yonatan Belinkov. MI...

    2025

  30. [30]

    A pragmatic vision for interpretability

    Neel Nanda, Josh Engels, Arthur Conmy, Senthooran Rajamanoharan, Bilal Chughtai, Callum McDougall, J \'a nos Kram \'a r, and Lewis Smith. A pragmatic vision for interpretability. Alignment Forum, 2025. URL https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability

    2025

  31. [31]

    Fine-tuning enhances existing mechanisms: A case study on entity tracking

    Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2402.14811

    Pith/arXiv arXiv 2024

  32. [32]

    Beyond accuracy: Behavioral testing of nlp models with checklist

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902--4912, 2020. doi:10.18653/v1/2020.acl-main.442

    doi:10.18653/v1/2020.acl-main.442 2020

  33. [33]

    Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. A benchmark for systematic generalization in grounded language understanding. In Advances in Neural Information Processing Systems, 2020

    2020

  34. [34]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath : Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. URL https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  35. [35]

    Claudia Shi, Nicolas Beltran-Velez, Achille Nazaret, Carolina Jiang Zheng, Adri \`a Garriga-Alonso, Andrew Jesson, Maggie Makar, and David M. Blei. Hypothesis testing the circuit hypothesis in LLM s. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/abccb8a90b30d45b948360b...

    2024

  36. [36]

    ProofWriter : Generating implications, proofs, and abductive statements over natural language

    Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter : Generating implications, proofs, and abductive statements over natural language. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621--3634, 2021. doi:10.18653/v1/2021.findings-acl.317

    doi:10.18653/v1/2021.findings-acl.317 2021

  37. [37]

    LLM circuit analyses are consistent across training and scale

    Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. LLM circuit analyses are consistent across training and scale. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017-1287. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/47c7edadfee365b394b2a3bd416048da-Abstract-Conference.html

    doi:10.52202/079017-1287 2024

  38. [38]

    Interpretability in the wild: A circuit for indirect object identification in GPT-2 Small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: A circuit for indirect object identification in GPT-2 Small . In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=NpsVSN6o4ul

    2023

  39. [39]

    Manning, and Christopher Potts

    Zhengxuan Wu, Christopher D. Manning, and Christopher Potts. ReCOGS : How incidental details of a logical form overshadow an evaluation of semantic interpretation. Transactions of the Association for Computational Linguistics, 11: 0 1719--1733, 2023. doi:10.1162/tacl_a_00623

    doi:10.1162/tacl_a_00623 2023

  40. [40]

    Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar

    Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo : Theorem proving with retrieval-augmented language models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  41. [41]

    Sheaf discovery with joint computation graph pruning and flexible granularity

    Lei Yu, Jingcheng Niu, Zining Zhu, Xi Chen, and Gerald Penn. Sheaf discovery with joint computation graph pruning and flexible granularity. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8822--8837. Association for Computational Linguistics, 2025. doi:10.18653/v1/2025.emnlp-main.446. URL https://aclantholo...

    work page doi:10.18653/v1/2025.emnlp-main.446 2025