Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

This paper argues that circuit discovery in mechanistic interpretability is a statistical estimation problem, and that a state-of-the-art method, EAP-IG, outputs circuits with high variance under data resampling and high sensitivity to hype

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 13:00 UTC pith:DLCEELPU

load-bearing objection Solid empirical stability audit of EAP-IG that is worth refereeing, but the abstract overclaims a base-layer instability that the experiments never directly measure. the 3 major comments →

arxiv 2510.00845 v4 pith:DLCEELPU submitted 2025-10-01 cs.LG cs.AIcs.CL

Mechanistic Interpretability as Statistical Estimation: A Variance Analysis

Maxime M\'eloux , Fran\c{c}ois Portet , Maxime Peyrard This is my paper
classification cs.LG cs.AIcs.CL
keywords mechanistic interpretabilitycircuit discoveryEAP-IGcausal mediation analysisbootstrap resamplingstatistical stabilityvariance analysisattribution patching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the circuits mined from a neural network by mechanistic interpretability methods are not fixed discoveries but statistical estimates, and that the widely used EAP-IG method produces estimates with substantial variance: resampling the input data changes the discovered graph, hyperparameter changes can push circuits far apart, and noise in the causal intervention degrades both fidelity and stability. A sympathetic reader should care because the field's promise depends on circuits being reproducible; if a single run can be an artifact of the data sample or pipeline choices, then claims about 'the circuit' for a task need revision. The paper decomposes the variance sources, quantifies them across three models and three tasks, and proposes routine reporting of stability metrics such as bootstrap pairwise Jaccard index and coefficient of variation of circuit error.

Core claim

The central claim is that the causal effect of a model component, measured by exact single-input causal mediation analysis, is a volatile random variable rather than a fixed property, and that circuit discovery pipelines inherit and amplify that variance. Concretely, the paper shows that EAP-IG — which approximates edge causal effects with integrated-gradient attributions and then greedily selects edges into a circuit — yields markedly different circuits when the discovery dataset is bootstrap-resampled (average pairwise Jaccard 0.561, coefficient of variation 0.335), that changing aggregation or intervention hyperparameters moves Jaccard similarity to a median circuit anywhere from 0.086 to

What carries the argument

The central object is EAP-IG (Edge Attribution Patching with Integrated Gradients), a method that assigns each edge of the computational graph a score approximating its causal effect — by combining gradient information with interpolated clean/corrupted inputs or activations — and then selects a circuit via greedy search that grows from 30 to up to 2000 edges until an input-to-output path exists. The paper's analytic machinery is the perturbation battery around it: bootstrap resampling, meta-distribution re-generation, prompt paraphrasing, hyperparameter sweeps, and noise injection in interventions, with pairwise Jaccard index and circuit error/CV as the stability metrics.

Load-bearing premise

The paper's strongest framing — that the causal effect of a component is a volatile random variable, not a fixed property — rests on an unmeasured assumption, since every experiment uses approximate EAP-IG scores combined with a greedy selection pipeline, and no experiment computes exact single-input CMA scores directly.

What would settle it

Compute exact single-input CMA scores (e.g., brute-force activation patching per token) on a small model and task, and measure their variance across inputs; if exact scores are stable while EAP-IG bootstrap Jaccard remains around 0.56, the base-layer instability claim fails and the variance is attributable to the approximation pipeline. Likewise, evaluate circuits on a held-out dataset disjoint from the discovery set: if circuit error drops and resample-to-resample Jaccard rises, part of the observed instability is discovery overfitting.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A single EAP-IG run on one dataset should not be interpreted as delivering the circuit; the result should be accompanied by bootstrap-based stability metrics (e.g., average pairwise Jaccard, CV of circuit error).
  • Circuits discovered under bootstrap resampling are not only structurally diverse but also less faithful (average circuit error 0.440), meaning unstable structure and degraded performance travel together.
  • Choice of EAP variant and hyperparameters can matter as much as the data: Jaccard-to-median ranges from 0.086 to 1.0 across configurations, so sensitivity analysis should accompany circuit reports.
  • Larger models (Llama-3.2-1B) yield less stable and less faithful circuits than gpt2-small, so stability results from small models may not extrapolate to the models of practical interest.
  • Noise injection in the intervention, at moderate amplitude (~0.2 in gpt2-small), is proposed as a practical robustness stress test that exposes structural variance without wrecking performance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the paper's framing is right, the instability should show up in exact patching methods too, not just gradient approximations; a direct side-by-side comparison of exact single-input CMA variance with EAP-IG variance would tell whether the problem is fundamental or largely an artifact of approximation.
  • The multimodal Jaccard distributions under bootstrap suggest multiple discrete 'solution basins'; this connects to non-identifiability and motivates treating circuit discovery as distributional (ensembles over resamples) rather than point estimation.
  • Because the paper evaluates circuits on the same data used for discovery (Deval = D), a held-out evaluation is a natural next test: if instability drops sharply out of sample, part of the measured variance is discovery overfitting rather than intrinsic mechanism variance.
  • A practical extension is to add a stability term to the discovery objective, searching for circuits that are both faithful and stable across resamples; the authors gesture at this, but the concrete optimization target is left for future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes reframing mechanistic interpretability circuit discovery as a statistical estimation problem and empirically assesses the stability of EAP-IG circuits. Experiments across three models (gpt2-small, Llama-3.2-1B, Llama-3.2-1B-Instruct) and three tasks (IOI, subject-verb agreement, greater-than) measure structural overlap (pairwise Jaccard) and performance (circuit error, KL divergence) under bootstrap resampling, meta-distribution shifts, prompt paraphrasing, hyperparameter variation, and injected intervention noise. The main quantitative results are an average bootstrap pairwise Jaccard of 0.561 with CV 0.335 (Table 1), a hyperparameter-sensitivity Jaccard-to-median range of 0.086–1.0 (Table 2), and noise-driven degradation of fidelity and stability (Fig. 3). The paper concludes that EAP-IG circuits are sample- and setting-dependent and recommends routine reporting of stability metrics.

Significance. If the reported results stand, the paper offers a valuable, directly observed demonstration that EAP-IG circuits are not stable point estimates. These measurements are a useful contribution to current debates on the reliability of mechanistic interpretability, and the recommendation to report bootstrap-based Jaccard and CV metrics is concrete and actionable. The empirical numbers are transparently presented in the main tables. However, the paper's abstract overreaches by asserting a fundamental instability of 'exact, single-input CMA scores' that is never measured, and several methodological reporting gaps limit immediate confidence in the quantitative claims. With appropriate revisions, this could be a solid empirical contribution.

major comments (3)
  1. [Abstract (metadata) and §1] The abstract states that the paper 'uncovers a fundamental instability at this base layer: exact, single-input CMA scores exhibit high intrinsic variance, implying that the causal effect of a component is a volatile random variable rather than a fixed property.' This claim is not supported by any experiment in the manuscript. All experiments use EAP-IG or its approximate variants; no exact single-input CMA score is computed. The phrase 'causal effect of a component is a volatile random variable' is a strong theoretical assertion that is neither derived nor tested. Since this is presented as a central discovery, it is a load-bearing unsupported claim. The authors should either remove this claim from the abstract and frame the paper strictly as an EAP-IG stability analysis, or add experiments that directly measure variance of exact single-input CMA scores.
  2. [§3.3 and Tables 1, 3–5] The paper does not report the number of bootstrap resamples, seeds, or circuits generated per condition. The text says 'For each experimental run (iterated over seed values)' but never gives the number of runs. Table 1 reports average CV values such as 0.335 for bootstrap Jaccard, and the appendix tables report variances, but without sample sizes the reader cannot assess the precision of these estimates or whether differences between resampling strategies are statistically meaningful. The paper should report N for every condition and, ideally, bootstrap confidence intervals for the mean Jaccard and circuit error.
  3. [§5.1, Table 1] The interpretation that a bootstrap Jaccard of 0.561 with CV 0.335 is 'high variance' lacks a baseline. Jaccard values depend strongly on circuit size and the size of the edge space; a value of 0.561 could be close to random overlap for large circuits or far from it for small ones. To make the 'high variance' claim load-bearing, the paper should compare against a null model (e.g., expected Jaccard for randomly selected circuits of the same size) or against another circuit discovery method's stability. Without such calibration, the claim is underdetermined.
minor comments (6)
  1. [§3.2] The paper explicitly sets Deval = D, which is acknowledged in the text. This limitation should be more prominently discussed when interpreting circuit error values, since evaluating on the discovery data may underestimate error. The structural (Jaccard) claims are less affected, but the performance numbers should be framed carefully.
  2. [Table 6 (appendix)] Several Greater-Than entries for Llama-3.2-1B are marked as '-' but the text does not explain why the circuit was not found or the entry is missing. Please provide an explanation (e.g., 'no faithful circuit found under the greedy search procedure').
  3. [§5.3, Figure 3] The text says 'Table 3 shows the effect of increasing noise amplitude' but Table 3 contains bootstrap results, not noise results. The noise results appear in Table 9 in the appendix. The cross-reference is confusing and should be corrected.
  4. [Table 1 caption] Typo: 'Bootstrap resampling tields' should be 'yields'.
  5. [§1 (metadata abstract)] The metadata/arXiv abstract differs from the full-text abstract; the exact-CMA claim appears only in the metadata. The two versions should be made consistent, and the overclaim removed from the metadata if not supported.
  6. [§6] The paper mentions a 'publicly available codebase' but gives no URL or repository identifier. If code is available, please provide a link; if not, state availability.

Circularity Check

0 steps flagged

No material circularity: the paper's central variance results are direct empirical measurements, not outputs of a derivation that presupposes them; the only self-citation is framing and not load-bearing.

full rationale

The paper is an empirical stability study of EAP-IG. Its headline quantities — bootstrap pairwise Jaccard 0.561 (CV 0.335), hyperparameter Jaccard-to-median values from 0.086 to 1.0, and noise-induced degradation — are reported measurements obtained by resampling data, varying hyperparameters, and perturbing interventions, not by fitting a parameter and then predicting a closely related quantity. No equation in the paper defines a predicted circuit property in terms of the same measured property; the formal setup (§3.1) merely describes the discovery pipeline, and the stability metrics (§3.2) are statistical summaries of observed circuits. The choice Deval = D is explicitly acknowledged as common practice and is a limitation for generalization claims, not a circular reduction. The main self-citation is Méloux et al. (2025), used to motivate non-identifiability and incompatible circuits. This is self-citational, but the paper's central variance findings do not depend on that prior result; they stand on the paper's own tables and figures. The abstract's claim about 'exact, single-input CMA scores' being unstable is not directly measured, but that is an overstatement/evidence gap rather than a circular derivation. Overall, the derivation chain is self-contained with respect to its empirical claims, so the circularity burden is minimal.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claims rest on standard statistical tools (bootstrap, coefficient of variation, Jaccard similarity) applied to circuit discovery; the paper's own contributions are the measurements and the variance/robustness formalization rather than new entities or fitted derivations. Upstream inputs: EAP-IG method and datasets from Hanna et al. 2024, task generators from Wang et al. 2023 / Warstadt et al. 2020 / Hanna et al. 2023, and the non-identifiability framing from Méloux et al. 2025 (same first author). The only hand-chosen quantities that shape the headline numbers are the noise level 0.2 recommendation and the fixed greedy-search initialization n=30.

free parameters (2)
  • noise amplitude recommendation = 0.2
    §6 recommendation 3: 'A noise level of 0.2 seems to be a good starting point for gpt2-small' — chosen by hand from the tested grid [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5]; it sits at the CV peak in Fig. 5, not at a principled optimum, and is extrapolated to general practice from two tasks on one model.
  • greedy search initial edge count n = n = 30, incremented up to 2000
    §3.1: 'We select an initial set of n edges based on the absolute values of their scores (starting with n=30), then incrementally increase n up to 2000 until a path from input to output is found.' Inherited from Hanna et al. 2024 and fixed across all runs, so every reported variance estimate is conditional on this selection setting.
axioms (5)
  • domain assumption Circuit discovery can be modeled as C = F_CD(Mθ, D, Λmethod), a composite of observations and a selection algorithm.
    §3.1 formalization; the entire variance/robustness decomposition presupposes this input-output model of circuit discovery.
  • domain assumption Bootstrap resampling of the discovery dataset estimates the sampling variance of discovered circuits.
    §3.3 invokes Efron & Tibshirani; standard statistics transferred to circuits. Requires datasets large enough for resampling to be meaningful — dataset sizes are never stated in the text.
  • domain assumption Circuit error evaluated with Deval = D is a valid faithfulness measure for comparing circuits.
    §3.2 explicitly sets Deval = D 'following common practice'; circuits are scored on the same sample they were found from, so errors (e.g., 0.44 under bootstrap) reflect fit to the discovery sample, not out-of-sample generalization.
  • ad hoc to paper Exact, single-input CMA scores exhibit high intrinsic variance.
    Abstract and §1 assert this 'fundamental instability at the base layer,' but the body's experiments only measure approximate EAP-IG scores on three tasks; no exact single-input CMA measurement or derivation is reported.
  • domain assumption The three tasks (IOI, SVA, Greater-Than) and three models are representative of MI circuit-discovery practice.
    §4 re-uses tasks and datasets from Hanna et al. 2024 (with generators from Wang et al. 2023, Warstadt et al. 2020, Hanna et al. 2023); conclusions about 'MI findings' generalize from this sample.

pith-pipeline@v1.3.0-alltime-deepseek · 17522 in / 18393 out tokens · 153436 ms · 2026-08-04T13:00:18.370963+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Mechanistic Interpretability as Statistical Estimation: A Variance Analysis." pith.science (2026). https://pith.science/paper/DLCEELPU

@misc{pith2026251000845,
  author       = {Pith},
  title        = {Pith review of: Mechanistic Interpretability as Statistical Estimation: A Variance Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DLCEELPU}},
  note         = {Machine review of arXiv:2510.00845}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Mechanistic Interpretability (MI) aims to reverse-engineer model behaviors by identifying functional sub-networks. Yet, the scientific validity of these findings depends on their stability. In this work, we argue that circuit discovery is not a standalone task but a statistical estimation problem built upon causal mediation analysis (CMA). We uncover a fundamental instability at this base layer: exact, single-input CMA scores exhibit high intrinsic variance, implying that the causal effect of a component is a volatile random variable rather than a fixed property. We then demonstrate that circuit discovery pipelines inherit this variance and further amplify it. Fast approximation methods, such as Edge Attribution Patching and its successors, introduce additional estimation noise, while aggregating these noisy scores over datasets leads to fragile structural estimates. Consequently, small perturbations in input data or hyperparameters yield vastly different circuits. We systematically decompose these sources of variance and advocate for more rigorous MI practices, prioritizing statistical robustness and routine reporting of stability metrics.

Figures

Figures reproduced from arXiv: 2510.00845 by Fran\c{c}ois Portet, Maxime M\'eloux, Maxime Peyrard.

Figure 1
Figure 1. Figure 1: In gpt2-small, varying multiple circuit finding parameters at once (resampling strategy, aggregation method, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Circuit error and pairwise Jaccard index of EAP-IG circuits found across the three models, tasks, and types of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average and standard deviation of the circuit error (left) and pairwise Jaccard (right) index of the circuits [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Full heatmap of the pairwise Jaccard index between circuits displayed in Figure 1 (circuits found in gpt2-small [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: CV of circuit metrics for different noise amplitudes in gpt2-small, averaged across tasks. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Many Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit Discovery

    cs.CL 2026-06 unverdicted novelty 7.0

    Structurally distinct circuits for literal sequence copying across token frequency bands implement the same computation, shown by broad transfer of band-specific edges, a shared core recovering 99% performance, and in...

  2. Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models

    cs.LG 2026-02 conditional novelty 7.0

    Protein language models complete repeats by combining induction heads that copy the aligned residue from the other repeat copy with neurons encoding amino-acid similarity; the approximate-repeat circuit contains and g...

  3. Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretability

    cs.LG 2026-07 accept novelty 6.0

    CIF certifies interventional interpretability metrics as bounded causal means with anytime-valid confidence sequences, including under adaptive sampling, cutting certification cost 10–30× with betting sequences.

  4. Validating Causal Abstraction Metrics on Simulated Complex Systems

    cs.LG 2026-06 unverdicted novelty 6.0

    Authors create a benchmark across discrete/continuous and static/dynamical systems and introduce the Causal Abstraction Error (CAE) metric that reliably distinguishes valid from invalid causal abstractions when it inc...

  5. Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers

    cs.AI 2026-06 unverdicted novelty 6.0

    Standard tests for mechanistic roles in transformer attention heads are insufficient because heads that pass them fail to transfer computations across prompts under matched controls.

  6. Circuit Claims Depend on What Is Extracted and How It Is Compared

    cs.LG 2026-07 conditional novelty 5.0

    On a synthetic Lean tactic-prediction task, exact circuit edge lists barely overlap across dense and sparse checkpoints while attention-head sets and size rankings do, so a circuit claim is well defined only once grap...

Reference graph

Works this paper leans on

61 extracted references · 4 canonical work pages · cited by 6 Pith papers

  1. [1]

    Sanity checks for saliency maps

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp.\ 9525–9536, Red Hook, NY, USA, 2018. Curran Associates Inc

    2018

  2. [2]

    Llama 3.2 model card

    AI@Meta. Llama 3.2 model card. 2024. URL https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md

    2024

  3. [3]

    Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai

    Alejandro Barredo Arrieta , Natalia Díaz-Rodríguez, Javier Del Ser , Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion...

    doi:10.1016/j.inffus.2019.12.012 2020

  4. [4]

    Network dissection: Quantifying interpretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017

    2017

  5. [5]

    Statistics for experimenters: Design, innovation, and discovery

    David Berengut. Statistics for experimenters: Design, innovation, and discovery. The American Statistician, 60 0 (4): 0 341--342, 2006. doi:10.1198/000313006X152991. URL https://doi.org/10.1198/000313006X152991

    work page doi:10.1198/000313006x152991 2006

  6. [6]

    Finding transformer circuits with edge pruning

    Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 18506--18534. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/pape...

    2024

  7. [7]

    Box, Stuart Hunter, and William Hunter

    G. Box, Stuart Hunter, and William Hunter. Statistics for experimenters. Design, innovation, and discovery. 2nd ed, volume 2. 01 2005

    2005

  8. [8]

    Nate Breznau, Eike Mark Rinke, Alexander Wuttke, Hung H. V. Nguyen, Muna Adem, Jule Adriaans, Amalia Alvarez-Benjumea, Henrik K. Andersen, Daniel Auer, Flavio Azevedo, Oke Bahnsen, Dave Balzer, Gerrit Bauer, Paul C. Bauer, Markus Baumann, Sharon Baute, Verena Benoit, Julian Bernauer, Carl Berning, Anna Berthold, Felix S. Bethke, Thomas Biegert, Katharina ...

    2022

  9. [9]

    Thread: Circuits

    Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. doi:10.23915/distill.00024. https://distill.pub/2020/circuits

    doi:10.23915/distill.00024 2020

  10. [10]

    Guidance on uncertainty analysis in scientific assessments

    EFSA Scientific Committee, Diane Benford, Thorhallur Halldorsson, Michael John Jeger, Helle Katrine Knutsen, Simon More, Hanspeter Naegeli, Hubert Noteborn, Colin Ockleford, Antonia Ricci, Guido Rychen, Josef R Schlatter, Vittorio Silano, Roland Solecki, Dominique Turck, Maged Younes, Peter Craig, Andrew Hart, Natalie Von Goetz, Kostas Koutsoumanis, Alicj...

    arXiv 2018

  11. [11]

    Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso

    Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=89ia77nZ8u

    2023

  12. [12]

    Transcoders find interpretable LLM feature circuits

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=J6zHcScAo0

    2024

  13. [13]

    Efron and R

    B. Efron and R. Tibshirani. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1 0 (1): 0 54--75, 1986. ISSN 08834237, 21688745. URL http://www.jstor.org/stable/2245500

    arXiv 1986

  14. [14]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

    2021

  15. [15]

    Alphaedit: Null-space constrained model editing for language models

    Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=HvSytvg3Jh

    2025

  16. [16]

    Statistical methods and scientific induction

    Ronald Fisher. Statistical methods and scientific induction. Journal of the Royal Statistical Society. Series B (Methodological), 17 0 (1): 0 69--78, 1955. ISSN 00359246. URL http://www.jstor.org/stable/2983785

    arXiv 1955

  17. [17]

    Causal abstractions of neural networks

    Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=RmuXDtjDhG

    2021

  18. [18]

    How does GPT -2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

    Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT -2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=p4PckNQR8k

    2023

  19. [19]

    Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview.net/forum?id=grXgesr5dT

    2024

  20. [20]

    Sparse autoencoders can interpret randomly initialized transformers, 2025

    Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison. Sparse autoencoders can interpret randomly initialized transformers, 2025. URL https://arxiv.org/abs/2501.17727

    arXiv 2025

  21. [21]

    Anna Hedström, Leander Weber, Daniel Krakowczyk, Dilyara Bareeva, Franz Motzkus, Wojciech Samek, Sebastian Lapuschkin, and Marina M.-C. Höhne. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24 0 (34): 0 1--11, 2023. URL http://jmlr.org/papers/v24/22-0142.html

    2023

  22. [22]

    Detecting edit failures in large language models: An improved specificity benchmark, 2023

    Jason Hoelscher-Obermaier, Julia Persson, Esben Kran, Ioannis Konstas, and Fazl Barez. Detecting edit failures in large language models: An improved specificity benchmark, 2023. URL https://arxiv.org/abs/2305.17553

    Pith/arXiv arXiv 2023

  23. [23]

    John P. A. Ioannidis. Why most published research findings are false. PLOS Medicine, 2 0 (8): 0 null, 08 2005. doi:10.1371/journal.pmed.0020124. URL https://doi.org/10.1371/journal.pmed.0020124

    doi:10.1371/journal.pmed.0020124 2005

  24. [24]

    BASIC GUIDELINES FOR RESEARCH: An Introductory Approach for All Disciplines

    Sajjad Kabir. BASIC GUIDELINES FOR RESEARCH: An Introductory Approach for All Disciplines. 07 2016. ISBN 978-984-33-9565-8

    2016

  25. [25]

    An Interpretability Illusion for Activation Patching of Arbitrary Subspaces

    Georg Lange, Alex Makelov, and Neel Nanda. An Interpretability Illusion for Activation Patching of Arbitrary Subspaces . August 2023. URL https://www.lesswrong.com/posts/RFtkRXHebkwxygDe2/an-interpretability-illusion-for-activation-patching-of

    2023

  26. [26]

    Subhash R. Lele. How Should We Quantify Uncertainty in Statistical Inference ? Frontiers in Ecology and Evolution, 8, March 2020. ISSN 2296-701X. doi:10.3389/fevo.2020.00035. URL https://www.frontiersin.org/journals/ecology-and-evolution/articles/10.3389/fevo.2020.00035/full

    arXiv 2020

  27. [27]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    2025

  28. [28]

    Mechanistic interpretability meets vision language models: Insights and limitations

    Yiming Liu, Yuhui Zhang, and Serena Yeung-Levy. Mechanistic interpretability meets vision language models: Insights and limitations. In ICLR Blogposts 2025, 2025. URL https://d2jud02ci9yv69.cloudfront.net/2025-04-28-vlm-understanding-29/blog/vlm-understanding/. https://d2jud02ci9yv69.cloudfront.net/2025-04-28-vlm-understanding-29/blog/vlm-understanding/

    2025

  29. [29]

    Error and the growth of experimental knowledge

    Deborah Mayo. Error and the growth of experimental knowledge. Bibliovault OAI Repository, the University of Chicago Press, 92, 04 1998. doi:10.1002/(SICI)1520-6696(199823)34:43.0.CO;2-E

    work page doi:10.1002/(sici)1520-6696(199823)34:43.0.co;2-e 1998

  30. [30]

    Everything, everywhere, all at once: Is mechanistic interpretability identifiable? In The Thirteenth International Conference on Learning Representations, 2025

    Maxime M \'e loux, Silviu Maniu, Fran c ois Portet, and Maxime Peyrard. Everything, everywhere, all at once: Is mechanistic interpretability identifiable? In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=5IWJBStfU7

    2025

  31. [31]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088

    2022

  32. [32]

    Mass-editing memory in a transformer

    Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=MkbcAHIYgyS

    2023

  33. [33]

    Opening the ai black box: Distilling machine-learned algorithms into code

    Eric Michaud, Isaac Liao, Vedang Lad, Ziming Liu, Anish Mudide, Chloe Loughridge, Zifan Guo, Tara Kheirkhah, Mateja Vukelić, and Max Tegmark. Opening the ai black box: Distilling machine-learned algorithms into code. Entropy, 26: 0 1046, 12 2024. doi:10.3390/e26121046

    work page doi:10.3390/e26121046 2024

  34. [34]

    Transformer circuit evaluation metrics are not robust

    Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=zSf8PJyQb2

    2024

  35. [35]

    A glitch in the matrix? locating and detecting language model grounding with fakepedia, 2024

    Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kıcıman, Hamid Palangi, Barun Patra, and Robert West. A glitch in the matrix? locating and detecting language model grounding with fakepedia, 2024. URL https://arxiv.org/abs/2312.02073

    Pith/arXiv arXiv 2024

  36. [36]

    The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability, 2024

    Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability, 2024. URL https://arxiv.org/abs/2408.01416

    arXiv 2024

  37. [37]

    Refining targeted syntactic evaluation of language models

    Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. Refining targeted syntactic evaluation of language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the A...

    2021

  38. [38]

    The building blocks of interpretability

    Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi:10.23915/distill.00010. https://distill.pub/2018/building-blocks

    doi:10.23915/distill.00010 2018

  39. [39]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

    doi:10.23915/distill.00024.001 2020

  40. [40]

    Direct and indirect effects

    Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI'01, pp.\ 411–420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001

    2001

  41. [41]

    K. R. Popper. The Logic of Scientific Discovery. Hutchinson, London, 1934

    1934

  42. [42]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  43. [43]

    Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

    Tilman Rauker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks . In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp.\ 464--483, Los Alamitos, CA, USA, February 2023. IEEE Computer Society. doi:10.1109/SaTML54575.2023.00039. URL htt...

    arXiv 2023

  44. [44]

    Claudia Shi, Nicolas Beltran-Velez, Achille Nazaret, Carolina Zheng, Adrià Garriga-Alonso, Andrew Jesson, Maggie Makar, and David M. Blei. Hypothesis testing the circuit hypothesis in llms, 2024. URL https://arxiv.org/abs/2410.13032

    Pith/arXiv arXiv 2024

  45. [45]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pp.\ 3319–3328. JMLR.org, 2017

    2017

  46. [46]

    Attribution patching outperforms automated circuit discovery

    Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In NeurIPS Workshop on Attributing Model Behavior at Scale, 2023 a . URL https://openreview.net/forum?id=tiLbFR4bJW

    2023

  47. [47]

    Attribution patching outperforms automated circuit discovery, 2023 b

    Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery, 2023 b . URL https://arxiv.org/abs/2310.10348

    Pith/arXiv arXiv 2023

  48. [48]

    Explanation in causal inference: developments in mediation and interaction

    Tyler J VanderWeele. Explanation in causal inference: developments in mediation and interaction. International Journal of Epidemiology, 45 0 (6): 0 1904--1908, 11 2016. ISSN 0300-5771. doi:10.1093/ije/dyw277. URL https://doi.org/10.1093/ije/dyw277

    work page doi:10.1093/ije/dyw277 1904

  49. [49]

    Investigating gender bias in language models using causal mediation analysis

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 12388--12401. Curran Associat...

    2020

  50. [50]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. ArXiv, abs/2004.12265, 2020 b . URL https://api.semanticscholar.org/CorpusID:216553696

    Pith/arXiv arXiv 2004

  51. [51]

    Fabian Walke, Lars Bennek, and Till J. Winkler. Artificial intelligence explainability requirements of the ai act and metrics for measuring compliance. In Daniel Beverungen, Christiane Lehrer, and Matthias Trier (eds.), Solutions and Technologies for Responsible Digitalization, pp.\ 113--129, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-80122-8

    2025

  52. [52]

    Interpretability in the wild: a circuit for indirect object identification in GPT -2 small

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT -2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=NpsVSN6o4ul

    2023

  53. [53]

    Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BL i MP : The benchmark of linguistic minimal pairs for E nglish. Transactions of the Association for Computational Linguistics, 8: 0 377--392, 2020. doi:10.1162/tacl_a_00321. URL https://aclanthology.org/2020.tacl-1.25/

    doi:10.1162/tacl_a_00321 2020

  54. [54]

    Wasserstein and Nicole A

    Ronald L. Wasserstein and Nicole A. Lazar. The asa statement on p-values: Context, process, and purpose. The American Statistician, 70 0 (2): 0 129--133, 2016. doi:10.1080/00031305.2016.1154108. URL https://doi.org/10.1080/00031305.2016.1154108

    Pith/arXiv arXiv 2016

  55. [55]

    Functional faithfulness in the wild: Circuit discovery with differentiable computation graph pruning

    Lei Yu, Jingcheng Niu, Zining Zhu, and Gerald Penn. Functional faithfulness in the wild: Circuit discovery with differentiable computation graph pruning. CoRR, abs/2407.03779, 2024. URL https://doi.org/10.48550/arXiv.2407.03779

    doi:10.48550/arxiv.2407.03779 2024

  56. [56]

    Zeiler and Rob Fergus

    Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision -- ECCV 2014, pp.\ 818--833, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10590-1

    2014

  57. [57]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Hf17y6u9BC

    2024

  58. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  59. [59]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  60. [60]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  61. [61]

    ground truth

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv