Pith. sign in

REVIEW 5 major objections 5 minor 66 references

Circuit Stability Characterizes Language Model Generalization

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model generalizes when its reasoning stays consistent across subtasks.

desk verdict A genuinely new formal definition of circuit stability, but the headline 'predicts generalization' claim is not actually tested out-of-sample, so the paper is a promising start rather than a finished contribution. read the letter →

arxiv 2505.24731 v1 pith:HUZWKCO2 submitted 2025-05-30 cs.CL

Alan Sun This is my paper
classification cs.CL
keywords circuitstabilitysoftcircuitsmechanisticinterpretabilitygeneralizationlanguagemodelsattributionpatchingchain-of-thoughtpromptingbenchmarksaturation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a language model generalizes when it applies the same reasoning process, its circuit, across related subtasks. The author formalizes "soft circuits" as edge-importance scores computed by ablating each edge of the model's computational graph, then defines circuit stability as the rank correlation between the soft circuits of different subtasks. Three case studies—arithmetic addition, Boolean expression evaluation, and sports understanding—show that high stability tracks successful length, structural, and compositional generalization, while instability flags where the model fails. If the claim holds, evaluating generalization would not require building ever-harder benchmarks; one could instead measure whether a model's internal reasoning is consistent. The paper offers this as a step toward tying interpretability directly to generalization.

What carries the argument

The central object is the soft circuit, a continuous relaxation of the discrete circuit used in mechanistic interpretability: instead of marking each edge in the computational graph as in or out, every edge receives a real score $c(e) = \mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathcal{L}(M_{\{e\}}(x),y) - \mathcal{L}(M(x),y)]$, the expected change in a performance metric when that edge is ablated. Soft circuits are computed with edge attribution patching with integrated gradients (EAP-IG), a Taylor-approximation of activation patching that needs only a constant number of forward and backward passes. Similarity between soft circuits is measured by Spearman's $\rho$, and this feeds the two formal definitions: $\varepsilon$-circuit stability (the expected pairwise similarity across subtasks exceeds $\varepsilon$) and $\alpha$-equivalence (two subtasks' circuits correlate at least $\alpha$). The paper's mechanism is to partition a task into subtasks, compute one soft circuit per subtask, and read generalization off the resulting cluster structure.

What would settle it

Find a task where a model's accuracy stays flat across a subtask boundary but the EAP-IG soft circuits change sharply; if circuit stability is low while generalization is perfect, stability is not necessary for generalization. The reverse—high stability with a large accuracy cliff—would also falsify sufficiency. A concrete candidate is addition with a carry boundary, where the correct algorithm may change with operand length while performance remains high.

Watch

Extended reading notes

Core claim

The central claim is that circuit stability—a model's tendency to reuse the same computational subgraph when solving inputs drawn from different subtasks—is a predictor and characterizer of generalization. Stability is measured on soft circuits, where each edge of the transformer's computational graph receives a real-valued importance equal to the expected change in a performance metric when that edge is ablated. Soft circuits are compared with Spearman's rank correlation, and two subtasks are $\alpha$-equivalent when their circuits correlate above a threshold. In gemma-2-2b on addition, subtasks cluster into circuit families that match performance: commutativity violations show up as different circuits for (o1,o2) versus (o2,o1), and within a family accuracy decays smoothly and predictably. In phi-1.5 on Boolean expressions, instability across parenthesization reveals whether the model has internalized operator precedence and associativity. On sports understanding, chain-of-thought prompting raises circuit stability while raising accuracy, suggesting stability is an actionable, not just descriptive, quantity.

Load-bearing premise

The load-bearing premise is that soft circuits, as computed by edge attribution patching with integrated gradients, faithfully reflect the model's actual reasoning, so that a high rank correlation between two soft circuits really means the model is reasoning consistently.

Editorial extensions

If this is right

  • For arithmetic, circuit instability across digit-length subtasks predicts where gemma-2-2b's exact-match accuracy drops, including the 20-plus percentage-point gap between commutative partners.
  • For Boolean expressions, the presence or absence of circuit equivalence across parenthesized and non-parenthesized subtasks tells whether phi-1.5 has internalized associativity and operator precedence.
  • Chain-of-thought prompting increases circuit stability on sports understanding for Llama-3.1-8b and Gemma-2-9b, linking interpretability metrics to an intervention that improves accuracy.
  • Because soft circuits can be estimated efficiently, circuit stability offers a benchmark-free signal for tracking and comparing model generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical extension the paper does not pursue: circuit stability could be computed on a held-out set of subtasks before new benchmarks are built, turning it into a cheap screen for which capability gaps are worth probing.
  • The faithfulness assumption could be tested directly: if EAP-IG scores are replaced by exact activation patching on a small subset of edges, the stability rankings should survive; if they flip, stability is an artifact of the approximation.
  • The framework suggests a definition of a learned skill as an equivalence class of soft circuits, which connects to compositionality research where a skill is a reusable subroutine.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper formalizes circuit stability and circuit equivalence for transformer language models, defining soft circuits via edge attribution patching and using Spearman rank correlation as a similarity measure. It reports three case studies: two-operand addition on gemma-2-2b, Boolean expression evaluation on phi-1.5, and sports understanding with chain-of-thought prompting on Llama-3.1-8b and Gemma-2-9b. The central claim, stated in the abstract, is that circuit stability and instability can characterize and predict different aspects of generalization, offering a way to assess models without constructing new benchmarks. The paper also contributes formal definitions, an open-source codebase, and a discussion of circuit discovery methods.

Significance. If the central claim were established, circuit stability would be a valuable and low-cost evaluation signal, connecting mechanistic interpretability to generalization prediction. The formalization of soft circuits and equivalence is a useful step, and the release of code plus detailed experimental settings supports reproducibility. However, the empirical evidence is confined to retrospective, in-sample analyses: the threshold that defines circuit families is chosen after inspecting the data, and the accuracy comparisons use the same subtasks. The predictive claim is therefore not actually tested, and several load-bearing assumptions, such as the faithfulness of EAP-IG soft circuits, are left unvalidated. The idea is promising, but the current evidence does not support the advertised prediction capability.

major comments (5)
  1. [Section 4.2, Figures 1–2, and Figure 4] The central predictive claim is not tested out-of-sample. The threshold alpha=0.6 is selected after inspecting Figure 2, and the resulting clusters are compared with accuracies measured on the same 64 subtasks in Figure 4. This is a retrospective, in-sample description: neither the threshold nor the clusters are used to forecast accuracy on held-out subtasks or held-out tasks. The t-SNE in Figure 1 is computed from the same circuits and does not provide independent validation because low perplexity t-SNE can create separated blobs even from noise. To support the abstract's 'predict' claim, the authors would need to fix the partition and alpha a priori, or select them on a training set and evaluate predictive accuracy on a held-out set of subtasks or tasks.
  2. [Section 3, Definition 4, and Section 4.2] Definition 4 (Eq. 2) is never instantiated in the experiments. The definition requires an infimum over a collection of partitions and an expectation over subtask pairs drawn from the partition distribution. Instead, the arithmetic case study computes pairwise Spearman correlations between all 64 soft circuits and thresholds them at alpha=0.6 to form connected components. This measures a different quantity: the empirical clustering of pairwise alpha-equivalence, not circuit stability as formally defined. At minimum, the paper should either compute the defined quantity or explicitly state that the experiments operationalize a weaker, pairwise notion and justify its relationship to Definition 4.
  3. [Appendix B, Definition 3, and Section 2] The faithfulness of EAP-IG soft circuits is assumed rather than established. Section 2 cites Miller et al. (2024) showing that circuit discovery is sensitive to the performance metric and threshold, but no robustness analysis is provided for the soft circuits used here. If the soft-circuit rankings are not faithful to the model's actual causal structure, then Spearman correlation between them does not measure consistency of reasoning. A concrete test would be to compare EAP-IG scores against activation patching on a subset of edges, or to verify that the top-ranked edges recover previously identified arithmetic circuits (e.g., for addition). Without such validation, the connection between 'circuit stability' and generalization is not established.
  4. [Section 5.2, Not subtask] The statement 'not is associative' is mathematically incorrect. The operator 'not' is unary, so associativity is not defined for it. The intended claim is likely that redundant parentheses do not change the semantics of an expression containing only 'not' and literals, which is true but requires a different justification. This error weakens the interpretation of the Not subtask results: the observed 40% performance drop after adding parentheses is still evidence of a problem, but the asserted reason (failure to understand associativity) is unsound.
  5. [Section 4.3, Figure 4] The claim that performance differences between subtask clusters are 'not merely an artifact of statistical noise' is not supported by uncertainty quantification. Figure 4 reports exact-match accuracies with n=1000 per subtask but no error bars, confidence intervals, or significance tests. Given that several adjacent cells differ by only a few percent, and the cluster boundary differences are sometimes around 10–20%, the lack of error bars makes it impossible to verify the claim. At minimum, the authors should report bootstrap intervals or pairwise significance tests for the cluster boundary differences.
minor comments (5)
  1. [Section 6] In the sentence 'we see that chain-of-thought significantly circuit stability', a verb appears to be missing; it should likely read 'significantly improves circuit stability'.
  2. [Section 5, introductory paragraph] The word 'meaingful' should be 'meaningful'.
  3. [Appendix A] The phrase 'well-studied in the the literature' contains a duplicated 'the'.
  4. [Figure 3] The multiple panels in Figure 3 are not labelled with the corresponding alpha values, which makes it difficult to follow the description of how circuit families emerge as alpha increases.
  5. [Appendix B] The abbreviation 'EA-IG' is used in one place; it should be 'EAP-IG' to match the edge attribution patching with integrated gradients terminology used elsewhere.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the circuit-stability measure is defined independently of accuracy, and the reported alignments are in-sample correlations rather than mathematical identities.

full rationale

The formal framework (Definitions 1-5) is self-contained: soft circuits are defined as expected loss differences (Eq. 1), and stability is defined as an infimum over expected rank correlations of those soft circuits. Accuracy does not enter the definition of a soft circuit or of circuit stability, so comparing circuit clusters to exact-match accuracy in Figures 4, 6, and 8 is an external correlation, not a restatement of the input. The alpha = 0.6 threshold is selected from the cluster-count curve in Figure 2, which is computed only from circuit similarities, not from accuracy; the subsequent alignment with accuracy is an in-sample association that limits the strength of the word 'predict' but does not make the circuit-stability measure equivalent to the accuracy it is compared with. Appendix C's exponential regressions are explicitly described as tight-fitting in-sample characterizations (R^2 > 0.99 on the fitted data), not as out-of-sample predictions, and they are not the central evidence for the paper's main claim. The only self-citation, Sun et al. (2024) in Appendix A, is a minor borrowing of a computational-graph construction and is not load-bearing. Section 6 candidly acknowledges that its random-partition procedure does not create true subtasks, which is a limitation rather than a circular step. Overall, no derived quantity in the paper reduces by construction to its own inputs, and no load-bearing argument rests on a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the choice of alpha, the faithfulness of EAP-IG soft circuits, and the validity of Spearman correlation as a measure of algorithmic consistency. None of these are independently validated.

free parameters (4)
  • alpha equivalence threshold = 0.6
    Chosen after inspecting the number-of-families curve (Figure 2); clusters derived with this threshold are then used to explain performance differences in the same data.
  • hard circuit size = 200 components
    Greedy decoding uses top 200 edges by soft circuit score; this cutoff is chosen by hand.
  • exponential regression parameters (a,b,c) = e.g., y = 1 - 0.027(0.95)^(4.066x), R^2 = 0.9978
    Fit to 8 data points per family in Appendix C; high R^2 with three free parameters on few points is not strong evidence.
  • few-shot count k = 3
    Chosen for the experimental setup; not the main concern.
assumptions (4)
  • domain assumption Computational graph representation: MLP layers as single nodes and each attention head split into key, query, value, output nodes.
    Borrowed from prior work; the level of graph abstraction affects all circuit computations.
  • domain assumption Soft circuit c(e) = E[L(M_{e}(x),y) - L(M(x),y)] is a meaningful importance measure for reasoning consistency.
    The central quantity of the paper; no validation that this expectation reflects algorithmic similarity.
  • domain assumption Spearman's rho is an appropriate similarity kernel K for circuits.
    The authors state this is chosen for interpretability, not derived from theory.
  • ad hoc to paper The paper claims 'not is associative' in Section 5.2.
    Mathematically incorrect: unary not is not an associative binary operation; this incorrect premise underpins the expected circuit equivalence for Not-only expressions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Circuit Stability Characterizes Language Model Generalization." pith.science (2026). https://pith.science/paper/HUZWKCO2

@misc{pith2026250524731,
  author       = {Pith},
  title        = {Pith review of: Circuit Stability Characterizes Language Model Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HUZWKCO2}},
  note         = {Machine review of arXiv:2505.24731}
}
read the original abstract

Extensively evaluating the capabilities of (large) language models is difficult. Rapid development of state-of-the-art models induce benchmark saturation, while creating more challenging datasets is labor-intensive. Inspired by the recent developments in mechanistic interpretability, we introduce circuit stability as a new way to assess model performance. Circuit stability refers to a model's ability to apply a consistent reasoning process-its circuit-across various inputs. We mathematically formalize circuit stability and circuit equivalence. Then, through three case studies, we empirically show that circuit stability and the lack thereof can characterize and predict different aspects of generalization. Our proposed methods offer a step towards rigorously relating the generality of models to their interpretability.

Figures

Figures reproduced from arXiv: 2505.24731 by the authors.

Figure 1
Figure 1. (top). To confirm that these clusters are not merely an artifact of a particular setting of 𝛼, we perform two complementary experiments. First, we directly compute a set of 𝑡-SNE embeddings using all the soft circuits (Maaten and Hinton, 2008). Notably, these embeddings are independent of 𝛼. The rela￾tive distances between the embedded circuits are visualized in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The number of 𝛼-circuit equivalence families as 𝛼 varies between [0, 1]. We omit visualization of 𝛼 < 0 since the number of circuit families is a monotonic function of 𝛼. The red region shows that 80% of circuit families emerge between 𝛼 = 0.58 and 𝛼 = 0.79 1,1 1,2 1,3 1,4 1,5 1,6 1,7 1,8 2,1 2,2 2,3 2,4 2,5 2,6 2,7 2,8 3,1 3,2 3,3 3,4 3,5 3,6 3,7 3,8 4,1 4,2 4,3 4,4 4,5 4,6 4,7 4,8 5,1 5,2 5,3 5,4 5,5 5,6 5,7 5,8 6… view at source ↗
Figure 3
Figure 3. The emergent families of 𝛼-equivalent sub￾tasks as 𝛼 varies between [0, 1]. gence and persistence of distinct families corre￾sponding to equal-digit, single-digit, and one-digit difference subtasks. The stability of these families over increasing 𝛼 suggests strong internal cohesion, further validating our proposed clustering depicted in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance of gemma-2-2b over arithmetic subtasks as Opr1 B 𝑜1 and Opr2 B 𝑜2 increase. Each cell denotes the exact string match accuracy. one (in circuit) and the remaining zero (out of cir￾cuit). In contrast to the sharp performance change across non-equivalent subta…
Figure 6
Figure 6. Figure 6: Circuit stability for phi-1.5 within and across six subtasks. A permutation test is performed between parenthetical and non-parenthetical pairs of subtasks. “∗” denotes a statistically significant difference with a setting of 𝑝 < 0.05. The error bars denote a 95% confi…
Figure 8
Figure 8. Figure 8: (right). Across both models, we see that chain-of-thought significantly circuit stability. It should be noted that technically the partition strategy we employ herein is not creating true sub￾tasks (see Definition 2). This is because we are sampling from 𝒟𝒳×𝒴 a finite …
Figure 9
Figure 9. Figure 9: Exponential regression on exact string accu [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Exponential regression on exact string accu [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 40 canonical work pages

  1. [1]

    Vilas, and Todd Wareham

    Federico Adolfi, Martina G. Vilas, and Todd Wareham. 2025. https://openreview.net/forum?id=QogcGNXJVw The computational complexity of circuit discovery for inner interpretability . In The Thirteenth International Conference on Learning Representations

    work page 2025

  2. [2]

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/f545448535dfde4f9786555403ab7c49-Paper-Conference.pdf Refusal in Language Models Is Mediated by a Single Direction . In Advances in Neural Information Processing Systems , volume 37, pages 136...

    work page 2024

  3. [3]

    Sanjeev Arora and Anirudh Goyal. 2023. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936

    arXiv 2023

  4. [4]

    Sander Beckers and Joseph Y Halpern. 2019. Abstracting causal models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 2678--2685. Issue: 01

    work page 2019

  5. [5]

    Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/20fdaf67581e6d7157376d1ed584040a-Paper-Conference.pdf Finding Transformer Circuits With Edge Pruning . In Advances in Neural Information Processing Systems , volume 37, pages 18506--18534. Curran Associates, Inc

    work page 2024

  6. [6]

    Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth. 1987. https://doi.org/10.1016/0020-0190(87)90114-1 Occam's Razor . Information Processing Letters, 24(6):377--380

    doi:10.1016/0020-0190(87)90114-1 1987

  7. [7]

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Ma \ n as, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. 2024. An introduction to vision-language modeling. arXiv preprint arXiv:2405.17247

    arXiv 2024

  8. [8]

    Hanseul Cho, Jaeyoung Cha, Srinadh Bhojanapalli, and Chulhee Yun. 2025. https://openreview.net/forum?id=eIgGesYKLG Arithmetic transformers can length-generalize in both operand length and count . In The Thirteenth International Conference on Learning Representations

    work page 2025

Show all 66 references
  1. [9]

    Bilal Chughtai, Lawrence Chan, and Neel Nanda. 2023. https://arxiv.org/pdf/2302.03025 A Toy Model of Universality : Reverse Engineering how Networks Learn Group Operations . In Proceedings of the 40th International Conference on Machine Learning

    2023 arXiv

  2. [10]

    Arthur Conmy, Augustine Parker-Mavor N., Aengus Lynch, Stefan Heimersheim, and Adria Alonso-Garriga. 2023. https://arxiv.org/abs/2304.14997 Towards Automated Circuit Discovery for Mechanistic Interpretability . In Thirty- Seventh Conference on Neural Information Processing Systems

    2023 arXiv

  3. [11]

    Ryan Cotterell, Anej Svete, Clara Meister, Tianyu Liu, and Li Du. 2023. Formal aspects of language modeling. arXiv preprint arXiv:2311.04329

    2023 arXiv

  4. [12]

    Patrick Cousot and Radhia Cousot. 1977. https://doi.org/10.1145/512950.512973 Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints . In Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles o...

    1977

  5. [13]

    Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2023. https://doi.org/10.18653/v1/2023.acl-long.543 A measure-theoretic characterization of tight language models . In Proceedings of the 61st Annual Meeting of the Association for ...

    2023 doi

  6. [14]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, DasSarma, Nova , Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Da...

    2021

  7. [15]

    Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/4f5c422f4d49a5a807eda27434231040-Paper.pdf Causal abstractions of neural networks . In Advances in Neural Information Processing Systems, volume 34,...

    2021

  8. [16]

    Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. https://proceedings.mlr.press/v236/geiger24a.html Finding alignments between interpretable causal variables and distributed neural representations . In Proceedings of the Third Conference on...

    2024

  9. [17]

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. 2024. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint ...

    2024 arXiv

  10. [18]

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R\' e . 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/05546b0e38ab9175cd905eebcc6ebb76-Paper.pdf Combining recurrent, convolutional, and continuous-time models with linear...

    2021

  11. [19]

    Rohan Gupta, Iván Arcuschin, Thomas Kwa, and Adrià Garriga-Alonso. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/a8f7d43ae092d9a5295775eb17f3f4f7-Paper-Datasets_and_Benchmarks_Track.pdf InterpBench : Semi - Synthetic Transformers for Evaluating Mechanistic I...

    2024

  12. [20]

    Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. https://openreview.net/forum?id=p4PckNQR8k How does GPT -2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  13. [21]

    Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. https://openreview.net/forum?id=TZ0CCGDcuT Have Faith in Faithfulness : Going Beyond Circuit Overlap When Finding Model Mechanisms . In First Conference on Language Modeling

    2024

  14. [22]

    Mark H Hansen and Bin Yu and. 2001. https://doi.org/10.1198/016214501753168398 Model selection and the principle of minimum description length . Journal of the American Statistical Association, 96(454):746--774

    2001 doi

  15. [23]

    Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. 2024. https://openreview.net/forum?id=aVh9KRZdRk Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks . In The Thirty-eighth Annual Conference on Neural Information Process...

    2024

  16. [24]

    Stefan Heimersheim and Neel Nanda. 2024. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255

    2024 arXiv

  17. [25]

    Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. 2024. https://openreview.net/forum?id=R2sVqqTf9p Grokking and the geometry of circuit formation . In ICML 2024 Workshop on Mechanistic Interpretability

    2024

  18. [26]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations

    2024

  19. [27]

    Greg Kamradt. 2023. https://github.com/gkamradt/LLMTest_NeedleInAHaystack LLM Test - Needle in a Haystack

    2023

  20. [28]

    J \'a nos Kram \'a r, Tom Lieberum, Rohin Shah, and Neel Nanda. 2024. Atp*: An efficient and scalable method for localizing llm behaviour to components. arXiv preprint arXiv:2403.00745

    2024 arXiv

  21. [29]

    Keito Kudo, Yoichi Aoki, Tatsuki Kuribayashi, Ana Brassard, Masashi Yoshikawa, Keisuke Sakaguchi, and Kentaro Inui. 2023. https://doi.org/10.18653/v1/2023.eacl-main.98 Do deep neural networks capture compositionality in arithmetic reasoning? In Proceedings of the 17th Conferen...

    2023 doi

  22. [30]

    Kummerfeld, and Rada Mihalcea

    Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. https://openreview.net/forum?id=dBqHGZPGZI A mechanistic understanding of alignment algorithms: A case study on DPO and toxicity . In Forty-first International Conference ...

    2024

  23. [31]

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. https://arxiv.org/abs/2309.05463 Textbooks are all you need ii: phi-1.5 technical report . Preprint, arXiv:2309.05463

    2023 arXiv

  24. [32]

    Tom Lieberum, Matthew Rahtz, J \'a nos Kram \'a r, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458

    2023 arXiv

  25. [33]

    Laurens van der Maaten and Geoffrey Hinton. 2008. http://jmlr.org/papers/v9/vandermaaten08a.html Visualizing Data using t- SNE . Journal of Machine Learning Research, 9(86):2579--2605

    2008

  26. [34]

    Andonian, and Yonatan Belinkov

    Kevin Meng, David Bau, Alex J. Andonian, and Yonatan Belinkov. 2022. https://openreview.net/forum?id=-h6WAS6eE4 Locating and Editing Factual Associations in GPT . In Advances in Neural Information Processing Systems

    2022

  27. [35]

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023. https://openreview.net/forum?id=fpoAYV6Wsk Circuit Component Reuse Across Tasks in Transformer Language Models . In The Twelfth International Conference on Learning Representations

    2023

  28. [36]

    Joseph Miller, Bilal Chughtai, and William Saunders. 2024. https://openreview.net/forum?id=zSf8PJyQb2 Transformer circuit evaluation metrics are not robust . In First Conference on Language Modeling

    2024

  29. [37]

    Neel Nanda. 2023. Attribution patching: Activation patching at industrial scale. URL: https://www. neelnanda. io/mechanistic-interpretability/attribution-patching

    2023

  30. [38]

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2022. https://openreview.net/forum?id=9XFSbDPmdW Progress measures for grokking via mechanistic interpretability . In The Eleventh International Conference on Learning Representations

    2022

  31. [39]

    Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2025. https://openreview.net/forum?id=O9YTt26r2P Arithmetic without algorithms: Language models solve math with a bag of heuristics . In The Thirteenth International Conference on Learning Representations

    2025

  32. [40]

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. https://doi.org/10.23915/distill.00024.001 Zoom in: An introduction to circuits . Distill. Https://distill.pub/2020/circuits/zoom-in

    2020 doi

  33. [41]

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  34. [42]

    Jun Otsuka and Hayato Saigo. 2022. https://proceedings.mlr.press/v177/otsuka22a.html On the Equivalence of Causal Models : A Category - Theoretic Approach . In Proceedings of the First Conference on Causal Learning and Reasoning , volume 177 of Proceedings of Machine Learning ...

    2022

  35. [43]

    Judea Pearl. 2009. Causality. Cambridge University Press

    2009

  36. [44]

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. 2022. https://arxiv.org/abs/2201.02177 Grokking: Generalization Beyond Overfitting on Small Datasets

    2022 arXiv

  37. [45]

    Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, et al. 2024. https://doi.org/10.48550/arXiv.2408.00118 Gemma 2: Improving open language models at a practical size . CoRR, abs/2408.00118

    2024 doi arXiv

  38. [46]

    Milad Sefidgaran, Abdellatif Zaidi, and Piotr Krasnowski. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/054e9f9a286671ababa3213d6e59c1c2-Paper-Conference.pdf Minimum Description Length and Generalization Guarantees for Representation Learning . In Advances i...

    2023

  39. [47]

    Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding machine learning: From theory to algorithms . Cambridge university press

    2014

  40. [48]

    Claudia Shi, Nicolas Beltran-Velez, Achille Nazaret, Carolina Zheng, Adri\` a Garriga-Alonso, Andrew Jesson, Maggie Makar, and David M. Blei. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/abccb8a90b30d45b948360ba41f5a20f-Paper-Conference.pdf Hypothesis testi...

    2024

  41. [49]

    Brown, et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, et al. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Transactions on...

    2023

  42. [50]

    Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.435 A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis . In Proceedings of the 2023 Conference on Empirical Methods ...

    2023 doi

  43. [51]

    Alan Sun, Chiyu Ma, Kenneth Ge, and Soroush Vosoughi. 2024. https://openreview.net/forum?id=v07KRLYxDX Achieving domain-independent certified robustness via knowledge continuity . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  44. [52]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  45. [53]

    Aaquib Syed, Can Rager, and Arthur Conmy. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.25 Attribution patching outperforms automated circuit discovery . In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407--416, ...

    2024 doi

  46. [54]

    Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. 2024. https://openreview.net/forum?id=3Ds5vNudIE LLM Circuit Analyses Are Consistent Across Training and Scale . In The Thirty -eighth Annual Conference on Neural Information Processing Systems

    2024

  47. [55]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30

    2017

  48. [56]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. https://proceedings.neurips.cc/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf Investigating gender bias in language models using causal mediation anal...

    2020

  49. [57]

    Vilas, Federuci Adolfi, David Poeppel, and Gemma Roig

    Martina G. Vilas, Federuci Adolfi, David Poeppel, and Gemma Roig. 2024. https://arxiv.org/pdf/2406.01352 Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Science . In Proceedings of the 41st International Conference on Machine Learning

    2024 arXiv

  50. [58]

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. https://openreview.net/forum?id=NpsVSN6o4ul Interpretability in the Wild : a Circuit for Indirect Object Identification in GPT -2 Small . In The Eleventh International Conference on ...

    2022

  51. [59]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of- Thought Prompting Elicits Reasonin...

    2022

  52. [60]

    Thadd \"a us Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. 2023. https://openreview.net/forum?id=LqOQ1uJmSx Compositional generalization from first principles . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  53. [61]

    Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/f6a8b109d4d4fd64c75e94aaf85d9697-Paper-Conference.pdf Interpretability at scale: Identifying causal mechanisms in alpaca . In Advan...

    2023

  54. [62]

    Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/c5c1cb0bebd56ae38817b251ad72bedb-Paper.pdf Towards a theoretical framework of out-of-distribution generalization . In Advances in Neu...

    2021

  55. [63]

    Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. 2024. https://openreview.net/forum?id=Jf5gplvglq SKILL - MIX : a flexible and expandable family of evaluations for AI models . In The Twelfth International Conference on Learning Representations

    2024

  56. [64]

    Zhiqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. 2023. https://arxiv.org/pdf/2306.17844 The Clock and the Pizza : Two Stories in Mechanistic Explanation of Neural Networks . In Advances in Neural Information Processing Systems

    2023 arXiv

  57. [65]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  58. [66]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.