Pith. sign in

REVIEW 3 major objections 5 minor 63 references

Explaining Neural Networks with Reasons

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single formula turns neuron activations into reasons for predictions

desk verdict A genuinely new reasons-based interpretability method with solid faithfulness results and an overclaimed correctness claim. read the letter →

arxiv 2505.14424 v1 pith:PRSNTZXX submitted 2025-05-20 cs.LG

Levin Hornischer , Hannes Leitgeb This is my paper · ORCID
classification cs.LG
keywords interpretabilityneuralnetworksreasonsepistemicfaithfulnessactivationpatchingrobustnessfairness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that every neuron in a trained neural network can be treated as an epistemic reason: a vector of its activations across a sample of inputs, with a formula that gives the strength with which that vector speaks for a proposition such as 'this image shows a 3' or 'this sentence has negative sentiment.' The method needs only forward passes, and the authors argue—with theory and experiments on image, tabular, and text models—that this reason reading is faithful (intervening on the neurons it flags changes outputs as predicted), correct (the model's reason structure mirrors the data's), trainable (models can be trained to have stronger, fairer reasons), and useful (improving reasons improves adversarial robustness and fairness). If correct, this would give a scalable, architecture-agnostic route to mechanistic interpretability that does not require training separate probes or sparse autoencoders.

What carries the argument

The load-bearing object is the reasons vector $r_u$ together with the strength functional $D$ from Equation (2). $r_u$ is simply the activation pattern of a neuron across a chosen set of sample worlds, and $D$ measures the change in log-odds that exponentially tilting the prior belief with $r_u$ produces for a proposition $A$. The paper uses two structural facts: aggregation of reasons by vector addition commutes with the belief update, and the underlying axiomatic theory of reasons yields a unique model (up to a multiplicative constant, set to $c=1$). This machinery converts interpretability into a forward-pass computation and connects it with both a logical reading (a neuron supports propositions) and a Bayesian reading (a neuron updates probabilities).

What would settle it

Train a model on a synthetic dataset with a known ground-truth reason (e.g., label is true exactly when feature $f$ is positive) and check whether the neuron with the highest $D$ for that label is the one that actually encodes feature $f$; if the top neurons do not match the causal drivers, or if intervening on them fails to flip outputs on out-of-distribution inputs, the faithfulness claim is refuted.

Watch

Extended reading notes

Core claim

The central discovery is the 'reasons method'. Fix a set $W$ of possible worlds (for example, 1024 input–label pairs from a test set). For each neuron $u$, form its reasons vector $r_u$ whose $k$-th component is $u$'s activation in world $w_k$. Relative to a prior belief $b$ over $W$, the strength with which $r_u$ speaks for a proposition $A$ is $D(r_u,A,b,W)=\frac{1}{2}\log\left(\frac{b*r_u(A)/b*r_u(A^c)}{b(A)/b(A^c)}\right)$, where $b*r_u$ is the exponential-tilt update of $b$ by $r_u$. The paper claims that this single scalar, computed by forward passes alone, gives a grounded, faithful, and correct interpretation of single neurons and of groups (whose reasons vectors are added), and that adding a loss that maximizes these strengths during training improves robustness and fairness without sacrificing accuracy.

Load-bearing premise

The method rests on assuming that a neuron's activation pattern across a sample of inputs can be treated as an epistemic reason whose strength is the log-odds change in Equation (2), and that 'internal similarity' between inputs is faithfully captured by Euclidean distance of their activation vectors and 'external similarity' by sharing the same label.

Editorial extensions

If this is right

  • Any neuron in any standard architecture—CNN, MLP, transformer—can be assigned a strength for any proposition using only forward passes, making interpretability scalable without training auxiliary networks.
  • Activation patching according to reason strengths should reliably flip model predictions, giving a causal check that the interpretation tracks the network's mechanism.
  • Adding a reasons loss to standard training improves adversarial robustness (e.g., against FGSM attacks) and can improve fairness metrics without harming accuracy.
  • The same method locates sentiment-direction neurons in an instruction-tuned language model, and intervening on them changes generated text tone as predicted.
  • Because the method applies to any subset of the sampled worlds, it offers a general language for interrogating a model's internal concepts beyond class labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ranking of neurons by $D$ should be stable under resampling of the world set $W$; testing this stability would tell whether the method reports a property of the model rather than an artifact of the chosen sample.
  • Because $D$ is defined by exponential tilting, it is closely related to pointwise mutual information between a neuron's activation and the proposition; making that link explicit could let practitioners import information-theoretic tools.
  • The reasons-difference fairness loss is unsupervised once the protected attribute is known, so it could in principle be applied to unlabeled data; the paper does not test this regime.
  • A stricter correctness test would use a synthetic dataset with known causal structure and ask whether the top-reason neurons are precisely the ones that causally drive the label; the PCA clustering test alone cannot distinguish a model that reasons correctly from one that merely correlates with labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an interpretability method ('reasons method') in which each neuron is assigned a vector of activations over a sample of input-label worlds W, and the strength with which this vector supports a proposition A is measured by D(r,A,b,W) as defined in Eq. (2), based on an axiomatic theory of reasons [29]. The paper claims that the method is grounded, uniform, scalable, faithful, correct, trainable, and useful, and it supports these claims with experiments on LeNet/MNIST (reason strengths, activation patching, PCA clustering), MLPs on a Folktables Adult income task (fairness and robustness), and Qwen2.5-0.5B on SST2 sentiment (intervention flips). The central algorithmic idea is clear and the faithfulness interventions are the strongest part of the empirical work, but the correctness evidence is non-discriminative and several quantitative claims lack statistical support.

Significance. If the central claims hold, this is a valuable addition to mechanistic interpretability: a forward-pass-only method that produces per-neuron reason strengths, supports group aggregation, and defines losses for training robustness and fairness. The faithfulness experiments are genuinely informative: MNIST interventions flip predictions at high rates, and the LLM sentiment interventions flip predictions in 97.9% and 98.6% of cases. The definition of D is crisp and the method is demonstrated across multiple architectures and modalities. However, the paper does not release code, the theoretical foundation is an unpublished co-authored manuscript, and the correctness test does not discriminate the reasons method from generic classifier behavior. The seven headline claims in the abstract are therefore not established to the standard the paper aims for, but the core ideas are promising and the gaps appear fixable.

major comments (3)
  1. [§4.1, Eq. (2), Figure 4] The correctness test operationalizes 'internal similarity' as Euclidean proximity of activation rows and 'external similarity' as equality of labels, then takes PCA monochromatic clusters as evidence that the model's reasons structure matches the data source. This test is non-discriminative: a high-accuracy classifier is expected to separate classes in its penultimate-layer activations regardless of the reasons method, and even raw logits would form label clusters because they are trained to do so. The paper provides no baseline comparison (e.g., random features, an untrained network, or another interpretability method) and no cluster-purity quantification. The absence of clusters in the first convolutional layer does not address this; it only shows that that layer does not separate labels. The paper's own Limitations section concedes that no theoretical guarantees for correctness are derived. Since correctness is one of the seven headline claims, this evidence is insufficient as it stands.
  2. [§4.2, Figure 14, Figure 16] The robustness and fairness conclusions are reported without error bars or significance tests. The FGSM comparison (78.6% vs 69.9% and 44.6% vs 27.1% at epsilon=0.15 and 0.25) appears to be a single pair of trained models; with different seeds the differences could lie within run-to-run variance. The fairness experiments are repeated 100 times, but the main text does not present standard deviations or a statistical test for DI, EoO, or RD. Because the abstract claims the method is 'useful' and 'trainable' on the basis of these improvements, the authors should report means over multiple seeds with confidence intervals, or at least show the distribution of the 100 runs.
  3. [§3, Eq. (1)–(2), paragraph 'Theory of reasons'] The axiomatic foundation and the claimed uniqueness theorem are taken from [29], an unpublished manuscript by one of the authors, and the paper works with the arbitrary normalization c=1. The definition D scales linearly with c: replacing a reason vector x by cx gives D(cx,A,b,W)=c D(x,A,b,W), so the reported absolute strengths such as '2.36' are convention-dependent. The paper should either include the axiomatization and uniqueness theorem in an appendix, or explicitly state that the theoretical grounding is conditional on [29] and that absolute strength values are meaningful only up to a global scale.
minor comments (5)
  1. [Eq. (1)] The denominator index appears to run from k=0 while the numerator runs from k=1; since W={w1,...,w2m}, the k=0 term is undefined. Please correct the indexing.
  2. [Appendix A] The heading 'Stastical robustness' should be 'Statistical robustness'.
  3. [§4.3] The phrase 'the 5 neurons peaking most against negativity' should be 'the 5 neurons speaking most against negativity'.
  4. [Abstract and GitHub link] The text says the source code 'will eventually be made available'; the wording should be updated to state whether the code is currently available, and the repository should be populated before publication to support reproducibility.
  5. [§4.2, Eq. (4)] The reasons difference loss would benefit from an explicit statement of the prior b and how the conditional measures b(·|Ap) and b(·|Au) are computed over the batch, so that the loss is fully reproducible.

Circularity Check

1 steps flagged · score 4.0 of 10

Load-bearing uniqueness theorem imported from co-author's unpublished manuscript; empirical interventions otherwise independent.

  1. uniqueness imported from authors [Section 3, 'Theory of reasons' paragraph; Ref. [29] (H. Leitgeb, manuscript under review)]
    "The theory of reasons [ 29] formalizes these philosophical ideas. ... The main mathematical result is that, surprisingly, the models of the overall axiomatic theory of reasons are unique up to a multiplicative constant c >0. Hence we will work here directly with the models given for c=1."

    Equations (1) and (2) are the operational core of the method. The paper does not derive them in this manuscript; it imports them from [29], an unpublished manuscript by co-author Leitgeb. The 'unique up to a multiplicative constant' theorem is used to declare the exponential-update model forced, and the arbitrary gauge c=1 is then baked into all reported strengths. Since [29] is not machine-checked, code-reproduced, or otherwise externally verified here, the claimed 'grounded' status and the uniqueness-based justification reduce to a self-citation chain: the authors' own prior theorem is invoked to make the modeling choice appear inevitable.

full rationale

The paper's empirical demonstrations are mostly self-contained. Faithfulness is tested by activation patching on held-out test images and on a validation sample disjoint from the worlds used for reasons vectors in the LLM experiment; robustness is measured by FGSM attacks on a reasons-trained versus a baseline model; these tests do not fit a parameter and then re-predict that parameter. The main circular element is the importation of the uniqueness theorem from [29], an unpublished manuscript by co-author Leitgeb, which forces the exponential update (Eq. 1) and the strength D (Eq. 2) and fixes the arbitrary gauge c=1. The Limitations section (Section 5) explicitly asks 'Which theoretical guarantees on faithfulness and correctness can one derive under plausible assumptions?', conceding that no such guarantees are given here. The §4.1 correctness operationalization (monochromatic PCA clusters) is additionally weak: any accurate classifier's representations tend to cluster by label, so it does not specifically validate the reasons method; however, that is a validity limitation, not a by-construction circularity. A score of 4 reflects one substantial load-bearing self-citation with substantial independent empirical content elsewhere.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method itself has one free scale constant (c=1) and relies on the axiomatic theory of reasons from an unpublished manuscript by a co-author. The correctness test uses a domain assumption equating Euclidean proximity in activation space with semantic similarity. No new physical or conceptual entities are introduced; the 'reasons vector' is a mathematical object derived from activations.

free parameters (1)
  • multiplicative constant c = 1
    The theory in [29] gives models unique up to c>0. The paper sets c=1 without justification, yet the numerical values of D(r,A,b,W) in Equation 2 depend on this scale. The log-odds strength is not invariant under scaling of the reasons vector.
assumptions (4)
  • domain assumption Axiomatic theory of epistemic reasons with unique models up to multiplicative constant
    Section 3 cites [29] for the primitive notions, axioms, and the result that models are unique up to c>0. This is a manuscript under review by co-author Leitgeb; it is not derived or appended, so the mathematical foundation is an unverified premise.
  • domain assumption A neuron's activation vector over worlds is a reason
    Section 3 defines r_u as the vector of activations for neuron u across the selected worlds and identifies it with an epistemic reason. This identification is a conceptual assumption, not a consequence of the theory.
  • domain assumption Uniform prior over the sampled worlds
    Section 4.1 takes b to be the uniform measure on W when computing strengths. D depends on b, and the paper does not test sensitivity to this choice.
  • ad hoc to paper Correctness is Euclidean clustering of activation rows by label
    Section 4.1 defines internal similarity as Euclidean distance between rows of the activation matrix and external similarity as same label, and then identifies monochromatic PCA clusters as evidence of correctness. This operationalization is introduced ad hoc for the experiment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explaining Neural Networks with Reasons." pith.science (2026). https://pith.science/paper/PRSNTZXX

@misc{pith2026250514424,
  author       = {Pith},
  title        = {Pith review of: Explaining Neural Networks with Reasons},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PRSNTZXX}},
  note         = {Machine review of arXiv:2505.14424}
}
read the original abstract

We propose a new interpretability method for neural networks, which is based on a novel mathematico-philosophical theory of reasons. Our method computes a vector for each neuron, called its reasons vector. We then can compute how strongly this reasons vector speaks for various propositions, e.g., the proposition that the input image depicts digit 2 or that the input prompt has a negative sentiment. This yields an interpretation of neurons, and groups thereof, that combines a logical and a Bayesian perspective, and accounts for polysemanticity (i.e., that a single neuron can figure in multiple concepts). We show, both theoretically and empirically, that this method is: (1) grounded in a philosophically established notion of explanation, (2) uniform, i.e., applies to the common neural network architectures and modalities, (3) scalable, since computing reason vectors only involves forward-passes in the neural network, (4) faithful, i.e., intervening on a neuron based on its reason vector leads to expected changes in model output, (5) correct in that the model's reasons structure matches that of the data source, (6) trainable, i.e., neural networks can be trained to improve their reason strengths, (7) useful, i.e., it delivers on the needs for interpretability by increasing, e.g., robustness and fairness.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 39 canonical work pages

  1. [29]

    H. Leitgeb. The additive logic of epistemic reasons: An axiomatic account, 2025. Manuscript under review

    work page 2025

  2. [1]

    Bau, J.-Y

    D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 117(48):30071–30078, Sept. 2020. ISSN 1091-6490. doi: https: //doi.org/10.1073/pnas.1907375117

    doi:10.1073/pnas.1907375117 2020

  3. [2]

    E. M. Bender and A. Koller. Climbing towards NLU: On meaning, form, and under- 12 standing in the age of data. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, 2020

    work page 2020

  4. [3]

    Bricken, A

    T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary...

    work page 2023

  5. [4]

    D. J. Chalmers. Propositional interpretability in artificial intelligence, 2025. URL https://arxiv.org/abs/2501.15740

    arXiv 2025

  6. [5]

    M. J. Colbrook, V . Antun, and A. C. Hansen. The difficulty of computing sta- ble and accurate neural networks: On the barriers of deep learning and smale’s 18th problem.Proceedings of the National Academy of Sciences, 119(12):e2107151119,

    work page

  7. [6]

    Cunningham, A

    H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. URL https://arxiv. org/abs/2309.08600. Also see [?]

    arXiv 2023

  8. [7]

    F. Ding, M. Hardt, J. Miller, and L. Schmidt. Retiring adult: New datasets for fair machine learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P . Liang, and J. W. Vaughan, editors,Advances in Neural Informa- tion Processing Systems, volume 34, pages 6478–6490. Curran Associates, Inc.,

    work page

Show all 63 references
  1. [8]

    Donnelly and A

    J. Donnelly and A. Roegiest. On interpretability and feature representations: an analysis of the sentiment neuron. InAdvances in Information Retrieval, ECIR 2019, pages 795–802. Springer, 2019. doi: https://doi.org/10.1007/978-3-030-15712-8_55

    2019 doi

  2. [9]

    Doshi-Velez and B

    F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning, 2017. URLhttps://arxiv.org/abs/1702.08608

    2017 arXiv

  3. [10]

    Dutta, D

    S. Dutta, D. Wei, H. Yueksel, P .-Y. Chen, S. Liu, and K. Varshney. Is there a trade-off between fairness and accuracy? A perspective using mismatched hypothesis testing. In H. D. III and A. Singh, editors,Proceedings of the 37th International Conference on Machine Learning, v...

    2020

  4. [11]

    G. K. Dziugaite, S. Ben-David, and D. M. Roy. Enforcing interpretability and its statistical impacts: Trade-offs between accuracy and interpretability, 2020. URL https://arxiv.org/abs/2010.13764

    2020 arXiv

  5. [12]

    Elhage, T

    N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition.Transformer Circuits Thread,

  6. [13]

    Feldman, S

    M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian. Certifying and removing disparate impact. InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’15, page 259–268, New York, NY, USA, 2015. As...

    2015

  7. [14]

    Fiotto-Kaufman, A

    J. Fiotto-Kaufman, A. R. Loftus, E. Todd, J. Brinkmann, C. Juang, K. Pal, C. Rager, A. Mueller, S. Marks, A. S. Sharma, F. Lucchetti, M. Ripa, A. Belfki, N. Prakash, S. Multani, C. Brodley, A. Guha, J. Bell, B. Wallace, and D. Bau. Nnsight and ndif: Democratizing access to fou...

    2024 arXiv

  8. [15]

    URLhttps://transformer-circuits.pub/2022/toy_model/index.html. 13

    2022

  9. [16]

    L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=tcsZt9ZNKD

    2025

  10. [17]

    A. D. Garcez and L. C. Lamb. Neurosymbolic ai: The 3rd wave.Artificial Intelligence Review, 56(11):12387–12406, 2023. doi: https://doi:10.1007/s10462-023-10448-w

    2023 doi

  11. [18]

    Frigg and J

    R. Frigg and J. Nguyen. Scientific Representation. In E. N. Zalta, editor,The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2021 edition, 2021

    2021

  12. [19]

    Geiger, D

    A. Geiger, D. Ibeling, A. Zur, M. Chaudhary, S. Chauhan, J. Huang, A. Arora, Z. Wu, N. Goodman, C. Potts, and T. Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability, 2025. URLhttps://arxiv.org/abs/2301.04709

    2025 arXiv

  13. [20]

    I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. InInternational Conference on Learning Representations (ICLR), 2015

    2015

  14. [21]

    Geiger, H

    A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586, 2021. URL https: //arxiv.org/abs/2106.02997

    2021 arXiv

  15. [22]

    Hardt, E

    M. Hardt, E. Price, E. Price, and N. Srebro. Equality of opportunity in super- vised learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 29. Curran As- sociates, Inc., 2016. URL https://proce...

    2016

  16. [23]

    T. Heap, T. Lawson, L. Farnik, and L. Aitchison. Sparse autoencoders can interpret randomly initialized transformers, 2025. URL https://arxiv.org/abs/2501.17727

    2025

  17. [24]

    J. Harding. Operationalising representation in natural language processing.The British Journal for the Philosophy of Science, forthcoming. URL https://www.journals. uchicago.edu/doi/abs/10.1086/728685?journalCode=bjps

    doi

  18. [25]

    J. F. Horty.Reasons as Defaults. Oxford University Press, 2012

    2012

  19. [26]

    Howard and M

    N. Howard and M. Schroeder.The Fundamentals of Reasons. Oxford University Press, 2024

    2024

  20. [27]

    D. A. Herrmann and B. A. Levinstein. Standards for belief representations in llms. Minds and Machines, 35(5), 2025. doi: https://doi.org/10.1007/s11023-024-09709-6. 14

    2025 doi

  21. [28]

    LeCun, L

    Y. LeCun, L. D. Jackel, L. Bottou, C. Cortes, J. S. Denker, H. Drucker, I. Guyon, U. A. Muller, E. Sackinger, P . Simard, et al. Learning algorithms for classification: A comparison on handwritten digit recognition.Neural networks: the statistical mechanics perspective, 261(27...

    1995

  22. [30]

    Ioffe and C

    S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. Bach and D. Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pa...

    2015

  23. [31]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/ forum?id=Bkg6RiCqY7

    2019

  24. [32]

    Marks and M

    S. Marks and M. Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URL https: //arxiv.org/abs/2310.06824

    2024 arXiv

  25. [33]

    Z. C. Lipton. The mythos of model interpretability.Commun. ACM, 61(10):36–43, sep 2018. ISSN 0001-0782. doi: 10.1145/3233231. URL https://doi.org/10.1145/ 3233231

    2018 doi

  26. [34]

    McInnes, J

    L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2020. URL https://arxiv.org/abs/1802. 03426

    2020

  27. [35]

    K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and edit- ing factual associations in gpt. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors,Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc.,

  28. [36]

    McCulloch and W

    W. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity.Bulletin of Mathematical Biophysics, 5:115–133, 1943. doi: https://doi.org/10. 1017/S0140525X00052791

    1943

  29. [37]

    C. Molnar. Interpretable machine learning, 2025. URL https://christophm.github. io/interpretable-ml-book. 15

    2025

  30. [38]

    W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Definitions, methods, and applications in interpretable machine learning.Proceedings of the National Academy of Sciences of the United States of America, 116(44):22071–22080, 2019. doi: https: //doi.org/10.1073/pna...

    2019 doi

  31. [39]

    URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf

    2022

  32. [40]

    Millière and C

    R. Millière and C. Buckner. A philosophical introduction to language models – part ii: The way forward, 2024. URLhttps://arxiv.org/abs/2405.03207

    2024 arXiv

  33. [41]

    C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits.Distill, 2020. doi: 10.23915/distill.00024.001. URL https://distill.pub/2020/circuits/zoom-in

    2020 doi

  34. [42]

    Radford, R

    A. Radford, R. Jozefowicz, and I. Sutskever. Learning to generate reviews and discovering sentiment, 2017. URLhttps://arxiv.org/abs/1704.01444

    2017 arXiv

  35. [43]

    C. Olah. Interpretability dreams.Transformer Circuits Thread, 2023. URL https: //transformer-circuits.pub/2023/interpretability-dreams/index.html. In- formal note

    2023

  36. [44]

    C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization.Distill, 2017. doi: 10.23915/distill.00007. URLhttps://distill.pub/2017/feature-visualization

    2017 doi

  37. [45]

    Smolensky

    P . Smolensky. On the proper treatment of connectionism.Behavioral and brain sciences, 11(1):1–74, 1988. doi: https://doi.org/10.1017/S0140525X00052791

    1988 doi

  38. [46]

    Socher, A

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Wa...

    2013

  39. [47]

    D. Rai, Y. Zhou, S. Feng, A. Saparov, and Z. Yao. A practical review of mechanistic interpretability for transformer-based language models, 2025. URL https://arxiv. org/abs/2407.02646

    2025

  40. [48]

    Sharkey, B

    L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimersheim, A. Ortega, J. Bloom, S. Biderman, A. Garriga-Alonso, A. Conmy, N. Nanda, J. Rumbelow, M. Wattenberg, N. Schoots, J. Miller, E. J. Michaud, S. Casper, M. Tegmark, W. Saunders, ...

    2025 arXiv

  41. [49]

    J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber. In- vestigating gender bias in language models using causal mediation analysis. In 16 H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors,Advances in Neural Information Processing ...

    2020

  42. [50]

    K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. InNeurIPS ML Safety Workshop, 2022. URLhttps://openreview.net/forum?id=rvi3Wa768B-

    2022

  43. [51]

    Templeton, T

    A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. McDougall, M. MacDiarmid, C. D. Freeman, T. R. Sumers, E. Rees, J. Batson, A. Jermyn, S. Carter, C. Olah, and T. Henighan. Scaling m...

    2024

  44. [52]

    Van der Maaten and G

    L. Van der Maaten and G. Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    2008

  45. [55]

    M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818–833. Springer, 2014

    2014

  46. [56]

    Zhang and N

    F. Zhang and N. Nanda. Towards best practices of activation patching in language models: Metrics and methods. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=Hf17y6u9BC. 17 Figure 6: Top: Computing reasons strengths ...

    2024

  47. [57]

    Random Forest Classifier

  48. [58]

    Each has 10 input neurons (for the 10 features) and 1 output neuron (indicating positive or negative classification) and uses ReLU as activation function

    C-Support Vector Classifier The second group is the following MLPs. Each has 10 input neurons (for the 10 features) and 1 output neuron (indicating positive or negative classification) and uses ReLU as activation function

  49. [59]

    MLP_s (‘small’): One hidden layer of size 100 followed by a second hidden layer of size 50

  50. [60]

    MLP_v (‘vanilla’): Four hidden layers each of size 128

  51. [61]

    We test all combinations of the following hyperparameters:

    MLP_dn (‘dropnorm’): Also four hidden layers each of size 128, but with 20%- dropout and batch norm [27]. We test all combinations of the following hyperparameters:

  52. [62]

    Learning rates: 1e-4, 1e-3, 1e-2

  53. [63]

    We train with binary cross entropy loss (with logits) using AdamW [31]

    Number of epochs: 5, 10, 20. We train with binary cross entropy loss (with logits) using AdamW [31]. The training-test split is 20% test data, and we scale the data using scikit-learn’s StandardScaler. Since the dataset is unbalanced, we report the F1 scores (rather than accur...

  54. [2021]

    URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 32e54441e6382a7fbacbbbaf3c450059-Paper.pdf

    2021

  55. [2022]

    URL https://www.pnas.org/doi/abs/10

    doi: 10.1073/pnas.2107151119. URL https://www.pnas.org/doi/abs/10. 1073/pnas.2107151119

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.