Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Most popular neuron-explanation evaluation metrics fail two simple sanity checks, leaving five reliable metrics.

desk verdict A useful sanity-check framework and meta-evaluation, but the claim that exactly five metrics pass is an artifact of averaging over settings; the paper needs per-setting pass/fail reporting. read the letter →

arxiv 2506.05774 v1 pith:76LJOWTY submitted 2025-06-06 cs.LG

Tuomas Oikarinen , Ge Yan , Tsui-Wei Weng This is my paper
classification cs.LG
keywords mechanisticinterpretabilityneuronexplanationsevaluationmetricssanitychecksconceptimbalanceEvalmeta-evaluationsparseautoencoders
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the field of neuron explanation evaluation has been using unreliable metrics, and that a small set of standard statistical scores are markedly safer. It unifies 18 evaluation metrics used across 19 studies into a single formal framework, NeuronEval, in which any evaluation score is a function of a neuron activation vector and a concept activation vector. It then proposes two sanity checks, the Missing Labels and Extra Labels Tests, which ask whether a metric scores a correct explanation above a randomly thinned or randomly inflated version of that explanation. Across experiments on vision and language models and a theoretical analysis of idealized neurons, the paper finds that most metrics fail at least one check, and identifies Pearson correlation, cosine similarity, AUPRC, IoU, and F1-score as the ones that pass. The broader point is that interpretability claims built on metrics like Recall, AUC, or top-and-random sampling can be illusory, because those metrics cannot distinguish a correct description from one that is too generic or too specific.

What carries the argument

The load-bearing object is the NeuronEval framework, which represents any explanation evaluation as a score $s_M(\mathbf{a}_k, \mathbf{c}_t)$ computed from the neuron activation vector $\mathbf{a}_k$ and the concept activation vector $\mathbf{c}_t$; all 19 surveyed prior evaluations are special cases obtained by choosing a metric $M$, a concept source, a granularity, and a probing dataset. The two sanity checks operate on this representation: the Missing Labels Test randomly zeros out half of the positive concept labels, and the Extra Labels Test randomly doubles the positive labels, and a metric passes a test if its average score decreases by more than $\epsilon=0.001$ on nearly all neurons, measured by the Decrease Acc statistic. The theoretical analysis studies idealized binary neurons whose activations perfectly match a concept, parametrized by activation frequency $\gamma$, and derives closed-form score differences; this is what exposes concept imbalance as the cause of failure.

What would settle it

Run the Missing Labels Test on a balanced dataset where every concept appears in exactly half of the inputs; the theory predicts Recall's score difference stays exactly zero under added extra labels, so if Recall shows a clear score drop there, the closed-form analysis fails. Conversely, if F1-score or AUPRC shows no score change when half the labels are removed on some real neuron, the claim that these metrics are imbalance-robust would be contradicted.

Watch

Extended reading notes

Core claim

The central discovery is that the dominant evaluation metrics for neuron explanations are not measuring what they appear to measure. The paper shows that many metrics, including Recall, AUC, Accuracy, Balanced Accuracy, and Correlation with top-and-random sampling, give nearly identical scores to a perfect explanation and to a deliberately corrupted explanation obtained by randomly adding or removing concept labels, especially when the concept is rare in the probing data. The authors attribute this failure to a common root cause: poor handling of imbalanced concept frequencies. A metric that is insensitive to imbalance collapses toward zero score difference as concepts become sparse, and real neurons, especially sparse autoencoder features, often activate very rarely. The metrics that maintain a stable score difference across all activation frequencies, Pearson correlation, cosine similarity, AUPRC, IoU, and F1-score, are the ones the paper recommends as reliable.

Load-bearing premise

The sanity checks rest on the assumption that if a text concept is a good description of a neuron, then a random subset of the concept's positive labels is necessarily a worse description; this monotonicity is verified only for one semantic relation on one dataset, so a metric could fail the tests for reasons unrelated to explanation quality if that assumption fails.

Editorial extensions

If this is right

  • Published evaluations that rely solely on Recall, AUC, Accuracy, MAD, or top-and-random sampled correlation should be re-examined, because those metrics can award top scores to explanations that are too generic or too specific.
  • Future neuron explanation studies should report at least one of the five passing metrics; in the known-concept comparison, Pearson correlation, cosine similarity, and AUPRC ranked highest at identifying correct explanations.
  • Metrics for sparse autoencoder features and other rarely activated units should be chosen specifically for imbalance robustness; the passing metrics keep their discriminative power at very low activation frequencies, while failing metrics do not.
  • Concept vectors built from generative model prompts are naturally missing labels, so evaluation pipelines using them should combine a generative precision estimate with a recall estimate, as in F1-score, rather than rely on metrics that fail the Missing Labels Test.
  • Because F1-score and IoU are monotonically related, and correlation and mean-centered cosine similarity coincide, the five passing metrics reduce to three distinct behaviors; the choice among them can be made on practical grounds like labeling cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's tests only randomize label sets; an untested extension is whether semantic subsets and supersets beyond ImageNet behave identically, since real explanation errors are semantic, and the single semantic experiment in Appendix E.3 cannot rule out metric-dependent surprises on other hierarchies.
  • The same Missing and Extra Labels logic transfers to any global explanation target with a concept-vector representation, such as attribution maps or graph motifs, though the paper does not claim this transfer.
  • A practical consequence the authors only sketch is that human studies, which overwhelmingly measure Recall for cost reasons, could be made to support F1-score or AUPRC by oversampling highly activating inputs and applying a sampling correction; this remains to be demonstrated in a real user study.
  • Because the theoretical analysis yields closed-form limits at activation frequency $\gamma \to 0$, future metrics could be screened a priori against these closed forms before any model experiments, without needing the full empirical test suite.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces NeuronEval, a unified mathematical framework that expresses many existing neuron-explanation evaluation metrics as functions of a neuron activation vector and a concept activation vector. The authors propose two sanity checks, the Missing Labels Test and the Extra Labels Test, which measure whether a metric assigns lower scores to concepts that are random subsets or supersets of the correct concept. They evaluate 18 metrics across 8 settings in vision and language models, and claim that only Correlation, Cosine Similarity, AUPRC, IoU, and F1-score pass both tests. A second meta-evaluation, meta-AUPRC on neurons with known concepts, is used to rank the metrics. The paper also provides theoretical analysis in Appendices C and D showing that most failures are due to poor handling of imbalanced concept frequencies.

Significance. If the central claims hold, this is a useful contribution to mechanistic interpretability: it gives the field a common notation for comparing explanation evaluations, proposes simple sanity checks that are easy to apply, and provides theoretical explanations for why popular metrics such as Recall, AUC, and Accuracy fail on imbalanced concepts. The paper is unusually systematic, shipping code, per-setting tables, ablations, and closed-form derivations for binary metrics. However, the headline claim that exactly five metrics pass is weakened by per-setting heterogeneity in Appendix F, so the practical guidance is not as clean as the abstract suggests.

major comments (3)
  1. [4.4, Table 3, Appendix F.2] The claim that only Correlation, Cosine, AUPRC, IoU, and F1-score pass the sanity checks is not robust to per-setting examination. Section 4.4 defines passing as Decrease Acc > 90% for all tests, but Table 3 reports values averaged over 8 settings. In Setting 3 (ResNet-18 on Places365, final layer), Table F.2 reports Missing Labels Test Decrease Acc of 56.77% for both IoU and F1-score and 84.44% for AUPRC, all below the stated 90% threshold. Averaging hides these failures. The paper should either adopt a per-setting pass criterion, report per-setting pass/fail in the main text, or explicitly justify why averaging is the right decision rule; as written, the list of 'solid passing metrics' in the abstract and Section 4.4 is overstated.
  2. [4.2, Appendix E.3] The Missing and Extra Labels Tests rest on the assumption that if concept tk is a good description for neuron k, a random subset of it should be a worse description. The only validation of this assumption against real semantic relations is Appendix E.3, which uses WordNet sub/superclasses on one ImageNet final-layer setting. This does not cover hidden-layer neurons, language models, polysemantic units, or compositional concepts. The authors should either add broader semantic-subset experiments or explicitly qualify the scope of the passing claim so that it does not overgeneralize to all neuron types.
  3. [4.2, Eq. (11), Appendix E.4] The pass/fail verdict depends on two thresholds, epsilon = 0.001 and Decrease Acc > 90%, but only epsilon is ablated. The epsilon ablation in Appendix E.4 shows robustness for that parameter, yet the 90% pass threshold is never varied. Since the borderline metrics WPMI and Inverse AUPRC are explicitly described as sensitive to hyperparameters, a small change in the pass threshold could change the reported 'passing' list. The authors should report sensitivity to the 90% threshold or justify its choice with a principled argument.
minor comments (5)
  1. [Abstract and Section 4.4] The abstract states 'only the following pass the tests' while Section 4.4 lists WPMI and Inverse AUPRC as 'borderline' cases; the wording should be consistent about the status of borderline metrics.
  2. [Table 3 footnote] The row for Inverse AUPRC has an extra space in '99.15 %'; please fix the formatting.
  3. [Section 5.1] The text says results are averaged 'across the four setups' but Table 4 and Appendix F describe ten settings; the sentence appears to be a leftover and should be corrected.
  4. [Appendix C.1] The caption of Tables C.1 and C.2 says a metric fails 'on any concept frequency gamma' but the main text says Decrease Acc > 90% for all tests; please align the terminology between the theoretical and experimental pass criteria.
  5. [Appendix D] The phrase 'a theoratical analysis' in the first sentence should be 'a theoretical analysis'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sanity checks are new external criteria, the metrics are evaluated on equal footing, and the passing list follows from explicit formulas and experiments rather than from construction or self-citation.

full rationale

The paper is a meta-evaluation rather than a derivation whose conclusion is built into its inputs. The NeuronEval framework is a notational unification of existing metrics under sM(ak, ct); it does not assume which metrics are reliable. The Missing and Extra Labels Tests are defined independently of the metrics being tested, and the paper evaluates 18 metrics on equal footing, including metrics introduced by the authors' own prior work (WPMI, Correlation), with WPMI failing the tests. The theoretical results in Appendices C and D derive the expected score changes from the metric definitions, so the passing/failing list is a computed consequence, not a fitted parameter renamed as a prediction. The only potentially self-referential choice is that for hidden-layer neurons the 'correct' concept is defined as the concept maximizing IoU (Appendix F), but the missing/extra-labels behavior of IoU/F1 follows from set monotonicity for any ground-truth concept, so this does not make the conclusion circular. The paper also validates the random subset/superset model against semantic WordNet subsets (Appendix E.3), providing external support for the test assumption. Per-setting heterogeneity in Appendix F (e.g., F1/IoU and AUPRC below the 90% threshold in Setting 3) is a robustness concern about averaging, not a circularity in the derivation. Self-citations are descriptive and none is load-bearing for the central claim.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the domain assumption that subset/superset perturbations of a correct concept are worse explanations. There are several hand-set thresholds (alpha, epsilon, 90% pass threshold) and a hidden-layer ground-truth definition that uses IoU itself. No new physical or model entities are postulated.

free parameters (4)
  • Binarization threshold alpha = varies per metric, selected on validation split
    Used for binary metrics (Recall, Precision, F1, IoU, Accuracy, etc.) to turn continuous activations into binary vectors. The paper selects alpha per metric by maximizing meta-AUPRC on a 5% validation split.
  • Epsilon threshold in sanity tests = 0.001
    Used in Decrease Acc definition (Eq. 11) to decide whether a score decrease is non-negligible. Ablations show results are not very sensitive to this choice, but it is a hand-set threshold.
  • Pass threshold for Decrease Acc = 90%
    A metric is said to pass a sanity test if its Decrease Acc exceeds 90%. This is an arbitrary threshold, though results are mostly clear-cut (either near 100% or below 50%).
  • WPMI lambda hyperparameter = not fixed in main text
    WPMI depends on a lambda hyperparameter, and the paper notes its pass/fail status depends on lambda choice. This is a free parameter for that metric.
assumptions (4)
  • domain assumption A random subset of a correct concept is a worse description of the neuron.
    Stated in Section 4.2 as the assumption behind the Missing and Extra Labels Tests. If false, the sanity checks would not measure what they claim.
  • domain assumption The goal of an input-based explanation is to approximate the function x -> f^0:l_k(x).
    Defined in Section 2.3 as the focus of the paper, which restricts the framework to input-based explanations and excludes output-based explanations.
  • ad hoc to paper For hidden layer neurons, the correct concept is the one that maximizes IoU with alpha=0.005.
    Used in Appendix F.1 to define ground truth for layer4 neurons. This choice is tied to one of the metrics being evaluated, which could bias the sanity check results for IoU/F1 on those settings.
  • domain assumption Passing the two sanity checks is a necessary (not sufficient) condition for a reliable evaluation metric.
    This is the paper's stated position in Section 4.2. It is a normative claim about what makes an evaluation metric trustworthy, not an empirical fact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Neuron Explanations: A Unified Framework with Sanity Checks." pith.science (2026). https://pith.science/paper/76LJOWTY

@misc{pith2026250605774,
  author       = {Pith},
  title        = {Pith review of: Evaluating Neuron Explanations: A Unified Framework with Sanity Checks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76LJOWTY}},
  note         = {Machine review of arXiv:2506.05774}
}
read the original abstract

Understanding the function of individual units in a neural network is an important building block for mechanistic interpretability. This is often done by generating a simple text explanation of the behavior of individual neurons or units. For these explanations to be useful, we must understand how reliable and truthful they are. In this work we unify many existing explanation evaluation methods under one mathematical framework. This allows us to compare existing evaluation metrics, understand the evaluation pipeline with increased clarity and apply existing statistical methods on the evaluation. In addition, we propose two simple sanity checks on the evaluation metrics and show that many commonly used metrics fail these tests and do not change their score after massive changes to the concept labels. Based on our experimental and theoretical results, we propose guidelines that future evaluations should follow and identify a set of reliable evaluation metrics.

Figures

Figures reproduced from arXiv: 2506.05774 by the authors.

Figure 1
Figure 1. An overview of contributions. We first unify many existing explanation evaluation methods into a single mathematical framework NeuronEval containing 18 different metrics sM. Next, we perform meta-evaluation via NeuronEval to answer the question: which evaluation metrics reliably measure how good an explanation is? 2. Definitions 2.1. What is an individual unit in a neural network? In this paper we are focused on ind… view at source ↗
Figure 2
Figure 2. Overview of our theoretical Missing and Extra Labels Tests. We can see the Recall metric fails the extra labels test as it cannot differentiate between the perfect explanation and an explanation with extra labels. On the other hand Recall clearly passes the missing labels test as −0.5 ≪ −ϵ = -0.001 For the sanity test, it is not important how much the modified labels decrease the score. We simply want it to decrease… view at source ↗
Figure 3
Figure 3. The effect of Concept/Neuron Activation frequency on the score difference in Theoretical Extra Labels test. We can see that all the metrics that fail the Extra Labels test approach 0 score difference on neurons/concepts that rarely activate, i.e. where the concept labels are imbalanced. The passing metrics maintain a non-zero score difference regardless of how rare the concept is. results align with previous statist… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Explainable AI (XAI): An Overdue Paradigm Shift and Post-XAI Research Directions

    cs.CY 2026-02 unverdicted novelty 4.0 of 10

    Current XAI methods for DNNs and LLMs rest on paradoxes and false assumptions that demand a paradigm shift to verification protocols, scientific foundations, context-aware design, and faithful model analysis rather th...

Reference graph

Works this paper leans on

55 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Sanity checks for saliency maps

    Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018

    work page 2018

  3. [3]

    A., Oikarinen, T., Kulkarni, A., and Weng, T.-W

    Bai, N., Iyer, R. A., Oikarinen, T., Kulkarni, A., and Weng, T.-W. Interpreting neurons in deep vision networks with language models. TMLR, 2025

    work page 2025

  4. [4]

    Network dissection: Quantifying interpretability of deep visual representations

    Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, 2017

    work page 2017

  5. [5]

    Understanding the role of individual units in a deep neural network

    Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., and Torralba, A. Understanding the role of individual units in a deep neural network. PNAS, 2020

    work page 2020

  6. [6]

    Bhatt, U., Weller, A., and Moura, J. M. F. Evaluating and aggregating feature-based model explanations. In Bessiere, C. (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , pp.\ 3016--3022. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi:10.24963/ijcai.2020/417. U...

    work page doi:10.24963/ijcai.2020/417 2020

  7. [7]

    Language models can explain neurons in language models

    Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

    2023

  8. [8]

    An interpretability illusion for bert

    Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Vi \'e gas, F., and Wattenberg, M. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143, 2021

    arXiv 2021

Show all 55 references
  1. [9]

    V., Lombrozo, T., Smith-Renner, A., and Tan, C

    Boyd-Graber, J., Carton, S., Feng, S., Liao, Q. V., Lombrozo, T., Smith-Renner, A., and Tan, C. Human-centered evaluation of explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...

    2022

  2. [10]

    E., Hume, T., Carter, S., Henighan, T., and Olah, C

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Car...

    2023

  3. [11]

    Bykov, K., Kopf, L., Nakajima, S., Kloft, M., and H \"o hne, M. M. Labeling neural representations with inverse recognition. In NeurIPS, 2023

    2023

  4. [12]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models, 2023

    2023

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255, 2009. doi:10.1109/CVPR.2009.5206848

    2009

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

    2021

  7. [15]

    Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis

    Fel, T., Cad \`e ne, R., Chalvidal, M., Cord, M., Vigouroux, D., and Serre, T. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in neural information processing systems, 34: 0 26005--26014, 2021

    2021

  8. [16]

    Craft: Concept recursive activation factorization for explainability

    Fel, T., Picard, A., Bethune, L., Boissin, T., Vigouroux, D., Colin, J., Cad \`e ne, R., and Serre, T. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 2711--2721, 2023

    2023

  9. [17]

    A holistic approach to unifying automatic concept extraction and concept importance estimation

    Fel, T., Boutin, V., B \'e thune, L., Cad \`e ne, R., Moayeri, M., And \'e ol, L., Chalvidal, M., and Serre, T. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  10. [18]

    A., and Steinhardt, J

    Gandelsman, Y., Efros, A. A., and Steinhardt, J. Interpreting the second-order effects of neurons in clip. arXiv preprint arXiv:2406.04341, 2024

    2024 arXiv

  11. [19]

    Y., and Kim, B

    Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. Towards automatic concept-based explanations. Advances in neural information processing systems, 32, 2019

    2019

  12. [20]

    Openwebtext corpus

    Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  13. [21]

    Enhancing automated interpretability with output-centric feature descriptions

    Gur-Arieh, Y., Mayan, R., Agassy, C., Geiger, A., and Geva, M. Enhancing automated interpretability with output-centric feature descriptions. arXiv preprint arXiv:2501.08319, 2025

    2025 arXiv

  14. [22]

    Finding neurons in a haystack: Case studies with sparse probing, 2023

    Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. Finding neurons in a haystack: Case studies with sparse probing, 2023

    2023

  15. [23]

    Natural language descriptions of deep visual features

    Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J. Natural language descriptions of deep visual features. In ICLR, 2022

    2022

  16. [24]

    A benchmark for interpretability methods in deep neural networks

    Hooker, S., Erhan, D., Kindermans, P.-J., and Kim, B. A benchmark for interpretability methods in deep neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, vol...

    2019

  17. [25]

    Rigorously assessing natural language explanations of neurons

    Huang, J., Geiger, A., D’Oosterlinck, K., Wu, Z., and Potts, C. Rigorously assessing natural language explanations of neurons. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp.\ 317--331, 2023

    2023

  18. [26]

    Benchmarking xai explanations with human-aligned evaluations

    Kazmierczak, R., Azzolin, S., Berthier, E., Hedstr \"o m, A., Delhomme, P., Bousquet, N., Frehse, G., Mancini, M., Caramiaux, B., Passerini, A., et al. Benchmarking xai explanations with human-aligned evaluations. arXiv preprint arXiv:2411.02470, 2024

    2024 arXiv

  19. [27]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp.\ 2668--2677. PMLR, 2018

    2018

  20. [28]

    Human-centered evaluation of explainable ai applications: a systematic review

    Kim, J., Maathuis, H., and Sent, D. Human-centered evaluation of explainable ai applications: a systematic review. Frontiers in Artificial Intelligence, 7: 0 1456486, 2024

    2024

  21. [29]

    W., Nguyen, T., Tang, Y

    Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International conference on machine learning, pp.\ 5338--5348. PMLR, 2020

    2020

  22. [30]

    o m, A., Lapuschkin, S., H \

    Kopf, L., Bommer, P. L., Hedstr \"o m, A., Lapuschkin, S., H \"o hne, M. M.-C., and Bykov, K. Cosy: Evaluating textual explanations of neurons. arXiv preprint arXiv:2405.20331, 2024

    2024 arXiv

  23. [31]

    Towards a fuller understanding of neurons with clustered compositional explanations

    La Rosa, B., Gilpin, L., and Capobianco, R. Towards a fuller understanding of neurons with clustered compositional explanations. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    The importance of prompt tuning for automated neuron explanations

    Lee, J., Oikarinen, T., Chatha, A., Chang, K.-C., Chen, Y., and Weng, T.-W. The importance of prompt tuning for automated neuron explanations. In NeurIPS ATTRIB Workshop, 2023

    2023

  25. [33]

    S., Iofinova, E., Frantar, E., and Alistarh, D

    Moakhar, A. S., Iofinova, E., Frantar, E., and Alistarh, D. Spade: Sparsity-guided debugging for deep neural networks. In Forty-first International Conference on Machine Learning, 2024

    2024

  26. [34]

    and Andreas, J

    Mu, J. and Andreas, J. Compositional explanations of neurons. In NeurIPS, 2020

    2020

  27. [35]

    and Weng, T.-W

    Oikarinen, T. and Weng, T.-W. Clip-dissect: Automatic description of neuron representations in deep vision networks. In ICLR, 2023

    2023

  28. [36]

    and Weng, T.-W

    Oikarinen, T. and Weng, T.-W. Linear explanations for individual neurons. In International Conference on Machine Learning, 2024

    2024

  29. [37]

    M., and Weng, T.-W

    Oikarinen, T., Das, S., Nguyen, L. M., and Weng, T.-W. Label-free concept bottleneck models. In International Conference on Learning Representations, 2023

    2023

  30. [38]

    Rise: Randomized input sampling for explanation of black-box models

    Petsiuk, V., Das, A., and Saenko, K. Rise: Randomized input sampling for explanation of black-box models. In BMVC, pp.\ 151. BMVA Press, 2018. URL http://bmvc2018.org/contents/papers/1064.pdf

    2018

  31. [39]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019

    2019

  32. [40]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021

    2021

  33. [41]

    Interpretable machine learning: Fundamental principles and 10 grand challenges

    Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., and Zhong, C. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16, 2022

    2022

  34. [42]

    Find: A function description benchmark for evaluating interpretability methods

    Schwettmann, S., Shaham, T., Materzynska, J., Chowdhury, N., Li, S., Andreas, J., Bau, D., and Torralba, A. Find: A function description benchmark for evaluating interpretability methods. Advances in Neural Information Processing Systems, 36: 0 75688--75715, 2023

    2023

  35. [43]

    R., Schwettmann, S., Wang, F., Rajaram, A., Hernandez, E., Andreas, J., and Torralba, A

    Shaham, T. R., Schwettmann, S., Wang, F., Rajaram, A., Hernandez, E., Andreas, J., and Torralba, A. A multimodal automated interpretability agent. In Forty-first International Conference on Machine Learning, 2024

    2024

  36. [44]

    R., Antonello, R., Jain, S., Huth, A

    Singh, C., Hsu, A. R., Antonello, R., Jain, S., Huth, A. G., Yu, B., and Gao, J. Explaining black box text modules in natural language with language models, 2023

    2023

  37. [45]

    A., Oikarinen, T., Srivastava, D., Weng, W.-H., and Weng, T.-W

    Srinivas, A. A., Oikarinen, T., Srivastava, D., Weng, W.-H., and Weng, T.-W. Sand: Enhancing open-set neuron descriptions through spatial awareness. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.\ 2993--3002. IEEE, 2025

    2025

  38. [46]

    Vlg-cbm: Training concept bottleneck models with vision-language guidance

    Srivastava, D., Yan, G., and Weng, T.-W. Vlg-cbm: Training concept bottleneck models with vision-language guidance. NeurIPS, 2024

    2024

  39. [47]

    Subramani, N., Suresh, N., and Peters, M. E. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pp.\ 566--581, 2022

    2022

  40. [48]

    L., McDougall, C., MacDiarmid, M., Freeman, C

    Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and H...

    2024

  41. [49]

    The caltech-ucsd birds-200-2011 dataset

    Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011

    2011

  42. [50]

    Post-hoc concept bottleneck models

    Yuksekgonul, M., Wang, M., and Zou, J. Post-hoc concept bottleneck models. International Conference on Learning Representations, 2023

    2023

  43. [51]

    Sigmoid loss for language image pre-training

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In ICCV, 2023

    2023

  44. [52]

    A., and Rubinstein, B

    Zhang, R., Madumal, P., Miller, T., Ehinger, K. A., and Rubinstein, B. I. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 11682--11690, 2021

    2021

  45. [53]

    Object detectors emerge in deep scene cnns

    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Object detectors emerge in deep scene cnns. In ICLR, 2015

    2015

  46. [54]

    Places: A 10 million image database for scene recognition

    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017

    2017

  47. [55]

    S., Klein, T., and Brendel, W

    Zimmermann, R. S., Klein, T., and Brendel, W. Scale alone does not improve mechanistic interpretability in vision models. In NeurIPS, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.