Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Mechanistic understanding and validation of large AI models with SemanticLens

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SemanticLens claims that every neuron of a vision model can be mapped into the shared embedding space of a multimodal foundation model such as CLIP, making a model's internal knowledge searchable by text and auditable against…

desk verdict Useful integration of CLIP-based neuron auditing, but the core semantic-proxy assumption is not directly validated and the 'universal' claim overreaches. read the letter →

arxiv 2501.05398 v1 pith:CP4KLXKB submitted 2025-01-09 cs.LG cs.AI

Maximilian Dreyer , Jim Berend , Tobias Labarta , Johanna Vielhaben , Thomas Wiegand , Sebastian Lapuschkin , Wojciech Samek This is my paper · ORCID
classification cs.LGcs.AI
keywords ExplainableAImechanisticinterpretabilityconceptlabellingsemanticembeddingCLIPauditingspuriouscorrelationsmeasures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SemanticLens sets out to make the inner components of a neural network as inspectable as the parts of a human-engineered machine. The method maps each neuron to a vector in the shared embedding space of a multimodal foundation model such as CLIP, by averaging the embeddings of the image patches that most strongly activate that neuron. Once components live in this semantic space, they can be searched with text prompts, labelled automatically, compared across models and layers, audited against user-defined valid and spurious concepts, and scored for clarity, redundancy, and polysemanticity. The paper demonstrates these operations on ImageNet classifiers and on a melanoma-detection model, where audits reveal that spurious concepts such as watermarks, palm trees, rulers, and band-aids are used in predictions, and that retraining with artefact augmentation removes their influence better than pruning does. If the approach works as claimed, it provides a scalable, human-free route from opaque model weights to component-level verification.

What carries the argument

The central object is the semantic component embedding $\vartheta_k = \frac{1}{|E_k|}\sum_{x\in E_k} F(x)$, where $E_k$ are the concept examples of neuron $k$ (the top-$m$ activating patches, cropped to the relevant region via CRP) and $F$ is a multimodal foundation model such as CLIP. All operations---search, labelling, comparison, audit, and interpretability scoring---are cosine similarities in this space, with a 'null' embedding $\vartheta_{\langle\rangle}$ subtracted to remove template and background effects. The interpretability measures are derived directly from the same embeddings: clarity is average pairwise similarity within a neuron's example set, redundancy is average maximal similarity to other neurons, and polysemanticity is one minus the clarity of clustered subsets. The link to decision-making is provided by CRP relevance scores $R$, which mark which components actually contributed to a prediction and allow attribution graphs and audit weighting.

What would settle it

A direct test is to take a neuron labelled by SemanticLens for a concept such as 'palm tree' and measure the neuron's activation on images that contain the concept versus closely matched images that do not, using the same activation-response metric the paper's supplement applies to synthetic labels. If a substantial fraction of labels assigned by the embedding search score no better than random labels on this held-out test, then the cosine-similarity proxy in the foundation model is not preserving what the neuron actually encodes. The same test should be run with a deliberately domain-shifted foundation model: if audit conclusions change while the audited model itself is unchanged, the semantic proxy, not the model's knowledge, is determining the verdict.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a model's hidden knowledge can be faithfully represented by a small set of semantic vectors: for each component, take the $m$ most activating image patches from the dataset, crop them to the relevant regions with Concept Relevance Propagation (CRP), embed each patch in a foundation model's space, and average (Eq. (2)). The resulting vector $\vartheta_k$ is then compared to text or image probes by cosine similarity with a null-embedding subtraction (Eqs. (3)-(4)). This representation supports the paper's claimed capabilities: search for concepts, automated labelling, model comparison via set similarity, and concept-alignment audits that split components into valid, spurious, and unexpected. On ImageNet, the audits expose neurons tied to Indian person, palm tree, and watermark for the class 'Ox' and find that no class among 26 examined is fully aligned with valid concepts; on the ISIC melanoma benchmark, they find spurious reliance on red skin, band-aid, and ruler concepts and show that retraining on augmented data, not pruning, materially reduces artefact sensitivity. The paper additionally introduces interpretability measures---clarity, redundancy, polysemanticity---that correlate above 0.74 with human judgments in user studies.

Load-bearing premise

Everything SemanticLens reports about a model's concepts rests on one premise: that cosine similarity in the chosen foundation model's embedding space is a faithful proxy for semantic similarity of the neuron's concept, and that averaging the top-activating cropped patches preserves that concept rather than averaging it away.

Editorial extensions

If this is right

  • Searching for a bias, artefact, or required concept in a trained vision model reduces to a text query over neuron embeddings, so audits that previously required manual inspection can be automated.
  • Models of different architectures, depths, and training recipes can be compared quantitatively on the same semantic space, revealing which concepts are shared, unique, or transformed by longer training.
  • Because each neuron embedding is linked to its activating training patches, detected spurious concepts can be traced to the specific data responsible, enabling targeted data cleaning.
  • The interpretability measures give a cheap, human-free proxy for how understandable a model's components are, so architecture and regularization choices such as dropout or sparsity can be optimized for interpretability at scale.
  • In the medical case, the audit framework can check whether a model follows domain rules such as the ABCDE rule and can identify which spurious cues, if removed by retraining rather than pruning, genuinely change artefact sensitivity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the semantic-proxy assumption transfers, the same embedding recipe should apply to audio or video models by swapping the foundation model; the paper demonstrates only vision but its formal steps are modality-agnostic.
  • The audit conclusions inherit the foundation model's own concept organization: in domains where generic CLIP is not a trustworthy semantic expert, the method needs a domain-specific foundation model, exactly as the paper uses a dermatology-specific CLIP for ISIC.
  • A stronger validation than the supplied synthetic-label faithfulness test would compare SemanticLens labels against human concept annotations on real images, which the paper does not provide; that comparison would reveal whether the averaged-embedding proxy or the foundation model is the limiting factor.
  • The pruning-versus-retraining result suggests that localizing spurious components is not by itself a repair: a testable extension is to use the neuron-level search to generate targeted training augmentations for other failure modes, not just the artefacts studied here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SemanticLens, a framework that represents each neuron (or other component) of a vision model by the mean embedding, in a foundation model such as CLIP, of its top-activating, CRP-cropped image patches. This semantic representation is then used for text-based neuron search, automatic labelling, model comparison, concept-alignment audits, and a set of human-interpretability measures (clarity, similarity, redundancy, polysemanticity). The method is evaluated on ImageNet classifiers (ResNet, VGG, ViT), on a binary melanoma classifier trained on ISIC 2019, and in a 218-participant user study, with code and a public demo provided.

Significance. If the central semantic-proxy assumption holds, SemanticLens is a practically valuable integration: it turns component-level interpretability into a searchable vector database and provides quantitative interpretability measures that correlate with human judgments at >0.74. The paper has real strengths: a label-faithfulness benchmark against INVERT and CLIP-Dissect, hyperparameter ablations, public code and demo, and an attempt to connect component semantics to both predictions and training data. The main risk is that every downstream capability inherits an unvalidated geometric assumption about the foundation model's embedding space, and the universality claim is supported only by vision experiments.

major comments (4)
  1. [§3.2, Eq. (2); §3.3, Eq. (3); Suppl. D.4, Eq. (1)] The core assumption that ϑ_k = (1/|E_k|) Σ F(x) over top-activating, cropped patches faithfully represents the neuron's concept is not tested in the direction the method uses it. The label-faithfulness benchmark in Supplementary Note D.4 computes ϕ(i,k) from the neuron's response to synthetic images of the candidate label, i.e., it tests label-to-neuron activation, not whether the mean embedding ϑ_k is semantically close to the label embedding in the foundation model's cosine geometry. All downstream capabilities—search (Eq. (4)), labelling, audit scores (Section 3.4), and interpretability measures (Eqs. (6)–(10))—are built on that cosine geometry. Please add a direct validation, for example measuring whether the rank order of cosine similarities between ϑ_k and a set of candidate label embeddings agrees with human judgments of the neuron's concept, or testing whether images whose embeddings are nearest to ϑ_k activate the neuron more than randomly selected images.
  2. [§3.1; Supplementary Note H] For Vision Transformers, concept examples are obtained by approximating CRP attributions with up-sampled spatial maps, an approximation the paper itself acknowledges in §3.1. Since the ViT interpretability conclusions in Fig. 5b and Section 4.4 are based on these concept examples, the ViT part of the universality claim rests on an unvalidated attribution step. Please report a quantitative comparison of the approximate spatial maps against a ground-truth or established attribution method (e.g., CRP once available, or attention-based attributions) and show that the resulting concept examples do not change the conclusions.
  3. [§4.3; Supplementary Note F.2] The medical audit uses WhyLesionCLIP because generic CLIP is not trusted for dermatology, but the paper provides no evidence that WhyLesionCLIP's cosine geometry ranks dermoscopic concepts correctly. The ABCDE audit results (Fig. 4) and the subsequent pruning/retraining evaluation therefore depend on an unstated assumption about the medical foundation model. Please provide a domain-specific validation, such as zero-shot classification or retrieval accuracy on dermoscopic images, or a rating study with dermatologists comparing the top label embeddings for a set of lesions.
  4. [Abstract; §1; §2] The 'universal' claim is broader than the evidence presented. The experiments cover CNN and ViT image classifiers and one binary skin-lesion model; there are no NLP, audio, or other modality results, and no sparse autoencoder or factorized-activation results, despite Section 2 stating that SemanticLens is also applicable to those component types. Either add demonstrations in other settings or restrict the abstract and central claims to vision components.
minor comments (6)
  1. [§3.1, Eq. (1)] The symbol CRP is used both for the attribution framework and for the cropping operation; please define the cropping function explicitly, e.g., via a threshold on the normalized attribution map.
  2. [§3.2; Suppl. Fig. D.6] The claim that |E_k|=30 yields converged ϑ_k is supported mainly for ResNet architectures; Supplementary Fig. D.6 shows a slight decline at 50 samples for ResNet and a monotone decline for VGG. Please state the convergence criterion and comment on the VGG behaviour.
  3. [§3.5.3, Eq. (10)] The polysemanticity measure uses h=2 and 'an off-the-shelf clustering method'; please specify the clustering algorithm and report sensitivity to the number of clusters h.
  4. [§3.3, Eq. (5); Suppl. Note E, Eq. (1)] The same set-similarity metric is defined twice; keep one canonical definition to avoid version skew.
  5. [Supplementary Note B.2.2] The text states that ISIC 2019 includes 'nine different diagnostic categories' but then lists eight; please correct the count.
  6. [Throughout] The paper alternates between 'SEMANTIC LENS' and 'SemanticLens' (title, abstract, demo URL); please unify the notation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SemanticLens builds on external tools (CLIP, CRP, Stable Diffusion) and validates labels and interpretability measures against independent behavioral and human benchmarks.

full rationale

SemanticLens is a compositional use of pre-existing, externally published components rather than a derivation that reduces to its own inputs. The central object is the semantic embedding ϑ_k = (1/|E_k|) Σ F(x) (Eq. 2), defined as an average of foundation-model embeddings of top-activating, CRP-cropped patches. Search, labelling, comparison, and audit then operate by cosine similarity over these ϑ vectors. This is a construction, not a predicted quantity, and no equation in the paper forces the reported conclusions purely by definition. The label-faithfulness evaluation (Supplementary Note D.4, Eq. 1 of D.4.1) provides an independent behavioral check: a label is scored by whether synthetic images generated from the label text (via Stable Diffusion) activate the neuron. This tests the neuron response, not a similarity recomputed from the same embeddings, so the label pipeline is externally anchored. The human-interpretability measures are likewise validated against Amazon Mechanical Turk judgments (Section 4.4.1), giving correlations above 0.74, an external benchmark. The paper's reliance on the authors' prior CRP framework is load-bearing for cropping and relevance attribution, but CRP is a published method with available code (Zennit-CRP), so it counts as independent support under the review rules rather than circular self-citation. The acknowledged ViT approximation of CRP via upsampled spatial maps (Section 3.1) is a stated limitation of tool availability, not a circular step. The substantive risk in the paper is that the semantic-proxy assumption—that cosine similarity in a chosen foundation model's space faithfully ranks neuron concepts, especially for the domain-specific WhyLesionCLIP medical case—is not fully validated. That is a correctness and generalization concern, not a circularity: the paper's claims do not become true by construction because of this assumption, and several of its outputs (spurious correlation detection, interpretability ratings) are checked against independent data or human judgment. No specific reduction of a prediction to a fitted input or to a self-citation chain could be exhibited, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method introduces no physical entities and fits no regression constants, but it depends on hand-chosen hyperparameters and several domain assumptions about foundation-model semantics, activation maximization, and CRP relevance.

free parameters (4)
  • concept_example_count_m = 30 for ImageNet, 20 for ISIC
    Number of top-activating samples averaged to form each neuron embedding; chosen by hand and argued to converge in Supplementary Note D.
  • alignment_threshold = 0.025 for default labelling, 0.01 for attribution graph
    Minimum cosine alignment above the null embedding required to assign a label; chosen by hand and varies across experiments.
  • relevance_filter_threshold = 2.8 percent in the Ox audit, 1 percent or 5 divided by neuron count in attribution graphs
    Filters out components considered irrelevant before auditing; chosen by hand and varies by experiment.
  • polysemanticity_cluster_count_h = 2 throughout
    Number of clusters used to split concept examples when measuring polysemanticity; fixed by hand.
assumptions (5)
  • domain assumption CLIP-style foundation model embeddings provide a semantically structured space where cosine similarity reflects human semantic similarity.
    Used in Eqs. (2) and (3) and in all search and labelling operations; not proven and depends on the foundation model matching the target domain.
  • domain assumption Highly activating image patches, after CRP cropping, represent the concept encoded by a neuron.
    Section 3.1, Eq. (1); this is the standard feature visualization assumption adopted by the method.
  • domain assumption CRP relevance scores faithfully quantify how much a component contributes to a prediction.
    Section 3.4 and Fig. 2c; adopted from prior work [15] without re-validation.
  • domain assumption Averaging the foundation model embeddings of top concept examples yields a stable semantic vector for a component.
    Section 3.2, Eq. (2); convergence is argued in Supplementary Note D but not formally proven.
  • ad hoc to paper For Vision Transformers, upsampled spatial maps approximate CRP attributions.
    Section 3.1 states that CRP is not available for ViTs and that attributions are approximated by up-sampled spatial maps; no validation of this approximation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mechanistic understanding and validation of large AI models with SemanticLens." pith.science (2026). https://pith.science/paper/CP4KLXKB

@misc{pith2026250105398,
  author       = {Pith},
  title        = {Pith review of: Mechanistic understanding and validation of large AI models with SemanticLens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CP4KLXKB}},
  note         = {Machine review of arXiv:2501.05398}
}
read the original abstract

Unlike human-engineered systems such as aeroplanes, where each component's role and dependencies are well understood, the inner workings of AI models remain largely opaque, hindering verifiability and undermining trust. This paper introduces SemanticLens, a universal explanation method for neural networks that maps hidden knowledge encoded by components (e.g., individual neurons) into the semantically structured, multimodal space of a foundation model such as CLIP. In this space, unique operations become possible, including (i) textual search to identify neurons encoding specific concepts, (ii) systematic analysis and comparison of model representations, (iii) automated labelling of neurons and explanation of their functional roles, and (iv) audits to validate decision-making against requirements. Fully scalable and operating without human input, SemanticLens is shown to be effective for debugging and validation, summarizing model knowledge, aligning reasoning with expectations (e.g., adherence to the ABCDE-rule in melanoma classification), and detecting components tied to spurious correlations and their associated training data. By enabling component-level understanding and validation, the proposed approach helps bridge the "trust gap" between AI models and traditional engineered systems. We provide code for SemanticLens on https://github.com/jim-berend/semanticlens and a demo on https://semanticlens.hhi-research-insights.eu.

Figures

Figures reproduced from arXiv: 2501.05398 by the authors.

Figure 1
Figure 1. Embedding the model components in an understandable semantic space allows to systematically and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SEMANTICLENS allows to systematically understand the internal knowledge and inference of neural networks. a) Via search engine-like queries, one can probe for knowledge referring to, e.g., (racial) biases, data artefacts, or specific knowledge of interest. b) A low-dimensional UMAP projection of the semantic embeddings provides a structured overview of the model’s knowledge, where each point corresponds to the encod… view at source ↗
Figure 3
Figure 3. Using SEMANTICLENS to audit models and check if their reasoning aligns with human expectation. a) 1 In a first step, a set of valid and spurious concepts is defined via text descriptions, e.g., curved horns or palm tree for “Ox” detection, respectively. 2 Afterwards, we check which model components encode for either spurious or valid concepts, both or neither. The size of each dot in the chart represents the importa… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Using SEMANTICLENS to find and correct bugs in medical models that detect melanoma skin cancer. a) The ABCDE-rule is a popular guide for visual melanoma clues. We expect models to learn several concepts corresponding to the ABCDE-rule, as well as other melanoma-unrelat…
Figure 5
Figure 5. Figure 5: We introduce computable human-interpretability measures that are useful to rate and improve model [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence

    cs.AI 2026-08 conditional novelty 7.0 of 10

    An agentic system called Mechanist autonomously discovers mechanisms of AI behavior, including hidden safety risks and separable internal 'belief heads', and uses them to steer model outputs.

  2. Relevance-driven Input Dropout: an Explanation-guided Regularization Technique

    cs.LG 2025-05 conditional novelty 6.0 of 10

    RelDrop, which occludes the most attribution-relevant input regions during training, improves generalization and occlusion robustness for image and point cloud classification.

  3. From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliance

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A new attribution framework combines sparse autoencoder components with gradient-based attribution to reveal how CLIP models rely on semantically unexpected concepts.

  4. Ensuring Medical AI Safety: Interpretability-Driven Detection and Mitigation of Spurious Model Behavior and Associated Data

    cs.AI 2025-01 conditional novelty 5.0 of 10

    Using concept vectors as bias representations, the framework semi-automatically labels and localizes spurious artifacts and partially unlearns them, though success varies by architecture and artifact type.

Reference graph

Works this paper leans on

122 extracted references · 64 canonical work pages · cited by 4 Pith papers

  1. [1]

    Unmasking clever hans predictors and assessing what machines really learn

    Sebastian Lapuschkin, Stephan W¨aldchen, Alexander Binder, Gr´egoire Montavon, Wojciech Samek, and Klaus-Robert M¨uller. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10:1096, 2019

    2019

  2. [2]

    The clever hans effect in unsupervised learning

    Jacob Kauffmann, Jonas Dippel, Lukas Ruff, Wojciech Samek, Klaus-Robert M ¨uller, and Gr´egoire Montavon. The clever hans effect in unsupervised learning. Nature Machine Intelligence, 2025

    2025

  3. [3]

    Explainable ai in medical imaging: An overview for clinical practitioners– beyond saliency-based xai approaches

    Katarzyna Borys, Yasmin Alyssa Schmitt, Meike Nauta, Christin Seifert, Nicole Kr¨amer, Christoph M Friedrich, and Felix Nensa. Explainable ai in medical imaging: An overview for clinical practitioners– beyond saliency-based xai approaches. European Journal of Radiology, 162:110786, 2023

    2023

  4. [4]

    Provably safe systems: the only path to controllable agi

    Max Tegmark and Steve Omohundro. Provably safe systems: the only path to controllable agi. arXiv preprint arXiv:2309.01933, 2023

    arXiv 2023

  5. [5]

    The Measure of All Minds: Evaluating Natural and Artificial Intelligence

    Jos´e Hern´andez-Orallo. The Measure of All Minds: Evaluating Natural and Artificial Intelligence . Cambridge University Press, Cambridge, UK, 2017. ISBN 9781316594179

    2017

  6. [6]

    Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 of LNCS

    Wojciech Samek, Gr´egoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert M¨uller, editors. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 of LNCS. Springer, Cham, Switzerland, 2019

    2019

  7. [7]

    Xai—explainable artificial intelligence

    David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. Xai—explainable artificial intelligence. Science Robotics, 4(37):eaay7120, 2019

    2019

  8. [8]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023

    2023

Show all 122 references
  1. [9]

    Explanation in artificial intelligence: Insights from the social sciences.Artificial Intelligence, 267:1–38, 2019

    Tim Miller. Explanation in artificial intelligence: Insights from the social sciences.Artificial Intelligence, 267:1–38, 2019

    2019

  2. [10]

    Mechanistic interpretability for AI safety - a review

    Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety - a review. Transactions on Machine Learning Research, 2024. ISSN 2835-8856

    2024

  3. [11]

    Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability

    Vikram V Ramaswamy, Sunnie SY Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10932–1...

    2023

  4. [12]

    A typology for exploring the mitigation of shortcut behaviour

    Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. A typology for exploring the mitigation of shortcut behaviour. Nature Machine Intelligence, 5(3):319–330, 2023

    2023

  5. [13]

    Understanding neural networks via feature visualization: A survey

    Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualization: A survey. In Explainable AI: interpreting, explaining and visualizing deep learning, volume 11700 of LNCS, pages 55–76. Springer, Cham, Switzerland, 2019

    2019

  6. [14]

    Network dissection: Quantifying interpretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6541–6549, 2017

    2017

  7. [15]

    From attribution maps to human-understandable explanations through concept relevance propagation

    Reduan Achtibat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojciech Samek, and Sebastian Lapuschkin. From attribution maps to human-understandable explanations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006–1019, 2023

    2023

  8. [16]

    Craft: Concept recursive activation factorization for explainability

    Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Remi Cadenc, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2023

  9. [17]

    A holistic approach to unifying automatic concept extraction and concept importance estimation

    Thomas Fel, Victor Boutin, Louis B ´ethune, R ´emi Cad `ene, Mazda Moayeri, L ´eo And ´eol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. In Advances in Neural Information Processing Systems...

    2024

  10. [18]

    Www: A unified framework for explaining what where and why of neural networks by interpretation of neuron concepts

    Yong Hyun Ahn, Hyeon Bae Kim, and Seong Tae Kim. Www: A unified framework for explaining what where and why of neural networks by interpretation of neuron concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10968–10977, 2024

    2024

  11. [19]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR, 2017. 19 Mechanistic understanding and validation of large AI models

    2017

  12. [20]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668–2677. PMLR, 2018

    2018

  13. [21]

    Making it possible for the auditing of ai: A systematic review of ai audits and ai auditability

    Yueqi Li and Sanjay Goel. Making it possible for the auditing of ai: A systematic review of ai audits and ai auditability. Information Systems Frontiers, pages 1–31, 2024

    2024

  14. [22]

    Foundational challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024

    2024 arXiv

  15. [23]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  16. [24]

    Evaluating explanations through llms: Beyond traditional user studies

    Francesco Bombassei De Bona, Gabriele Dominici, Tim Miller, Marc Langheinrich, and Martin Gjoreski. Evaluating explanations through llms: Beyond traditional user studies. arXiv preprint arXiv:2410.17781, 2024

    2024 arXiv

  17. [25]

    Object detectors emerge in deep scene CNNs

    Bolei Zhou, Aditya Khosla, `Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene CNNs. In 3rd International Conference on Learning Representations (ICLR), 2015

    2015

  18. [26]

    Feature visualization

    Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017

    2017

  19. [27]

    Learning to generate reviews and discovering sentiment

    Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444, 2017

    2017 arXiv

  20. [28]

    On completeness-aware concept-based explanations in deep neural networks

    Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. In Advances in neural information processing systems, volume 33, pages 20554–20565, 2020

    2020

  21. [29]

    Understanding deep image representations by inverting them

    Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5188–5196, 2015

    2015

  22. [30]

    Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

    Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, pages 3387–3395, 2016

    2016

  23. [31]

    Dreaming to distill: Data-free knowledge transfer via deepinversion

    Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8715–8724, 2020

    2020

  24. [32]

    Unlocking feature visualization for deep network with magnitude constrained optimization

    Thomas FEL, Thibaut Boissin, Victor Boutin, Agustin PICARD, Paul Novello, Julien Colin, Drew Linsley, Tom ROUSSEAU, Remi Cadene, Lore Goetschalckx, et al. Unlocking feature visualization for deep network with magnitude constrained optimization. In Advances in Neural Informatio...

    2024

  25. [33]

    Dig-in: Diffusion guidance for investigating networks-uncovering classifier differences neuron visualisations and visual counterfactual explanations

    Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. Dig-in: Diffusion guidance for investigating networks-uncovering classifier differences neuron visualisations and visual counterfactual explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  26. [34]

    Labeling neural representations with inverse recognition

    Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina H ¨ohne. Labeling neural representations with inverse recognition. In Advances in Neural Information Processing Systems , volume 37, 2024

    2024

  27. [35]

    Clip-dissect: Automatic description of neuron representations in deep vision networks

    Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. In International Conference on Learning Representations, 2022

    2022

  28. [36]

    Natural language descriptions of deep visual features

    Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. InInternational Conference on Learning Representations, 2021

    2021

  29. [37]

    Describe-and-dissect: Interpreting neurons in vision networks with language models

    Nicholas Bai, Rahul A Iyer, Tuomas Oikarinen, and Tsui-Wei Weng. Describe-and-dissect: Interpreting neurons in vision networks with language models. arXiv preprint arXiv:2403.13771, 2024. 20 Mechanistic understanding and validation of large AI models

    2024 arXiv

  30. [38]

    Identifying interpretable subspaces in image representations

    Neha Kalibhat, Shweta Bhardwaj, Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces in image representations. In International Conference on Machine Learning, volume 202, pages 15623–15638, 2023

    2023

  31. [39]

    Understanding the (extra-)ordinary: Validating deep model decisions with prototypical concept-based explanations

    Maximilian Dreyer, Reduan Achtibat, Wojciech Samek, and Sebastian Lapuschkin. Understanding the (extra-)ordinary: Validating deep model decisions with prototypical concept-based explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2024

  32. [40]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519–3529. PMLR, 2019

    2019

  33. [41]

    Teaching matters: In- vestigating the role of supervision in vision transformers

    Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava. Teaching matters: In- vestigating the role of supervision in vision transformers. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , pages 7486–7496, 2022. URL https: //api.s...

    2022

  34. [42]

    Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems, volume 34, pages 12116–12128, 2021

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems, volume 34, pages 12116–12128, 2021

    2021

  35. [43]

    What do self-supervised vision transformers learn? In International Conference on Learning Representations , 2023

    Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self-supervised vision transformers learn? In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=azCKuYyS74

    2023

  36. [44]

    Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks

    Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8730–8738, 2018

    2018

  37. [45]

    Beyond scalars: Concept-based alignment analysis in vision transformers

    Johanna Vielhaben, Dilyara Bareeva, Jim Berend, Wojciech Samek, and Nils Strodthoff. Beyond scalars: Concept-based alignment analysis in vision transformers. arXiv preprint arXiv:2412.06639, 2024

    2024 arXiv

  38. [46]

    DORA: Exploring outlier representations in deep neural networks

    Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC H¨ohne. DORA: Exploring outlier representations in deep neural networks. In ICLR 2023 Workshop on Pitfalls of Limited Data and Computation for Trustworthy ML, 2023. URL https://openreview.net/forum? ...

    2023

  39. [47]

    Can: Concept-aligned neurons for visual comparison of deep neural network models

    M Li, S Jeong, S Liu, and M Berger. Can: Concept-aligned neurons for visual comparison of deep neural network models. In Computer Graphics Forum, page e15085. Wiley Online Library, 2024

    2024

  40. [48]

    Concept evolution in deep learning training: A unified interpretation framework and discoveries

    Haekyu Park, Seongmin Lee, Benjamin Hoover, Austin P Wright, Omar Shaikh, Rahul Duggal, Nilaksh Das, Kevin Li, Judy Hoffman, and Duen Horng Chau. Concept evolution in deep learning training: A unified interpretation framework and discoveries. In Proceedings of the 32nd ACM Int...

    2023

  41. [49]

    Uncov- ering unique concept vectors through latent space decomposition

    Mara Graziani, Laura O’Mahony, An phi Nguyen, Henning M¨uller, and Vincent Andrearczyk. Uncov- ering unique concept vectors through latent space decomposition. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=LT4DXqUJTD

    2023

  42. [50]

    Multi-dimensional concept discovery (mcd): A unifying framework with completeness guarantees

    Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (mcd): A unifying framework with completeness guarantees. Transactions on Machine Learning Research, 2023

    2023

  43. [51]

    Disentangling neuron representations with concept vectors

    Laura O’Mahony, Vincent Andrearczyk, Henning M¨uller, and Mara Graziani. Disentangling neuron representations with concept vectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3769–3774, 2023

    2023

  44. [52]

    Local vs distributed representations: What is the right basis for interpretability? arXiv preprint arXiv:2411.03993, 2024

    Julien Colin, Lore Goetschalckx, Thomas Fel, Victor Boutin, Jay Gopal, Thomas Serre, and Nuria Oliver. Local vs distributed representations: What is the right basis for interpretability? arXiv preprint arXiv:2411.03993, 2024

    2024

  45. [53]

    Sparse autoencoders find highly interpretable features in language models

    Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, 2023

    2023

  46. [54]

    Pure: Turning polysemantic neurons into pure features by identifying relevant circuits

    Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojciech Samek, and Sebastian Lapuschkin. Pure: Turning polysemantic neurons into pure features by identifying relevant circuits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Worksho...

    2024

  47. [55]

    Tackling polysemanticity with neuron embeddings

    Alex Foote. Tackling polysemanticity with neuron embeddings. InICML 2024 Workshop on Mechanistic Interpretability, 2024

    2024

  48. [56]

    Evaluating readability and faithfulness of concept-based explanations

    Meng Li, Haoran Jin, Ruixuan Huang, Zhihao Xu, Defu Lian, Zijia Lin, Di Zhang, and Xiting Wang. Evaluating readability and faithfulness of concept-based explanations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 607–625, 2024

    2024

  49. [57]

    A concept- based explainability framework for large multimodal models

    Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. A concept- based explainability framework for large multimodal models. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  50. [58]

    Under- standing visual feature reliance through the lens of complexity

    Thomas Fel, Louis B´ethune, Andrew Kyle Lampinen, Thomas Serre, and Katherine Hermann. Under- standing visual feature reliance through the lens of complexity. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  51. [59]

    What you can cram into a single vector: Probing sentence embeddings for linguistic properties

    Alexis Conneau, German Kruszewski, Guillaume Lample, Lo¨ıc Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol...

    2018

  52. [60]

    Best of both worlds: local and global explanations with human-understandable concepts

    Jessica Schrouff, Sebastien Baur, Shaobo Hou, Diana Mincu, Eric Loreaux, Ralph Blanes, James Wexler, Alan Karthikesalingam, and Been Kim. Best of both worlds: local and global explanations with human-understandable concepts. arXiv preprint arXiv:2106.08641, 2021

    2021 arXiv

  53. [61]

    S ummit: Scaling deep learning interpretability by visualizing activation and attribution summarizations

    Fred Hohman, Haekyu Park, Caleb Robinson, and Duen Horng Polo Chau. S ummit: Scaling deep learning interpretability by visualizing activation and attribution summarizations. IEEE Transactions on Visualization and Computer Graphics, 26(1):1096–1106, 2019

    2019

  54. [62]

    Spurious features everywhere-large-scale detection of harmful spurious features in imagenet

    Yannic Neuhaus, Maximilian Augustin, Valentyn Boreiko, and Matthias Hein. Spurious features everywhere-large-scale detection of harmful spurious features in imagenet. In Proceedings of the IEEE International Conference on Computer Vision, pages 20235–20246, 2023

    2023

  55. [63]

    A multimodal automated interpretability agent

    Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A multimodal automated interpretability agent. In International Conference on Machine Learning, 2024

    2024

  56. [64]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  57. [65]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1):1–9, 2018

    2018

  58. [66]

    Mobileclip: Fast image-text models through multi-modal reinforced training

    Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15963–15974, 2024

    2024

  59. [67]

    V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, et al

    Maxime Oquab, Timoth´ee Darcet, Th ´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=a68SUt6zFt

    2024

  60. [68]

    A textbook remedy for domain shifts: Knowledge priors for medical image analysis

    Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael S Yao, Chris Callison-Burch, James Gee, and Mark Yatskar. A textbook remedy for domain shifts: Knowledge priors for medical image analysis. In Advancements In Medical Foundation Models: Explainability, Robustness, Security, ...

    2024

  61. [69]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770– 778, 2016

    2016

  62. [70]

    Efficient unsupervised shortcut learning detection and mitigation in transformers

    Lukas Kuhn, Sari Sadiya, Jorg Schlotterer, Christin Seifert, and Gemma Roig. Efficient unsupervised shortcut learning detection and mitigation in transformers. arXiv preprint arXiv:2501.00942, 2025

    2025 arXiv

  63. [71]

    Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations

    Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3619–3629, 2021

    2021

  64. [72]

    Reveal to revise: An explainable ai life cycle for iterative bias correction of deep models

    Frederik Pahde, Maximilian Dreyer, Wojciech Samek, and Sebastian Lapuschkin. Reveal to revise: An explainable ai life cycle for iterative bias correction of deep models. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 596–606. S...

    2023

  65. [73]

    Cosy: Evaluating textual explanations of neurons

    Laura Kopf, Philine Lou Bommer, Anna Hedstr¨om, Sebastian Lapuschkin, Marina M-C H¨ohne, and Kirill Bykov. Cosy: Evaluating textual explanations of neurons. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  66. [74]

    Towards understanding grokking: An effective theory of representation learning

    Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 34651–34663, 2022

    2022

  67. [75]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015

    2015

  68. [76]

    Skin lesion analysis toward melanoma detection

    Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analysis toward melanoma detection. In IEEE 15th International Symposium on Biomedical Imaging,...

    2018

  69. [77]

    Bcn20000: Dermoscopic lesions in the wild

    Carlos Hern´andez-P´erez, Marc Combalia, Sebastian Podlipnik, Noel CF Codella, Veronica Rotem- berg, Allan C Halpern, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Brian Helba, et al. Bcn20000: Dermoscopic lesions in the wild. Scientific Data, 11(1):641, 2024

    2024

  70. [78]

    Clinical ABCDE rule for early melanoma detection

    Ana F Duarte, Bernardo Sousa-Pinto, Lu´ıs F Azevedo, Ana M Barros, Susana Puig, Josep Malvehy, Eckart Haneke, and Osvaldo Correia. Clinical ABCDE rule for early melanoma detection. European Journal of Dermatology, 31(6):771–778, December 2021

    2021

  71. [79]

    Analysis of the isic image datasets: Usage, benchmarks and recommendations

    Bill Cassidy, Connah Kendrick, Andrzej Brodzicki, Joanna Jaworek-Korjakowska, and Moi Hoon Yap. Analysis of the isic image datasets: Usage, benchmarks and recommendations. Medical Image Analysis, 75:102305, 2022

    2022

  72. [80]

    Gadgil, Alex J

    Chanwoo Kim, Soham U. Gadgil, Alex J. DeGrave, Jesutofunmi A. Omiye, Zhuo Ran Cai, Roxana Daneshjou, and Su-In Lee. Transparent medical image ai via an image–text foundation model grounded in medical literature. Nature Medicine, 2024

    2024

  73. [81]

    innspector: Visual, interactive deep model debugging

    Thilo Spinner, Daniel F¨urst, and Mennatallah El-Assady. innspector: Visual, interactive deep model debugging. arXiv preprint arXiv:2407.17998, 2024

    2024 arXiv

  74. [82]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In Advances in Neural Info...

    2022

  75. [83]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi...

    2022

  76. [84]

    Polysemanticity and capacity in neural networks

    Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022

    2022 arXiv

  77. [85]

    Large- scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large- scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2023

    2023

  78. [86]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural L...

    2021

  79. [87]

    From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai

    Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, J ¨org Schl¨otterer, Maurice Van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing ...

    2023

  80. [88]

    Challenges and opportunities in text generation explainability

    Kenza Amara, Rita Sevastjanova, and Mennatallah El-Assady. Challenges and opportunities in text generation explainability. In World Conference on Explainable Artificial Intelligence, pages 244–264. Springer, 2024

    2024

  81. [89]

    Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead

    Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206–215, 2019. 23 Mechanistic understanding and validation of large AI models

    2019

  82. [90]

    Explainable artificial intelligence (xai) 2.0: A manifesto of open challenges and interdisciplinary research directions

    Luca Longo, Mario Brcic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger, et al. Explainable artificial intelligence (xai) 2.0: A manifesto of open challenges and interdisciplinary res...

    2024

  83. [91]

    Pytorch: An imperative style, high- performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems,...

    2019

  84. [92]

    rachtibat/zennit-crp: v0.6.0

    Reduan Achtibat, Maximilian Dreyer, and Sebastian Lapuschkin. rachtibat/zennit-crp: v0.6.0. Zenodo,

  85. [93]

    Software for dataset-wide xai: from local explanations to global insights with zennit, corelay, and virelay

    Christopher J Anders, David Neumann, Wojciech Samek, Klaus-Robert M ¨uller, and Sebastian La- puschkin. Software for dataset-wide xai: from local explanations to global insights with zennit, corelay, and virelay. arXiv preprint arXiv:2106.13200, 2021

    2021 arXiv

  86. [94]

    On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

    Sebastian Bach, Alexander Binder, Gr´egoire Montavon, Frederick Klauschen, Klaus-Robert M¨uller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7):e0130140, 2015

    2015

  87. [95]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017

    2017

  88. [96]

    Torchvision the machine-vision package of torch

    S´ebastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. InProceedings of the 18th ACM international conference on Multimedia, pages 1485–1488, 2010

    2010

  89. [97]

    Pytorch image models

    Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models, 2019

    2019

  90. [98]

    Batch normalization: accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, page 448–456, 2015

    2015

  91. [99]

    Umap: Uniform manifold approximation and projection

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29), 2018

    2018

  92. [100]

    Network dissection: Quantifying interpretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3319–3327, 2017. doi: 10....

    2017 doi

  93. [101]

    Paco: Parts and attributes of common objects

    Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, Amir Mousavi, Yiwen Song, Abhimanyu Dubey, and Dhruv Mahajan. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/C...

    2023

  94. [102]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022

    2022

  95. [103]

    Dermoscopy: what’s new? Clinics in Dermatology, 27(1):26–34, 2009

    Ralph Peter Braun, Margaret Oliviero, Isabel Kolm, Lars E French, Ashfaq A Marghoob, and Harold Rabinovitz. Dermoscopy: what’s new? Clinics in Dermatology, 27(1):26–34, 2009

    2009

  96. [104]

    Dermoscopy of squamous cell carcinoma: from actinic keratosis to invasive forms

    M ´A Salafranca and P Zaballos. Dermoscopy of squamous cell carcinoma: from actinic keratosis to invasive forms. Actas Dermo-Sifiliograficas, pages S0001–7310, 2024

    2024

  97. [105]

    Association of shiny white blotches and strands with nonpigmented basal cell carcinoma: evaluation of an additional dermoscopic diagnostic criterion

    Cristi´an Navarrete-Dechent, Shirin Bajaj, Michael A Marchetti, Harold Rabinovitz, Stephen W Dusza, and Ashfaq A Marghoob. Association of shiny white blotches and strands with nonpigmented basal cell carcinoma: evaluation of an additional dermoscopic diagnostic criterion. JAMA...

    2016

  98. [106]

    The diagnostic value and histologic correlate of distinct patterns of shiny white streaks for the diagnosis of melanoma: A retrospective, case-control study

    Anna Eliza Verzi, Victor L Quan, Kara E Walton, Mary C Martini, Ashfaq A Marghoob, Erin M Garfield, Betty Y Kong, Maria Cristina Isales, Timothy VandenBoom, Bin Zhang, et al. The diagnostic value and histologic correlate of distinct patterns of shiny white streaks for the diag...

    2018

  99. [107]

    In- terventions for basal cell carcinoma of the skin

    Jason Thomson, Sarah Hogan, Jo Leonardi-Bee, Hywel C Williams, and Fiona J Bath-Hextall. In- terventions for basal cell carcinoma of the skin. Cochrane Database of Systematic Reviews , (11), 2020

    2020

  100. [108]

    The significance of crys- talline/chrysalis structures in the diagnosis of melanocytic and nonmelanocytic lesions

    Yevgeniy Balagula, Ralph P Braun, Harold S Rabinovitz, Stephen W Dusza, Alon Scope, Tracey N Liebman, Ines Mordente, Katherine Siamas, and Ashfaq A Marghoob. The significance of crys- talline/chrysalis structures in the diagnosis of melanocytic and nonmelanocytic lesions. Jour...

    2012

  101. [109]

    Dermoscopy in the diagnosis and management of basal cell carcinoma

    Aimilios Lallas, Zoe Apalla, Dimitrios Ioannides, Giuseppe Argenziano, Fabio Castagnetti, Elvira Moscarella, Caterina Longo, Tamara Palmieri, Dafne Ramundo, and Iris Zalaudek. Dermoscopy in the diagnosis and management of basal cell carcinoma. Future Oncology, 11(22):2975–2984, 2015

    2015

  102. [110]

    Dermoscopy–pathology relationship in seborrheic keratosis

    Akane Minagawa. Dermoscopy–pathology relationship in seborrheic keratosis. The Journal of Derma- tology, 44(5):518–524, 2017

    2017

  103. [111]

    Dermoscopy: a useful tool for the diagnosis of angiokeratoma

    Jong Hoon Kim, Mi Ri Kim, Si-Hyung Lee, Sang Eun Lee, and Seung Hun Lee. Dermoscopy: a useful tool for the diagnosis of angiokeratoma. Annals of Dermatology, 24(4):468–471, 2012

    2012

  104. [112]

    Cherry angioma: A case–control study

    Ramah I Nazer, Rahaf H Bashihab, Wedad H Al-Madani, Aamir A Omair, and Mohammed I AlJasser. Cherry angioma: A case–control study. Journal of Family and Community Medicine, 27(2):109–113, 2020

    2020

  105. [113]

    Talon noir: A case report and literature review

    Sampa Choudhury and Ashish Mandal. Talon noir: A case report and literature review. Cureus, 15(3), 2023

    2023

  106. [114]

    concept examples

    Md Mehrab Tanjim, Krishna Kumar Singh, Kushal Kafle, Ritwik Sinha, and Garrison W Cottrell. Discovering and mitigating biases in clip-based image editing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2984–2993, 2024. 25 Mechanistic...

    2024

  107. [116]

    a photo of a green car

    Probe Selection The searched concept can be specified in any modality supported by the foundation model within SEMANTIC LENS, or as combinations thereof. For instance, the search prompt may take the form of one or multiple samples from the model’s data domain (e.g., images of ...

  108. [117]

    If multiple prompts are provided, their embeddings are aggregated (e.g., via average pooling) to produce a single probe embedding, ϑprobe

    Probe Embedding The selected search prompt is embedded into the semantic space. If multiple prompts are provided, their embeddings are aggregated (e.g., via average pooling) to produce a single probe embedding, ϑprobe

  109. [118]

    animal snout

    Similarity Search The final step is a similarity search within the set of semantic embeddings VM wrt. ϑprobe: ϑ∗ = argmax ϑ∈VM {a(ϑ, ϑprobe)} with a(ϑ, ϑprobe) := s(ϑ, ϑprobe)−s(ϑ, ϑ<>). (1) 31 Mechanistic understanding and validation of large AI models Supplementary Fig. C.1:...

  110. [119]

    This relevance score is used to filter out irrelevant neurons

    Relevance filtering of components We compute the highest relevance of a component on the test set for a specific output target (here “Ox”). This relevance score is used to filter out irrelevant neurons. Concretely, we filter out neurons with relevance that is below 1 % or 5 #neurons %

  111. [120]

    <concept>

    Labelling of components To label neurons, we first embed the following textual descriptions via the text model of CLIP: Horns, Eyes, Nose, Ears, Mouth, Neck, Back, Chest, Belly, Tail, Hooves, Legs, Brown, Black, Lined texture, Striped texture, Spotted texture, Brown colour, Sh...

  112. [121]

    Concretely, we start with layer 3 and attribute component groups using the LRP εz+♭-rule wrt

    Relevance propagation We use CRP to attribute groups of components. Concretely, we start with layer 3 and attribute component groups using the LRP εz+♭-rule wrt. to the “Ox” class on the test set. Subsequently, all lower-level (layer 2) component groups are attributed wrt. an ...

  113. [122]

    block 2”) and layer 3 (“block 3

    Investigate the attribution graph The full attribution graph is shown in Supplementary Fig. D.7 for layer 2 (“block 2”) and layer 3 (“block 3”). Notably, unaligned components are marked with “?”, indicating concepts that have not been though of, i.e., included in the list of e...

  114. [2023]

    URL https://doi.org/10.5281/zenodo.7962574

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.