Pith. sign in

REVIEW 3 major objections 6 minor 55 references

Multi-Depth Concept Extraction for Post-Hoc Vision Encoder Explanation

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read NAVE is a post-hoc, unsupervised method that turns the multi-depth activations of a frozen vision encoder into cluster-based segmentation maps, and the paper argues these clusters capture semantic concepts that can be used for model…

desk verdict Useful and simple encoder-inspection tool, but the headline quantitative claim is undermined by an oracle segment-selection protocol; qualitative and inspection results stand. read the letter →

arxiv 2411.19700 v5 pith:WNNODUDP submitted 2024-11-29 cs.CV cs.LG

Ahcène Boubekki , Samuel G. Fadel , Sebastian Mair This is my paper · ORCID
classification cs.CVcs.LG
keywords ExplainableAIConceptextractionClusteringVisiontransformersEncoderexplanationUnsupervisedsegmentationPost-hocObjectlocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NAVE is a post-hoc, unsupervised method for explaining what a frozen vision encoder has internalized, without any fine-tuning or architectural changes. The method upsamples the feature activations of several encoder layers to a common resolution, normalizes each per-position vector, concatenates the layers, and clusters the resulting multi-depth activations with k-means, so that every cluster is a segment in an explanation map of the input image. The paper's central claim is that these clusters correspond to semantically meaningful image regions: on object-localization proxies over VOC07, VOC12, and COCO20k, at least one NAVE segment overlaps an object's ground-truth box with IoU above 50 percent in roughly 61-69 percent of test images, depending on model and training, on par with specialized localization baselines. The paper further uses NAVE to inspect trained models, showing that the artifacts addressed by vision-transformer registers mostly affect backgrounds, and that shortcut-inducing watermarks in a training set visibly distort how an encoder segments a scene. If these claims hold, NAVE provides a lightweight, architecture-agnostic inspection tool for understanding the representation stage of a vision model before any classification head is applied.

What carries the argument

The load-bearing object is the multi-depth composite activation vector. For each layer $j$ of the frozen encoder, NAVE upsamples the feature map to a common resolution $(h,w)$, reshapes it into one vector per spatial position of dimension $c_j$, normalizes each vector by its $\ell^2$ norm, and scales it by $1/(1+c_j)$ to balance layers with different channel counts. The per-position vectors from all selected depths are concatenated, and $k$-means clusters the full set of positions, so each pixel inherits a cluster label and the explanation map is a segmentation. Because no gradient flows and no labels are used, the method is post-hoc, unsupervised, and architecture-agnostic across CNNs and ViTs; the number of clusters $K$ controls the granularity of the extracted concepts. For class-level explanations, activations from multiple images of the same class are clustered together, in the qualitative study with hierarchical Ward linkage, so that a cluster's patches reveal a recurring concept such as head, tail, or uniform background.

What would settle it

Run the same AP@50 evaluation using a fixed, no-peek rule for choosing which NAVE segment to score, for example always the largest cluster or the cluster closest to the image center, and compare with the paper's oracle selection. If the fixed-rule scores drop to the SLIC baseline level (about 43/46/39 on VOC07/VOC12/COCO20k) instead of staying near 60+, the apparent semantic alignment is an artifact of picking the best segment after seeing the answer.

Watch

Extended reading notes

Core claim

The paper claims that the internal representations of a frozen vision encoder can be read directly by clustering composite feature activations collected from multiple depths, and that the resulting clusters are the encoder's own concepts: image regions it processes similarly. Using object localization as the proxy, NAVE is shown to find segments whose bounding boxes match ground-truth objects with IoU above 50 percent in most test images, even though NAVE never sees labels and never modifies the network. A supervised ViT-Small trained on ImageNet1K gives the best localization (68.7/69.5/64.1 AP@50 on VOC07/VOC12/COCO20k) among the tested encoders, DINO and DINOv2 self-supervised encoders come close, and a randomly initialized encoder performs no better than generic SLIC superpixels. The paper also claims two inspection results: registers in DINOv2 vision transformers do not significantly change localization scores, so the artifacts they remove mostly affect backgrounds, and encoders trained with watermark shortcuts produce degenerate explanation maps that lock onto the shortcut region. The intended contribution is a practical, post-hoc inspection tool rather than a semantic segmentation method; NAVE is meant to show what information the encoder supplies to the classifier head.

Load-bearing premise

The proof that NAVE's segments match objects depends on letting the method peek at the correct answer and pick the one segment that overlaps the object best; a person or system that has to choose a segment without that answer may see much worse alignment.

Editorial extensions

If this is right

  • Frozen encoders can be audited without fine-tuning: NAVE maps reveal which regions a model treats as alike, so a practitioner can inspect what the representation stage has learned before any classification head is attached.
  • Training data and training scheme shape the encoder's concepts: a model trained on a dataset containing a concept such as person isolates it in NAVE maps, while a model trained without it does not; self-supervised DINO and DINOv2 encoders nearly match a supervised ImageNet classifier in this proxy.
  • Artifacts in vision transformers are mostly a background phenomenon: adding registers to DINOv2 barely changes NAVE's object-localization scores, so the semantic segments tied to main objects are preserved.
  • Shortcut learning is visible as distorted segmentations: when watermarks are available as a shortcut during training, NAVE maps lock onto the watermark region and lose the surrounding anatomy, giving a direct visual signature of a degenerate encoder.
  • NAVE's segmentation can be produced at any granularity by changing $K$, making it a tunable lens for moving from coarse scene layout to fine object parts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is to replace the oracle segment selection with a heuristic or learned selector, such as the largest segment or the segment closest to the image center, and re-measure localization; this would reveal how much of the reported alignment a user could see without knowing the answer.
  • Because NAVE is unsupervised and works on any frozen encoder, it could serve as a dataset-auditing tool: clusters that persistently capture watermarks, hospital tags, or other unintended markers would expose shortcut structure in a training set before a classifier is deployed.
  • The per-layer normalization by $1/(1+c_j)$ suggests that depth-balancing is doing real work; an ablation that removes or reweights the scaling would reveal whether multi-depth fusion, rather than the deepest layer alone, drives the localization scores.
  • NAVE's maps might serve as pseudo-labels for weakly supervised semantic segmentation, since the clusters align with objects without label supervision, though the paper explicitly avoids claiming NAVE is a segmentation method.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces NAVE (Neuro-Activated Vision Explanations), a post-hoc, unsupervised method that clusters normalized feature activations extracted from multiple depths of a frozen vision encoder to produce interpretable segmentation maps. The authors claim that NAVE's clusters correspond to semantically meaningful image regions, and they evaluate this claim qualitatively on STL-10 bird images and quantitatively through an object localization proxy using AP@50% on VOC07, VOC12, and COCO20k. They also use NAVE as an inspection tool to study the effect of registers in vision transformers and the effect of data augmentation and shortcut learning in a Chest-X-Ray pneumonia model. The central contribution is a simple, architecture-agnostic encoder inspection method that requires no fine-tuning.

Significance. If the quantitative claims were fully supported, NAVE would be a useful and accessible tool for post-hoc encoder inspection, with the practical appeal of being architecture-agnostic and requiring no fine-tuning or architectural changes. The qualitative demonstrations, especially the class-concept visualizations and the PCA comparison, are suggestive and visually compelling. The paper also contributes two application studies—registers in ViTs and shortcut learning in chest X-ray models—that are of independent interest. The method is simple enough to be easily reproduced, and the authors state that code will be made public. However, the main quantitative evidence currently rests on an oracle-based segment selection procedure and on a single-depth variant of the method, which limits what the experiments establish about the multi-depth NAVE algorithm described in the paper.

major comments (3)
  1. [Section 4.3, Predictions] The quantitative evaluation uses an oracle selection rule: after computing NAVE's K=5 clusters, the segment whose inner box has the largest IoU with the ground-truth bounding box is selected and scored. This gives NAVE (and, by construction, SLIC and CONE-SHAP) access to the labels during evaluation. The resulting AP@50% is therefore an upper bound on what a label-free version of NAVE would achieve, and it invalidates the direct comparison with LOST, TokenCut, and Lv et al. in Table 1, since those methods emit a single box without peeking at the target. The relative ranking among NAVE, SLIC, and CONE-SHAP is meaningful because all three use the same oracle, but the statements that NAVE is 'on par with the state of the art' and that the results 'suggest the concepts captured by NAVE align with the semantics' are not supported by the current protocol. The authors should either provide a fixed, label-free selection rule or explicitly reframe the reported numbers as oracle upper bounds and remove the SOTA comparison.
  2. [Section 4.3, Baselines] The main quantitative tables use only the output of the last transformer block: the text states that 'NAVE is computed image-wise using the output of the last transformer block (without the attention) and relies on k-means with K=5 clusters.' This is not the multi-depth algorithm defined in Algorithm 1, which concatenates activations from all encoder depths. Consequently, Tables 1, 2, and 3 do not actually measure the multi-depth NAVE that the paper motivates as its core contribution. The qualitative experiments use multiple depths (e.g., residual blocks 2, 3, and 4 in Section 4.2), but the quantitative evidence for the headline multi-depth claim is missing. The authors should add experiments using the full multi-depth composite activations, or clearly separate the claims made for the single-depth and multi-depth variants.
  3. [Appendix, Table 7] The number of clusters K is selected on the evaluation datasets themselves. The appendix reports AP@50% for K=3, 5, and 7 on VOC07, VOC12, and COCO20k and concludes that 'setting K=5 achieves the best performance.' Since K is chosen after seeing the test-set scores, the numbers in Tables 1, 2, and 3 are optimistically biased. The authors should select K on a held-out validation set or report results averaged over a range of K without tuning on the test data. This issue is load-bearing because the quantitative comparisons between methods and between training schemes depend on the chosen K.
minor comments (6)
  1. [Section 4.2] The experimental setup is confusing: the text says the class explanation is 'trained on the 800 test images' of the bird class, but the explanations are shown for 'four selected bird training images.' Please clarify which split is used for constructing the class concepts and why using the test split is appropriate for this qualitative illustration.
  2. [Section 4.3] The appendix's outer-box results (e.g., NAVE ImageNet outer-box 74.9 on VOC07 in Table 4) are substantially higher than the inner-box results reported in the main text. Since the main text reports only inner-box numbers, the reader cannot see how sensitive the results are to the box-extraction strategy; the relationship between the two should be stated in the main text.
  3. [Section 4.1 and Conclusion] The paper acknowledges k-means initialization variability but does not quantify it, despite stating that 'we did not notice a significant variance between multiple runs.' A small multi-seed experiment reporting mean and standard deviation of AP@50% would make this claim concrete.
  4. [Algorithm 1] The per-layer normalization divides by (1+c_j), where c_j is the number of channels. This is an ad hoc weighting that is not motivated or ablated; please clarify the rationale and, if possible, include an ablation showing its effect.
  5. [Section 5.2] The phrase 'the internal clustering is optimized over 200 images with K=10' is unclear. Please specify what is optimized (e.g., K, k-means initialization, or some other hyperparameter) and how those 200 images were selected.
  6. [Throughout] There are several typographical and grammatical errors, including 'does not refines' near Figure 4, 'an a posteriori' constructions, and inconsistent capitalization of 'Vision Transformers Need Registers' in the reference list. A careful proofreading pass is needed.

Circularity Check

1 steps flagged · score 6.0 of 10

Quantitative alignment claim rests on an oracle segment-selection rule, making AP@50% a best-case upper bound rather than a label-free prediction.

  1. self definitional [Section 4.3, Predictions (page 8) and Table 1 (page 9)]
    "Since NAVE explains only the encoder's part of the neural network, we instead select the segment with the bounding box that has the largest IoU with the bounding box target. ... Following the protocol used in [46], we report the Average Precision at 50% (AP@50%), also known as the Correct Localization metric, which is the frequency of the predicted boxes with an Intersection over Union score (IoU) larger than 50% with at least one ground truth bounding box."

    The 'predicted box' is not produced by a fixed, label-free rule: it is the box of the segment that maximizes IoU with the target ground-truth box. The same ground-truth box is then used to score whether the prediction is correct. Hence AP@50% measures, by construction, the best-case overlap among the K candidate segments, i.e., an oracle upper bound rather than an autonomous localization accuracy. Comparing this number with LOST, TokenCut, and Lv et al., which emit a single box without peeking at the target, is protocol-unfair. The conclusion that 'the concepts captured by NAVE align with the semantics present in the images' is therefore supported only by an evaluation whose selection rule already knows the target.

full rationale

NAVE itself is a straightforward unsupervised clustering of normalized multi-layer activations and is not circular. The circularity is confined to the quantitative object-localization evaluation in Section 4.3. There, the segment used for the 'prediction' is chosen by maximizing IoU with the very ground-truth box against which the prediction is scored, so AP@50% is a best-case oracle upper bound rather than a label-free prediction. Because LOST, TokenCut, and Lv et al. produce one box without access to the target, Table 1 does not compare like with like. The same oracle is applied to CONE-SHAP and SLIC, so the internal ranking among those three is potentially informative, but the paper's absolute claim that NAVE concepts align with semantics is supported only by this oracle metric. Note also that the headline multi-depth aspect of Algorithm 1 is not what is measured in the main tables, which use only the last transformer block; this is a correctness or validity gap rather than a circularity. No load-bearing self-citation chain was found; the one self-citation [21] is related-work context. Overall score 6.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

NAVE's central operation is a small set of design choices and assumptions rather than a derivation. The ledger lists the hand-picked scaling, the cluster count, and two assumptions about what activation clustering means and how it is measured. The evaluation proxy is the most important assumption: the reported AP@50% is an oracle-selected score. No invented entities are introduced; concept is a label on a cluster, not a new physical or model entity.

free parameters (3)
  • Number of clusters K = K=5 for localization; K=10 for class explanations
    Granularity hyperparameter; the appendix tests K=3, 5, 7 and K=5 is declared best on the evaluation datasets, so it is fitted on test.
  • Per-layer depth scale 1/(1+c_j) = None, hand-chosen
    Introduced in Algorithm 1 to weight layers; no ablation is shown and the method's behavior may change with this choice.
  • Explanation resolution (h,w) = Recommended to match first extracted layer
    A user-chosen trade-off between detail, artifacts, and compute; not systematically selected in the main experiments.
assumptions (3)
  • domain assumption If the encoder considers two images similar, it must have recognized similar concepts within them, and those concepts are identifiable in intermediate activations.
    Section 1, second paragraph; this is the conceptual foundation for NAVE and is not independently verified.
  • ad hoc to paper After upsampling to a common grid and normalizing each layer by l2 norm and 1/(1+c_j), the concatenated activation vector at each pixel can be clustered with k-means into spatially coherent, semantically meaningful regions.
    Algorithm 1; the depth scaling 1/(1+c_j) is introduced for this paper with no ablation or theoretical justification.
  • domain assumption AP@50% after selecting the segment with the largest IoU against the ground truth measures whether concepts align with input semantics.
    Section 4.3, Predictions; this underlies all quantitative alignment claims and is the main source of circular evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Depth Concept Extraction for Post-Hoc Vision Encoder Explanation." pith.science (2026). https://pith.science/paper/WNNODUDP

@misc{pith2026241119700,
  author       = {Pith},
  title        = {Pith review of: Multi-Depth Concept Extraction for Post-Hoc Vision Encoder Explanation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNNODUDP}},
  note         = {Machine review of arXiv:2411.19700}
}
read the original abstract

Explainable AI methods for vision models aim to identify the parts of the input that are important for the final prediction and subsequently relate these regions to human-understandable concepts. Here, we propose focusing solely on the encoder and relating its intermediate outputs to the input, instead. We introduce Neuro-Activated Vision Explanations (NAVE), a post-hoc, unsupervised, and architecture-agnostic (across CNNs and ViTs) method for extracting and visualizing internal representations from frozen vision model encoders. Specifically, NAVE clusters composite feature activations from multiple encoder depths to produce interpretable segmentation maps with controllable granularity, requiring no fine-tuning or architectural modifications. Through extensive experiments, we quantitatively demonstrate that NAVE's concepts align with input semantics and can be used in downstream tasks. We further demonstrate NAVE as an inspection tool by analyzing how training strategies and architectures affect encoder representations. Overall, our results establish NAVE as an effective tool for post-hoc model inspection and enhancing transparency in vision models. \texttt{https://github.com/Ahcene-B/NAVE}

Figures

Figures reproduced from arXiv: 2411.19700 by the authors.

Figure 1
Figure 1. Comparison of class concepts extracted by different methods on bird images of STL-10. NAVE produces consistent and interpretable explanations, capturing semantic features such as heads, tails, and feathers. Concepts of CONE-SHAP, CRAFT, and VCC exhibit inconsistencies across similar regions and images [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Patches of NAVE’s class concepts. Concept 1 captures backgrounds, concept 8 body parts and concept 10 feather patterns. The segmentations of SAM succesfuly isolate the main objects except for the second image. Note the cluster outlining the objects. Also note how both eyes of the owl are not assigned the same cluster. SLIC’s results are less precise and do not respect the semantic of the images, such as the backgrou… view at source ↗
Figure 3
Figure 3. Projections of the output of a DINOv2-ViTS/14 into three components of using PCA (DINO) or k-means (NAVE). The color channels are used to represent, respectively, the importance of each direction or the closeness to each centroid [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Explanations derived from DINO and NAVE with increasing number of com￾ponents/clusters (K). While NAVE uses the extra clusters to refine the decomposition of the scene, the PCA-based one barely changes beyond 6 components. On the other hand, scenes can be inferred from…
Figure 5
Figure 5. Figure 5: Average pixel color of 20 NAVE explanations extracted from a ResNet18 either untrained (Random Init.) or trained on STL-10 or ImageNet1K. Visualization. In [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: NAVE on four sample images taken from [14, 39] (with K = 5 clusters) reveals that registers do not fully safeguard ViTs trained with DINOv2 against artifacts [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Explanations of course a ResNet50 encoder trained under various regimes of data augmentation and watermarking. Remark that if concepts are capturing the watermark, they are disproportionately large compared to the watermark itself. 5.2 Augmentation and Shortcuts Augmen…
Figure 8
Figure 8. Figure 8: Example taken from VOC07 of the computed outer (red, center) and inner (blue, bottom) boxes of the top left connected component and their respective IoU score with respect to the ground truth bounding box (green). Algorithm 2 Python pseudocode to extract the outer-box …
Figure 9
Figure 9. Figure 9: Ablation study of the number of clusters for NAVE for a ResNet50 and a ViTB16 pretrained on ImageNet1K [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Ablation study of the layers used to compute NAVE for a ResNet50 and a ViTB16 pretrained on ImageNet1K [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: Ablation study of the clustering algorithm used in NAVE for a ResNet50 and a ViTB16 pretrained on ImageNet1K [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: NAVE’s class concepts patches of birds from STL-10 with interpretations [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 54 canonical work pages

  1. [1]

    IEEE Transactions on Pattern Analysis and Machine Intelligence34(11), 2274–2282 (2012) 3, 5

    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: SLIC superpix- els compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence34(11), 2274–2282 (2012) 3, 5

    work page 2012

  2. [2]

    In: Advances in Neural Information Processing Systems

    Alvarez Melis, D., Jaakkola, T.: Towards robust interpretability with self-explaining neural networks. In: Advances in Neural Information Processing Systems. vol. 31 (2018) 3

    work page 2018

  3. [3]

    Biomedical Signal Processing and Control90, 105831 (2024) 5, 13

    Arias-Londoño, J.D., Godino-Llorente, J.I.: Analysis of the clever hans effect in covid-19 detection using chest x-ray images and Bayesian deep learning. Biomedical Signal Processing and Control90, 105831 (2024) 5, 13

    work page 2024

  4. [4]

    In: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algo- rithms

    Arthur, D., Vassilvitskii, S.:k-means++: The advantages of careful seeding. In: Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algo- rithms. pp. 1027–1035. SODA ’07, Society for Industrial and Applied Mathematics (2007) 5

    work page 2007

  5. [5]

    PloS ONE10(7), e0130140 (2015) 1

    Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS ONE10(7), e0130140 (2015) 1

    work page 2015

  6. [6]

    European Journal of Radiology162, 110786 (2023) 1

    Borys, K., Schmitt, Y.A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C.M., Nensa, F.: Explainable ai in medical imaging: An overview for clinical practitioners–beyond saliency-based xai approaches. European Journal of Radiology162, 110786 (2023) 1

    work page 2023

  7. [7]

    In: International Conference on Medical Image Computing and Computer- Assisted Intervention

    Bransby, K.M., Beqiri, A., Cho Kim, W.J., Oliveira, J., Chartsias, A., Gomez, A.: Backmix: Mitigating shortcut learning in echocardiography with minimal supervi- sion. In: International Conference on Medical Image Computing and Computer- Assisted Intervention. pp. 570–579. Springer (2024) 13

    work page 2024

  8. [8]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9650–9660 (2021) 1, 3, 5, 7, 8

    work page 2021

Show all 55 references
  1. [9]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visual- ization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 782–791 (2021) 1 16 Ahcène Boubekki, Samuel G. Fadel, and Sebastian Mair

    2021

  2. [10]

    In: Advances in Neural Information Processing Systems

    Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., Su, J.K.: This looks like that: deep learning for interpretable image recognition. In: Advances in Neural Information Processing Systems. vol. 32 (2019) 1, 3

    2019

  3. [11]

    In: International Conference on Machine Learning

    Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning. pp. 1597–1607. PMLR (2020) 13, 14

    2020

  4. [12]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024) 1, 3

    Chormai, P., Herrmann, J., Müller, K.R., Montavon, G.: Disentangled explanations of neural network predictions by finding relevant subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024) 1, 3

    2024

  5. [13]

    In: Proceedings of the 14th International Conference on Arti- ficial Intelligence and Statistics

    Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the 14th International Conference on Arti- ficial Intelligence and Statistics. pp. 215–223. JMLR Workshop and Conference Proceedings (2011) 5

    2011

  6. [14]

    In: International Conference on Learning Representations (2024) 12

    Darcet, T., Oquab, M., Mairal, J., Bojanowski, P.: Vision transformers need registers. In: International Conference on Learning Representations (2024) 12

    2024

  7. [15]

    In: International Conference on Learning Representations (2021) 1, 5

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on L...

    2021

  8. [16]

    Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higher-layer features of a deep network. Tech. rep., University of Montreal (2009) 3

    2009

  9. [17]

    http://www.pascal- network.org/challenges/VOC/voc2007/workshop/index.html (2007) 5

    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PAS- CAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal- network.org/challenges/VOC/voc2007/workshop/index.html (2007) 5

    2007

  10. [18]

    http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html (2012) 5

    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PAS- CAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html (2012) 5

    2012

  11. [19]

    In: Advances in Neural Information Processing Systems

    Fel, T., Boutin, V., Béthune, L., Cadène, R., Moayeri, M., Andéol, L., Chalvidal, M., Serre, T.: A holistic approach to unifying automatic concept extraction and concept importance estimation. In: Advances in Neural Information Processing Systems. vol. 36, pp. 54805–54818 (2023) 3

    2023

  12. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Fel, T., Picard, A., Bethune, L., Boissin, T., Vigouroux, D., Colin, J., Cadène, R., Serre, T.: Craft: Concept recursive activation factorization for explainability. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2711–2721 (2023) 3, 5

    2023

  13. [21]

    Transactions on Machine Learning Research (2024) 1, 3

    Gautam, S., Boubekki, A., Höhne, M.M., Kampffmeyer, M.: Prototypical self- explainable models without re-training. Transactions on Machine Learning Research (2024) 1, 3

    2024

  14. [22]

    In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)

    Gautam, S., Höhne, M.M.C., Hansen, S., Jenssen, R., Kampffmeyer, M.: Demon- strating the risk of imbalanced datasets in chest x-ray image-based diagnostics by prototypical relevance propagation. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). pp. 1–5. ...

    2022

  15. [23]

    Nature Machine Intelligence 2(11), 665–673 (2020) 13

    Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11), 665–673 (2020) 13

    2020

  16. [24]

    In: Advances in Neural Information Processing Systems

    Ghorbani, A., Wexler, J., Zou, J.Y., Kim, B.: Towards automatic concept-based explanations. In: Advances in Neural Information Processing Systems. vol. 32 (2019) 2

    2019

  17. [25]

    Journal of the Royal Statistical Society: Series A (General)150(2), 119–137 (1987) 4 Neuro-Activated Vision Explanations 17

    Gordon, A.D.: A review of hierarchical classification. Journal of the Royal Statistical Society: Series A (General)150(2), 119–137 (1987) 4 Neuro-Activated Vision Explanations 17

    1987

  18. [26]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016) 5

    2016

  19. [27]

    Nature Methods 18(2), 203–211 (2021) 1

    Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203–211 (2021) 1

    2021

  20. [28]

    IEEE Transactions on Neural Networks and Learning Systems (2022) 1, 3

    Kauffmann, J., Esders, M., Ruff, L., Montavon, G., Samek, W., Müller, K.R.: From clustering to cluster explanations via neural networks. IEEE Transactions on Neural Networks and Learning Systems (2022) 1, 3

    2022

  21. [29]

    In: International Conference on Machine Learning

    Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.: Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In: International Conference on Machine Learning. pp. 2668–2677. PMLR (2018) 3

    2018

  22. [30]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4015–4026 (2023) 3, 5

    2023

  23. [31]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kowal, M., Wildes, R.P., Derpanis, K.G.: Visual concept connectome (vcc): Open world concept discovery and their interlayer connections in deep models. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10895–10905 (2024) 3

    2024

  24. [32]

    Nature Communications 10(1), 1096 (2019) 4, 13

    Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., Müller, K.R.: Unmasking clever hans predictors and assessing what machines really learn. Nature Communications 10(1), 1096 (2019) 4, 13

    2019

  25. [33]

    In: Proceedings of the 29th ACM International Conference on Multimedia

    Li, J., Kuang, K., Li, L., Chen, L., Zhang, S., Shao, J., Xiao, J.: Instance-wise or class-wise? a tale of neighbor shapley for concept-based explanation. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 3664–3672 (2021) 3, 5

    2021

  26. [34]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2117–2125 (2017) 1

    2017

  27. [35]

    In: European Conference on Computer Vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference on Computer Vision. pp. 740–755. Springer (2014) 5

    2014

  28. [36]

    Information Fusion103, 102094 (2024) 13

    Linhardt, L., Müller, K.R., Montavon, G.: Preemptively pruning clever-hans strate- gies in deep neural networks. Information Fusion103, 102094 (2024) 13

    2024

  29. [37]

    IEEE Transactions on Information Theory 28(2), 129–137 (1982) 4

    Lloyd, S.: Least squares quantization in PCM. IEEE Transactions on Information Theory 28(2), 129–137 (1982) 4

    1982

  30. [38]

    IEEE Transactions on Image Processing (2024) 5, 9

    Lv, Y., Zhang, J., Barnes, N., Dai, Y.: Weakly-supervised contrastive learning for unsupervised object discovery. IEEE Transactions on Image Processing (2024) 5, 9

    2024

  31. [39]

    Transactions on Machine Learning Research (2024) 1, 5, 7, 8, 12

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, ...

    2024

  32. [40]

    Scientific Reports14(1), 5369 (2024) 1 18 Ahcène Boubekki, Samuel G

    Panfilova, A.S., Turdakov, D.Y.: Applying explainable artificial intelligence methods to models for diagnosing personal traits and cognitive abilities by social network data. Scientific Reports14(1), 5369 (2024) 1 18 Ahcène Boubekki, Samuel G. Fadel, and Sebastian Mair

    2024

  33. [41]

    In: Advances in Neural Information Processing Systems

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An imperative style, high- performance deep learning library. In: Advances in Neural Information Processing Systems. vol. 32 (2019) 5

    2019

  34. [42]

    Journal of Machine Learning Research12, 2825–2830 (2011) 5

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Mach...

    2011

  35. [43]

    International Journal of Computer Vision115, 211–252 (2015) 5

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recog- nition challenge. International Journal of Computer Vision115, 211–252 (2015) 5

    2015

  36. [44]

    In: Proceedings of the IEEE International Conference on Computer Vision

    Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- CAM: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 618–626 (2017) 1

    2017

  37. [45]

    In: International Conference on Document Analysis and Recognition (2003) 13

    Simard, P.Y., Steinkraus, D., Platt, J.C., et al.: Best practices for convolutional neural networks applied to visual document analysis. In: International Conference on Document Analysis and Recognition (2003) 13

    2003

  38. [46]

    In: Proceedings of the British Machine Vision Conference (November 2021) 3, 5, 8, 9

    Siméoni, O., Puy, G., Vo, H.V., Roburin, S., Gidaris, S., Bursuc, A., Pérez, P., Marlet, R., Ponce, J.: Localizing objects with self-supervised transformers and no labels. In: Proceedings of the British Machine Vision Conference (November 2021) 3, 5, 8, 9

    2021

  39. [47]

    arXiv preprint arXiv:1312.6034 (2013) 3

    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional net- works: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013) 3

    2013 arXiv

  40. [48]

    In: Advances in Neural Information Processing Systems

    Sun, A., Ma, P., Yuan, Y., Wang, S.: Explain any concept: Segment anything meets concept-based explanation. In: Advances in Neural Information Processing Systems. vol. 36, pp. 21826–21840 (2023) 3

    2023

  41. [49]

    In: International Conference on Machine Learning

    Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: International Conference on Machine Learning. pp. 3319–3328. PMLR (2017) 1

    2017

  42. [50]

    In: IEEE/CVF International Conference on Computer Vision

    Tian, Z., Shen, C., Chen, H., He, T.: Fcos: Fully convolutional one-stage object detection. In: IEEE/CVF International Conference on Computer Vision. pp. 9626– 9635 (2019) 1

    2019

  43. [51]

    In: Conference on Computer Vision and Pattern Recognition

    Wang, Y., Shen, X., Hu, S.X., Yuan, Y., Crowley, J.L., Vaufreydaz, D.: Self- supervised transformers for unsupervised object discovery using normalized cut. In: Conference on Computer Vision and Pattern Recognition. New Orleans, LA, USA (June 2022) 3, 5, 9

    2022

  44. [52]

    In: European Conference on Computer Vision

    Williford, J.R., May, B.B., Byrne, J.: Explainable face recognition. In: European Conference on Computer Vision. pp. 248–263. Springer (2020) 1

    2020

  45. [53]

    In: Advances in Neural Information Processing Systems

    Yeh, C.K., Kim, B., Arik, S., Li, C.L., Pfister, T., Ravikumar, P.: On completeness- aware concept-based explanations in deep neural networks. In: Advances in Neural Information Processing Systems. vol. 33, pp. 20554–20565 (2020) 1

    2020

  46. [54]

    PLoS Medicine15(11), e1002683 (2018) 13

    Zech, J.R., Badgeley, M.A., Liu, M., Costa, A.B., Titano, J.J., Oermann, E.K.: Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS Medicine15(11), e1002683 (2018) 13

    2018

  47. [55]

    Zhou, B., Bau, D., Oliva, A., Torralba, A.: Interpreting deep visual representa- tions via network dissection. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(9), 2131–2145 (2018) 1, 3 Neuro-Activated Vision Explanations 19 7 Appendix 7.1 Inner and Outer Boxe...

    2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.