{"id":"3bc103e5-f4b0-430a-8b35-5df3bf44c903","arxiv_id":"2411.19700","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NAVE clusters multi-depth feature activations of frozen CNNs and vision transformers into interpretable segmentation maps for post-hoc model inspection.","lead":"NAVE segments an image into colored regions based on the internal activations of a frozen vision model, then clusters these activations to expose the concepts the model uses. The paper is relevant to explainable AI because it offers a simple, unsupervised way to inspect what an image encoder has learned without fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative semantic-alignment claim rests on an oracle segment-selection rule: the reported AP@50% measures best-segment overlap with the ground-truth box, not a label-free localization, so the comparison to LOST/TokenCut is protocol-unfair.","rationale":"The paper has genuine qualitative merit: the multi-depth concept patches (Figures 1, 2, 12) and the inspection use cases (Section 5) are suggestive, and the method is simple, unsupervised, and architecture-agnostic. I see no grounds for rejection, and the implementation appears reproducible. The decisive weakness is exactly the one the reader identified: the quantitative backbone, Table 1 and the derived claims in Tables 2 and 3, uses an oracle segment-selection rule. The AP@50% metric then answers 'does some segment overlap the object?' rather than 'does the method localize the object?', so the absolute numbers and the comparison to single-box SOTA baselines are not valid evidence for semantic alignment. I also note two secondary issues that do not change the verdict: (i) the main quantitative tables use only the last transformer block, not the multi-depth composite that defines NAVE, so the central method is not what is being measured; (ii) Appendix Table 6 shows PCA sometimes outperforming k-means on DINOv2 encoders, which undercuts the main-text claim that k-means is superior (Section 4.2). These reinforce the need for a conditional acceptance: the method is salvageable, but the evaluation protocol must be fixed before the central quantitative claim is accepted.","tokens_in":16777,"tokens_out":8064,"duration_ms":71026,"concrete_test":"Recompute Table 1 rows for NAVE, SLIC, and CONE-SHAP using a fixed, label-free selection rule instead of the oracle: for each image, pick the segment with the highest mean feature norm (or the largest connected component), compute its inner box, and report AP@50% on VOC07, VOC12, and COCO20k. Compare these numbers to LOST/TokenCut under the same protocol. If NAVE's non-oracle AP falls to the SLIC level or below the reported 62.7/62.3/61.0, the oracle selection is load-bearing for the semantic-alignment claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4.3 (Predictions), the method does not produce a predicted box. After computing NAVE's K=5 clusters, it selects the segment whose inner box has the largest IoU with the ground-truth target and reports that IoU. This oracle selection makes AP@50% an upper bound on what NAVE would deliver without access to labels. It also invalidates the direct comparison to LOST, TokenCut, and Lv et al. in Table 1, since those methods emit a single box without peeking at the target. The paper acknowledges the selection rule but still states that NAVE is 'on par with the state of the art' and uses the scores to conclude that 'the concepts captured by NAVE align with the semantics.' The same oracle is applied to SLIC and CONE-SHAP, so the relative ranking among those three is meaningful, but the absolute claim and the SOTA comparison are not. A further compounding issue is that the quantitative experiments use only the output of the last transformer block, not the multi-depth composite activations that define NAVE in Algorithm 1; thus the headline multi-depth claim is not actually what is measured in the main tables. If a fixed, label-free selection rule drops NAVE's AP to near SLIC's level, the central quantitative demonstration of semantic alignment fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NAVE (Neuro-Activated Vision Explanations), a post-hoc, unsupervised method that clusters normalized feature activations extracted from multiple depths of a frozen vision encoder to produce interpretable segmentation maps. The authors claim that NAVE's clusters correspond to semantically meaningful image regions, and they evaluate this claim qualitatively on STL-10 bird images and quantitatively through an object localization proxy using AP@50% on VOC07, VOC12, and COCO20k. They also use NAVE as an inspection tool to study the effect of registers in vision transformers and the effect of data augmentation and shortcut learning in a Chest-X-Ray pneumonia model. The central contribution is a simple, architecture-agnostic encoder inspection method that requires no fine-tuning.","tokens_in":17047,"tokens_out":4162,"duration_ms":37205,"significance":"If the quantitative claims were fully supported, NAVE would be a useful and accessible tool for post-hoc encoder inspection, with the practical appeal of being architecture-agnostic and requiring no fine-tuning or architectural changes. The qualitative demonstrations, especially the class-concept visualizations and the PCA comparison, are suggestive and visually compelling. The paper also contributes two application studies—registers in ViTs and shortcut learning in chest X-ray models—that are of independent interest. The method is simple enough to be easily reproduced, and the authors state that code will be made public. However, the main quantitative evidence currently rests on an oracle-based segment selection procedure and on a single-depth variant of the method, which limits what the experiments establish about the multi-depth NAVE algorithm described in the paper.","major_comments":[{"comment":"The quantitative evaluation uses an oracle selection rule: after computing NAVE's K=5 clusters, the segment whose inner box has the largest IoU with the ground-truth bounding box is selected and scored. This gives NAVE (and, by construction, SLIC and CONE-SHAP) access to the labels during evaluation. The resulting AP@50% is therefore an upper bound on what a label-free version of NAVE would achieve, and it invalidates the direct comparison with LOST, TokenCut, and Lv et al. in Table 1, since those methods emit a single box without peeking at the target. The relative ranking among NAVE, SLIC, and CONE-SHAP is meaningful because all three use the same oracle, but the statements that NAVE is 'on par with the state of the art' and that the results 'suggest the concepts captured by NAVE align with the semantics' are not supported by the current protocol. The authors should either provide a fixed, label-free selection rule or explicitly reframe the reported numbers as oracle upper bounds and remove the SOTA comparison.","section":"Section 4.3, Predictions"},{"comment":"The main quantitative tables use only the output of the last transformer block: the text states that 'NAVE is computed image-wise using the output of the last transformer block (without the attention) and relies on k-means with K=5 clusters.' This is not the multi-depth algorithm defined in Algorithm 1, which concatenates activations from all encoder depths. Consequently, Tables 1, 2, and 3 do not actually measure the multi-depth NAVE that the paper motivates as its core contribution. The qualitative experiments use multiple depths (e.g., residual blocks 2, 3, and 4 in Section 4.2), but the quantitative evidence for the headline multi-depth claim is missing. The authors should add experiments using the full multi-depth composite activations, or clearly separate the claims made for the single-depth and multi-depth variants.","section":"Section 4.3, Baselines"},{"comment":"The number of clusters K is selected on the evaluation datasets themselves. The appendix reports AP@50% for K=3, 5, and 7 on VOC07, VOC12, and COCO20k and concludes that 'setting K=5 achieves the best performance.' Since K is chosen after seeing the test-set scores, the numbers in Tables 1, 2, and 3 are optimistically biased. The authors should select K on a held-out validation set or report results averaged over a range of K without tuning on the test data. This issue is load-bearing because the quantitative comparisons between methods and between training schemes depend on the chosen K.","section":"Appendix, Table 7"}],"minor_comments":[{"comment":"The experimental setup is confusing: the text says the class explanation is 'trained on the 800 test images' of the bird class, but the explanations are shown for 'four selected bird training images.' Please clarify which split is used for constructing the class concepts and why using the test split is appropriate for this qualitative illustration.","section":"Section 4.2"},{"comment":"The appendix's outer-box results (e.g., NAVE ImageNet outer-box 74.9 on VOC07 in Table 4) are substantially higher than the inner-box results reported in the main text. Since the main text reports only inner-box numbers, the reader cannot see how sensitive the results are to the box-extraction strategy; the relationship between the two should be stated in the main text.","section":"Section 4.3"},{"comment":"The paper acknowledges k-means initialization variability but does not quantify it, despite stating that 'we did not notice a significant variance between multiple runs.' A small multi-seed experiment reporting mean and standard deviation of AP@50% would make this claim concrete.","section":"Section 4.1 and Conclusion"},{"comment":"The per-layer normalization divides by (1+c_j), where c_j is the number of channels. This is an ad hoc weighting that is not motivated or ablated; please clarify the rationale and, if possible, include an ablation showing its effect.","section":"Algorithm 1"},{"comment":"The phrase 'the internal clustering is optimized over 200 images with K=10' is unclear. Please specify what is optimized (e.g., K, k-means initialization, or some other hyperparameter) and how those 200 images were selected.","section":"Section 5.2"},{"comment":"There are several typographical and grammatical errors, including 'does not refines' near Figure 4, 'an a posteriori' constructions, and inconsistent capitalization of 'Vision Transformers Need Registers' in the reference list. A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core idea is attractive and the qualitative results are suggestive, but the main quantitative evidence needs substantial revision. The oracle-based segment selection and the single-depth evaluation are central to the paper's claim that NAVE's concepts align with semantics; these can be fixed within the paper's scope by adding a label-free selection rule or clearly reframing the results as upper bounds, and by evaluating the full multi-depth algorithm. The K-selection issue also needs to be addressed. If the authors can provide these experiments, the paper could become a useful contribution to the explainability literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: NAVE is a plausible and simple post-hoc encoder-explanation method. It clusters normalized activations from several depths of a frozen vision encoder and produces segmentation maps whose clusters often track semantic regions. The multi-depth concatenation with per-layer normalization is a new combination, and the paper is careful to place itself relative to DINO, CONE-SHAP, CRAFT, and VCC. The qualitative results are the best part: the bird-class concepts (heads, tails, feathers) and the register/artifact analysis are convincing and suggest the tool has real value for model inspection.\n\nThe soft spot is exactly where the stress-test note points. The quantitative evaluation in Section 4.3 selects the segment whose inner box has the largest IoU with the ground-truth target and then reports that IoU as the localization score. That is an oracle rule: it uses the labels to pick the segment, so the resulting AP@50% is an upper bound and the comparison to LOST and TokenCut, which emit a single box without peeking, is protocol-unfair. The paper states the rule openly, but it still claims NAVE is 'on par with the state of the art' and uses those numbers to conclude semantic alignment. That conclusion does not follow from the reported experiments. The relative ranking among NAVE, SLIC, and CONE-SHAP under the same oracle is meaningful, but the absolute claim and the SOTA comparison are not.\n\nThere are two more issues, both real but smaller. First, the main localization table is computed using only the output of the last transformer block, not the multi-depth composite activations that define NAVE in Algorithm 1. The appendix has layer ablations, so the multi-depth claim has some support, but the headline configuration is not what is measured in Table 1. Second, the paper claims k-means separates information better than PCA, yet its own Table 6 shows PCA beating k-means on several settings (DINOv2-ViTS/14, for instance). That subclaim needs to be walked back.\n\nThe method itself is not circular—the evaluation is. The qualitative inspection results, including the watermark and register experiments, do not depend on the oracle and stand on their own. The fix is clear: use a label-free selection rule (or a different proxy) and report the actual NAVE configuration in the main table.\n\nWho is this for? Anyone who wants a cheap, architecture-agnostic way to look inside frozen encoders, and researchers studying training artifacts. It deserves a serious referee but not acceptance as-is. I would recommend major revision focused on the evaluation protocol.","headline":"Useful and simple encoder-inspection tool, but the headline quantitative claim is undermined by an oracle segment-selection protocol; qualitative and inspection results stand.","tokens_in":17554,"tokens_out":2638,"would_cite":true,"duration_ms":22812,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NAVE is a post-hoc, unsupervised method that turns the multi-depth activations of a frozen vision encoder into cluster-based segmentation maps, and the paper argues these clusters capture semantic concepts that can be used for model…","keywords":["Explainable AI","Concept extraction","Clustering","Vision transformers","Encoder explanation","Unsupervised segmentation","Post-hoc explanation","Object localization"],"falsifier":"Run the same AP@50 evaluation using a fixed, no-peek rule for choosing which NAVE segment to score, for example always the largest cluster or the cluster closest to the image center, and compare with the paper's oracle selection. If the fixed-rule scores drop to the SLIC baseline level (about 43/46/39 on VOC07/VOC12/COCO20k) instead of staying near 60+, the apparent semantic alignment is an artifact of picking the best segment after seeing the answer.","tokens_in":16542,"feed_emoji":"🔍","tokens_out":12462,"duration_ms":88767,"temperature":0.7,"pith_summary":"NAVE is a post-hoc, unsupervised method for explaining what a frozen vision encoder has internalized, without any fine-tuning or architectural changes. The method upsamples the feature activations of several encoder layers to a common resolution, normalizes each per-position vector, concatenates the layers, and clusters the resulting multi-depth activations with k-means, so that every cluster is a segment in an explanation map of the input image. The paper's central claim is that these clusters correspond to semantically meaningful image regions: on object-localization proxies over VOC07, VOC12, and COCO20k, at least one NAVE segment overlaps an object's ground-truth box with IoU above 50 percent in roughly 61-69 percent of test images, depending on model and training, on par with specialized localization baselines. The paper further uses NAVE to inspect trained models, showing that the artifacts addressed by vision-transformer registers mostly affect backgrounds, and that shortcut-inducing watermarks in a training set visibly distort how an encoder segments a scene. If these claims hold, NAVE provides a lightweight, architecture-agnostic inspection tool for understanding the representation stage of a vision model before any classification head is applied.","feed_headline":"NAVE turns frozen encoders into readable semantic maps","feed_subtitle":"Post-hoc clustering of multi-depth activations finds objects without fine-tuning.","key_machinery":"The load-bearing object is the multi-depth composite activation vector. For each layer $j$ of the frozen encoder, NAVE upsamples the feature map to a common resolution $(h,w)$, reshapes it into one vector per spatial position of dimension $c_j$, normalizes each vector by its $\\ell^2$ norm, and scales it by $1/(1+c_j)$ to balance layers with different channel counts. The per-position vectors from all selected depths are concatenated, and $k$-means clusters the full set of positions, so each pixel inherits a cluster label and the explanation map is a segmentation. Because no gradient flows and no labels are used, the method is post-hoc, unsupervised, and architecture-agnostic across CNNs and ViTs; the number of clusters $K$ controls the granularity of the extracted concepts. For class-level explanations, activations from multiple images of the same class are clustered together, in the qualitative study with hierarchical Ward linkage, so that a cluster's patches reveal a recurring concept such as head, tail, or uniform background.","core_discovery":"The paper claims that the internal representations of a frozen vision encoder can be read directly by clustering composite feature activations collected from multiple depths, and that the resulting clusters are the encoder's own concepts: image regions it processes similarly. Using object localization as the proxy, NAVE is shown to find segments whose bounding boxes match ground-truth objects with IoU above 50 percent in most test images, even though NAVE never sees labels and never modifies the network. A supervised ViT-Small trained on ImageNet1K gives the best localization (68.7/69.5/64.1 AP@50 on VOC07/VOC12/COCO20k) among the tested encoders, DINO and DINOv2 self-supervised encoders come close, and a randomly initialized encoder performs no better than generic SLIC superpixels. The paper also claims two inspection results: registers in DINOv2 vision transformers do not significantly change localization scores, so the artifacts they remove mostly affect backgrounds, and encoders trained with watermark shortcuts produce degenerate explanation maps that lock onto the shortcut region. The intended contribution is a practical, post-hoc inspection tool rather than a semantic segmentation method; NAVE is meant to show what information the encoder supplies to the classifier head.","pith_inferences":["A natural extension the paper does not run is to replace the oracle segment selection with a heuristic or learned selector, such as the largest segment or the segment closest to the image center, and re-measure localization; this would reveal how much of the reported alignment a user could see without knowing the answer.","Because NAVE is unsupervised and works on any frozen encoder, it could serve as a dataset-auditing tool: clusters that persistently capture watermarks, hospital tags, or other unintended markers would expose shortcut structure in a training set before a classifier is deployed.","The per-layer normalization by $1/(1+c_j)$ suggests that depth-balancing is doing real work; an ablation that removes or reweights the scaling would reveal whether multi-depth fusion, rather than the deepest layer alone, drives the localization scores.","NAVE's maps might serve as pseudo-labels for weakly supervised semantic segmentation, since the clusters align with objects without label supervision, though the paper explicitly avoids claiming NAVE is a segmentation method."],"forward_implications":["Frozen encoders can be audited without fine-tuning: NAVE maps reveal which regions a model treats as alike, so a practitioner can inspect what the representation stage has learned before any classification head is attached.","Training data and training scheme shape the encoder's concepts: a model trained on a dataset containing a concept such as person isolates it in NAVE maps, while a model trained without it does not; self-supervised DINO and DINOv2 encoders nearly match a supervised ImageNet classifier in this proxy.","Artifacts in vision transformers are mostly a background phenomenon: adding registers to DINOv2 barely changes NAVE's object-localization scores, so the semantic segments tied to main objects are preserved.","Shortcut learning is visible as distorted segmentations: when watermarks are available as a shortcut during training, NAVE maps lock onto the watermark region and lose the surrounding anatomy, giving a direct visual signature of a degenerate encoder.","NAVE's segmentation can be produced at any granularity by changing $K$, making it a tunable lens for moving from coarse scene layout to fine object parts."],"supporting_citations":[{"why":"Lloyd's k-means is the clustering algorithm that converts normalized multi-depth activations into segment labels; it is the central mechanism of NAVE.","marker":"[37]"},{"why":"DINO establishes that self-supervised ViT features carry semantic information; NAVE uses DINO models and compares against its PCA visualization.","marker":"[8]"},{"why":"DINOv2 provides the ViT-S and ViT-B encoders used in the training-scheme and register experiments.","marker":"[39]"},{"why":"LOST supplies the object-localization protocol (AP@50) and evaluation code that NAVE adopts, and is an object-localization baseline.","marker":"[46]"},{"why":"SLIC superpixels serve as the non-semantic baseline whose scores are much lower, and they underpin the CONE-SHAP baseline.","marker":"[1]"},{"why":"CRAFT is the concept-attribution baseline compared qualitatively for class-concept extraction.","marker":"[20]"},{"why":"CONE-SHAP is the concept-explanation baseline compared quantitatively in the object-localization proxy.","marker":"[33]"},{"why":"The registers paper defines the ViT artifacts that NAVE inspects to show that artifacts mainly affect backgrounds.","marker":"[14]"}],"fun_headline_variants":["NAVE turns frozen encoders into readable concept maps","Clustering multi-depth activations reveals a vision encoder's concepts","No fine-tuning needed: NAVE extracts concepts from any frozen encoder","Read a vision encoder's mind with NAVE's post-hoc concept maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that NAVE's segments match objects depends on letting the method peek at the correct answer and pick the one segment that overlaps the object best; a person or system that has to choose a segment without that answer may see much worse alignment.","fun_headline_variants_meta":{"raw":{"variants":["NAVE turns frozen encoders into readable concept maps","Clustering multi-depth activations reveals a vision encoder's concepts","No fine-tuning needed: NAVE extracts concepts from any frozen encoder","Read a vision encoder's mind with NAVE's post-hoc concept maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000587,"raw_usage":{"total_tokens":2771,"prompt_tokens":972,"completion_tokens":1799,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1727}},"tokens_in":588,"tokens_out":1799,"duration_ms":11154,"temperature":1.0,"reasoning_tokens":1727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:55:43.454388+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same AP@50 evaluation using a fixed, no-peek rule for choosing which NAVE segment to score, for example always the largest cluster or the cluster closest to the image center, and compare with the paper's oracle selection. If the fixed-rule scores drop to the SLIC baseline level (about 43/46/39 on VOC07/VOC12/COCO20k) instead of staying near 60+, the apparent semantic alignment is an artifact of picking the best segment after seeing the answer.","supporting_citations":[{"cited_title":"IEEE Transactions on Information Theory 28(2), 129–137 (1982) 4","cited_arxiv_id":null,"evidence_quote":"Lloyd's k-means is the clustering algorithm that converts normalized multi-depth activations into segment labels; it is the central mechanism of NAVE."},{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"DINO establishes that self-supervised ViT features carry semantic information; NAVE uses DINO models and compares against its PCA visualization."},{"cited_title":"Transactions on Machine Learning Research (2024) 1, 5, 7, 8, 12","cited_arxiv_id":null,"evidence_quote":"DINOv2 provides the ViT-S and ViT-B encoders used in the training-scheme and register experiments."},{"cited_title":"In: Proceedings of the British Machine Vision Conference (November 2021) 3, 5, 8, 9","cited_arxiv_id":null,"evidence_quote":"LOST supplies the object-localization protocol (AP@50) and evaluation code that NAVE adopts, and is an object-localization baseline."},{"cited_title":"IEEE Transactions on Pattern Analysis and Machine Intelligence34(11), 2274–2282 (2012) 3, 5","cited_arxiv_id":null,"evidence_quote":"SLIC superpixels serve as the non-semantic baseline whose scores are much lower, and they underpin the CONE-SHAP baseline."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"CRAFT is the concept-attribution baseline compared qualitatively for class-concept extraction."},{"cited_title":"In: Proceedings of the 29th ACM International Conference on Multimedia","cited_arxiv_id":null,"evidence_quote":"CONE-SHAP is the concept-explanation baseline compared quantitatively in the object-localization proxy."},{"cited_title":"In: International Conference on Learning Representations (2024) 12","cited_arxiv_id":null,"evidence_quote":"The registers paper defines the ViT artifacts that NAVE inspects to show that artifacts mainly affect backgrounds."}],"review_version":1}