{"id":"48dfcb7a-f53c-43f9-8744-8d5fc05af7e8","arxiv_id":"2501.05398","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SemanticLens maps each neuron of a vision model to a CLIP-space vector, enabling text-based search, labelling, audit, and interpretability scoring of model internals.","lead":"SemanticLens maps individual neurons of a vision model into the semantic space of a foundation model such as CLIP, turning them into searchable text-like descriptions. It lets practitioners query what concepts a model uses, expose spurious correlations and the training images behind them, and check whether reasoning matches human rules such as the ABCDE rule for melanoma.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core semantic-proxy assumption (mean CLIP embedding of top patches = neuron concept) is unvalidated; audits and search would fail if cosine similarity in foundation-model space is not faithful for the target domain.","rationale":"The reader identifies the semantic proxy as the weakest assumption; I agree. The central claim is that SemanticLens maps neuron concepts into a semantically structured space, and every capability described in the paper (search, labelling, comparison, audit, interpretability measures) is computed via cosine similarity on ϑ. If that geometric mapping is not faithful, the entire framework produces plausible but unreliable outputs. The paper's own evidence for faithfulness is indirect: the label benchmark in Suppl. D.4 checks that synthetic images of a label activate the neuron, not that the ϑ vector is close to the label embedding. Likewise, the user studies in Section 4.4 validate the interpretability measures, not the semantic accuracy of the embeddings. Therefore, a direct retrieval test against ground-truth neuron concepts is needed. The test I propose is feasible with existing data (Broden) and, if it fails, would invalidate the universality claim. The reader's CONDITIONAL verdict remains appropriate: the method may work, but the key assumption is not yet established. This is not a disagreement with the reader; it is a sharper specification of the same load-bearing concern.","tokens_in":52668,"tokens_out":7132,"duration_ms":70474,"concrete_test":"Use Network Dissection (Bau et al., 2017) with Broden to obtain ground-truth concept labels for 200 neurons in a ResNet50 and 200 neurons in a ViT. For each neuron, compute SemanticLens ϑ exactly as in the paper: select top-30 activating images from ImageNet, crop with the paper's CRP (or its ViT approximation), embed with Mobile-CLIP, and average. Then rank all 1,344 Broden concept labels (as text prompts) by cosine similarity to ϑ. Compute top-1/top-5 retrieval accuracy. Repeat with WhyLesionCLIP on 50 dermoscopy neurons labeled by a board-certified dermatologist who inspects top-activating patches. If top-1 accuracy is at or near chance level (e.g., <10% for 1,344 labels), the semantic proxy assumption fails and all downstream audit claims are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All downstream capabilities in SemanticLens flow from a single assumption: that the mean of foundation-model embeddings of the top-activating cropped patches, ϑ_k = (1/|E_k|) Σ F(x) (Eq. 2), is a faithful semantic representation of neuron k's concept, and that cosine similarity in that space (Eq. 3) ranks concepts the way humans would. The paper's label-faithfulness evaluation (Suppl. D.4) does not test this: it measures whether synthetic images of the assigned label activate the neuron (ϕ in Eq. 1 of Suppl. D.4), which is a label-to-neuron activation test, not a test of whether the ϑ vector is semantically close to the label embedding. For the medical use case, WhyLesionCLIP is chosen because generic CLIP is untrusted, but no evidence shows that its cosine geometry ranks dermatological concepts correctly. The paper also applies SemanticLens to ViTs using an unvalidated upsampled-spatial-map approximation of CRP (§3.1), so the concept examples themselves may be corrupted for ViTs. If the semantic proxy fails—e.g., for polysemantic neurons, where averaging mixes unrelated concepts into an off-manifold vector—every search result, audit conclusion, and interpretability measure in Sections 4.1–4.4 is unreliable, including the headline spurious-correlation findings. The framework's universality claim therefore rests on an unverified geometric assumption rather than on an established property of the foundation model.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SemanticLens, a framework that represents each neuron (or other component) of a vision model by the mean embedding, in a foundation model such as CLIP, of its top-activating, CRP-cropped image patches. This semantic representation is then used for text-based neuron search, automatic labelling, model comparison, concept-alignment audits, and a set of human-interpretability measures (clarity, similarity, redundancy, polysemanticity). The method is evaluated on ImageNet classifiers (ResNet, VGG, ViT), on a binary melanoma classifier trained on ISIC 2019, and in a 218-participant user study, with code and a public demo provided.","tokens_in":52970,"tokens_out":7929,"duration_ms":77723,"significance":"If the central semantic-proxy assumption holds, SemanticLens is a practically valuable integration: it turns component-level interpretability into a searchable vector database and provides quantitative interpretability measures that correlate with human judgments at >0.74. The paper has real strengths: a label-faithfulness benchmark against INVERT and CLIP-Dissect, hyperparameter ablations, public code and demo, and an attempt to connect component semantics to both predictions and training data. The main risk is that every downstream capability inherits an unvalidated geometric assumption about the foundation model's embedding space, and the universality claim is supported only by vision experiments.","major_comments":[{"comment":"The core assumption that ϑ_k = (1/|E_k|) Σ F(x) over top-activating, cropped patches faithfully represents the neuron's concept is not tested in the direction the method uses it. The label-faithfulness benchmark in Supplementary Note D.4 computes ϕ(i,k) from the neuron's response to synthetic images of the candidate label, i.e., it tests label-to-neuron activation, not whether the mean embedding ϑ_k is semantically close to the label embedding in the foundation model's cosine geometry. All downstream capabilities—search (Eq. (4)), labelling, audit scores (Section 3.4), and interpretability measures (Eqs. (6)–(10))—are built on that cosine geometry. Please add a direct validation, for example measuring whether the rank order of cosine similarities between ϑ_k and a set of candidate label embeddings agrees with human judgments of the neuron's concept, or testing whether images whose embeddings are nearest to ϑ_k activate the neuron more than randomly selected images.","section":"§3.2, Eq. (2); §3.3, Eq. (3); Suppl. D.4, Eq. (1)"},{"comment":"For Vision Transformers, concept examples are obtained by approximating CRP attributions with up-sampled spatial maps, an approximation the paper itself acknowledges in §3.1. Since the ViT interpretability conclusions in Fig. 5b and Section 4.4 are based on these concept examples, the ViT part of the universality claim rests on an unvalidated attribution step. Please report a quantitative comparison of the approximate spatial maps against a ground-truth or established attribution method (e.g., CRP once available, or attention-based attributions) and show that the resulting concept examples do not change the conclusions.","section":"§3.1; Supplementary Note H"},{"comment":"The medical audit uses WhyLesionCLIP because generic CLIP is not trusted for dermatology, but the paper provides no evidence that WhyLesionCLIP's cosine geometry ranks dermoscopic concepts correctly. The ABCDE audit results (Fig. 4) and the subsequent pruning/retraining evaluation therefore depend on an unstated assumption about the medical foundation model. Please provide a domain-specific validation, such as zero-shot classification or retrieval accuracy on dermoscopic images, or a rating study with dermatologists comparing the top label embeddings for a set of lesions.","section":"§4.3; Supplementary Note F.2"},{"comment":"The 'universal' claim is broader than the evidence presented. The experiments cover CNN and ViT image classifiers and one binary skin-lesion model; there are no NLP, audio, or other modality results, and no sparse autoencoder or factorized-activation results, despite Section 2 stating that SemanticLens is also applicable to those component types. Either add demonstrations in other settings or restrict the abstract and central claims to vision components.","section":"Abstract; §1; §2"}],"minor_comments":[{"comment":"The symbol CRP is used both for the attribution framework and for the cropping operation; please define the cropping function explicitly, e.g., via a threshold on the normalized attribution map.","section":"§3.1, Eq. (1)"},{"comment":"The claim that |E_k|=30 yields converged ϑ_k is supported mainly for ResNet architectures; Supplementary Fig. D.6 shows a slight decline at 50 samples for ResNet and a monotone decline for VGG. Please state the convergence criterion and comment on the VGG behaviour.","section":"§3.2; Suppl. Fig. D.6"},{"comment":"The polysemanticity measure uses h=2 and 'an off-the-shelf clustering method'; please specify the clustering algorithm and report sensitivity to the number of clusters h.","section":"§3.5.3, Eq. (10)"},{"comment":"The same set-similarity metric is defined twice; keep one canonical definition to avoid version skew.","section":"§3.3, Eq. (5); Suppl. Note E, Eq. (1)"},{"comment":"The text states that ISIC 2019 includes 'nine different diagnostic categories' but then lists eight; please correct the count.","section":"Supplementary Note B.2.2"},{"comment":"The paper alternates between 'SEMANTIC LENS' and 'SemanticLens' (title, abstract, demo URL); please unify the notation.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is an integrative paper from the group that developed CRP; the novelty is more in the end-to-end workflow than in a new algorithmic component, but the workflow is genuinely useful for the XAI community. The main request should be for a direct validation of the semantic-proxy assumption; without it, the audit and search conclusions are conditional on the foundation model's cosine geometry. The 'universal' framing in the abstract should also be tempered unless evidence from other modalities or component types is added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing that matters: SemanticLens is a well-engineered integration of known interpretability pieces—CRP-based concept examples, CLIP embeddings, neuron labeling, and interpretability metrics—into a searchable audit workflow. That integration is genuinely useful, and the paper is refreshingly honest about most of its moving parts. But the central assumption that a mean CLIP embedding of a neuron's top-activating patches faithfully represents that neuron's concept is never directly tested, and the \"universal, no-human-input\" framing outruns the evidence.\n\nWhat the paper does well: it ships code and a demo, compares labeling against CLIP-Dissect and INVERT on three label sets with careful ablations, runs a 218-participant user study to ground the interpretability measures, and walks through a realistic medical audit with pruning and retraining. The ViT attribution limitation is disclosed rather than hidden. The citation pattern is fine; building on the authors' own CRP line is legitimate and properly credited.\n\nThe soft spots are concentrated in one load-bearing beam. The stress-test concern is on target: the label-faithfulness benchmark in Suppl. D.4 measures whether synthetic images of the assigned label activate the neuron, not whether the neuron's embedding is actually close to the label embedding. That is a label-to-activation test, not an embedding-validity test. So every downstream search, audit, and similarity score inherits an unverified geometric assumption. This is especially delicate for the medical case, where WhyLesionCLIP's cosine geometry for dermatology concepts is assumed without validation. The ViT attribution via upsampled spatial maps is another unvalidated approximation that could corrupt concept examples. Several audit results also lack error bars, and the \"universal\" and \"without human input\" claims overstate, since the concept sets and thresholds are user-defined.\n\nNone of this sinks the framework; it is a solid integration with a useful toolbox. But the claims should be scaled to the evidence. I would send this to peer review—the engineering is real and the benchmarks will be widely cited—but I would require either direct validation of the semantic-proxy assumption (e.g., human or ground-truth similarity judgments against CLIP cosine distances in the target domains) or a more cautious framing. As it stands, conditional accept, not reject.","headline":"Useful integration of CLIP-based neuron auditing, but the core semantic-proxy assumption is not directly validated and the 'universal' claim overreaches.","tokens_in":53496,"tokens_out":2902,"would_cite":true,"duration_ms":30388,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SemanticLens claims that every neuron of a vision model can be mapped into the shared embedding space of a multimodal foundation model such as CLIP, making a model's internal knowledge searchable by text and auditable against…","keywords":["Explainable AI","mechanistic interpretability","concept labelling","semantic embedding","CLIP","AI auditing","spurious correlations","interpretability measures"],"falsifier":"A direct test is to take a neuron labelled by SemanticLens for a concept such as 'palm tree' and measure the neuron's activation on images that contain the concept versus closely matched images that do not, using the same activation-response metric the paper's supplement applies to synthetic labels. If a substantial fraction of labels assigned by the embedding search score no better than random labels on this held-out test, then the cosine-similarity proxy in the foundation model is not preserving what the neuron actually encodes. The same test should be run with a deliberately domain-shifted foundation model: if audit conclusions change while the audited model itself is unchanged, the semantic proxy, not the model's knowledge, is determining the verdict.","tokens_in":52474,"feed_emoji":"🔍","tokens_out":7805,"duration_ms":73559,"temperature":0.7,"pith_summary":"SemanticLens sets out to make the inner components of a neural network as inspectable as the parts of a human-engineered machine. The method maps each neuron to a vector in the shared embedding space of a multimodal foundation model such as CLIP, by averaging the embeddings of the image patches that most strongly activate that neuron. Once components live in this semantic space, they can be searched with text prompts, labelled automatically, compared across models and layers, audited against user-defined valid and spurious concepts, and scored for clarity, redundancy, and polysemanticity. The paper demonstrates these operations on ImageNet classifiers and on a melanoma-detection model, where audits reveal that spurious concepts such as watermarks, palm trees, rulers, and band-aids are used in predictions, and that retraining with artefact augmentation removes their influence better than pruning does. If the approach works as claimed, it provides a scalable, human-free route from opaque model weights to component-level verification.","feed_headline":"SemanticLens turns every neuron into a text-searchable concept","feed_subtitle":"By embedding a model's components in a text-image space, bias audits become search queries instead of manual inspections.","key_machinery":"The central object is the semantic component embedding $\\vartheta_k = \\frac{1}{|E_k|}\\sum_{x\\in E_k} F(x)$, where $E_k$ are the concept examples of neuron $k$ (the top-$m$ activating patches, cropped to the relevant region via CRP) and $F$ is a multimodal foundation model such as CLIP. All operations---search, labelling, comparison, audit, and interpretability scoring---are cosine similarities in this space, with a 'null' embedding $\\vartheta_{\\langle\\rangle}$ subtracted to remove template and background effects. The interpretability measures are derived directly from the same embeddings: clarity is average pairwise similarity within a neuron's example set, redundancy is average maximal similarity to other neurons, and polysemanticity is one minus the clarity of clustered subsets. The link to decision-making is provided by CRP relevance scores $R$, which mark which components actually contributed to a prediction and allow attribution graphs and audit weighting.","core_discovery":"On its own terms, the paper's discovery is that a model's hidden knowledge can be faithfully represented by a small set of semantic vectors: for each component, take the $m$ most activating image patches from the dataset, crop them to the relevant regions with Concept Relevance Propagation (CRP), embed each patch in a foundation model's space, and average (Eq. (2)). The resulting vector $\\vartheta_k$ is then compared to text or image probes by cosine similarity with a null-embedding subtraction (Eqs. (3)-(4)). This representation supports the paper's claimed capabilities: search for concepts, automated labelling, model comparison via set similarity, and concept-alignment audits that split components into valid, spurious, and unexpected. On ImageNet, the audits expose neurons tied to Indian person, palm tree, and watermark for the class 'Ox' and find that no class among 26 examined is fully aligned with valid concepts; on the ISIC melanoma benchmark, they find spurious reliance on red skin, band-aid, and ruler concepts and show that retraining on augmented data, not pruning, materially reduces artefact sensitivity. The paper additionally introduces interpretability measures---clarity, redundancy, polysemanticity---that correlate above 0.74 with human judgments in user studies.","pith_inferences":["If the semantic-proxy assumption transfers, the same embedding recipe should apply to audio or video models by swapping the foundation model; the paper demonstrates only vision but its formal steps are modality-agnostic.","The audit conclusions inherit the foundation model's own concept organization: in domains where generic CLIP is not a trustworthy semantic expert, the method needs a domain-specific foundation model, exactly as the paper uses a dermatology-specific CLIP for ISIC.","A stronger validation than the supplied synthetic-label faithfulness test would compare SemanticLens labels against human concept annotations on real images, which the paper does not provide; that comparison would reveal whether the averaged-embedding proxy or the foundation model is the limiting factor.","The pruning-versus-retraining result suggests that localizing spurious components is not by itself a repair: a testable extension is to use the neuron-level search to generate targeted training augmentations for other failure modes, not just the artefacts studied here."],"forward_implications":["Searching for a bias, artefact, or required concept in a trained vision model reduces to a text query over neuron embeddings, so audits that previously required manual inspection can be automated.","Models of different architectures, depths, and training recipes can be compared quantitatively on the same semantic space, revealing which concepts are shared, unique, or transformed by longer training.","Because each neuron embedding is linked to its activating training patches, detected spurious concepts can be traced to the specific data responsible, enabling targeted data cleaning.","The interpretability measures give a cheap, human-free proxy for how understandable a model's components are, so architecture and regularization choices such as dropout or sparsity can be optimized for interpretability at scale.","In the medical case, the audit framework can check whether a model follows domain rules such as the ABCDE rule and can identify which spurious cues, if removed by retraining rather than pruning, genuinely change artefact sensitivity."],"supporting_citations":[{"why":"Supplies the multimodal foundation model whose shared embedding space and cosine-similarity design SemanticLens adopts.","marker":"[23]"},{"why":"Provides Concept Relevance Propagation, used to crop concept examples and to compute component relevance scores R linking components to predictions.","marker":"[15]"},{"why":"Establishes Network Dissection, the labelling-by-concept-set baseline and the neuron-count comparison used for model comparison.","marker":"[14]"},{"why":"CLIP-Dissect is the closest foundation-model-based labelling method; SemanticLens is benchmarked against it on faithfulness.","marker":"[35]"},{"why":"Shows that averaging embeddings of strongly activating images is a meaningful pooling strategy; Eq. (2) adopts this proposal.","marker":"[48]"},{"why":"Supplies the model test that separates outputs on spurious-feature images via AUC, used in the audit step.","marker":"[62]"},{"why":"Provides the domain-specific CLIP for the dermatology experiments, used as the semantic expert for ISIC.","marker":"[68]"},{"why":"MobileCLIP is the foundation model used throughout the ImageNet search, labelling, and audit experiments.","marker":"[66]"},{"why":"The ISIC 2019 dataset is the medical benchmark on which the melanoma audit and correction are demonstrated.","marker":"[65]"},{"why":"Defines the ABCDE rule, the expected-concept set against which the melanoma model's reasoning is audited.","marker":"[78]"}],"fun_headline_variants":["SemanticLens: turn every neuron into a searchable concept","Query AI neurons by meaning with SemanticLens","Explain any neural net: SemanticLens maps neurons to concepts","Audit AI decisions at neuron level via SemanticLens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything SemanticLens reports about a model's concepts rests on one premise: that cosine similarity in the chosen foundation model's embedding space is a faithful proxy for semantic similarity of the neuron's concept, and that averaging the top-activating cropped patches preserves that concept rather than averaging it away.","fun_headline_variants_meta":{"raw":{"variants":["SemanticLens: turn every neuron into a searchable concept","Query AI neurons by meaning with SemanticLens","Explain any neural net: SemanticLens maps neurons to concepts","Audit AI decisions at neuron level via SemanticLens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000458,"raw_usage":{"total_tokens":2346,"prompt_tokens":1045,"completion_tokens":1301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":1233}},"tokens_in":661,"tokens_out":1301,"duration_ms":10227,"temperature":1.0,"reasoning_tokens":1233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:12:42.665303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to take a neuron labelled by SemanticLens for a concept such as 'palm tree' and measure the neuron's activation on images that contain the concept versus closely matched images that do not, using the same activation-response metric the paper's supplement applies to synthetic labels. If a substantial fraction of labels assigned by the embedding search score no better than random labels on this held-out test, then the cosine-similarity proxy in the foundation model is not preserving what the neuron actually encodes. The same test should be run with a deliberately domain-shifted foundation model: if audit conclusions change while the audited model itself is unchanged, the semantic proxy, not the model's knowledge, is determining the verdict.","supporting_citations":[{"cited_title":"Spurious features everywhere-large-scale detection of harmful spurious features in imagenet","cited_arxiv_id":null,"evidence_quote":"Supplies the model test that separates outputs on spurious-feature images via AUC, used in the audit step."},{"cited_title":"A textbook remedy for domain shifts: Knowledge priors for medical image analysis","cited_arxiv_id":null,"evidence_quote":"Provides the domain-specific CLIP for the dermatology experiments, used as the semantic expert for ISIC."},{"cited_title":"Mobileclip: Fast image-text models through multi-modal reinforced training","cited_arxiv_id":null,"evidence_quote":"MobileCLIP is the foundation model used throughout the ImageNet search, labelling, and audit experiments."},{"cited_title":"The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions","cited_arxiv_id":null,"evidence_quote":"The ISIC 2019 dataset is the medical benchmark on which the melanoma audit and correction are demonstrated."},{"cited_title":"Clinical ABCDE rule for early melanoma detection","cited_arxiv_id":null,"evidence_quote":"Defines the ABCDE rule, the expected-concept set against which the melanoma model's reasoning is audited."}],"review_version":1}