{"id":"d7f8b694-dc43-42ab-a346-26b93bc9d213","arxiv_id":"2508.14562","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A label-free concept bottleneck model using per-concept prototypes aligned by CLIP to localize concept predictions to the correct image regions.","lead":"This paper adds a prototype-learning step to label-free concept bottleneck models so that each concept, like 'black beak', is predicted from the image region where it actually appears. The result is a more reliable visual explanation for an image classifier's decisions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The prototype-concept alignment is supervised by CLIP patch-text similarity; if CLIP's local semantics are biased for fine-grained attributes, LCBM's localized GradCAM maps may reflect CLIP's spatial priors rather than human-defined concept regions, undermining the central localization claim.","rationale":"The reader's weakest assumption already identifies CLIP similarity as the weak point, and I agree. I did not find a separate internal inconsistency. The method's whole point is to use CLIP to teach prototypes where concepts are; the evaluation uses human labels; the gap between the two is the risk. The paper's own limitation statement makes the concern concrete rather than speculative. A single correlational check on CUB would settle it. Because the supplied text omits Section 3 and the quantitative tables, I cannot go further than the reader; the appropriate verdict remains UNVERDICTED until the missing evidence and the CLIP-agreement check are available.","tokens_in":7724,"tokens_out":4498,"duration_ms":58023,"concrete_test":"On CUB-200-2011, for each concept in LCBM's evaluation set, compute a zero-shot CLIP localization map: patch-level cosine similarity between the image patch embeddings and the concept text, thresholded as in §4.2, and measure its hit rate/IoU against the human-annotated point/bbox used for LCBM's evaluation. Split concepts by median CLIP-GT agreement. If LCBM's per-concept localization is significantly worse on the low-agreement half, or if its GradCAM maxima on those concepts align with CLIP's wrong regions rather than GT, the CLIP-alignment premise is the bottleneck. If LCBM remains accurate even where zero-shot CLIP maps disagree with human annotations, the concern is resolved and the method learns beyond CLIP.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: LCBM improves spatial localization of concept predictions while preserving classification (Abstract; §6). The mechanism is a per-concept prototype trained so that local image patches similar to the prototype are CLIP-similar to the concept text (Abstract; §3). Thus CLIP similarity is not a diagnostic; it is the training target for where each concept lives. Localization is then evaluated against human-annotated points/boxes (§4.2). This creates a load-bearing empirical premise: CLIP's patch-level text-image similarity must agree with human-annotated locations for the tested fine-grained concepts (e.g., 'barred black feather', 'black beak'). If CLIP is biased or weakly calibrated at the patch level, the prototype is attracted to the region CLIP associates with the phrase, not the region a human annotator labels. GradCAM maps will be concentrated (hence 'localized' in a weak sense), but they will be localized around the wrong structure; the reported agreement with human annotations would then be coincidental or limited to concepts where CLIP's spatial prior happens to match. The authors explicitly concede in §5 that LCBM 'inherits inherent biases from CLIP' and that CLIP is 'not optimized for identifying detailed concepts within an image.' That is a direct admission that the training signal for the paper's core contribution is exactly what the central claim requires to be reliable. The visible portions of the manuscript do not contain a quantitative check that CLIP patch-similarity maps correlate with the human-annotated regions used in §4.2, nor the localization tables that would show per-concept performance. Without such a check, the improvement could be an artifact of optimizing toward CLIP's biases rather than toward human-meaningful concept localization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LCBM (Locality-aware Concept Bottleneck Model), a label-free CBM in which each concept is assigned a prototype that is trained to encode local image features associated with that concept. Prototype learning is guided by CLIP similarity scores between local patches and concept text, and the prototypes are then used to predict concept presence from local regions. The central claim is that LCBM improves the spatial localization of concept predictions—measured via GradCAM alignment with human-annotated points/boxes and via a concept-removal protocol—while maintaining classification performance comparable to prior label-free CBMs. The supplied manuscript text includes the abstract, introduction, related work, a qualitative results section, the limitation section, and the conclusion, but Sections 3 (method) and the quantitative portions of Section 4 (experiments) are not visible in the provided version. The limitation section explicitly concedes that LCBM inherits biases from CLIP and that CLIP is not optimized for identifying detailed concepts within an image.","tokens_in":8046,"tokens_out":3899,"duration_ms":50103,"significance":"If the central claim is substantiated, the work addresses an important and practical gap: label-free CBMs often produce concept explanations that attend to visually irrelevant regions. The prototype-based locality mechanism is a sensible and potentially effective approach, and the two proposed evaluation protocols (GradCAM vs. human annotations, and concept removal) are appropriate ways to assess localization fidelity. The paper also provides qualitative evidence and an appendix with additional examples. However, the significance is conditional: the current supplied text lacks the quantitative experimental tables needed to verify the 'significantly improves' claim, and the method's reliance on CLIP patch-text similarity as the training target creates a risk of circularity when evaluation also uses CLIP-derived concepts. These issues must be resolved before the contribution can be assessed reliably.","major_comments":[{"comment":"The core training signal for prototype localization is CLIP similarity between local image patches and concept text. The paper's own Limitation section (Section 5) admits that LCBM 'inherits inherent biases from CLIP' and that 'CLIP is not optimized for identifying detailed concepts within an image.' This is load-bearing because the central claim—localization improvement—is evaluated against human-annotated points/boxes in Section 4.2. If CLIP's patch-level similarity is biased or weakly calibrated for fine-grained concepts such as 'barred black feather' or 'black beak,' the prototype will be attracted to the regions CLIP associates with the phrase rather than the regions a human annotator labels. The manuscript provides no quantitative check that CLIP patch-level similarity agrees with human spatial annotations for the tested concepts. I would ask for an analysis of this agreement, or a","section":"Section 5 (Limitation) and Section 3 (method, as described in the Abstract)"},{"comment":"The Conclusion states that LCBM 'significantly improves the localization ability of concept predictions,' but the supplied manuscript does not contain the quantitative experimental tables that would support this. The only visible evidence is the qualitative Figure 6 and selected appendix examples. Without exact metrics—for example, localization point-hit rates or IoU, GradCAM agreement with human points/boxes, concept-removal score decreases, classification accuracy, and concept-presence F1—it is impossible to verify the central claim or compare with baselines such as LfCBM and VLG-CBM. The manuscript needs to report these numbers, including error bars and significance tests, and to specify how many concepts and images are used.","section":"Section 4 and Conclusion (quantitative results missing in supplied text)"},{"comment":"There is a potential circularity in the evaluation. The concept vocabulary and concept-presence pseudo-labels are derived from CLIP (or from a language model in prior label-free CBMs), and LCBM is trained to align prototypes with CLIP similarity. GradCAM maps are then computed from the concept scores. Consequently, evaluating concept-presence accuracy and localization against CLIP-derived pseudo-labels partly measures how well the model reproduces CLIP's own outputs, rather than how well it captures human-aligned concepts. The manuscript mentions comparison to human-annotated points/boxes for localization, which is the right direction, but it should also report the correlation between CLIP pseudo-labels and human labels, and separate the CLIP-supervised components from the claims validated against human ground truth.","section":"Evaluation protocol (Section 4.2, as referenced in the text)"},{"comment":"The paper claims 'to the best of our knowledge, this is the first work to perform a quantitative analysis of concept-level localization,' yet the preceding paragraph discusses SALF-CBM, which 'aims to resolve the issue of concept localization.' The distinction drawn—that SALF-CBM 'cannot determine which concept is most likely to be present in a specific region'—needs to be made concrete with a quantitative comparison against SALF-CBM. If SALF-CBM already performs localization and is evaluated quantitatively, the claimed novelty is weakened. Please clarify what exactly is new and include SALF-CBM in the experimental comparison.","section":"Section 2 (Related Works, SALF-CBM discussion)"}],"minor_comments":[{"comment":"The description of the GradCAM maps says 'thresholded GradCAM maps' but does not specify the threshold value or how the color scale is defined. Please provide this information so the qualitative results are reproducible.","section":"Figure 6 (Qualitative Analysis, Section 4.6)"},{"comment":"Figures 22–24 in the appendix display quantities such as $\\chi$, $\\hat{X}$, and $\\hat{P}$ without explanation in the visible text. If these are part of the local-explanation visualization, the notation should be defined and connected to the concept contribution bar graphs.","section":"Appendix A.5.2 (Local Explanation Examples)"},{"comment":"Several references (e.g., [10], [19], [27], [29], [33]) appear only in the appendix or future-work discussion; the main text should cite them where they are first relevant. Also, the supplied text omits Sections 3 and the experimental tables, making the manuscript incomplete as submitted; please ensure the full version is used in review.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The supplied version of the manuscript is missing Section 3 and the quantitative experimental tables, which makes it impossible to verify the central claim from the provided text. The most serious substantive concern is the circularity arising from training prototype localization with CLIP patch-text similarity and then evaluating against CLIP-derived concepts; the Limitation section's admission that CLIP is biased for fine-grained concepts reinforces this. The paper's idea is plausible and the qualitative results are suggestive, but the authors need to supply the missing quantitative material and address the CLIP-agreement question. I would recommend major revision rather than rejection, assuming the full version contains the missing experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before anything else: the problem it targets is real, but the version I read never lets me check the actual claim. The text jumps from related work to qualitative results, and the quantitative localization tables and the full method section (Section 3) are missing. So the headline result—improved localization without hurting classification—is running on trust, not evidence.\n\nWhat is actually new: assigning one learned prototype per concept and using CLIP patch-text similarity to align each prototype to a local region, in a label-free CBM setting. That combination is not in the cited ProtoCBM or SALF-CBM, and the authors are upfront that they are the first to do quantitative concept-level localization. The evaluation protocols they describe are sensible: GradCAM maps compared against human-annotated points/boxes, and a concept-removal test to see if the model relies on the localized region. They also explicitly acknowledge that CLIP carries biases and is not built for fine-grained concepts. That is honest.\n\nThe soft spot is not small, though. The prototype alignment loss is trained on CLIP similarity between local patches and concept text. That means CLIP's notion of where a concept lives is the training target, not a diagnostic. If CLIP's patch-level semantics are biased or poorly calibrated for fine-grained attributes like \"barred black feather\" or \"black beak,\" the prototypes will be pulled toward whatever CLIP associates with the phrase, and the GradCAM maps will be concentrated around those possibly wrong regions. The localization evaluation against human annotations would then be the only thing that could catch this, but we don't see per-concept localization scores or any sanity check that CLIP similarity maps actually correlate with the human-annotated regions. The authors' own limitation statement—\"CLIP is not optimized for identifying detailed concepts within an image\"—makes this a load-bearing concern, not a minor one. There is also secondary circularity for concept-presence accuracy, since the pseudo-labels come from CLIP as well.\n\nFor a reader working on interpretable ML or label-free CBMs, this is worth knowing about as a plausible incremental advance. But I would not cite it on the basis of the visible text, because the central number is missing. It deserves a serious referee: the problem is real, the method is coherent, and the qualitative examples look encouraging. The referee should demand the full localization tables, error bars, and preferably an explicit check—e.g., per-concept correlation between CLIP patch similarity and human-annotated regions—before believing that the improvement is more than the model learning CLIP's spatial priors.","headline":"The paper plausibly improves concept localization in label-free CBMs, but the visible text omits the method and all quantitative results, so the central claim is unverifiable and the CLIP-based training signal raises a real circularity concern.","tokens_in":8591,"tokens_out":1711,"would_cite":false,"duration_ms":21523,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LCBM claims that per-concept prototype learning, supervised by CLIP region similarity, makes label-free concept bottleneck models localize concepts correctly without losing classification accuracy.","keywords":["concept bottleneck models","label-free concept bottleneck","prototype learning","concept localization","interpretable machine learning","CLIP","GradCAM","fine-grained classification"],"falsifier":"Take a synthetic dataset with known part locations and a concept whose CLIP text embedding is confounded (e.g., \"white belly\" on an animal with a white background). Train LCBM on images where the confound is decorrelated, then evaluate whether GradCAM peaks at the belly or the background; if it tracks CLIP's highest-scoring region rather than the annotated belly, the prototype mechanism is just relaying CLIP's patch-level errors. A stronger version uses human part masks during training as an oracle and asks whether LCBM approaches oracle localization; if it does not, the gap is the cost of tru","tokens_in":7636,"feed_emoji":"🎯","tokens_out":8581,"duration_ms":89082,"temperature":0.7,"pith_summary":"Label-free concept bottleneck models predict human-readable concepts from images but often explain a concept like \"barred black feather\" by attending to the wrong region. LCBM is an attempt to fix that by giving each concept a learnable prototype that encodes the concept's prototypical local appearance, with CLIP similarity scores keeping the prototype semantically tied to the concept's text. Concept presence is then read from similarity between image regions and prototypes, so each prediction carries a spatial answer to \"where is this concept?\". The paper reports improved agreement between concept-level GradCAM maps and human-annotated regions on CUB, ImageNet-animal, and Stanford Cars, and shows that erasing those regions lowers concept presence scores while classification accuracy stays comparable.","feed_headline":"Prototypes pin concept explanations to the right pixels","feed_subtitle":"A per-concept prototype, aligned by CLIP, makes concept explanations point at the object part they name—without losing accuracy.","key_machinery":"Each concept is assigned one prototype vector, a learnable embedding meant to represent the prototypical local appearance of that concept. Two additional losses drive the learning: one auxiliary classification loss makes the prototype encode features shared by visually similar local regions, and one alignment loss uses CLIP similarity scores to keep the prototype tied to its concept's text. At inference, similarity between an image's local features and a prototype produces a spatial relevance map, and the concept presence score is read from the regions that match the prototype. The prototype is the load-bearing object: it turns a global concept score into a \"this looks like that\" region, and","core_discovery":"The central claim is that label-free concept bottleneck models misbehave spatially—they predict a concept like \"barred black feather\" while their activation lands on the wrong part of the image—and that this is fixable without dense concept annotations. The proposed mechanism is to learn one prototype per concept so that concept prediction is mediated by local-region-to-prototype similarity. Prototypes are trained with an auxiliary classification objective and with CLIP similarity between prototype features and concept text, which teaches each prototype which local appearance it stands for. On CUB-200-2011, ImageNet-animal, and Stanford Cars, LCBM's concept-level GradCAM maps fall inside hum","pith_inferences":["A natural next step, not explored in the paper, is treating the learned prototype-region similarity maps as soft part segmentations; if they are as well-localized as the GradCAM results suggest, they could seed part discovery on datasets that lack part annotations.","Because CLIP is used only as a teacher for the alignment loss, the same architecture could be retrained with a different vision-language teacher; comparing localization quality across teachers would quantify how much of the gain comes from the prototype mechanism versus from CLIP's own spatial knowledge.","The paper's localization metric (GradCAM overlap with annotated regions) can be gamed by models that produce peaked heatmaps for the wrong reason; the removal experiment is the stronger evidence, and a direct comparison of both metrics across baselines would separate true causal use from saliency artifacts.","For concepts where CLIP is known to be systematically biased (e.g., color adjectives that actually describe the background), LCBM would inherit that bias; testing on a bias-controlled set would clarify how far the locality guarantee extends."],"forward_implications":["Concept predictions in label-free CBMs acquire a usable spatial reference: the region of highest prototype similarity is where the concept is claimed to be, so a user can verify each concept against the image instead of trusting a score.","Because classification accuracy remains comparable, LCBM can replace existing label-free CBM backbones as a drop-in module for interpretable fine-grained classification.","The removal experiment gives evidence that the localized region is actually used: deleting the region decreases the concept presence score, tying explanations to the model's computation rather than only to post-hoc saliency.","No dense concept annotations are needed to obtain this localization, which matters for scaling interpretable models to new datasets and domains."],"supporting_citations":[{"why":"Defines the label-free concept bottleneck formulation that LCBM inherits and improves on.","marker":"[25]"},{"why":"Supplies the vision-language teacher whose similarity scores align prototypes to concept texts.","marker":"[26]"},{"why":"The VLG-CBM baseline that uses bounding-box supervision; LCBM compares localization against it.","marker":"[34]"},{"why":"Provides the GradCAM protocol used to measure where concept predictions activate.","marker":"[30]"},{"why":"Identifies the concept-trustworthiness and localization failure in CBMs that motivates LCBM.","marker":"[9]"},{"why":"A concurrent localization-focused CBM; LCBM contrasts its richer prototype-to-region relationship.","marker":"[2]"},{"why":"Supplies the prototype-based interpretability paradigm (this looks like that) adopted for per-concept prototypes.","marker":"[3]"},{"why":"A label-free CBM baseline (Language in a Bottle) used in experiments.","marker":"[38]"},{"why":"Supplies the CUB benchmark with fine-grained concept annotations used for localization evaluation.","marker":"[35]"}],"fun_headline_variants":["Per-concept prototypes lock explanations to true pixels","Concept bottleneck models that look where they label","Locality-aware concepts: right region, same accuracy","CLIP-aligned prototypes keep concept heatmaps on target","Stop concept mislocalization without dense annotations"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The entire localization improvement rests on trusting CLIP's similarity scores on small image regions as ground truth for whether a concept is present there; if those scores are unreliable or biased for fine-grained concepts, the prototypes learn to attend to the wrong places and the reported localization is an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Per-concept prototypes lock explanations to true pixels","Concept bottleneck models that look where they label","Locality-aware concepts: right region, same accuracy","CLIP-aligned prototypes keep concept heatmaps on target","Stop concept mislocalization without dense annotations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1019,"prompt_tokens":709,"completion_tokens":310,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":453,"completion_tokens_details":{"reasoning_tokens":239}},"tokens_in":453,"tokens_out":310,"duration_ms":4707,"temperature":1.0,"reasoning_tokens":239,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:25:31.683689+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic dataset with known part locations and a concept whose CLIP text embedding is confounded (e.g., \"white belly\" on an animal with a white background). Train LCBM on images where the confound is decorrelated, then evaluate whether GradCAM peaks at the belly or the background; if it tracks CLIP's highest-scoring region rather than the annotated belly, the prototype mechanism is just relaying CLIP's patch-level errors. A stronger version uses human part masks during training as an oracle and asks whether LCBM approaches oracle localization; if it does not, the gap is the cost of tru","supporting_citations":[{"cited_title":"Nguyen, and Tsui- Wei Weng","cited_arxiv_id":null,"evidence_quote":"Defines the label-free concept bottleneck formulation that LCBM inherits and improves on."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the vision-language teacher whose similarity scores align prototypes to concept texts."},{"cited_title":"On the concept trustworthiness in concept bottleneck models","cited_arxiv_id":null,"evidence_quote":"Identifies the concept-trustworthiness and localization failure in CBMs that motivates LCBM."},{"cited_title":"Show and tell: Visually explainable deep neural nets via spatially-aware concept bot- tleneck models","cited_arxiv_id":null,"evidence_quote":"A concurrent localization-focused CBM; LCBM contrasts its richer prototype-to-region relationship."},{"cited_title":"This looks like that: deep learn- ing for interpretable image recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the prototype-based interpretability paradigm (this looks like that) adopted for per-concept prototypes."},{"cited_title":"Language in a bottle: Language model guided concept bottlenecks for interpretable image classification","cited_arxiv_id":null,"evidence_quote":"A label-free CBM baseline (Language in a Bottle) used in experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CUB benchmark with fine-grained concept annotations used for localization evaluation."}],"review_version":1}