{"id":"e03ec8f0-0d44-49d9-a30a-8032f386fd71","arxiv_id":"2501.09333","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Learning class-specific prompts on a frozen ViT yields attention maps that localize the discriminative traits used in fine-grained image classification.","lead":"Prompt-CAM adds one learnable token per class to a frozen pre-trained vision transformer and uses the attention maps of the winning class token to highlight the subtle traits that separate similar species or objects. The method is a near drop-in change to Visual Prompt Tuning, so it may give biologists a cheap way to discover traits automatically.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 7's guarantee is broken by the residual/prompt-identity path: a class-specific prompt can drive its logit without attending to trait patches, so 'must attend to traits' is not established for the full ViT.","rationale":"The paper's empirical contributions are real: broad evaluation across 13 datasets, a human study, trait-manipulation counterfactual, and taxonomy-key experiments. These provide support for the method as a useful trait-visualization tool. However, the load-bearing theoretical justification for why attention maps must reveal traits (abstract, Section 2.5) is not valid for the full architecture. The residual connection and prompt-token identity provide an alternative channel for class-discriminative logits; the simplified Eq. 7 explicitly assumes these away. This is precisely the weakest assumption flagged by the reader, and it is more than a technicality—it is a concrete failure mode that can be tested. The proposed intervention would settle whether patch attention is causally necessary. Because the paper currently lacks such a causal test (the greedy head-blurring is illustrative and on selected species), the verdict should remain CONDITIONAL: accept the method's empirical promise but require authors to either provide the uniform-attention intervention results or weaken the 'must attend' claim. I do not see grounds to reject outright, since the empirical trait localization could still be sound; nor to accept as-is, since the central claim is stronger than the proof supports.","tokens_in":21489,"tokens_out":6775,"duration_ms":74951,"concrete_test":"Intervention on CUB with the trained DINO PROMPT-CAM-DEEP: for every correctly classified validation image, recompute logits after replacing each true-class prompt's patch-attention weights in the final layer (Eq. 5) by uniform over image patches, while leaving residual connections, MLP, CLS attention, and prompt-to-prompt attention unchanged. Measure top-1 accuracy and mean drop in true-class logit. If accuracy stays far above chance (or the logit drop is small), the model classifies via prompt identity/self-attention rather than patch attention, falsifying the Eq. 7-based guarantee; also run the symmetric map-swap (feed class c' attention map to class c and see if c' logit rises) to confirm whether attention maps are causal. Report these numbers across all species, not selected examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Section 2.5's Eq. 7, but Eq. 7 is derived for a single-head linear attention layer with no residual connection, no MLP, and explicitly 'no self-attention among the prompts.' In the actual PROMPT-CAM-DEEP forward pass, the final-layer output z_c for class c is not sum_j alpha_c[j] v_j; the prompt's own input token p_c survives via the residual connection, and the prompt also attends to the [CLS] token and to the other C-1 prompts. Thus w^T z_c can encode class identity through p_c and prompt-to-prompt attention alone, even if every patch attention map alpha_c is uniform. The 'no spatial resolution' argument collapses because it assumes the only route to class-discriminative scores is the attention-weighted sum of shared patch values. Moreover, even if attention were the only route, Eq. 7 forces only that alpha_c differ across classes; it does not force attention onto unique traits rather than background, pose, or other spurious patches, since those can also yield class-predictive w^T v_j. So the abstract's 'must attend to unique image patches... (i.e., traits)' is an overclaim; what is shown is at most that class-specific attention is one sufficient mechanism under a stripped model. This is the load-bearing gap because the paper's interpretability promise is specifically that attention maps point to traits, not merely that they are class-specific and causally used.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Prompt-CAM, an interpretability method for frozen Vision Transformers in fine-grained classification. It adds C learnable class-specific prompt tokens to the last (or first) transformer layer, computes logits as inner products of the prompt outputs with a single shared vector w, and trains only the prompts and w with cross-entropy. The core claim is that, because the classifier is shared and has no per-class fully connected layer, the true-class prompt must attend to image patches unique to that class (\"traits\"), so the resulting multi-head attention maps provide faithful trait localization. The paper validates this with insertion/deletion faithfulness scores, a human trait-recognition study, comparisons to other interpretable and explainable methods, a counterfactual trait-removal experiment, and a preliminary hierarchical taxonomy application, across 13 fine-grained datasets and multiple ViT backbones.","tokens_in":21805,"tokens_out":5739,"duration_ms":58069,"significance":"If the central claim holds, Prompt-CAM is a genuinely attractive method: it is simple, training-efficient, requires only frozen pre-trained encoders, and is supported by code and a demo. The empirical scope is unusually broad for an interpretability paper, and the counterfactual red-wing-patch experiment plus the human study are steps in the right direction. The method also connects to an active area of parameter-efficient transfer learning and to biological trait discovery. However, the paper's advertised guarantee that attention maps 'must' point to traits is not established by the theoretical argument in Section 2.5, and the trait-validation evidence is currently too thin to carry the full weight of the claim. With a careful revision of the theoretical claim and additional systematic trait evaluations, this could be a valuable contribution.","major_comments":[{"comment":"The central claim in the Abstract that the true-class prompt \"must attend to unique image patches ... (i.e., traits)\" is proven only in a stripped-down model that, as the text itself states, omits layer normalization, residual connections, MLP blocks, and self-attention among prompts. In the actual PROMPT-CAM-DEEP forward pass (Eq. (3) plus the final layer LN), the output z_c for class c is not equal to sum_j alpha_c[j] v_j: the prompt's own token survives via the residual connection, it attends to the [CLS] token and to the other C-1 prompts, and the MLP block operates on the residual stream. Therefore w^T z_c can encode class identity through the prompt's own pathway even if every patch attention map alpha_c is uniform, and the \"no spatial resolution\" argument in Section 2.5 collapses. What is shown is that class-specific attention is a sufficient mechanism in a simplified model, not a necessary one in the full architecture. Please either soften the \"must\" claim to an \"encourages\" statement, or provide an empirical test (e.g., setting all patch attention weights to uniform while keeping the prompt's residual and other-token contributions intact and comparing logits) demonstrating that the trait-attention route is the one the trained model actually uses for classification.","section":"2.5, Eq. (7)"},{"comment":"Even if attention were the only route to class-discriminative scores, Eq. (7) forces only that alpha_c differ across classes so that the shared inner products w^T v_j are combined class-specifically. It does not force the attended patches to be unique to the class or to correspond to semantically meaningful traits: background, pose, and other class-predictive patches can equally generate class-specific scores. The paper's definition of traits as class-unique patches is thereby made true by construction only if the optimized attention happens to choose such patches, which is not proven. The human study (Supplementary Appendix F) covers only four species and four images per method, and the counterfactual in Figure 8 is a single image, so the evidence for trait localization is anecdotal relative to the breadth of the claim. Please evaluate quantitatively against an external trait or part annotation, e.g., the CUB part attributes the paper already uses or the Fish-Vista trait annotations it cites, reporting hit rates or IoU for the top-ranked attention heads over a large set of images.","section":"2.5, Eq. (7); 3.2 human study"},{"comment":"In the greedy head-blurring algorithm, replacing a head's attention vector with the uniform distribution and recomputing s[c] does not remove the trait from the image; the residual connection, the MLP, and the prompt's own token still contribute to z_c even when the patch attention is uniform. As a result, the logits may remain high even if no patch is effectively attended to, and the ranking of \"least important\" heads may under-estimate the number of heads that are actually necessary. Please report the relationship between the greedy head rankings and the human/annotation-based trait importance, and state how often the prediction flips when only the retained heads are left active, so the reader can judge whether the retained heads are indeed sufficient.","section":"2.3, D. Inner Workings of Visualization"}],"minor_comments":[{"comment":"The method name appears in multiple inconsistent forms, including \"P ROMPT -CAM\", \"PROMPT-CAM\", and \"Prompt-CAM\"; please standardize the typography.","section":"Throughout"},{"comment":"The evaluation dataset list includes \"Stanford Cars\" twice; please remove the duplicate entry.","section":"1, Introduction"},{"comment":"The DINOv2 learning-rate description is duplicated verbatim; the second block likely refers to BioCLIP and should state the correct learning rate for that backbone.","section":"E.1, Supplementary"},{"comment":"The uniform-distribution notation \"1/M 1\" is ambiguous; please introduce a symbol such as u for the uniform vector and write alpha_c,r <- u.","section":"2.3, Eq. (5)"},{"comment":"Accuracy is reported for only four of the thirteen datasets in the main paper; consider moving the fuller accuracy table (Supplementary Table 5) to the main text, since the Abstract emphasizes evaluation on more than a dozen datasets.","section":"Table 2"},{"comment":"Reference [14] is cited for attention roll-out, but the original attention roll-out method is Abnar and Zuidema [1]; please reconcile the two citations so that credit is given to the original source.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical contribution from a group that has produced related work (INTR), and the method is simple enough to be widely adopted if the trait-localization claims hold up. My main concern is the gap between the advertised theoretical guarantee and what Eq. (7) actually proves; this is fixable by rewording the claim and adding targeted experiments (e.g., a uniform-attention probe, and large-scale trait-annotation evaluation). The paper should also be careful in the related-work discussion to state plainly how much of the design is inherited from INTR, since the shared-head versus per-class-head difference is the key novelty. I would not reject, but the revision should be substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thought you'd want a quick read on Prompt-CAM. The punchline: the method is simple and it delivers. Class-specific prompts on a frozen DINO-style ViT, with a shared scoring vector, produce attention maps that genuinely seem to point at discriminative traits—yellow bellies, wing patches, black crowns—across a dozen datasets. The greedy head-blurring to rank traits per species is a neat addition, and the counterfactual where removing a red wing patch flips a Red-winged Blackbird to a Boat-tailed Grackle is the strongest evidence in the paper. You can train this in a few lines on top of VPT, and the code is out. That is a real, useful contribution to fine-grained interpretability.\n\nThe soft spot is the theory, and it's not minor. Equation 7 is derived for a single-head linear attention layer with no residual, no MLP, and no self-attention among prompts. The actual forward pass has all of those. The prompt's own input token survives via the residual connection, and the prompt also attends to CLS and the other C-1 prompts. So a class-specific prompt could drive its logit through a pathway that has nothing to do with attending to unique trait patches. Even within the patch attention, Eq. 7 only forces alpha_c to differ across classes; it doesn't force the differences to be traits rather than background or pose. The abstract's 'must attend to unique image patches (i.e., traits)' is an overclaim. The paper actually flags the simplification in Section 2.5, but then proceeds as if the conclusion survived the simplification. This needs a rewrite, not just a footnote.\n\nThe empirical side is decent but not bulletproof: the faithfulness numbers in Table 1 have no error bars or significance tests, the human study is 35 participants on 4 species, and the comparisons to ProtoPFormer, INTR etc. are mostly qualitative. None of these are fatal; they are tighten-up items. The broad dataset coverage and the taxonomy-key experiment do most of the work in convincing me the method generalizes.\n\nPer the stress-test question: yes, the skeptic's concern lands on the theoretical claim, but it doesn't sink the method. What survives is an empirically validated tool for trait localization. Who should read this: anyone doing fine-grained recognition in biology, and the interpretability crowd that wants a cheap alternative to prototype methods. I'd send it to review—the method is solid and broadly evaluated—but the authors should be pushed to either prove a real guarantee for the full architecture or state plainly that 'must attend' is an intuition, not a theorem. If they soften the language and add a few error bars, this is a strong paper.","headline":"Prompt-CAM is a cheap, broadly evaluated trait-localization method that mostly works, but its 'must attend' theoretical claim is overreaching and should be softened.","tokens_in":22357,"tokens_out":3102,"would_cite":true,"duration_ms":57115,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a pre-trained, frozen Vision Transformer can be made interpretable for fine-grained analysis by learning one class-specific prompt per class and scoring each with a single shared vector.","keywords":["Prompt-CAM","vision transformer interpretability","fine-grained visual categorization","class-specific prompt tokens","attention visualization","trait localization","visual prompt tuning"],"falsifier":"A decisive test would be to compute Prompt-CAM attention on images where the class label can be predicted from background or global shape alone: if deleting or masking the highest-attended trait patches leaves the predicted class unchanged, while masked-out low-attended patches change it, the attention maps are not showing the traits that drive classification.","tokens_in":21302,"feed_emoji":"🐦","tokens_out":5092,"duration_ms":49764,"temperature":0.7,"pith_summary":"The paper establishes a simple recipe for extracting fine-grained explanations from a pre-trained Vision Transformer: attach one learnable prompt token per class, freeze the backbone, and score each class with a single shared vector. Because the classifier is shared, class-discriminative information cannot be stored in patch features; it must be encoded in where each class's prompt attends. The resulting multi-head attention maps therefore localize the diagnostic traits of each class—the red wing patch of a Red-winged Blackbird, the yellow breast of a Scott Oriole—rather than blurring over the whole object. This makes the method nearly a free lunch on top of Visual Prompt Tuning, and the paper validates it on more than a dozen fine-grained datasets plus human and counterfactual tests.","feed_headline":"Prompt-CAM localizes the traits that separate visually similar classes","feed_subtitle":"One learnable prompt per class and a shared scorer turn a frozen transformer's attention into trait maps.","key_machinery":"The load-bearing objects are the class-specific prompt tokens \\(p^c\\) and the shared scoring vector \\(w\\). Each prompt is injected into the frozen transformer, and after the last layer its output \\(z^c_N\\) is scored as \\(s[c] = w^\\top z^c_N\\); the prompts are trained with cross-entropy while the backbone stays frozen. In the simplified analysis of Section 2.5, Equation 7—\\(\\hat{y} = \\arg\\max_c \\sum_j \\$\\alpha$^c[j] (w^\\top v_j)\\)—is the identity that carries the argument: with one shared \\(w\\), the model cannot store class-discriminative information in the patch-value channels, so attention weights \\(\\$\\alpha$^c\\) must become class-specific and trait-focused. In the deep variant, class-agnostic prompts in earlier layers adapt the frozen features while only the last-layer prompts are class-specific.","core_discovery":"The paper claims that a pre-trained, frozen ViT can deliver interpretable fine-grained classification if classification is done by class-specific prompt tokens scored by a shared vector w. In the simplified single-head picture, replacing the per-class head \\(\\sum_j \\$\\alpha$^*[j] (w_c^\\top v_j)\\) with \\(\\sum_j \\$\\alpha$^c[j] (w^\\top v_j)\\) removes the detour that lets a conventional model store class information in patch values and attend where it likes. With no class-specific \\(w_c\\), the only way to separate classes is for each prompt's attention \\(\\$\\alpha$^c\\) to concentrate on patches that carry the class's traits. The paper argues that after cross-entropy training on a frozen backbone, the true class's multi-head attention maps in the last transformer layer reveal those traits and their locations, and the experiments support this through faithfulness metrics, a human study, trait-manipulation counterfactuals, and hierarchical taxonomy-key discovery.","pith_inferences":["If the shared-vector mechanism is truly what forces trait-focused attention, then a conventional per-class head on the same prompts should diffuse the attention maps; this is a direct ablation the paper does not report.","The method suggests a trait-based image retrieval or similarity search: images could be matched by which attention heads fire on the same class-specific traits, not just by global embeddings.","Applied to non-biological fine-grained domains, the same recipe may expose diagnostic features in medical or industrial imagery, but the paper's evidence is limited to natural-object datasets and human-viewable traits.","The greedy head-blurring procedure could be turned into a quantitative trait-necessity score per patch, enabling automated trait discovery at scale."],"forward_implications":["The true class's multi-head attention maps can be visualized per head, so a single prediction can expose several distinct traits instead of one coarse heatmap.","Attention maps of other classes' prompts on the same image show which traits are shared between classes, explaining confusions and enabling trait comparison.","A greedy head-blurring procedure ranks the most discriminative traits per class; the remaining heads highlight what is sufficient for classification.","When an image is misclassified, comparing the true-class and predicted-class attention maps explains why: traits are invisible, occluded, or the object resembles the wrong class.","The same recipe extends to hierarchical taxonomy: prompts trained at each tree level localize coarser-to-finer group-level traits."],"supporting_citations":[{"why":"Supplies the Visual Prompt Tuning design that Prompt-CAM modifies by moving the prediction head onto the prompt outputs.","marker":"[12]"},{"why":"Provides the shared-vector classification formulation and an interpretable-transformer baseline that Prompt-CAM extends.","marker":"[31]"},{"why":"Provides the DINO backbone whose localized, self-supervised features are what the prompts attend to.","marker":"[4]"},{"why":"Contributes the idea of class-specific tokens for localization, here repurposed for fine-grained trait discovery.","marker":"[50]"},{"why":"Another source of class-specific token learning, applied to multi-label classification.","marker":"[20]"},{"why":"Supplies the main fine-grained bird dataset used for faithfulness metrics, human study, and trait visualizations.","marker":"[45]"}],"fun_headline_variants":["Prompt-CAM turns frozen ViTs into trait finders","Class prompts make ViTs show fine-grained traits","Prompt-CAM: see the traits ViTs use to distinguish species","One prompt per class, frozen ViT, trait maps","Prompt-CAM: easy interpretable fine-grained analysis with prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that conclusions drawn from a stripped-down single-head transformer—where only attention can carry class information—still hold for the full frozen ViT, which has residual connections, MLPs, layer norms, and many heads that could hide class information in patch values or background cues.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-CAM turns frozen ViTs into trait finders","Class prompts make ViTs show fine-grained traits","Prompt-CAM: see the traits ViTs use to distinguish species","One prompt per class, frozen ViT, trait maps","Prompt-CAM: easy interpretable fine-grained analysis with prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000639,"raw_usage":{"total_tokens":2974,"prompt_tokens":1006,"completion_tokens":1968,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1886}},"tokens_in":622,"tokens_out":1968,"duration_ms":14967,"temperature":1.0,"reasoning_tokens":1886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:05:15.511232+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would be to compute Prompt-CAM attention on images where the class label can be predicted from background or global shape alone: if deleting or masking the highest-attended trait patches leaves the predicted class unchanged, while masked-out low-attended patches change it, the attention maps are not showing the traits that drive classification.","supporting_citations":[{"cited_title":"Vi- sual prompt tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the Visual Prompt Tuning design that Prompt-CAM modifies by moving the prediction head onto the prompt outputs."},{"cited_title":"A simple interpretable transformer for fine-grained image classifica- tion and analysis","cited_arxiv_id":null,"evidence_quote":"Provides the shared-vector classification formulation and an interpretable-transformer baseline that Prompt-CAM extends."},{"cited_title":"Emerg- ing properties in self-supervised vision transformers","cited_arxiv_id":null,"evidence_quote":"Provides the DINO backbone whose localized, self-supervised features are what the prompts attend to."},{"cited_title":"Multi-class token transformer for weakly supervised semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Contributes the idea of class-specific tokens for localization, here repurposed for fine-grained trait discovery."}],"review_version":1}