Pith. sign in

REVIEW 3 major objections 6 minor 54 references

Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a pre-trained, frozen Vision Transformer can be made interpretable for fine-grained analysis by learning one class-specific prompt per class and scoring each with a single shared vector.

desk verdict Prompt-CAM is a cheap, broadly evaluated trait-localization method that mostly works, but its 'must attend' theoretical claim is overreaching and should be softened. read the letter →

arxiv 2501.09333 v2 pith:C2QROUHF submitted 2025-01-16 cs.CV cs.AI

Arpita Chowdhury , Dipanjyoti Paul , Zheda Mai , Jianyang Gu , Ziheng Zhang , Kazi Sajeed Mehrab , Elizabeth G. Campolongo , Daniel Rubenstein
show 5 more authors
Charles V. Stewart Anuj Karpatne Tanya Berger-Wolf Yu Su Wei-Lun Chao
This is my paper · ORCID
classification cs.CVcs.AI
keywords Prompt-CAMvisiontransformerinterpretabilityfine-grainedvisualcategorizationclass-specificprompttokensattentionvisualizationtraitlocalizationtuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a simple recipe for extracting fine-grained explanations from a pre-trained Vision Transformer: attach one learnable prompt token per class, freeze the backbone, and score each class with a single shared vector. Because the classifier is shared, class-discriminative information cannot be stored in patch features; it must be encoded in where each class's prompt attends. The resulting multi-head attention maps therefore localize the diagnostic traits of each class—the red wing patch of a Red-winged Blackbird, the yellow breast of a Scott Oriole—rather than blurring over the whole object. This makes the method nearly a free lunch on top of Visual Prompt Tuning, and the paper validates it on more than a dozen fine-grained datasets plus human and counterfactual tests.

What carries the argument

The load-bearing objects are the class-specific prompt tokens \(p^c\) and the shared scoring vector \(w\). Each prompt is injected into the frozen transformer, and after the last layer its output \(z^c_N\) is scored as \(s[c] = w^\top z^c_N\); the prompts are trained with cross-entropy while the backbone stays frozen. In the simplified analysis of Section 2.5, Equation 7—\(\hat{y} = \arg\max_c \sum_j \$\alpha$^c[j] (w^\top v_j)\)—is the identity that carries the argument: with one shared \(w\), the model cannot store class-discriminative information in the patch-value channels, so attention weights \(\$\alpha$^c\) must become class-specific and trait-focused. In the deep variant, class-agnostic prompts in earlier layers adapt the frozen features while only the last-layer prompts are class-specific.

What would settle it

A decisive test would be to compute Prompt-CAM attention on images where the class label can be predicted from background or global shape alone: if deleting or masking the highest-attended trait patches leaves the predicted class unchanged, while masked-out low-attended patches change it, the attention maps are not showing the traits that drive classification.

Watch

Extended reading notes

Core claim

The paper claims that a pre-trained, frozen ViT can deliver interpretable fine-grained classification if classification is done by class-specific prompt tokens scored by a shared vector w. In the simplified single-head picture, replacing the per-class head \(\sum_j \$\alpha$^*[j] (w_c^\top v_j)\) with \(\sum_j \$\alpha$^c[j] (w^\top v_j)\) removes the detour that lets a conventional model store class information in patch values and attend where it likes. With no class-specific \(w_c\), the only way to separate classes is for each prompt's attention \(\$\alpha$^c\) to concentrate on patches that carry the class's traits. The paper argues that after cross-entropy training on a frozen backbone, the true class's multi-head attention maps in the last transformer layer reveal those traits and their locations, and the experiments support this through faithfulness metrics, a human study, trait-manipulation counterfactuals, and hierarchical taxonomy-key discovery.

Load-bearing premise

The load-bearing premise is that conclusions drawn from a stripped-down single-head transformer—where only attention can carry class information—still hold for the full frozen ViT, which has residual connections, MLPs, layer norms, and many heads that could hide class information in patch values or background cues.

Editorial extensions

If this is right

  • The true class's multi-head attention maps can be visualized per head, so a single prediction can expose several distinct traits instead of one coarse heatmap.
  • Attention maps of other classes' prompts on the same image show which traits are shared between classes, explaining confusions and enabling trait comparison.
  • A greedy head-blurring procedure ranks the most discriminative traits per class; the remaining heads highlight what is sufficient for classification.
  • When an image is misclassified, comparing the true-class and predicted-class attention maps explains why: traits are invisible, occluded, or the object resembles the wrong class.
  • The same recipe extends to hierarchical taxonomy: prompts trained at each tree level localize coarser-to-finer group-level traits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shared-vector mechanism is truly what forces trait-focused attention, then a conventional per-class head on the same prompts should diffuse the attention maps; this is a direct ablation the paper does not report.
  • The method suggests a trait-based image retrieval or similarity search: images could be matched by which attention heads fire on the same class-specific traits, not just by global embeddings.
  • Applied to non-biological fine-grained domains, the same recipe may expose diagnostic features in medical or industrial imagery, but the paper's evidence is limited to natural-object datasets and human-viewable traits.
  • The greedy head-blurring procedure could be turned into a quantitative trait-necessity score per patch, enabling automated trait discovery at scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Prompt-CAM, an interpretability method for frozen Vision Transformers in fine-grained classification. It adds C learnable class-specific prompt tokens to the last (or first) transformer layer, computes logits as inner products of the prompt outputs with a single shared vector w, and trains only the prompts and w with cross-entropy. The core claim is that, because the classifier is shared and has no per-class fully connected layer, the true-class prompt must attend to image patches unique to that class ("traits"), so the resulting multi-head attention maps provide faithful trait localization. The paper validates this with insertion/deletion faithfulness scores, a human trait-recognition study, comparisons to other interpretable and explainable methods, a counterfactual trait-removal experiment, and a preliminary hierarchical taxonomy application, across 13 fine-grained datasets and multiple ViT backbones.

Significance. If the central claim holds, Prompt-CAM is a genuinely attractive method: it is simple, training-efficient, requires only frozen pre-trained encoders, and is supported by code and a demo. The empirical scope is unusually broad for an interpretability paper, and the counterfactual red-wing-patch experiment plus the human study are steps in the right direction. The method also connects to an active area of parameter-efficient transfer learning and to biological trait discovery. However, the paper's advertised guarantee that attention maps 'must' point to traits is not established by the theoretical argument in Section 2.5, and the trait-validation evidence is currently too thin to carry the full weight of the claim. With a careful revision of the theoretical claim and additional systematic trait evaluations, this could be a valuable contribution.

major comments (3)
  1. [2.5, Eq. (7)] The central claim in the Abstract that the true-class prompt "must attend to unique image patches ... (i.e., traits)" is proven only in a stripped-down model that, as the text itself states, omits layer normalization, residual connections, MLP blocks, and self-attention among prompts. In the actual PROMPT-CAM-DEEP forward pass (Eq. (3) plus the final layer LN), the output z_c for class c is not equal to sum_j alpha_c[j] v_j: the prompt's own token survives via the residual connection, it attends to the [CLS] token and to the other C-1 prompts, and the MLP block operates on the residual stream. Therefore w^T z_c can encode class identity through the prompt's own pathway even if every patch attention map alpha_c is uniform, and the "no spatial resolution" argument in Section 2.5 collapses. What is shown is that class-specific attention is a sufficient mechanism in a simplified model, not a necessary one in the full architecture. Please either soften the "must" claim to an "encourages" statement, or provide an empirical test (e.g., setting all patch attention weights to uniform while keeping the prompt's residual and other-token contributions intact and comparing logits) demonstrating that the trait-attention route is the one the trained model actually uses for classification.
  2. [2.5, Eq. (7); 3.2 human study] Even if attention were the only route to class-discriminative scores, Eq. (7) forces only that alpha_c differ across classes so that the shared inner products w^T v_j are combined class-specifically. It does not force the attended patches to be unique to the class or to correspond to semantically meaningful traits: background, pose, and other class-predictive patches can equally generate class-specific scores. The paper's definition of traits as class-unique patches is thereby made true by construction only if the optimized attention happens to choose such patches, which is not proven. The human study (Supplementary Appendix F) covers only four species and four images per method, and the counterfactual in Figure 8 is a single image, so the evidence for trait localization is anecdotal relative to the breadth of the claim. Please evaluate quantitatively against an external trait or part annotation, e.g., the CUB part attributes the paper already uses or the Fish-Vista trait annotations it cites, reporting hit rates or IoU for the top-ranked attention heads over a large set of images.
  3. [2.3, D. Inner Workings of Visualization] In the greedy head-blurring algorithm, replacing a head's attention vector with the uniform distribution and recomputing s[c] does not remove the trait from the image; the residual connection, the MLP, and the prompt's own token still contribute to z_c even when the patch attention is uniform. As a result, the logits may remain high even if no patch is effectively attended to, and the ranking of "least important" heads may under-estimate the number of heads that are actually necessary. Please report the relationship between the greedy head rankings and the human/annotation-based trait importance, and state how often the prediction flips when only the retained heads are left active, so the reader can judge whether the retained heads are indeed sufficient.
minor comments (6)
  1. [Throughout] The method name appears in multiple inconsistent forms, including "P ROMPT -CAM", "PROMPT-CAM", and "Prompt-CAM"; please standardize the typography.
  2. [1, Introduction] The evaluation dataset list includes "Stanford Cars" twice; please remove the duplicate entry.
  3. [E.1, Supplementary] The DINOv2 learning-rate description is duplicated verbatim; the second block likely refers to BioCLIP and should state the correct learning rate for that backbone.
  4. [2.3, Eq. (5)] The uniform-distribution notation "1/M 1" is ambiguous; please introduce a symbol such as u for the uniform vector and write alpha_c,r <- u.
  5. [Table 2] Accuracy is reported for only four of the thirteen datasets in the main paper; consider moving the fuller accuracy table (Supplementary Table 5) to the main text, since the Abstract emphasizes evaluation on more than a dozen datasets.
  6. [References] Reference [14] is cited for attention roll-out, but the original attention roll-out method is Abnar and Zuidema [1]; please reconcile the two citations so that credit is given to the original source.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Prompt-CAM's trait evaluations are grounded in external CUB annotations and counterfactual tests; the Eq. 7 simplification is a theoretical gap, not a circular derivation.

full rationale

The paper's central derivation (Section 2.5, Eqs. 6-7) is a simplified, self-contained argument about why a shared classifier w discourages class information in patch values. It does not fit parameters to its own outputs or rename a fitted quantity as a prediction. The claim that 'the true-class prompt must attend to unique image patches' is an over-idealization for the full ViT because Eq. 7 explicitly assumes no residual connections, no MLP, and no self-attention among prompts; however, this is a correctness/robustness concern, not a circularity. The interpretability claims are validated independently: the human study compares model heatmaps against external CUB trait annotations, and the counterfactual trait-removal test checks causal sensitivity. Self-citations to INTR and Fish-Vista are present, but they are architectural/dataset references, not load-bearing uniqueness theorems or fitted inputs. The greedy head-blurring algorithm is self-referential in that it uses the model's own predictions to rank heads, but the resulting trait identifications are subsequently checked against external trait knowledge, so the loop is not closed internally. No step in the derivation reduces by construction to its own input, and no fitted parameter is renamed as a prediction. Score 0 reflects the absence of circularity; the Eq. 7 gap should be addressed as a limitation but does not constitute a circular derivation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities or ex nihilo constants. It learns prompts and a classifier from data, so the main preconditions are the informativeness of frozen ViT features, the validity of the simplified head analysis, and the reliability of CUB trait annotations as an evaluation benchmark.

free parameters (3)
  • Class-specific prompts pc = not reported
    C learnable D-dimensional tokens trained per dataset; the core mechanism that produces class-specific attention maps.
  • Shared classification vector w = not reported
    Learnable vector in Eq. 1 that projects prompt outputs to logits; central to the claimed trait-forcing behavior.
  • Learning rates per dataset = 0.01/0.1/0.125 (DINO), 0.005/0.01 (DINOv2), 0.05/0.01 (BioCLIP)
    Selected based on dataset-specific characteristics; a hyperparameter tuning choice reported in the supplementary.
assumptions (3)
  • domain assumption Frozen pre-trained ViT patch features retain enough spatial, trait-specific information for fine-grained discrimination.
    The whole method relies on DINO/DINOv2/BioCLIP features; if these features lost local discriminative detail, prompts could not localize traits.
  • ad hoc to paper The simplified single-head analysis of Section 2.5 extends to the full ViT with residuals, norm layers, MLPs, and multiple heads.
    Equations 6-7 omit most of the architecture; the 'must attend to traits' conclusion depends on this extrapolation.
  • domain assumption CUB expert trait annotations are a valid ground truth for trait identification.
    The human study uses CUB text attributes as reference; if these annotations are noisy or not what experts would call traits, the evaluation is weakened.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis." pith.science (2026). https://pith.science/paper/C2QROUHF

@misc{pith2026250109333,
  author       = {Pith},
  title        = {Pith review of: Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C2QROUHF}},
  note         = {Machine review of arXiv:2501.09333}
}
read the original abstract

We present a simple approach to make pre-trained Vision Transformers (ViTs) interpretable for fine-grained analysis, aiming to identify and localize the traits that distinguish visually similar categories, such as bird species. Pre-trained ViTs, such as DINO, have demonstrated remarkable capabilities in extracting localized, discriminative features. However, saliency maps like Grad-CAM often fail to identify these traits, producing blurred, coarse heatmaps that highlight entire objects instead. We propose a novel approach, Prompt Class Attention Map (Prompt-CAM), to address this limitation. Prompt-CAM learns class-specific prompts for a pre-trained ViT and uses the corresponding outputs for classification. To correctly classify an image, the true-class prompt must attend to unique image patches not present in other classes' images (i.e., traits). As a result, the true class's multi-head attention maps reveal traits and their locations. Implementation-wise, Prompt-CAM is almost a ``free lunch,'' requiring only a modification to the prediction head of Visual Prompt Tuning (VPT). This makes Prompt-CAM easy to train and apply, in stark contrast to other interpretable methods that require designing specific models and training processes. Extensive empirical studies on a dozen datasets from various domains (e.g., birds, fishes, insects, fungi, flowers, food, and cars) validate the superior interpretation capability of Prompt-CAM. The source code and demo are available at https://github.com/Imageomics/Prompt_CAM.

Figures

Figures reproduced from arXiv: 2501.09333 by the authors.

Figure 1
Figure 1. Illustration of PROMPT-CAM. By learning class-specific prompts for a pre-trained Vision Transformer (ViT), PROMPT-CAM enables multiple functionalities. (a) PROMPT-CAM achieves fine-grained image classification using the output logits from the class￾specific prompts. (b) PROMPT-CAM enables trait localization by visualizing the multi-head attention maps queried by the true-class prompt. (c) PROMPT-CAM identifies commo… view at source ↗
Figure 2
Figure 2. PROMPT-CAM vs. Visual Prompt Tuning (VPT). (a) VPT [12] adds the prediction head on top of the [CLS] token’s out￾put, a default design to use ViTs for classification. (b) PROMPT￾CAM adds the prediction head on top of the injected prompts’ outputs, making them class-specific to identify and localize traits. show PROMPT-CAM’s extendability by applying it to dis￾covering taxonomy keys. Our contributions are two-fold. •… view at source ↗
Figure 3
Figure 3. Overview of Prompt Class Attention Map (PROMPT-CAM). We explore two variants, given a pre-trained ViT with N layers and a downstream task with C classes: (a) PROMPT-CAM-DEEP: insert C learnable “class-specific” tokens to the last layer’s input and C learnable “class-agnostic” tokens to each of the other N − 1 layers’ input; (b) PROMPT-CAM-SHALLOW: insert C learnable “class￾specific” tokens to the first layer’s input… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Visualization of PROMPT-CAM on different datasets. We show the top four attention maps (from left to right) per correctly classified test example triggered by the ground-truth classes. Species Input Ours Linear Red Breasted Grosbeak Laysan Albatross Input Ours Linear T…
Figure 5
Figure 5. Figure 5: Images misclassified by PROMPT-CAM but correctly classified by Linear Probing. Species-specific traits—such as the red breast of “Red-breasted Grosbeak”—are barely visible in mis￾classified images while Linear Probing uses global features such as body shapes, poses, an…
Figure 6
Figure 6. Figure 6: Comparison of interpretable models. Visual demon￾stration (heatmaps and bounding boxes) of the four most activated responses of attention heads (PROMPT-CAM and INTR) or proto￾types of each method on a “Lazuli Bunting” example image. age of Scott Oriole, highlighting co…
Figure 7
Figure 7. Figure 7: PROMPT-CAM on different backbones. Here we show the top attention maps for PROMPT-CAM on (a) DINO, (b) DINOv2, and (c) BioCLIP backbone. All three sets of attention heads point to consistent key traits of the species “Scott Oriole”— yellow belly, black head, and black …
Figure 8
Figure 8. Figure 8: Trait manipulation. The top row shows attention maps for a correctly classified “Red-winged Blackbird” image. In the second row, the red spot on the bird’s wings was removed, and PROMPT-CAM subsequently classified it as a “Boat-tailed Grackle,” as depicted in the refer…
Figure 10
Figure 10. Figure 10: Accuracy versus the number of layers (from last layer to first) attended by class-specific prompts. As the num￾ber of attended layers increases in class-specific prompts, accuracy decreases, highlighting the importance of class-agnostic prompts. The more class-agnosti…
Figure 11
Figure 11. Figure 11: Visualization of attention maps for different configurations of PROMPT-CAM. For a random image of the “Red-Winged Blackbird” species, twelve attention heads of the last layer of PROMPT-CAM on the DINO backbone are shown for the ground truth class prompt. The first row…
Figure 12
Figure 12. Figure 12: Example Image of a “Western Gull” and its closest bird species, highlighting overlapping traits. Correctly classify￾ing the “Western Gull” requires attention to multiple subtle traits, as it shares many traits with similar species. This highlights the need to examine …
Figure 13
Figure 13. Figure 13: Greedy approach to identify and rank important traits for species classification. For the species “Ruby Throated Hum￾mingbird”, we progressively blur attention heads (from top to bottom), retaining only the traits necessary for correct classification, using the PROMPT…
Figure 14
Figure 14. Figure 14: Visualization of ground truth class probability vs. the number of masked heads at the species level in PROMPT-CAM. The left plots show how the probability of the ground truth class changes for all correctly classified images in a species, as heads are progressively ma…
Figure 15
Figure 15. Figure 15: Comparison of top attention heads for PROMPT￾CAM and Linear probing on two images of the species “Painted Bunting.” For the correctly classified image by both, PROMPT-CAM focuses on meaningful traits such as the blue head, wings, tail, and red lower belly, while Linea…
Figure 16
Figure 16. Figure 16: Comparison of attention heatmaps for Linear Probing and PROMPT-CAM. On random images of “Yellow Headed Black￾bird” and “Scott Oriole” from the CUB dataset, in (a), Linear Probing consistently focuses on similar body parts (e.g., tail, head, under-tail, wings) across a…
Figure 17
Figure 17. Figure 17: Attention heatmaps of cls-token for Linear Probing on misclassified images. For some random images of “Scarlet Tanager” from the CUB dataset, Linear Probing highlights the same body parts across images, failing to provide meaningful insights into misclassi￾fications. …
Figure 18
Figure 18. Figure 18: Visualization of top attention heads of PROMPT￾CAM for DINO, DINOv2 and BioCLIP backbones. For ran￾dom correctly classified images from “Ruby Throated Humming￾bird” and “Chattering Lori” species from Bird Dataset, top-4 at￾tention heads (from left to right) are shown.…
Figure 19
Figure 19. Figure 19: Visualization of attention heads for pre-trained DINO backbone variants. For correctly classified images of “Red winged blackbird”, with PROMPT-CAM, both DINO ViT b/16 and DINO ViT b/8 backbones can capture traits for classifi￾cation. tively. These findings demonstrat…
Figure 20
Figure 20. Figure 20: Visualization of PROMPT-CAM on Bird Dataset. We show the top four attention maps (from left to right) per correctly classified test example, triggered by the ground-truth classes. As top head indices per image may vary, traits may not align across columns. 10 [PITH_F…
Figure 21
Figure 21. Figure 21: Visualization of PROMPT-CAM on Flower Dataset. We show the top four attention maps (from left to right) per correctly classified test example, triggered by the ground-truth classes. As top head indices per image may vary, traits may not align across columns. 11 [PITH…
Figure 22
Figure 22. Figure 22: Visualization of PROMPT-CAM on Dog Dataset. We show the top four attention maps (from left to right) per correctly classified test example, triggered by the ground-truth classes. As top head indices per image may vary, traits may not align across columns. 12 [PITH_FU…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 38 canonical work pages

  1. [1]

    Quantifying atten- tion flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying atten- tion flow in transformers. arXiv preprint arXiv:2005.00928,

    arXiv 2005

  2. [2]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part VI 13, pages 446–461. Springer, 2014. 2, 6

    work page 2014

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

    work page 2020

  4. [4]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv ´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the International Conference on Computer Vi- sion (ICCV), 2021. 2, 6, 1

    work page 2021

  5. [5]

    Transformer inter- pretability beyond attention visualization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer inter- pretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782–791, 2021. 2

    work page 2021

  6. [6]

    This looks like that: deep learn- ing for interpretable image recognition

    Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learn- ing for interpretable image recognition. Advances in neural information processing systems, 32, 2019. 2, 1

    work page 2019

  7. [7]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR,

  8. [8]

    Robust learning with progressive data expansion against spurious correlation

    Yihe Deng, Yu Yang, Baharan Mirzasoleiman, and Quan- quan Gu. Robust learning with progressive data expansion against spurious correlation. Advances in neural information processing systems, 36, 2024. 5

    work page 2024

Show all 54 references
  1. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  2. [10]

    Transfg: A trans- former architecture for fine-grained recognition

    Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, and Changhu Wang. Transfg: A trans- former architecture for fine-grained recognition. In Proceed- ings of the AAAI conference on artificial intelligence , pages 852–860, 2022. 2

    2022

  3. [11]

    The spectre of ‘spurious’ correlations

    Darneisha A Jackson and Keith M Somers. The spectre of ‘spurious’ correlations. Oecologia, 86:147–151, 1991. 5

    1991

  4. [12]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022. 2, 3, 4, 1

    2022

  5. [13]

    Layercam: Exploring hierarchical class activation maps for localization

    Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing, 30:5875–5888, 2021. 2, 6

    2021

  6. [14]

    Explainability of vision transform- ers: A comprehensive review and new perspectives

    Rojina Kashefi, Leili Barekatain, Mohammad Sabokrou, and Fatemeh Aghaeipoor. Explainability of vision transform- ers: A comprehensive review and new perspectives. arXiv preprint arXiv:2311.06786, 2023. 2, 6

    2023 arXiv

  7. [15]

    Novel dataset for fine-grained image categorization: Stanford dogs

    Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proceedings CVPR work- shop on fine-grained visual categorization (FGVC), 2011. 2, 6

    2011

  8. [16]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on com- puter vision workshops, pages 554–561, 2013. 2, 6

    2013

  9. [17]

    Transcam: Transformer attention-based cam refinement for weakly supervised semantic segmentation

    Ruiwen Li, Zheda Mai, Zhibo Zhang, Jongseong Jang, and Scott Sanner. Transcam: Transformer attention-based cam refinement for weakly supervised semantic segmentation. Journal of Visual Communication and Image Representation, 92:103800, 2023. 2

    2023

  10. [18]

    Re- moteclip: A vision language foundation model for remote sensing

    Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Re- moteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024. 1

    2024

  11. [19]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, 2024. 1

    2024

  12. [20]

    Query2label: A simple transformer way to multi-label clas- sification

    Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way to multi-label clas- sification. arXiv preprint arXiv:2107.10834, 2021. 2, 3

    2021 arXiv

  13. [21]

    This looks like those: Illuminating prototypical con- cepts using multiple visualizations

    Chiyu Ma, Brandon Zhao, Chaofan Chen, and Cynthia Rudin. This looks like those: Illuminating prototypical con- cepts using multiple visualizations. Advances in Neural In- formation Processing Systems, 36, 2024. 6, 1, 8

    2024

  14. [22]

    Fine-tuning is fine, if cal- ibrated

    Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Vardaan Pahuja, Tanya Berger-Wolf, Song Gao, Charles Stewart, Yu Su, et al. Fine-tuning is fine, if cal- ibrated. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 1

    2024

  15. [23]

    Lessons learned from a unifying empirical study of parameter-efficient transfer learning (petl) in visual recognition

    Zheda Mai, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Li Zhang, and Wei-Lun Chao. Lessons learned from a unifying empirical study of parameter-efficient transfer learning (petl) in visual recognition. arXiv preprint arXiv:2409.16434 ,

    arXiv

  16. [24]

    Fish-vista: A multi-purpose dataset for understanding & identification of traits from images

    Kazi Sajeed Mehrab, M Maruf, Arka Daw, Harish Babu Manogaran, Abhilash Neog, Mridul Khurana, Bahadir Al- tintas, Yasin Bakis, Elizabeth G Campolongo, Matthew J Thompson, et al. Fish-vista: A multi-purpose dataset for understanding & identification of traits from images. arXiv ...

    2024 arXiv

  17. [25]

    Eigen-cam: Class activation map using principal compo- nents

    Mohammed Bany Muhammad and Mohammed Yeasin. Eigen-cam: Class activation map using principal compo- nents. In 2020 international joint conference on neural net- works (IJCNN), pages 1–7. IEEE, 2020. 2, 6, 1

    2020

  18. [26]

    Neural prototype trees for interpretable fine-grained image recogni- tion

    Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recogni- tion. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 14933–14943,

  19. [27]

    Dreamcreature: Crafting photorealistic virtual creatures from imagination

    Kam Woh Ng, Xiatian Zhu, Yi-Zhe Song, and Tao Xi- ang. Dreamcreature: Crafting photorealistic virtual creatures from imagination. arXiv preprint arXiv:2311.15477, 2023. 2

    2023 arXiv

  20. [28]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & im- age processing, pages 722–729. IEEE, 2008. 2, 6

    2008

  21. [29]

    Maxime Oquab, Timoth ´ee Darcet, Theo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Rus- sell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Ni...

    2023

  22. [30]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505,

    2012

  23. [31]

    A simple interpretable transformer for fine-grained image classifica- tion and analysis

    Dipanjyoti Paul, Arpita Chowdhury, Xinqi Xiong, Feng-Ju Chang, David Carlyn, Samuel Stevens, Kaiya Provost, Anuj Karpatne, Bryan Carstens, Daniel Rubenstein, Charles Stew- art, Tanya Berger-Wolf, Yu Su, and Wei-Lun Chao. A simple interpretable transformer for fine-grained imag...

    2024

  24. [32]

    Rise: Randomized input sampling for explanation of black-box models

    V Petsiuk, A Das, and K Saenko. Rise: Randomized input sampling for explanation of black-box models. arxiv 2018. arXiv preprint arXiv:1806.07421, 1806. 6

    2018 arXiv

  25. [33]

    Birds 525 species - image classification

    Gerald Piosenka. Birds 525 species - image classification

  26. [34]

    Attention-based interpretabil- ity with concept transformers

    Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. Attention-based interpretabil- ity with concept transformers. In International conference on learning representations, 2021. 1

    2021

  27. [35]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  28. [36]

    Medicinal Leaf Dataset, 2020

    Roopashree S and Anitha J. Medicinal Leaf Dataset, 2020. Mendeley Data, V1. 2, 6

    2020

  29. [37]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE in- ternational conference on computer vision , pages 618–626,

  30. [38]

    Bioclip: A vision foundation model for the tree of life

    Samuel Stevens, Jiaman Wu, Matthew J Thompson, Eliza- beth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger- Wolf, et al. Bioclip: A vision foundation model for the tree of life. In Proceedings of the IEEE/CVF Conference...

  31. [39]

    Emergent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Pro- cessing Systems, 36:1363–1389, 2023. 2

    2023

  32. [40]

    Weakly supervised posture mining for fine-grained classi- fication

    Zhenchao Tang, Hualin Yang, and Calvin Yu-Chian Chen. Weakly supervised posture mining for fine-grained classi- fication. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23735– 23744, 2023. 2

    2023

  33. [41]

    Rare Species Dataset, 2023

    Imageomics Team. Rare Species Dataset, 2023. Dataset with 400 classes of rare species images and descriptions sourced from the Encyclopedia of Life and the IUCN Red List. 2, 6

    2023

  34. [42]

    Visual query tuning: Towards effective usage of intermediate representa- tions for parameter and memory efficient transfer learning

    Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. Visual query tuning: Towards effective usage of intermediate representa- tions for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7725–7735...

    2023

  35. [43]

    Benchmarking rep- resentation learning for natural world image collections

    Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking rep- resentation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 12884–12893,...

    2021

  36. [44]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Pro- cessing Systems, page 6000–6010, 2017. 3

    2017

  37. [45]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 2, 6

    2011

  38. [46]

    Score-cam: Score-weighted visual explanations for convolutional neural networks

    Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops ...

    2020

  39. [47]

    Interpretable image recognition by constructing transparent embedding space

    Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by constructing transparent embedding space. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 895–904, 2021. 6, 1, 8 10

    2021

  40. [48]

    Open-set fine-grained retrieval via prompting vision-language evaluator

    Shijie Wang, Jianlong Chang, Haojie Li, Zhihui Wang, Wanli Ouyang, and Qi Tian. Open-set fine-grained retrieval via prompting vision-language evaluator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19381–19391, 2023. 2

    2023

  41. [49]

    Ip102: A large-scale benchmark dataset for insect pest recognition

    Xiaoping Wu, Chi Zhan, Yu-Kun Lai, Ming-Ming Cheng, and Jufeng Yang. Ip102: A large-scale benchmark dataset for insect pest recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8787–8796, 2019. 2, 6

    2019

  42. [50]

    Multi-class token transformer for weakly supervised semantic segmentation

    Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multi-class token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4310–4319, 2022. 2, 3

    2022

  43. [51]

    Protopformer: Concentrating on prototypical parts in vision transform- ers for interpretable image recognition

    Mengqi Xue, Qihan Huang, Haofei Zhang, Lechao Cheng, Jie Song, Minghui Wu, and Mingli Song. Protopformer: Concentrating on prototypical parts in vision transform- ers for interpretable image recognition. arXiv preprint arXiv:2208.10431, 2022. 6, 1

    2022

  44. [52]

    Learning deep features for discrimina- tive localization

    Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discrimina- tive localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929,

  45. [53]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In- ternational Journal of Computer Vision, 130(9):2337–2348,

  46. [54]

    learnable prototypes

    Haowei Zhu, Wenjing Ke, Dong Li, Ji Liu, Lu Tian, and Yi Shan. Dual cross-attention learning for fine-grained visual categorization and object re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4692–4702, 2022. 2 11 P...

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.