Pith. sign in

REVIEW 4 major objections 5 minor 54 references

Verbalized Representation Learning for Interpretable Few-Shot Generalization

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a vision-language model can write task-specific verbalized features from a few pairs of images, and that these features beat prior attribute-generation pipelines and human labels.

desk verdict Core VRL method works and shows consistent few-shot gains, but the advertised 24% claim is not supported by the method alone and the evaluation needs error bars and less selection bias. read the letter →

arxiv 2411.18651 v3 pith:DUT2DQ4T submitted 2024-11-27 cs.CV cs.CLcs.LG

Cheng-Fu Yang , Da Yin , Wenbo Hu , Heng Ji , Nanyun Peng , Bolei Zhou , Kai-Wei Chang This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords VerbalizedrepresentationlearningFew-shotclassificationInterpretablefeaturesVision-languagemodelsInter-classdifferenceIntra-classcommonalityConceptbottleneckNaturallanguageattributes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a vision-language model can produce human-interpretable, task-specific features for image classification from only a handful of examples per class. Instead of relying on the model's memorized knowledge of class names, the method asks the model to compare actual images: what differs between two species, and what is shared within a species. Those verbalized answers are then converted into binary or continuous vectors that any standard classifier can use. The paper reports a 24% absolute improvement over prior state-of-the-art methods while using 95% less data and a smaller model, and a 20% absolute gain over human-labeled attributes. A sympathetic reader would care because it offers a path to interpretable few-shot recognition without manual annotation or model fine-tuning.

What carries the argument

The machinery is a prompted pairing loop over the few-shot training set. For every sampled pair of images from different classes, the vision-language model is asked to name the most distinctive difference ($y_{\text{diff}}$); for every sampled pair from the same class, it is asked to list features shared by both images that make the class distinct ($y_{\text{comm}}$). A second pass converts these descriptions into yes/no or class-choice questions, and each image is scored against every question by the VLM, yielding a feature vector $F = \mathrm{VLM}(y,x)$ whose dimensions are verbalized features. These vectors are fed to off-the-shelf classifiers such as logistic regression, random forests, or MLPs, optionally ensembled, and optionally combined with CLIP or DINO embeddings. The verbalized format is what carries the argument: it grounds features in the data, makes them inspectable by a human, and prevents the collapse-to-constant-output failure of ordinary contrastive self-supervised learning.

What would settle it

A direct check is to corrupt the verbalized feature bridge: replace the VLM's feature-mapping answers with shuffled or random values while keeping the classifier training identical. If accuracy drops by far less than the reported 24-point gain, the claimed grounding in language is not doing the work. A cleaner test is to collect human ground-truth labels for a sample of the actual features used in the main tables and measure per-feature agreement on held-out images; low agreement would falsify the claim that the features are reliably grounded.

Watch

Extended reading notes

Core claim

The central discovery is that the features which separate classes and the features which bind a class together can be written down as natural-language descriptions by a vision-language model, grounded in the actual few-shot images rather than in the model's priors. The paper calls this Verbalized Representation Learning (VRL). With 10 images per species on iNaturalist, the 7B variant reaches 58.46% average accuracy and the 72B variant reaches 67.92%, while prior methods using 200+ images per class and 70B models reach 50.06%. On Kiki-Bouba, a dataset of novel abstract objects, the 7B variant reaches 83.00% with 10 images per object, versus 69.30% for the prior 70B pipeline trained on 800+ images per object. The paper also reports that these automatically extracted features outperform human-annotated attributes by 20% absolute when both are used for downstream classification.

Load-bearing premise

The method assumes the vision-language model reliably decides, for each unseen test image, whether a verbalized feature is present or absent; if it often hallucinates or misses a feature, the entire feature vector is noise and the downstream classifier cannot recover.

Editorial extensions

If this is right

  • VRL with a 7B vision-language model outperforms prior 70B attribute-generation pipelines on iNaturalist while using only 10 images per species instead of 200+.
  • On Kiki-Bouba, VRL reaches 83.00% accuracy with 10 images per object, surpassing prior state-of-the-art methods that used 800+ images per object.
  • The verbalized features combine additively with CLIP and DINO embeddings, raising iNaturalist accuracy to 79.92% via classifier ensembling.
  • VRL-extracted features beat human-labeled attributes by 20% absolute on Kiki-Bouba, suggesting that automatic feature extraction can replace manual attribute annotation in low-resource settings.
  • The same verbalized features work across multiple classifier families and both LLaVA and CLIP feature-mapping models, indicating the representation is not tied to one classifier or one VLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit is that the verbalized format makes the classifier's reasoning auditable: one can read which features drove a prediction and edit a single text description to correct a systematic error.
  • A testable extension would apply the same pairing loop to domains where the vision-language model has weak grounding, such as medical or satellite imagery, and measure whether per-feature agreement degrades faster than accuracy.
  • Because feature generation scales with the number of image pairs, deliberately searching for the most informative pairs rather than sampling them could raise the ceiling of few-shot performance even further.
  • The method's assumption is about the mapping model's reliability, not its size, so the same pipeline could in principle run on a much smaller or closed-source VLM if the mapping step remains faithful.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Verbalized Representation Learning (VRL), a few-shot image classification method in which a vision-language model generates verbalized features by contrasting pairs of images from different classes (inter-class difference) and by identifying commonalities between images of the same class (intra-class commonality). The same VLM then maps each image to a binary or continuous feature vector indicating the presence of each verbalized feature, and these vectors are used to train classifiers such as logistic regression, MLP, or ensembles with CLIP/DINO features. Experiments are conducted on iNaturalist for fine-grained classification, on Kiki-Bouba for novel-object classification, and on mini-ImageNet in the appendix, with additional ablations and a human study of learned features.

Significance. The central idea—using a VLM to verbalize discriminative and shared visual features from few-shot pairs and then using those verbalized features as an interpretable bottleneck—is timely and potentially useful. The paper ships code, reproduces few-shot baselines under matched data conditions, provides ablations over classifiers and feature-mapping models, and attempts a human evaluation of feature quality. If the claims are properly scoped, the method offers a data-efficient, interpretable alternative to fine-tuning and prior attribute-generation approaches. However, the headline quantitative claims are not supported by the tables as written, and several evaluation-protocol details must be clarified before the paper can be accepted.

major comments (4)
  1. [Abstract; §4.3; Table 1] The headline '24% absolute improvement over prior state-of-the-art methods while using 95% less data and a smaller model' is not supported by any single row in Table 1. VRL-7B (58.46%) is 8.40 points above the full-data LLM-Mutate-70B (50.06%) and 14.33 points above the reproduced 10-shot LLM-Mutate-70B† (44.13%); the approximately 24-point gap appears only for VRL-72B (67.92%) versus LLM-Mutate-70B†, which is not a smaller-model comparison. The numbers above 74% in Tables 3 and 9 come from ensembling VRL features with CLIP and DINO image features, not from VRL alone. The abstract, Figure 1, and the conclusion should either cite the exact comparison being made or delete the 24% figure; as written, the paper's central quantitative claim is overstated.
  2. [§3.1, F = VLM(y, x); Appendix B.5] The mapping from images to feature vectors is performed by the same VLM that generated the features, and the paper provides no direct validation that this mapping is reliable on held-out images. The human study in Appendix B.5 samples only 20 features per super class (100 total) on iNaturalist and does not cover the full feature sets used in Tables 1–5 or any Kiki-Bouba features; the 93% relevance score therefore does not establish the reliability of the full feature vectors on unseen test images. Please add a per-feature human-agreement study on held-out images for the actual feature sets, and quantify the effect of replacing the mapping model's outputs with human ground-truth feature labels.
  3. [§4.2; Tables 1, 5, 10] The evaluation protocol is not transparent about how the reported accuracy is selected. Section 4.2 says 'We report the best performance achieved across different classifiers, selecting the optimal results from a single classifier,' but Table 1 reports VRL-7B at 58.46, while Table 10 shows 62.06 for the 7B 'both' configuration with logistic regression and LLaVA-based mapping. Since logistic regression is among the classifiers and 62.06 is higher than 58.46, the best-classifier rule cannot reproduce the headline number unless the two tables use different training-pair samples or feature sets, which is not explained. The paper should state a fixed evaluation protocol and report variance over multiple few-shot splits.
  4. [Table 4] The comparison with human-labeled attributes on Kiki-Bouba is missing a description of how the human attributes were collected. The paper says they are 'human-annotated attributes provided by [12]' but never states whether the annotators worked under the same 10-shot condition as VRL or whether the attributes were produced with full-dataset supervision. If the human attributes were obtained with substantially more data or different annotation instructions, the reported 20% gain would not demonstrate superiority over human feature labeling under matched conditions. Please describe the annotation protocol and, if possible, collect human attributes under the same few-shot setting.
minor comments (5)
  1. [Abstract] The abstract contains the typo 'a smaller mode' for 'a smaller model'; also, the phrase 'at the same model scale' is contradicted by the cross-scale comparisons in Table 1 and should be reworded.
  2. [Table 2] The table title spells the dataset as 'Kiki-Kouba' while the rest of the paper uses 'Kiki-Bouba'; please correct the spelling.
  3. [§3.1] The expression 'CC2 × CN2' is not typeset clearly; please use standard binomial notation, e.g., C(C,2) × C(N,2), and define the pair-counting formulas for both positive and negative pairs.
  4. [Appendix B.3] References [9, 30, 31] appear in the appendix comparison table but are not cited in the main text; please ensure all references are introduced consistently where they are first used.
  5. [Figure 3] The caption states that key attributes are highlighted in bold, but the printed figure does not show clear bold formatting; consider using color or an explicit marker in the figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: VRL's feature-extraction and classification pipeline is self-contained; the same-VLM proposer/evaluator loop is a mild self-reference but does not reduce the accuracy claim to its inputs.

full rationale

The paper's derivation chain is not circular. VRL extracts verbalized features y_diff and y_comm by prompting a fixed VLM on labeled training pairs, then builds numeric vectors F = VLM(y, x) for training and test images, and trains a standard classifier (logistic regression/MLP) on those vectors. The target test labels never enter the feature-generation or classifier-training procedure, so the accuracy numbers are not forced by construction. The use of the same VLM to both propose features and score their presence is a mild self-referential design that could make the features appear more interpretable to that VLM, but it is not an equation-level reduction: the VLM is a fixed pretrained model, and the downstream classifier is a separate fitted model whose predictions can fail. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling citation was found. The abstract's '24% absolute improvement... with a smaller model' is inconsistent with Table 1, but that is a reporting/correctness concern, not circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method relies on a fixed pretrained VLM for both feature generation and feature mapping; no new physical entities are introduced. The main free parameters are sampling choices and the post-hoc selection of the best classifier and ensemble.

free parameters (4)
  • Number of image pairs sampled for feature generation = C*N (e.g., 50 for 5 classes, 10 shots); 100 in scalability experiment
    The paper states 'sampling C*N pairs' is sufficient; this hand-chosen number affects the diversity and quality of verbalized features.
  • Number of VLM generations per pair (b) = 1
    Set to 1; more generations could produce more features but are not used.
  • CLIP similarity threshold for binary features = Not specified
    When CLIP is used as mapping model, a threshold converts similarities to binary values; the threshold value is not reported.
  • Classifier and ensemble choice = Best of LR/MLP/RF etc.; soft ensemble LR+LR+MLP chosen
    The paper reports the best classifier and ensemble combination selected on the validation set, which is a form of model selection that inflates reported accuracy.
assumptions (3)
  • domain assumption The VLM (LLaVA-OneVision) can reliably determine whether an image possesses a previously generated verbalized feature.
    Assumed in Sec 3.1 when mapping F = VLM(y, x). If the VLM's judgments are unreliable, the feature vectors are unreliable.
  • domain assumption The few-shot training images are representative of the class distribution for generating discriminative features.
    Features are generated from only 10 images per class; if these images are atypical, the features may not generalize.
  • domain assumption The verbalized features capture visual characteristics that generalize beyond the training images, rather than overfitting to image-specific background or lighting.
    The human study (Appendix B.5) partially supports this for 100 features, but the full feature sets are not validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Verbalized Representation Learning for Interpretable Few-Shot Generalization." pith.science (2026). https://pith.science/paper/DUT2DQ4T

@misc{pith2026241118651,
  author       = {Pith},
  title        = {Pith review of: Verbalized Representation Learning for Interpretable Few-Shot Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUT2DQ4T}},
  note         = {Machine review of arXiv:2411.18651}
}
read the original abstract

Humans recognize objects after observing only a few examples, a remarkable capability enabled by their inherent language understanding of the real-world environment. Developing verbalized and interpretable representation can significantly improve model generalization in low-data settings. In this work, we propose Verbalized Representation Learning (VRL), a novel approach for automatically extracting human-interpretable features for object recognition using few-shot data. Our method uniquely captures inter-class differences and intra-class commonalities in the form of natural language by employing a Vision-Language Model (VLM) to identify key discriminative features between different classes and shared characteristics within the same class. These verbalized features are then mapped to numeric vectors through the VLM. The resulting feature vectors can be further utilized to train and infer with downstream classifiers. Experimental results show that, at the same model scale, VRL achieves a 24% absolute improvement over prior state-of-the-art methods while using 95% less data and a smaller mode. Furthermore, compared to human-labeled attributes, the features learned by VRL exhibit a 20% absolute gain when used for downstream classification tasks. Code is available at: https://github.com/joeyy5588/VRL/tree/main.

Figures

Figures reproduced from arXiv: 2411.18651 by the authors.

Figure 1
Figure 1. (a) Humans can identify key differences between similar species and recognize common traits within a class, even when appear￾ance varies, and express these insights in concise language. Similarly, our Verbalized Representation Learning (VRL) extracts meaningful features by querying Vision-Language Models (VLMs) to capture inter-class discriminative features and intra-class commonalities. (b) These verbalized feature… view at source ↗
Figure 2
Figure 2. The overview of our Verbalized Representation Learning (VRL) framework. (a) Given N samples per class from C different classes, VRL is able to generate a diverse set (exponentially scaling with N × C) of verbalized features by: 1) extracting key differences between samples from different classes, and 2) identifying commonalities shared among objects within the same class. (b) Given an image, a Vision-and-Language mo… view at source ↗
Figure 3
Figure 3. Qualitative examples of the features extracted by VRL. We highlight the key attributes in bold. (a) Verbalized features extracted by comparing images from different classes. (b) Verbalized features extracted by comparing the images within the same class. Method v1 v2 Avg. Human 73.8 52.5 63.15 VRL-Fdif f 88.4 75.2 81.80 VRL-Fcomm 89.2 74.0 81.60 VRL-both 89.4 76.6 83.00 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 30 canonical work pages

  1. [12]

    Evolving interpretable visual classifiers with large language models

    Mia Chiquier, Utkarsh Mall, and Carl V ondrick. Evolving interpretable visual classifiers with large language models. arXiv preprint arXiv:2404.09941, 2024. 1, 2, 5, 6, 7

    arXiv 2024

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [2]

    Kiki or bouba? sound symbolism in vision-and-language models

    Morris Alper and Hadar Averbuch-Elor. Kiki or bouba? sound symbolism in vision-and-language models. Advances in Neural Information Processing Systems, 36, 2024. 2, 5

    work page 2024

  4. [3]

    Network dissection: Quantifying inter- pretability of deep visual representations

    David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 6541–6549, 2017. 2

    2017

  5. [4]

    Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020

    David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 117(48):30071–30078, 2020. 2

    2020

  6. [5]

    Deep clustering for unsupervised learning of visual features

    Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European confer- ence on computer vision (ECCV), pages 132–149, 2018. 3

    work page 2018

  7. [6]

    Unsupervised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Pi- otr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Ad- vances in neural information processing systems , 33:9912– 9924, 2020. 2, 3, 4

    work page 2020

  8. [7]

    This looks like that: deep learn- ing for interpretable image recognition

    Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learn- ing for interpretable image recognition. Advances in neural information processing systems, 32, 2019. 2

    2019

Show all 54 references
  1. [8]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 2, 3, 4

    2020

  2. [9]

    Semantic prompt for few-shot image recognition

    Wentao Chen, Chenyang Si, Zhang Zhang, Liang Wang, Zilei Wang, and Tieniu Tan. Semantic prompt for few-shot image recognition. arXiv preprint arXiv:2303.14123, 2023. 4

    2023 arXiv

  3. [10]

    Exploring simple siamese rep- resentation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese rep- resentation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 15750–15758, 2021. 2, 3, 4

    2021

  4. [11]

    Solo: A single transformer for scalable vision-language modeling

    Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. Solo: A single transformer for scalable vision-language modeling. In Transactions on Machine Learning Research, 2024. 2

    2024

  5. [13]

    What is one grain of sand in the desert? analyzing individual neurons in deep nlp models

    Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Be- linkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, pages 6309–6317, 2019. 2

    2019

  6. [14]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022. 5

    2022 arXiv

  7. [15]

    Describing objects by their attributes

    Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE con- ference on computer vision and pattern recognition , pages 1778–1785. IEEE, 2009. 1, 2

    2009

  8. [16]

    Learning visual at- tributes

    Vittorio Ferrari and Andrew Zisserman. Learning visual at- tributes. Advances in neural information processing systems, 20, 2007. 1

    2007

  9. [17]

    De- vise: A deep visual-semantic embedding model

    Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. De- vise: A deep visual-semantic embedding model. Advances in neural information processing systems, 26, 2013. 2

    2013

  10. [18]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altch ´e, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neur...

    2020

  11. [19]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9729–9738, 2020. 2, 3, 4

    2020

  12. [20]

    Grounding visual explanations

    Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explanations. In Proceed- ings of the European conference on computer vision (ECCV), pages 264–279, 2018. 2

    2018

  13. [21]

    Natural language descriptions of deep visual features

    Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In Interna- tional Conference on Learning Representations, 2021. 2

    2021

  14. [22]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5

    2021 arXiv

  15. [23]

    Part- stacked cnn for fine-grained visual categorization

    Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. Part- stacked cnn for fine-grained visual categorization. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 1173–1182, 2016. 2

    2016

  16. [24]

    Visualiz- ing and understanding recurrent networks

    Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualiz- ing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015. 2

    2015 arXiv

  17. [25]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673,

  18. [26]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020. 2

    2020

  19. [27]

    Attribute-based classification for zero-shot visual object categorization

    Christoph H Lampert, Hannes Nickisch, and Stefan Harmel- ing. Attribute-based classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intelligence, 36(3):453–465, 2013. 2

    2013

  20. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 4, 5

    2024 arXiv

  21. [29]

    Clip-event: Connecting text and images with event structures

    Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. In Proc. Conference on Computer Vi- sion and Pattern Recognition (CVPR2022), 2022. 2

    2022

  22. [30]

    Making large vision lan- guage models to be good few-shot learners

    Fan Liu, Wenwen Cai, Jian Huo, Chuanyi Zhang, De- long Chen, and Jun Zhou. Making large vision lan- guage models to be good few-shot learners. arXiv preprint arXiv:2408.11297, 2024. 4

    2024 arXiv

  23. [31]

    Envisioning class entity reasoning by large language models for few-shot learning

    Mushui Liu, Fangtai Wu, Bozheng Li, Ziqian Lu, Yun- long Yu, and Xi Li. Envisioning class entity reasoning by large language models for few-shot learning. arXiv preprint arXiv:2408.12469, 2024. 4

    2024 arXiv

  24. [32]

    Visual classification via description from large language models

    Sachit Menon and Carl V ondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183, 2022. 1, 2, 5, 6

    2022 arXiv

  25. [33]

    Language can shape the perception of oriented objects

    Eduardo Navarrete, Michele Miozzo, and Francesca Peres- sotti. Language can shape the perception of oriented objects. Scientific reports, 10(1):8409, 2020. 1, 3

    2020

  26. [34]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 2

    2018 arXiv

  27. [35]

    OpenAI. ChatGPT. 2022. 5

    2022

  28. [36]

    Multimodal explanations: Justifying deci- sions and pointing to the evidence

    Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Mar- cus Rohrbach. Multimodal explanations: Justifying deci- sions and pointing to the evidence. In Proceedings of the IEEE conference on computer vision and pattern recogni- ti...

    2018

  29. [37]

    Scikit-learn: Machine learning in python

    Fabian Pedregosa, Ga ¨el Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011. 3

    2011

  30. [38]

    What does a platypus look like? generating customized prompts for zero-shot image classification

    Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15691–15701, 2023. 1, 2

    2023

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    Synaesthesia–a window into perception, thought and lan- guage

    Vilayanur S Ramachandran and Edward M Hubbard. Synaesthesia–a window into perception, thought and lan- guage. Journal of consciousness studies, 8(12):3–34, 2001. 5

    2001

  33. [41]

    Optimization as a model for few-shot learning

    Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International conference on learn- ing representations, 2017. 5

    2017

  34. [42]

    An embarrass- ingly simple approach to zero-shot learning

    Bernardino Romera-Paredes and Philip Torr. An embarrass- ingly simple approach to zero-shot learning. InInternational conference on machine learning, pages 2152–2161. PMLR,

  35. [43]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE in- ternational conference on computer vision , pages 618–626,

  36. [44]

    A multimodal automated interpretability agent

    Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and An- tonio Torralba. A multimodal automated interpretability agent. In Forty-first International Conference on Machine Learning, 2024. 2

    2024

  37. [45]

    Incremental residual con- cept bottleneck models

    Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. Incremental residual con- cept bottleneck models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11030–11040, 2024. 2

    2024

  38. [46]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 5

    2023 arXiv

  39. [47]

    The inaturalist species classification and de- tection dataset

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and de- tection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages...

  40. [48]

    Learning concise and descriptive attributes for visual recognition

    An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 3090–3100,

  41. [49]

    Paraphrasing is all you need for novel object caption- ing

    Cheng-Fu Yang, Yao-Hung Hubert Tsai, Wan-Cyuan Fan, Russ R Salakhutdinov, Louis-Philippe Morency, and Frank Wang. Paraphrasing is all you need for novel object caption- ing. Advances in Neural Information Processing Systems , 35:6492–6504, 2022

    2022

  42. [50]

    Language in a bottle: Language model guided concept bottlenecks for interpretable image classification

    Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2023

  43. [51]

    Post-hoc concept bottleneck models

    Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205.15480, 2022. 2

    2022 arXiv

  44. [52]

    Visualizing and understanding convolutional net- works

    MD Zeiler. Visualizing and understanding convolutional net- works. In European conference on computer vision/arXiv ,

  45. [53]

    Sglang: Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2023. 1 Verbalized Represe...

    2023 arXiv

  46. [54]

    role": "user

    For intra-class commonality features (ycomm), we assign a value of 1 if the model responds with ‘Yes’ and 0 if it responds with ‘No’. A.2. Time Complexity Given a classification task with C classes and N-shot ex- amples per class, we are able to construct C C 2 × C N 2 and C ×...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.