Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationality

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a vision-language model improves accuracy while increasing the share of correct predictions that rest on invalid evidence, a gap that standard accuracy metrics hide.

desk verdict Plausible and important claim that fine-tuning hurts prediction rationality, but the uncalibrated RMA threshold and single-run results make the evidence conditional, not conclusive. read the letter →

arxiv 2412.13333 v2 pith:IZDVGM2L submitted 2024-12-17 cs.LG

Qitong Wang , Tang Li , Kien X. Nguyen , Xi Peng This is my paper · ORCID
classification cs.LG
keywords predictiontrustworthinessinferencereliabilityvision-languagemodelfine-tuningexplanationheatmapsrelevantmassaccuracyrationalityout-of-distributionrobustness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning a vision-language model such as CLIP on a downstream classification task reliably improves accuracy, but the paper argues that this gain is accompanied by a hidden cost: a larger fraction of the model's correct answers rest on evidence that would not survive human scrutiny. To make this visible, the paper proposes two metrics. Prediction Trustworthiness measures, among all correct predictions, the share whose explanation heatmap concentrates on the target object rather than on background; Inference Reliability measures, among all cases where the model's attention lands on the target, the share of correct predictions. Across four VLM backbones, four datasets, and out-of-distribution corruptions, fine-tuning consistently lowers Prediction Trustworthiness while raising Inference Reliability relative to zero-shot. The message is that accuracy alone overstates the quality of fine-tuned VLMs.

What carries the argument

The machinery has three parts. First, explanation heatmaps are produced by gradient-weighted attention, a method that combines attention maps with their gradients to the output; these heatmaps stand in for the model's evidence. Second, the Relevant Mass Accuracy (RMA) score, defined as $\text{RMA}(H,M)=\frac{\sum H\odot M}{\sum H}$, measures what fraction of heatmap mass falls inside the ground-truth target mask, and a hand-set threshold of 0.5 labels a prediction's evidence as valid or invalid. Third, the four quadrant counts RR, RW, WR, and WW define Prediction Trustworthiness as $\text{PT}=\frac{RR}{RR+RW}$ and Inference Reliability as $\text{IR}=\frac{RR}{RR+WR}$. The separation of PT from IR is what distinguishes 'right for wrong reasons' from 'reliable when focused.'

What would settle it

Recompute PT and IR while varying the validity threshold between roughly 0.3 and 0.7, or replacing bounding-box masks with pixel-level segmentation masks on ImageNet; if for some threshold in that range the fine-tuned methods no longer show a PT drop relative to zero-shot, the paper's central finding is an artifact of the chosen cutoff.

Watch

Extended reading notes

Core claim

The central claim is that mainstream fine-tuning methods, including linear probing, finetune-like-CLIP-pretrain, and full fine-tuning, produce more correct predictions based on invalid evidence than zero-shot does, as measured by the paper's Prediction Trustworthiness metric. Concretely, for CLIP-ViT-B/16 on ImageNet-1K, PT drops by 6.4%, 5.65%, and 4.07% for LP, FLCP, and FT, while accuracy rises; on CalTech-101 with ALBEF, the drops are 17.2, 13.85, and 27.31 percentage points. The same tuned models show higher Inference Reliability: given that valid evidence is found, they are more often correct, with CLIP-ViT-B/16 on ImageNet improving IR by 12.6, 8.67, and 16.92 percentage points. Both findings persist on ImageNet-C and across ablation settings, including different optimizers, learning rates, explanation methods, and prompt- and adapter-based tuning.

Load-bearing premise

The claim rests on labeling a prediction's evidence as valid whenever its RMA score is at least 0.5, and this single threshold is applied uniformly across all images, datasets, and models.

Editorial extensions

If this is right

  • Reporting accuracy alone will overstate the value of fine-tuning a VLM for a safety-critical task.
  • A model can become simultaneously more accurate and less trustworthy, so the two properties need to be tracked independently.
  • Fine-tuning methods that improve accuracy are not interchangeable: the paper shows that LP, FLCP, and FT differ in how much PT drops and how much IR rises.
  • On corrupted images the same pattern holds, so the trustworthiness drop is not a quirk of clean evaluation data.
  • The four-way RR/RW/WR/WW accounting gives a template for auditing any classifier's evidence, not just VLM fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not test it, but the PT/IR decomposition suggests a concrete fine-tuning objective: adding a term that penalizes attention mass outside the target region should improve PT, and the paper's metrics provide the evaluation recipe.
  • Because RMA uses ground-truth boxes as masks, the valid-evidence label is easiest to earn on datasets where the box covers much of the image; a reader should recheck rankings on fine-grained masks before relying on exact numbers.
  • A practical audit protocol follows directly: in a safety-critical deployment, log heatmaps alongside predictions and reject correct answers whose RMA is below the threshold, which would recover some of the trust that fine-tuning loses.
  • The finding that IR rises with fine-tuning suggests the failure mode is attention selection rather than classification itself, so interventions aimed at focusing attention may be more fruitful than changing the classifier.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes two metrics, Prediction Trustworthiness (PT) and Inference Reliability (IR), to evaluate whether vision-language model predictions are grounded in valid evidence. PT is the fraction of correct predictions whose attention heatmap has RMA ≥ 0.5; IR is the fraction of predictions that are correct among those with RMA ≥ 0.5. Across CLIP, ALBEF, and BLIP models on ImageNet, CalTech-101, Stanford-Dogs, and CUB-200-2011, and for zero-shot, linear probing, FLCP, and full fine-tuning, the paper reports that fine-tuning lowers PT but raises IR, and claims these patterns persist under ImageNet-C corruption and across optizer, learning-rate, explanation-method, prompt-tuning, and adapter-tuning ablations.

Significance. If the central finding is robust, the paper adds a useful evaluation dimension to VLM fine-tuning: accuracy gains may come with a trade-off in the rationality of the evidence, a point that is relevant to safety-critical deployments. The paper contributes two simple, easy-to-compute metrics, releases code, and includes a broad experimental matrix with multiple models, methods, datasets, ablations, and OOD tests. However, the quantitative claim currently rests on an arbitrary validity threshold and a single run per configuration, so its significance is contingent on the sensitivity and uncertainty analyses described in the major comments.

major comments (4)
  1. [Our Proposed Evaluations / Eq. (1)] The binary valid/invalid evidence label is assigned by the single threshold RMA ≥ 0.5, with no calibration to object size. For the bounding-box masks used in ImageNet, ImageNet-C, and Stanford-Dogs, RMA of a uniform heatmap equals the fraction of image pixels inside the box; when that fraction exceeds 0.5, a non-informative explanation is automatically scored as valid evidence. Conversely, for the small segmentation masks in CUB-200-2011, a diffuse but object-covering heatmap can fall below 0.5. Because the correct-prediction sets of ZS, LP, FLCP, and FT differ, the PT and IR differences in Tables 2–4 are confounded by the distribution of mask sizes within each method's correct and wrong prediction sets. The paper does not provide a sensitivity analysis over thresholds, mask-size normalization, or the distribution of RMA scores. I request that the authors report PT and IR as functions of the RMA threshold (e.g., 0.3, 0.4, 0.6, 0.7), normalize RMA by the mask fraction, or otherwise demonstrate that the ordering of methods is not an artifact of the uncalibrated 0.5 cut-off.
  2. [Supplementary: More VLM Finetuning Details] The supplementary states 'We provide the results from a single run.' No confidence intervals, error bars, or significance tests appear in any table or figure in the paper. Because the central claims are quantitative comparisons (for example, ImageNet PT differences of 4.07–6.40 percentage points in Table 2), a single run cannot distinguish the reported pattern from optimization or evaluation noise. Please provide multiple seeds with error bars, or a paired statistical test over the same images, for at least the main tables.
  3. [Explanation Heatmap Generation] Heatmaps are generated for the ground-truth class ('based on the index given by the annotations instead of predictions'), not for the predicted class. This is inconsequential for PT, where correct predictions have predicted label equal to the ground truth, but it matters for IR, which includes WR samples (wrong prediction with 'right' rationale). For a wrong prediction, the heatmap for the ground-truth class does not necessarily reflect the evidence the model actually used to make its incorrect prediction. Please either generate heatmaps for the predicted class, which is the natural target for 'evidence for the prediction,' or provide a justification and show that the IR conclusions are unchanged under both choices.
  4. [Abstract / Weaknesses of Fine-tuning] The abstract asserts without qualification that 'the well-adopted fine-tuning methods led to more correct predictions based on invalid evidence.' Yet Table 2 contains multiple settings where PT increases after fine-tuning (e.g., ALBEF-ViT-B/16 FLCP on CUB: 64.33 vs 49.31 for ZS; CLIP-ViT-B/32 FT on CalTech-101: 80.01 vs 73.44). The body acknowledges these as 'rare exceptions' and attributes them to low zero-shot accuracy, but the headline claim is broader than the data. Please qualify the abstract and conclusion (e.g., 'in most settings') and quantify how often PT decreases across the full experimental matrix.
minor comments (5)
  1. [Supplementary: More VLM Finetuning Details] The acronym FLCP is used in the main text, but the supplementary refers to 'the FLYP method' in the paragraph about the contrastive loss temperature. Please use a single, consistent acronym throughout.
  2. [Supplementary: Additional Visualizations] The phrase 'deteriorate the prediction ratinoality' contains a typo; it should read 'rationality.'
  3. [Conclusion] The phrase 'high-stack applications' in the conclusion should be 'high-stakes applications.'
  4. [Figure 1] The caption lists IoU and RMA values for (a) and (b) but does not state which models or explanation methods produced these heatmaps; please clarify so the example can be interpreted.
  5. [Table 2] The table layout with separate PT and IR blocks is easy to misread; consider adding explicit row labels to every row (e.g., a 'Metric' column) or using a multi-level header.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: PT and IR are direct conditional proportions computed from observed correctness and RMA values, with no fitted parameters, no prediction renamed from a fit, and no load-bearing self-citation.

full rationale

The paper's derivation chain is empirical rather than circular. Prediction Trustworthiness is defined as PT = RR/(RR+RW) and Inference Reliability as IR = RR/(RR+WR), where the four quadrants are obtained by intersecting observed prediction correctness with an RMA threshold (RMA >= 0.5). These are definitions of conditional proportions, not derivations whose conclusions are hidden in the inputs. No parameter is fitted to the datasets and then reported as a prediction; the central findings (fine-tuning lowers PT while raising IR relative to zero-shot) are read directly off the measured counts. The RMA >= 0.5 threshold is an arbitrary operationalization of 'valid evidence', and the choice of Generic Attention Attribution as the heatmap method is justified by external prior work (Liu et al. 2022) rather than by the authors' own claims, so these are independent assumptions about measurement validity, not circular dependencies. The authors' self-citations in Related Works are contextual and do not carry the argument. The supplementary note that all results come from a single run is a statistical limitation — there is no variance estimate — but it is not a circularity. A potential critique that large bounding-box masks make RMA >= 0.5 easier to satisfy is a calibration/validity concern, not a case of the paper defining its conclusion into existence. Accordingly, no specific circular step can be quoted, and the paper receives score 0.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The central claim depends on one hand-set threshold (0.5 on RMA) and two domain assumptions about masks and heatmap faithfulness. No new entities are introduced.

free parameters (1)
  • RMA threshold = 0.5
    Binarizes valid evidence; no sensitivity analysis and no correction for object size.
assumptions (2)
  • domain assumption Bounding-box annotations for ImageNet, ImageNet-C, and Stanford-Dogs are usable as ground-truth explanation masks.
    RMA masks are generated by filling bounding boxes with ones; this conflates object region with a rectangular crop.
  • domain assumption Generic Attention Attribution heatmaps faithfully reflect the evidence used by the model.
    The paper relies on Chefer et al. 2021 and Liu et al. 2022 for faithfulness, but faithfulness of attention-based explanations is debated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationality." pith.science (2026). https://pith.science/paper/IZDVGM2L

@misc{pith2026241213333,
  author       = {Pith},
  title        = {Pith review of: Beyond Accuracy: On the Effects of Fine-tuning Towards Vision-Language Model's Prediction Rationality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZDVGM2L}},
  note         = {Machine review of arXiv:2412.13333}
}
read the original abstract

Vision-Language Models (VLMs), such as CLIP, have already seen widespread applications. Researchers actively engage in further fine-tuning VLMs in safety-critical domains. In these domains, prediction rationality is crucial: the prediction should be correct and based on valid evidence. Yet, for VLMs, the impact of fine-tuning on prediction rationality is seldomly investigated. To study this problem, we proposed two new metrics called Prediction Trustworthiness and Inference Reliability. We conducted extensive experiments on various settings and observed some interesting phenomena. On the one hand, we found that the well-adopted fine-tuning methods led to more correct predictions based on invalid evidence. This potentially undermines the trustworthiness of correct predictions from fine-tuned VLMs. On the other hand, having identified valid evidence of target objects, fine-tuned VLMs were more likely to make correct predictions. Moreover, the findings are also consistent under distributional shifts and across various experimental settings. We hope our research offer fresh insights to VLM fine-tuning.

Figures

Figures reproduced from arXiv: 2412.13333 by the authors.

Figure 1
Figure 1. Both (a) and (b) have low responses to the back [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the four quadrants (RR, RW, WR, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visualization comparisons among different methods. Compared with zero-shot (ZS), current mainstream fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Experimental results on out-of-distribution data. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Additional results of testing on out-of-distribution [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Additional visualizations among different methods on ImageNet-1K validation sets. Here we also select the samples [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Experimental results on out-of-distribution data with CLIP-ViT-B/16 model. Due to the space limit in this figure, [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 29 canonical work pages

  1. [1]

    L.; and Parikh, D

    Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Zitnick, C. L.; and Parikh, D. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, 2425--2433

    2015

  2. [2]

    Arras, L.; Osman, A.; and Samek, W. 2022. CLEVR-XAI: A Benchmark Dataset for the Ground Truth Evaluation of Neural Network Explanations. Inf. Fusion, 81(C): 14–40

    work page 2022

  3. [3]

    Brandt, R.; Raatjens, D.; and Gaydadjiev, G. 2023. Precise Benchmarking of Explainable AI Attribution Methods. arXiv preprint arXiv:2308.03161

    arXiv 2023

  4. [4]

    Chefer, H.; Gur, S.; and Wolf, L. 2021. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 397--406

    work page 2021

  5. [5]

    Chen, R.; Liu, Y.; Kong, L.; Zhu, X.; Ma, Y.; Li, Y.; Hou, Y.; Qiao, Y.; and Wang, W. 2023. CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7020--7030

    work page 2023

  6. [6]

    Chen, X.; and Wang, X. 2022. PaLI: Scaling Language-Image Learning in 100+ Languages. In Conference on Neural Information Processing Systems (NeurIPS)

    work page 2022

  7. [7]

    Chen, Y.-C.; Li, L.; Yu, L.; El Kholy, A.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, 104--120. Springer

    work page 2020

  8. [8]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    arXiv 2020

Show all 53 references
  1. [9]

    Goyal, S.; Kumar, A.; Garg, S.; Kolter, Z.; and Raghunathan, A. 2023. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19338--19347

    2023

  2. [10]

    A.; Akata, Z.; Rohrbach, M.; Donahue, J.; Schiele, B.; and Darrell, T

    Hendricks, L. A.; Akata, Z.; Rohrbach, M.; Donahue, J.; Schiele, B.; and Darrell, T. 2016. Generating visual explanations. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, 3--19. Springer

    2016

  3. [11]

    Hendrycks, D.; and Dietterich, T. 2018. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations

    2018

  4. [12]

    Jia, C.; Yang, Y.; Xia, Y.; Chen, Y.-T.; Parekh, Z.; Pham, H.; Le, Q.; Sung, Y.-H.; Li, Z.; and Duerig, T. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, 4904--4916. PMLR

    2021

  5. [13]

    H.; and Albanie, S

    Jonathan Roberts, K. H.; and Albanie, S. 2023. SATIN: A Multi-Task Metadataset for Classifying Satellite Imagery using Vision-Language Models. arXiv preprint arXiv:2304.11619

    2023 arXiv

  6. [14]

    Khosla, A.; Jayadevaprakash, N.; Yao, B.; and Fei-Fei, L. 2011. Novel Dataset for Fine-Grained Image Categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO

    2011

  7. [15]

    Kim, W.; Son, B.; and Kim, I. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, 5583--5594. PMLR

    2021

  8. [16]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  9. [17]

    Kumar, A.; Ma, T.; Liang, P.; and Raghunathan, A. 2022. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift. In Uncertainty in Artificial Intelligence, 1041--1051. PMLR

    2022

  10. [18]

    Li, B.; Zhang, Y.; Chen, L.; Wang, J.; Yang, J.; and Liu, Z. 2023. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726

    2023 arXiv

  11. [19]

    C.; and Hoi, S

    Li, D.; Li, J.; Li, H.; Niebles, J. C.; and Hoi, S. C. 2022 a . Align and prompt: Video-and-language pre-training with entity prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4953--4963

    2022

  12. [20]

    Li, F.-F.; Andreeto, M.; Ranzato, M.; and Perona, P. 2022 b . Caltech 101

    2022

  13. [21]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022 c . Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, 12888--12900. PMLR

    2022

  14. [22]

    Li, J.; Selvaraju, R.; Gotmare, A.; Joty, S.; Xiong, C.; and Hoi, S. C. H. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34: 9694--9705

    2021

  15. [23]

    H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; Hwang, J.-N.; et al

    Li, L. H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; Hwang, J.-N.; et al. 2022 d . Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10965--10975

    2022

  16. [24]

    Li, T.; Ma, M.; and Peng, X. 2024 a . Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales. In Conference on Neural Information Processing Systems (NeurIPS)

    2024

  17. [25]

    Li, T.; Ma, M.; and Peng, X. 2024 b . DEAL: Disentangle and Localize Concept-level Explanations for VLMs. In Proceedings of the European Conference on Computer Vision (ECCV)

    2024

  18. [26]

    Li, Y.; Liang, F.; Zhao, L.; Cui, Y.; Ouyang, W.; Shao, J.; Yu, F.; and Yan, J. 2022 e . Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm. In International Conference on Learning Representations

    2022

  19. [27]

    Liu, Y.; Li, H.; Guo, Y.; Kong, C.; Li, J.; and Wang, S. 2022. Rethinking attention-model explainability through faithfulness violation test. In International Conference on Machine Learning, 13807--13824. PMLR

    2022

  20. [28]

    Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  21. [29]

    Mao, C.; Teotia, R.; Sundar, A.; Menon, S.; Yang, J.; Wang, X.; and Vondrick, C. 2023. Doubly Right Object Recognition: A Why Prompt for Visual Rationales. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2722--2732

    2023

  22. [30]

    Petsiuk, V.; Das, A.; and Saenko, K. 2018. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421

    2018 arXiv

  23. [31]

    Qiao, F.; and Peng, X. 2023. Topology-aware Robust Optimization for Out-of-Distribution Generalization. In The Eleventh International Conference on Learning Representations

    2023

  24. [32]

    Qiao, F.; Zhao, L.; and Peng, X. 2020. Learning to Learn Single Domain Generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  25. [33]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

    2021

  26. [34]

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211--252

    2015

  27. [35]

    R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D

    Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2020. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int. J. Comput. Vis., 128(2): 336--359

    2020

  28. [36]

    Serrano, S.; and Smith, N. A. 2019. Is Attention Interpretable? In Korhonen, A.; Traum, D.; and M \`a rquez, L., eds., Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2931--2951. Florence, Italy: Association for Computational Linguistics

    2019

  29. [37]

    Singh, A.; Hu, R.; Goswami, V.; Couairon, G.; Galuba, W.; Rohrbach, M.; and Kiela, D. 2022. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15638--15650

    2022

  30. [38]

    Suhr, A.; Lewis, M.; Yeh, J.; and Artzi, Y. 2017. A Corpus of Natural Language for Visual Reasoning. In Annual Meeting of the Association for Computational Linguistics

    2017

  31. [39]

    Tian, Y.; Krishnan, D.; and Isola, P. 2020. Contrastive multiview coding. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XI 16, 776--794. Springer

    2020

  32. [40]

    Vinyals, O.; and Le, Q. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869

    2015 arXiv

  33. [41]

    Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011. The caltech-ucsd birds-200-2011 dataset

    2011

  34. [42]

    Wang, J.; Chen, D.; Wu, Z.; Luo, C.; Zhou, L.; Zhao, Y.; Xie, Y.; Liu, C.; Jiang, Y.-G.; and Yuan, L. 2022 a . Omnivl: One foundation model for image-language and video-language tasks. Advances in neural information processing systems, 35: 5696--5710

    2022

  35. [43]

    Wang, Q.; Zhao, L.; Yuan, L.; Liu, T.; and Peng, X. 2023. Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 3307--3317

    2023

  36. [44]

    Wang, Z.; Wu, Z.; Agarwal, D.; and Sun, J. 2022 b . MedCLIP: Contrastive Learning from Unpaired Medical Images and Text. In Conference on Empirical Methods in Natural Language Processing

    2022

  37. [45]

    W.; Li, M.; Kornblith, S.; Roelofs, R.; Lopes, R

    Wortsman, M.; Ilharco, G.; Kim, J. W.; Li, M.; Kornblith, S.; Roelofs, R.; Lopes, R. G.; Hajishirzi, H.; Farhadi, A.; Namkoong, H.; et al. 2022 a . Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7959--7971

    2022

  38. [46]

    W.; Li, M.; Kornblith, S.; Roelofs, R.; Lopes, R

    Wortsman, M.; Ilharco, G.; Kim, J. W.; Li, M.; Kornblith, S.; Roelofs, R.; Lopes, R. G.; Hajishirzi, H.; Farhadi, A.; Namkoong, H.; et al. 2022 b . Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7959--7971

    2022

  39. [47]

    Xie, N.; Lai, F.; Doran, D.; and Kadav, A. 2019. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706

    2019 arXiv

  40. [48]

    Yuan, L.; Chen, D.; Chen, Y.-L.; Codella, N.; Dai, X.; Gao, J.; Hu, H.; Huang, X.; Li, B.; Li, C.; et al. 2021. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432

    2021 arXiv

  41. [49]

    Zhang, R.; Zhang, W.; Fang, R.; Gao, P.; Li, K.; Dai, J.; Qiao, Y.; and Li, H. 2022. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, 493--510. Springer

    2022

  42. [50]

    H.; Zhou, L.; Dai, X.; Yuan, L.; Li, Y.; et al

    Zhong, Y.; Yang, J.; Zhang, P.; Li, C.; Codella, N.; Li, L. H.; Zhou, L.; Dai, X.; Yuan, L.; Li, Y.; et al. 2022. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16793--16803

    2022

  43. [51]

    C.; and Liu, Z

    Zhou, K.; Yang, J.; Loy, C. C.; and Liu, Z. 2022. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9): 2337--2348

    2022

  44. [52]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.