Pith. sign in

REVIEW 5 major objections 6 minor 47 references

Probing Vision-Language Understanding through the Visual Entailment Task: promises and pitfalls

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that the Visual Entailment task, as evaluated on e-SNLI-VE with Llama 3.2 Vision, does not reliably measure vision-language understanding: the model is highly sensitive to label order, performs near chance on black images…

desk verdict A useful black-image BERTScore control makes this a worthwhile cautionary paper, but single-run numbers and an unmatched SOTA comparison need tightening before the specifics are quoted. read the letter →

arxiv 2507.17467 v1 pith:XYWUEZUW submitted 2025-07-23 cs.CV cs.AI

Elena Pitta , Tom Kouwenhoven , Tessa Verhoef This is my paper · ORCID
classification cs.CVcs.AI
keywords visualentailmentvision-languageunderstandingmultimodallargelanguagemodelsLlama3.2Visione-SNLI-VEpromptsensitivityhallucinationBERTScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether the Visual Entailment (VE) task, deciding whether an image supports, contradicts, or is neutral toward a text hypothesis, actually measures vision-language understanding in a multimodal model. Using Llama 3.2 Vision on the e-SNLI-VE dataset, it finds that task performance is brittle: label order in the prompt shifts predictions, few-shot gains are unstable, and with entirely black images the model still performs near chance while inventing visual details. Fine-tuning reaches 83.3% accuracy and beats the OFA-X baseline, and the fine-tuned model's explanations score 89.2% BERTScore against human references. However, black-image explanations score 87.1%, nearly as high, which the authors take to mean that high BERTScore, and even high accuracy, does not prove the model is grounding its decisions in the image. The upshot, if right, is that VE on this dataset is not a trustworthy probe of multimodal understanding without controls for linguistic priors and prompt effects.

What carries the argument

The load-bearing object is the Visual Entailment task itself: given an image as premise and a sentence as hypothesis, output one of Entailment, Contradiction, or Neutral, as implemented in the e-SNLI-VE dataset, which pairs Flickr30k images with SNLI-style hypotheses and human explanations. The argument is carried by controlled perturbations of that task: permuting the order of the three class labels in the prompt across all six permutations, varying the number and order of in-context examples in three-shot and six-shot settings, and degrading or removing the visual input with random crops and then fully black images. The explanation condition adds a second measurement, the BERTScore F1 between model-generated and human explanations, which the paper uses to separate sounding like reasoning from actually using the image. The black-image condition is the decisive instrument: it holds the linguistic input fixed while removing all visual information, so any residual accuracy, confident hallucination, or high explanation similarity exposes reliance on language priors.

What would settle it

Re-running the three-shot and six-shot experiments with, say, ten randomly selected in-context example sets and reporting confidence intervals would settle whether the 48.7% versus 36.5% ordering and the label-order sensitivity hold; evaluating OFA-X under the same generation prompt would settle the 83.3% versus 80.9% claim; and measuring the black-image BERTScore across multiple runs would settle whether the 0.871 versus 0.892 gap is real.

Watch

Extended reading notes

Core claim

The central discovery is that a state-of-the-art vision-language model can appear to solve visual entailment without reliably using vision. On e-SNLI-VE, Llama 3.2 Vision's zero-shot accuracy is only 41% across six label-order permutations, and majority voting drops to 33.7% because the model changes its answer for the same image when the order of the three class labels changes. With randomly cropped or entirely black images, accuracy falls to near chance, 36.0–38.0% for black images, yet the model confidently describes content that is not there, inventing basketball jerseys and sinks from a black square. Providing three in-context examples helps modestly, with a best accuracy of 48.7%, but six examples hurt, dropping to 36.5%, and the first example's class strongly biases predictions. Fine-tuning on the training set lifts accuracy to 83.3%, above OFA-X's 80.9%, and generated explanations reach a BERTScore F1 of 89.2%; however, black-image zero-shot explanations reach 87.1%, so the explanation metric does not discriminate visually grounded reasoning from language-only guessing. The authors conclude that the VE task as evaluated rewards linguistic priors and prompt-sensitive heuristics as much as or more than visual understanding.

Load-bearing premise

The load-bearing premise is that the reported single-run numbers, such as 41% zero-shot, 48.7% three-shot, 36.5% six-shot, 83.3% fine-tuned, and the 0.871 versus 0.892 BERTScore gap, are stable, and that the OFA-X comparison uses an identical evaluation protocol.

Editorial extensions

If this is right

  • Accuracy on e-SNLI-VE cannot be read as evidence of visual grounding: label-order permutations change zero-shot predictions for more than half the test samples, and majority-vote accuracy falls below single-prompt accuracy.
  • Because six-shot inference (36.5%) performs worse than three-shot (48.7%), adding in-context examples does not reliably improve VE performance and may introduce class biases, such as overprediction of the neutral class.
  • Fine-tuning with QLoRA for one epoch yields 83.3% accuracy, surpassing the reported OFA-X state of the art of 80.9%, so the model's embeddings are rich enough for the task once adapted.
  • Explanation similarity to humans (BERTScore F1 around 0.89) does not certify visual reasoning, since black-image explanations score around 0.87; explanation quality must be evaluated under degraded vision to count as evidence of grounding.
  • The e-SNLI-VE dataset still contains mislabeled or ambiguous examples, so benchmark scores inherit label noise and cases where more than one answer is arguably correct.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not run is to replace generated answers with log-probability-based selection over the three class tokens; if label-order sensitivity largely disappears, the effect is a decoding artifact rather than a learned linguistic prior.
  • The black-image probe could be turned into a cheap diagnostic for other benchmarks: any multimodal task where a model with black images still achieves high explanation similarity or above-chance classification is likely confounded by language priors.
  • The results suggest a design rule for future VE datasets: report accuracy on a no-vision control, such as black images or text-only premises, alongside the main score so readers can quantify the visual contribution.
  • The finding that fine-tuning sharply raises accuracy while black-image explanations stay high hints that fine-tuning may teach dataset-specific label associations as much as visual reasoning; inspecting attention maps or counterfactual image edits could separate those.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper evaluates Llama 3.2 Vision 11B on the e-SNLI-VE visual entailment benchmark across zero-shot, few-shot, and fine-tuned settings, with additional manipulations: six label-order permutations, in-context example order, random crops, and fully black images. The authors report that zero-shot accuracy is near chance (41% overall, 44.5% best single prompt), that predictions are unstable under label reordering, that black images still yield near-chance accuracy with hallucinated explanations, and that fine-tuning reaches 83.3% accuracy, which they claim outperforms the OFA-X state of the art. Explanation BERTScore for the fine-tuned model is 89.2%, while black-image zero-shot explanations score 87.1%; the authors interpret the closeness of these scores as evidence that the VE task is not reliably visually grounded. The paper concludes that VE on e-SNLI-VE is a flawed probe of vision-language understanding.

Significance. If substantiated, the paper would make a useful contribution to the growing literature on benchmark reliability: the black-image control and label-order sensitivity analysis are simple, interpretable probes that could be adopted by other evaluations. The study also provides one of the first public evaluations of Llama 3.2 Vision on VE, with qualitative examples that are instructive. However, the strength of the central claim is currently limited by the lack of variance estimates, ambiguity in the explanation-generation protocol, and unverified comparability with OFA-X. The authors should be credited for explicitly acknowledging the single-run limitation in the Discussion.

major comments (5)
  1. [§5.3, Table 9; Discussion (limitations)] The claim that black-image BERTScore (0.8709/0.8714) is 'comparable' to the fine-tuned model's 0.8916 is load-bearing for the visual-grounding conclusion, but no variance or significance test is reported. Because these scores are averaged over the same test items, a paired bootstrap (or paired test) over items, and ideally over fine-tuning seeds, is needed to determine whether the 0.02 gap is meaningful. If the gap is significant, the wording should be revised; if it is not, that should be stated explicitly. The authors themselves note in the Discussion that 'every experiment was evaluated once,' which makes this omission central rather than cosmetic.
  2. [§5.3, Table 8] The statement that the fine-tuned model 'outperforms the state-of-the-art OFA-X model' (80.9%) assumes that the published OFA-X result was obtained under an identical evaluation protocol. The OFA-X number comes from a different paper and may involve different decoding, prompt, label extraction, or even a different test subset. To make the comparison valid, the authors should either replicate OFA-X under their protocol or reframe the claim as 'higher accuracy than the published OFA-X number under our evaluation pipeline.' Without this, the headline improvement over SOTA is not established.
  3. [§4.3.3 and Table 9] The explanation-generation procedure for the fine-tuned model is not specified. Prompt 1 explicitly instructs the model to 'not include explanations, commentary, or any additional text,' yet Table 9 reports BERTScore for 'Fine-tuned model - Prompt 1.' If explanations were elicited with Prompt 3 (or a variant), the paper should say so; if they were elicited with Prompt 1, the authors need to explain how this is possible. This ambiguity directly affects the validity of the reported 89.2% BERTScore and its comparison with the zero-shot conditions.
  4. [§4.3.2, Tables 6-7; Discussion (limitations)] All few-shot results come from a single random selection of in-context examples, with no seed or example IDs provided. The conclusion that three-shot inference improves over zero-shot rests on the best of six configurations (48.7% vs 44.5%), while the other three-shot configurations achieve 41-43%, overlapping with zero-shot performance. A different random selection could change the ordering, as the authors acknowledge in the Discussion. Please report the selected examples or seed and, ideally, average over multiple selections with standard deviations.
  5. [§5.1, Table 5] Describing the black-image accuracy (36.0-36.9%) as 'at chance' is imprecise because the e-SNLI-VE test set is imbalanced: from Table 1, the majority class (Contradiction) is about 38.8% of the test labels. A trivial majority-class classifier would therefore achieve roughly 38.8%, and the black-image model is actually below that baseline. This is a stronger result than 'at chance' and should be reported as such, with the majority-class baseline explicitly stated.
minor comments (6)
  1. [§4.3.1, Table 2] The term 'majority vote accuracy' is defined as requiring at least four of six outputs to match the correct label; with six prompts, a 3-3 tie is possible, so please clarify how ties are handled and whether excluded items are counted as errors.
  2. [Table 6] The 'Exa. Order' abbreviations (CEN, ECN, NEC) should be defined in the caption rather than only in the text.
  3. [§4.3.3, Table 9] The BERTScore configuration (e.g., the underlying model and layer, and whether rescaled scores were used) is not stated; this is needed for reproducibility.
  4. [Throughout] The paper uses 'Llama', 'LLaMA', and 'Llama 3.2 Vision' inconsistently; please standardize.
  5. [§4.2, References] Reference [17] is cited for the claim that 39% of neutral labels were incorrectly labeled, but the e-SNLI-VE corrected dataset is described in reference [10]; please verify the intended citation.
  6. [Discussion] The phrases 'significantly affects' and 'significant improvement' are used descriptively without statistical tests; consider replacing them with 'substantially' to avoid confusion with statistical significance.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims are external-benchmark evaluations, and the only self-citations are non-load-bearing background.

full rationale

The paper makes no formal derivation that reduces to its own inputs. Its central claim, that VE on e-SNLI-VE does not reliably probe vision-language understanding, rests on measured accuracies (zero-shot 41.0%, three-shot 48.7%, six-shot 36.5%, fine-tuned 83.3%) and BERTScore values (0.8916 fine-tuned vs. 0.8709-0.8714 for black images) computed on the fixed external e-SNLI-VE benchmark (ref. [17]) against an external SOTA baseline, OFA-X (ref. [28]). These quantities are observed outputs of Llama 3.2 Vision, not parameters fitted to make the conclusion true. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no ansatz is smuggled in via self-citation. The only self-citations (refs. [19] and [33]) support a background sentence about conflicting evidence on cross-modal associations and are not load-bearing for any conclusion. The stated single-run limitation (Sections 5 and 6) and the absence of confidence intervals or significance tests weaken statistical robustness, but that is a correctness or evidence concern rather than circularity. The paper's evaluation chain is therefore self-contained with respect to circularity, and a non-finding is appropriate.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

Empirical evaluation paper; no new theoretical entities. The main load-bearing choices are the dataset, the BERTScore metric, the single-run protocol, and the fine-tuning hyperparameters.

free parameters (6)
  • LoRA rank r = 8
    Chosen by hand for QLoRA fine-tuning; table in Appendix B. It controls adapter capacity and is not derived from data.
  • LoRA alpha = 16
    Chosen scaling factor for LoRA updates; affects update magnitude and thus fine-tuning accuracy.
  • Learning rate = 2e-4
    Selected for SFT fine-tuning; no schedule search is reported. It influences the 83.3% accuracy number.
  • Number of epochs = 1
    Training for one epoch; changing it would change the fine-tuned result and the SOTA comparison.
  • Max sequence length = 2048
    Context window for prompts; a larger or smaller value could alter handling of few-shot examples.
  • Temperature = 0
    Set for deterministic decoding in zero- and few-shot experiments; results depend on this decoding choice.
assumptions (4)
  • domain assumption e-SNLI-VE test labels are treated as ground truth for accuracy calculations
    The paper itself shows examples where labels are debatable (Figure 4) and cites known relabeling issues; accuracy numbers inherit label quality.
  • domain assumption BERTScore F1 is a valid proxy for explanation quality
    The paper uses BERTScore as the main explanation metric; it is correlated with human judgment, but the black-image results show it does not track visual grounding.
  • domain assumption Generated text answers reflect the model's belief for the classification
    The paper acknowledges that generated multiple-choice answers can be inconsistent with model beliefs, citing Wang et al. [35] and Khatun and Brown [18].
  • domain assumption OFA-X's reported 80.9% is directly comparable to the authors' 83.3%
    The comparison assumes the same dataset split, prompt, decoding, and label-order conditions; the paper does not demonstrate protocol equivalence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Probing Vision-Language Understanding through the Visual Entailment Task: promises and pitfalls." pith.science (2026). https://pith.science/paper/XYWUEZUW

@misc{pith2026250717467,
  author       = {Pith},
  title        = {Pith review of: Probing Vision-Language Understanding through the Visual Entailment Task: promises and pitfalls},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XYWUEZUW}},
  note         = {Machine review of arXiv:2507.17467}
}
read the original abstract

This study investigates the extent to which the Visual Entailment (VE) task serves as a reliable probe of vision-language understanding in multimodal language models, using the LLaMA 3.2 11B Vision model as a test case. Beyond reporting performance metrics, we aim to interpret what these results reveal about the underlying possibilities and limitations of the VE task. We conduct a series of experiments across zero-shot, few-shot, and fine-tuning settings, exploring how factors such as prompt design, the number and order of in-context examples and access to visual information might affect VE performance. To further probe the reasoning processes of the model, we used explanation-based evaluations. Results indicate that three-shot inference outperforms the zero-shot baselines. However, additional examples introduce more noise than they provide benefits. Additionally, the order of the labels in the prompt is a critical factor that influences the predictions. In the absence of visual information, the model has a strong tendency to hallucinate and imagine content, raising questions about the model's over-reliance on linguistic priors. Fine-tuning yields strong results, achieving an accuracy of 83.3% on the e-SNLI-VE dataset and outperforming the state-of-the-art OFA-X model. Additionally, the explanation evaluation demonstrates that the fine-tuned model provides semantically meaningful explanations similar to those of humans, with a BERTScore F1-score of 89.2%. We do, however, find comparable BERTScore results in experiments with limited vision, questioning the visual grounding of this task. Overall, our results highlight both the utility and limitations of VE as a diagnostic task for vision-language understanding and point to directions for refining multimodal evaluation methods.

Figures

Figures reproduced from arXiv: 2507.17467 by the authors.

Figure 1
Figure 1. An example premise, hypothesis, model prediction, and explanation of the visual entailment task. Hypothesis: A woman carrying a stick. Label: Entailment. Prompt 1 prediction: Contradiction. Explanation: "The image provides sufficient evidence to confirm that the woman is indeed carrying a stick." Prompt 2 prediction: Entailment. Explanation: "The image shows a woman holding a stick, which is consistent with the desc… view at source ↗
Figure 2
Figure 2. Consistency of model predictions across six prompts [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. The original image premise is shown, but the model saw an entirely black image. Hypothesis: 2 basketball players Label: Contradiction. Prompt 1 prediction: Entailment. Explanation: "The image shows two individuals wearing basketball jerseys, which is consistent with the description of basketball players." Prompt 2 prediction: Entailment. Explanation: "The image shows two individuals wearing basketball jerseys, which… view at source ↗
Figures from the paper (1 more)
Figure 6
Figure 6. Figure 6: The original image premise is shown, but the model saw an entirely black image. Hypothesis: the man is inside washing dishes Label: Contradiction. Prompt 1 prediction: Entailment. Explanation: "The image shows a person standing in front of a sink, actively engaged in w…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 35 canonical work pages

  1. [1]

    Agrawal, D

    A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi. Don’t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971–4980, 2018

    work page 2018

  2. [2]

    Alper and H

    M. Alper and H. Averbuch-Elor. Kiki or bouba? sound sym- bolism in vision-and-language models. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, ed- itors, Advances in Neural Information Processing Systems , vol- ume 36, pages 78347–78359. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file...

    work page 2023

  3. [3]

    Baltrušaitis, C

    T. Baltrušaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analy- sis and machine intelligence, 41(2):423–443, 2018

    work page 2018

  4. [4]

    Bowman, G

    S. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Process- ing, pages 632–642, 2015

    work page 2015

  5. [5]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Am...

    work page 1901

  6. [6]

    Conwell, J

    C. Conwell, J. S. Prince, K. N. Kay, G. A. Alvarez, and T. Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv, 2023. doi: 10. 1101/2022.03.28.485868. URL https://www.biorxiv.org/content/early/ 2023/07/01/2022.03.28.485868

    work page 2023

  7. [7]

    Dagan, O

    I. Dagan, O. Glickman, and B. Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evalu- ating Predictive Uncertainty, Visual Object Classification, and Recog- nising Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, volum...

    work page 2005

  8. [8]

    Dettmers, A

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Ef- ficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36:10088–10115, 2023

    work page 2023

Show all 47 references
  1. [9]

    Diwan, L

    A. Diwan, L. Berry, E. Choi, D. Harwath, and K. Mahowald. Why is winoground hard? investigating failures in visuolinguistic composition- ality. In Y . Goldberg, Z. Kozareva, and Y . Zhang, editors, Proceed- ings of the 2022 Conference on Empirical Methods in Natural Lan- guage...

    2022

  2. [10]

    Do, O.-M

    V . Do, O.-M. Camburu, Z. Akata, and T. Lukasiewicz. e-SNLI-VE: Corrected visual-textual entailment with natural language explanations,

  3. [11]

    H. Face. What is llama 3.2 vision?, 2025. URL https://huggingface.co/ blog/llama32?utm_source=chatgpt.com#what-is-llama-32-vision. Ac- cessed: 2025-02-25

    2025

  4. [12]

    Goyal, T

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Mak- ing the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6325–6334,

  5. [13]

    S. Gu, C. Clark, and A. Kembhavi. I can’t believe there’s no im- ages!: Learning visual tasks using only language supervision. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2672–2683. IEEE, 2023

    2023

  6. [14]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR) , 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  7. [15]

    Jabri, A

    A. Jabri, A. Joulin, and L. Van Der Maaten. Revisiting visual ques- tion answering baselines. In European Conference on Computer Vision, pages 727–739. Springer, 2016

    2016

  8. [16]

    Kamath, J

    A. Kamath, J. Hessel, and K.-W. Chang. What’s “up” with vision- language models? investigating their struggle with spatial reasoning. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 91...

    2023 doi

  9. [17]

    Kayser, O.-M

    M. Kayser, O.-M. Camburu, L. Salewski, C. Emde, V . Do, Z. Akata, and T. Lukasiewicz. e-vil: A dataset and benchmark for natural lan- guage explanations in vision-language tasks. In 2021 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 1224–1234. IEEE, 2021

    2021

  10. [18]

    Khatun and D

    A. Khatun and D. G. Brown. A study on large language models’ limitations in multiple-choice question answering, 2024. URL https: //arxiv.org/abs/2401.07955

    2024 arXiv

  11. [19]

    Kouwenhoven, K

    T. Kouwenhoven, K. Shahrasbi, and T. Verhoef. Cross-modal associa- tions in vision and language models: Revisiting the bouba-kiki effect,

  12. [20]

    Lapuschkin, S

    S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications, 10(1):1096, 2019

    2019

  13. [21]

    P. P. Liang, A. Zadeh, and L.-P. Morency. Foundations & Trends in Mul- timodal Machine Learning: Principles, challenges, and open questions. ACM Computing Surveys, 56(10):1–42, 2024

    2024

  14. [22]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157– 173, 2024. doi: 10.1162/tacl_a_00638. URL https://aclanthology...

    2024 doi

  15. [23]

    R. T. McCoy, E. Pavlick, and T. Linzen. Right for the wrong rea- sons: Diagnosing syntactic heuristics in natural language inference. In A. Korhonen, D. Traum, and L. Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, p...

    2019 doi

  16. [24]

    Millière and C

    R. Millière and C. Rathkopf. Anthropocentric bias and the possibility of artificial cognition. In ICML 2024 Workshop on LLMs and Cognition ,

    2024

  17. [25]

    M. Mina, V . Ruiz-Fernández, J. Falcão, L. Vasquez-Reina, and A. Gonzalez-Agirre. Cognitive biases, task complexity, and result in- tepretability in large language models. In O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, and S. Schockaert, ed- itors, Proce...

    2025

  18. [26]

    Mitchell and D

    M. Mitchell and D. C. Krakauer. The debate over understanding in AI’s large language models. Proceedings of the National Academy of Sciences, 120(13):e2215907120, 2023

    2023

  19. [27]

    Muttenthaler, J

    L. Muttenthaler, J. Dippel, L. Linhardt, R. A. Vandermeulen, and S. Kornblith. Human alignment of neural network representa- tions. In The Eleventh International Conference on Learning Rep- resentations (ICLR) , 2023. URL https://openreview.net/forum?id= ReDQ1OUQR0X

    2023

  20. [28]

    Plüster, J

    B. Plüster, J. Ambsdorf, L. Braach, J. H. Lee, and S. Wermter. Harness- ing the power of multi-task pretraining for ground-truth level natural language explanations, 2023. URL https://arxiv.org/abs/2212.04231

    2023 arXiv

  21. [29]

    E. S. Salido, J. Gonzalo, and G. Marco. None of the others: a gen- eral technique to distinguish reasoning from memorization in multiple- choice llm evaluation benchmarks, 2025. URL https://arxiv.org/abs/ 2502.12896

    2025

  22. [30]

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vision , 128(2):336–359, Feb. 2020. ISSN 0920-5691. doi: 10.1007/s11263-019-01228-7. URL https://doi.or...

    2020 doi

  23. [31]

    Shiri, X.-Y

    F. Shiri, X.-Y . Guo, M. G. Far, X. Yu, R. Haf, and Y .-F. Li. An empirical analysis on spatial reasoning capabilities of large multimodal models. In Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language...

    2024 doi

  24. [32]

    Thrush, R

    T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) , pages 5238–5248...

    2022

  25. [33]

    Verhoef, K

    T. Verhoef, K. Shahrasbi, and T. Kouwenhoven. What does kiki look like? cross-modal associations between speech sounds and visual shapes in vision-and-language models. In T. Kuribayashi, G. Ram- belli, E. Takmaz, P. Wicke, and Y . Oseki, editors, Proceedings of the Workshop on...

    2024 doi

  26. [34]

    P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. OFA: Unifying Architectures, Tasks, and Modal- ities Through a Simple Sequence-to-Sequence Learning Framework. In International Conference on Machine Learning , pages 23318–23340. PMLR, 2022

    2022

  27. [35]

    my answer is C

    X. Wang, B. Ma, C. Hu, L. Weber-Genzel, P. Röttger, F. Kreuter, D. Hovy, and B. Plank. “my answer is C”: First-token probabili- ties do not match text answers in instruction-tuned language mod- els. In L.-W. Ku, A. Martins, and V . Srikumar, editors, Findings of the Associatio...

    2024 doi

  28. [36]

    Weber, E

    L. Weber, E. Bruni, and D. Hupkes. Mind the instructions: a holis- tic evaluation of consistency and interactions in prompt-based learn- ing. In J. Jiang, D. Reitter, and S. Deng, editors, Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), p...

    2023 doi

  29. [37]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    2022

  30. [38]

    N. Xie, F. Lai, D. Doran, and A. Kadav. Visual entailment task for visually-grounded language learning, 2019. URL https://arxiv.org/abs/ 1811.10582

    2019 arXiv

  31. [39]

    Zhang, V

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations (ICLR), 2020. A Prompts Example prompt used to assess three-shot and six-shot performance on the VE task. In ...

    2020

  32. [46]

    Hypothesis: A text description. Your task is to classify the relationship between the Premise (image) and Hypothesis (text) into one of the following three categories: - Entailment: The image provides enough evidence to conclude that the Hypothesis is true. - Contradiction: Th...

  33. [47]

    Premise: An image described as follows (attached below)

  34. [48]

    Hypothesis: A text description. Your task is to classify the relationship between the Premise (image) and Hypothesis (text) into one of the following three categories: - Entailment: The image provides enough evidence to conclude that the Hypothesis is true. - Contradiction: Th...

  35. [1195]

    URL https://aclanthology.org/2024.emnlp-main.1195/

    2024

  36. [2017]

    doi: 10.1109/CVPR.2017.670

    2017 doi

  37. [2021]

    URL https://arxiv.org/abs/2004.03744

    2004 arXiv

  38. [2024]

    URL https://openreview.net/forum?id=wrZ6mLelzu

  39. [2025]

    URL https://arxiv.org/abs/2507.10013

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.