Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

NegVQA: Can Vision Language Models Understand Negation?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new 7,379-question benchmark shows vision-language models lose an average of 29.7 accuracy points when a question is negated.

desk verdict A useful negation benchmark with a real main finding, but the headline gap is likely inflated by training-data contamination and the U-shaped scaling claim is overdrawn. read the letter →

arxiv 2505.22946 v1 pith:OJ33KDHS submitted 2025-05-28 cs.CL cs.AIcs.CVcs.CYcs.LG

Yuhui Zhang , Yuchang Su , Yiming Liu , Serena Yeung-Levy This is my paper
classification cs.CLcs.AIcs.CVcs.CYcs.LG
keywords negationunderstandingvisionlanguagemodelsvisualquestionansweringbenchmarkmultimodalreasoningU-shapedscalingzero-shotevaluationinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NegVQA tests whether vision-language models (VLMs) understand negation by converting 7,379 existing multiple-choice visual questions into negated two-choice questions and flipping which answer is correct. Evaluated zero-shot on 20 models from seven families, every model loses accuracy on the negated set; the best model falls from 92.2% to 72.7%, and the average drop across models is 29.7 percentage points. A human baseline on a 100-question sample scores 89%, well above every model, so the gap reflects a real linguistic difficulty rather than impossible items. The paper also reports a U-shaped scaling curve: across model sizes, negation accuracy first gets worse as models grow, then improves at the largest scales, with the dip strongest in reasoning and document/chart categories. The authors argue the weakness tracks scarce negation in training data and recommend adding curated negated examples to instruction-tuning data.

What carries the argument

The central object is the NegVQA dataset itself, built by a two-step pipeline. First, GPT-4o rewrites each original four-choice question from the VMCBench collection into a minimally negated version (inserting 'not', 'do not', 'isn't', etc.) and flags questions that cannot be sensibly negated; after filtering, 7,379 of 9,018 questions remain. Second, the answer labels are inverted: the original correct answer becomes the wrong choice and a randomly sampled original distractor becomes correct, so a model that ignores the negation and picks the old answer fails. This design isolates negation understanding by letting the paper compare each model's accuracy on the identical image and question content with and without negation, and the multi-size model families allow the scaling analysis.

What would settle it

Have human experts audit all 7,379 NegVQA items for conversion errors, such as ungrammatical negation, negation applied to background conditions, or questions without a unique answer after inversion. If the audit finds an error rate well above the 3% seen in the 100-question sample and re-scoring the 20 models on only the clean items shrinks the average drop substantially below 29.7 points, the central claim would be an artifact of flawed items rather than a genuine negation deficit.

Watch

Extended reading notes

Core claim

The paper's central claim is that current VLMs do not reliably process negation, and that this failure is masked by strong performance on ordinary (non-negated) visual question answering. The evidence is the accuracy gap between matched positive and negated questions: although all 20 tested models score high on the original two-choice questions, every one of them drops substantially when asked the negated version, and the best model still answers only 72.7% of negated items correctly. The paper further claims that this weakness follows a U-shaped scaling pattern, explained as the sum of a steadily improving base VQA ability and a negation-specific ability that only 'turns on' at sufficient scale; mid-sized models are competent enough on the original questions that ignoring the negation hurts them more. It also attributes part of the problem to training data: in one typical model family's fine-tuning data, only 1.1% of conversations contain the word 'not'.

Load-bearing premise

The load-bearing premise is that GPT-4o's negated questions are correct and that inverting the two answer choices always yields exactly one right answer; the authors' manual check of 100 questions found 97% correct, so roughly 3% of items may be flawed, and a higher error rate in the full set would inflate the measured drop.

Editorial extensions

If this is right

  • In any deployed system that takes verbal instructions, a user's 'do not...' or 'is not...' may be silently misread; the measured gaps imply this is a concrete failure mode, not a theoretical one.
  • Overall VQA accuracy is not a reliable safety signal: a model can look excellent on ordinary questions while mishandling negated ones.
  • Because the U-shaped trend means some larger models are worse than their smaller counterparts on negation, scaling alone is not a dependable remedy.
  • Adding curated negated examples to instruction-tuning data is a concrete, testable fix suggested by the paper's finding that negations are rare in current fine-tuning data.
  • The benchmark provides a reusable diagnostic: 7,379 paired positive/negated items across general, reasoning, OCR, and document/chart domains for future models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the U-shape holds beyond these seven families, the most dangerous deployment window may be intermediate model sizes: large enough to be trusted on ordinary questions, but still prone to ignoring negation.
  • The two-choice format likely makes the benchmark easier than free-form negation, so the measured gap probably understates real-world difficulty for open-ended negated instructions.
  • A direct follow-up experiment would be to fine-tune one family on negated instruction data and re-run NegVQA; the paper stops at recommending this, so a before/after comparison would test the training-data explanation.
  • The same generation-and-inversion recipe could be ported to other languages or to action-oriented instructions, since the bottleneck appears to be linguistic rather than purely visual.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces NegVQA, a two-choice visual question answering benchmark of 7,379 negated questions created by GPT-4o from VMCBench, covering 20 source datasets and diverse negation types. The authors evaluate 20 VLMs from 7 families zero-shot on both the original non-negated two-choice questions and the negated versions. They report a large and consistent performance drop on negated questions, with an average drop of 29.7 percentage points and the best model (Qwen2-VL-72B) falling from 92.2% to 72.7%. They also report a U-shaped scaling trend across model sizes and a human baseline of 89% on 100 questions. The paper concludes that current VLMs lack robust negation understanding and that scaling shows a U-shaped pattern.

Significance. If the reported drop is a clean measure of negation-specific failure, NegVQA would be a valuable diagnostic for VLM safety and capability assessment. The benchmark is large, covers a broad range of tasks, evaluates 20 models from 7 families, and provides a human baseline; the observation that all evaluated models degrade on negated questions is consistent and likely robust. However, the headline magnitude of the drop is confounded by potential training-data contamination of the positive baseline, and the U-shaped scaling claim is supported by only one clean model family. These issues do not undermine the existence of a negation difficulty, but they do affect the interpretation and strength of the central claims. The manuscript includes a Limitations section but does not mention the contamination concern.

major comments (3)
  1. [§3.2, Table 1; §2.1] The central performance-drop claim (average 29.7-pt drop; Qwen2-VL-72B from 92.2% to 72.7%) is confounded by training-data contamination. The positive baseline 'PosVQA' consists of original non-negated two-choice questions drawn from public VQA datasets (VQAv2, GQA, TextVQA, etc., per §2.1), which are widely used in pretraining and instruction tuning of the evaluated VLMs. A model that memorized the original question-answer mapping will score artificially high on the positive baseline, while NegVQA's answer inversion ensures that the memorized answer is now the wrong option, actively penalizing models on the negated version. The reported gap therefore conflates negation-specific failure with a novelty/familiarity effect. The Limitations section does not address this. To support the claim that VLMs 'struggle significantly with negation' specifically, the authors should either control for memorization (e.g., evaluate on novel non-negated questions or use paraphrased positive controls) or demonstrate via per-dataset contamination analysis that the positive baseline is not inflated.
  2. [§3.2, Figure 3, Table 1] The U-shaped scaling trend is not supported by the reported data. In Table 1, only Cambrian (8B: 54.6, 13B: 50.3, 34B: 59.9) shows a clean dip-and-recover pattern. VILA (3B: 44.8, 8B: 56.2, 13B: 51.2, 40B: 70.5) rises from 3B to 8B, then dips, then rises; it is not a simple U. Molmo, Qwen2-VL, DeepSeek-VL, and InstructBLIP increase monotonically across the available sizes, and LLaVA has only two sizes. The text itself describes the pattern as 'a hint' in one sentence and as a 'trend' in the abstract. The abstract's stronger wording 'uncovers a U-shaped scaling trend' is not commensurate with the evidence. The authors should either restrict the claim to a 'hint' or 'suggestive pattern', or provide additional model families/sizes or a statistical test before claiming a U-shape.
  3. [§2.1, Figure 4] The manual verification of 100 sampled negated questions found 97% correctness, and the three errors shown in Figure 4 include improper negation and negation applied to the condition rather than the question. This implies roughly 220 flawed items in the full 7,379 benchmark. While a 3% error rate cannot by itself explain the 29.7-point drop, the benchmark's validity rests on the accuracy of the GPT-4o transformations and the correctness of the answer inversion. The authors should report the verification procedure in more detail (sampling method, number of annotators, inter-annotator agreement), provide a confidence interval for the 97% estimate, and perform a sensitivity analysis by re-evaluating models on the cleaned subset of verified-correct questions (or on the remaining 97 verified examples) to show that the main conclusions are unchanged when flawed items are removed.
minor comments (5)
  1. [Throughout] The model name 'LLaV A' appears with a space in several places; it should be 'LLaVA'.
  2. [§3.1] The evaluation prompt in §3.1 lists options A through D, but NegVQA is a two-choice benchmark; please clarify whether the prompt is adapted to two options and how model outputs are parsed for a two-choice setting.
  3. [§3.2] The human baseline of 89% on 100 questions lacks details on how the questions were sampled, who the annotators were, and the variability of the estimate; please provide this information or cite a protocol.
  4. [§2.1] The statement that the 97% verification rate 'confirms the high reliability' of the method would be more informative with a confidence interval (for example, a 95% Wilson interval for 97/100 is approximately 91.5%–99.4%).
  5. [§3.2] The claim that only 1.1% of LLaVA fine-tuning conversations contain the word 'not' is presented without a citation; please add a reference or clarify that this is the authors' own observation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark construction and evaluation are external to the tested models, and the central performance-drop claim does not reduce to a fitted parameter or self-citation.

full rationale

NegVQA is an empirical benchmark paper, not a derivation. The claim that 20 VLMs drop by an average of 29.7 points on negated questions rests on (i) GPT-4o-generated negations of source questions taken from VMCBench, (ii) manual verification of 97/100 negated items, and (iii) zero-shot accuracy measurements on 20 external models selected from the OpenVLM Leaderboard. None of these steps fits a circularity pattern: GPT-4o is the generator but is not among the evaluated models; the manual check provides an independent label-quality anchor; and there are no fitted parameters later renamed as predictions. The human baseline (89% on 100 questions) further anchors the comparison externally. The paper does cite the authors' own prior work—VMCBench as the question source and Zhang et al. 2023 for U-shaped scaling—but neither citation carries the central claim: the drop is measured directly on NegVQA, and the U-shape is observed in the paper's own scaling curves. The Limitations section explicitly notes the multiple-choice format and zero-shot evaluation as coverage limits, and Appendix Figure 4 discloses GPT-4o negation errors; these are validity caveats, not circularity. A separate threat—that the non-negated baseline may reuse public VQA questions that some models memorized, inflating the gap—is a correctness/contamination concern outside the seven circularity patterns, so it does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is an empirical benchmark paper, so the ledger contains no fitted parameters or invented entities. The central claims rest on the correctness of the synthetic negation data and the fairness of the two-choice control design.

assumptions (2)
  • domain assumption GPT-4o's generated negations preserve the meaning of the original question and invert the correctness of the answer choices.
    Section 2.1 relies on this for benchmark construction; only a 100-question manual check (97% accuracy) supports it.
  • domain assumption The two-choice format with the original correct answer and a randomly sampled incorrect answer is a fair control for the original (non-negated) questions.
    Section 2.1 states this conversion; it assumes the random incorrect choice does not introduce ambiguity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NegVQA: Can Vision Language Models Understand Negation?." pith.science (2026). https://pith.science/paper/OJ33KDHS

@misc{pith2026250522946,
  author       = {Pith},
  title        = {Pith review of: NegVQA: Can Vision Language Models Understand Negation?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OJ33KDHS}},
  note         = {Machine review of arXiv:2505.22946}
}
read the original abstract

Negation is a fundamental linguistic phenomenon that can entirely reverse the meaning of a sentence. As vision language models (VLMs) continue to advance and are deployed in high-stakes applications, assessing their ability to comprehend negation becomes essential. To address this, we introduce NegVQA, a visual question answering (VQA) benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. We construct NegVQA by leveraging large language models to generate negated versions of questions from existing VQA datasets. Evaluating 20 state-of-the-art VLMs across seven model families, we find that these models struggle significantly with negation, exhibiting a substantial performance drop compared to their responses to the original questions. Furthermore, we uncover a U-shaped scaling trend, where increasing model size initially degrades performance on NegVQA before leading to improvements. Our benchmark reveals critical gaps in VLMs' negation understanding and offers insights into future VLM development. Project page available at https://yuhui-zh15.github.io/NegVQA/.

Figures

Figures reproduced from arXiv: 2505.22946 by the authors.

Figure 1
Figure 1. NegVQA dataset overview. (Middle) NegVQA comprises a diverse set of negated questions, totaling 7,379 instances sourced from various VQA datasets and domains (general, document/chart, reasoning, OCR). (Left/Right) Example questions from different datasets and domains, with correct answers highlighted in green. VLMs process negation and how to scale up VLMs to enhance negation understanding abilities. In summary, we … view at source ↗
Figure 2
Figure 2. Detailed prompts for adding the negation using GPT-4o. reasoning tasks (MathVision (Wang et al., 2024a), GQA (Hudson and Manning, 2019), MMMU (Yue et al., 2024), RealWorldQA (xAI, 2024), MathVista (Lu et al., 2024b), Sci￾enceQA (Lu et al., 2022)), OCR-based VQA (OCRVQA (Mishra et al., 2019), TextVQA (Singh et al., 2019)), and document/chart compre￾hension (DocVQA (Mathew et al., 2021), In￾foVQA (Mathew et al., 2022)… view at source ↗
Figure 3
Figure 3. Model performance and scaling analysis on NegVQA across different VLM families and task categories. (Top left) Performance on the original non-negated two-choice questions shows high accuracy and a positive scaling trend. (Top right) Performance on NegVQA (negated two-choice questions) is significantly lower, with models exhibiting a U-shaped scaling pattern—initially decreasing before improving as model size increa… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Errors in negated questions generated by GPT-4o. The first question cannot be negated, while the second and third questions are negated in the condition, whereas the negation should apply to the main question. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Model performance and scaling analysis on NegVQA across different VLM families and datasets. For each of the 20 subsets in NegVQA, we present scaling curves for both the original non-negated dataset and the negated dataset from left to right, resulting in a total of 40…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Native Hierarchical and Compositional Representations with Subspace Embeddings

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Concepts are embedded as learned linear subspaces, so entailment becomes subspace inclusion and logical operations become intersection, sum, and orthogonal complement; the method claims SOTA on WordNet and surpasses b...

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    John L Ackrill et al. 1975. https://library.uoh.edu.iq/admin/ebooks/30887-aristotle-categories-and-de-interpretatione-clarendon-aristotle-series--1975-2_compressed.pdf Categories and De interpretatione . Clarendon Press

    work page 1975

  4. [4]

    Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. 2025. Vision-language models do not understand negation. arXiv preprint arXiv:2501.09425

    arXiv 2025

  5. [5]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-family Introducing the next generation of claude

    work page 2024

  6. [6]

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. 2023. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818

    arXiv 2023

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS

    2020

  8. [8]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024. Are we on the right way for evaluating large vision-language models? In NeurIPS

    work page 2024

Show all 52 references
  1. [9]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS

    2023

  2. [10]

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146

    2024 arXiv

  3. [11]

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. In ICML

    2023

  4. [12]

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. 2024. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In MM

    2024

  5. [13]

    Federico Fancellu and Bonnie Webber. 2015. Translating negation: A manual error analysis. In Proceedings of the Second Workshop on Extra-Propositional Aspects of Meaning in Computational Semantics ( E x P ro M 2015)

    2015

  6. [14]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR

    2017

  7. [15]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR

    2018

  8. [16]

    Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. An analysis of natural language inference benchmarks through the lens of negation. In EMNLP

    2020

  9. [17]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR

    2019

  10. [18]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  11. [19]

    Nora Kassner and Hinrich Sch \"u tze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In ACL

    2020

  12. [20]

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In ECCV

    2016

  13. [21]

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. 2024 a . Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246

    2024 arXiv

  14. [22]

    Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. 2024 b . Tablevqa-bench: A visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404.19205

    2024 arXiv

  15. [23]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In CVPR

    2024

  16. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597

    2023 arXiv

  17. [25]

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. Vila: On pre-training for visual language models. In CVPR

    2024

  18. [26]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. T ruthful QA : Measuring how models mimic human falsehoods. In ACL

    2022

  19. [27]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In NeurIPS

    2023

  20. [28]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024 a . Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525

    2024 arXiv

  21. [29]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024 b . Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR

    2024

  22. [30]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS

    2022

  23. [31]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR

    2019

  24. [32]

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL Findings

    2022

  25. [33]

    Minesh Mathew, Viraj Bagal, Rub \`e n Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In WACV

    2022

  26. [34]

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In WACV

    2021

  27. [35]

    Ian R McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, et al. 2023. Inverse scaling: When bigger isn't better. TMLR

    2023

  28. [36]

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In ICDAR

    2019

  29. [37]

    OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  30. [38]

    Vincent Quantmeyer, Pablo Mosteiro, and Albert Gatt. 2024. How and where does clip process negation? arXiv preprint arXiv:2407.10488

    2024 arXiv

  31. [39]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML

    2021

  32. [40]

    Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. 2024. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938

    2024 arXiv

  33. [41]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV

    2022

  34. [42]

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In CVPR

    2019

  35. [43]

    no" to say

    Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. 2024. Learn" no" to say" yes" better: Improving vision-language models via negations. arXiv preprint arXiv:2403.20312

    2024 arXiv

  36. [44]

    Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, ADITHYA JAIRAM IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. 2024. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. In NeurIPS

    2024

  37. [45]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024 a . Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.14804

    2024 arXiv

  38. [46]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024 b . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  39. [47]

    Jason Wei, Yi Tay, and Quoc V Le. 2022. Inverse scaling can become u-shaped. arXiv preprint arXiv:2211.02011

    2022 arXiv

  40. [48]

    xAI. 2024. https://x.ai/blog/grok-1.5v Realworldqa dataset

    2024

  41. [49]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML

    2024

  42. [50]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR

    2024

  43. [51]

    Yuhui Zhang, Yuchang Su, Yiming Liu, Xiaohan Wang, James Burgess, Elaine Sui, Chenyu Wang, Josiah Aklilu, Alejandro Lozano, Anjiang Wei, et al. 2025. Automated generation of challenging multiple-choice questions for vision language model evaluation. In CVPR

    2025

  44. [52]

    Yuhui Zhang, Michihiro Yasunaga, Zhengping Zhou, Jeff Z HaoChen, James Zou, Percy Liang, and Serena Yeung. 2023. Beyond positive scaling: How negation impacts scaling trends of language models. In ACL 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.