Pith. sign in

REVIEW 4 major objections 3 minor 53 references

Mitigating Easy Option Bias in Multiple-Choice Question Answering

T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports an Easy-Options Bias in multiple-choice VQA benchmarks: a vision-language model can pick the correct answer from the image and options alone, without seeing the question.

desk verdict A genuine and potentially important observation that multiple-choice VQA lets models bypass the question, but the 'EOB-free' claim is only as strong as the CLIP-similarity measure used to both build and validate it. read the letter →

arxiv 2508.13428 v1 pith:XAPJYJAB submitted 2025-08-19 cs.CV cs.AIcs.MM

Hao Zhang , Chen Li , Basura Fernando This is my paper
classification cs.CVcs.AIcs.MM
keywords Easy-OptionsBiasmultiple-choiceVQAvision-languagemodelsshortcutlearningCLIPsimilarityhardnegativesbenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This early study reports that several multiple-choice visual question answering benchmarks—including MMStar, RealWorldQA, SEED-Bench, NExT-QA, STAR, and Video-MME—contain an Easy-Options Bias. Across four VLM families, models given only the image and option list reach 51.57% mean accuracy, compared with 61.11% when the question is included, and a CLIP-based image–option matching baseline reaches 42.1%. The paper attributes the bias to an imbalance in visual relevance: the correct option is typically more similar to the image in feature space than the distractors. To correct it, GroundAttack generates hard negative options whose visual similarity is balanced with the correct answer. On the resulting EOB-free annotations, V+O accuracy approaches chance and full V+Q+O accuracy falls to non-saturated levels, indicating that the original numbers do not purely measure question-guided reasoning.

What carries the argument

The central objects are the Easy-Options Bias and the CLIP-based visual-similarity score $\mathrm{CLIP}(V,O)$ that exposes it. The bias is quantified by the accuracy gap between $(V,Q,O)$ and $(V,O)$ inputs; a small gap means the question adds little. GroundAttack is the corrective machinery: it automatically rewrites negative options until their CLIP similarity to the image matches the correct answer's, removing the feature-space imbalance. The load-bearing operation is this balancing—it converts a benchmark in which the correct answer is visually distinctive into one in which distractors are visually interchangeable with it.

What would settle it

Take a benchmark re-annotated by GroundAttack and run a VLM on (V+O) with hard negatives picked by a different visual encoder; if accuracy stays well above chance, then CLIP similarity was not the operative shortcut and the claim that the annotations are EOB-free would be called into question. Alternatively, for any original benchmark, replace the negative options with options whose CLIP similarity is balanced and check whether V+O accuracy still stays above chance.

Watch

Extended reading notes

Core claim

The central claim is that current multiple-choice VQA benchmarks overstate VLMs' reasoning ability because the correct answer is, on average, more visually aligned with the image than the negative options. This makes the question unnecessary: a model can select the correct option by matching image and option features alone. The supporting experiment shows mean accuracy of 51.57% with only (V,O) versus 61.11% with (V,Q,O), and 42.1% using CLIP similarity directly. The paper then argues this imbalance is a dataset construction artifact rather than a sign of competence, and shows that once negative options are made visually plausible—NExT-QA and MMStar re-annotated with GroundAttack—VLMs fall t

Load-bearing premise

The diagnosis and the fix both assume that CLIP visual similarity between an option and the image is the same cue the VLM itself exploits when the question is removed; if models use some other clue, balancing CLIP scores may not remove the bias or may introduce new artifacts.

Editorial extensions

If this is right

  • Benchmark scores on the affected datasets partly measure option-set artifacts: on EOB-free versions, V+O accuracy approaches chance, so the question is no longer redundant.
  • The V+Q+O accuracy itself drops to non-saturated levels once hard negatives are balanced, meaning previously reported numbers overstate how much question-guided reasoning these models perform.
  • GroundAttack can serve as an automatic audit: running the (V,Q,O) versus (V,O) comparison over an MCQ-VQA dataset reveals whether the option set is EOB-prone.
  • The same de-biasing process can be applied to other multiple-choice VQA benchmarks, including video ones such as Video-MME, though the paper's evidence there is early.
  • If EOB is widespread, ranking of VLMs may change under EOB-free evaluation, because models that lean on the shortcut lose their advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the (V,Q,O) versus (V,O) accuracy gap is a cheap audit that any benchmark builder can run; a large gap is evidence the options are doing too much work.
  • Editorial extension: because GroundAttack balances with CLIP similarity, the annotations are only as good as that proxy; human plausibility ratings or another vision encoder would test whether the bias is CLIP-specific.
  • Editorial extension: automatically generated benchmarks that use model-produced options may inherit EOB; applying GroundAttack there is a direct stress test.
  • Editorial extension: if EOB is as widespread as this paper suggests, comparing models on EOB-free sets could separate those that consult the question from those that pattern-match images and options.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper identifies an 'Easy-Options Bias' (EOB) in multiple-choice VQA benchmarks: vision-language models (VLMs) can select the correct answer using only the image (V) and the answer options (O), without the question (Q). The authors report aggregate mean accuracies (V+O 51.57%, V+Q+O 61.11%, CLIP-based selection 42.1%) across six benchmarks and four VLM series, and attribute the bias to an imbalance in visual relevance between correct and incorrect options in feature space. They propose GroundAttack, a toolkit that automatically generates hard negative options balanced in CLIP-based visual relevance, apply it to NExT-QA and MMStar to create 'EOB-free' annotations, and claim that on these annotations VLMs approach random accuracy under V+O and drop to non-saturated accuracy under V+Q+O. The manuscript as reviewed contains only the abstract, introduction, and two figure captions; the methods, experimental protocol, and results sections are absent.

Significance. If the claims are correct, this work is significant because it suggests that multiple-choice VQA benchmarks substantially overestimate the visual reasoning capabilities of VLMs: the question is often unnecessary for selecting the answer. The proposed hard-negative rebalancing (GroundAttack) could lead to more realistic evaluation benchmarks, and the paper promises release of code and annotations, which is a strength. However, the current manuscript provides insufficient evidence. The CLIP-based grounding is a reasonable first step, but it accounts for only part of the observed V+O accuracy (42.1% vs. 51.57%), and the construction and validation of GroundAttack share the same CLIP-similarity measure. The central claims, while plausible, are not yet established with the rigorous experimental support expected in a journal submission.

major comments (4)
  1. [Abstract/Figure 2] The central quantitative evidence for EOB is the aggregate gap between V+O accuracy (51.57%) and V+Q+O accuracy (61.11%), and the CLIP-only accuracy of 42.1%. These are presented as means over six benchmarks and four VLM series with no variance measures, per-dataset breakdowns, or significance tests. The 9.54 percentage-point gap between V+O and V+Q+O could be within model or benchmark variability. Without error bars or a paired statistical test, the existence and magnitude of EOB are not established, which is load-bearing for the paper's first claim.
  2. [Abstract/Introduction] GroundAttack is the main mitigation contribution, but the reviewed text contains no description of its algorithm. The paper does not state how hard negatives are generated, how 'visually plausible' is quantified beyond mentioning CLIP, which CLIP model/features are used, how balance is enforced (e.g., matching mean/max similarity), or how the NExT-QA and MMStar EOB-free annotations were produced and validated. Without this information, the proposed toolkit cannot be evaluated or reproduced, and the EOB-free claim is unsupported.
  3. [Figure 2 / GroundAttack] The grounding experiment and the mitigation share the same measure. GroundAttack constructs negatives balanced in CLIP visual similarity, and the success criterion is the drop in V+O accuracy. The reported CLIP(V,O) accuracy (42.1%) is substantially lower than the VLM V+O accuracy (51.57%), leaving a residual shortcut that is not explained by CLIP similarity. The paper does not show that this residual is eliminated by the CLIP-balanced negatives. Consequently, 'EOB-free' is a stronger claim than 'CLIP-balanced' and requires an independent validation of distractor quality (e.g., human plausibility, or probing for object-presence/OCR/language-prior shortcuts).
  4. [Abstract / EOB-free evaluation] The claim that VLMs 'approach random accuracies' under V+O on the new annotations is not quantitatively defined. For a multiple-choice task with K options the chance level is 1/K, but the paper never states K for NExT-QA/MMStar, how the random baseline is computed (uniform versus empirical marginal), or whether the observed accuracies are statistically indistinguishable from that baseline. 'Approach' is too vague to support the EOB-free claim without error bars and a formal test.
minor comments (3)
  1. [Abstract] Typographical spacing: 'anEasy-Options' should read 'an Easy-Options'.
  2. [Figure 1 caption] Typographical spacing: 'Easy-Options Biaslets' should read 'Easy-Options Bias lets'.
  3. [Figure 2 caption] The abbreviations V, Q, O, and A are defined in Figure 1 but not in this caption; please define them for self-contained reading.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EOB is measured empirically, GroundAttack uses CLIP relevance, and EOB-free is validated by VLM accuracy, a distinct measure.

full rationale

The paper's central claim is an empirical observation: VLMs can pick the correct answer from vision-plus-options without the question, supported by V+O accuracy (51.57%) versus V+Q+O accuracy (61.11%) across six benchmarks and four VLM families, plus a CLIP-based grounding result (42.1%). The attribution to visual-relevance imbalance is a hypothesis tested by these grounding experiments, not a definitional consequence. GroundAttack generates hard negatives by balancing CLIP-style visual similarity, which is a specific, mechanistically motivated intervention. The subsequent claim that the new annotations are 'EOB-free' is then evaluated by measuring VLM V+O accuracy on the modified benchmarks. Construction (CLIP similarity) and evaluation (VLM accuracy) are different measures, and no disclosed equation or fitted constant makes the V+O accuracy drop follow by construction. The residual risk that CLIP balancing may not capture all shortcut mechanisms is a validity concern, not an instance of circularity. There is no load-bearing self-citation or imported uniqueness theorem. Therefore no specific circular step can be quoted, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the assumption that CLIP similarity captures the VLM shortcut, that the V+O versus V+Q+O accuracy gap is a clean measure of question necessity, and that random accuracy without the question is the correct fairness baseline. No fitted constants are reported in the available text, and no new physical or conceptual entities are introduced beyond the named bias and toolkit.

assumptions (3)
  • domain assumption CLIP-style visual similarity between an option and the image is a faithful measure of the shortcut VLMs use under V+O inputs.
    The grounding experiments use CLIP to rank options by visual similarity and attribute EOB to this imbalance, but it is assumed that this similarity score correlates with the internal representations VLMs rely on when the question is absent.
  • domain assumption The difference between V+O accuracy and V+Q+O accuracy isolates the contribution of the question, with no other confounds such as language priors or model memorization.
    The paper compares these two settings to conclude that the question is redundant, but other factors (e.g., option wording, dataset answer distribution) could also produce high V+O accuracy without being a visual-relevance shortcut.
  • ad hoc to paper Multiple-choice VQA accuracy should be at chance when the question is removed; any accuracy above chance indicates an unwanted shortcut.
    This is the paper's operational definition of EOB: if a model can pick the correct answer without the question, the benchmark is flawed. This is a reasonable evaluation criterion but is imposed by the authors rather than derived from prior theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mitigating Easy Option Bias in Multiple-Choice Question Answering." pith.science (2026). https://pith.science/paper/XAPJYJAB

@misc{pith2026250813428,
  author       = {Pith},
  title        = {Pith review of: Mitigating Easy Option Bias in Multiple-Choice Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XAPJYJAB}},
  note         = {Machine review of arXiv:2508.13428}
}
read the original abstract

In this early study, we observe an Easy-Options Bias (EOB) issue in some multiple-choice Visual Question Answering (VQA) benchmarks such as MMStar, RealWorldQA, SEED-Bench, Next-QA, STAR benchmark and Video-MME. This bias allows vision-language models (VLMs) to select the correct answer using only the vision (V) and options (O) as inputs, without the need for the question (Q). Through grounding experiments, we attribute the bias to an imbalance in visual relevance: the correct answer typically aligns more closely with the visual contents than the negative options in feature space, creating a shortcut for VLMs to infer the answer via simply vision-option similarity matching. To fix this, we introduce GroundAttack, a toolkit that automatically generates hard negative options as visually plausible as the correct answer. We apply it to the NExT-QA and MMStar datasets, creating new EOB-free annotations. On these EOB-free annotations, current VLMs approach to random accuracies under (V+O) settings, and drop to non-saturated accuracies under (V+Q+O) settings, providing a more realistic evaluation of VLMs' QA ability. Codes and new annotations will be released soon.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 45 canonical work pages

  1. [1]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA : V isual Q uestion A nswering. In International Conference on Computer Vision (ICCV) , 2015

    work page 2015

  2. [2]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608--3617, 2018

    work page 2018

  3. [3]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6700--6709, 2019

    work page 2019

  4. [4]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 9127--9134, 2019

    work page 2019

  5. [5]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9777--9786, 2021

    work page 2021

  6. [6]

    Star: A benchmark for situated reasoning in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Josh Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , volume 1, 2021

    work page 2021

  7. [7]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for exp...

    work page 2024

  8. [8]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330 , 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330 , 2024

    arXiv 2024

Show all 53 references
  1. [9]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  2. [10]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition , pages 3195--3204, 2019

    2019

  3. [11]

    Physreason: A comprehensive benchmark towards physics-based reasoning

    Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. Physreason: A comprehensive benchmark towards physics-based reasoning. arXiv preprint arXiv:2502.12054 , 2025

    2025 arXiv

  4. [12]

    Neuro symbolic knowledge reasoning for procedural video question answering

    Thanh-Son Nguyen, Hong Yang, Tzeh Yuan Neoh, Hao Zhang, Ee Yeo Keat, and Basura Fernando. Neuro symbolic knowledge reasoning for procedural video question answering. arXiv preprint arXiv:2503.14957 , 2025

    2025

  5. [13]

    Causalchaos! dataset for comprehensive causal action question answering over longer causal chains grounded in dynamic visual scenes

    Paritosh Parmar, Eric Peh, Ruirui Chen, Ting En Lam, Yuhan Chen, Elston Tan, and Basura Fernando. Causalchaos! dataset for comprehensive causal action question answering over longer causal chains grounded in dynamic visual scenes. Advances in Neural Information Processing Syst...

    2024

  6. [14]

    Dissecting multimodality in V ideo QA transformer models by impairing modality fusion

    Ishaan Singh Rawal, Alexander Matyasko, Shantanu Jaiswal, Basura Fernando, and Cheston Tan. Dissecting multimodality in V ideo QA transformer models by impairing modality fusion. In Proceedings of the 41st International Conference on Machine Learning , volume 235, pages 42213-...

    2024

  7. [15]

    Shortcut learning in deep neural networks

    Robert Geirhos, J \"o rn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665--673, 2020

    2020

  8. [16]

    Being negative but constructively: Lessons learnt from creating better visual question answering datasets

    Wei-Lun Chao, Hexiang Hu, and Fei Sha. Being negative but constructively: Lessons learnt from creating better visual question answering datasets. arXiv preprint arXiv:1704.07121 , 2017

    2017 arXiv

  9. [17]

    What gives the answer away? question answering bias analysis on video qa datasets

    Jianing Yang, Yuying Zhu, Yongxin Wang, Ruitao Yi, Amir Zadeh, and Louis-Philippe Morency. What gives the answer away? question answering bias analysis on video qa datasets. arXiv preprint arXiv:2007.03626 , 2020

    2007 arXiv

  10. [18]

    Explicit bias discovery in visual question answering models

    Varun Manjunatha, Nirat Saini, and Larry S Davis. Explicit bias discovery in visual question answering models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9562--9571, 2019

    2019

  11. [19]

    Rubi: Reducing unimodal biases for visual question answering

    Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems , 32, 2019

    2019

  12. [20]

    Don't take the easy way out: Ensemble based methods for avoiding known dataset biases

    Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv:1909.03683 , 2019

    1909 arXiv

  13. [21]

    Video question answering: Datasets, algorithms and challenges

    Yaoyao Zhong, Junbin Xiao, Wei Ji, Yicong Li, Weihong Deng, and Tat-Seng Chua. Video question answering: Datasets, algorithms and challenges. arXiv preprint arXiv:2203.01225 , 2022

    2022 arXiv

  14. [22]

    Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering

    Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1574--1583, 2021

    2021

  15. [23]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904--6913, 2017

    2017

  16. [24]

    Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International conference on learning representations , 2018

    2018

  17. [25]

    Don't just assume; look and answer: Overcoming priors for visual question answering

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4971--4980, 2018

    2018

  18. [26]

    Language prior is not the only shortcut: A benchmark for shortcut learning in VQA

    Qingyi Si, Fandong Meng, Mingyu Zheng, Zheng Lin, Yuanxin Liu, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. Language prior is not the only shortcut: A benchmark for shortcut learning in VQA . In Findings of the Association for Computational Linguistics: EMNLP 2022 , 2022

    2022

  19. [27]

    On the role of visual grounding in vqa

    Daniel Reich and Tanja Schultz. On the role of visual grounding in vqa. arXiv preprint arXiv:2406.18253 , 2024

    2024 arXiv

  20. [28]

    Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Conference on Neural Information Processing Syst...

    2024

  21. [29]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075 , 2024

    2024 arXiv

  22. [30]

    Grok-1.5v: Multimodal model with visual understanding

    xAI. Grok-1.5v: Multimodal model with visual understanding. https://x.ai/news/grok-1.5v, 2024

    2024

  23. [31]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In CVPR , pages 13299--13308, 2024

    2024

  24. [32]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025

    2025 arXiv

  25. [33]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 , 2024

    2024 arXiv

  26. [34]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 26689--26699, 2024

    2024

  27. [35]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 , 2024

    2024 arXiv

  28. [36]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  29. [37]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11975--11986, 2023

    2023

  30. [38]

    Disgem: Distractor generation for multiple choice questions with span masking

    Devrim C avu s o g lu, Se c il S en, and Ula s Sert. Disgem: Distractor generation for multiple choice questions with span masking. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 9714--9732, 2024

    2024

  31. [39]

    A bert-based distractor generation scheme with multi-tasking and negative answer training strategies

    Ho-Lam Chung, Ying-Hong Chan, and Yao-Chung Fan. A bert-based distractor generation scheme with multi-tasking and negative answer training strategies. arXiv preprint arXiv:2010.05384 , 2020

    2010 arXiv

  32. [40]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning , pages 12888--12900. PMLR, 2022

    2022

  33. [41]

    Recovering generalization via pre-training-like knowledge distillation for out-of-distribution visual question answering

    Yaguang Song, Xiaoshan Yang, Yaowei Wang, and Changsheng Xu. Recovering generalization via pre-training-like knowledge distillation for out-of-distribution visual question answering. IEEE Transactions on Multimedia , 26:837--851, 2023

    2023

  34. [42]

    Bias mimicking: A simple sampling approach for bias mitigation

    Maan Qraitem, Kate Saenko, and Bryan A Plummer. Bias mimicking: A simple sampling approach for bias mitigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20311--20320, 2023

    2023

  35. [43]

    Reassessing evaluation practices in visual question answering: A case study on out-of-distribution generalization

    Aishwarya Agrawal, Ivana Kaji \'c , Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, and Aida Nematzadeh. Reassessing evaluation practices in visual question answering: A case study on out-of-distribution generalization. arXiv preprint arXiv:2205.12191 , 2022

    2022 arXiv

  36. [44]

    Human-adversarial visual question answering

    Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Magana, Tristan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Human-adversarial visual question answering. Advances in Neural Information Processing Systems , 34:20346--20359, 2021

    2021

  37. [45]

    Adversarial vqa: A new benchmark for evaluating the robustness of vqa models

    Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. Adversarial vqa: A new benchmark for evaluating the robustness of vqa models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2042--2051, 2021

  38. [46]

    On evaluating adversarial robustness of large vision-language models

    Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems , 36:54111--54138, 2023

    2023

  39. [47]

    Next-ood: Overcoming dual multiple-choice vqa biases

    Xi Zhang, Feifei Zhang, and Changsheng Xu. Next-ood: Overcoming dual multiple-choice vqa biases. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(4):1913--1931, 2023

    1913

  40. [48]

    Dataset bias mitigation in multiple-choice visual question answering and beyond

    Zhecan Wang, Long Chen, Haoxuan You, Keyang Xu, Yicheng He, Wenhao Li, Noel Codella, Kai-Wei Chang, and Shih-Fu Chang. Dataset bias mitigation in multiple-choice visual question answering and beyond. arXiv preprint arXiv:2310.14670 , 2023

    2023 arXiv

  41. [49]

    Robust visual question answering: Datasets, methods, and future challenges

    Jie Ma, Pinghui Wang, Dechen Kong, Zewei Wang, Jun Liu, Hongbin Pei, and Junzhou Zhao. Robust visual question answering: Datasets, methods, and future challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024

    2024

  42. [50]

    A closer look at the robustness of vision-and-language pre-trained models

    Linjie Li, Zhe Gan, and Jingjing Liu. A closer look at the robustness of vision-and-language pre-trained models. arXiv preprint arXiv:2012.08673 , 2020

    2012 arXiv

  43. [51]

    Enhancing robust vqa via contrastive and self-supervised learning

    Runlin Cao, Zhixin Li, Zhenjun Tang, Canlong Zhang, and Huifang Ma. Enhancing robust vqa via contrastive and self-supervised learning. Pattern Recognition , 159:111129, 2025

    2025

  44. [52]

    Unsupervised distractor generation via large language model distilling and counterfactual contrastive decoding

    Fanyi Qu, Hao Sun, and Yunfang Wu. Unsupervised distractor generation via large language model distilling and counterfactual contrastive decoding. arXiv preprint arXiv:2406.01306 , 2024

    2024 arXiv

  45. [53]

    Break, perturb, build: Automatic perturbation of reasoning paths through question decomposition

    Mor Geva, Tomer Wolfson, and Jonathan Berant. Break, perturb, build: Automatic perturbation of reasoning paths through question decomposition. Transactions of the Association for Computational Linguistics , 10:111--126, 2022

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.