Pith. sign in

REVIEW 3 major objections 5 minor 49 references

Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Counterfactual images show VLM priors persist despite extra visual evidence

desk verdict A well-built counterfactual benchmark with a genuinely new idea—controlled tool-evidence conditions—but the headline closed-vs-open split is confounded with different text-only judges and needs a single-judge re-scoring before it can be believed. read the letter →

arxiv 2607.16311 v1 pith:MCSKYJ3U submitted 2026-07-14 cs.CV cs.AI

Jingyu Sun , Jiachen Tu , Yuyang Xue , Yaoxin Jiang , Guoyi Xu , Zhengtao Yao , Rui Qian , Yizheng Sun
show 3 more authors
Hongpeng Zhou Jingyuan Sun Yan Lin
This is my paper
classification cs.CVcs.AI
keywords vision-languagemodelscounterfactualimagesprior-followingvisualgroundingagenticvisiontool-augmentedVLMsbenchmarkPFER
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether showing a vision-language model more visual evidence can make it stop relying on learned priors. The authors build a benchmark of paired original and counterfactual images that contradict common expectations, and a controlled tool-condition extension that adds bounding boxes, crops, zoom panels, and contours. Across eight models, counterfactual inputs cause systematic prior-following errors rather than ambiguous failures. The central finding is that tool-derived evidence helps closed-source models substantially but has weak or negative effects for most open-source models, so exposing relevant evidence is not sufficient: the model must also be able to integrate localized evidence against its priors. This matters because agentic vision systems increasingly rely on such tools.

What carries the argument

The central object is the counterfactual image pair: an original image plus a targeted intervention that violates a canonical expectation (missing part, changed count, altered logo, swapped MRI region). This pair makes the prior–evidence conflict explicit and allows responses to be classified as correct, biased (prior-following), or other. PriVE-Tools then varies only the evidence view — bounding box, crop, zoom panel, or contour — while keeping image, question, correct answer, biased answer, and rubric fixed, isolating the effect of the evidence presentation itself. The key metric is PFER (prior-following error rate), which distinguishes prior-consistent errors from ambiguity.

What would settle it

Re-score a random subset of open-source model responses using the judge used for closed-source models and recompute the tool deltas; if the open-source deltas turn positive under the other judge, the model-family split is a scoring artifact. Alternatively, run the tool conditions with automatically generated (non-manually-filtered) evidence to test whether the oracle-like selection explains the gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that counterfactual images expose a systematic prior-following failure mode in VLMs, and that additional visual evidence does not guarantee grounding. Across five domains, counterfactual-only accuracy drops sharply while prior-following error rate (PFER) dominates, meaning models commit to canonical wrong answers rather than hedging. Paired images help most models but not all. Tool-derived evidence, such as crops and zoom panels, improves accuracy and reduces PFER for closed-source models (up to +11.8 points accuracy and −10.6 PFER) but the same views are weak or negative for open-source models, sometimes shifting errors from bias to uncertainty. The paper conclu

Load-bearing premise

The headline closed-versus-open split rests on two different text-only judges scoring the responses; if those judges differ in strictness or prior alignment, the aggregate difference in tool effectiveness could be a measurement artifact rather than a property of the models.

Editorial extensions

If this is right

  • Agentic vision systems cannot assume that supplying more visual evidence will correct prior-driven answers; they need mechanisms that force models to ground responses in the evidence.
  • Evaluation of VLMs on counterfactual inputs should report PFER separately from accuracy, because high accuracy on canonical images can mask prior-following.
  • Paired-image comparison helps in several domains (counting, fashion, industry, medical) but hurts object-existence judgments, so comparison is not a universal remedy.
  • Tool usefulness is domain-specific: local crops help industry attribute edits, zoom panels help medical modality consistency, and neither solves counting; evidence design must match the task.
  • The closed-versus-open split suggests models differ in their ability to integrate localized evidence, pointing to a concrete capability that could be targeted by training or prompting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The closed-versus-open difference in tool deltas may be inflated by the use of different text-only judges for the two model groups; if those judges differ in strictness, the aggregate split could be partly a scoring artifact rather than a pure capability difference.
  • Because evidence views were manually filtered to ensure they highlight the target attribute, the tool condition is partly oracle-like; real agents that must select or segment evidence automatically may see smaller gains.
  • A direct test of the paper's claim: re-score open-source responses with the closed-source judge and see whether tool deltas change sign, or train an open-source model with a grounding objective and check whether the same tool views become effective.
  • The medical results suggest interventions can trade bias for uncertainty (lower PFER, higher Other rate); for high-stakes applications, moving a model from wrong-but-confident to uncertain-but-correct may still be a safety improvement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces PriVE-Bench, a counterfactual benchmark with paired original/counterfactual images across five domains (object existence, counting, fashion, industry attributes, medical modality), and PriVE-Tools, which augments counterfactual inputs with controlled tool-derived evidence views (bounding boxes, crops, zoom panels, contours). Eight open- and closed-source VLMs are evaluated under original-only, counterfactual-only, paired-image, and tool conditions using accuracy, prior-following error rate (PFER), and other-response rate. The central claim is that counterfactual images expose systematic prior-following failures across models, that paired images help only partially, and that additional localized visual evidence is not a universal remedy: it improves grounding for GPT-5 and Claude Sonnet 4 (e.g., zoom ΔAcc up to +11.8, ΔPFER −10.6) but is weak or negative for most open-source models (e.g., Qwen3-VL-32B crop ΔAcc −2.2, ΔPFER +2.9). The authors frame PriVE-Tools as a controlled diagnostic rather than an end-to-end agentic evaluation, and they acknowledge oracle-like evidence generation and limited tool coverage.

Significance. If the central claims hold, PriVE-Bench and PriVE-Tools are a valuable contribution: the paired counterfactual design with predefined correct/biased labels cleanly separates visually grounded answers from prior-consistent errors, and the PFER/Other distinction prevents conflating ambiguity with prior-following. The domain diversity, including a carefully constructed MRI modality-swap sub-benchmark, is a strength, as is the matched raw-versus-tool comparison protocol. The headline finding that localized evidence helps closed-source models but not most open-source models, if confirmed, has direct implications for the design of tool-augmented and agentic vision systems. However, that specific model-family split currently rests on a comparison across two different text-only judges, and the absence of uncertainty quantification makes small deltas hard to interpret. The raw prior-following phenomenon (high PFER, large counterfactual accuracy drops) is more robust and less threatened by these concerns.

major comments (3)
  1. [Appendix D, Table 4; Section 5; RQ3 (Table 1, Tables 12–13)] The central closed-vs-open tool split is confounded with judge choice: yes/no and open-ended responses are scored by GPT-4o-mini for GPT-5/Claude and by Qwen3-8B for the six open-source models, with no judge agreement or cross-judge calibration reported. Since the split is exactly the load-bearing RQ3 result (e.g., Claude Zoom +11.8/−10.6 vs. Qwen3-VL-32B Crop −2.2/+2.9), the group-level difference could be a scoring artifact rather than a model property. Please rescore all tool-conditioned and baseline responses with a single judge (or at least with one fixed judge per model plus agreement statistics), and verify that the closed/open ΔAcc and ΔPFER pattern persists.
  2. [Section 5, Eqs. (1)–(6); Tables 1, 12–13] All reported deltas are point estimates with no confidence intervals, significance tests, or per-delta sample sizes. Deltas in the range of 1–5 percentage points (e.g., GPT-5 Crop +4.5/−4.3, Qwen3-VL-8B Zoom −1.4/+1.2, InternVL3.5-8B Zoom +0.5/−5.8) are small relative to macro-averaging across domains with 160–455 examples each (Table 2), and the matched evidence-valid subset denominators are not stated. Without paired bootstrap intervals or significance tests, RQ2/RQ3 conclusions such as ‘paired images help seven of eight models’ or ‘tools help closed but not open models’ are difficult to distinguish from sampling noise. Please report uncertainty estimates for the paired deltas.
  3. [Section 4; Section 8 (Limitations)] The tool evidence views are manually filtered so that the highlighted region corresponds to the manipulated target attribute, and the medical evidence comes directly from BraTS segmentation labels. This is acknowledged as oracle-like in the Limitations, but the acknowledged worry is still load-bearing for the positive tool claims: the closed-source gains may reflect receiving a human/annotation-selected region rather than any realistic tool output. The negative open-source result is less affected, since even oracle-like evidence fails to help, but the paper should quantify the fraction of examples filtered during evidence generation and, if feasible, include an unfiltered or automatically segmented condition to show how much of the positive effect depends on the oracle selection.
minor comments (5)
  1. [Abstract / Section 8] The paper states that the benchmark and codebase ‘can be accessed at counterfactual-vlm-benchmark’ but no actual URL, repository identifier, or data-release link is provided. Please include the full link and, ideally, a dataset card and evaluation code in the final version.
  2. [Table 1] The heading ‘Pair ΔA/P’ is unclear at first glance. Write ‘Pair ΔAcc/ΔPFER’ and define the notation in the caption.
  3. [Section 6] Typo: ‘Claude Sonnet 4. 1 The open-source models...’ should read ‘Claude Sonnet 4. The open-source models...’.
  4. [Appendix G, Table 14] The BBox coverage is incomplete for Fashion and Industry, and those cells are blank. It would help to state explicitly whether ‘empty’ means ‘not run’ as opposed to ‘not applicable’ in the caption, since a reader may otherwise infer an invalid or zero result.
  5. [Section 5 / Appendix D] The prior-conflict prime is always enabled in the main evaluation. This is a reasonable stress-test choice, but it should be stated more prominently in the main text that PriVE-Bench measures grounding under explicitly primed priors, not naturalistic prompt behavior.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark measures external VLM behavior against author-defined rubrics; the two main caveats (oracle-like evidence and split judges) are acknowledged limitations or measurement risks, not input-equivalence.

full rationale

PriVE-Bench and PriVE-Tools form an empirical benchmark paper, not a derivation chain. The only equations (1)-(6) define sample proportions and matched deltas; no parameter is fitted and no reported quantity is mathematically entailed by another by construction. The correct/biased labels are author-defined rubrics, and PFER is by definition the rate of biased responses, but the paper's claims are measurements of external VLM outputs against those rubrics, not predictions derived from the rubrics. There are no self-citations and no imported uniqueness or ansatz results. Two validity caveats appear in the paper itself and are correctly disclosed: (i) Section 8 acknowledges the tool evidence is 'partly oracle-like' because regions are generated around known target regions, which limits generalization of tool-condition effects to real agents but is a transparent input-construction choice, not a fitted parameter renamed as a prediction; (ii) Appendix D Table 4 uses GPT-4o-mini as the text-only judge for closed-source runs and Qwen3-8B for open-source runs, so the headline closed-vs-open tool split is statistically confounded with judge identity. This is a correctness/measurement risk rather than circularity: the deltas are still computed from actual model responses, and no equation reduces the conclusion to the judge assignment. The oracle filter, if anything, makes the negative open-source result conservative. No circular step meets the required evidence bar.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The paper contributes a benchmark and an evaluation protocol; there are no fitted physical parameters. The hand-chosen evaluation settings (judge choice, prior-conflict prime, medical subset size, macro-averaging) and the domain assumptions about image resolvability and judge reliability carry the central claim. The benchmark itself is an invented evaluation entity without independent falsifiable handles.

free parameters (4)
  • Medical evaluation subset ratio = 10% (125 per swap direction)
    Hand-chosen to cut evaluation cost; the main medical results come from a stratified 10% subset of 2,502 constructed records, so domain-level magnitudes may not reflect the full pool.
  • Prior-conflict prime = On (main setting)
    Hand-chosen to make canonical priors salient before the visual question; this turns PriVE-Bench into an explicit stress test and directly affects absolute PFER and tool deltas.
  • Judge model per model family = GPT-4o-mini (closed-source), Qwen3-8B (open-source)
    Hand-chosen; different judge models for the two groups create a potential measurement confound in cross-family tool comparisons.
  • Macro-averaging across domains = Equal weight per domain
    Hand-chosen; treats all five domains equally even though medical has 250 of 1,469 evaluation records, affecting aggregate deltas.
assumptions (4)
  • domain assumption Counterfactual interventions are visually resolvable — the edited attribute is actually apparent in the pixels.
    If editing/generation artifacts make the change invisible or confounded, the 'correct' vs 'biased' labels lose meaning. The authors acknowledge this in Limitations (Section 8).
  • domain assumption Text-only LLM judges produce reliable labels without seeing the image.
    The judge receives the question, response, and rubrics but not the image (Appendix D). Judge errors remain possible, and no agreement stats are given.
  • domain assumption The five chosen domains are representative of VLM prior-following behavior.
    Generalization from object existence, counting, fashion, industry attributes, and medical modality to other VLM use cases is assumed.
  • domain assumption AI-generated images do not fully inherit generator priors that confound the diagnostic.
    AI-generated non-medical images (e.g., via GPT Image 1.5) may themselves reflect visual priors of the generators; the authors state this as a limitation rather than controlling for it.
invented entities (2)
  • PriVE-Bench dataset
    purpose: Paired original/counterfactual image benchmark across five domains to elicit and measure prior-following behavior.
    The benchmark is an evaluation artifact; it provides no falsifiable handle outside the authors' own results. Correctness depends on the authors' construction and rubrics.
  • PriVE-Tools evidence conditions
    purpose: Controlled tool-derived visual views (bbox, crop, zoom, contour) added to inputs to test whether extra evidence reduces prior-following.
    These are controlled input conditions, not independently testable entities; their usefulness is the empirical object of study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs." pith.science (2026). https://pith.science/paper/MCSKYJ3U

@misc{pith2026260716311,
  author       = {Pith},
  title        = {Pith review of: Seeing What Is Actually There: PriVE-Bench and PriVE-Tools for Counterfactual Evaluation of Agentic Visual Evidence in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MCSKYJ3U}},
  note         = {Machine review of arXiv:2607.16311}
}
read the original abstract

Vision-language models (VLMs) often answer visual questions using learned language and category priors rather than grounding their predictions in the image itself. Counterfactual images provide a natural diagnostic setting for this failure mode: when visible evidence contradicts what is usually true, a grounded model should answer from the pixels, while a prior-following model will produce a canonical but visually incorrect response. However, existing counterfactual benchmarks mainly ask whether such prior-following behavior exists. In this paper, we ask a further question motivated by the rise of tool-augmented and agentic vision systems: can additional visual evidence views help VLMs reason against their priors? We introduce PriVE-Bench, a Prior-vs-Visual Evidence Benchmark that uses paired original and counterfactual images to distinguish visually grounded answers from prior-consistent errors. We further introduce PriVE-Tools, a controlled agentic-vision-inspired extension that evaluates whether tool-derived visual evidence -- including bounding boxes, crops, zoom panels, and contours -- improves grounding under the same counterfactual conflicts. Across open- and closed-source VLMs, we compare raw, paired-image, and tool-conditioned inputs using accuracy, prior-following error rate, and other-response rate. Our results show that visual evidence tools can help in some settings, especially when models can use localized evidence effectively, but they are not a universal remedy: several models continue to follow language and category priors even when relevant visual evidence is explicitly provided.

Figures

Figures reproduced from arXiv: 2607.16311 by the authors.

Figure 1
Figure 1. Motivation of PriVE-Bench and PriVE￾Tools. Top: Counterfactual images reveal whether VLMs answer from visible evidence or learned lan￾guage/category priors. Bottom: PriVE-Tools tests whether controlled agentic-vision-inspired evidence views help models recover from prior-following errors. provided. The benchmark and codebase can be accessed at counterfactual-vlm-benchmark. 1 Introduction Vision-language models (VLMs… view at source ↗
Figure 2
Figure 2. Representative PriVE-Bench examples and PriVE-Tools visual evidence. Rows correspond to the five benchmark domains. Columns show the original image, counterfactual image, and three visual-evidence views: localization, crop, and zoom panel. counterfactual editing, without assuming that either web-collected or generated images are guaranteed unseen or unbiased. Each instance records source metadata, intervention type,… view at source ↗
Figure 3
Figure 3. Dataset Generation and Question-Format Design Example for PriVE-Bench and PriVE-Tools. The same counting counterfactual image is evaluated with three question formats: yes/no, multiple choice, and open-ended response. All formats share the same target attribute, visible wing count, and the same prior conflict. test whether models can describe or infer the vis￾ible evidence without fixed options. The bench￾mark is ev… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Original and counterfactual counts by domain in the formal evaluation set. Counts are computed from active benchmark metadata. Counting reuses some originals across multiple edits, while medical modality reports the stratified evaluation subset; the full constructed me…
Figure 5
Figure 5. Figure 5: Source composition of original images. Non-medical domains combine web-collected or real images with AI-generated images to balance natural visual diversity and controllable counterfactual construction. Medical examples are derived from BraTS2023 MRI data and are repor…
Figure 6
Figure 6. Figure 6: Original-only and counterfactual-only raw-image performance. Panel A compares answerability under original images with performance under counterfactual images. Panel B decomposes counterfactual-only responses into correct, prior-following, and other labels. The large P…
Figure 7
Figure 7. Figure 7: Heatmap of paired-image deltas by model and domain. Panel A reports ∆Acc and Panel B reports ∆PFER relative to counterfactual-only inputs. Positive ∆Acc and negative ∆PFER indicate improved visual grounding through direct comparison. Tool Coverage All models Closed-sou…
Figure 8
Figure 8. Figure 8: Macro-averaged effects of common PriVE-Tools evidence conditions across models and domains. Bars report ∆Acc, ∆PFER, and ∆Other relative to matched raw counterfactual baselines. Positive ∆Acc and negative ∆PFER indicate improved grounding. domain-specific evidence view…
Figure 9
Figure 9. Figure 9: Domain-by-tool heatmaps for displayed PriVE-Tools evidence conditions. Panel A reports ∆Acc and Panel B reports ∆PFER relative to matched raw counterfactual baselines. Fashion and Industry BBox cells are left blank because open-source bbox runs were not completed [PIT…
Figure 10
Figure 10. Figure 10: Model-by-tool heatmaps for displayed PriVE-Tools evidence conditions. Panel A reports ∆Acc and Panel B reports ∆PFER. BBox values are macro-averaged over domains with complete bbox coverage across all models. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 12 linked inside Pith

  1. [1]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  2. [2]

    Advances in neural information processing systems , volume=

    Flamingo: a visual language model for few-shot learning , author=. Advances in neural information processing systems , volume=

  3. [3]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  4. [4]

    Advances in neural information processing systems , volume=

    Visual instruction tuning , author=. Advances in neural information processing systems , volume=

  5. [5]

    Proceedings of the IEEE international conference on computer vision , pages=

    Vqa: Visual question answering , author=. Proceedings of the IEEE international conference on computer vision , pages=

  6. [6]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Don't just assume; look and answer: Overcoming priors for visual question answering , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

  7. [7]

    Advances in neural information processing systems , volume=

    Rubi: Reducing unimodal biases for visual question answering , author=. Advances in neural information processing systems , volume=

  8. [8]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Vlind-bench: Measuring language priors in large vision-language models , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

Show all 49 references
  1. [11]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Segment anything , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  2. [15]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    V?: Guided visual search as a core mechanism in multimodal llms , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  3. [17]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  4. [18]

    Advances in Neural Information Processing Systems , volume=

    Are we on the right way for evaluating large vision-language models? , author=. Advances in Neural Information Processing Systems , volume=

  5. [19]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  6. [20]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Words or vision: Do vision-language models have blind faith in text? , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  7. [23]

    IEEE transactions on medical imaging , volume=

    The multimodal brain tumor image segmentation benchmark (BRATS) , author=. IEEE transactions on medical imaging , volume=. 2014 , publisher=

    2014

  8. [25]

    ArXiv , pages=

    The brain tumor segmentation (brats) challenge 2023: Brain mr image synthesis for tumor segmentation (brasyn) , author=. ArXiv , pages=

    2023

  9. [28]

    ArXiv , year=

    Gemma 3 Technical Report , author=. ArXiv , year=

  10. [29]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    G-eval: NLG evaluation using gpt-4 with better human alignment , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  11. [30]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  12. [31]

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971--4980

    2018

  13. [32]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing syst...

    2022

  14. [33]

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433

    2015

  15. [34]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631

    2025 arXiv

  16. [35]

    Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, and 1 others. 2021. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic cla...

    2021 arXiv

  17. [36]

    Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. 2023. Breaking common sense: Whoops! a vision-and-language benchmark of synthetic and compositional images. In Proceedings of the IEEE/CVF International Confe...

    2023

  18. [37]

    Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, and 1 others. 2019. Rubi: Reducing unimodal biases for visual question answering. Advances in neural information processing systems, 32

    2019

  19. [38]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056--27087

    2024

  20. [39]

    Ailin Deng, Tri Cao, Zhirui Chen, and Bryan Hooi. 2025. Words or vision: Do vision-language models have blind faith in text? In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3867--3876

    2025

  21. [40]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language mod...

    2024

  22. [41]

    Gemma Team Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram'e, Morgane Rivi \`e re, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casb...

    2025 arXiv

  23. [42]

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, and 1 others. 2023. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015--4026

    2023

  24. [43]

    Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. Vlind-bench: Measuring language priors in large vision-language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4129--4144

    2025

  25. [44]

    Hongwei Bran Li, Gian Marco Conte, Qingqiao Hu, Syed Muhammad Anwar, Florian Kofler, Ivan Ezhov, Koen van Leemput, Marie Piraud, Maria Diaz, Byrone Cole, and 1 others. 2024. The brain tumor segmentation (brats) challenge 2023: Brain mr image synthesis for tumor segmentation (b...

    2024

  26. [45]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  27. [46]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 a . Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  28. [47]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 2511--2522

    2023

  29. [48]

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. 2024. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403.12966

    2024 arXiv

  30. [49]

    Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. 2024. Probing visual language priors in vlms. arXiv preprint arXiv:2501.00569

    2024 arXiv

  31. [50]

    Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, and 1 others. 2014. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical ...

    2014

  32. [51]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine...

    2021

  33. [52]

    An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. 2025. Vision language models are biased. arXiv preprint arXiv:2505.23941

    2025 arXiv

  34. [53]

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265

    2025 arXiv

  35. [54]

    Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, and 1 others. 2024. Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:2409.15310

    2024 arXiv

  36. [55]

    Penghao Wu and Saining Xie. 2024. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13084--13094

    2024

  37. [56]

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023 a . Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441

    2023 arXiv

  38. [57]

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023 b . Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381

    2023 arXiv

  39. [58]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629

    2022 arXiv

  40. [59]

    Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. 2023. Towards perceiving small visual details in zero-shot visual question answering with multimodal llms. arXiv preprint arXiv:2310.16033

    2023 arXiv

  41. [60]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595--46623

    2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.