Pith. sign in

REVIEW 4 major objections 5 minor 61 references

SABRE: Scalable and Automated Benchmarking of VLMs under Stress

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SABRE automates VLM stress-test construction; its world-prior benchmark leaves six frontier models at 17.8–31.3% macro accuracy.

desk verdict A genuinely useful benchmark-construction pipeline whose headline difficulty numbers rest on an unmeasured human-verification step; worth serious review but only as a provisional contribution until data and reliability checks are released. read the letter →

arxiv 2608.07435 v1 pith:SAOT6P6Z submitted 2026-08-07 cs.CV cs.AIcs.CL

Zixuan Lan , Luzhe Sun , Matthew R. Walter , Jiawei Zhou This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords VLMstresstestingbenchmarkconstructionpipelineworldpriorsvisualgroundingmodel-in-the-loopfilteringhumanverificationimageeditingvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that VLM stress-test benchmarks can be produced by a reusable pipeline rather than by hand-curated fixed sets. The pipeline, SABRE, takes a one-page Markdown Test Primer and automatically produces structured sample specifications, generated or edited images, and question-answer pairs, then discards candidates that a filtering VLM answers correctly and sends the rest to human reviewers for verification and localized repair. In its first full instantiation, SABRE-Prior, built to test whether models follow visual evidence when it contradicts world priors, six frontier VLMs score between 17.8% and 31.3% macro accuracy, with a mean of 22.6%. Two small pilots for counting and spatial reasoning use the same workflow with different task descriptions, supporting the claim that the framework generalizes. A sympathetic reader should care because it argues that benchmark construction can keep pace with model development and expose weaknesses that saturated fixed benchmarks miss.

What carries the argument

The load-bearing mechanism is the model-in-the-loop pressure screen combined with human verification. A Filtering VLM evaluates each candidate generated from a schema-validated sample specification; any candidate it answers correctly is discarded, so retained candidates are ones the filter fails. Because failure alone does not prove validity, each retained candidate must then pass human review of the image, question, and reference answer, with support for editing the question, correcting the reference answer, and repairing local image defects through a patch-based soft-blend tool. The paired Base-Edited design in Context and Texture uses four yes/no probes per case, scored only if all four are correct, and isolates whether the model updates its answer after a controlled visual intervention. Attribute and Language Elicitation apply the same pressure screen to open-ended counting and four-option multiple-choice formats, showing that the screening works across question types.

What would settle it

Take a random sample of SABRE-Prior cases and have two independent reviewer teams, one blind to the filtering model's predictions and reference answers, re-verify the images; if a substantial fraction of reference answers are judged wrong or ambiguous, the reported 17.8%–31.3% range would overstate genuine model failure. Alternatively, rebuild the benchmark with a different Filtering VLM; if model scores rise sharply, the difficulty is filter-specific rather than general.

Watch

Extended reading notes

Core claim

The central claim is that benchmark construction itself can be mechanized: from a natural-language task design plus a data schema, SABRE generates candidate samples, pressure-filters them by discarding any that a Filtering VLM gets right, and retains only candidates that human reviewers confirm are valid, where validity means the required visual evidence is present, the edit is correctly applied, the question is unambiguous, and the reference answer matches the image. Using this workflow, SABRE-Prior places unexpected objects in familiar scenes (Context), gives objects counterfactual materials (Texture), alters canonical component counts (Attribute), and asks questions whose wording suggests an answer the image cannot support (Language Elicitation). Across six frontier VLMs, macro-average accuracy is 17.8%–31.3%, and a real-image Attribute control is comparably hard for the filtering model, which the authors take as evidence that the difficulty is not an artifact of generated images. The two additional pilots demonstrate that the same pipeline, with different Test Primers, produces challenging counting and spatial-reasoning tests, establishing SABRE as a reusable framework rather than a single fixed benchmark.

Load-bearing premise

The benchmark's validity rests on human reviewers giving correct reference answers during verification; the paper reports no inter-annotator agreement and no test of whether showing reviewers the filtering model's predictions biased their decisions.

Editorial extensions

If this is right

  • SABRE-Prior's macro accuracy of 17.8%–31.3% across six frontier VLMs implies that current state-of-the-art models systematically fall back on world priors when visual evidence contradicts them, at least on these screened cases.
  • The real-image Attribute control (30% vs 26% for the filtering model) implies that the low scores are not primarily generated-image artifacts.
  • VCD and SoM, two visual-enhancement methods, do not improve Qwen 3.5 27B's macro-average on SABRE-Prior (19.5% and 16.8% vs 23.0%), implying that these failures resist generic inference-time fixes.
  • Counting and Spatial pilots, on which all six models score near zero, imply that the pipeline can generate hard stress tests for new capabilities from a changed Test Primer alone.
  • Because the filter is a frontier VLM and can be swapped, the pipeline can refresh benchmarks as models improve, rather than being frozen at release time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: Since SABRE-Prior screens with one filter model, the benchmark may be biased toward cases that happen to fool that particular VLM; re-running with a different filter could yield different subsets and different difficulty levels, an implicit consequence the paper does not test.
  • Inference: The reliance on human verification with no reported inter-annotator agreement means the true validity-error rate of the benchmark is unknown; a blinded re-annotation study would test whether reference answers are unbiased.
  • Inference: The pipeline's illustrative cost estimate, roughly $43-$55 of API cost and 1.5-2.9 hours of human review per 100 retained Context cases, suggests that continuously refreshing benchmarks against each new model generation is economically plausible, not just technically possible.
  • Inference: If the world-prior failure pattern persists across refreshed instantiations, it would suggest a structural bias in VLM training, optimizing for predictive priors over image-grounded evidence, rather than a benchmark quirk.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SABRE, a pipeline that converts a natural-language Test Primer and data schema into structured sample specifications, generates or edits images, builds question-answer pairs, filters out candidates that a Filtering VLM answers correctly, and sends the remaining candidates through human verification with localized image repair. The authors instantiate SABRE-Prior, a 600-image, 1,000-question benchmark with Context, Texture, Attribute, and Language Elicitation subsets, and report that six frontier VLMs score between 17.8% and 31.3% macro accuracy. They also report a real-image Attribute control, two 20-sample pilots for Counting and Spatial reasoning, and a user study of the repair module, and argue that SABRE is a reusable framework rather than a single fixed benchmark.

Significance. If the central claims hold, the paper makes a useful contribution: it proposes a modular, scalable pipeline for constructing controlled VLM stress tests, and the SABRE-Prior instantiation covers several distinct prior-conflict phenomena with a shared verification workflow. The paper is creditable for acknowledging that the Filtering VLM's own low score is partly by construction, for including a real-image control, for publishing illustrative cost and human-time estimates, and for providing detailed appendices on the sample schema and annotation interface. The main risk is that the human verification stage is the only quality gate for reference answers, and its reliability is not established; because the empirical claim that models fail to follow visual evidence depends on the correctness of those reference answers, this is a load-bearing gap. The two 20-sample pilots and the 20-session user study are small but are presented as pilot evidence, which is acceptable if their limitations are stated clearly.

major comments (4)
  1. [§3.5, Appendix D.1, Figure 9] The paper reports no inter-annotator agreement, no reviewer calibration, and no independent audit of the final 1,000 samples, even though human verification is the sole quality gate between pressure-filtered candidates and the final benchmark. The annotation interface displays the intended reference answer, the filtering model's prediction, and whether that prediction was correct, which can bias reviewers on borderline cases; a ground-truth error rate of even 5–10% could account for a substantial fraction of the reported 17.8–31.3% accuracy figures. Please report inter-annotator agreement on a held-out set, perform an independent blind re-verification of the final benchmark, and report the resulting error rate.
  2. [Table 7, Context Q3] Under the strict All4 metric, Context accuracy is near zero largely because models answer Q3 (expected 'no' for the source object in the Edited image) with 'yes' at rates of 0–33%. The paper interprets this as failure to suppress the world prior, but Appendix D.2 states that edit models 'may leave visible remnants of the original entity,' and if any final Edited image still contains a recognizable source remnant, the intended answer is wrong and all models are penalized. Without an audit confirming the absence of source remnants in the final Context cases, the near-zero Q3 accuracy is not yet interpretable as a world-prior failure.
  3. [Table 3, §5.5] The real-image Attribute control contains only 20 cases, reports no confidence interval, and is evaluated on a single model and a single subset. The comparison of 30% versus 26% is presented as evidence that benchmark difficulty is not driven by generated-image artifacts, but the sample size is too small to support that claim statistically. Please expand the control, report confidence intervals, and ideally cover additional subsets and models before drawing this conclusion.
  4. [§4.1, §5.6, Figure 7] The Counting and Spatial pilots contain only 20 samples each, and every model scores at most 1/20 on Counting and 0/20 on Spatial. These near-floor results are too sparse to establish that the workflow 'supports other stress-test settings' without controlling for generation failures or annotation errors. At a minimum, report rejection and repair rates for these pilots, include a per-sample validity audit, and provide confidence intervals; as is, the extensibility claim rests on very thin evidence.
minor comments (5)
  1. [§3.2] The text reads 'an data schema' and should read 'a data schema.'
  2. [Figure 9 caption] The caption contains 'SABREannonation platform'; this should be 'SABRE annotation platform.'
  3. [Figure 4, §5.1] The figure caption states that whiskers are confidence intervals, but the numeric interval values are not reported anywhere; please include them in a table so that differences between models and subsets can be assessed.
  4. [Appendix D.3, Table 8] The repair-quality user study analyzes only 20 of 40 initiated sessions, a 50% completion rate; the manuscript should acknowledge this and report any available information about participant background or selection.
  5. [Appendix A, response parsing] For Context and Texture, unparseable yes/no responses are marked incorrect; please report how many responses fell into this category, since it affects the strict All4 scores.

Circularity Check

2 steps flagged · score 4.0 of 10

The Filtering VLM's own SABRE-Prior score is partly by construction, and the real-image control is uninformative if it ran through the same failure-only screening; the five unscreened models keep the central claim partially independent.

  1. self definitional [Section 3.4 Automated Filtering; Section 3.5 final benchmark definition; Section 5.2 Table 1 discussion]
    "The automated filtering retains only the candidate samples that M answers incorrectly: Cpressure(S;M) ={x i ∈ C(S)|ei = 0}. ... Each item in D(S;M) originates from a pressure-selected candidate sample and passes human verification. ... Its low score on SABRE-Prior partly reflects its role as the Filtering VLM. However, the other five models, which were not used during pressure screening, also score below 32%."

    By the paper's own definition, every final benchmark item descends from a candidate that the Filtering VLM M (Gemini 3.5 Flash) answered incorrectly. Re-evaluating M on D(S;M) therefore measures the model on a set selected to be its failures; a low Gemini score is forced or strongly biased by construction, not discovered. The paper acknowledges this, and the inclusion of Gemini in the six-model 17.8-31.3% aggregate means one of the six numbers is at least partly an artifact of the selection rule. The claim does not collapse, since the other five models were not used in screening.

  2. fitted input called prediction [Section 1 (real-image workflow); Section 5.5 Real-Image Control, Table 3]
    "The same interface also accepts uploaded real images, which pass through the same screening and curation stages. ... We construct 20 control cases from real images using the same editing and evaluation procedure as the Attribute. ... The similarly low accuracies suggest that the benchmark difficulty is not primarily caused by generated-image artifacts."

    If the real-image Attribute control 'pass[es] through the same screening ... stages,' then those 20 cases were also retained by discarding candidates the Filtering VLM answered correctly. The control is therefore a second set of Gemini failures, not an independent sample of real-image difficulty. Comparing Gemini's 30% on the real control with 26% on the generated subset cannot establish that generated-image artifacts are not responsible for the low scores: both sets were selected to be cases Gemini gets wrong. The conclusion is an artifact of the selection filter.

full rationale

The derivation chain is not globally circular. The core framework claim and the primary empirical evidence for non-filter models rest on five VLMs (Claude 4.6, Kimi-k2.6, Qwen 3.5, GPT-5.4, Grok-4.3) that were not used in pressure screening, and all five still score 17.8-31.3% macro accuracy; that part is not forced by construction. There is no load-bearing self-citation: the reference list is external, and no uniqueness theorem or ansatz is imported from the authors' prior work. The two genuine reductions concern the Filtering VLM's own reported accuracy and the real-image control: the former is explicitly disclosed, the latter is uninformative if it used the same failure-only screening. I do not count the missing inter-annotator agreement and the reviewer-interface bias (Appendix D.1 shows reviewers see reference answers, filter predictions, and correctness) as circularity; they are ground-truth validity risks that could amplify the selection bias but are not a by-construction equivalence. On balance, the centrality of the construction is partial, so score 4 rather than 0-2; 6+ would be inappropriate because the main cross-model result does not reduce to a fit or a self-citation chain.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on assumptions about image fidelity, human verification reliability, and the generalizability of filter-selected difficulty. No physical constants are fitted, but the Filtering VLM is a hand-chosen design parameter that shapes the benchmark. The paper introduces no explanatory entities such as new particles or forces.

free parameters (2)
  • Filtering VLM selection = Gemini 3.5 Flash
    Hand-chosen design parameter; every candidate retained or discarded depends on this model's errors (Section 3.4), so the final benchmark composition is a function of this choice.
  • Subset sample sizes = 100 cases per SABRE-Prior subset; 20 per pilot
    Chosen by hand; the 20-sample pilot size limits the statistical strength of the extensibility claim.
assumptions (3)
  • domain assumption Generated and edited images realize their textual specifications with sufficient fidelity for the reference answers to be visually verifiable.
    The validity of every question-answer pair depends on image generation and editing models producing the specified visual evidence (Section 3.3); the paper itself cites GenEval and T2I-CompBench to motivate this concern and handles it with human verification.
  • domain assumption Human reviewers provide trustworthy ground truth in the verification step.
    The final benchmark includes only candidates marked valid by reviewers (Section 3.5), but no inter-annotator agreement or reviewer calibration is reported (Appendix D.1), so human judgment is an unmeasured load-bearing premise.
  • domain assumption Filtering VLM mistakes are a useful proxy for stress-test difficulty beyond the filtering model itself.
    Pressure screening keeps only candidates the Filtering VLM answers incorrectly (Section 3.4); the claim that the resulting set is hard for other models rests on the five non-filter models also scoring low, which is evidence for but not a guarantee of this premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SABRE: Scalable and Automated Benchmarking of VLMs under Stress." pith.science (2026). https://pith.science/paper/SAOT6P6Z

@misc{pith2026260807435,
  author       = {Pith},
  title        = {Pith review of: SABRE: Scalable and Automated Benchmarking of VLMs under Stress},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SAOT6P6Z}},
  note         = {Machine review of arXiv:2608.07435}
}
read the original abstract

Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify. Building stress tests is costly: samples must satisfy controlled conditions, remain answerable, and challenge current models. We present SABRE, a scalable, automated pipeline that converts a Test Primer (a Markdown Task Design with Data Schema) into structured specifications, generated or edited images, and question-answer pairs. Automated filtering removes candidates solved by a Filtering VLM, while human review verifies candidate validity and supports annotation correction and localized image repair. We instantiate SABRE-Prior to test whether VLMs follow visual evidence instead of relying on world priors -- learned expectations about familiar objects and scenes. Its 600 images and 1,000 questions span Context (unexpected entities in familiar scenes), Texture (counterfactual materials), Attribute (noncanonical component counts), and Language Elicitation (answers suggested by language but unsupported by the image). Across six VLMs, macro-average accuracy ranges from 17.8% to 31.3% (22.6% mean). A real-image Attribute control is comparably difficult for the Filtering VLM. SABRE-Counting and SABRE-Spatial pilots show that the workflow supports other stress-test settings. These results establish SABRE as a reusable framework for constructing and refreshing VLM stress tests rather than a single fixed benchmark.

Figures

Figures reproduced from arXiv: 2608.07435 by the authors.

Figure 1
Figure 1. Overview of the SABRE benchmark-construction pipeline. ular pipeline in which a benchmark designer pro￾vides a Test Primer which consists of a natural￾language Markdown Task Design, a Data Schema that defines task-specific fields and validation rules, and question format, such as open generation or multiple choice. The pipeline converts this spec￾ification into structured sample specifications. It then generates the… view at source ↗
Figure 2
Figure 2. Samples of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. SABRE-counting and SABRE-spatial original patches as P ⋆ i = Ai ⊙ Pei + (1 − Ai) ⊙ Pi . It then places P ⋆ i back into the original image. The center of the mask preserves the repaired content, while the smooth boundary gradually transitions to the original pixels. This soft blending reduces rect￾angular seams and integrates the repaired region more naturally into the surrounding scene. The reviewer compares the rep… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Accuracy (%) of six frontier VLMs on SABRE-Prior. Bars show point estimates, and whiskers indicate the corresponding confidence intervals. Context and Texture report strict All4 accuracy: a case is correct only when all four Base–Edited yes/no probes are answered corre…
Figure 5
Figure 5. Figure 5: Examples of generated images from SABRE [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Additional examples of generated images from [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Results on two additional stress-test instan￾tiations. Each cell represents one sample, and colored cells indicate correct model responses. dress specific VLM failures. 5.5 Real-Image Control We test whether the low model accuracy on SABRE-Prior is caused by generated-…
Figure 8
Figure 8. Figure 8: Abbreviated Context Sample Specification. The record connects visual construction, question definition, and human verification within a schema-validated specification. Long prompts are shortened for presentation. Context Task Design. We use the Context subset as a runn…
Figure 9
Figure 9. Figure 9: Overview of the SABRE annonation platform. the lattice orientation is visually recoverable, the queried cells or layers remain visible, the target objects are distinguishable from distractors, and the reference answer can be verified without relying on the generation p…
Figure 11
Figure 11. Figure 11: Instructions for the method-anonymous repair-quality study. Participants are asked to select the most natural result that removes only the marked entity while preserving all unrelated image content [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 29 canonical work pages

  1. [1]

    2024 , eprint=

    MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs , author=. 2024 , eprint=

    work page 2024

  2. [2]

    ImageNet: A large-scale hierarchical image database , year=

    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Kai Li and Li Fei-Fei , booktitle=. ImageNet: A large-scale hierarchical image database , year=

  3. [3]

    2015 , eprint=

    Microsoft COCO: Common Objects in Context , author=. 2015 , eprint=

    2015

  4. [5]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Zhang, Yuhui and Su, Yuchang and Liu, Yiming and Wang, Xiaohan and Burgess, James and Sui, Elaine and Wang, Chenyu and Aklilu, Josiah and Lozano, Alejandro and Wei, Anjiang and Schmidt, Ludwig and Yeung-Levy, Serena , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2025 , pages =

    2025

  5. [6]

    , booktitle=

    Hu, Yushi and Liu, Benlin and Kasai, Jungo and Wang, Yizhong and Ostendorf, Mari and Krishna, Ranjay and Smith, Noah A. , booktitle=. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering , year=

    work page

  6. [7]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Zhang, Chenshuang and Pan, Fei and Kim, Junmo and Kweon, In So and Mao, Chengzhi , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

    work page 2024

  7. [8]

    Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping , url =

    Yang, Yue and Zhang, Shuibo and Zhang, Kaipeng and Bin, Yi and Wang, Yu and Luo, Ping and Shao, Wenqi , booktitle =. Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping , url =

    work page

  8. [9]

    2026 , eprint=

    Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Models , author=. 2026 , eprint=

    work page 2026

Show all 61 references
  1. [10]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Wang, Haoming and Xue, Qiyao and Gao, Wei , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2026 , pages =

    2026

  2. [11]

    2025 , eprint=

    T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation , author=. 2025 , eprint=

    2025

  3. [12]

    2023 , eprint=

    GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment , author=. 2023 , eprint=

    2023

  4. [13]

    2025 , eprint=

    PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset , author=. 2025 , eprint=

    2025

  5. [15]

    Probing Visual Language Priors in

    Luo, Tiange and Cao, Ang and Lee, Gunhee and Johnson, Justin and Lee, Honglak , booktitle =. Probing Visual Language Priors in. 2025 , editor =

    2025

  6. [16]

    2026 , eprint=

    Vision Language Models are Biased , author=. 2026 , eprint=

    2026

  7. [17]

    2024 , eprint=

    HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models , author=. 2024 , eprint=

    2024

  8. [18]

    2025 , eprint=

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models , author=. 2025 , eprint=

    2025

  9. [19]

    2024 , eprint=

    MMBench: Is Your Multi-modal Model an All-around Player? , author=. 2024 , eprint=

    2024

  10. [20]

    2024 , editor =

    Yu, Weihao and Yang, Zhengyuan and Li, Linjie and Wang, Jianfeng and Lin, Kevin and Liu, Zicheng and Wang, Xinchao and Wang, Lijuan , booktitle =. 2024 , editor =

    2024

  11. [21]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Tong, Shengbang and Liu, Zhuang and Zhai, Yuexiang and Ma, Yi and LeCun, Yann and Xie, Saining , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =. 2024 , pages =

    2024

  12. [23]

    Gemini 3.5 Flash: Model Card , year =

  13. [24]

    2026 , month = mar, howpublished =

    Introducing. 2026 , month = mar, howpublished =

    2026

  14. [25]

    2026 , month = feb, howpublished =

    Introducing. 2026 , month = feb, howpublished =

    2026

  15. [26]

    2026 , month = apr, howpublished =

    2026

  16. [27]

    2026 , month = feb, howpublished =

    2026

  17. [28]

    2026 , howpublished =

    2026

  18. [29]

    2023 , eprint=

    Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V , author=. 2023 , eprint=

    2023

  19. [30]

    Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding , year=

    Leng, Sicong and Zhang, Hang and Chen, Guanzheng and Li, Xin and Lu, Shijian and Miao, Chunyan and Bing, Lidong , booktitle=. Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding , year=

  20. [31]

    2025 , month = nov, howpublished =

    2025

  21. [32]

    2026 , month = feb, howpublished =

    Raisinghani, Naina , title =. 2026 , month = feb, howpublished =

    2026

  22. [33]

    Anthropic . 2026. Introducing Claude Sonnet 4.6 . https://www.anthropic.com/news/claude-sonnet-4-6. Accessed: 2026-07-20

    2026

  23. [34]

    Black Forest Labs . 2025. FLUX.2 : Frontier visual intelligence. https://bfl.ai/blog/flux-2. Accessed: 2026-07-20

    2025

  24. [35]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. https://doi.org/10.1109/CVPR.2009.5206848 Imagenet: A large-scale hierarchical image database . In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248--255

    2009

  25. [36]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. 2025. https://arxiv.org/abs/2306.13394 Mme: A comprehensive evaluation benchmark for multimodal large langua...

    2025 arXiv

  26. [37]

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, Caifeng Shan, and Ran He. 2024. https://arxiv.org/abs/2411.15296 Mme-survey: A comprehensive survey on evaluation of multimodal llms . Preprint, arXiv:2411.15296

    2024 arXiv

  27. [38]

    Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. 2023. https://arxiv.org/abs/2310.11513 Geneval: An object-focused framework for evaluating text-to-image alignment . Preprint, arXiv:2310.11513

    2023 arXiv

  28. [39]

    Google DeepMind . 2026. Gemini 3.5 flash: Model card. https://deepmind.google/models/model-cards/gemini-3-5-flash/. Accessed: 2026-07-20

    2026

  29. [40]

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. https://arxiv.org/abs/2310.14566 Hallusionbench: An advanced diagnostic suite for entangled language hallucinat...

    2024 arXiv

  30. [41]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. 2023. https://doi.org/10.1109/ICCV51070.2023.01866 Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering . In 2023 IEEE/CVF Internationa...

    2023

  31. [42]

    Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2025. https://arxiv.org/abs/2307.06350 T2i-compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation . Preprint, arXiv:2307.06350

    2025 arXiv

  32. [43]

    Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, and Kyomin Jung. 2025. https://doi.org/10.18653/v1/2025.findings-naacl.231 VL ind-bench: Measuring language priors in large vision-language models . In Findings of the Association for Computatio...

    2025 doi

  33. [44]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. https://doi.org/10.1109/CVPR52733.2024.01316 Mitigating object hallucinations in large vision-language models through visual contrastive decoding . In 2024 IEEE/CVF Conference on ...

    2024

  34. [45]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.20 Evaluating object hallucination in large vision-language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi...

    2023 doi

  35. [46]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. https://arxiv.org/abs/1405.0312 Microsoft coco: Common objects in context . Preprint, arXiv:1405.0312

    2015 arXiv

  36. [47]

    Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. 2025. https://arxiv.org/abs/2403.11116 Phd: A chatgpt-prompted visual hallucination evaluation dataset . Preprint, arXiv:2403.11116

    2025 arXiv

  37. [48]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024. https://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281

    2024 arXiv

  38. [49]

    Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, and Honglak Lee. 2025. https://proceedings.mlr.press/v267/luo25b.html Probing visual language priors in VLM s . In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learni...

    2025

  39. [50]

    Moonshot AI . 2026. Kimi K2.6 : Advancing open-source coding. https://www.kimi.com/blog/kimi-k2-6. Accessed: 2026-07-20

    2026

  40. [51]

    OpenAI . 2026. Introducing GPT-5.4 . https://openai.com/index/introducing-gpt-5-4/. Accessed: 2026-07-20

    2026

  41. [52]

    Qwen Team . 2026. Qwen3.5 : Towards native multimodal agents. https://qwen.ai/blog?id=qwen3.5. Accessed: 2026-07-20

    2026

  42. [53]

    Naina Raisinghani. 2026. Nano banana 2: Combining pro capabilities with lightning-fast speed. https://blog.google/innovation-and-ai/technology/ai/nano-banana-2/. Accessed: 2026-07-20

    2026

  43. [54]

    Cristian Sbrolli, Matteo Matteucci, and Toshihiko Yamasaki. 2026. https://arxiv.org/abs/2602.02043 Auto-comp: An automated pipeline for scalable compositional probing of contrastive vision-language models . Preprint, arXiv:2602.02043

    2026

  44. [55]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9568--9578

    2024

  45. [56]

    An Vo, Khai-Nguyen Nguyen, Mohammad Reza Taesiri, Vy Tuong Dang, Anh Totti Nguyen, and Daeyoung Kim. 2026. https://arxiv.org/abs/2505.23941 Vision language models are biased . Preprint, arXiv:2505.23941

    2026 arXiv

  46. [57]

    Haoming Wang, Qiyao Xue, and Wei Gao. 2026. Infinibench: Infinite benchmarking for visual spatial reasoning with customizable scene complexity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21594--21605

    2026

  47. [58]

    Atabuzzaman, Hammad Ayyubi, Haoxuan You, Alvi Ishmam, Kai-Wei Chang, Shih-Fu Chang, and Chris Thomas

    Zhecan Wang, Junzhang Liu, Chia-Wei Tang, Hani Alomari, Anushka Sivakumar, Rui Sun, Wenhao Li, Md. Atabuzzaman, Hammad Ayyubi, Haoxuan You, Alvi Ishmam, Kai-Wei Chang, Shih-Fu Chang, and Chris Thomas. 2024. https://doi.org/10.52202/079017-2015 Journeybench: A challenging one-s...

    2024 doi

  48. [59]

    xAI . 2026. Grok 4.3 . https://docs.x.ai/developers/models/grok-4.3. Accessed: 2026-07-20

    2026

  49. [60]

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. https://arxiv.org/abs/2310.11441 Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v . Preprint, arXiv:2310.11441

    2023 arXiv

  50. [61]

    Yue Yang, Shuibo Zhang, Kaipeng Zhang, Yi Bin, Yu Wang, Ping Luo, and Wenqi Shao. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/file/36d9468ebdb76b9b229fbd343fff84d5-Paper-Conference.pdf Dynamic multimodal evaluation with flexible complexity by vision-language boots...

    2025

  51. [62]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. https://proceedings.mlr.press/v235/yu24o.html MM -vet: Evaluating large multimodal models for integrated capabilities . In Proceedings of the 41st International Co...

    2024

  52. [63]

    Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. 2024. Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21752--21762

    2024

  53. [64]

    Yuhui Zhang, Yuchang Su, Yiming Liu, Xiaohan Wang, James Burgess, Elaine Sui, Chenyu Wang, Josiah Aklilu, Alejandro Lozano, Anjiang Wei, Ludwig Schmidt, and Serena Yeung-Levy. 2025. Automated generation of challenging multiple-choice questions for vision language model evaluat...

    2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.