pith. machine review for the scientific record. sign in

arxiv: 2604.21523 · v1 · submitted 2026-04-23 · 💻 cs.CV · cs.CL

Recognition: unknown

Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models

Mohammed Safi Ur Rahman Khan , Sanjay Suryanarayanan , Tushar Anand , Mitesh M. Khapra
Authors on Pith no claims yet

Pith reviewed 2026-05-09 22:48 UTC · model grok-4.3

classification 💻 cs.CV cs.CL
keywords vision-language modelsevaluator modelsblind spotsoutput perturbationshallucinationsspatial reasoningimage-to-text evaluationtext-to-image evaluation
0
0 comments X

The pith

Current vision-language models used as evaluators often fail to detect errors in the outputs they judge, missing more than half of perturbed responses in some tests.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large vision-language models are increasingly asked to score or compare outputs from other models on tasks that combine images and text. This work creates thousands of deliberately degraded outputs that contain specific mistakes such as wrong objects, incorrect spatial relations, or details that contradict the source image. It then asks four prominent VLMs to evaluate these outputs under single-score, pairwise, and reference-guided setups. The evaluators frequently assign good scores to the degraded outputs, especially when the errors are fine-grained or compositional. The results indicate that reliance on these automated judges can lead to overestimation of model quality in both image-to-text and text-to-image work.

Core claim

Prominent VLMs used as evaluators for image-to-text and text-to-image tasks exhibit substantial blind spots. Targeted perturbations degrade outputs along forty dimensions covering object hallucinations, spatial reasoning, factual grounding, and visual fidelity. Across more than four thousand test instances, the models often fail to penalize the perturbed outputs, with failure rates exceeding fifty percent in some conditions. They are particularly insensitive to fine-grained compositional and spatial errors and to hallucinated content that contradicts the input image. Pairwise comparison is more reliable than single-answer scoring, yet failure rates remain high.

What carries the argument

Targeted perturbations that introduce controlled errors into model outputs, applied across single-answer scoring, pairwise comparison, and reference-guided evaluation paradigms on a benchmark spanning over four thousand instances and forty error dimensions.

If this is right

  • VLM-based evaluation of image-to-text and text-to-image models can systematically overestimate output quality.
  • Pairwise comparison should be preferred over single scoring when VLMs are used as judges, though it does not eliminate the problem.
  • Development and benchmarking decisions that depend on VLM evaluators may select models that still contain undetected hallucinations or spatial mistakes.
  • Research papers that report VLM-evaluated results may present an inflated picture of progress on visual grounding and compositionality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Developers of new VLMs might add explicit training signals for detecting contradictions between text and image to reduce these blind spots.
  • Similar reliability gaps could appear when VLMs are used to judge outputs in related areas such as video or audio, suggesting the need for domain-specific checks.
  • Until the blind spots are closed, hybrid evaluation pipelines that combine VLM judges with targeted human review on fine-grained errors may be necessary for trustworthy benchmarking.

Load-bearing premise

The artificial perturbations create the same kinds of mistakes that real model outputs contain and that human judges would penalize.

What would settle it

A side-by-side test in which human evaluators rate the same perturbed outputs and show strong agreement with the VLM evaluators on which outputs contain errors would falsify the reported blind spots.

Figures

Figures reproduced from arXiv: 2604.21523 by Mitesh M. Khapra, Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand.

Figure 1
Figure 1. Figure 1: FOCUS is a meta-evaluation benchmark to evaluate robustness of Evaluator VLMs. to-image tasks, it should be able to judge whether a generated image faithfully reflects the prompt, including objects, attributes, physical plausibility, and rendered text (Huang et al., 2023; Meng et al., 2024). Can current VLMs reliably perform such fine-grained, multi-dimensional assessments, or do they exhibit systematic bl… view at source ↗
Figure 2
Figure 2. Figure 2: Comparing the performance of different evaluator paradigms across perturbation [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of reasoning budget on eval￾uator performance. We plot the percentage of instances where the score/verdict of the evaluator is not affected by the perturbation - lower is better. Single Vanilla Single Axes Compare Vanilla Compare Axes 0 20 40 60 80 100 % Perturbations I2T Single Vanilla Single Axes Compare Vanilla Compare Axes T2I I2T Score only I2T - Justifcation Still undetected T2I Score only T2I… view at source ↗
Figure 5
Figure 5. Figure 5: User Application Interface for Validating Image-to-Text (I2T) Perturbations. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: User Application Interface for Validating Text-to-Image (T2I) Perturbations. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Evaluator prompts for I2T - Vanilla 45 [PITH_FULL_IMAGE:figures/full_fig_p045_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Evaluator prompts for I2T - Axes 46 [PITH_FULL_IMAGE:figures/full_fig_p046_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evaluator prompts for I2T - Rubrics/Rules [PITH_FULL_IMAGE:figures/full_fig_p047_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Evaluator prompts for I2T - Single Axes + Rubrics [PITH_FULL_IMAGE:figures/full_fig_p048_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Evaluator prompts for I2T - Compare Axes + Rules [PITH_FULL_IMAGE:figures/full_fig_p049_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Evaluator prompts for I2T - Reference 50 [PITH_FULL_IMAGE:figures/full_fig_p050_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Evaluator prompts for T2I - Vanilla 51 [PITH_FULL_IMAGE:figures/full_fig_p051_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Evaluator prompts for T2I - Axes 52 [PITH_FULL_IMAGE:figures/full_fig_p052_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Evaluator prompts for T2I - Rubrics/Rules [PITH_FULL_IMAGE:figures/full_fig_p053_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Evaluator prompts for T2I - Single Axes + Rubrics [PITH_FULL_IMAGE:figures/full_fig_p054_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Evaluator prompts for T2I - Compare Axes + Rules [PITH_FULL_IMAGE:figures/full_fig_p055_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Evaluator prompts for T2I - Reference 56 [PITH_FULL_IMAGE:figures/full_fig_p056_18.png] view at source ↗
read the original abstract

Large Vision-Language Models (VLMs) are increasingly used to evaluate outputs of other models, for image-to-text (I2T) tasks such as visual question answering, and text-to-image (T2I) generation tasks. Despite this growing reliance, the reliability of these Evaluator VLMs remains under explored. In this work, we systematically evaluate the reliability of Evaluator VLMs across both I2T and T2I tasks. We introduce targeted perturbations that degrade output quality along key error dimensions, including object hallucinations, spatial reasoning, factual grounding, and visual fidelity. These perturbations test whether Evaluator VLMs can reliably account for these quality degrading errors in their evaluations. Using a comprehensive benchmark of over 4000 perturbed instances spanning 40 perturbation dimensions, we evaluate 4 prominent VLMs using single-answer scoring, pairwise comparison, and reference-guided paradigms. Our findings reveal that current VLM evaluators exhibit substantial blind spots: they often fail to detect perturbed outputs - in some cases exceeding 50%, struggle particularly with fine-grained compositional and spatial errors, and are often insensitive to hallucinated content that contradicts the input image. Pairwise comparison proves more reliable, though failure rates persist. These results highlight the unreliable nature of current Evaluator VLMs and urge caution in their deployment for benchmarking and development decisions. Code and data have been made publicly available.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript claims that current Vision-Language Models (VLMs) used as evaluators for image-to-text (I2T) and text-to-image (T2I) tasks exhibit substantial blind spots. Using programmatically generated perturbations across 40 dimensions (object hallucinations, spatial reasoning, factual grounding, visual fidelity) on a benchmark of over 4000 instances, the authors evaluate four prominent VLMs under single-answer, pairwise, and reference-guided paradigms. They report failure rates often exceeding 50% in detecting degraded outputs, with particular weaknesses on fine-grained compositional/spatial errors and insensitivity to image-contradicting hallucinations; pairwise comparison is more reliable but still flawed. The work concludes that VLM evaluators are unreliable for benchmarking and releases code and data publicly.

Significance. If the central empirical findings hold after addressing validation gaps, this work would be significant for the multimodal AI community, where VLM-based evaluation is increasingly adopted for VQA, captioning, and generation tasks. The large-scale benchmark, public code/data release, and multi-paradigm testing provide a reproducible foundation for future work on evaluator reliability. The results could prompt shifts toward hybrid human-VLM evaluation pipelines.

major comments (3)
  1. [§3] §3 (Perturbation Design and Generation): The 40 perturbation dimensions are defined programmatically (e.g., object swaps, spatial flips, factual contradictions), but the manuscript provides no human validation study rating original vs. perturbed pairs on the same rubric used for VLMs. This is load-bearing for the 'blind spots' interpretation, as the observed insensitivity could reflect either VLM limitations or perturbations that are too artificial/subtle for any evaluator to penalize.
  2. [§4.2] §4.2 (Evaluation Paradigms and Prompting): The three paradigms (single-answer scoring, pairwise comparison, reference-guided) are tested, yet there are no controls or ablations for prompt sensitivity, exact prompt templates, or confirmation that the setups match real-world VLM evaluator deployments. This leaves the reported failure rates (including >50% cases) potentially dependent on unexamined prompting choices.
  3. [§5] §5 (Results and Analysis): Failure rates and category-specific weaknesses (compositional/spatial/hallucination) are reported without statistical significance tests, confidence intervals, or inter-annotator details for any human component. This weakens the robustness of claims about particular error dimensions and the superiority of pairwise comparison.
minor comments (2)
  1. [Abstract / §4.1] The abstract states 'over 4000 perturbed instances' but the exact split between I2T and T2I tasks, and per-dimension counts, should be tabulated in §4.1 for clarity.
  2. [Figures in §3] Example figures illustrating perturbations (e.g., before/after images or outputs) would benefit from higher resolution and explicit annotations highlighting the introduced error.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback on our manuscript. The comments highlight key areas for strengthening the empirical foundation of our claims regarding VLM evaluator blind spots. We respond point-by-point to each major comment below and indicate the revisions we will make in the next version of the paper.

read point-by-point responses

  1. Referee: §3 (Perturbation Design and Generation): The 40 perturbation dimensions are defined programmatically (e.g., object swaps, spatial flips, factual contradictions), but the manuscript provides no human validation study rating original vs. perturbed pairs on the same rubric used for VLMs. This is load-bearing for the 'blind spots' interpretation, as the observed insensitivity could reflect either VLM limitations or perturbations that are too artificial/subtle for any evaluator to penalize.

    Authors: We agree that explicit human validation of the perturbations is important to confirm they induce perceptible quality degradations. Our perturbations were constructed to target well-documented error categories from prior literature on I2T and T2I failures, using deterministic programmatic rules that produce clear mismatches (e.g., object swaps that violate scene consistency). However, the original submission did not include a human rating study. We will add a targeted human validation experiment on a stratified subset of 200 instances, where annotators rate original-perturbed pairs for quality difference using a rubric aligned with the VLM evaluation criteria. Results and inter-annotator agreement will be reported in the revised manuscript to support the blind-spot interpretation. revision: yes

  2. Referee: §4.2 (Evaluation Paradigms and Prompting): The three paradigms (single-answer scoring, pairwise comparison, reference-guided) are tested, yet there are no controls or ablations for prompt sensitivity, exact prompt templates, or confirmation that the setups match real-world VLM evaluator deployments. This leaves the reported failure rates (including >50% cases) potentially dependent on unexamined prompting choices.

    Authors: We acknowledge that prompt engineering can influence VLM outputs and that our original experiments used fixed templates without systematic sensitivity analysis. The templates were chosen to reflect common practices in recent VLM evaluation papers for scoring, comparison, and reference-guided assessment. To address this, we will include an ablation study in the revision that varies prompt phrasing (e.g., explicitness of scoring instructions and comparison criteria) across a subset of the benchmark and reports the resulting variance in failure rates. We will also add a section clarifying alignment with real-world usage in related works. revision: yes

  3. Referee: §5 (Results and Analysis): Failure rates and category-specific weaknesses (compositional/spatial/hallucination) are reported without statistical significance tests, confidence intervals, or inter-annotator details for any human component. This weakens the robustness of claims about particular error dimensions and the superiority of pairwise comparison.

    Authors: We appreciate this observation on statistical rigor. The manuscript presents aggregate and per-category failure rates but does not include formal tests or intervals. In the revision we will add paired statistical tests (e.g., McNemar’s test for paradigm comparisons and chi-squared tests for error-type differences) together with 95% confidence intervals computed via bootstrap resampling. Because the core evaluation relies on programmatic perturbations rather than human annotations, inter-annotator agreement metrics do not apply to the main results; however, the human validation study added in response to the first comment will include agreement statistics. These changes will strengthen the claims about category-specific weaknesses and the relative reliability of pairwise comparison. revision: yes

Circularity Check

0 steps flagged

Empirical study with no self-referential derivations or fitted predictions

full rationale

The paper conducts an empirical evaluation by programmatically generating 4000+ perturbed instances across 40 dimensions and measuring VLM evaluator performance under three standard paradigms. No equations, parameters, or derivations are present that could reduce results to inputs by construction. Findings rest on direct observation of failure rates rather than any self-citation chain or ansatz. The central claim of blind spots is therefore independent of the paper's own prior outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard assumptions about VLM prompting and the validity of synthetic perturbations as error proxies; no free parameters are fitted and no new entities are postulated.

axioms (1)
  • domain assumption VLMs can be reliably prompted to act as evaluators using single-answer scoring, pairwise comparison, and reference-guided paradigms
    The paper tests these three paradigms without questioning their basic applicability to evaluator roles.

pith-pipeline@v0.9.0 · 5560 in / 1184 out tokens · 27675 ms · 2026-05-09T22:48:34.351105+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    MMBench: Is Your Multi-modal Model an All-around Player?

    doi: 10.48550/arXiv.2307.06281. Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv: 2406.11069, 2024. Fanqing Meng, Wenqi Shao, Lixin Luo, Yahong Wang, Yiran Chen, Quanfeng Lu, Yue Yang, Tianshuo Yang, Kaipeng Zhang, ...

    work page internal anchor Pith review doi:10.48550/arxiv.2307.06281 2024

  2. [2]

    Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

    doi: 10.48550/ARXIV .2305.17926. URL https://doi.org/10.48550/arXiv. 2305.17926. Zengbin Wang, Xuecai Hu, Yong Wang, Feng Xiong, Man Zhang, and Xiangxiang Chu. Everything in its place: Benchmarking spatial intelligence of text-to-image models.arXiv preprint arXiv: 2601.20354, 2026. Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Swami Manohar...

    work page internal anchor Pith review doi:10.48550/arxiv 2026

  3. [3]

    MMBench - Liu et al. (2023)

    2023

  4. [4]

    MMDocBench - Zhu et al. (2024)

    2024

  5. [5]

    TouchStone - Bai et al. (2023)

    2023

  6. [6]

    VisIT-Bench - Bitton et al. (2023)

    2023

  7. [7]

    WildVision - Lu et al. (2024)

    2024

  8. [8]

    SimpleVQA - Cheng et al. (2025)

    2025

  9. [9]

    Under review

    RealWorldQA-https://huggingface.co/datasets/xai-org/ RealworldQA 15 Preprint. Under review. Category Perturbation Dimension # Image-to-Text (I2T) Visual Grounding (VG) Entity Substitution 105 Attribute Distortion 91 Spatial Relation Swap 90 Phantom Details Injection 83 Over Generalization 67 Important Detail Omission 64 Semantic Interpretation (SI) Contex...

  10. [10]

    T2I-CoReBench - Li et al. (2026a)

  11. [11]

    T2I-ReasonBench - Sun et al. (2025a)

  12. [12]

    T2I-CompBench++ - Huang et al. (5555)

  13. [13]

    R2I-Bench - Chen et al. (2025)

    2025

  14. [14]

    MJBench - Chen et al. (2024b)

  15. [15]

    TextAtlasEval - Wang et al. (2025)

    2025

  16. [16]

    SpatialGen Eval - Wang et al. (2026) B Human-in-the-Loop Validation of Perturbations To ensure the quality of the constructed perturbations, we conducted a human-in-the- loop validation process using a custom-built annotation interface, PerturbVal. This tool streamlines the validation of perturbations for both Image-to-Text (I2T) and Text-to-Image (T2I) t...

    2026

  17. [17]

    A relevant response stays on-topic and provides information pertinent to the requested task

    Relevance:Measures how closely and directly the response addresses the question about the image. A relevant response stays on-topic and provides information pertinent to the requested task

  18. [18]

    It checks for factual correctness and reliability of the generated content

    Trustworthiness:Evaluates whether the response is accurate, grounded in the image, and free from hallucinated or unsupported claims. It checks for factual correctness and reliability of the generated content

  19. [19]

    Claims that cannot be verified from the image are treated as weakly grounded

    Visual Grounding:Measures whether the response relies on visible evidence from the image rather than assumed or fabricated details. Claims that cannot be verified from the image are treated as weakly grounded

  20. [20]

    Clarity:Assesses how easy the response is to understand, including clear expression, well-organized ideas, and the absence of ambiguity or confusion

  21. [21]

    It ensures that ideas are connected logically and the narrative progresses smoothly without abrupt jumps or disjointed sections

    Coherence:Evaluates the logical flow and internal consistency of the response. It ensures that ideas are connected logically and the narrative progresses smoothly without abrupt jumps or disjointed sections

  22. [22]

    Completeness:Measures whether the response fully addresses all aspects of the question based on the image, including all necessary components, details, and depth

  23. [23]

    Text-to-Image (T2I) Axes We evaluate generated images along the following axes:

    Creativity(pairwise only):Assesses the originality and inventiveness of the re- sponse, encompassing imagination, freshness of content, and whether the output introduces something genuinely new or less commonly encountered. Text-to-Image (T2I) Axes We evaluate generated images along the following axes:

  24. [24]

    Prompt Adherence:Measures how closely and faithfully the generated image reflects the given text prompt, including described objects, attributes, actions, and scene elements

  25. [25]

    Text-Image Alignment:Assesses how accurately the image reflects the prompt. Checks whether all elements—including background, object attributes (e.g., color, gender, skin texture, count), actions, and spatial relationships (e.g., positioning)— are correctly represented

  26. [26]

    These elements should be accurate and aesthetically pleasant

    Image Quality:Identifies visual artifacts in the image, such as distortion, blurriness, or illogical representation of facial features, limbs, fingers, objects, or rendered text. These elements should be accurate and aesthetically pleasant

  27. [27]

    Accuracy:Measures whether the depicted objects, attributes, spatial relationships, and scene elements are correctly rendered as described in the prompt, without errors or hallucinations

  28. [28]

    Completeness:Measures whether all elements, objects, and attributes described in the text prompt are present and visible in the generated image

  29. [29]

    19 Preprint

    Safety:Ensures the image does not contain inappropriate content, including hate speech, harassment, violence, self-harm, sexual content, NSFW material, shocking visuals, or illegal activities. 19 Preprint. Under review

  30. [30]

    A" if assistant A is better,

    Bias:Assesses whether the occupation or education specified in the prompt is represented without stereotyping towards a particular demographic group. Eval- uators are instructed to ensure their judgment does not reflect bias towards any demographic. In both settings, evaluators also produce anOverall Qualityscore that provides a holistic assessment of the...