Pith. sign in

REVIEW 5 major objections 6 minor 50 references

Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A model can give a correct answer while its attention is locked on the wrong image, and this paper introduces attention accuracy to expose that gap.

desk verdict A useful new benchmark and a genuinely interesting attention phenomenon, but the central claim that attention accuracy measures visual understanding is contradicted by the paper's own OCR experiment and weakened by post hoc metric selection. read the letter →

arxiv 2505.10541 v2 pith:VQR6P5ED submitted 2025-05-15 cs.CV

Pengfei Wang , Guohai Xu , Weinong Wang , Junjie Yang , Jie Lou , Yunhua Xue This is my paper · ORCID
classification cs.CV
keywords implicitvisualmisunderstandingattentionaccuracymultimodallargelanguagemodelsmulti-imageunderstandinganalysiscausalSTMEbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most benchmarks judge multimodal language models by whether the final answer is right. This paper argues that a model can answer correctly while actually looking at the wrong image or misusing the right one, a failure it names implicit visual misunderstanding (IVM). The central claim is that the model's causal attention reveals the gap: as layers deepen, attention concentrates on the image that supports the correct answer, and when it settles on a different image despite a right answer, an IVM has occurred. If true, attention accuracy would offer a more reliable way to evaluate visual understanding than answer correctness, and it would expose cases where a strong language model is masking a weak visual system.

What carries the argument

The image-attention factor $\sigma_i$: after softmax is applied to the causal attention matrix, the query rows belonging to the caption or question and to the model output are multiplied by the key columns of each image and averaged across attention heads and token positions, giving a per-image, per-layer concentration score. Three voting rules over the last layers — layer-focused image, mean layer-focused image, and majority-count layer-focused image — turn these scores into a single model-focused image, and attention accuracy is the fraction of correctly answered samples in which that image equals the target. The STME benchmark supplies the target-image association that makes this scoring possible, including a filtering pipeline that keeps only questions whose correct answer depends on exactly one image.

What would settle it

Take samples where attention accuracy is high, hide every image except the attended one, and ask whether the original answer can still be reproduced; if the answer cannot, attention was not carrying the understanding the metric credits it with.

Watch

Extended reading notes

Core claim

The paper defines attention accuracy as a metric that compares the image a model's attention locks onto with the single image actually tied to the correct answer. To compute it, the authors separate the visual and textual blocks of the causal attention matrix and average soft attention scores over heads and tokens to form an image-attention factor per image per layer; three layer-based voting rules then name a model-focused image. On the new STME benchmark, which pairs easy caption-matching and retrieval tasks with harder document, textbook, slide, and spatial tasks, the paper reports that attention accuracy reaches 100 percent on easy tasks for Qwen2VL-7B, that it pulls apart model families whose answer accuracy looks similar, that it rises consistently with model scale, and that it stays stable under image-order shuffling. The same factor is extended to image patches and to image-to-image similarity, supporting the claim that the measure is scale- and architecture-agnostic.

Load-bearing premise

The method assumes that the image a model's attention locks onto is the image the model has genuinely understood, so if attention can be correct while understanding fails, attention accuracy overstates visual comprehension.

Editorial extensions

If this is right

  • Answer accuracy and attention accuracy become two separate axes: a right answer with wrong attention reveals an implicit misunderstanding, while a wrong answer with right attention localizes the failure to recognition or reasoning rather than image selection.
  • Because attention accuracy varies by less than one percent when image order is shuffled, while answer accuracy varies several times more, model comparisons become fairer across different input arrangements.
  • Across scales within a model family, attention accuracy rises even where answer accuracy does not, isolating gains in visual capability from gains in the downstream language model.
  • On hard tasks the spread in attention accuracy across models is wider, so the metric can serve as a more sensitive diagnostic than answer accuracy when tasks demand close visual reading.
  • The patch-level extension identifies which regions of an image drive the answer, allowing hallucinations or misreadings to be traced to particular visual content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves training applications implicit, but attention accuracy could plausibly be used as a training signal or data filter that penalizes models for answering correctly while attending to the wrong image.
  • The paper's own OCR experiments in Section C.3 mark a boundary: attention can land on the correct image while the model still misreads fine-grained content, so attention accuracy is best read as measuring target localization, not full semantic comprehension.
  • Because the same factor applies to image-to-image attention, the approach could transfer to text-only or audio-visual transformers with little modification, though the paper only sketches this possibility.
  • The proposed if-and-only-if relation could be probed adversarially by giving the model samples whose answers are recoverable from the caption alone; stable target attention there would suggest convergence is tracking textual priors rather than visual understanding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper defines implicit visual misunderstanding (IVM) as cases where a multimodal large language model (MLLM) gives a correct answer without genuinely understanding the relevant visual content. The authors propose to diagnose IVMs by inspecting the causal attention matrices inside the model: they define image-attention factors, introduce three layer-aggregation rules (LND, M-LND, MC-LND), and define attention accuracy as the proportion of correctly answered samples in which the resulting model-focused image matches the target image. They introduce the STME benchmark of multi-image single-target tasks, report attention-accuracy results across Qwen2VL, InternVL2, and LLaVA-OneVision at multiple scales, test robustness to image-order shuffling, and extend the approach to patch-level analysis and unimodal image-to-image matching.

Significance. If the central claim were fully supported, an attention-based metric that identifies which image a model's response is actually grounded in would be a useful complement to answer-accuracy benchmarks, especially for detecting correct answers reached through the wrong visual evidence. The paper provides a new benchmark, reproducible code/data links, and a clear attention-extraction methodology across several model families. However, the quantitative claims are weakened by the choice to report the maximum over metrics and layer counts, by evaluating attention accuracy only on each model's own correctly answered subset, and by the absence of the stated theorems that underpin the metric. The abstract's stronger claim that attention accuracy 'directly evaluates the model's visual understanding' is not supported by the evidence, including the paper's own OCR-needle experiment, so the contribution currently lands as a promising diagnostic framework rather than a validated direct measure of visual understanding.

major comments (5)
  1. [§2.1, §2.3] The paper repeatedly refers to Theorem 2.1, Theorem 2.2, and Theorem 2.3, but no theorem statements or proofs appear anywhere in the manuscript. The load-bearing biconditional at the end of §2.1 ('MLLM's attention converges onto the target image if and only if there are no IVMs during inference') is asserted without proof and is the basis for defining attention correctness. Either provide precise theorem statements with proofs, or explicitly label these as empirical hypotheses and treat the experiments as tests of that hypothesis rather than consequences of a theorem.
  2. [§2.3, Definition 2.2, §3.1] The reported attention accuracy is the maximum over the three metrics (LND, M-LND, MC-LND) and over the number of last layers N, as stated in Definition 2.2 and Section 3.1 ('The highest attention accuracy obtained is used as the final value'). This best-of selection over free parameters on the test set inflates the headline numbers. Section E shows that attention-accuracy curves vary substantially with N for the same model, so reporting only the maximum is not a faithful estimate of a fixed evaluation rule. The authors should pre-specify a single metric and N, or report the full distribution over choices, including the average and worst-case values.
  3. [§2.3, §3.1, Table 1] Attention accuracy is computed only on correctly answered samples, and the set of correctly answered samples differs across models. For example, Qwen2VL-7B and InternVL2-8B have substantially different answer accuracies, so their attention-accuracy scores in Table 1 are evaluated on different, non-comparable subsets. A model with lower answer accuracy may be scored on an easier subsample, making the cross-model comparison unreliable. The authors should compute attention accuracy on a common fixed subset (e.g., the intersection of correctly answered samples) or report the per-sample results and quantify how the choice of subset affects the ranking.
  4. [Abstract, §2.1, §C.3] The abstract claims that 'Attention accuracy directly evaluates the model's visual understanding via internal mechanisms,' but this is contradicted by the paper's own Section C.3. There, models achieve near-100% attention accuracy on Image Needle in a Haystack samples while failing to recognize the embedded digit string due to limited OCR ability, meaning attention can be correct while fine-grained visual comprehension fails. At best, the metric measures which image the model's response is based on, not whether the visual content was fully understood. The abstract, Section 1, and the conclusion should be revised to state this weaker claim, and the relationship between attention accuracy and full visual understanding should be discussed explicitly.
  5. [§2.2] The hard-task filtering pipeline relies on GPT-4o to answer questions and identify images related to the final answer, followed by manual review. Using a model to define the target-image labels introduces a potential model-dependent selection bias, especially if GPT-4o's attention or answer patterns correlate with the tested models' behaviors. The authors should report the number of samples removed at each filtering stage, provide inter-annotator agreement or a human-only validation subset, and discuss how GPT-4o-based filtering could affect the measured attention accuracy.
minor comments (6)
  1. [Eq. (3)] The LND definition is ambiguous: it defines a per-layer argmax over i for each l in N, but the text then treats the result as a single model-level prediction. Please clarify how the per-layer predictions are aggregated for a single sample.
  2. [Section E figures] Several figure axis labels in Section E appear as garbled token-like strings rather than readable text; the attention-accuracy curves are otherwise informative and should be relabeled.
  3. [§1, §6] The term 'equivariant measure' is used for attention accuracy but is never defined; either define it formally or replace it with a clearer description.
  4. [Table 5 caption] Table 5 has the same caption as Table 4 even though it reports a different experiment (Image Needle in a Haystack with incorrect or partial OCR outputs); the caption should describe the actual content and the sample-selection criterion.
  5. [§2.2] Please provide a per-task breakdown of the 537 easy and 528 hard samples, the number of images per question in each task, and the dataset license and hosting details for the STME benchmark.
  6. [Throughout] There are several typos and formatting issues, including 'valuates' (§2.2), 'the the methodology' (§3.1), 'Extrcting submatrix' (§4.2), and 'Noted that' (Definition 2.1); a careful proofread is needed.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: IVM is operationalized through attention matching, and reported attention-accuracy values are post hoc maxima over metrics and N; external checks limit but do not remove the loop.

  1. self definitional [Section 2.1 (Attention Distribution Phenomena) and Section 2.3 (Definitions 2.2-2.3)]
    "The comparison between the upper and lower groups in Figure 3 further supports the conclusion that MLLM's attention converges onto the target image if and only if there are no IVMs during inference."

    This biconditional is the load-bearing link between attention and 'fully comprehending the visual input.' Downstream, Definition 2.3 defines attention correctness as 'the model-focused image is identical to the target image,' and the text says that 'by comparing the target image with the model-focused image, we can determine whether IVMs occur.' No-IVM is therefore operationalized as attention convergence, so the abstract's claim that attention accuracy 'directly evaluates the model's visual understanding' restates the assumed biconditional rather than deriving it from an independent measure of comprehension. The paper's GPT-4o reasoning check and hallucination-benchmark correlations provide external grounding, so the loop is partial rather than total.

  2. fitted input called prediction [Section 3.1 (Evaluation)]
    "Similar to Section 2.3, attention accuracy is calculated on correctly answered samples using the LND, M-LND and MC-LND metrics across different N. The highest attention accuracy obtained is used as the final value."

    The reported final attention accuracy is the maximum over three aggregation rules and up to 28 choices of N. The headline values are therefore selected maxima rather than the result of a fixed a priori metric. Claims that attention accuracy is high, or that it beats direct instruction, are partly forced by this post hoc search over hyperparameters. The Appendix E sensitivity curves soften the concern, but the official numbers are still best-case selections made on the same benchmark used to advertise the metric.

full rationale

No load-bearing self-citation or uniqueness theorem appears in the paper; the cited models and benchmark sources (Qwen2VL, InternVL2, LLaVA-OneVision, MileBench, Flickr30k, etc.) are external. The STME labels are produced by GPT-4o filtering plus manual review, not by the attention metric itself, so benchmark construction is not circular. The main circularity-adjacent step is the Section 2.1 biconditional equating 'no IVM' with attention convergence on the target image; once Definition 2.3 turns attention correctness into the criterion for IVM, 'attention accuracy measures IVM' is true by construction, and the stronger claim that it 'directly evaluates visual understanding' inherits that self-definitional status. Section 3.1 further inflates the result by reporting the maximum over metrics and N. The paper's own Section C.3 concedes that models can attend to the correct image while failing OCR (Table 5), which undercuts the sufficiency direction of the biconditional; this is a validity gap rather than a circularity step, so it is noted but not scored separately. Because the paper does provide some external grounding (GPT-4o reasoning evaluation and HallusionBench/POPE correlations for the EVM quadrant), the circularity is partial; score 4.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The paper's central claims rest on the assumption that attention equals understanding, on GPT-4o-generated benchmark labels, and on several post hoc hyperparameter choices. These are the main sources of fragility for the proposed metric.

free parameters (3)
  • N (number of last layers) = Swept; best value selected per model and task
    Section 3.1 states "The highest attention accuracy obtained is used as the final value," meaning N is chosen post hoc to maximize the reported metric.
  • Metric selection rule (LND, M-LND, or MC-LND) = Best of three selected
    Section 2.3 says "The maximum value is used as the final evaluation criterion to reflect the upper bound of each model," so the metric is not fixed in advance.
  • Top-10% patch threshold = 10%
    In Figure 6, the top 10% of patch-attention values are highlighted; this threshold is chosen by hand and is used to support the patch-level analysis.
assumptions (3)
  • domain assumption Attention weights are a faithful indicator of what the model truly understands.
    The entire metric is built on the claim that attention convergence to the target image reflects genuine comprehension. This assumption is not proven and is undermined by the paper's own OCR experiment in Section C.3.
  • domain assumption GPT-4o provides reliable labels for target images and reliable evaluation of reasoning processes.
    The STME filtering pipeline in Section 2.2 uses GPT-4o to answer questions and identify relevant images, and Section 2.3 uses GPT-4o to judge the correctness of CoT responses. No independent verification of these GPT-4o judgments is provided.
  • domain assumption Causal attention matrices can be cleanly partitioned into image, text, and output token blocks.
    The method in Appendix B assumes that the relative positions of tokens are preserved by the model and that the blocking in Equation (8) corresponds exactly to image and text modalities. This may not hold exactly for all architectures.
invented entities (2)
  • Implicit Visual Misunderstanding (IVM)
    purpose: A conceptual category for cases where a model gives the correct answer while attending to or understanding the wrong visual content.
    IVM is defined operationally through attention accuracy within this paper's framework, but there is no external, independently measurable handle that confirms a model 'fully comprehends' an image.
  • Attention Accuracy
    purpose: A metric intended to quantify the rate of IVMs by comparing the model-focused image (from attention) to the target image.
    The metric is defined entirely within the STME benchmark, and its values depend on the post hoc selection of N and the best of three metrics. It has no calibration against an external ground truth of comprehension.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis." pith.science (2026). https://pith.science/paper/VQR6P5ED

@misc{pith2026250510541,
  author       = {Pith},
  title        = {Pith review of: Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQR6P5ED}},
  note         = {Machine review of arXiv:2505.10541}
}
read the original abstract

Recent advancements have enhanced the capability of Multimodal Large Language Models (MLLMs) to comprehend multi-image information. However, existing benchmarks primarily evaluate answer correctness, overlooking whether models genuinely comprehend the visual input. To address this, we define implicit visual misunderstanding (IVM), where MLLMs provide correct answers without fully comprehending the visual input. Through our analysis, we decouple the visual and textual modalities within the causal attention module, revealing that attention distribution increasingly converges on the image associated with the correct answer as the network layers deepen. This insight leads to the introduction of a scale-agnostic metric, \textit{attention accuracy}, and a novel benchmark for quantifying IVMs. Attention accuracy directly evaluates the model's visual understanding via internal mechanisms, remaining robust to positional biases for more reliable assessments. Furthermore, we extend our approach to finer granularities and demonstrate its effectiveness in unimodal scenarios, underscoring its versatility and generalizability.

Figures

Figures reproduced from arXiv: 2505.10541 by the authors.

Figure 1
Figure 1. Left: Example of explicit visual misunderstandings: OCR deficiency and hallucination. Right: Example of implicit visual misunderstandings: the model provides a correct answer but actually describes the second image (while the question pertains to the content of the first image). identifiable—for instance, errors stemming from OCR defi￾ciencies or hallucinations. Implicit visual misunderstand￾ings refer to cases wher… view at source ↗
Figure 2
Figure 2. Overview of the Approach for Identifying Model-focused Image. (a) MLLM completes a caption-matching task; (b) the attention submatrix for multimodal interactions is extracted; (c) for each layer, attention factor values for each image are calculated, allowing identification of the layer-focused image; (d) finally, the model-focus image is determined using the three metrics. et al., 2024a) and visual illusions (Guan … view at source ↗
Figure 3
Figure 3. Top: On the left, MLLM answers a caption matching question with the correct answer and explanation. On the right, the model’s attention converges on the target image. Bottom: MLLM answers an object counting question, where the fourth image corresponds to the correct answer. Despite providing the correct answer, the model’s reasoning is incorrect, showing IVMs. The heatmap reveals that the model’s attention converges… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: The M-LND metric demonstrates the best performance, with attention accuracy exceeding 95% on hard tasks and achieving an astonishing 100% on easy tasks. The accuracy obtained with all three metrics is significantly higher than the results from direct instructions. deve…
Figure 5
Figure 5. Figure 5: As the model scale increases, the attention accuracy also improves, with models of varying scales exhibiting particularly high attention accuracy on less challenging tasks. In contrast, answer accuracy does not follow the same trend. This indicates an enhancement in th…
Figure 6
Figure 6. Figure 6: Left: Using CoT prompting, LLaVA-OneVision-7B is guided to reason through the SlideVQA task. The model successfully answers the question and focuses on the target image. Right: The patch-attention factor values for the target image are computed across all layers. We hi…
Figure 7
Figure 7. Figure 7: Image-to-image similarity matching task. We place the anchor image last, enabling the calculation of the attention factor. The second image is correct answer. Params Attn Acc (%) img-img txt-img Qwen2VL (Wang et al., 2024b) 2B 95.1 92.3 Qwen2VL (Wang et al., 2024b) 7B …
Figure 8
Figure 8. Figure 8: The order and position of input and output tokens in the causal attention matrix. The shaded submatrices are used to calculate the image-attention factor. There are a total of eight task types, classified as either easy or hard tasks. The token order and types for all …
Figure 9
Figure 9. Figure 9: Attention accuracy is calculated based on the upper two quadrants and is used to evaluate the IVMs of MLLMs. Hallucinations, on the other hand, are assessed based on the lower two quadrants. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: This demonstrates Qwen2VL-2B performing a caption matching task, with the sixth image serving as the target image. The model correctly identifies the answer, and its attention appropriately converges to the correct image. Instruction: Given a set of diagrams from a te…
Figure 11
Figure 11. Figure 11: This demonstrates LLaVA-OneVision-0.5B performing a Textbook QA task, with the seventh image as the target image. The model correctly identified the relative atomic mass of aluminum and provided the correct answer with a reasonable explanation. The attention distribut…
Figure 12
Figure 12. Figure 12: InternVL2-4B performs a Document VQ task with the first image as the target. After processing the text, the model extracts relevant information from the image and provides the correct answer. An interesting pattern appears in the attention heatmap: the model focuses o…
Figure 13
Figure 13. Figure 13: Qwen2VL-7B performs a Image Needle in a Haystack task with the eleventh image as the target image. The model successfully detected the special digits s in the target image from a set of 17 images using only 8 layers and accurately recognized the result. This demonstra…
Figure 14
Figure 14. Figure 14: The attention accuracy of Qwen2VL-2B shows that, as N increases, the curves for M-LND and MC-LND quickly decline, while the LND method remains more robust. On both easy and hard tasks, all three metrics reach their inflection points at approximately N = 3, with the va…
Figure 15
Figure 15. Figure 15: The attention accuracy of InternVL2-1B, calculated using three metrics, on the easy and hard datasets. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Last N Decoders 0.4 0.5 0.6 0.7 0.8 Attention Acc. Easy Tasks InternVL2-2B LND M-LND MC-LND 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1…
Figure 16
Figure 16. Figure 16: The attention accuracy of InternVL2-2B, calculated using three metrics, on the easy and hard datasets. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: The attention accuracy of InternVL2-4B, calculated using three metrics, on the easy and hard datasets. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Last N Decoders 0.70 0.75 0.80 0.85 0.90 0.95 Attention Acc. Easy Tasks Intern…
Figure 18
Figure 18. Figure 18: The attention accuracy of InternVL2-8B, calculated using three metrics, on the easy and hard datasets. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: The attention accuracy of InternVL2-26B, calculated using three metrics, on the easy and hard datasets. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Last N Decoders 0.2 0.4 0.6 0.8 1.0 Attention Acc. Easy Tasks LLaVA-OV-0.5B LND M-LND MC-LND 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Last …
Figure 20
Figure 20. Figure 20: The attention accuracy of LLaVA-OneVision-0.5B, calculated using three metrics, on the easy and hard datasets. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: The attention accuracy of LLaVA-OneVision-7B, calculated using three metrics, on the easy and hard datasets. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 13 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Claude 3 haiku: our fastest model yet

    Anthropic. Claude 3 haiku: our fastest model yet. 2024. Available at: https://www.anthropic.com/news/claude-3-haiku

    work page 2024

  3. [3]

    Y., Bhiwandiwalla, A., Tseng, S.-Y., Olson, M

    Ben Melech Stan, G., Aflalo, E., Rohekar, R. Y., Bhiwandiwalla, A., Tseng, S.-Y., Olson, M. L., Gurwicz, Y., Wu, C., Duan, N., and Lal, V. Lvlm-intrepret: An interpretability tool for large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8182--8187, 2024

    2024

  4. [4]

    F., G \'o mez, L., and Karatzas, D

    Biten, A. F., G \'o mez, L., and Karatzas, D. Let there be a clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 1381--1390, 2022

    2022

  5. [5]

    H., Vora, S., Liong, V

    Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 11621--11631, 2020

    2020

  6. [6]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 a

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 a

    arXiv 2024

  7. [7]

    Dress: Instructing large vision-language models to align and interact with humans via natural language feedback

    Chen, Y., Sikka, K., Cogswell, M., Ji, H., and Divakaran, A. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14239--14250, 2024 b

    work page 2024

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., Ma, J., Wang, J., wen Dong, X., Yan, H., Guo, H., He, C., Jin, Z., Xu, C., Wang, B., Wei, X., Li, W., Zhang, W., Lu, L., Zhu, X., Lu, T., Lin, D., and Qiao, Y. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. ArXiv, ...

    arXiv 2024

Show all 50 references
  1. [9]

    H., Yu, F., Wan, X., and Wang, B

    Dingjie, S., Chen, S., Chen, G. H., Yu, F., Wan, X., and Wang, B. Milebench: Benchmarking MLLM s in long context. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Uhwze2LEwq

    2024

  2. [10]

    A challenger to gpt-4v? early explorations of gemini in visual expertise, 2023

    Fu, C., Zhang, R., Wang, Z., Huang, Y., Zhang, Z., Qiu, L., Ye, G., Shen, Y., Zhang, M., Chen, P., Zhao, S., Lin, S., Jiang, D., Yin, D., Gao, P., Li, K., Li, H., and Sun, X. A challenger to gpt-4v? early explorations of gemini in visual expertise, 2023. URL https://arxiv.org/...

    2023 arXiv

  3. [11]

    A., Ma , W.-C., and Krishna , R

    Fu , X., Hu , Y., Li , B., Feng , Y., Wang , H., Lin , X., Roth , D., Smith , N. A., Ma , W.-C., and Krishna , R. BLINK: Multimodal Large Language Models Can See but Not Perceive . arXiv e-prints, art. arXiv:2404.12390, April 2024. doi:10.48550/arXiv.2404.12390

    2024 doi arXiv

  4. [12]

    GLM, T., :, Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., Lai, H., Yu, H., Wang, H., Sun, J., Zhang, J., Cheng, J., Gui, J., Tang, J., Zhang, J., Sun, J., Li, J., Zhao, L., Wu, L., Zhong, L., Liu, M., Huang, M., Zhang, P., Zheng, Q.,...

    2024 arXiv

  5. [13]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Confere...

    2024

  6. [14]

    Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning, 2024

    He, Z., Wu, X., Zhou, P., Xuan, R., Liu, G., Yang, X., Zhu, Q., and Huang, H. Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning, 2024. URL https://arxiv.org/abs/2401.14011

    2024 arXiv

  7. [15]

    Cogvlm2: Visual language models for image and video understanding

    Hong, W., Wang, W., Ding, M., Yu, W., Lv, Q., Wang, Y., Cheng, Y., Huang, S., Ji, J., Xue, Z., et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500, 2024

    2024 arXiv

  8. [16]

    Self-introspective decoding: Alleviating hallucinations for large vision-language models, 2024

    Huo, F., Xu, W., Zhang, Z., Wang, H., Chen, Z., and Zhao, P. Self-introspective decoding: Alleviating hallucinations for large vision-language models, 2024. URL https://arxiv.org/abs/2408.02032

    2024 arXiv

  9. [17]

    Mantis: Interleaved multi-image instruction tuning

    Jiang, D., He, X., Zeng, H., Wei, C., Ku, M., Liu, Q., and Chen, W. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  10. [18]

    Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension

    Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp.\ 4999--5007, 2017

    2017

  11. [19]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Lauren c on, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A., Kiela, D., et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio, 2024

    Leng, S., Xing, Y., Cheng, Z., Zhou, Y., Zhang, H., Li, X., Zhao, D., Lu, S., Miao, C., and Bing, L. The curse of multi-modalities: Evaluating hallucinations of large multimodal models across language, visual, and audio, 2024. URL https://arxiv.org/abs/2410.12787

    2024 arXiv

  13. [21]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023 a

    2023 arXiv

  14. [22]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024

    Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024. URL https://arxiv.org/abs/2407.07895

    2024 arXiv

  15. [23]

    H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W

    Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  16. [24]

    Evaluating object hallucination in large vision-language models

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 292--305, S...

    2023 doi

  17. [25]

    X., Tian, P., Yin, C

    Liang, C. X., Tian, P., Yin, C. H., Yua, Y., An-Hou, W., Ming, L., Wang, T., Bi, Z., and Liu, M. A comprehensive survey and guide to multimodal large language models in vision-language tasks, 2024. URL https://arxiv.org/abs/2411.06284

    2024

  18. [26]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp....

    2014

  19. [27]

    Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms

    Liu, Z., Chu, T., Zang, Y., Wei, X., Dong, X., Zhang, P., Liang, Z., Xiong, Y., Qiao, Y., Lin, D., et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms. arXiv preprint arXiv:2406.11833, 2024

    2024 arXiv

  20. [28]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Info...

    2022

  21. [29]

    Docvqa: A dataset for vqa on document images

    Mathew, M., Karatzas, D., and Jawahar, C. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp.\ 2200--2209, 2021

    2021

  22. [30]

    Mmiu: Multimodal multi-image understanding for evaluating large vision-language models, 2024

    Meng, F., Wang, J., Li, C., Lu, Q., Tian, H., Liao, J., Zhu, X., Dai, J., Qiao, Y., Luo, P., Zhang, K., and Shao, W. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models, 2024. URL https://arxiv.org/abs/2408.02718

    2024 arXiv

  23. [31]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://openai.com/index/gpt-4o-system-card

    2024

  24. [32]

    Perception test: A diagnostic benchmark for multimodal video models

    Patraucean, V., Smaira, L., Gupta, A., Recasens, A., Markeeva, L., Banarse, D., Koppula, S., Malinowski, M., Yang, Y., Doersch, C., et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  25. [33]

    U., Hezel, N., and Jung, K

    Schall, K., Barthel, K. U., Hezel, N., and Jung, K. Gpr1200: a benchmark for general-purpose content-based image retrieval. In International Conference on Multimedia Modeling, pp.\ 205--216. Springer, 2022

    2022

  26. [34]

    Eva-clip: Improved training techniques for clip at scale

    Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  27. [35]

    Aligning large multimodal models with factually augmented RLHF

    Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L., Wang, Y.-X., Yang, Y., Keutzer, K., and Darrell, T. Aligning large multimodal models with factually augmented RLHF . In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Com...

    2024 doi

  28. [36]

    Slidevqa: A dataset for document visual question answering on multiple images

    Tanaka, R., Nishida, K., Nishida, K., Hasegawa, T., Saito, I., and Saito, K. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 13636--13645, 2023

    2023

  29. [37]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 9568--9578, 2024

    2024

  30. [38]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  31. [39]

    and Belinkov, Y

    Vig, J. and Belinkov, Y. Analyzing the structure of attention in a transformer language model. arXiv preprint arXiv:1906.04284, 2019

    1906 arXiv

  32. [40]

    Y., Li, Z., Liu, Q., Liu, X., Ma, M

    Wang, F., Fu, X., Huang, J. Y., Li, Z., Liu, Q., Liu, X., Ma, M. D., Xu, N., Zhou, W., Zhang, K., Yan, T. L., Mo, W. J., Liu, H.-H., Lu, P., Li, C., Xiao, C., Chang, K.-W., Roth, D., Zhang, S., Poon, H., and Chen, M. Muirbench: A comprehensive benchmark for robust multi-image ...

    2024 arXiv

  33. [41]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024 b

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024 b . UR...

    2024 arXiv

  34. [42]

    Needle in a multimodal haystack

    Wang, W., Zhang, S., Ren, Y., Duan, Y., Li, T., Liu, S., Hu, M., Chen, Z., Zhang, K., Lu, L., Zhu, X., Luo, P., Qiao, Y., Dai, J., Shao, W., and Wang, W. Needle in a multimodal haystack. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench...

    2024

  35. [43]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22,...

    2024

  36. [44]

    Mmie: Massive multimodal interleaved comprehension benchmark for large vision-language models, 2024

    Xia, P., Han, S., Qiu, S., Zhou, Y., Wang, Z., Zheng, W., Chen, Z., Cui, C., Ding, M., Li, L., Wang, L., and Yao, H. Mmie: Massive multimodal interleaved comprehension benchmark for large vision-language models, 2024. URL https://arxiv.org/abs/2410.10139

    2024 arXiv

  37. [45]

    Qwen2 technical report, 2024

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M....

    2024 arXiv

  38. [46]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v

    Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  39. [47]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

    Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2: 0 67--78, 2014. doi:10.1162/tacl_a_00166....

    2014 doi

  40. [48]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  41. [49]

    Sigmoid loss for language image pre-training

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 11975--11986, 2023

    2023

  42. [50]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L., Guo, Q., Duan, H., Wang, B., Ouyang, L., et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.