Pith. sign in

REVIEW 5 major objections 6 minor 39 references

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MagiC is a benchmark and metric suite that measures whether vision-language models ground their reasoning in the right image regions, and its main result is that grounding fidelity predicts answer correctness.

desk verdict Worth a look for the human-annotated reasoning data, but the headline grounding result is not established because the metric is underspecified and the relevant-box construction looks wrong. read the letter →

arxiv 2507.07297 v1 pith:SJXFSJWU submitted 2025-07-09 cs.CV

Chengfei Wu , Ronald Seoh , Bingxuan Li , Liqiang Zhang , Fengrong Han , Dan Goldwasser This is my paper
classification cs.CV
keywords groundedmultimodalcognitionvisualreasoningbenchmarkMagiScorevision-languagemodelsgroundingregionfocusself-correctionGQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MagiC, a benchmark for asking whether vision-language models truly ground their answers in the image or rely on shortcuts. It is built from GQA images and questions, human eye-tracking saliency maps that mark relevant regions, and densely annotated step-by-step reasoning chains. The central finding is that a model's answer accuracy tracks how well the regions it mentions in its reasoning overlap with those human-marked relevant regions, and this holds for both small and large models. The authors argue that grounding fidelity is therefore a measurable, scalable signal that should be evaluated and improved alongside answer accuracy.

What carries the argument

The load-bearing object is MagiScore, computed from a region extractor $\phi$ that maps each reasoning step to bounding-box indices from the benchmark's $\mathrm{Box}_q$. Relevant boxes are formed by merging human eye-tracking saliency hotspots with GQA object annotations, and adversarial boxes irrelevant to the question are added; a model's reasoning yields a binary vector $\hat{y}$ over boxes, and precision, recall, and $F_1$ against the ground-truth vector $y^*$ give MagiScore. The same human-corrected reasoning chains drive StepSense, the human-rated correctness of individual reasoning steps, and Self-Heal, which measures whether a model, given an injected erroneous prefix, produces the human correction in its continuation.

What would settle it

Take a random sample of MagiC questions and re-annotate the relevant regions using a stricter criterion: only the objects that must be inspected to determine the answer. Recompute MagiScore against this reference. If the correlation with answer accuracy weakens or disappears, the benchmark's result is an artifact of the saliency-based relevance definition.

Watch

Extended reading notes

Core claim

The paper's central claim is that selective attention to relevant visual regions is a strong indicator of whether a model will answer a visual question correctly. Concretely, MagiScore is defined as the precision, recall, and F1 between the set of bounding boxes a model's reasoning steps reference and the set of human-relevant regions, and across 15 models from 7B to 70B+ parameters higher MagiScore tracks higher answer accuracy. The paper also reports that this region-focus ability improves with model scale, that the test-time-scaling model QvQ reasons over the injected boxes but covers them exhaustively, and that sentence-level reasoning quality (StepSense) and self-correction success (Self-Heal) move together with answer correctness. Even the strongest models still fail in characteristic ways: exhaustive coverage of irrelevant boxes, locating objects in the wrong box, wrong spatial relations, and missing details inside a relevant region.

Load-bearing premise

The whole correlation rests on the assumption that the human eye-tracking saliency regions plus GQA boxes are exactly the visual evidence a model must use to answer each question; if they are not, MagiScore measures gaze prediction rather than answer grounding.

Editorial extensions

If this is right

  • Final-answer benchmarks should report grounding fidelity alongside accuracy, because high accuracy alone does not tell whether a model used the right visual evidence.
  • MagiScore can act as a cheap diagnostic: a model that does not mention the relevant boxes is likely to be wrong even before its answer is graded.
  • Scaling model size improves region focus and answer accuracy, but even the largest open models and GPT-4.1 still make grounding errors, so scale alone is not enough.
  • Test-time scaling helps a model use injected boxes, but QvQ's exhaustive coverage shows that more reasoning tokens do not automatically mean more selective attention.
  • Improving the visual-grounding link in the perception–reasoning–answer chain is a direct path to better final answers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's correlation between MagiScore and accuracy is observational; an implied test is to fine-tune a model with a reward that penalizes mentioning adversarial boxes and rewards mentioning relevant ones, and then check whether final-answer accuracy improves.
  • The adversarial boxes are drawn on the input images but the paper does not compare accuracy with and without them; such a comparison would quantify how much irrelevant visual cues actually hurt answers.
  • Because relevance is defined by eye-tracking saliency merged with GQA object boxes, a version using only GQA scene-graph boxes could show whether the expensive eye-tracking data is needed for the correlation to hold.
  • StepSense currently depends on human annotation, but the paper's own LLM-judge procedure could automate it, letting reasoning quality be measured on the 5,500 weakly supervised examples instead of only the human-curated set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript presents MagiC, a benchmark for evaluating grounded multimodal cognition in large vision-language models (LVLMs). The dataset is built from GQA image-question pairs, human saliency maps from AiR-D, and human annotations of reasoning steps with corrections and bounding-box groundings. The authors evaluate 15 LVLMs across four dimensions: final answer accuracy, reasoning validity (StepSense), grounding fidelity (MagiScore), and self-correction (Self-Heal). The central claim, stated in Section 5.2, is that models with higher MagiScore tend to answer questions more accurately, and that this grounding ability improves with model scale.

Significance. If the benchmark and its metrics are valid, MagiC would be a useful addition to LVLM evaluation, moving beyond answer accuracy to assess whether reasoning steps reference relevant visual evidence. The paper includes human annotation with reported inter-annotator agreement, a relatively broad model suite of 15 systems, and diagnostic settings for adversarial regions and self-correction. However, the central quantitative claim rests on MagiScore, whose computation is under-specified in a load-bearing way: the region extractor φ is never defined, and the pseudocode in Appendix E is inconsistent with the textual description. The correlation claim in Section 5.2 is also not supported by any statistical test, and Table 2 itself contains counterexamples. These issues prevent the claims from being verified or reproduced as written, so the significance is conditional on substantial revision and release of implementation details.

major comments (5)
  1. [Section 4.1] The region extractor function φ(s_i) is never defined. The text says 'we introduce the region extractor function φ(·)' but gives no implementation, prompt, parsing rule, or validation. Since every MagiScore value in Table 2 is computed through this function, the central correlation in Section 5.2 cannot be reproduced or checked. If φ is an LLM call, its own errors are confounded with the model's grounding; if it is a rule-based parser, models that refer to boxes indirectly (e.g., 'the leftmost region') are penalized. The authors must specify φ completely, including the prompt or parsing rules, and ideally release the extracted regions for all evaluated models.
  2. [Appendix E, Algorithm 1] The pseudocode as written does not implement the 'relevant regions for question Q' construction described in Section 3.1. The loop sets G <- ground-truth objects (gqa objects) and B <- merge(G, S) with no per-question filter, so all GQA object boxes in an image enter the relevant set. Under this code, y* would mark many boxes that are irrelevant to Q, and MagiScore would reward exhaustive coverage rather than selective attention, reversing the paper's stated goal. The pseudocode must either include the question-dependent filtering step or be amended to match what was actually run. In addition, the thresholds thresh, areamin, coverage, and user_thr are never given values, and the function RemoveOverlap uses an unspecified 'temporal rule', so the construction is not reproducible.
  3. [Section 5.2, Table 2] The central claim that 'models with higher MagiScore tend to perform better in answering the question itself' is supported only by informal comparison; no correlation coefficient, significance test, or confidence interval is reported across the 15 models. Moreover, Table 2 contains explicit counterexamples: InternVL 2.5-MPO 26B has micro-F1 62.86 but short accuracy 40.69, while Qwen2.5-VL 32B has micro-F1 58.95 and short accuracy 63.32. The authors should report a rank correlation (e.g., Spearman's rho) between MagiScore and Acc_short with a p-value, and should discuss whether the relationship persists after removing Qwen models, whose training recipes they already flag as different.
  4. [Section 4.2 and Appendix prompts (Figures 15-17)] The LLM-based judges used for short/full answer accuracy and for Self-Heal are not validated against human judgments. The accuracy numbers in Table 2 and the correlation in Section 5.2 inherit any systematic bias of the judge model (Qwen2.5-72B-Instruct) — for example, leniency toward certain phrasings or over-rejection of terse answers. The authors should report agreement between the LLM judge and the human annotations they already collected, or at least run a sample-based validation, before these accuracy numbers are used as the ground truth for the main claim.
  5. [Abstract and Table 1] There is a numerical inconsistency in the dataset size. The abstract states 'a held-out test set of 689 instances', while Table 1 reports 698 instances in the test split; Section 3 also says 'we introduce MagiC, which contains 698 test question-and-reasoning pairs' and then in the same paragraph says '181 are reserved for development and 689 for test.' Since the benchmark's test-set size is a headline number, this must be reconciled. The abstract's 'over 15,000 annotated reasoning steps' also conflicts with Table 1, which lists 8,403 test steps and 2,373 dev steps (10,776 total); please clarify what is being counted.
minor comments (6)
  1. [Section 3, p. 4] The phrase 'In which, 181 are reserved for development and 689 for test' is grammatically awkward and internally inconsistent with the 698 in Table 1; it should read 'Of which' and use a consistent test-set count.
  2. [Figure 4] The scaling curve shows no error bars, fitted line, or quantitative trend; the claim in Section 5 that 'accuracy climbs almost linearly' needs at least a fitted slope or correlation to be meaningful.
  3. [Section 3.4] The weakly supervised split is said to use 'relevant object annotations provided by GQA', but the procedure for selecting which objects are relevant per question is not described; please specify how relevance is determined in the absence of saliency maps.
  4. [Section 3.1 and references] The text refers to 'AiR-D' but the cited reference is titled 'AiR: Attention with reasoning capability'; please use a consistent dataset name and provide the correct citation for the AiR-D dataset.
  5. [Appendix C] The hardware description is incomplete: 'All experiments are performed on either a single Nvidia A100-80GB GPU or Nvidia A100-40GB GPU' does not say which models used which GPU, and the paper reports no multiple runs or seed variation despite stochastic sampling at default temperatures.
  6. [Appendix E and general reproducibility] The Project Repository link in the header is not included in the references or in a data-availability statement; given that φ and MagiScore are central, the authors should release code and the exact threshold values used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MagiScore is built from external AiR-D/GQA region labels and correlated with independently judged GQA answer accuracy, so the headline claim is not forced by construction.

full rationale

The paper's central claim, that higher MagiScore tends to imply better final-answer performance, is an empirical correlation rather than a definitional identity. The region ground truth, RBoxq, is constructed from human eye-tracking saliency (AiR-D) and GQA object annotations, not from the model outputs being scored and not from the answer-accuracy labels. Final answer correctness is evaluated separately by an LLM judge against GQA ground-truth answers, which are external to the region-focus computation. No parameter is fitted to make the correlation hold, and no equation defines MagiScore in terms of Accshort or Accfull. The only overlapping-author citation (VISCO, Wu et al. 2025) appears in a related-work list and is not load-bearing for the benchmark's main result. Two weaknesses are real but non-circular: Section 4.1 names the region extractor phi without giving an implementation, so MagiScore cannot be reproduced from the paper alone; and Algorithm 1 merges all GQA objects into B, so the reference relevant set is broader than question-specific evidence. These are validity and reproducibility concerns, not reductions of the claimed result to its own inputs. The human step-annotation protocol in Section 3.3 compares final answers to ground truth before rating individual steps, a possible label-contamination risk, but StepSense is not defined as answer accuracy and the correlation is not algebraically forced. Therefore no circular step is established.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are posited; MagiC, MagiScore, StepSense, and Self-Heal are new benchmark or metric names rather than unexplained postulates. The central claim rests on hand-chosen thresholds in the box-generation pipeline and on domain assumptions about saliency, the LLM judge, and the region extractor.

free parameters (3)
  • saliency map binarization threshold (thresh) and minimum area (areamin)
    Algorithm 1 uses M > thresh and area(r) >= areamin to convert eye-tracking maps into bounding boxes; these thresholds determine the ground-truth relevant regions and thus every MagiScore value, but their values are not given.
  • overlap/coverage thresholds for box merging (coverage, user_thr)
    Algorithm 1 prunes and merges boxes using IoU/coverage and user threshold; these hand-chosen values affect which boxes are considered relevant.
  • number of adversarial regions per question = 3
    Section 3.1 fixes three adversarial boxes per question, a design choice that shapes precision/recall of MagiScore.
assumptions (4)
  • domain assumption Human eye-tracking saliency maps from AiR-D, taken from correct human responses, identify exactly the image regions needed to answer each GQA question.
    Section 3.1: relevant regions RBoxq are built from AiR-D saliency maps; if saliency reflects gaze rather than answer-necessary evidence, ground-truth boxes are mis-specified.
  • domain assumption The LLM judge (Qwen2.5-72B) reliably reproduces human judgments of answer correctness and self-correction.
    Section 4.2 and 4.3 rely on fLM for both Acc and Self-Heal; no human agreement for the judge is reported.
  • domain assumption The region extractor phi can reliably map reasoning-step text to bounding-box indices.
    Section 4.1 defines phi but never specifies its implementation; all MagiScore results depend on it.
  • domain assumption GQA val split answers and scene graphs are clean and no models were exposed to these images during training.
    Section 3.1 says the val split is chosen 'to minimize the likelihood of dataset overlap with existing model knowledge'; this is an assumption, not verified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning." pith.science (2026). https://pith.science/paper/SJXFSJWU

@misc{pith2026250707297,
  author       = {Pith},
  title        = {Pith review of: MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SJXFSJWU}},
  note         = {Machine review of arXiv:2507.07297}
}
read the original abstract

Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, it remains unclear whether these models genuinely perform grounded visual reasoning or rely on superficial patterns and dataset biases. In this work, we introduce MagiC, a comprehensive benchmark designed to evaluate grounded multimodal cognition, assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence. Our benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. We evaluate 15 vision-language models ranging from 7B to 70B parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability. MagiC further includes diagnostic settings to probe model robustness under adversarial visual cues and assess their capacity for introspective error correction. We introduce new metrics such as MagiScore and StepSense, and provide comprehensive analyses that reveal key limitations and opportunities in current approaches to grounded visual reasoning.

Figures

Figures reproduced from arXiv: 2507.07297 by the authors.

Figure 1
Figure 1. Example scenarios from MagiC. Existing work mainly focuses on evaluating the final answer for a given task ignoring the steps model takes to answer. reasoning. This ability is central to human intelligence and is critical for building interpretable, trustworthy AI systems. For instance, as illustrated in our example in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Question type distributions. De￾tailed distribution of detailed types can be found in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of Task-Input Processing 4 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Scaling curve for MagiScore Sound intermediate reasoning, sound final answer. Human-annotated evaluation from [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Examples of failed reasoning. Adversarial boxes are labeled in green. Exhaustive coverage of all regions In many cases, the models tend to exhaustively consider all bounding boxes that appear in the image, despite our instruction to selectively utilize bounding boxes t…
Figure 6
Figure 6. Figure 6: Detailed question type distributions for [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Example where both reasoning and answer are correct [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Example where reasoning is partially incorrect and answer are correct [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Example where both reasoning and answer are incorrect, red boxes without correction [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Example where reasoning is partially incorrect and answer are correct [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Example on how MagiScore are calculated. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Annotation User Interface 17 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Prompt for reasoning and answer generation. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Prompt for self correction. {false_reasoning} is the injected incorrect sub-reasoning. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Prompt for long-form answer judgment. You will be given three pieces of information: 1. A **Question** 2. A **Model Output** (the answer generated by the model) 3. A **Ground Truth Answer** (the correct answer) **Your task**: Determine if the **Model Output** correctl…
Figure 16
Figure 16. Figure 16: Prompt for short-form answer judgment. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: Prompt for self-correction judgment. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 16 canonical work pages

  1. [1]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

    2024

  2. [2]

    Introducing the next generation of claude

    Anthropic. Introducing the next generation of claude. 2024. URL https://www.anthropic.com/news/claude-3-family

    work page 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    Qwen2.5-vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2...

    arXiv 2025

  5. [5]

    AiR : Attention with reasoning capability

    Shi Chen, Ming Jiang, Jinhui Yang, and Qi Zhao. AiR : Attention with reasoning capability. In European Conference on Computer Vision (ECCV), 2020

    work page 2020

  6. [6]

    Measuring and improving chain-of-thought reasoning in vision-language models

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measuring and improving chain-of-thought reasoning in vision-language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum...

    doi:10.18653/v1/2024.naacl-long.1 2024

  7. [7]

    See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://arxiv.org/abs/2301.05226

    arXiv 2023

  8. [8]

    Spatialrgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 135062--135093. Curran ...

    work page 2024

Show all 39 references
  1. [9]

    Aya-vision model card

    CohereForAI. Aya-vision model card. 2025. URL https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/aya_vision.md

    2025

  2. [10]

    Gemini: A family of highly capable multimodal models

    Google. Gemini: A family of highly capable multimodal models. ArXiv, abs/2312.11805, 2023. URL https://api.semanticscholar.org/CorpusID:266361876

    2023 arXiv

  3. [11]

    Improved visual grounding through self-consistent explanations

    Yujie He, Yujia Wang, Yujia Li, Wei Xu, Kai Xu, and Xin Wang. Improved visual grounding through self-consistent explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. URL https://arxiv.org/abs/2312.04554

    2024 arXiv

  4. [12]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  5. [13]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  6. [14]

    Weakly supervised grounding for vqa in vision-language transformers

    Aisha Urooj Khan, Hilde Kuehne, Chuang Gan, Niels Da Vitoria Lobo, and Mubarak Shah. Weakly supervised grounding for vqa in vision-language transformers. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pa...

    2022 doi

  7. [15]

    Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023 a

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023 a . URL https://arxiv.org/abs/2307.16125

    2023 arXiv

  8. [16]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023 b

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023 b . URL https://arxiv.org/abs/2301.12597

    2023 arXiv

  9. [17]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 34892--34916. Curran Associates, Inc., 2023....

    2023

  10. [18]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, Se...

    2024 doi

  11. [19]

    Deepseek-vl: Towards real-world vision-language understanding, 2024

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding, 2024

    2024

  12. [20]

    Whiteboard-of-thought: Thinking step-by-step across modalities, 2024

    Sachit Menon, Richard Zemel, and Carl Vondrick. Whiteboard-of-thought: Thinking step-by-step across modalities, 2024. URL https://arxiv.org/abs/2406.14562

    2024 arXiv

  13. [21]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. URL https://api.semanticscholar.org/CorpusID:263218031

    2023

  14. [22]

    Introducing gpt-4.1 in the api

    OpenAI. Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/, 2025

    2025

  15. [23]

    Qvq: To see the world with wisdom, December 2024

    QwenTeam. Qvq: To see the world with wisdom, December 2024. URL https://qwenlm.github.io/blog/qvq-72b-preview/

    2024

  16. [24]

    Uncovering the full potential of visual grounding methods in VQA

    Daniel Reich and Tanja Schultz. Uncovering the full potential of visual grounding methods in VQA . In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2024 doi

  17. [25]

    Visual chain of thought: Bridging logical gaps with multimodal infillings, 2024

    Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings, 2024. URL https://arxiv.org/abs/2305.02317

    2024 arXiv

  18. [26]

    Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning, 2024. URL https://arxiv.org/abs/2403.16999

    2024 arXiv

  19. [27]

    Gemma 3 technical report, 2025

    Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786

    2025 arXiv

  20. [28]

    Llamav-o1: Rethinking step-by-step visual reasoning in llms, 2025

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, and Salman Khan. Llamav-o1: Rethinking step-by-step visual reaso...

    2025 arXiv

  21. [29]

    Contrastive region guidance: Improving grounding in vision-language models without training

    David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LX...

    2024 doi

  22. [30]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  23. [31]

    Visco: Benchmarking fine-grained critique and correction towards self-improvement in visual reasoning, 2025

    Xueqing Wu, Yuheng Ding, Bingxuan Li, Pan Lu, Da Yin, Kai-Wei Chang, and Nanyun Peng. Visco: Benchmarking fine-grained critique and correction towards self-improvement in visual reasoning, 2025. URL https://arxiv.org/abs/2412.02172

    2025 arXiv

  24. [32]

    Llava-onevision-chat: Improving chat with preference learning, September 2024

    Tianyi Xiong, Bo Li, Dong Guo, Huizhuo Yuan, Quanquan Gu, and Chunyuan Li. Llava-onevision-chat: Improving chat with preference learning, September 2024. URL https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_OneVision_Chat.md

    2024

  25. [33]

    Llava-cot: Let vision language models reason step-by-step, 2025

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2025. URL https://arxiv.org/abs/2411.10440

    2025 arXiv

  26. [34]

    Improving visual grounding by encouraging consistent gradient-based explanations

    Xiaoyu Yang, Yujia Li, Yunchao Wang, Wei Xu, Kai Xu, and Xin Wang. Improving visual grounding by encouraging consistent gradient-based explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. URL https://arxiv.org/abs/2206.15462

    2023 arXiv

  27. [35]

    Mm-vet: evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  28. [36]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  29. [37]

    Improve vision language model chain-of-thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. In Proceedings of the 2024 International Conference on Learning Representations (ICLR), 2024 a . U...

    2024 arXiv

  30. [38]

    Multimodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Research, 2024 b . ISSN 2835-8856. URL https://openreview.net/forum?id=y1pPWFVfvR

    2024

  31. [39]

    Cot-vla: Visual chain-of-thought reasoning for vision-language-action models

    Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Ming-Yu Liu, Donglai Xiang, Gordon Wetzstein, and Tsung-Yi Lin. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models....

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.