Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VFaith-Bench shows that large multimodal models often succeed on visual questions by repeating learned patterns rather than reasoning from the image in front of them.

desk verdict VFaith-Bench is a genuinely useful new benchmark for probing visual faithfulness in MLLMs, but the headline memory/data-leakage conclusion rests on a metric that conflates perceptual failure with memory and needs to be softened or re-supported. read the letter →

arxiv 2506.11571 v2 pith:QGYL42RE submitted 2025-06-13 cs.CV

Jiachen Yu , Yufei Zhan , Ziheng Wu , Yousong Zhu , Jinqiao Wang , Minghui Qiu This is my paper
classification cs.CV
keywords visualfaithfulnessmultimodallargelanguagemodelschain-of-thoughtreasoninghallucinationevaluationcueeditingbenchmarkconstructionrepeatratioperceptiontask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether multimodal large language models solve visual-reasoning problems by genuinely reading the image, and it answers with VFaith-Bench, a benchmark of 755 isomorphic image-question pairs. Starting from 644 questions in existing multimodal reasoning sets, the authors extract the visual cues that a model's chain of thought explicitly says it relies on, edit one such cue so that the correct answer flips to a different option, and keep the question text unchanged. Across every tested model, accuracy drops after this edit, with several models losing more than ten points overall and more than twenty in a single category; models also frequently repeat the now-incorrect original answer, a behavior the paper quantifies as the repeat ratio. The authors conclude that strong performance on the original images may come partly from brittle, memory-like patterns rather than robust perception-and-reasoning, and that current models show substantial visual-cue hallucination on a dedicated perception task. If right, the benchmark provides a way to separate visual faithfulness from raw accuracy when comparing reasoning models.

What carries the argument

The load-bearing object is the cue-driven automatic and controllable editing pipeline. A reasoner is prompted to mark every visual cue it uses with explicit <vcues_*> tags; an edit-suggestion model proposes a minimal change to one selected cue that makes a different option correct; GPT-image-1 performs the edit; and human reviewers verify that the edit is coherent, the original answer is no longer valid, and the new answer is unique. This yields isomorphic pairs: the same question, a visually similar image, and a different ground truth. The repeat-ratio metric then isolates the memory-like failure: among all cases where the edited answer is wrong, it measures the fraction in which the model repeats its originally correct answer, a signature of pattern adherence rather than re-reading the image. The perception subset, with questions about which cues were actually changed, separates failures of seeing from failures of reasoning.

What would settle it

A direct falsifier is a controlled human study on a random sample of VFaith-Bench pairs: if human annotators shown only the edited image and question do not reliably pick the new ground-truth answer, or if an independent edit-quality check finds systematic artifacts or difficulty shifts in the edited images, the memory interpretation loses its footing. A complementary control experiment would edit image regions that are irrelevant to the correct answer; if accuracy drops as much as in the task-relevant edits, the drop reflects general sensitivity to editing rather than reliance on memorized answer patterns.

Watch

Extended reading notes

Core claim

In the paper's own terms, VFaith-Bench establishes that current multimodal reasoning models are not reliably faithful to the visual input: all evaluated models show a significant average accuracy drop on questions whose images were edited only in task-critical visual cues, while maintaining coherent-looking chain-of-thought output. The paper reports that closed-source flagship models lead in raw accuracy but also show statistically higher repeat ratios, indicating greater reliance on memorized response patterns, and that the InternVL series shows the strongest visual-cue perception yet underperforms in overall edited accuracy, suggesting a perception-reasoning misalignment. The authors deliberately interpret the raw-versus-edited gap as evidence that success on the original problems may not necessarily stem from true visual observation coupled with robust reasoning, but potentially from brittle patterns.

Load-bearing premise

The load-bearing premise is that GPT-image-1 edits change only the targeted visual cue, leaving the question, the rest of the image, and the difficulty of the task otherwise intact, and that human review guarantees each edited pair has exactly one correct answer; if edits introduce artifacts, ambiguity, or shifts in difficulty, the raw-versus-edit gap conflates visual faithfulness with image quality.

Editorial extensions

If this is right

  • If the central claim is correct, raw accuracy on existing multimodal reasoning benchmarks overstates true visual reasoning, since part of the score can come from repeating memorized answer patterns.
  • VFaith-Bench lets researchers rank models by visual faithfulness, and the paper's results identify concrete failure modes: most models lose points after cue edits, while repeat ratio and perception accuracy vary independently.
  • The reported pattern that closed-source models have higher repeat ratios suggests that larger, text-heavy training makes models more likely to stick to learned answers rather than update from the new image.
  • The repeat-ratio metric could be used as a diagnostic signal when training or fine-tuning reasoning models, with lower repetition after cue edits indicating better visual grounding.
  • The benchmark construction itself provides a reusable recipe for synthesizing challenging, answer-flipping test cases from any multimodal reasoning dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not in the paper: if the memory interpretation holds, the repeat ratio becomes a natural training target, e.g. a reward that penalizes carrying over the pre-edit answer after the image has changed.
  • Not in the paper: the isomorphic-pair design transfers to other modalities, such as short video clips or audio diagrams, where one event-level cue determines the answer.
  • As an extension: a control set with irrelevant edits would tell whether some of the reported accuracy drop is generic image-edit sensitivity rather than cue-specific memory reliance.
  • As an extension: the closed-source-versus-open-source repeat-ratio difference hints at memorization scaling with training mixture, but confirming that would require controlling pretraining data, which the paper does not do.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces VFaith-Bench, a benchmark for evaluating whether large multimodal models reason from seen visual evidence or from memorized patterns. The authors build an automatic pipeline using visual-cue extraction and GPT-image-1-based editing to create 644 isomorphic image-question pairs (plus a 91-item perception task) in which a targeted visual cue is modified so that the correct answer changes. They evaluate 13 models in raw and edited conditions, reporting accuracy drops, a 'repeat ratio' (how often an error on the edited image still outputs the original answer), and perception accuracy. The central claim is that a significant performance drop on edited images, together with high repeat ratios, indicates hallucination and potential data leakage or over-reliance on training patterns rather than genuine visual reasoning.

Significance. VFaith-Bench is a useful and timely diagnostic resource: the paired design controls for question text and image content, making the raw-versus-edit accuracy comparison a relatively clean measure of sensitivity to specific visual cues. The perception task is a valuable addition, and the repeat-ratio metric, while currently under-validated, points at an interesting failure mode. The paper also provides a reproducible-looking pipeline and promises public code and data. If the central memory/data-leakage interpretation is supported by additional analyses, the benchmark would be an important tool for studying MLLM hallucination and reasoning faithfulness. However, the current evidence for the strongest claim is confounded by perceptual failure, and the lack of statistical rigor weakens the quantitative conclusions.

major comments (4)
  1. [Section 3.2.2 and Tables 2-3] The repeat-ratio metric, defined as the fraction of edited-image errors in which the model outputs the original ground-truth answer, is interpreted in Section 4.2.1 as evidence of 'data leakage or over-reliance on training patterns.' However, because the question text is identical and the edited image preserves most content, a model that simply fails to perceive the altered cue will, by construction, produce the original answer. The paper's own perception results (Perception column, 16.48-50.50%) show that all tested models frequently fail to detect modified cues, so the high repeat ratios (68.85-87.50%) are exactly what low perception predicts. The metric cannot separate 'saw the change but answered from memory' from 'did not see the change and answered from the unchanged image.' To support the memory/data-leakage conclusion, the authors should provide a per-item analysis linking perception failures to repeat responses (e.g., repeat ratio conditioned on whether the model correctly describes the edited cue in its CoT), or add a control condition that edits a task-irrelevant cue and show that repeat ratios are not elevated there.
  2. [Section 4, Tables 2 and 3] The text repeatedly uses 'significant' (e.g., 'a significant average performance drop', 'statistically higher repeat ratio') without any confidence intervals, bootstrap estimates, or hypothesis tests. Sample sizes are small in several categories (CRE=57, TIF=71, REA=132), so per-category deltas such as the +10.53 point increase for Seed1.5-VL on CRE or the -1.19 for Gemini on PFK are within plausible noise. The headline claims about degradation and about closed-source versus open-source repeat ratios therefore are not quantitatively substantiated. The authors should report at least bootstrap 95% confidence intervals and, if feasible, paired significance tests on the raw-versus-edit comparisons.
  3. [Sections 3.1.2 and 3.2.1] The validity of the benchmark hinges on the assertion that edits change only the targeted visual cue and yield a unique correct answer (Principles 1-3) and that manual verification ensures this (Section 3.2.1). However, the paper provides no details on the number of annotators, inter-annotator agreement, or the criteria applied; the appendix only mentions that a manual review was conducted. Moreover, the example in Figure 7 (train dirt) edits a cue about rain streaks that may still leave option C ('All options are possible') plausible, illustrating the difficulty of guaranteeing uniqueness. Without a documented verification protocol and ideally an independent human-accuracy study, the raw-versus-edit accuracy gap conflates visual faithfulness with edit quality and ambiguity.
  4. [Section 1] The paper justifies the interpretation of model accuracy drops by stating that humans can solve the edited variants 'with near-perfect accuracy,' but no human-subject data are presented anywhere in the manuscript. Since this assertion is the normative baseline that makes a model drop a 'pathology,' it should be replaced with a measured human accuracy on a sample (or at least on a subset) of VFaith-Bench, with the same question format.
minor comments (6)
  1. [Abstract] There is a grammatical error: 'it is challenging to analysis with quantitative results' should be 'it is challenging to analyze with quantitative results.'
  2. [Section 4.2.2] The subsection heading contains a typo: 'Self-Evloving' should be 'Self-Evolving.'
  3. [Table 2] The Repeat column lists numbers such as 86.36 and 72.92; the caption or text should explicitly state that these are percentages and should define the denominator used for the repeat ratio in this table.
  4. [Section 4.2.1] The statement that closed-source models have a 'statistically higher repeat ratio' is not accompanied by any test statistic or p-value; if the claim is qualitative, remove the word 'statistically' to avoid implying a test was run.
  5. [Figure 1] The example option D contains a typo, 'winder outing,' which should be 'winter outing.'
  6. [Evaluation pipeline] The paper uses Claude-3.5-Sonnet to grade answers when direct extraction fails; since Claude-3.7-Sonnet is one of the evaluated models, the authors should discuss whether using a model from the same family as a grader could introduce systematic bias in the scoring of that model's responses.

Circularity Check

1 steps flagged · score 6.0 of 10

Repeat-ratio 'memory' conclusion reduces by construction to perceptual failure on near-identical edited images.

  1. self definitional [Section 3.2.2 (Repeat Ratio Metric), interpreted in Section 1 'Pattern Adherence' bullet and Section 4.2.1]
    "Repeat Ratio Metric: Beyond errors caused by the inability to correctly interpret visual cues, we also seek to explore issues arising from the exact replication of the original question content and largely unchanged image content and structure, which may lead to hallucinations. This could be due to the model's exposure to similar data during training, preventing it from breaking away from prior knowledge to produce accurate results. Alternatively, it could indicate a bias towards certain paradigms or difficulty in following specific instructions."

    By construction, the question text is identical and the edited image is 'largely unchanged' relative to the original. A model that simply fails to perceive the modified visual cue will therefore output the same answer it gave for the original image, so a_edit = a_ori = gt_ori is the default outcome under perceptual miss, with no appeal to training data needed. The paper's own perception results show that all evaluated models frequently miss the modified cues (Perception accuracy 16.48–50.50 in Tables 2–3). Hence the numerator of the Repeat Ratio is populated by perceptual failures by construction.

full rationale

The core raw-versus-edit accuracy comparison and the perception task are empirical benchmark measurements with no fitted parameters, so they are not circular in themselves. The circular step is confined to the 'Pattern Adherence' / repeat-ratio conclusion. Because the question text is identical and the edited image is 'largely unchanged' by the paper's own description, any model that fails to perceive the targeted edit will output the same answer as before the edit; hence a_edit = a_ori = gt_ori is forced under perceptual miss. The paper's perception tables show all models frequently miss the modified cues (16.48–50.50 on the perception subset), so the repeat-ratio numerator is dominated by perception failures by construction. Interpreting this as 'data leakage or over-reliance on training patterns' labels the metric's own numerator as memory without a control condition that separates perceptual miss from memorized recall. The benchmark's other findings, such as performance drops after edits, stand independently; only the memory/data-leakage interpretation is partially circular. No load-bearing self-citation chain was found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted numerical parameters appear in the benchmark construction or metrics. The load-bearing assumptions are qualitative: edit fidelity, human verification quality, an unmeasured human baseline, and the interpretation of repeat answers as memory. These are listed as axioms because the central conclusions collapse if any of them fails.

assumptions (4)
  • domain assumption GPT-image-1 edits modify only the targeted visual cues and preserve all other task-relevant content.
    Invoked in Section 3.1.2 Principles 2 and 3 and Figure 3; the entire before-and-after comparison depends on the edit being localized and not changing difficulty.
  • domain assumption Manual verification guarantees each edited image-question pair has a unique correct answer among the original options and the original answer is no longer correct.
    Section 3.2.1 describes manual verification of all 644 entries; without this guarantee, raw-versus-edit accuracy differences cannot be attributed to the cue change.
  • ad hoc to paper Humans solve the edited variants near-perfectly after seeing the original reasoning, so a model accuracy drop is a pathology.
    Section 1 states this as an observation, but the paper reports no human-subject data or inter-annotator agreement.
  • ad hoc to paper Repeating the original answer on the edited image indicates memory or data leakage rather than misperception or edit artifact.
    Section 3.2.2 defines repeat ratio and Section 4.2 interprets it as evidence of memorization; the metric does not rule out the model perceiving the edited image as similar to the original.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?." pith.science (2026). https://pith.science/paper/QGYL42RE

@misc{pith2026250611571,
  author       = {Pith},
  title        = {Pith review of: VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGYL42RE}},
  note         = {Machine review of arXiv:2506.11571}
}
read the original abstract

Recent extensive works have demonstrated that by introducing long CoT, the capabilities of MLLMs to solve complex problems can be effectively enhanced. However, the reasons for the effectiveness of such paradigms remain unclear. It is challenging to analysis with quantitative results how much the model's specific extraction of visual cues and its subsequent so-called reasoning during inference process contribute to the performance improvements. Therefore, evaluating the faithfulness of MLLMs' reasoning to visual information is crucial. To address this issue, we first present a cue-driven automatic and controllable editing pipeline with the help of GPT-Image-1. It enables the automatic and precise editing of specific visual cues based on the instruction. Furthermore, we introduce VFaith-Bench, the first benchmark to evaluate MLLMs' visual reasoning capabilities and analyze the source of such capabilities with an emphasis on the visual faithfulness. Using the designed pipeline, we constructed comparative question-answer pairs by altering the visual cues in images that are crucial for solving the original reasoning problem, thereby changing the question's answer. By testing similar questions with images that have different details, the average accuracy reflects the model's visual reasoning ability, while the difference in accuracy before and after editing the test set images effectively reveals the relationship between the model's reasoning ability and visual perception. We further designed specific metrics to expose this relationship. VFaith-Bench includes 755 entries divided into five distinct subsets, along with an additional human-labeled perception task. We conducted in-depth testing and analysis of existing mainstream flagship models and prominent open-source model series/reasoning models on VFaith-Bench, further investigating the underlying factors of their reasoning capabilities.

Figures

Figures reproduced from arXiv: 2506.11571 by the authors.

Figure 1
Figure 1. Overall view of VFaith Bench. • We developed VFaith-Bench, a novel and diverse benchmark to evaluate MLLMs’ visual reasoning ability with an emphasis on the visual faithness, and conducted extensive evaluations of mainstream models. • We introduce a cue-driven automatic and controllable editing pipeline, which is the first to leverage instruction-following image editing models for generating multimodal benchmark dat… view at source ↗
Figure 2
Figure 2. Our dual cue-driven image editing pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of different image editing models given specific image editing [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Few-shot prompt we used during generate visual cues towards origin question and answer. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Few-shot prompt we used during generate edit suggestions towards visual cues. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Zero-shot prompt we used during evaluate models on VFaith-Bench. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: An example in non-perception subset [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: The corresponding example image in Figure 7. The left side is the original image, and the [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: An example in perception subset. Note that the perception subset used in actual testing only [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The corresponding example image in Figure 9. The left side is the original image, and the [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models

    cs.LG 2026-07 conditional novelty 7.0 of 10

    CoT-Mediate shows that whether medical VLMs follow their own edited reasoning depends mostly on where it sits in the context window, not on who the text is attributed to.

Reference graph

Works this paper leans on

63 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hallucination of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

  2. [2]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier vision-language models. arXiv preprint arXiv:2504.15271, 2025

    arXiv 2025

  3. [3]

    Sft or rl? an early investigation into training r1-like reasoning large vision-language models

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468, 2025

    arXiv 2025

  4. [4]

    Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks

    Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks. arXiv preprint arXiv:2410.10563, 2024

    arXiv

  5. [5]

    M ˆ3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M ˆ3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv preprint arXiv:2405.16473, 2024

    arXiv 2024

  6. [6]

    See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning. arXiv preprint arXiv:2301.05226, 2023

    arXiv 2023

  7. [7]

    Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges

    Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287, 2023

    arXiv 2023

  8. [8]

    Video- of-thought: Step-by-step video reasoning from perception to cognition

    Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video- of-thought: Step-by-step video reasoning from perception to cognition. arXiv preprint arXiv:2501.03230, 2024

    arXiv 2024

Show all 63 references
  1. [9]

    Mme-survey: A comprehensive survey on evaluation of multimodal llms

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296, 2024

    2024 arXiv

  2. [10]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proce...

    2024

  3. [11]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025

    2025 arXiv

  4. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402....

    2024 arXiv

  6. [14]

    Ciem: Contrastive instruction evaluation method for better instruction tuning

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning. arXiv preprint arXiv:2309.02301, 2023

    2023 arXiv

  7. [15]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  8. [16]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  9. [17]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  10. [18]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 11

    2023 arXiv

  11. [19]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  12. [20]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2507–2...

    2022

  13. [21]

    Visual-o1: Understanding ambiguous instructions via multi-modal multi-turn chain-of-thoughts reasoning

    Minheng Ni, Yutao Fan, Lei Zhang, and Wangmeng Zuo. Visual-o1: Understanding ambiguous instructions via multi-modal multi-turn chain-of-thoughts reasoning. arXiv preprint arXiv:2410.03321, 2024

    2024 arXiv

  14. [22]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284, 2024

    2024 arXiv

  15. [23]

    Vlm-r1: A stable and generalizable r1-style large vision-language model

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  16. [24]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  17. [25]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  18. [26]

    Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models

    Andrés Villa, Juan Carlos León Alcázar, Alvaro Soto, and Bernard Ghanem. Behind the magic, merlim: Multi-modal evaluation benchmark for large image-language models. arXiv preprint arXiv:2312.02219, 2023

    2023 arXiv

  19. [27]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hong- sheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    2024

  20. [28]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  21. [29]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  22. [30]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  23. [31]

    Valley2: Exploring multimodal models with scalable vision-language design

    Ziheng Wu, Zhenghao Chen, Ruipu Luo, Can Zhang, Yuan Gao, Zhentao He, Xian Wang, Haoran Lin, and Minghui Qiu. Valley2: Exploring multimodal models with scalable vision-language design. arXiv preprint arXiv:2501.05901, 2025

    2025 arXiv

  24. [32]

    Llava-o1: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  25. [33]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  26. [34]

    Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319, 2024

    2024 arXiv

  27. [35]

    Tree of thoughts: Deliberate problem solving with large language models, 2023

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL https://arxiv. org/abs/2305.10601, 3, 2023

    2023 arXiv

  28. [36]

    Dapo: An open-source llm reinforcement learning system at scale

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  29. [37]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 12 A Appendix A.1 ...

    2025 arXiv

  30. [38]

    **Format Extraction**: - Always wrap reasoning in <think></think> - Extract answer to <answer></answer> without prefixes and postfixes

  31. [39]

    **Validate and correct visual cue**: - Cross-check visual cues between: a) Model’s <think> content b) Image context - If visual cues mismatch: - Correct the model’s answer based on the following rules: - Prioritize corrections in this order:

  32. [40]

    Image - Update subsequent reasoning accordingly and ensure logical consistency

  33. [41]

    Annotate visual cues: - Wrap each distinct visual cue in <vcues_*></vcues_*> - Start numbering from 1 (* = 1,2...) - Tag specific phrases, not entire paragraphs

  34. [42]

    **Output Constraints**: - Maintain original reasoning contents and flow structure as possible as you can - Ensure that all annotations (<think>, <answer>, <vcues_*>, <aha>) are properly formatted and inserted in the correct locations. - Corrected reasoning in <think> tags - An...

  35. [43]

    The screen visible in the image is small and not a touch screen

    **Option A: It has a large touch screen display** - The image shows a flip phone, which typically has a small screen. The screen visible in the image is small and not a touch screen. Therefore, this option is incorrect

  36. [44]

    This option is speculative and cannot be confirmed from the image alone

    **Option B: It cannot be used in low light conditions** - The image does not provide any information about the phone’s camera or its performance in low light. This option is speculative and cannot be confirmed from the image alone

  37. [45]

    This option is likely correct

    **Option C: It is able to take pictures** - The image shows a camera lens on the flip phone, which suggests that the phone is capable of taking pictures. This option is likely correct

  38. [46]

    The image does not show any indication of this technology

    **Option D: It has facial recognition technology** - Facial recognition technology is a modern feature not typically found in flip phones. The image does not show any indication of this technology. Therefore, this option is incorrect. Given the analysis, the most reasonable co...

  39. [47]

    <vcues_2>The screen visible in the image is small and not a touch screen</vcues_2>

    **Option A: It has a large touch screen display** - <vcues_1>The image shows a flip phone</vcues_1>, which typically has a small screen. <vcues_2>The screen visible in the image is small and not a touch screen</vcues_2>. Therefore, this option is incorrect

  40. [48]

    </vcues_3>

    **Option B: It cannot be used in low light conditions** - <vcues_3>The illuminated keys imply that it can be used in low light conditions. </vcues_3>. Therefore, this option is incorrect

  41. [49]

    This option is likely correct

    **Option C: It is able to take pictures** - <vcues_4>The image shows a camera lens on the flip phone</vcues_4>, which suggests that the phone is capable of taking pictures. This option is likely correct

  42. [50]

    <vcues_5>The image does not show any indication of this technology</vcues_5>

    **Option D: It has facial recognition technology** - Facial recognition technology is a modern feature not typically found in flip phones. <vcues_5>The image does not show any indication of this technology</vcues_5>. Therefore, this option is incorrect. Given the analysis, the...

  43. [51]

    <vcues_2>The screen visible in the image is very small and not a touch screen</vcues_2>

    Option A: It has a large touch screen display - <vcues_1>The image shows a flip phone</vcues_1>, which typically has a small screen. <vcues_2>The screen visible in the image is very small and not a touch screen</vcues_2>. Therefore, this option is incorrect

  44. [52]

    Therefore, this option is incorrect

    Option B: It cannot be used in low light conditions - <vcues_3>The illuminated keys visible in the image imply that the phone can be used in low light conditions</vcues_3>. Therefore, this option is incorrect

  45. [53]

    This option is correct

    Option C: It is able to take pictures - <vcues_4>The image shows a camera lens on the flip phone</vcues_4>, which indicates that the phone is capable of taking pictures. This option is correct

  46. [54]

    new_option

    Option D: It has facial recognition technology - Facial recognition technology is a modern feature not typically found in flip phones. <vcues_5>The image does not show any indication of facial recognition technology</vcues_5>. Therefore, this option is incorrect. Given the ana...

  47. [55]

    Condition of the Train: <vcues_1>The front of the train has visible dirt and grime</vcues_1>, which suggests it has been in use for a while and has traveled through various conditions

  48. [56]

    The presence of dirt could be from the environment it has traveled through, such as dust, debris, or even road salt in some areas

    Environment: <vcues_2>The train is on a railway track with overhead electric lines</vcues_2>, indicating it’s in a operational setting. The presence of dirt could be from the environment it has traveled through, such as dust, debris, or even road salt in some areas

  49. [57]

    Maintenance: <vcues_3>The train appears to be in service</vcues_3>, and trains often accumulate dirt and grime over time, especially if they are not cleaned regularly

  50. [58]

    The train was caught in a rainstorm: While rain can cause dirt to accumulate, the image does not show signs of recent rain, such as wet surfaces or water streaks

    Options Analysis: - A. The train was caught in a rainstorm: While rain can cause dirt to accumulate, the image does not show signs of recent rain, such as wet surfaces or water streaks. - B. The train has not been washed in some time: This is a plausible reason for the dirt on...

  51. [59]

    <vcues_1>The room has a bed with a cat lying on it, which suggests a personal living space</vcues_1>

  52. [60]

    <vcues_2>There is a television on a stand with two lamps on either side, indicating a setup for watching TV</vcues_2>

  53. [61]

    <vcues_3>The blinds on the windows are closed, and the room is lit by the lamps</vcues_3>, which suggests it might be nighttime or the room is meant to be cozy and dimly lit

  54. [62]

    <vcues_4>The presence of personal items like a cat, a bed, and a TV setup indicates a residential setting, not a hotel room</vcues_4>

  55. [63]

    Given these observations, the most reasonable inference is that the setting is a small residential room, likely a bedroom in an apartment or a small house

    <vcues_5>The room appears to be small and compact, which is more indicative of an apartment or a small house rather than a big house</vcues_5>. Given these observations, the most reasonable inference is that the setting is a small residential room, likely a bedroom in an apart...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.