Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Spotting one fake caption word improves VLM perception.

desk verdict A genuinely new RL proxy task for VLM perception with consistent external gains, but the 'unambiguous reward' claim is weakened by unaudited GPT-4 labels and a matcher relaxation that looks gameable as written. read the letter →

arxiv 2506.10128 v1 pith:7LEFEJ6I submitted 2025-06-11 cs.CV cs.LG

Xiyao Wang , Zhengyuan Yang , Chao Feng , Yongyuan Liang , Yuhang Zhou , Xiaoyu Liu , Ziyi Zang , Ming Li
show 5 more authors
Chung-Ching Lin Kevin Lin Linjie Li Furong Huang Lijuan Wang
This is my paper
classification cs.CVcs.LG
keywords reinforcementlearningvision-languagemodelshallucinationdetectionverifiablerewardvisualperceptionproxytaskViCritVLbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning has transformed text reasoning because math and code tasks have answers a script can check, but vision-language models have lacked an equally verifiable perceptual task. ViCrit closes that gap by swapping one noun phrase in a ~200-word human caption for a visually plausible but wrong description and rewarding the model for finding the wrong span via exact string match, so the reward is cheap and deterministic while the task still forces full-image scrutiny. Training Qwen2.5-VL-7B and 72B on this task improves hallucination metrics and general vision-language benchmarks, with the largest out-of-domain gains on visual math and abstract-image reasoning. The paper reads these results as evidence that fine-grained hallucination criticism teaches a transferable strategy of checking textual claims against visual evidence rather than memorizing object lists. It also releases ViCrit-Bench, a 607-sample diagnostic set whose scores track overall VLM performance.

What carries the argument

The load-bearing mechanism is the span-level exact-match reward on a single injected hallucination. A human-written caption of about 200 words forces the model to check every noun phrase against the image, while the injected span, usually two words, is short enough to grade by string equality; the paper relaxes matching so extra words copied from the caption are not penalized. Training uses Group Relative Policy Optimization (GRPO), a policy-gradient update that compares each sample's reward with the group mean, combining a 0.9 weight on answer correctness and 0.1 on format. This design is what makes the proxy task challenging yet verifiable, the property the paper identifies as the missing ingredient for RL in visual perception.

What would settle it

Take a random sample of, say, 500 training pairs and have independent human annotators verify that the injected span is actually inconsistent with the image and is the only plausible wrong span; if a substantial fraction of labels fail, the reward signal is not as clean as claimed. A stronger experiment would train a model on the same ViCrit task with shuffled or reverse labels, rewarding any span except the true one, and compare benchmark gains, which would distinguish perception learning from generic RL effects.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that ViCrit is a task that is both fully perceptually demanding and unambiguously verifiable, and that reinforcement learning on this task improves visual perception in VLMs. The 875K training instances are generated by prompting GPT-4 to alter a single noun phrase — an object, attribute, count, scene-text string, or spatial relation — in PixMo-Cap captions, and GRPO with an exact-match span reward is applied to Qwen2.5-VL-Instruct checkpoints. The measured effect is consistent across scales: at 72B, CHAIRs drops from 26.4 to 21.0, MMHal rises to 3.91, and the average of eight general VL benchmarks rises from 59.78 to 63.16, including MathVision 35.2→40.1, VLMsAreBlind 61.3→65.8, and Charxiv 45.5→49.4. The authors argue that the transfer to document, chart, and abstract-image domains, which are underrepresented in the training data, shows the model has learned how to inspect an image rather than a list of seen objects.

Load-bearing premise

The central assumption is that the 875K automatically generated hallucination labels are correct and unambiguous, so the exact-match reward is a true signal of perception rather than a noisy or superficial one.

Editorial extensions

If this is right

  • ViCrit RL training reduces hallucination on both scales: at 72B, CHAIRs falls from 26.4 to 21.0 and MMHal reaches 3.91, surpassing the open- and closed-source models compared in the paper.
  • General vision-language accuracy rises by 2.40 points averaged over eight benchmarks for the 7B model and 3.38 points for the 72B model, with the biggest gains on MathVision, VLMsAreBlind, and Charxiv.
  • Supervised fine-tuning on the same caption data reduces hallucinations but does not produce the same general benchmark gains, isolating the RL mechanism as the source of generalization.
  • ViCrit-Bench accuracy orders models almost identically to their average performance on eight VL benchmarks (r = 0.96), so zero-shot performance on the benchmark may serve as an indicator of overall VLM capability.
  • The benchmark's results also expose persistent weaknesses: spatial hallucinations remain hard even for the best models, and RL training can hurt spatial and scene-text error detection, which the paper attributes to training-set imbalance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension implied by the paper's interpretation is to train with a random or reversed reward, such as rewarding any span except the true one; if the same benchmark gains appear, they may come from generic RL regularization rather than perception-specific learning.
  • If ViCrit-Bench scores truly track general VLM capability with near-monotonic ordering, the benchmark could be used as a low-cost early-stopping or model-selection signal before expensive RL runs, a use the paper does not develop.
  • The 72B model's relatively small ViCrit-Bench gain (42.4 to 43.0) is explained by the paper as saturation on easy training data; a testable extension is to generate harder instances, such as multiple corruptions per image or errors requiring multi-step inference, and see whether the 72B gains scale.
  • Since the training set is 59% natural images and only 7% abstract images, the large abstract-image and math gains suggest the learned strategy transfers across image types; an implicit extension is that adding more abstract, document, or 3D data to ViCrit-style training could drive further gains on those benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces ViCrit, a reinforcement learning proxy task for vision-language models in which a model must localize a single synthetic hallucination injected into a long human-written image caption, with a deterministic exact-match reward. The authors train Qwen2.5-VL-7B and 72B with GRPO on 875K GPT-4-generated hallucination examples and report consistent gains across ten vision-language benchmarks, including hallucination metrics, mathematical reasoning, abstract visual reasoning, and chart understanding. They also release ViCrit-Bench, a 607-sample human-validated diagnostic benchmark spanning four image domains and eight hallucination types, and report that ViCrit-Bench scores correlate strongly with average performance on general VL tasks (r=0.96). The central claim is that fine-grained hallucination criticism is a challenging yet unambiguously verifiable RL objective that improves generalizable visual perception.

Significance. If the central claim holds, ViCrit would be an important step toward bringing outcome-reward RL to visual perception, an area where verifiable tasks are scarce. The paper has notable strengths: it ships code, model weights, training data, and a benchmark; it includes a caption-SFT control (Appendix B) showing that captioning supervision alone does not reproduce the general benchmark gains; and the reported improvements on external benchmarks such as MathVision, VLMsAreBlind, and Charxiv provide evidence that effects transfer beyond the training distribution. The ViCrit-Bench construction includes human validation and cross-checking, which is a meaningful quality signal. The main risk is that the 'unambiguous exact-match reward' premise depends on the validity of 875K unaudited GPT-4 labels and on the exact string-matching relaxation, neither of which is fully established in the manuscript.

major comments (5)
  1. [Section 3.1, Data] The validity of the exact-match reward rests on the assumption that each of the 875K GPT-4-injected hallucinations is visually wrong, semantically plausible, and unique. No human validation, annotator agreement, or automatic quality filter is reported for the training data. If a nontrivial fraction of injected spans are in fact consistent with the image, or if multiple equally plausible alternative spans exist, then the reward assigns +1 or 0 to a non-perceptual decision and the RL signal is partly noise. I request a label-validity audit: sample at least a few hundred training pairs, have human annotators judge whether each injected span is a genuine hallucination and whether the ground-truth span is the unique plausible correction, and report agreement and error rates. Also, running an image-blind baseline (caption-only) on a sample of the training set would estimate how much of the task can be solved from caption priors alone.
  2. [Section 3.2, reward relaxation] The string-matching relaxation is described as: 'the model is not penalized for copying additional words before or after the selected span, as long as they are an exact copy from the original caption C.' As written, this permits a prediction equal to the entire corrupted caption: such a prediction contains the gold span, and every additional word is an exact copy from the original caption (the prefix and suffix around the replaced span are unchanged). That would make the reward trivially gameable by copying the input rather than localizing the hallucination, undermining the claim that the reward is unambiguous. Please specify the exact matching procedure—for example, whether the predicted span must be a contiguous substring of the corrupted caption and must match the gold span with at most a one-token boundary tolerance—and provide the matcher code or pseudocode. If the full-caption prediction is actually penalized by the released implementation, state that explicitly in the paper.
  3. [Sections 4 and 5.2, ViCrit-Bench] ViCrit-Bench is constructed from the same PixMo-Cap source and the same hallucination-injection procedure as the ViCrit training data, and the paper does not report any deduplication between the 607 benchmark images and the 875K training pairs. Consequently, the ViCrit-RL gains on ViCrit-Bench in Table 2 are partly attributable to train/eval distribution overlap rather than to generalizable perception. I request: (i) an explicit statement of whether any of the ViCrit-Bench images or captions appear in ViCrit-Train; (ii) if overlap exists, results reported on the non-overlapping subset; and (iii) the correlation in Figure 5 recomputed either on models not trained on ViCrit data or after excluding the ViCrit-RL checkpoints, with the number of points and a confidence interval reported.
  4. [Section 5.1, Table 1 and 5.2] All benchmark numbers are single runs with no error bars or seed counts. RL training is stochastic, and several reported deltas are small—for example, at 7B scale MMHal improves by +0.03 and MMStar by +0.2—so the 'consistent gains' claim would be more convincing with variance estimates from multiple runs or at least multiple evaluation samples. I request reporting standard deviations or confidence intervals, or clarifying the evaluation protocol (e.g., greedy decoding vs. sampling) and whether any of the reported numbers are averaged over seeds.
  5. [Section 5.2, spatial and text regression] The authors report a 'significant drop in accuracy for the Spatial and Text tasks after RL training' on ViCrit-Bench. This is consequential for the narrative that ViCrit improves fine-grained visual perception across perceptual dimensions, and the attribution to data imbalance is not supported by any reported training-set statistics. Please report the frequency of each of the eight hallucination types in the 875K training instances, and provide per-type training accuracy or a stratified analysis showing that the Spatial and Text regressions are explained by under-representation rather than by a systematic weakness induced by the RL objective.
minor comments (5)
  1. [Abstract and Section 3.1] There are typos: 'Starting from a 200-word captions' in the abstract and 'task the model to identity the visual hallucination' in Section 3.1 should be corrected to 'identify.'
  2. [Table 1] The InternVL2.5-78B row appears malformed: the entry '51.768.768.9' and nearby values lack clear separators, making the MathVerse, MMMU, and MMStar columns hard to read. Please reformat the table.
  3. [Figure 5] The correlation figure does not state the number of points, the model selection, or a confidence interval; please specify which models are included and report the sample size and uncertainty.
  4. [Section 5.2] The sentence 'The higher number is because of an easier question subset on foreground objects' is vague and informal; please specify which subset is easier and how that was determined.
  5. [Appendix B] The table caption 'Comparison between ViCrit-RL and ViCrit-RL with using same captioning data for SFT' is garbled; it should read something like 'Comparison between ViCrit-RL and caption-SFT using the same base data.' Also clarify that the SFT used 900k captioning samples while ViCrit used 875K pairs, and why these are treated as comparable.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the central claims rest on external benchmarks, with only minor in-domain benchmark overlap.

full rationale

The paper's central claim is empirical: reinforcement learning on the ViCrit proxy task transfers to ten vision-language benchmarks. Those benchmarks (CHAIR, MMHal, MathVista, MathVision, MathVerse, MMMU, MMStar, MM-Vet, VLMsAreBlind, Charxiv) are external, pre-existing evaluation suites that are not used to construct the RL reward, and no parameter is fitted to them. The reported gains on these benchmarks are therefore not circular. The ViCrit-Bench diagnostic is drawn from the same PixMo-Cap caption source used for training, so the in-domain ViCrit-Bench improvements are partly self-referential, but the paper does not use ViCrit-Bench as the primary evidence for generalization; it explicitly compares against external benchmarks and against caption-SFT. Concerns about unaudited GPT-4-generated labels and the string-match relaxation are reward-validity or data-quality issues, not circularity: they do not make a prediction reduce to its inputs by construction. No load-bearing self-citation, fitted-input-as-prediction, or definitional equivalence was found.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or model entities are postulated; the new artifacts are a task and a benchmark, not entities requiring independent evidence. The central claim rests on data quality assumptions rather than mathematical axioms.

free parameters (1)
  • reward weighting = 0.9 * r_answer + 0.1 * r_format
    Hand-chosen combination of answer and format rewards (Section 3.2); no sensitivity analysis is reported, so the contribution of the format term to the results is unknown.
assumptions (3)
  • domain assumption PixMo-Cap captions are accurate, exhaustive, and human-quality
    Used as the substrate for both RL training data and ViCrit-Bench (Sections 3.1 and 4.2); caption errors would corrupt labels and rewards.
  • ad hoc to paper GPT-4 injected hallucinations are visually wrong, unambiguous, and unique
    The 875K training labels are generated without human validation (Section 3.1, Data); the paper's 'unambiguous' reward claim depends on this.
  • domain assumption Exact string match is a valid correctness measure for hallucination localization
    The reward and ViCrit-Bench metric equate correctness with span match (Sections 3.2 and 4.4); paraphrased answers are counted wrong even if perceptually correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs." pith.science (2026). https://pith.science/paper/7LEFEJ6I

@misc{pith2026250610128,
  author       = {Pith},
  title        = {Pith review of: ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7LEFEJ6I}},
  note         = {Machine review of arXiv:2506.10128}
}
read the original abstract

Reinforcement learning (RL) has shown great effectiveness for fine-tuning large language models (LLMs) using tasks that are challenging yet easily verifiable, such as math reasoning or code generation. However, extending this success to visual perception in vision-language models (VLMs) has been impeded by the scarcity of vision-centric tasks that are simultaneously challenging and unambiguously verifiable. To this end, we introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions. Starting from a 200-word captions, we inject a single, subtle visual description error-altering a few words on objects, attributes, counts, or spatial relations-and task the model to pinpoint the corrupted span given the image and the modified caption. This formulation preserves the full perceptual difficulty while providing a binary, exact-match reward that is easy to compute and unambiguous. Models trained with the ViCrit Task exhibit substantial gains across a variety of VL benchmarks. Crucially, the improvements transfer beyond natural-image training data to abstract image reasoning and visual math, showing promises of learning to perceive rather than barely memorizing seen objects. To facilitate evaluation, we further introduce ViCrit-Bench, a category-balanced diagnostic benchmark that systematically probes perception errors across diverse image domains and error types. Together, our results demonstrate that fine-grained hallucination criticism is an effective and generalizable objective for enhancing visual perception in VLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.

  2. DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences

    cs.CV 2026-06 conditional novelty 6.0 of 10

    Rule-reward training on controllable cross-video differences (Grounding + MCQ) improves Video MLLM local spatiotemporal evidence localization and transfers to general video QA benchmarks.

  3. LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model

    cs.CV 2025-08 conditional novelty 6.0 of 10

    RL training on preference-labeled critic data transforms a 7B vision-language model into both a stronger critic and a stronger generative policy, improving average benchmark accuracy by 5.7% and enabling self-critique...

  4. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

80 extracted references · 17 canonical work pages · cited by 4 Pith papers

  1. [1]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425--2433, 2015

    work page 2015

  2. [2]

    Blip3-kale: Knowledge augmented large-scale dense captions

    Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, et al. Blip3-kale: Knowledge augmented large-scale dense captions. arXiv preprint arXiv:2411.07461, 2024

    arXiv 2024

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2 0 (3): 0 8, 2023

    2023

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

    1901

  6. [6]

    R1-v: Reinforcing super generalization ability in vision language models with less than \ 3

    Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision language models with less than \ 3. https://github.com/Deep-Agent/R1-V, 2025. Version \(v0.2\) released 3 Feb 2025

    work page 2025

  7. [7]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370--387. Springer, 2024 a

    work page 2024

  8. [8]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 b

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024 b

    arXiv 2024

Show all 80 references
  1. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024 c

    2024 arXiv

  2. [10]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  3. [11]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  4. [12]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  5. [13]

    Enhancing large vision language models with self-training on image comprehension

    Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Y Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training on image comprehension. Advances in Neural Information Processing Systems, 37: 0 131369--131397, 2024

    2024

  6. [14]

    Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement, 2025. URL https://arxiv.org/abs/2503.17352

    2025 arXiv

  7. [15]

    Virtex: Learning visual representations from textual annotations

    Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11162--11173, 2021

    2021

  8. [16]

    Refocus: Visual editing as a chain of thought for structured image understanding

    Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, and Cha Zhang. Refocus: Visual editing as a chain of thought for structured image understanding. arXiv preprint arXiv:2501.05452, 2025. URL https://arxiv.org/abs/2501.05452

    2025 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, and many others. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  10. [18]

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444, 2025

    2025 arXiv

  11. [19]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. Includes AIME and other competition-level problems

    2021

  12. [20]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models, 2025. URL https://arxiv.org/abs/2503.06749

    2025 arXiv

  13. [21]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709, 2019

    2019

  14. [22]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  15. [23]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  16. [24]

    Graph chain-of-thought: Augmenting large language models by reasoning on graphs

    Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. Graph chain-of-thought: Augmenting large language models by reasoning on graphs. arXiv preprint arXiv:2404.07103, 2024. URL https://arxiv....

    2024 arXiv

  17. [25]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 0 22199--22213, 2022

    2022

  18. [26]

    Mawps: A math word problem repository

    Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages...

    2016

  19. [27]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024 a

    2024 arXiv

  20. [28]

    Multimodal foundation models: From specialists to general-purpose assistants

    Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends in Computer Graphics and Vision , 16 0 (1-2): 0 1--214, 2024 b

    2024

  21. [29]

    Describe anything: Detailed localized image and video captioning

    Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming-Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504.16072, 2025

    2025 arXiv

  22. [30]

    Let’s verify step by step

    Hunter Lightman, Karl Cobbe, Vineet Kosaraju, Yura Burda, Harri Edwards, Jan Leike, and Ilya Sutskever. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023. URL https://arxiv.org/abs/2305.20050

    2023 arXiv

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  24. [32]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306, 2024

    2024

  25. [33]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai - Wei Chang, Song - Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems (Neu...

    2022

  26. [34]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representat...

    2024

  27. [35]

    Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. URL https://arxiv.org/abs/1906.00067

    2019 arXiv

  28. [37]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025 b

    2025 arXiv

  29. [38]

    Swe-lancer: Can frontier llms earn \ 1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115, 2025

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier llms earn \ 1 million from real-world freelance software engineering? arXiv preprint arXiv:2502.12115, 2025

    2025 arXiv

  30. [39]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  31. [40]

    Point-rft: Improving multimodal reasoning with visually grounded reinforcement finetuning

    Minheng Ni, Zhengyuan Yang, Linjie Li, Chung-Ching Lin, Kevin Lin, Wangmeng Zuo, and Lijuan Wang. Point-rft: Improving multimodal reasoning with visually grounded reinforcement finetuning. arXiv preprint arXiv:2505.19702, 2025

    2025 arXiv

  32. [41]

    American invitational mathematics examination (aime) 2024: Competition problems

    Mathematical Association of America. American invitational mathematics examination (aime) 2024: Competition problems. https://artofproblemsolving.com/wiki/index.php/2024_AIME_I_Problems, 2024. Accessed 13 May 2025

    2024

  33. [42]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. URL https://api.semanticscholar.org/CorpusID:263218031

    2023

  34. [43]

    Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021

    2021 arXiv

  35. [44]

    Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl

    Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.07536, 2025. URL https://arxiv.org...

    2025 arXiv

  36. [45]

    Connecting vision and language with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part V 16, pages 647--66...

    2020

  37. [46]

    Vision language models are blind

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pages 18--34, 2024

    2024

  38. [47]

    Object hallucination in image captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156, 2018

    2018 arXiv

  39. [48]

    Visual chain of thought: bridging logical gaps with multimodal infillings

    Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: bridging logical gaps with multimodal infillings. arXiv preprint arXiv:2305.02317, 2023

    2023 arXiv

  40. [49]

    Learning visual representations with caption annotations

    Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VIII 16, pages 153--170. Springer, 2020

    2020

  41. [50]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  42. [51]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326, 2019

    2019

  43. [52]

    Aligning large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  44. [53]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processi...

    2024

  45. [54]

    Image captioners are scalable vision learners too

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances in Neural Information Processing Systems, 36: 0 46830--46855, 2023

    2023

  46. [55]

    Solving math word problems with process-and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022

    2022 arXiv

  47. [56]

    Git: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022 a

    2022 arXiv

  48. [57]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 202...

    2024

  49. [58]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Rui Xia, Damai Dai, Yifei Li, De Li Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023. URL https://arxiv.org/abs/2312.08935

    2023 arXiv

  50. [59]

    Enhancing visual-language modality alignment in large vision language models via self-improvement

    Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, et al. Enhancing visual-language modality alignment in large vision language models via self-improvement. arXiv preprint arXiv:2405.15973, 2024 b

    2024 arXiv

  51. [60]

    Scaling inference-time search with vision value model for improved visual comprehension

    Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Scaling inference-time search with vision value model for improved visual comprehension. arXiv preprint arXiv:2412.03704, 2024 c

    2024 arXiv

  52. [61]

    Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement

    Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint arXiv:2504.07934, 2025

    2025 arXiv

  53. [62]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022 b . URL https://arxiv.org/abs/2203.11171

    2022 arXiv

  54. [63]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37: 0 113569--1136...

    2024

  55. [64]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  56. [65]

    Grit: A generative region-to-text transformer for object understanding

    Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. In European Conference on Computer Vision, pages 207--224. Springer, 2024 a

    2024

  57. [66]

    Mind's eye of llms: Visualization-of-thought elicits spatial reasoning in large language models

    Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind's eye of llms: Visualization-of-thought elicits spatial reasoning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b

    2024

  58. [67]

    Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy Lillicrap, Kenji Kawaguchi, and Michael Q. Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.00451, 2024. URL https://arxiv.org/abs/2405.00451

    2024 arXiv

  59. [68]

    Llava-critic: Learning to evaluate multimodal models

    Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02712, 2024

    2024 arXiv

  60. [69]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9 0 (1): 0 1, 2023 a

    2023 arXiv

  61. [70]

    Idea2img: Iterative self-refinement with gpt-4v (ision) for automatic image design and generation

    Zhengyuan Yang, Jianfeng Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. Idea2img: Iterative self-refinement with gpt-4v (ision) for automatic image design and generation. arXiv preprint arXiv:2310.08541, 2023 b

    arXiv

  62. [71]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 0 11809--11822, 2023

    2023

  63. [72]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2024 a

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2024 a . URL https://arxiv.org/abs/2308.02490

    2024 arXiv

  64. [73]

    Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Lingfeng Ren, Linjie Li, Jianfeng Wang, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.00765, 2024 b

    2024 arXiv

  65. [74]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  66. [75]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024 a

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024 a

    2024 arXiv

  67. [76]

    Improve vision language model chain-of-thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024 b

    2024 arXiv

  68. [77]

    Multimodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. Trans. Machine Learning Research, 2024 c

    2024

  69. [78]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Sage Zheng, Darwin Hou, Yujie Pan, Xinyun Li, Amanpreet Singh, Dawn Song, Percy Liang, Jason Wei, et al. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    2023 arXiv

  70. [79]

    Easyr1: An efficient, scalable, multi-modality rl training framework

    Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyr1: An efficient, scalable, multi-modality rl training framework. https://github.com/hiyouga/EasyR1, 2025

    2025

  71. [80]

    Aligning modalities in vision large language models via preference fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024 a

    2024 arXiv

  72. [81]

    Calibrated self-rewarding vision language models

    Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622, 2024 b

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.