VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought

Chaoya Jiang; Fei Huang; Haiyang Xu; Han Yang; Ji Zhang; Ming Yan; Shikun Zhang; Wei Ye; Yongrui Heng

REVIEW 4 major objections 5 minor 6 cited by

VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VLM-R3 claims that a vision-language model which interleaves textual reasoning with region crops and zoom-ins outperforms its base model on every benchmark tested, with the largest gain on ScienceQA (87.9 vs. 73.6).

desk verdict Real training idea, shaky evaluation: DocVQA overlap is explicit, ScienceQA gains are plausibly contaminated via M3CoT, yet the core R-GRPO method still deserves a careful referee. read the letter →

arxiv 2505.16192 v2 pith:RK65EMHD submitted 2025-05-22 cs.CV cs.AI

Chaoya Jiang , Yongrui Heng , Wei Ye , Han Yang , Haiyang Xu , Ming Yan , Ji Zhang , Fei Huang

show 1 more author

Shikun Zhang

This is my paper · ORCID

classification cs.CVcs.AI

keywords multimodalchain-of-thoughtregiongroundingreinforcementlearningGRPOvisualreasoningcropandzoominterleavedVQA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that complex visual reasoning fails when a model looks at the image once and then reasons only in text. Its proposed fix is an interleaved chain-of-thought in which the model can write text, emit a crop command with a bounding box, receive the zoomed-in sub-image, and continue. Training is bootstrapped on a curated corpus of interleaved rationales (VLIR) and then optimized by R-GRPO, a reinforcement policy that rewards correct answers, well-formed region commands, and non-redundant reasoning. On the reported benchmarks the trained 7B model beats its base model everywhere, and the largest gain is on ScienceQA (87.9% vs. 73.6%). A sympathetic reader would take this as evidence that dynamic, iterative grounding is a practical way to get finer-grained visual reasoning without scaling the model.

What carries the argument

The load-bearing object is R-GRPO, a variant of group-relative policy optimization that treats the model's own tokens and bounding-box commands as actions and the injected cropped images as environmental state updates; gradients are masked on image tokens so the policy is rewarded for choosing informative regions, not for generating pixels. The reward mixes a terminal accuracy reward, a format reward, a region-validity reward capped at 0.5 per episode, and a small reasoning-length reward, which together encourage the model to look again only when it helps. The companion VLIR corpus cold-starts the policy with step-level demonstrations, and a piecewise zoom-scaling rule maps the selected bounding-box area to a zoom factor between 1.0 and 2.0. The paper's grounding-accuracy sweep, which degrades performance as bounding boxes are perturbed, is the direct evidence that the selected region, not the reasoning text alone, carries the benefit.

What would settle it

Re-run DocVQA evaluation on the subset of test examples that were never used to build VLIR rationales or R-GRPO rollouts; if accuracy drops materially below 96.8, the reported DocVQA gain is training exposure, not generalization. A second check: fix all bounding boxes to a random region of the same size; if ScienceQA stays near 87.9, region quality is not actually what the reward is optimizing.

Watch

Extended reading notes

Core claim

The central claim is that region recognition and region reasoning belong inside the generation loop, not before it. VLM-R3 is the paper's demonstration: a 7B vision-language model whose output stream can switch between natural language and structured crop commands, with the environment injecting the requested visual detail so the model can revise its reasoning mid-chain. The paper argues this yields a new state of the art in zero-shot and few-shot settings for fine-grained visually grounded inference, with consistent gains across MathVista, MathVision, MMMU, MME, ScienceQA, DocVQA, and HallusionBench. The mechanism it credits is not the extra text but the extra visual evidence: the ablation that removes the interleaved image while keeping the localized text drops sharply on fine-grained benchmarks such as ScienceQA.

Load-bearing premise

The load-bearing premise is that evaluation benchmarks, especially DocVQA, are cleanly separated from the data used to build VLIR and to sample R-GRPO rollouts, so the reported scores measure generalization rather than training exposure.

Editorial extensions

If this is right

If the claim is correct, interleaved region grounding is a transferable recipe: the same crop-command loop should work for chart reading, OCR-heavy documents, and spatial-relation questions without scaling parameters.
The masking of injected image tokens in the policy gradient is a general trick for any tool-augmented language model whose environment returns images.
The ablation results imply that textual descriptions of regions are a poor substitute for the actual cropped pixels, so future systems should keep image evidence in the context window.
The grounding-accuracy sensitivity result implies that improving region proposal quality, not just the reasoning text, is a direct lever on final accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the paper does not fully separate the DocVQA benchmark from the DocVQA samples used to build VLIR and R-GRPO, so the reported DocVQA score of 96.8 should be read as an upper bound until a clean split is verified.
Editorial extension: one testable prediction of the grounding-sensitivity result is that an oracle region proposal (perfect bounding boxes) would lift scores further; the paper's sweep from 40% to 90% grounding accuracy implies a monotone continuation toward the oracle.
Editorial extension: because the region-validity reward is syntactic and non-redundancy based, a future comparison could measure whether the model's chosen regions match human-marked informative regions, which would test whether the reward is selecting semantically useful evidence or merely well-formed boxes.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: there is a real idea here, but the evaluation is not clean enough to trust as written. The core of the paper—make the model emit crop commands, inject the cropped image back into the context, and mask gradients for those injected image tokens—is a sensible and novel adaptation of GRPO. The ablations support that each component contributes, and the gains on MathVista, MathVision, and MMMU are on benchmarks not listed among the training sources. The VLIR dataset, if released, would be a useful resource. So this is not a throwaway.

The problem is benchmark hygiene. DocVQA is explicitly a training source: Section 3.1.1 lists it in VLIR construction and A.3 lists it in R-GRPO sampling, yet Table 1 reports it as a test benchmark with no statement about splits. That row should not be there. The bigger concern is ScienceQA, where the paper claims the largest gain (87.9 vs 73.6). R-GRPO sampling uses M3CoT, a composite benchmark that commonly inherits items from existing datasets, and ScienceQA is one of the usual constituents. The stress-test note is right: the paper does not disclose the composition of the M3CoT sample or which split was used. That makes the headline "state of the art" claim load-bearing on an unresolved overlap. If part of the ScienceQA gain is memorization, what remains is smaller margins on MathVista/MMMU—still positive, but not the story the abstract tells.

Other soft spots are minor in comparison: no error bars, no code or data release, and no random-crop baseline. A random-crop control would isolate whether the benefit comes from selecting the region or merely from having an extra view. The region-validity reward also mirrors the VLIR filtering criterion; as a training mechanism that is fine, but it should be acknowledged.

My take: this deserves a serious referee, not a desk reject. The method is substantive and the masking adaptation is correct in principle. But the authors need to re-evaluate on truly held-out benchmarks, state exactly what was in the M3CoT sample, and release code/data before I would trust the numbers. As written, I would treat the ScienceQA and DocVQA rows as unverified.

Referee Report

4 major / 5 minor

Summary. The paper proposes VLM-R3, a framework built on Qwen2.5-VL 7B that performs interleaved textual and visual chain-of-thought reasoning. During inference, the model can emit crop/zoom commands (as JSON bounding boxes), after which the cropped image tokens are injected back into the context and the model continues reasoning. The method has two training stages: supervised fine-tuning on a newly constructed VLIR dataset of interleaved rationales, and a reinforcement learning stage called R-GRPO that rewards accuracy, format compliance, region validity, and reasoning length. Experiments compare VLM-R3 against open- and closed-source models on MathVista, MathVision, MMMU, MME, ScienceQA, DocVQA, and HallusionBench, reporting gains over the base model on all benchmarks and an overall claim of state-of-the-art performance.

Significance. If the reported results are reliable, the work addresses a genuine and timely limitation of multimodal LLMs: the inability to dynamically re-focus on specific image regions during long reasoning chains. The design of masking environment-injected image tokens in the policy gradient is a thoughtful and sensible detail, and the ablation study attempts to attribute gains to the interleaved framework, VLIR fine-tuning, and R-GRPO. The region-grounding accuracy analysis in Section 4.5.1 also provides useful evidence that the model's performance depends on the quality of grounding. However, the significance is currently conditional on resolving train/evaluation overlap concerns, the correctness of the R-GRPO objective as written, and the lack of statistical reliability measures. The reported gains on ScienceQA and DocVQA are the empirical keystone of the claims, and both are threatened by possible leakage through the training data sources.

major comments (4)

[Sections 3.1.1, 4.2, A.3, Table 1] The paper does not establish that the evaluation benchmarks are disjoint from the data used in either training stage. DocVQA appears as a VLIR training source (1,497 samples in Section 4.2), as an R-GRPO sampling source (Section A.3), and as an evaluation benchmark (Table 1). The paper never states that the DocVQA evaluation split was excluded from both training stages, so the reported 96.8% is uninterpretable as a generalization result unless such a split is explicitly documented. In addition, Section A.3 lists M3CoT as an R-GRPO source; M3CoT is a composite benchmark that inherits items from existing datasets, potentially including ScienceQA-derived questions. Since ScienceQA shows the largest gain (87.9 vs. 73.6), the authors must disclose the exact composition of the M3CoT sample and verify that no evaluation item from ScienceQA or any other reported benchmark was present in VLIR or R-GRPO. Without this disjointness guarantee, the central generalization claim is not supported.
[Section 3.3, Eq. (2)] The importance-sampling ratio in the R-GRPO objective is written as πθ(ci|Q) / πθ(ci|Q)|no grad, which is identically 1 and therefore makes the first term in the loss constant. As written, this cannot provide the intended advantage-weighted policy gradient. The ratio should presumably be πθ/πθ_old or involve the reference policy πref. Please correct the equation and clarify how the 'no grad' annotation is meant to be interpreted. This is a load-bearing technical detail of the R-GRPO contribution.
[Abstract and Section 1] The claim of 'state of the art in zero-shot and few-shot settings' is not supported by the experimental protocol. All main results in Table 1 are obtained after supervised fine-tuning on VLIR and subsequent R-GRPO training; no zero-shot or few-shot evaluation procedure is described anywhere in the paper. Furthermore, Table 1 itself shows that Gemini-2 Flash and o1 exceed VLM-R3 on MathVista, MMMU, and DocVQA. The global 'new state of the art' wording is therefore inaccurate unless it is explicitly restricted to open-source reasoning MLLMs, and the zero-shot/few-shot claim should either be removed or supported by dedicated experiments.
[Section 4.3, Tables 1 and 2] All accuracy numbers are single-run point estimates without error bars, confidence intervals, or significance tests. Several of the reported improvements over the base model are small (MathVista +2.2, DocVQA +1.1, HallusionBench +0.7), so it is impossible to assess whether these differences are stable or within run-to-run noise. The authors should report results over multiple seeds (at least three) with means and variances, and state the decoding protocol (e.g., greedy vs. sampling) for each benchmark.

minor comments (5)

[Section 4.1 and Table 1] The benchmark is referred to as 'DocQA' in Section 4.1 but 'DocVQA' in Table 1 and elsewhere; please unify the naming.
[Table 2] The parenthetical deltas in Table 2 are relative to the 'Full VLM-R3' row, not to the 'Base Model' row, but this is not stated. The Avg column (e.g., 74.4 for 'w/o Interleaved Chain-of-Thought' vs. 74.0 for 'Base Model') can otherwise appear contradictory. Please add a note explaining the reference for the deltas.
[Section 4.5.2] The text says 'As shown in the lower portion of Figure 4' when discussing the attention comparison; the correct reference is Figure 5.
[Section 1 and 3.1] The paper describes VLIR as a 'benchmark dataset,' but it is used only for training and no evaluation split is defined. Please clarify that VLIR is a training corpus, and consider releasing it with clear train/validation splits to support reproducibility.
[A.3 and overall reproducibility] No code, data, or model checkpoints are released. Given that the method is data- and RL-heavy, the lack of release plans makes the exact filtering, training, and evaluation pipeline difficult to reproduce. A supplementary reproducibility statement or release plan would strengthen the paper.

Circularity Check

1 steps flagged · score 4.0 of 10

DocVQA appears as both a training source and a reported benchmark, so that particular gain is not an independent prediction; the R-GRPO/VLIR derivation itself is otherwise non-circular.

fitted input called prediction [Section 3.1.1 and 4.2 (VLIR sources), Section A.3 (R-GRPO sources), Table 1 (DocVQA evaluation)]
"For the R-GRPO stage, we sampled approximately 5,000 data points from TextVQA [47], GQA [17], VSR [25], DocVQA [32] and M3CoT [9] datasets. ... These samples are drawn from five distinct source datasets: GQA (4,057 samples), TextVQA (3,267 samples), DocVQA (1,497 samples), InfographicsVQA (1,497 samples), and VSR (1,492 samples). ... Ours 7B 70.4 30.2 62.2 2432 87.9 96.8 62.0"

The paper uses DocVQA as a data source in both training stages: it is one of the VLIR source datasets (1,497 samples) and one of the R-GRPO sampling sources. Table 1 then reports DocVQA as an evaluation benchmark, with the model scoring 96.8 versus the base model's 95.7. The paper never states that the DocVQA evaluation split was excluded from the VLIR corpus or the R-GRPO sample. As reported, the DocVQA gain is therefore not demonstrably an out-of-training prediction; at the benchmark level, the evaluated quantity overlaps with the training input by construction unless a split separation is supplied, which the paper does not provide.

full rationale

The core method derivation is not circular: R-GRPO's rewards are defined from final-answer correctness, format adherence, syntactic region validity, and reasoning length, none of which encode the benchmark answers or reduce to the evaluation metrics. The VLIR corpus provides step-level supervision rather than a renamed version of the reported scores. The similarity between the region-validity reward and the VLIR data filter is an alignment of training objectives, not a by-construction identity with the target result. There is no load-bearing self-citation: Qwen2.5-VL and GRPO are external base artifacts, and the paper does not invoke a uniqueness theorem or prior-work ansatz by the same authors to force its design. The one directly evidenced circularity concern is DocVQA, which appears both as a training source and as a reported evaluation benchmark without disclosed split separation. The concern that M3CoT may carry ScienceQA-derived items into R-GRPO is plausible but not established by the manuscript text, so it is treated as an unresolved data-hygiene risk rather than a second circular step. The abstract's few-shot claim is not substantiated by reported experiments, but that is a completeness issue, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a small number of hand-tuned parameters (zoom thresholds, reward coefficients) and on the assumption that the benchmark evaluations are independent of training data. The DocVQA overlap is the most significant additional burden. No fundamentally new physical entities or external postulates are introduced.

free parameters (4)

Zoom scaling thresholds = r<0.125 -> scale 2.0; r>=0.5 -> scale 1.0; linear between
Hand-chosen piecewise function in Eq. (4) determines how much to zoom a cropped region; not fitted to data but affects the visual input the model sees.
Region validity reward coefficient = 0.5 per valid bbox, capped at 0.5 per episode
Hand-tuned to encourage the model to output syntactically correct, non-redundant bounding boxes; affects the policy optimization.
Reasoning length reward coefficient = 0.001 per character, capped at 0.25
Hand-tuned to encourage longer reasoning chains without excessive verbosity; a free parameter that shapes the RL objective.
RL training hyperparameters = M=5, 300 steps, learning rate 1e-6, beta=0
These GRPO settings are chosen without reported sensitivity analysis; the final scores may depend on these values.

assumptions (4)

domain assumption Cropping and zooming a selected region and feeding it back as visual tokens improves the model's ability to answer questions requiring fine-grained detail.
This is the central design premise of the pipeline (Section 3.2). The ablations support it but do not compare against a random-crop baseline.
domain assumption The benchmark evaluations use split protocols that do not overlap with the training data.
This is violated for DocVQA because it appears in both the VLIR corpus and the R-GRPO training data (Section 3.1.1 and A.3).
ad hoc to paper The filtering process using a smaller VLM and DeepSeek V3 produces high-quality rationales with valid semantic units and logical reasoning.
Rejection sampling relies on two auxiliary models; no analysis of the filter accuracy or its effect on the final dataset is provided.
standard math The GRPO advantage formulation with M=5 trajectories is a valid estimator of the policy gradient.
The method is adapted from the established GRPO estimator of Shao et al. 2024, cited as [45].

how reviews work

0 comments

Cite this review

Pith. "Pith review of VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought." pith.science (2026). https://pith.science/paper/RK65EMHD

@misc{pith2026250516192,
  author       = {Pith},
  title        = {Pith review of: VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RK65EMHD}},
  note         = {Machine review of arXiv:2505.16192}
}

abstract

Recently, reasoning-based MLLMs have achieved a degree of success in generating long-form textual reasoning chains. However, they still struggle with complex tasks that necessitate dynamic and iterative focusing on and revisiting of visual regions to achieve precise grounding of textual reasoning in visual evidence. We introduce \textbf{VLM-R$^3$} (\textbf{V}isual \textbf{L}anguage \textbf{M}odel with \textbf{R}egion \textbf{R}ecognition and \textbf{R}easoning), a framework that equips an MLLM with the ability to (i) decide \emph{when} additional visual evidence is needed, (ii) determine \emph{where} to ground within the image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved chain-of-thought. The core of our method is \textbf{Region-Conditioned Reinforcement Policy Optimization (R-GRPO)}, a training paradigm that rewards the model for selecting informative regions, formulating appropriate transformations (e.g.\ crop, zoom), and integrating the resulting visual context into subsequent reasoning steps. To bootstrap this policy, we compile a modest but carefully curated Visuo-Lingual Interleaved Rationale (VLIR) corpus that provides step-level supervision on region selection and textual justification. Extensive experiments on MathVista, ScienceQA, and other benchmarks show that VLM-R$^3$ sets a new state of the art in zero-shot and few-shot settings, with the largest gains appearing on questions demanding subtle spatial reasoning or fine-grained visual cue extraction.

Figures

Figures reproduced from arXiv: 2505.16192 by the authors.

**Figure 1.** This figure visualizes our proposed VLM-R3 approach, which integrates region grounding and refinement in an interleaved visual-textual reasoning chain. While conventional text-based reasoning fails when analyzing scenes that require dynamic, iterative, and finegrained interaction with specific visual regions, our approach succeeds by precisely identifying and focusing on critical visual elements, such as the ’Hong… view at source ↗

**Figure 2.** The visualization of the reason pipline Of our proposed methods. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Distribution of the VLIR dataset: (a) number of crops per image, (b) samples across [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Impact of region grounding accuracy on model performance across three benchmarks. Each [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Comparison of attention distribution patterns between the interleaved reasoning chain with [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: This figure presents how VLM-R3 captures details in a natural image and reasons about them. 5 Conclusion This paper introduced VLM-R3 , a novel framework enabling MLLMs to perform dynamic visual reasoning through region recognition, reasoning, and refinement. By integr…

**Figure 7.** Figure 7: This figure presents how VLM-R3 gets the text details from the document image [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 8.** Figure 8: This figure presents how VLM-R3 iteratively tracks multiple visual cues in an image. a maximum of 1605632 pixels. Because the value of the bounding box is related to the number of pixels in the input image, the setting of the range of pixels needs to be unified. 11 [P…

**Figure 9.** Figure 9: This figure presents how VLM-R3 performs complex interleaved text-image CoT reasoning. A.1.2 Zoom Scaling Rule In our pipeline, when a region is selected for closer inspection (e.g., via a "Crop" operation), a zoom operation is applied. The scaling factor for this zoom…

**Figure 10.** Figure 10: This figure presents how VLM-R3 performs complex interleaved text-image CoT reasoning. formulation(2), we set M = 5. Following the experience of related studies, we set β = 0.0, i.e., we eliminate the KL divergence constraint. Our experiments for R-GRPO were performe…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Zoom-IQA: Image Quality Assessment with Reliable Region-Aware Reasoning
cs.CV 2026-01 conditional novelty 7.0 of 10

Zoom-IQA lets a vision-language model iteratively crop and zoom into image regions before giving a quality score, improving reasoning and restoration guidance over single-pass IQA models.
InSight-doc: Agentic Visual Perception for Long-Document Understanding
cs.CV 2026-08 conditional novelty 6.0 of 10

InSight-doc trains an 8B vision-language model to zoom into document sub-regions on demand, improving long-document VQA accuracy by up to 16.4 points while cutting latency by 41-68%.
OPLD: On-Policy Latent Distillation for Multimodal Reasoning
cs.CV 2026-07 conditional novelty 6.0 of 10

On-policy distillation of a CoT-privileged teacher’s token preferences and latent trajectories yields a CoT-free student that beats prior visual-latent methods on several multimodal benchmarks.
Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing
cs.CV 2026-07 conditional novelty 6.0 of 10

Training on multi-tool visual reasoning trajectories (zoom, grounding, lines) with an attention-focused RL objective improves UHR remote-sensing VQA accuracy over single-tool zoom-in and larger base models.
Imagination Helps Visual Reasoning, But Not Yet in Latent Space
cs.CL 2026-02 conditional novelty 5.0 of 10

Intervening on latent 'imagination' tokens in three visual-reasoning models changes almost nothing, while replacing them with explicit text descriptions (CapImagine) improves benchmark scores.
Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models
cs.CV 2025-05 conditional novelty 4.0 of 10

A survey organizes multimodal reasoning research into a staged roadmap and proposes native large multimodal reasoning models that unify perception, generation, and agentic planning.

Reference graph

Works this paper leans on

65 extracted references · 19 canonical work pages · cited by 6 Pith papers

[1]

https://deepmind.google/technologies/gemini/

Gemini 2.5: Our most intelligent ai model, 2024. https://deepmind.google/technologies/gemini/

work page 2024
[2]

https://openai.com/index/introducing-o3-and-o4- mini/

Introducing openai o3 and o4-mini, 2024. https://openai.com/index/introducing-o3-and-o4- mini/

work page 2024
[3]

https://qwenlm.github.io/blog/qvq-72b-preview/

Qvq: To see the world with wisdom, 2024. https://qwenlm.github.io/blog/qvq-72b-preview/

work page 2024
[4]

https://huggingface.co/datasets/TheEighthDay/SeekWorld

Seekworld: Geolocation is a natural rl task for o3-like visual clue-tracking reasoning, 2025. https://huggingface.co/datasets/TheEighthDay/SeekWorld

work page 2025
[5]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: a visual language model for few-shot learning. InNeurIPS, volume 35, 2022

work page 2022
[6]

Gemini: A family of highly capable multimodal models.ArXiv, abs/2312.11805, 2023

Gemini Team Google Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, and Johan Schalkwyk... Gemini: A family of highly capable multimodal models.ArXiv, abs/2312.11805, 2023

arXiv 2023
[7]

Qwen-vl: A frontier large vision-language model with versatile abilities.ArXiv, abs/2308.12966, 2023

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.ArXiv, abs/2308.12966, 2023

arXiv 2023
[8]

Qwen2.5-vl technical report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. A...

arXiv 2025

Show all 65 references

[9]

M 3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024

Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought, 2024

2024
[10]

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.Trans. Mach. Learn. Res., 2023, 2022

2023
[11]

Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi...

2025
[12]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Be...

2025 arXiv
[13]

Virgo: A preliminary exploration on reproducing o1-like mllm, 2025

Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing o1-like mllm, 2025

2025
[14]

Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

2023 arXiv
[15]

Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models, 2024

Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

2024
[16]

Reinforce++: A simple and efficient approach for aligning large language models

Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025

2025 arXiv
[17]

Gqa: A new dataset for real-world visual reasoning and compositional question answering

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

2019
[18]

Mathprompter: Mathematical reasoning using large language models.arXiv preprint arXiv:2303.05398, 2023

Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models.arXiv preprint arXiv:2303.05398, 2023

2023 arXiv
[19]

Tab-cot: Zero-shot tabular chain of thought

Ziqi Jin and Wei Lu. Tab-cot: Zero-shot tabular chain of thought. InAnnual Meeting of the Association for Computational Linguistics, 2023

2023
[20]

Imagine while reasoning in space: Multimodal visualization-of-thought, 2025

Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli´c, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought, 2025

2025
[21]

Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024

Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024

2024
[22]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

Junnan Li, Dongxu Li, Silvio Savarese, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InICML, 2023

2023
[23]

Let’s verify step by step, 2023

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023

2023
[24]

Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

2024 arXiv
[25]

Visual spatial reasoning, 2023

Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, 2023

2023
[26]

Improved baselines with visual instruction tuning

Haotian Liu, Chunyuan Li, Yuheng Li, et al. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023

2023 arXiv
[27]

Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement, 2025

Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement, 2025. 17

2025
[28]

Visual-rft: Visual reinforcement fine-tuning, 2025

Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, 2025

2025
[29]

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

2024
[30]

Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

2022
[31]

V Jawahar

Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021

2021
[32]

Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021

2021
[33]

Visual-o1: Understanding am- biguous instructions via multi-modal multi-turn chain-of-thoughts reasoning.arXiv preprint arXiv:2410.03321, 2024

Minheng Ni, Yutao Fan, Lei Zhang, and Wangmeng Zuo. Visual-o1: Understanding am- biguous instructions via multi-modal multi-turn chain-of-thoughts reasoning.arXiv preprint arXiv:2410.03321, 2024

2024 arXiv
[34]

Skeleton- of-thought: Prompting llms for efficient parallel generation.arXiv preprint arXiv:2307.15337, 2023

Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton- of-thought: Prompting llms for efficient parallel generation.arXiv preprint arXiv:2307.15337, 2023

2023 arXiv
[35]

OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex ...

2024
[36]

OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

2024
[37]

Gpt-4v(ision) system card

OpenAI. Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV_System_ Card.pdf, 2023

2023
[38]

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

2022
[39]

Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl, 2025

Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl, 2025

2025
[40]

Mutual reasoning makes smaller llms stronger problem-solvers.arXiv preprint arXiv:2408.06195, 2024

Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller llms stronger problem-solvers.arXiv preprint arXiv:2408.06195, 2024

2024 arXiv
[41]

Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3...

2020
[42]

Visual chain of thought: bridging logical gaps with multimodal infillings.arXiv preprint arXiv:2305.02317, 2023

Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: bridging logical gaps with multimodal infillings.arXiv preprint arXiv:2305.02317, 2023

2023 arXiv
[43]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

2017 arXiv
[44]

Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning, 2024

Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning, 2024

2024
[45]

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

2024
[46]

Vlm-r1: A stable and generalizable r1-style large vision-language model, 2025

Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-r1: A stable and generalizable r1-style large vision-language model, 2025. 20

2025
[47]

Towards vqa models that can read

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

2019
[48]

Scaling llm test-time compute optimally can be more effective than scaling model parameters.ArXiv, abs/2408.03314, 2024

Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters.ArXiv, abs/2408.03314, 2024

2024 arXiv
[49]

A survey of reasoning with foundation models.arXiv preprint arXiv:2312.11562, 2023

Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models.arXiv preprint arXiv:2312.11562, 2023

2023 arXiv
[50]

Mm-verify: Enhancing multimodal reasoning with chain-of-thought verification, 2025

Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Tianpeng Li, Fan Yang, Zenan Zhou, and Wentao Zhang. Mm-verify: Enhancing multimodal reasoning with chain-of-thought verification, 2025

2025
[51]

Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

2025 arXiv
[52]

Llamav-o1: Rethinking step-by-step visual reasoning in llms, 2025

Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, and Salman Khan. Llamav-o1: Rethinking step-by-step visual reaso...

2025
[53]

Measuring multimodal mathematical reasoning with math-vision dataset, 2024

Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset, 2024

2024
[54]

Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

2022 arXiv
[55]

Chain-of-thought prompting elicits reasoning in large language models, 2023

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

2023
[56]

Boosting multimodal reasoning with mcts-automated structured thinking.arXiv preprint arXiv:2502.02339, 2025

Jinyang Wu, Mingkuan Feng, Shuai Zhang, Ruihan Jin, Feihu Che, Zengqi Wen, and Jianhua Tao. Boosting multimodal reasoning with mcts-automated structured thinking.arXiv preprint arXiv:2502.02339, 2025

2025
[57]

Llava-cot: Let vision language models reason step-by-step, 2025

Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2025

2025
[58]

R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization, 2025

Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization, 2025

2025
[59]

Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search, 2024

Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, and Dacheng Tao. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search, 2024

2024
[60]

Griffiths, Yuan Cao, and Karthik Narasimhan

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models.ArXiv, abs/2305.10601, 2023

2023 arXiv
[61]

mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

2023 arXiv
[62]

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

2024
[63]

Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning.Advances in Neural Information Processing Systems, 35:15476–15488, 2022

2022
[64]

Mul- timodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models.arXiv preprint arXiv:2302.00923, 2023

2023 arXiv
[65]

Reflection of thought: Inversely eliciting numerical reasoning in language models via solving linear systems

Fan Zhou, Haoyu Dong, Qian Liu, Zhoujun Cheng, Shi Han, and Dongmei Zhang. Reflection of thought: Inversely eliciting numerical reasoning in language models via solving linear systems. arXiv preprint arXiv:2210.05075, 2022. 22

2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

https://deepmind.google/technologies/gemini/

Gemini 2.5: Our most intelligent ai model, 2024. https://deepmind.google/technologies/gemini/

work page 2024

[2] [2]

https://openai.com/index/introducing-o3-and-o4- mini/

Introducing openai o3 and o4-mini, 2024. https://openai.com/index/introducing-o3-and-o4- mini/

work page 2024

[3] [3]

https://qwenlm.github.io/blog/qvq-72b-preview/

Qvq: To see the world with wisdom, 2024. https://qwenlm.github.io/blog/qvq-72b-preview/

work page 2024

[4] [4]

https://huggingface.co/datasets/TheEighthDay/SeekWorld

Seekworld: Geolocation is a natural rl task for o3-like visual clue-tracking reasoning, 2025. https://huggingface.co/datasets/TheEighthDay/SeekWorld

work page 2025

[5] [5]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: a visual language model for few-shot learning. InNeurIPS, volume 35, 2022

work page 2022

[6] [6]

Gemini: A family of highly capable multimodal models.ArXiv, abs/2312.11805, 2023

Gemini Team Google Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, and Johan Schalkwyk... Gemini: A family of highly capable multimodal models.ArXiv, abs/2312.11805, 2023

arXiv 2023

[7] [7]

Qwen-vl: A frontier large vision-language model with versatile abilities.ArXiv, abs/2308.12966, 2023

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.ArXiv, abs/2308.12966, 2023

arXiv 2023

[8] [8]

Qwen2.5-vl technical report

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. A...

arXiv 2025