Pith. sign in

REVIEW 2 major objections 3 minor 82 references

Exploratory visual reasoning can be split into planning and following, and current VLMs are much better at following.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:42 UTC pith:JO4T37HP

load-bearing objection A solid new benchmark for exploratory visual reasoning that cleanly separates planning from following, but the planning metric is only as clean as the GPT-5/Qwen-generated distractors and needs extra validation before the headline claims are fully trusted. the 2 major comments →

arxiv 2512.11995 v2 pith:JO4T37HP submitted 2025-12-12 cs.CV cs.AIcs.LG

V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions

Chenrui Fan , Yijun Liang , Shweta Bhardwaj , Kwesi Cobbina , Ming Li , Tianyi Zhou This is my paper
classification cs.CV cs.AIcs.LG
keywords exploratory visual reasoningChain-of-QuestionsV-REX benchmarkvision-language modelsplanning vs. followingmulti-step visual reasoningintermediate-step evaluationvisual question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

V-REX is a benchmark for multi-step visual reasoning: 702 images-and-questions tasks, 2,504 sub-questions, spanning deduction, guessing, navigation, and retrieval. It represents each solution as a Chain-of-Questions — a sequence of sub-questions and answers — and scores models separately on Planning (choosing the useful sub-question) and Following (answering it). The paper's central claim is that this decomposition is meaningful and measurable, and that across the 32 models tested, Planning is the weaker, more variable skill. The benchmark also shows that giving a model a good chain of hints usually improves its final answer, while errors in answering sub-questions are harder to recover from than errors in choosing them. The reader should care because most visual-reasoning benchmarks only grade the final answer, leaving the exploratory process that leads to it unevaluated.

Core claim

Exploratory visual reasoning is not one ability but two, and a finite multiple-choice Chain-of-Questions can measure them separately. Ground-truth answers are supplied during Planning and ground-truth questions during Following, isolating question-selection from answer-execution. Across 32 VLMs, both abilities scale with model size and correlate with end-to-end accuracy (Pearson 0.858 Planning, 0.948 Following), but Planning is the scarcer skill: same-size models vary more in Planning, smaller models are better at Following, and models recover better from bad plans than bad answers. The bottleneck in multi-step visual reasoning is asking the right next question.

What carries the argument

The central object is the Chain-of-Questions (CoQ), a manually constructed QA chain of 2–6 interconnected sub-questions leading to a final answer. The benchmark turns each chain into multiple-choice probes: at each step, Planning presents the model with the ground-truth question alongside unrelated 'distractor' questions and asks it to pick the helpful one (ground-truth answers are given, so only question choice is tested); Following presents the ground-truth question with plausible wrong answers and asks the model to pick the correct one (ground-truth questions are given, so only answer choice is tested). This decoupling is what allows the two skills to be scored independently.

Load-bearing premise

The load-bearing premise is that the questions labeled 'distractors' in the Planning task are genuinely unhelpful to every evaluated model; they are generated by GPT-5 and filtered by Qwen3-VL-32B (Appendix 9.1), so a model's Planning score partly measures how closely its notion of a useful next question matches those two LLMs, not just its intrinsic planning skill.

What would settle it

Regenerate the Planning distractor set using a different LLM generator (or human annotators) and drop the model-based filtering step; then compare model Planning rankings to the paper's. If rankings shift substantially, the scores reflect distractor provenance rather than planning skill.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Providing a CoQ chain as a hint improves most models' final-answer accuracy, so structured intermediate questions can serve as test-time scaffolding for VLMs.
  • Planning remains the larger differentiator at every scale: models of the same size vary far more in Planning than in Following, so planning ability is not just a matter of parameter count.
  • Both intermediate skills contribute to end-to-end ability, with Following the stronger correlate; a model strong at both should beat a model strong at only one even when final answers alone look similar.
  • Errors in Following propagate sharply into the final answer, whereas wrong Planning steps are often recoverable, meaning verification effort on answers matters more than on question choices.
  • Final-answer-only benchmarks systematically miss where VLMs fail; V-REX shows the failure often sits in the question space, not the answer space.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If Planning is the genuine bottleneck, training objectives that reward choosing a useful next question (for example, preference learning over question choices) might improve open-ended, agent-like visual reasoning more than training on final-answer accuracy alone.
  • The benchmark's Planning difficulty is defined by GPT-5-generated distractors filtered by Qwen3-VL-32B; a replication with distractors from a different generator (or human-written distractors) would test whether planning scores measure a model-general skill or agreement with those generators' notion of helpfulness.
  • The same finite-option CoQ protocol could be applied to audio, video, or embodied navigation tasks, where exploration is naturally sequential; the key constraint is that each step must be reducible to a small menu of candidate questions/answers.
  • The asymmetry in recovery (models recover better from failed Planning than failed Following) suggests an actionable design: in deployed systems, let the user or an external planner pick sub-questions, and spend verification effort on the answers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper introduces V-REX, a benchmark of 702 samples and 2,504 questions spanning four reasoning categories (Deduction, Guessing, Navigation, Retrieval) and 15 scenarios. Each sample is annotated with a human-constructed Chain-of-Questions (CoQ) of 2–6 steps. The benchmark disentangles multi-step exploratory visual reasoning into two tasks: Planning, where the model selects the most helpful next sub-question among LLM-generated distractors (Eq. 3), and Following, where the model selects the correct answer among distractors at each step (Eq. 4). The authors evaluate 32 VLMs and report that CoQ hints generally improve final-answer accuracy, that performance scales with model size, that Planning and Following both correlate with overall performance, that small models favor Following over Planning, and that models recover better from failed Planning than from failed Following.

Significance. If the proposed metrics are construct-valid, V-REX is a useful contribution: it is one of the first benchmarks to evaluate exploration in question space separately from answer-following, it uses a controlled MCQ protocol, and it includes a broad model suite with both open and proprietary VLMs. The explicit disentanglement of Planning and Following, together with the detailed taxonomy and the planned public release, could support more diagnostic evaluations of multi-step visual reasoning. However, the core Planning measure depends on the quality and neutrality of the distractor questions, and the current paper does not yet provide sufficient evidence that those distractors are model-independent. The central findings, especially the planning/following asymmetry and the claim that Planning is the primary differentiator, would be stronger with explicit construct-validity checks.

major comments (2)
  1. [§4.2.2 and Appendix 9.1, Eq. (3)] The Planning task's distractor questions are generated by GPT-5 and then filtered by Qwen3-VL-32B-Instruct, which 'automatically identify[ies] the chain that causes the highest confusion.' GPT-5 is itself one of the evaluated models in Table 1. Without a human rating of distractor helpfulness, an alternative distractor set, or per-item choice distributions, the Planning metric risks measuring agreement with the two LLMs that constructed the distractors rather than a model-independent planning ability. This is load-bearing because Findings 2 and 5 and the claimed planning/following disentanglement rely on comparing Planning scores across models. Please add construct-validity evidence: e.g., expert labels of distractor helpfulness, choice distributions showing whether strong models systematically prefer any distractor, or a robustness run with a differently generated distractor set.
  2. [Appendix 12, Figures 10–11, and Table 2] The text says that 'the accuracy drops more sharply in Figure 11 than in Figure 10, which further verifies that models are generally more robust to wrong following steps than wrong planning steps.' This is inverted: Figure 11, the wrong-Following condition, drops to 0.000 at five errors, while Figure 10, the wrong-Planning condition, remains at 0.429; Table 2 also shows higher recovery from failed Planning than from failed Following. The appendix sentence contradicts the main text's Finding 5 and should be corrected, with the stepwise analysis re-interpreted accordingly.
minor comments (3)
  1. [§5.2 and Table 3] The performance-changing ratio (AccCoQ − Acc)/Acc is reported without confidence intervals or per-category sample sizes. Since some categories are small (e.g., Retrieval), adding variability measures would help assess the strength of Finding 1.
  2. [§4.3 and Table 1] The number of MCQ options varies across steps, so reporting chance-level accuracy or a random-choice baseline would aid interpretation of absolute Planning and Following scores.
  3. [Appendix 9.2] The generation prompts include an 'unhelpfulness_score', but the paper does not state how this score is used in the filtering pipeline. Please clarify whether it influences the final distractor selection.

Circularity Check

0 steps flagged

No significant circularity: human-annotated QA chains define both metrics; LLM-generated distractors affect difficulty, not the gold target.

full rationale

V-REX does not contain a derivation chain in which a prediction reduces to its input. The ground-truth QA chains are human-annotated and cross-verified by five PhD-level annotators with two rounds of verification (Section 4.2.1). Planning and Following metrics (Eqs. 3-4) are indicator-function matches against that human ground truth, not against any model-generated output. The LLM pipeline in Section 4.2.2 and Appendix 9.1 uses GPT-5 to create candidate distractors and Qwen3-VL-32B to select challenging chains; this affects the difficulty and construct validity of the Planning items, and no human rating of distractor unhelpfulness is reported. But it does not define the gold label: a model is scored correct only when it selects the human-authored question or answer. The concern that GPT-5's Planning score partly measures self-agreement with its own distractor style is a legitimate validity caveat, not a circularity in the paper's definitional or statistical chain. No parameters are fitted to the target results, and no load-bearing self-citation or imported uniqueness theorem is used.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The benchmark operates on a curated, human-annotated reasoning chain and LLM-generated distractors; the main assumptions are about the validity of the ground truth, the representativeness of MCQs, and the absence of contamination.

axioms (4)
  • domain assumption The manually annotated ground-truth QA chains are the unique correct reasoning paths for each sample.
    The benchmark's validity depends on the human experts' correctness; the paper uses two rounds of cross-verification to enforce helpfulness and ordering.
  • domain assumption Multiple-choice selection among curated distractors is a faithful proxy for open-ended planning/following ability.
    Section 3.1 argues this makes evaluation tractable, but it reduces the exploration space and may not capture the model's natural question-generation behavior.
  • ad hoc to paper GPT-5-generated distractors are contextually relevant yet unhelpful, and the filtering by Qwen3-VL-32B selects the most challenging chains.
    Appendix 9.1; the benchmark's difficulty is defined by these LLM judgments, which may not transfer to other models.
  • domain assumption Images from public benchmarks (PhysBench, MathVision, CV-Bench, InfographicQA, GeoGuessr, TimeGuessr) have not been seen by the evaluated VLMs during training.
    Appendix 7 lists sources but no contamination check is performed, so performance could be inflated.

pith-pipeline@v1.3.0-alltime-deepseek · 24534 in / 9046 out tokens · 81715 ms · 2026-08-03T16:42:20.274466+00:00 · methodology

0 comments
read the original abstract

While many vision-language models (VLMs) are developed to answer well-defined, straightforward questions with highly specified targets, as in most benchmarks, they often struggle in practice with complex open-ended tasks, which usually require multiple rounds of exploration and reasoning in the visual space. Such visual thinking paths not only provide step-by-step exploration and verification as an AI detective but also produce better interpretations of the final answers. However, these paths are challenging to evaluate due to the large exploration space of intermediate steps. To bridge the gap, we develop an evaluation suite, ``Visual Reasoning with multi-step EXploration (V-REX)'', which is composed of a benchmark of challenging visual reasoning tasks requiring native multi-step exploration and an evaluation protocol. V-REX covers rich application scenarios across diverse domains. V-REX casts the multi-step exploratory reasoning into a Chain-of-Questions (CoQ) and disentangles VLMs' capability to (1) Planning: breaking down an open-ended task by selecting a chain of exploratory questions; and (2) Following: answering curated CoQ sequentially to collect information for deriving the final answer. By curating finite options of questions and answers per step, V-REX achieves a reliable quantitative and fine-grained analysis of the intermediate steps. By assessing SOTA proprietary and open-sourced VLMs, we reveal consistent scaling trends, significant differences between planning and following abilities, and substantial room for improvement in multi-step exploratory reasoning.

Figures

Figures reproduced from arXiv: 2512.11995 by Chenrui Fan, Kwesi Cobbina, Ming Li, Shweta Bhardwaj, Tianyi Zhou, Yijun Liang.

Figure 1
Figure 1. Figure 1: Overview of Chain-of-Questions (CoQ). The left repre￾sents the manually formulated ground truth QA chain. The middle represents the Planning task, evaluating the model’s capability in selecting sub-questions that are helpful to answer the original question. The right represents the Following task, evaluating the model’s capability in answering each sub-question. objects of interest, and rarely adjust their… view at source ↗
Figure 2
Figure 2. Figure 2: An example from V-REX with corresponding planning and following tasks. In the planning task, the model is given the original question and asked to select a sub-question in each step that is necessary and helpful for solving the original question. In the following task, the model is asked to answer the ground truth sub-questions step-by-step. can be addressed by single-round QA or language space reasoning. … view at source ↗
Figure 3
Figure 3. Figure 3: Scenarios in V-REX. With various samples, V-REX spans 15 real-world scenarios across 4 reasoning categories (Deduction, Guessing, Navigation, & Retrieval), covering diverse settings such as diagrams, time estimation, GUI interpretation, and others. strating the importance of exploration in visual reasoning. 2. The scaling law holds for V-REX, and models of the same size show much less variance in Following… view at source ↗
Figure 4
Figure 4. Figure 4: Statistics of V-REX, including question distributions (left) and reasoning-step distributions (right). in the chain, we augment the correct answer with several incorrect alternatives. The model p is tasked with selecting the correct answer to question Q∗ t at t step, conditioned on the image and the accumulated conversation history: AQ∗ t ∼ p(·|Q ∗ 1, AQ∗ 1 , Q∗ 2, AQ∗ 2 , . . . , Q∗ t−1, AQ∗ t−1 , Q∗ t ) … view at source ↗
Figure 5
Figure 5. Figure 5: The ratio of change on final accuracy brought by CoQ evaluation across all VLMs. The X-axis denotes task categories, and the Y-axis represents the performance changing ratio. 5.2. Main Findings Finding 1 By following the hints of CoQ, VLMs consistently achieve better performance on final questions, demon￾strating the importance of exploration in visual reasoning. To assess the effect of the hints in CoQ in… view at source ↗
Figure 8
Figure 8. Figure 8: presents the ratio of Following to Planning abil￾ity as a function of model size. The results indicate that smaller models exhibit a pronounced imbalance, with sub￾0.4 0.5 0.6 0.7 0.8 Planning Ability 0.4 0.5 0.6 0.7 0.8 Overall Performance R=0.858 p<1e-4 0.5 0.6 0.7 0.8 Following Ability 0.4 0.5 0.6 0.7 0.8 R=0.948 p<1e-4 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 6
Figure 6. Figure 6: Following and Planning ability on models of different sizes (logarithmic scale for the x-axis). Overall, the model’s ability on both tasks positively correlates with model size. Notably, the variance of following ability among same-sized models is smaller than that of Planning ability. Finding 4 Smaller models are better at Following than Planning while larger models have more balanced performance [PITH_F… view at source ↗
Figure 7
Figure 7. Figure 7: Correlation between Following and Overall performance (left), and between Planning and Overall performance (right). Both Following and Planning abilities are positively correlated with the overall performance of models. Balanced Following > Planning [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Generation Pipeline for Planning. For each QA chain, GPT-5 generates candidate distractors using two complementary strategies: (1) step-level, which introduces diverse distracting questions at each reasoning step, and (2) chain-level, which constructs full distracting reasoning chains. For each sample, we select 1 most challenging distracting chain and 2 diverse distracting questions per step to form the f… view at source ↗
Figure 10
Figure 10. Figure 10: Stepwise result of recovery from failed planning 0 1 2 3 4 5 Number of Wrong Following Steps 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy (Averaged over All Models) 0.880 0.626 0.468 0.324 0.204 0.000 Final Accuracy by Number of Wrong Following Steps [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Stepwise result of recovery from failed following offer little value and may not influence the final prediction [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Success case for Planning task. Without CoQ, the model fails to reach the correct final answer due to perceptual and counting errors. Under the Planning setting, however, once the model selects the correct intermediate question, it receives the corresponding answer, which compensates for its perceptual limitations and enables it to arrive at the correct final prediction. Final Question: What is the comple… view at source ↗
Figure 13
Figure 13. Figure 13: Success case for Following task. By decomposing the final question into informative sub-questions, the model correctly identifies the algorithm in the flowchart and uses this intermediate insight to arrive at the correct final answer. 13.4. Failure cases of Following Although the human-designed CoQ path is intended to guide the model to answer the final question with manually-crafted decomposition steps, … view at source ↗
Figure 14
Figure 14. Figure 14: Failure case in the Retrieval category where CoQ provides limited benefit. The model can directly identify the correct word from the puzzle without CoQ. However, when required to follow or plan using the CoQ chain, the model becomes misled by intermediate steps and incorrectly answers “Whistle.” The example shows how structured CoQ exploration can interfere with tasks that primarily require precise visual… view at source ↗
Figure 15
Figure 15. Figure 15: Failure case in the Retrieval category where CoQ provides limited benefit. Without CoQ, the model naturally checks each car one by one and correctly identifies three fully visible cars. Under Planning, the imposed CoQ chain forces the model into a reasoning path that diverges from its native strategy, leading to incorrect intermediate questions and an incorrect final count. Even under Following, the model… view at source ↗
Figure 16
Figure 16. Figure 16: Failure case in the Planning task. Although the model correctly identifies the repeating cycle and the position of the fourteenth animal, its third step selects an irrelevant question about birds. This detour misleads the final prediction, causing the model to answer with a bird rather than the correct frog. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Failure case in the Planning task. The model selects a chain of CoQ steps centered on color descriptions, which are uninformative for determining the intended concept of the drawing. This distractive chain introduces additional noise and prevents the model from connecting the pig and canned meat sketches to the correct answer “Spam.” 26 [PITH_FULL_IMAGE:figures/full_fig_p026_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Failure case in the Following task. Although the model correctly answers all the intermediate questions about the book cover, it fails to leverage the information extracted from previous reasoning steps to answer the final question correctly. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Failure case in the Following task. The model incorrectly answers the intermediate question about the least numbers of protusions and indentations of the missing puzzle piece, leading to an incorrect final answer. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

82 extracted references · 7 linked inside Pith

  1. [1]

    Qwen2.5- vl technical report, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5- vl technical report, ...

    2025

  2. [2]

    Multi-step visual reasoning with visual tokens scaling and verification

    Tianyi Bai, Zengjie Hu, Fupeng Sun, Qiu Jiantao, Yizhen Jiang, Guangxin He, Bohan Zeng, Conghui He, Binhang Yuan, and Wentao Zhang. Multi-step visual reasoning with visual tokens scaling and verification. InThe Thirty-ninth An- nual Conference on Neural Information Processing Systems,

  3. [3]

    Comparebench: A benchmark for visual comparison reason- ing in vision-language models, 2025

    Jie Cai, Kangning Yang, Lan Fu, Jiaming Ding, Jinlong Li, Huiming Sun, Daitao Xing, Jinglin Shen, and Zibo Meng. Comparebench: A benchmark for visual comparison reason- ing in vision-language models, 2025. 3

    2025

  4. [4]

    Nicol`o De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M Frankland, Thomas L

    Declan Iain Campbell, Sunayana Rane, Tyler Giallanza, C. Nicol`o De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M Frankland, Thomas L. Griffiths, Jonathan D. Cohen, and Taylor Whittington Webb. Understanding the lim- its of vision language models through the lens of the binding problem. InThe Thirty-eighth Annual Conference on Neural Information...

    2024

  5. [5]

    Unveiling chain of step reasoning for vision-language models with fine-grained rewards, 2025

    Honghao Chen, Xingzhou Lou, Xiaokun Feng, Kaiqi Huang, and Xinlong Wang. Unveiling chain of step reasoning for vision-language models with fine-grained rewards, 2025. 3

    2025

  6. [6]

    Visr-bench: An empirical study on visual retrieval-augmented generation for multilingual long document understanding,

    Jian Chen, Ming Li, Jihyung Kil, Chenguang Wang, Tong Yu, Ryan Rossi, Tianyi Zhou, Changyou Chen, and Ruiyi Zhang. Visr-bench: An empirical study on visual retrieval-augmented generation for multilingual long document understanding,

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024. 19

    Pith/arXiv arXiv 2024

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024. 6

    2024

  9. [9]

    Evaluating mllms with multimodal multi-image reasoning benchmark, 2025

    Ziming Cheng, Binrui Xu, Lisheng Gong, Zuhe Song, Tian- shuo Zhou, Shiqi Zhong, Siyu Ren, Mingxiang Chen, Xi- angchao Meng, Yuxin Zhang, Yanlin Li, Lei Ren, Wei Chen, Zhiyuan Huang, Mingjie Zhan, Xiaojie Wang, and Fangxi- ang Feng. Evaluating mllms with multimodal multi-image reasoning benchmark, 2025. 3

    2025

  10. [10]

    Physbench: Benchmarking and enhancing vision-language models for physical world under- standing.arXiv preprint arXiv:2501.16411, 2025

    Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing vision-language models for physical world under- standing.arXiv preprint arXiv:2501.16411, 2025. 13

    Pith/arXiv arXiv 2025

  11. [11]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025. 6, 19

    Pith/arXiv arXiv 2025

  12. [12]

    timeguessr, 2025

    Saloni Dalal. timeguessr, 2025. 13

    2025

  13. [13]

    Gemini 2.0 flash, 2025

    Google DeepMind. Gemini 2.0 flash, 2025. 6, 19

    2025

  14. [14]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junx- iao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, and et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. 3

    2025

  15. [15]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Win- ston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9062–9072, 2025. 3

    2025

  16. [16]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 19 9

    Pith/arXiv arXiv 2024

  17. [17]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 6, 19

    2024

  18. [18]

    Caughtcheating: Is your mllm a good cheating detective? exploring the boundary of visual perception and reasoning, 2025

    Ming Li, Chenguang Wang, Yijun Liang, Xiyao Wang, Yuhang Zhou, Xiyang Wu, Yuqing Zhang, Ruiyi Zhang, and Tianyi Zhou. Caughtcheating: Is your mllm a good cheating detective? exploring the boundary of visual perception and reasoning, 2025. 1, 3

    2025

  19. [19]

    Towards visual text grounding of multimodal large language model, 2025

    Ming Li, Ruiyi Zhang, Jian Chen, Chenguang Wang, Jiux- iang Gu, Yufan Zhou, Franck Dernoncourt, Wanrong Zhu, Tianyi Zhou, and Tong Sun. Towards visual text grounding of multimodal large language model, 2025. 1

    2025

  20. [20]

    Perception, reason, think, and plan: A survey on large multimodal reasoning models, 2025

    Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, and Min Zhang. Perception, reason, think, and plan: A survey on large multimodal ...

    2025

  21. [21]

    Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness, 2025

    Yijun Liang, Ming Li, Chenrui Fan, Ziyue Li, Dang Nguyen, Kwesi Cobbina, Shweta Bhardwaj, Jiuhai Chen, Fuxiao Liu, and Tianyi Zhou. Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness, 2025. 3

    2025

  22. [22]

    Image-based geolocation using large vision-language models, 2024

    Yi Liu, Junchen Ding, Gelei Deng, Yuekang Li, Tianwei Zhang, Weisong Sun, Yaowen Zheng, Jingquan Ge, and Yang Liu. Image-based geolocation using large vision-language models, 2024. 1

    2024

  23. [23]

    Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts, 2024. 1

    2024

  24. [24]

    Jigsaw-puzzles: From seeing to understanding to reasoning in vision-language models, 2025

    Zesen Lyu, Dandan Zhang, Wei Ye, Fangdi Li, Zhihang Jiang, and Yao Yang. Jigsaw-puzzles: From seeing to understanding to reasoning in vision-language models, 2025. 1

    2025

  25. [25]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 13

    2022

  26. [26]

    geoguessr, 2024

    Marcelo Moreno. geoguessr, 2024. 13

    2024

  27. [27]

    Gpt-5 system card, 2025

    OpenAI. Gpt-5 system card, 2025. 6, 14, 19

    2025

  28. [28]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. Technical report, OpenAI, 2025. 6, 19

    2025

  29. [29]

    Openai o1 system card, 2024

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, and et al. Openai o1 system card, 2024. 3, 6

    2024

  30. [30]

    Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and etc

    OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and etc. Gpt-4o system card, 2024. 6, 19

    2024

  31. [31]

    NTSEBENCH: Cog- nitive reasoning benchmark for vision language models

    Pranshu Pandya, Vatsal Gupta, Agney S Talwarr, Tushar Kataria, Dan Roth, and Vivek Gupta. NTSEBENCH: Cog- nitive reasoning benchmark for vision language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3680–3708, Albuquerque, New Mexico,

    2025

  32. [32]

    Capture: Evaluating spatial reasoning in vision lan- guage models via occluded object counting

    Atin Pothiraj, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Capture: Evaluating spatial reasoning in vision lan- guage models via occluded object counting. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8001–8010, 2025. 1

    2025

  33. [33]

    Vgrp- bench: Visual grid reasoning puzzle benchmark for large vision-language models, 2025

    Yufan Ren, Konstantinos Tertikas, Shalini Maiti, Junlin Han, Tong Zhang, Sabine S¨usstrunk, and Filippos Kokkinos. Vgrp- bench: Visual grid reasoning puzzle benchmark for large vision-language models, 2025. 1, 3

    2025

  34. [34]

    Atkinson, Aa- ditya Baranwal, Alexandru Coca, Mikah Dang, Sebastian Dziadzio, Jakob D

    Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, Vishaal Udandarao, Jingyi Lu, Shiyang Chen, Sam Purkis, Tianshuo Yan, Wenye Lin, Gyungin Shin, Qiaochu Yang, Anh Totti Nguyen, David I. Atkinson, Aa- ditya Baranwal, Alex...

    2025

  35. [35]

    Seeing culture: A benchmark for visual reasoning and grounding

    Burak Satar, Zhixin Ma, Patrick Amadeus Irawan, Wil- fried Ariel Mulyawan, Jing Jiang, Ee-Peng Lim, and Chong- Wah Ngo. Seeing culture: A benchmark for visual reasoning and grounding. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22238–22254, Suzhou, China, 2025. Association for Compu- tational Linguistics. 3

    2025

  36. [36]

    Vlm- r1: A stable and generalizable r1-style large vision-language model, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm- r1: A stable and generalizable r1-style large vision-language model, 2025. 3

    2025

  37. [37]

    Visualpuzzles: Decoupling multi- modal reasoning evaluation from domain knowledge, 2025

    Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. Visualpuzzles: Decoupling multi- modal reasoning evaluation from domain knowledge, 2025. 1

    2025

  38. [38]

    Reason-rft: Reinforcement fine-tuning for visual reasoning of vision language models, 2025

    Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Xiansheng Chen, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning of vision language models, 2025. 3

    2025

  39. [39]

    LlamaV-o1: Rethink- ing step-by-step visual reasoning in LLMs

    Omkar Thawakar, Dinura Dissanayake, Ketan Pravin More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Ilmuz Zaman Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fa- had Shahbaz Khan, and Salman Khan. LlamaV-o1: Rethink- ing step-by-step visual reasoning in LLMs. InFindings of the Association for Computati...

    2025

  40. [40]

    Cambrian-1: A fully open, vision-centric exploration of mul- timodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of mul- timodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024. 13

    2024

  41. [41]

    Mea- suring multimodal mathematical reasoning with math-vision 10 dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision 10 dataset.Advances in Neural Information Processing Systems, 37:95095–95169, 2024. 13

    2024

  42. [42]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. 6, 19

    Pith/arXiv arXiv 2025

  43. [43]

    Multimodal chain- of-thought reasoning: A comprehensive survey, 2025

    Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain- of-thought reasoning: A comprehensive survey, 2025. 1

    2025

  44. [44]

    Benchmarking multi- modal mathematical reasoning with explicit visual depen- dency, 2025

    Zhikai Wang, Jiashuo Sun, Wenqi Zhang, Zhiqiang Hu, Xin Li, Fan Wang, and Deli Zhao. Benchmarking multi- modal mathematical reasoning with explicit visual depen- dency, 2025. 1

    2025

  45. [45]

    Llava-cot: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. InProceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 2087–2098, 2025. 1

    2087

  46. [46]

    Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models, 2025

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wen- gang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, and Jinguo Zhu. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models, 2025. 1, 3

    2025

  47. [47]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 6, 19

    Pith/arXiv arXiv 2025

  48. [48]

    IdealGPT: Iteratively decomposing vision and language rea- soning via large language models

    Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. IdealGPT: Iteratively decomposing vision and language rea- soning via large language models. InFindings of the Asso- ciation for Computational Linguistics: EMNLP 2023, pages 11289–11303, Singapore, 2023. Association for Computa- tional Linguistics. 1

    2023

  49. [49]

    R1-zero’s ”aha moment” in visual reasoning on a 2b non-sft model, 2025

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero’s ”aha moment” in visual reasoning on a 2b non-sft model, 2025. 3

    2025

  50. [50]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 6, 19 11 Table of Contents for Appendix 7 . Image Sources . . . . . . . . . . . . . . . . . . . ....

    Pith/arXiv arXiv 2025

  51. [52]

    Experiment Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 10.1 . Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 10.2 . Evaluation Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

  52. [53]

    Final Accuracy for VLMs on V-REX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

  53. [54]

    Stepwise Recovery from Failure Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

  54. [55]

    Cases Study. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 13.1 . Success cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 13.2 . Failure cases of CoQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2...

  55. [56]

    Specifically, our dataset incorporates images from PhysBench[10], MathVision[41], CV-Bench[40], and InfographicQA[25]

    Image Sources We collect images for V-REX from a combination of website sources and publicly available visual reasoning benchmarks. Specifically, our dataset incorporates images from PhysBench[10], MathVision[41], CV-Bench[40], and InfographicQA[25]. In addition, we include real-world images sourced from the online platforms GeoGuessr[26] and TimeGuessr[1...

  56. [57]

    Detailed Taxonomy Deduction.This category addresses rule-based and logical reasoning, requiring models to explore potential causal or relational patterns to reach consistent conclusions. Exploration involves hypothesizing, verifying, and generalizing implicit rules from limited observations.Flowchart Deductionassesses whether models can reason through mul...

  57. [58]

    Data Curation forPlanning 9.1. Generation Process To evaluate VLMs’ exploratory abilities within the question space, we design aPlanningtask in which models must navigate reasoning chains while confronted with multiple plausible yet distracting questions. This setup probes whether a model can select the most informative sub-questions amid competing altern...

  58. [59]

    A final question that must be answered based on an image

  59. [60]

    If the input is xxx

    (Optional) Intermediate questions and confusing questions for earlier steps. - If this is step 1, you will only receive the final question. - If this is step N > 1, you will receive all intermediate and confusing questions from steps 1 to N-1 as context. Your task: For the current reasoning step, generate 5 confusing or misleading questions that: - Are vi...

  60. [61]

    Coherent with any prior intermediate steps

  61. [62]

    Shares entities or context with the main task

  62. [63]

    Does NOT provide causal, counting, spatial, or comparative reasoning

  63. [64]

    Focuses on irrelevant attributes or secondary objects

  64. [65]

    side-path

    Appears plausible as a next-step question. FORBIDDEN QUESTION TYPES: - Overlap with any intermediate question’s reasoning step. - Provide causal, temporal, or numerical clues needed for the final answer. - Help the model disambiguate the correct reasoning step. Prefer "side-path" confusion questions: - About background objects or irrelevant actions. - Abo...

  65. [66]

    A final question that must be answered based on the image

  66. [67]

    If the input is X

    Several intermediate questions that form the correct reasoning chain. Your task: Generate 2 confusing reasoning chains. For each gold step i, produce: - A paraphrase of the gold intermediate question_i. - One misleading question (plus its answer) for Chain 1. - One misleading question (plus its answer) for Chain 2. Each confusing question must: - Be visua...

  67. [68]

    Shares entities or scene context with the gold question

  68. [69]

    Maintains a consistent distractor storyline across steps

  69. [70]

    Avoids reasoning patterns (causal, temporal, spatial, numerical, comparative) that lead to the correct answer

  70. [71]

    Side-path

    Focuses on irrelevant or secondary attributes, actions, or objects. FORBIDDEN HELPFUL QUESTION TYPES: - Repeating or clarifying any gold reasoning step. - Providing causal, numerical, or spatial clues related to the final answer. - Helping to disambiguate or verify the correct reasoning path. PREFERRED QUESTION STYLE: - "Side-path" questions about backgro...

  71. [72]

    A final question to be answered from the image

  72. [73]

    paraphrased_gt_questions

    The gold intermediate questions forming the correct chain. YOUR GOAL: Produce multiple WRONG but PLAUSIBLE reasoning chains that appear consistent but do not help solve the final question. REQUIREMENTS: A. PARAPHRASE (MASK THE GOLD): Paraphrase each gold question_i concisely. B. TWO WRONG CHAINS: For each step, generate exactly one misleading question (wi...

  73. [74]

    Experiment Details 10.1. Implementation Details To comprehensively evaluate the exploratory capabilities of VLMs, we assess a total of32 models spanning a broad range of state-of-the-art families, including both proprietary and open-source systems. The evaluated models include GPT-4o[30], GPT- 5[27], O1[16], O3[28], Gemini-2 Flash[13], Gemini-2.5 Flash[11...

  74. [75]

    Table 3 reports the final-question accuracies of all 32 evaluated VLMs across these three evaluation modes

    Final Accuracy for VLMs on V-REX The accuracies on final questions under different settings (w/o CoQ, under thePlanningtask, and under theFollowingtask) are shown in Table 3. Table 3 reports the final-question accuracies of all 32 evaluated VLMs across these three evaluation modes. Under the w/o CoQ setting ( Acc), the model answers the final question dir...

  75. [76]

    As shown in Figure 10 and Figure 11, they display slightly different patterns

    Stepwise Recovery from Failure Analysis To study the model’s ability to recover from failure, we investigate the relationship between the number of wrong planning or following steps in CoQ versus the final accuracy of the model, averaged over all models. As shown in Figure 10 and Figure 11, they display slightly different patterns. The final accuracy decr...

  76. [77]

    Whistle

    Cases Study 13.1. Success cases We showcase representative success cases enabled by CoQ across different task settings in Figure 12 and 13. Decomposing the final question into sub-questions helps the model correct perceptual mistakes or stay aligned with the intended reasoning path, ultimately enabling it to reach the correct final answer. 13.2. Failure c...

  77. [78]

    The black SUV is fully visible in the lower left part of the image

  78. [79]

    The silver sedan is fully visible in the lower right part of the image

  79. [80]

    The silver SUV is also fully visible in the lower right part of the image

  80. [81]

    Thus, there are three cars that are completely visible: the black SUV, the silver sedan, and the silver SUV

    The black car is partially visible on the left side of the image, but only the front part is visible. Thus, there are three cars that are completely visible: the black SUV, the silver sedan, and the silver SUV. Therefore, the correct answer is (B) 3 cars are fully visible. Human-designed CoQ Where is the scene depicted in the image taking place? Next to a...

Showing first 80 references.