Pith. sign in

REVIEW 4 major objections 5 minor 88 references

Agentic visual models should call tools only when text fails—and those calls should extend capability, not cancel it out.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 02:43 UTC pith:3YHFXYJ6

load-bearing objection Solid systems paper: clearer MA/TE diagnostics plus a usable NAAR+HCE recipe that actually moves open 8B agentic VLMs, with the usual teacher-distillation caveat on how “genuine” the tool gains are. the 4 major comments →

arxiv 2607.28595 v1 pith:3YHFXYJ6 submitted 2026-07-30 cs.CV

Beacon: Knowing When and How to Perform Agentic Visual Reasoning

Qixun Wang , Yang Shi , Letian Cheng , Zhuoran Zhang , Yan He , Yuqi Tang , Qi Zhang , Xinlei Yu
show 6 more authors
Ruizhe Chen Tianrun Xu Yuanxing Zhang Pengfei Wan Haotian Wang Xianghua Ying
This is my paper
classification cs.CV
keywords agentic visual reasoningmultimodal large language modelstool usemode adaptivenesstool effectreinforcement learningcode executionvisual tools
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most agentic visual reasoners look powerful because they can crop, annotate, and run code, yet on careful measurement they barely beat plain text reasoning. Gains on hard images are largely wiped out by mistakes tools introduce on easy ones, and the models do not reliably know when tools are needed. This paper defines two yardsticks—Mode Adaptiveness (call tools when text fails, skip them when text works) and Tool Effect (net gain after subtracting tool-caused harm)—and shows prior systems sit near chance on both. Beacon is trained so that rewards favor text when any text-only rollout in the group is already correct, and so that expert answer-free hints rescue all-wrong groups so the policy can learn real tool skill. Across 13 benchmarks it leads open-source averages and posts a clearly positive tool-gain-minus-harm gap, arguing that progress should be scored by when and whether tools truly help, not by how often they are invoked.

Core claim

Existing agentic visual reasoning models have limited Mode Adaptiveness and near-zero net Tool Effect: tool gains on text-hard items are largely offset by tool harm on text-easy items. Beacon, trained with a Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion, improves overall accuracy while raising adaptiveness and producing a genuine positive gap between tool gain and tool harm.

What carries the argument

Necessity-Aware Adaptive Reward (NAAR): in each GRPO group, if any correct text-only answer exists, correct text gets full reward and correct code gets partial credit (0.25); if no text answer is correct, correct code gets full reward. Paired with Hint-Guided Capability Expansion, which injects expert answer-free hints only on all-wrong groups, then optimizes the trajectories under the original hint-free prompt.

Load-bearing premise

That a handful of on-policy rollouts (plus optional expert hints on total failures) correctly label when tools are necessary and transfer tool skill without making the model over-call tools or copy the teacher.

What would settle it

Re-run the five-rollout MA/TE protocol on the same hard/easy splits: if Beacon’s average ΔTE collapses toward zero or MAmean falls to the always-tool / never-tool 50% baseline while accuracy stays flat versus strong text-only ablations, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarking agentic VLMs should report Mode Adaptiveness and Tool-Gain vs Tool-Harm, not only aggregate accuracy.
  • RL for tool use should condition rewards on whether text-only already solves the item, rather than always encouraging or always discouraging tools.
  • Hard all-wrong groups can be recycled via answer-free hints instead of being discarded as zero-advantage noise.
  • Open-source 8B agentic models can close more of the gap to strong closed models when tool use is necessity-aware rather than habitual.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same MA/TE split could audit tool-using agents outside vision (code interpreters, web search) where “always tool” is equally common.
  • Partial credit for correct-but-unnecessary tool use may generalize as a soft efficiency prior in any multi-mode RL setting.
  • If hint transfer works mainly by teaching crop/rotate/count patterns, lighter self-generated curricula might eventually replace external expert hints.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper argues that agentic visual reasoning should be judged by Mode Adaptiveness (MA)—whether an MLLM invokes tools only when text-only reasoning is unreliable—and Tool Effect (TE)—whether tools raise success on text-hard items without harming text-easy ones. After multi-run analysis of Thyme, CodeV, DeepEyes-V2, and Metis showing weak MA and near-zero net TE, the authors train Beacon (Qwen3-VL-8B) via SFT on Gemini-synthesized code trajectories plus GRPO with Necessity-Aware Adaptive Reward (NAAR; Eq. 1) and Hint-Guided Capability Expansion (HCE; §4.4.2, Eqs. 5–6). On 13 benchmarks Beacon leads open-source averages and reports higher MA_mean and ΔTE≈+3.14% versus near-zero for priors (Table 3, Fig. 2).

Significance. If the MA/TE framing and Beacon results hold, the work usefully reframes agentic VLM evaluation away from aggregate accuracy and indiscriminate tool calling toward necessity-aware, net-positive tool use. Strengths include a clearer multi-rollout solvability protocol than single-shot labels, explicit gain/harm decomposition, ablations of NAAR and HCE (Tables 4, 6), training-dynamics evidence of mode-alignment rather than pure mode collapse (Figs. 5–7), and broad benchmark coverage. The NAAR design also addresses concrete flaws the authors identify in CodeDance, AdaTooler-V, and Metis. The contribution is primarily empirical/methodological rather than theoretical, but it is timely for the agentic multimodal literature.

major comments (4)
  1. [§4.4.2, Eqs. (5)–(6), Table 4] §4.4.2 and Eqs. (5)–(6): HCE re-rolls all-wrong groups under Gemini-3.1-Pro answer-free hints, then optimizes with the hint stripped only from the new-policy ratio while retained in the old-policy context. Table 4 attributes part of ΔTE to HCE (+2.96 GRPO+HCE vs +1.40 GRPO; full model +3.14), yet every hard trajectory lineage still originates from the same proprietary teacher used for SFT (§4.2). The manuscript does not isolate how much Tool-Gain is reusable policy skill versus distillation of teacher-chosen crops/rotations/pixel ops that standard GRPO would not discover. A load-bearing control is needed: e.g., HCE with a weaker/open teacher, frozen-hint imitation without GRPO, or TE measured only on items never seen by the teacher. Without this, the claim of “genuine tool-induced capability extension” via RL (§1, §5.3 Obs. 3, conclusion) is only partially supported.
  2. [§3.1, Table 3, Fig. 2(a)] §3.1 definitions of Tool-Gain and Tool-Harm normalize by 5N over the full test set, while MA_text/MA_tool average only over E and H. Table 3 shows Beacon’s MA_text is often low (e.g., 22.91% average; 10.77% on BabyVision) with very high tool-call rates under p_tool (70–95%), so much of ΔAcc and Tool-Gain may reflect frequent tool use plus stronger base competence rather than selective necessity detection. The paper acknowledges a code preference but still markets “improved Mode Adaptiveness.” Please report MA conditioned on difficulty bins more finely (Fig. 2a-style curves for all models on the full 13-benchmark suite), tool-call rate vs. text-only accuracy calibration error, and ΔTE restricted to samples where the model actually switches mode relative to its text-only success rate—so that adaptiveness is not conflated with a high baseline tool prior.
  3. [§4.4.1 Eq. (1), §3.1, Appendix E.1] §4.4.1 Eq. (1): I_text(G) labels necessity from whether any of N=8 on-policy rollouts is correct text-only. Appendix E.1 correctly notes that forced extra text rollouts biased labels toward text; the dual risk remains that small-N groups mislabel “text-sufficient” problems as code-necessary (or vice versa), and that the same group structure used for reward is reused in the MA/TE story. Sensitivity of final MA_mean, ΔTE, and overall accuracy to N, to the 0.25 partial-credit constant, and to the ≥4 / ≤1 easy/hard thresholds should be reported. Absent that, it is unclear whether NAAR’s gains are robust or tuned to these free parameters listed implicitly throughout §3–4.
  4. [Tables 1–3, §5.2–5.3] Tables 1–2 vs. Table 3: Main headline gains (+6.07 avg over Qwen3-VL-8B) mix perception-heavy sets where cropping helps with math sets where ΔAcc is small or negative for several models (MathVista Beacon ΔAcc +0.62, ΔTE −0.38). The abstract’s “genuine tool-induced performance gains” should be qualified by task family. Please break out MA/TE and tool-available vs tool-free accuracy for all 13 benchmarks (not only five in Table 3), and state clearly on which families tools extend capability versus where gains are mostly SFT/RL general improvement visible already in Tool-Free Acc (51.57% vs base ~49.75% on the five-set slice).
minor comments (5)
  1. [Figure 2] Figure 2(c) and related text use “DeepEyes V2” / “DeepEyes-V2” inconsistently; unify naming with the body (DeepEyesV2).
  2. [§3.2] §3.2 “Issue attribution” invokes a general limitation of RLVR without citing quantitative evidence inside this paper beyond the later HCE motivation; a forward pointer to §4.4.2 and Fig. 7 would tighten the narrative.
  3. [Appendix C.3–C.4] Prompt boxes and appendix refer to both <tool_response> and <observation> conventions; a short inference-time state machine diagram would help reproducibility alongside the promised code release.
  4. [Table 1] Table 1 lists PyVision-7B with many dashes; either complete the re-evaluation under the same protocol or mark averages as non-comparable more explicitly in the caption.
  5. [§5.3, captions] Typos/style: “T ool Effect” spacing artifacts in several figure captions; “amost zero” → “almost zero” (§5.3); arXiv ID/date in footer is fine for preprint but ensure camera-ready cleans “Preprint. Work in progress.” headers.

Circularity Check

0 steps flagged

No load-bearing circular derivation: Beacon is an empirical methods paper whose claims rest on external benchmarks and independently measured MA/TE, not on quantities forced by construction from their own fits.

full rationale

The paper does not present a first-principles derivation chain in which a claimed prediction or uniqueness result reduces to its inputs. It (1) defines Mode Adaptiveness and Tool Effect operationally via multi-rollout text-only solvability and tool-enabled correctness on held-out problems (§3.1), (2) diagnoses prior agentic models under those metrics (Fig. 2–3), (3) trains with SFT trajectories plus GRPO augmented by Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion (§4.4), and (4) reports accuracy on 13 external benchmarks plus the same MA/TE suite (Tables 1–4). Optimizing a reward that encourages adaptive tool use and then measuring MA/TE is standard goal-directed training, not self-definitional circularity: evaluation uses fixed prompts, five independent text-only rollouts, and test sets disjoint from training after overlap removal (§4.2). Teacher (Gemini) trajectories in SFT and HCE raise attribution questions—how much TE gain is policy discovery versus distilled tool strategies—but that is a validity/credit-assignment concern, not a reduction of Eq. X to Eq. Y by construction, nor a uniqueness theorem imported from overlapping authors. Self-citations are ordinary related-work pointers. No fitted parameter is renamed as an out-of-sample prediction of the same quantity. Score 1 only for the mild, non-load-bearing proximity between group-level necessity labels used in training (I_text(G)) and the evaluation notion of text-easy/text-hard; they are not identical procedures and do not force the reported ΔTE or benchmark wins.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 4 invented entities

The work is empirical ML. Load-bearing premises are standard RL assumptions, the validity of multi-sample text-only solvability as a necessity oracle, and that Gemini-synthesized trajectories/hints are useful teachers without locking the policy to non-transferable behaviors. Free parameters are training/reward knobs chosen by the authors rather than physical constants.

free parameters (5)
  • Radaptive partial credit for correct code when text succeeds = 0.25
    Set to 0.25 vs 1.0 for correct text-only when I_text(G)=1; controls the soft preference against unnecessary tools.
  • Format vs adaptive reward mix = 0.1 / 0.9
    R = 0.1 R_format + 0.9 R_adaptive; hand-chosen weighting of format compliance vs task/mode objectives.
  • Text-easy / text-hard thresholds = 4-of-5 and 1-of-5
    ≥4/5 correct → text-easy; ≤1/5 → text-hard; middle band dropped. Defines MA/TE denominators and analysis cohorts.
  • RL/SFT filtering thresholds and group size = ≤2/5 SFT, ≤3/5 RL, N=8
    SFT keeps ≤2/5 base correct; RL keeps ≤3/5 SFT correct; N=8 rollouts, batch 128, etc. Shape which problems and signals enter training.
  • GRPO clip ε and learning rates = ε=0.2; lr 1e-5 / 1e-6
    Standard PPO/GRPO hyperparameters (ε=0.2, SFT lr 1e-5, RL lr 1e-6) that affect stability and final scores.
axioms (5)
  • domain assumption Group-relative policy optimization (GRPO) with verifiable answer rewards improves tool-use policies for MLLMs.
    Base algorithm choice in §4.4; standard in recent reasoning RL but not proved optimal here.
  • ad hoc to paper Five stochastic text-only rollouts sufficiently classify tool-free solvability for mode-adaptiveness analysis.
    §3.1 metric definition; improves on single-shot labels but remains a finite-sample proxy for true necessity.
  • domain assumption Correct code-assisted trajectories and answer-free hints from Gemini 3.1 Pro are valid supervision for expanding the student policy.
    §4.2–4.4.2 data synthesis and HCE; teacher quality bounds student tool skill.
  • domain assumption Permitting rather than mandating tools in evaluation prompts reveals intrinsic mode preference.
    §3.1 prompt modification for fair MA comparison across models.
  • domain assumption Standard cross-entropy SFT with masked tool outputs plus injected easy text trajectories yields a usable cold start without catastrophic forgetting.
    §4.3 cold-start design.
invented entities (4)
  • Mode Adaptiveness (MA_text, MA_tool, MA_mean) no independent evidence
    purpose: Quantify whether the model avoids tools on text-easy items and invokes them on text-hard items.
    Author-defined metrics in §3.1; useful diagnostics but not independently standardized outside this paper.
  • Tool Effect (Tool-Gain, Tool-Harm, ΔTE) no independent evidence
    purpose: Separate capability extension on hard items from regressions on easy items under tool-available decoding.
    Author-defined in §3.1; central to the 'genuine gains' claim.
  • Necessity-Aware Adaptive Reward (NAAR) no independent evidence
    purpose: Soft preference for text-only success when available, full credit for tools when text fails in-group.
    Core training mechanism §4.4.1; validated only via ablations in this work.
  • Hint-Guided Capability Expansion (HCE) no independent evidence
    purpose: Recover learning signal from all-wrong groups via expert answer-free hints and off-hint importance sampling.
    Core training mechanism §4.4.2; related to adaptive guidance literature but specific recipe is paper-local.

pith-pipeline@v1.2.0-daily-grok45 · 33667 in / 3929 out tokens · 71841 ms · 2026-07-31T02:43:33.528163+00:00 · methodology

0 comments
read the original abstract

The fundamental goal of agentic visual reasoning is to improve the success rate of multimodal large language models (MLLMs) on complex tasks, rather than merely equipping them with a sophisticated yet inefficient reasoning paradigm. In this work, we rethink agentic visual reasoning through two key dimensions of tool use: Mode Adaptiveness (MA) and Tool Effect (TE). Mode Adaptiveness characterizes whether an MLLM can recognize when tools are truly necessary and invoke them accordingly, thereby avoiding unnecessary computational overhead while improving performance on challenging problems that require tool assistance. Tool Effect characterizes the actual impact of tool use: tools should extend the model's capabilities on problems unsolvable through text-only reasoning, while avoiding additional errors on problems that the model can already solve without tools. We conduct a comprehensive analysis to quantify these two properties and empirically reveal that existing agentic visual reasoning models exhibit limited Mode Adaptiveness, while the gains produced by tool use on hard examples are largely offset by the harm introduced on easy examples that the models can already solve. Motivated by these observations, we propose Beacon, a novel agentic visual reasoning model that achieves stronger overall performance, improved Mode Adaptiveness, and genuine tool-induced performance gains. At the core of Beacon are the Necessity-Aware Adaptive Reward and the Hint-Guided Capability Expansion mechanism in the reinforcement learning stage, which respectively encourage adaptive tool invocation based on task necessity and strengthen the model's tool-use capability on the most challenging problems. Extensive experiments across diverse benchmarks demonstrate the strong overall performance of Beacon and its substantial improvements in both Mode Adaptiveness and Tool Effect.

Figures

Figures reproduced from arXiv: 2607.28595 by Haotian Wang, Letian Cheng, Pengfei Wan, Qixun Wang, Qi Zhang, Ruizhe Chen, Tianrun Xu, Xianghua Ying, Xinlei Yu, Yang Shi, Yan He, Yuanxing Zhang, Yuqi Tang, Zhuoran Zhang.

Figure 1
Figure 1. Figure 1: Agentic visual reasoning models should use tools adaptively and effectively. ABSTRACT The fundamental goal of agentic visual reasoning is to improve the success rate of multimodal large language models (MLLMs) on complex tasks, rather than merely equipping them with a sophisticated yet inefficient reasoning paradigm. In this work, we rethink agentic visual reasoning through two key dimensions of tool use: … view at source ↗
Figure 2
Figure 2. Figure 2: (a) The tool-call ratio at different levels of text-only accuracy. (b) Average performance across 13 benchmarks. (c) Tool-induced performance gains and harms (see Section 3.1 for de￾tails). Takeaway 1 (a): Beacon exhibits adaptive tool-invocation behavior that is largely absent from existing agentic visual reasoning models, invoking tools more frequently on samples that are difficult to solve through tool-… view at source ↗
Figure 3
Figure 3. Figure 3: (a) The accuracy of tool-available and tool-free reasoning. (b) The MAtext, MAtool, and MAmean of different models. (c) The Tool-Gain and Tool-Harm of different models. Observation 1 ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the RL process of Beacon. The Necessity-Aware Adaptive Reward (NAAR) and Hint-Guided Capability Expansion (HCE) are designed to improve reasoning-mode adaptiveness and expand the model’s tool-use capabilities on the most challenging problems, re￾spectively. 4.3 COLD-START SFT We use a standard cross-entropy loss for cold-start SFT, and mask the code output (i.e., the content between <tool_c… view at source ↗
Figure 5
Figure 5. Figure 5: Training accuracy and rewards of the RL stage. (a) Mean training accuracy. (b) Adaptive reward. (c) Format reward. with no clear shift toward either reasoning mode. In contrast, [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: (a) Proportions of pure-text and code-assisted responses during training. (b) Proportions of training groups assigned the “text” and “code” adaptive labels for reward computation. (c) Accu￾racy of reasoning-mode selection, measured as the proportion of responses whose reasoning mode matches the adaptive label of their corresponding group. 1 20 40 60 80 99 Training Step 0 20 40 60 80 100 Ratio (%) Training … view at source ↗
Figure 7
Figure 7. Figure 7: (a) Proportions of groups containing at least one correct pure-text response (“w/ text correct”), groups containing no correct pure-text response but at least one correct code-assisted response (“w/o text correct & w/ code correct”), and hint-induced groups (“hinted”). (b) Proportion of initially all-incorrect groups that are converted into groups with non-zero accuracy advantages through hint-guided re-ro… view at source ↗
Figure 8
Figure 8. Figure 8: Training data sources of Beacon. AgentVista [31] provides tool-oriented and agentic visual reasoning data, where models are encour￾aged to perform long-horizon reasoning with intermediate visual operations. Finally, MMMU [43], MuirBench [35], and Vero [21] introduce broader multimodal reasoning scenarios across academic, multi-image, and general visual reasoning tasks. As shown in [PITH_FULL_IMAGE:figures… view at source ↗
Figure 9
Figure 9. Figure 9: Task category distribution of the SFT and RL data. 10 1 10 2 10 3 10 4 10 5 Number of Samples Geometry3K OlympiadBench Vero STEM Vero Chart and OCR HRScene CV-Bench Vero Grounding Counting Vero Spatial AgentVista MMMU Muir-Bench Vero Knowledge Recognition 257 136 1,618 3,282 106 210 2,570 4,090 16 454 88 2,878 Retention 12.2% 3.4% 3.5% 6.8% 12.6% 8.0% 14.3% 6.9% 7.7% 11.5% 4.1% 11.3% Original samples Retai… view at source ↗
Figure 10
Figure 10. Figure 10: Detailed statistics of the SFT data. The numbers in the bars indicate the number of samples retained after rejection sampling, which removes easy samples, and refinement, which removes code trajectories where code does not actually help to derive the answer. “Retention” denotes the percentage of retained samples relative to the original samples. • Structured trajectory format. We require the model to gene… view at source ↗
Figure 11
Figure 11. Figure 11: Detailed statistics of the RL data. The numbers in the bars indicate the number of samples retained after rejection sampling, which removes easy samples. “Retention” denotes the percentage of retained samples relative to the original samples. (e.g., q0, q1, g0, and g1), which can be accessed through get_image(ref). The final line within each <tool_call>...</tool_call> block must be the executable expressi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

88 extracted references · 15 linked inside Pith

  1. [1]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jian- qiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report....

    Pith/arXiv arXiv 2025

  3. [3]

    Babyvision: Visual reasoning beyond language

    Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Yiping Bao, et al. Babyvision: Visual reasoning beyond language. arXiv preprint arXiv:2601.06521, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. InACM MM, 2024

    2024

  5. [5]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, pp. 148–166. Springer, 2024

    2024

  6. [6]

    Gemini 3 flash.https://deepmind.google/models/ model-cards/gemini-3-flash/, 2025

    Google DeepMind. Gemini 3 flash.https://deepmind.google/models/ model-cards/gemini-3-flash/, 2025

    2025

  7. [7]

    Gemini 3.1 Pro: Model Card.https://deepmind.google/ models/model-cards/gemini-3-1-pro/, 2026

    Google DeepMind. Gemini 3.1 Pro: Model Card.https://deepmind.google/ models/model-cards/gemini-3-1-pro/, 2026

    2026

  8. [8]

    Do multimodal agents really benefit from tool use? a systematic study of capability gains.arXiv preprint arXiv:2606.02357, 2026

    Garvin Guo, Donglei Yu, Yu Chen, Xiang Wang, Shuai Li, Xinpei Zhao, Huaxing Liu, Qinghao Wang, and Minpeng Liao. Do multimodal agents really benefit from tool use? a systematic study of capability gains.arXiv preprint arXiv:2606.02357, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. InACL, 2024

    2024

  10. [10]

    Deep- eyesv2: Toward agentic multimodal model

    Jack Hong, Chenxiao Zhao, ChengLIn Zhu, Weiheng Lu, Guohai Xu, and XingYu. Deep- eyesv2: Toward agentic multimodal model. InICLR, 2026. URLhttps://openreview. net/forum?id=yDKawwfJ5O

    2026

  11. [11]

    Codev: Code with images for faithful visual reasoning via tool-aware policy optimization

    Xinhai Hou, Shaoyuan Xu, Manan Biyani, Mayan Li, Jia Liu, Todd C Hollon, and Bryan Wang. Codev: Code with images for faithful visual reasoning via tool-aware policy optimization. In CVPR, 2026

    2026

  12. [12]

    Mini-o3: Scaling up reasoning patterns and interaction turns for visual search.arXiv preprint arXiv:2509.07969, 2025

    Xin Lai, Junyi Li, Wei Li, Tao Liu, Tianjian Li, and Hengshuang Zhao. Mini-o3: Scaling up reasoning patterns and interaction turns for visual search.arXiv preprint arXiv:2509.07969, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    Tir-bench: A comprehensive benchmark for agen- tic thinking-with-images reasoning.arXiv preprint arXiv:2511.01833, 2025

    Ming Li, Jike Zhong, Shitian Zhao, Haoquan Zhang, Shaoheng Lin, Yuxiang Lai, Chen Wei, Konstantinos Psounis, and Kaipeng Zhang. Tir-bench: A comprehensive benchmark for agen- tic thinking-with-images reasoning.arXiv preprint arXiv:2511.01833, 2025

    arXiv 2025

  14. [14]

    Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. InACL, 2021. 14 Preprint. Work in progress

    2021

  15. [15]

    Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemati- cal reasoning of foundation models in visual contexts. InICLR, 2024

    2024

  16. [16]

    What does vision tool-use reinforcement learning really learn? disentangling tool-induced and intrinsic effects for crop-and-zoom

    Yan Ma, Weiyu Zhang, Tianle Li, Du Linge, Xuyang Shen, and Pengfei Liu. What does vision tool-use reinforcement learning really learn? disentangling tool-induced and intrinsic effects for crop-and-zoom. InICML, 2026

    2026

  17. [17]

    Chartqapro: A more diverse and challenging benchmark for chart question answering

    Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmo- hammadi, et al. Chartqapro: A more diverse and challenging benchmark for chart question answering. InACL Findings, 2025

    2025

  18. [18]

    Adaptive guidance accelerates reinforcement learning of reasoning models.arXiv preprint arXiv:2506.13923, 2025

    Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma, Nikhil Baharte, and Sean Hendryx. Adaptive guidance accelerates reinforcement learning of reasoning models.arXiv preprint arXiv:2506.13923, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Cogcom: A visual language model with chain-of- manipulations reasoning

    Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, and Jie Tang. Cogcom: A visual language model with chain-of- manipulations reasoning. InICLR, 2025

    2025

  20. [20]

    V-thinker: Interactive thinking with images

    Runqi Qiao, Qiuna Tan, Minghan Yang, Guanting Dong, Peiqing Yang, Shiqiang Lang, Enhui Wan, Xiaowan Wang, Yida Xu, Lan Yang, et al. V-thinker: Interactive thinking with images. arXiv preprint arXiv:2511.04460, 2025

    arXiv 2025

  21. [21]

    Vero: An open rl recipe for general visual reasoning.arXiv preprint arXiv:2604.04917, 2026

    Gabriel Sarch, Linrong Cai, Qunzhong Wang, Haoyang Wu, Danqi Chen, and Zhuang Liu. Vero: An open rl recipe for general visual reasoning.arXiv preprint arXiv:2604.04917, 2026

    Pith/arXiv arXiv 2026

  22. [22]

    Tarr, Aviral Kumar, and Katerina Fragkiadaki

    Gabriel Herbert Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning. In NeurIPS, 2025

    2025

  23. [23]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Mavors: Multi-granularity video repre- sentation for multimodal large language model

    Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, et al. Mavors: Multi-granularity video repre- sentation for multimodal large language model. InProceedings of the 33rd ACM International Conference on Multimedia, pp. 10994–11003, 2025

    2025

  26. [26]

    Megatron-lm: Training multi-billion parameter language models using model par- allelism.arXiv preprint arXiv:1909.08053, 2019

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model par- allelism.arXiv preprint arXiv:1909.08053, 2019

    Pith/arXiv arXiv 1909

  27. [27]

    Codedance: A dynamic tool-integrated mllm for executable visual reason- ing.arXiv preprint arXiv:2512.17312, 2025

    Qi Song, Honglin Li, Yingchen Yu, Haoyi Zhou, Lin Yang, Song Bai, Qi She, Zilong Huang, and Yunqing Zhao. Codedance: A dynamic tool-integrated mllm for executable visual reason- ing.arXiv preprint arXiv:2512.17312, 2025

    arXiv 2025

  28. [28]

    Visualpuz- zles: Decoupling multimodal reasoning evaluation from domain knowledge.arXiv preprint arXiv:2504.10342, 2025

    Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. Visualpuz- zles: Decoupling multimodal reasoning evaluation from domain knowledge.arXiv preprint arXiv:2504.10342, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Pixel reasoner: In- centivizing pixel space reasoning via curiosity-driven reinforcement learning

    Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: In- centivizing pixel space reasoning via curiosity-driven reinforcement learning. InNeurIPS, 2025

    2025

  30. [30]

    Openthinkimg: Learning to think with images via visual tool reinforcement learning.arXiv preprint arXiv:2505.08617, 2025

    Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, and Yu Cheng. Openthinkimg: Learning to think with images via visual tool reinforcement learning.arXiv preprint arXiv:2505.08617, 2025. 15 Preprint. Work in progress

    Pith/arXiv arXiv 2025

  31. [31]

    Agentvista: Evaluating multimodal agents in ultra-challenging realistic visual scenarios.arXiv preprint arXiv:2602.23166, 2026

    Zhaochen Su, Jincheng Gao, Hangyu Guo, Zhenhua Liu, Lueyang Zhang, Xinyu Geng, Shijue Huang, Peng Xia, Guanyu Jiang, Cheng Wang, et al. Agentvista: Evaluating multimodal agents in ultra-challenging realistic visual scenarios.arXiv preprint arXiv:2602.23166, 2026

    arXiv 2026

  32. [32]

    Game-rl: Synthesizing multimodal verifiable game data to boost vlms’ general reasoning.arXiv preprint arXiv:2505.13886, 2025

    Jingqi Tong, Jixin Tang, Hangcheng Li, Yurong Mou, Ming Zhang, Jun Zhao, Yanbo Wen, Fan Song, Jiahao Zhan, Yuyang Lu, et al. Game-rl: Synthesizing multimodal verifiable game data to boost vlms’ general reasoning.arXiv preprint arXiv:2505.13886, 2025

    arXiv 2025

  33. [33]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.NeurIPS, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.NeurIPS, 2024

    2024

  34. [34]

    Adatooler-v: Adaptive tool-use for images and videos.arXiv preprint arXiv:2512.16918, 2026

    Chaoyang Wang, Kaituo Feng, Dongyang Chen, Zhongyu Wang, Zhixun Li, Sicheng Gao, Meng Meng, Xu Zhou, Manyuan Zhang, Yuzhang Shang, and Xiangyu Yue. Adatooler-v: Adaptive tool-use for images and videos.arXiv preprint arXiv:2512.16918, 2026

    Pith/arXiv arXiv 2026

  35. [35]

    Muirbench: A comprehensive benchmark for robust multi-image understanding

    Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. InICLR, 2025

    2025

  36. [36]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. NeurIPS, 2024

    2024

  37. [37]

    Monet: Reasoning in latent visual space beyond images and language.arXiv preprint arXiv:2511.21395, 2025

    Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, and Yisen Wang. Monet: Reasoning in latent visual space beyond images and language.arXiv preprint arXiv:2511.21395, 2025

    arXiv 2025

  38. [38]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. InAAAI, 2025

    2025

  39. [39]

    V*: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. InCVPR, pp. 13084–13094, 2024

    2024

  40. [40]

    RealWorldQA.https://huggingface.co/datasets/xai-org/ RealworldQA, 2024

    xAI. RealWorldQA.https://huggingface.co/datasets/xai-org/ RealworldQA, 2024. A benchmark for evaluating real-world spatial understanding

    2024

  41. [41]

    Act wisely: Cultivating meta-cognitive tool use in agentic multimodal models.arXiv preprint arXiv:2604.08545, 2026

    Shilin Yan, Jintao Tong, Hongwei Xue, Xiaojun Tang, Yangyang Wang, Kunyu Shi, Guannan Zhang, Ruixuan Li, and Yixiong Zou. Act wisely: Cultivating meta-cognitive tool use in agentic multimodal models.arXiv preprint arXiv:2604.08545, 2026

    Pith/arXiv arXiv 2026

  42. [42]

    Position: Your VLM may not be thinking with interleaved images

    Wenjie Yang, Siqi Zhu, and Zengfeng Huang. Position: Your VLM may not be thinking with interleaved images. InICML Position Paper Track, 2026

    2026

  43. [43]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. InCVPR, 2024

    2024

  44. [44]

    Thyme: Think beyond images

    Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, Haonan Fan, Kaibing Chen, Jiankang Chen, Haojie Ding, Kaiyu Tang, Zhang Zhang, Liang Wang, Fan Yang, Tingting Gao, and Guorui Zhou. Thyme: Think beyond images. InICLR, 2026

    2026

  45. [45]

    Hrscene: How far are vlms from effective high-resolution image understanding? InICCV, 2025

    Yusen Zhang, Wenliang Zheng, Aashrith Madasu, Peng Shi, Ryo Kamoi, Hao Zhou, Zhuoyang Zou, Shu Zhao, Sarkar Snigdha Sarathi Das, Vipul Gupta, et al. Hrscene: How far are vlms from effective high-resolution image understanding? InICCV, 2025

    2025

  46. [46]

    Pyvision-rl: Forging open agentic vision models via rl.arxiv preprint arxiv:2602.20739, 2026

    Shitian Zhao, Shaoheng Lin, Ming Li, Haoquan Zhang, Wenshuo Peng, Kaipeng Zhang, and Chen Wei. Pyvision-rl: Forging open agentic vision models via rl.arxiv preprint arxiv:2602.20739, 2026. 16 Preprint. Work in progress

    arXiv 2026

  47. [47]

    Swift:a scalable lightweight infrastructure for fine-tuning.arXiv preprint arXiv:2408.05517, 2024

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scalable lightweight infrastructure for fine-tuning.arXiv preprint arXiv:2408.05517, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Chain-of-focus prompting: Leveraging sequential visual cues to prompt large autoregres- sive vision models

    Jiyang Zheng, Jialiang Shen, Yu Yao, Min Wang, Yang Yang, Dadong Wang, and Tongliang Liu. Chain-of-focus prompting: Leveraging sequential visual cues to prompt large autoregres- sive vision models. InICLR, 2025

    2025

  49. [49]

    Vtc-bench: Evaluating agentic multi- modal models via compositional visual tool chaining.arXiv preprint arXiv:2603.15030, 2026

    Xuanyu Zhu, Yuhao Dong, Rundong Wang, Yang Shi, Zhipeng Wu, Yinlun Peng, YiFan Zhang, Yihang Lou, Yuanxing Zhang, Ziwei Liu, et al. Vtc-bench: Evaluating agentic multi- modal models via compositional visual tool chaining.arXiv preprint arXiv:2603.15030, 2026. 17 Preprint. Work in progress. APPENDIXCONTENTS A Experimental Details 19 A.1 Details of the Metr...

    arXiv 2026

  50. [50]

    Retention

    Prompt for SFT code-reasoning trajectory synthesis.The prompt is provided in Box 2. Its key design choices are summarized below: •Representative code actions.We provide five representative code actions, including crop,draw line,draw box,numeric calculation, androtation, covering common visual reasoning operations. For each action, we specify its purpose, ...

  51. [51]

    Even if you can already guess or know the answer from the image, you should still use sandboxed Python snippets to mark, isolate, or verify the relevant visual evidence before answering

  52. [52]

    Original question images use stable ids like q0, q1, and images returned by earlier code calls use stable ids like g0, g1

    Use get_image(ref) to access runtime images. Original question images use stable ids like q0, q1, and images returned by earlier code calls use stable ids like g0, g1

  53. [53]

    For example: annotated = get_image("q0").copy()

    If you need to draw on an image, call copy() first so you do not modify the source image directly. For example: annotated = get_image("q0").copy()

  54. [54]

    If the current call should return a numeric result, the last line should be the numeric object itself, such as result

    If the current call should return an image result, the last line inside < tool_call> should be the PIL image object itself, such as cropped, rotated, or annotated. If the current call should return a numeric result, the last line should be the numeric object itself, such as result

  55. [55]

    Use Python code

    Do not do arithmetic mentally. Use Python code. You may import math when needed

  56. [56]

    It is not something you should generate yourself

    <tool_response>...</tool_response> is inserted by the system after the code finishes running. It is not something you should generate yourself. Never output <tool_response> tags on your own

  57. [57]

    After each code execution returns a result, you must immediately output an < observation>...</observation> tag that explicitly analyzes the returned result before deciding whether to call code again

  58. [58]

    After that, you may continue with additional <tool_call> -> <tool_response> -> <observation> cycles

    The order must be strictly <tool_call>...</tool_call> -> <tool_response >...</tool_response> -> <observation>...</observation>. After that, you may continue with additional <tool_call> -> <tool_response> -> <observation> cycles

  59. [59]

    q0").copy() draw = ImageDraw.Draw(annotated) draw.rectangle((600, 290, 660, 380), outline=

    Your final answer must be grounded in the tool results and observations you obtained. If a tool result is insufficient, unclear, or unhelpful, explicitly say so in <observation> and call code again to get better evidence instead of relying on your own unaided visual guess. Full example: Question: Is the man phone both blue and closed? Assistant: <tool_cal...

  60. [60]

    Work in progress

    Prompt for refining synthesized code-reasoning trajectories.Although Gemini 3.1 Pro [7] can synthesize high-quality code-reasoning trajectories using the prompt in Box 2, the generated trajectories still exhibit several common issues: 24 Preprint. Work in progress. •✗Redundant or repetitive tool calls.The model may repeatedly invoke similar code operation...

  61. [61]

    Tool returns are wrapped in <tool_response>...</tool_response> and images are inserted in place

  62. [62]

    <tool_call>...</tool_call> is executable code

  63. [63]

    <observation>...</observation> is the observation after a tool returns

  64. [64]

    <answer>...</answer> is the final answer

  65. [65]

    Your task:

    Text thoughts may be interleaved anywhere in the trajectory. Your task:

  66. [66]

    If the tools did not provide helpful information and the answer was effectively derived from tool-free reasoning, especially in the last step, discard the trajectory

    Check whether the trajectory actually relied on useful tool information to answer the question. If the tools did not provide helpful information and the answer was effectively derived from tool-free reasoning, especially in the last step, discard the trajectory

  67. [67]

    Remove the unhelpful repeated tool-call steps and keep only the step that produced the key useful information

    Check whether the same tool was called repeatedly with unhelpful retries. Remove the unhelpful repeated tool-call steps and keep only the step that produced the key useful information

  68. [68]

    For each tool call, if the tool result is followed by no meaningful < observation>, add one

  69. [69]

    <tool_call>result = 1result</tool_call>

    Remove clearly meaningless tool calls or meaningless code, such as trivial assignments, like "<tool_call>result = 1result</tool_call>"

  70. [70]

    Remove meaningless thought fragments

  71. [71]

    Note that ‘img.size‘ is necessary for the model to know the image size, so keep the ‘img.size‘ tool call that corresponds to it even if it looks trivial

  72. [72]

    If not, try to refine the <observation> to make it more accurate and specific to the actual tool response

    Check whether each <observation> block precisely describes the preceding tool call result. If not, try to refine the <observation> to make it more accurate and specific to the actual tool response. Output requirements:

  73. [73]

    decision

    If no refinement is needed, output {"decision":"keep"}

  74. [74]

    decision

    If the trajectory should be discarded, output {"decision":"drop","reason ":"..."}

  75. [75]

    decision

    If refinement is needed, output JSON in this schema: { "decision": "refine", "messages": [ {"kind": "assistant", "text": "... assistant message text ..."}, {"kind": "original_message", "source_message_id": "M3"} ] }

  76. [76]

    messages

    In "messages", do not repeat the question or system prompt. Only output the messages after the original question. 25 Preprint. Work in progress

  77. [77]

    kind":"original_message

    Use {"kind":"original_message","source_message_id":"Mx"} only for original non-assistant messages that should be kept exactly, especially tool-response user messages

  78. [78]

    If a tool- response step is kept, keep its <tool_response> in the corresponding original_message instead of rewriting it

    For assistant messages, put the full assistant text in "text". If a tool- response step is kept, keep its <tool_response> in the corresponding original_message instead of rewriting it

  79. [79]

    Keep the final answer unchanged

  80. [80]

    You should decide whether to use code

    If a <tool_call> step is removed, also remove the corresponding < tool_response> and <observation>. C TRAININGDETAILS C.1 DETAILS OFCOLD-STARTSFT For SFT, we train Qwen3-VL-8B-Instruct for 4 epochs using Megatron [26] on the ms-swift frame- work [47]. We optimize the model with Adam using a peak learning rate of1×10−5, a weight decay of 0.1, and a cosine ...

Showing first 80 references.