Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VisTA claims that reinforcement learning alone—without demonstrations or tool descriptions—can teach an agent which vision tools to invoke, improving accuracy on chart and geometry reasoning.

desk verdict A solid RL-for-tool-selection recipe with a real transfer result, but the query-adaptive claim outruns the evidence; worth a revision, not a rejection. read the letter →

arxiv 2505.20289 v2 pith:HV4DE772 submitted 2025-05-26 cs.CV

Zeyi Huang , Yuyang Ji , Anirudh Sundara Rajan , Zefan Cai , Wen Xiao , Haohan Wang , Junjie Hu , Yong Jae Lee This is my paper
classification cs.CV
keywords reinforcementlearningtoolselectionvision-languagemodelsGRPOchartunderstandinggeometryreasoningout-of-distributiongeneralizationvisualaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes VisTA, a reinforcement learning framework that trains a vision-language agent to choose which external tools (chart-to-table converters, SVG extractors, captioners, geometry parsers) to feed to a frozen reasoning model for a given visual question. The central claim is that end-to-end training with Group Relative Policy Optimization, using only the final answer's correctness as reward, lets the agent discover query-specific tool selections without human demonstrations or explicit tool descriptions. On ChartQA, the trained agent improves accuracy from 76.4 to 79.4 over the best training-free baseline, and the gain grows on an out-of-distribution variant (66.8 to 73.2) where chart labels are removed and shapes distorted. The learned policy also transfers to GPT-4o as a frozen reasoner, improving ChartQA from 84.6 to 88.9. If correct, the framework offers a way to make tool-augmented visual reasoning adaptive and experience-driven rather than dependent on prompt engineering or supervised fine-tuning.

What carries the argument

The load-bearing mechanism is the pair (GRPO, differential reward). The agent is a vision-language model outputting a comma-separated list of tool indices; GRPO samples a group of tool sets per query and computes a group-relative advantage. The reward function is the key innovation: it credits a tool set only for changing an incorrect base prediction into a correct one, penalizes flipping a correct answer to incorrect, and treats the two no-change cases asymmetrically. The frozen reasoner produces both a no-tool baseline prediction and a tool-augmented prediction at train time, so the reward isolates the causal contribution of the selected tools to the final answer.

What would settle it

Compute the accuracy of a fixed policy that always selects Tool 1 and Tool 2, the two chart-to-table tools with the highest standalone accuracy, on ChartQA and ChartQA-OoD; if that fixed policy matches or beats VisTA's 79.4 and 73.2, then the learned policy has not demonstrated query-conditioned selection beyond a static preference. Additionally, an oracle comparison that asks whether the agent's chosen tool set ever outperforms the single best tool on the same query would directly test the adaptation claim.

Watch

Extended reading notes

Core claim

VisTA's central discovery is that a tool-selection policy for visual reasoning can be trained entirely from task-outcome feedback: the agent samples a set of tools, the tools process the image, the frozen reasoner answers, and a reward that compares the answer with and without tools (+1 for help, -0.5 for hurt, 0 for no change, +1 for neutral) drives a GRPO update. The learned policy strongly prefers the two table-extraction tools that have the highest standalone accuracy, avoids poorly performing tools, and still outperforms every single-tool baseline, indicating that selection adapts per query. The same policy, trained once with a 7B agent, improves a frozen GPT-4o reasoner across ChartQA, Geometry3K, MathVerse, and BlindTest without retraining. The paper argues this is evidence that RL can replace both training-free prompting and supervised fine-tuning as the mechanism for tool integration in multimodal reasoning.

Load-bearing premise

The reward feedback collected from 800 training questions with only four sampled tool sets per question is enough for the agent to learn a policy that picks tools per query rather than simply always calling the tools with the best average accuracy.

Editorial extensions

If this is right

  • RL-trained tool selection beats the best single tool and the all-tools baseline, showing that selective augmentation matters: 79.4 versus 78.3 for the best single tool and 74.6 for all tools on ChartQA with a QwenVL-7B reasoner.
  • Gains are larger on the out-of-distribution ChartQA variant (73.2 versus 66.8), suggesting the learned policy relies less on textual labels and more on tool-derived visual structure.
  • The policy transfers without retraining to stronger frozen reasoners such as GPT-4o and GPT-o1, meaning tool-selection skill and reasoning skill can be decoupled and recombined.
  • The method outperforms directly RL-training the reasoner itself (ChartQA +1.9 points, ChartQA-OoD +8.9 points), implying tool selection is a cheaper, complementary axis for improving visual reasoning.
  • A policy trained only on Geometry3K transfers zero-shot to the MathVerse plane-geometry subset, improving GPT-o1 from 67.7 to 73.1.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reward function also pays +1 when the base reasoner is already correct and the tools change nothing, so the agent is partly rewarded for not disturbing correct answers; distinguishing helpful from harmless selection would require a reward that conditions on whether the tool output actually contributed.
  • Because the learned policy concentrates on the two highest-accuracy tools, the cleanest test of genuine per-query adaptation is a comparison against an oracle that always emits those same two tools; the paper's current numbers do not settle that comparison.
  • A natural extension is to unroll the same GRPO loop over sequential tool pipelines, where the reward must be credited across tool order; the paper explicitly leaves sequential composition for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. VisTA proposes an end-to-end reinforcement learning framework for visual tool selection. A vision-language agent is trained with GRPO to choose tools from a curated, benchmark-specific library; a frozen reasoner consumes the tool outputs along with the original image and question. The method is evaluated on ChartQA, ChartQA-OoD, Geometry3K, MathVerse, and BlindTest, reporting gains over training-free prompting baselines and an RL-only reasoner baseline, as well as zero-shot transfer of the learned policy to different reasoners. The paper argues that end-to-end RL enables the agent to discover query-specific tool-selection pathways without explicit reasoning supervision.

Significance. If the central claim holds, VisTA is a practical contribution: it shows that a relatively small agent (QwenVL-7B) can be trained with RL to select and combine existing perception tools for a frozen reasoner, yielding consistent improvements across multiple visual reasoning benchmarks and transferring across reasoners without retraining. The paper includes several useful ablations (random/all-tool selection, individual tool performance, pseudo-upper bound, transfer to GPT-4o) and explicitly studies an out-of-distribution chart variant. However, the significance is currently limited by the absence of a baseline that tests the central 'adaptive, query-specific' claim directly: a static top-2 policy may reproduce most of the reported gains, and the reported evidence does not yet separate query-conditional adaptation from a learned global preference over tools.

major comments (3)
  1. [§4.2, §4.3, Fig. 5] The central claim of query-conditional tool selection is not yet supported by the reported experiments. On ChartQA, the margin over the best individual tool is only 79.4 vs. 78.3 (§4.3, Fig. 3), and Fig. 5 shows a strong preference for Tools 1 and 2, which are the two highest standalone-accuracy tools. A static policy that always selects Tools 1 and 2 would therefore likely match or closely approach the reported result, but no static top-1 or top-2 baseline is reported. Please add such baselines and report the per-query agreement between the learned selection and the pseudo-upper-bound oracle (§4.3) to demonstrate that the policy does more than recover a global preference ordering.
  2. [§3.3, §4.1] The reward function assigns +1 to the neutral case (correct with and without tools), so on the majority of easy queries where the reasoner already answers correctly, every candidate tool that does not harm is rewarded equally. With only 800 training queries and G=4 sampled tool sets per query (§4.1), the group-relative advantages are noisy and likely encode average tool utility rather than query-dependent distinctions. The paper should provide evidence that the learned policy is genuinely query-conditional, for example by showing on a query subset with known oracle-optimal tools that the policy's per-query choices correlate with the oracle, or by reporting the reward breakdown (help/hurt/neutral) over training.
  3. [§4.4] The Pearson correlation between usage frequency and standalone tool accuracy (Fig. 4) is equally consistent with a static top-2 policy, since such a policy would also select high-accuracy tools almost always. This analysis therefore does not substantiate the statement in §4.3 that the policy 'learns to go beyond fixed tool usage.' A more direct test of adaptivity would be to compute, for each query, whether the selected tool set matches the best single tool for that query (as derived from the pseudo-upper-bound construction) and to compare this overlap against the static top-2 policy on the same metric.
minor comments (6)
  1. [§4.6] The text states that the policy 'selects approximately 1.2 tools per query on average,' but this statistic is not reported elsewhere; please include the selection-count distribution over the test set.
  2. [Table 1] The 'RL' row in Table 1 refers to a GRPO-trained reasoner without tools, which could be confused with the proposed VisTA RL framework; please rename this row (e.g., 'GRPO reasoner (no tools)') in the caption or table.
  3. [§4.11] The ChartQA-OoD construction is described in detail, but it is not stated whether similar perturbations are applied to Geometry3K or why not; please clarify the scope of the OoD evaluation.
  4. [§4.1 and References] The tool is called 'ChartQCR' in Section 4.1, but the cited work [38] is titled 'ChartOCR'; please verify the tool name and citation.
  5. [Table 2] For the Geometry3K rows, the reasoner is not specified in the table body; please add a note in the caption that 'Ours' uses QwenVL-7B as reasoner for Geometry3K and GPT-4o for ChartQA.
  6. [Fig. 3] The caption should explicitly identify what 'No' denotes (the no-tool baseline) and which configuration corresponds to 'Ours' (agent and reasoner both QwenVL-7B in this figure).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; VisTA's reported gains are measured on held-out test sets and are not derived from the training reward by construction.

full rationale

VisTA's derivation chain is: sample 800 training queries, sample tool sets from the policy, execute tools with a frozen reasoner, compute a GRPO reward from ground-truth labels via the four-case reward in Sec 3.3, update only the agent's selection policy, and then evaluate on full held-out test sets (ChartQA, ChartQA-OoD, Geometry3K, MathVerse subset, BlindTest). The training reward uses ground-truth labels and the reasoner's correctness with and without tools, but the reported accuracy is measured on disjoint test queries, so the headline numbers are not encoded in the training objective by construction. The ChartQA-OoD set is author-constructed (Sec 4.11), which is an external-validity concern rather than a circularity concern: the same perturbed inputs are presented to the trained agent and to all baselines. The pseudo-upper bound in Sec 4.3 is explicitly a loose upper limit, not a fitted target, and the claim that the policy closes part of the gap is a measured, falsifiable result. The only self-citations are background references to LLaVA works ([5], [56]) for VLM capabilities; they are not load-bearing, and no uniqueness theorem, ansatz, or fitted parameter is imported from prior work. The reviewer concern that a static top-2 policy may match VisTA (Sec 4.5 shows a strong preference for Tools 1 and 2) is an evidential limitation about whether query-adaptive selection is demonstrated, not a demonstration that the result reduces to its inputs. A missing static-top-k baseline and small margins affect the strength of the adaptive-selection claim, but they do not make the derivation circular. No circular step meets the required standard of quote-plus-reduction.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim depends on a small set of hand-set reward weights, a small training budget, a hand-curated tool pool, and an author-constructed OoD benchmark. No new physical or mathematical entities are introduced.

free parameters (4)
  • reward_schedule = +1 help, +1 neutral, 0 no change, -0.5 hurt
    Hand-chosen reward values in Sec 3.3 shape the learned policy; giving +1 to the neutral case (correct with and without tools) rewards safe tool use rather than only tools that rescue wrong answers, which weakens the per-query adaptation signal.
  • training_subset_size = 800 examples per benchmark
    Training on 800 randomly sampled examples for 100 iterations with batch size 8 (Sec 4.1) is a small budget that may only support learning a global tool preference rather than per-query conditional selection.
  • group_size_G = 4 answer generations per query
    G=4 tool candidates per query is used for GRPO advantage estimation (Sec 4.1); small groups give noisy relative advantages and limit exploration diversity.
  • ood_stretch_frequency = 0.5 probability of stretching charts to twice width or height
    The ChartQA-OoD test set is constructed by the authors with a 50% random geometric stretch (Sec 4.11); this hand-set perturbation rate defines the difficulty of the OoD benchmark.
assumptions (6)
  • standard math GRPO clipping and KL regularization objective is a valid optimization target for the tool-selection policy.
    Sec 3.3 Eq. (2) adopts the GRPO objective with ratio clipping and KL divergence to a reference policy.
  • domain assumption Each tool can be executed on the input image and its output can be appended to the prompt for the frozen reasoner.
    Sec 3.2 assumes tool outputs o(i) = T(i)(I) are usable by the reasoner in an augmented prompt.
  • domain assumption Ground-truth answers y* are available for the sampled training queries.
    The reward function in Sec 3.3 compares y_img and y_img+tools against y*; this requires labeled training data.
  • ad hoc to paper The manually curated tool pools (9 tools for ChartQA, 4 for Geometry3K, 6 for BlindTest) are sufficient and are the only tools the agent can choose from.
    Sec 4.1 defines benchmark-specific tool libraries; the agent's search space is limited to these preselected tools.
  • ad hoc to paper The ChartQA-OoD perturbation protocol (removing text labels and random stretching) measures robustness rather than introducing a different task.
    Sec 4.11 constructs the OoD set; validity of the OoD comparison depends on this protocol being an appropriate robustness probe.
  • ad hoc to paper The reward value for the neutral case (correct with and without tools) is set equal to the help case.
    Sec 3.3 assigns +1 when y_img = y* and y_img+tools = y*; this design choice is not derived from any principle and affects the learned policy's tool usage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection." pith.science (2026). https://pith.science/paper/HV4DE772

@misc{pith2026250520289,
  author       = {Pith},
  title        = {Pith review of: VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HV4DE772}},
  note         = {Machine review of arXiv:2505.20289}
}
read the original abstract

We introduce VisTA, a new reinforcement learning framework that empowers visual agents to dynamically explore, select, and combine tools from a diverse library based on empirical performance. Existing methods for tool-augmented reasoning either rely on training-free prompting or large-scale fine-tuning; both lack active tool exploration and typically assume limited tool diversity, and fine-tuning methods additionally demand extensive human supervision. In contrast, VisTA leverages end-to-end reinforcement learning to iteratively refine sophisticated, query-specific tool selection strategies, using task outcomes as feedback signals. Through Group Relative Policy Optimization (GRPO), our framework enables an agent to autonomously discover effective tool-selection pathways without requiring explicit reasoning supervision. Experiments on the ChartQA, Geometry3K, and BlindTest benchmarks demonstrate that VisTA achieves substantial performance gains over training-free baselines, especially on out-of-distribution examples. These results highlight VisTA's ability to enhance generalization, adaptively utilize diverse tools, and pave the way for flexible, experience-driven visual reasoning systems.

Figures

Figures reproduced from arXiv: 2505.20289 by the authors.

Figure 1
Figure 1. Overview of VisTA. (Left) Our method trains an agent to autonomously discover effective combinations of visual tools without human supervision. (Right) By decoupling the agent from the reasoner, the learned policy can be seamlessly integrated with a wide range of reasoning models. on empirical performance rather than pre-specified rules. Through iterative interactions with its envi￾ronment, an RL agent can learn ada… view at source ↗
Figure 2
Figure 2. Policy Optimization. Given a user query, the agent selects tools from a pre-defined set of external tools. The tools are applied to the image, and their outputs and the query are fed to a frozen rea￾soner model. Both the Direct Path (query+image) and the Tool-Augmented Path (query+tools+image) are evaluated to compute a reward signal, which is used to update the agent’s tool-selection policy. We define the agent’s o… view at source ↗
Figure 3
Figure 3. Comparison of ChartQA accuracy across individual tools (T0–T8), the no-tool baseline (No), [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Pearson correlation between tool usage frequency and individual tool performance. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Tool selection frequency across our RL-trained agent, QwenVL-7B, and GPT-4o. Our method [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Tool Selection for a Geometry Question. Our agent selects the formal diagram parser, Inter-GPS, which accurately extracts essential details from the diagram and represents them in a formal structure. Leveraging this representation, the reasoner is able to determine the…
Figure 7
Figure 7. Figure 7: Tool Selection for a Chart Question. An example demonstrating a case where the agent calls a set of complementary tools. This question requires understanding both numeric values (extracted from the table) and color information (extracted from the SVG). The reasoner com…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Training on multi-tool visual reasoning trajectories (zoom, grounding, lines) with an attention-focused RL objective improves UHR remote-sensing VQA accuracy over single-tool zoom-in and larger base models.

  2. OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

    cs.CV 2026-02 conditional novelty 6.0 of 10

    A 4B model fine-tuned on tool-augmented geospatial reasoning traces outperforms larger general-purpose models on executable GIS/spectral tool-use benchmarks and matches frontier models on trajectory fidelity.

Reference graph

Works this paper leans on

59 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

  2. [2]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 12

    work page 1901

  3. [3]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  6. [6]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayi- heng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  7. [7]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    arXiv 2024

  8. [8]

    Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022

    arXiv 2022

Show all 59 references
  1. [9]

    Tora: A tool-integrated reasoning agent for mathematical problem solving

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452, 2023

    2023 arXiv

  2. [10]

    Pal: Program-aided language models

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR, 2023

    2023

  3. [11]

    Gorilla: Large language model connected with massive apis

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37:126544– 126565, 2024

    2024

  4. [12]

    Visual sketchpad: Sketching as a visual chain of thought for multimodal language models

    Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.09403, 2024

    2024 arXiv

  5. [13]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023

    2023

  6. [14]

    Vipergpt: Visual inference via python execution for reasoning

    D ´ ıdac Sur ´ ıs, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023

    2023

  7. [15]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess ` ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551, 2023

    2023

  8. [16]

    Llava-plus: Learning to use tools for creating multimodal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. In European Conference on Computer Vision, pages 126–142. Springer, 2024

    2024

  9. [17]

    Octotools: An agentic framework with extensible tools for complex reasoning

    Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octotools: An agentic framework with extensible tools for complex reasoning. arXiv preprint arXiv:2502.11271, 2025

    2025 arXiv

  10. [18]

    Reinforcement learning: An introduction, volume 1

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  11. [19]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  12. [20]

    Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. 13 arXiv preprint arXiv:2105.04165, 2021

    2021 arXiv

  13. [21]

    Internet-augmented dialogue generation

    Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566, 2021

    2021 arXiv

  14. [22]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  15. [23]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-llms/ , 2024. Accessed: 2025-05-13

    2024

  16. [24]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  17. [25]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  18. [26]

    Visual-rft: Visual reinforcement fine-tuning

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  19. [27]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  20. [28]

    Retool: Reinforcement learning for strategic tool use in llms

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536, 2025

    2025 arXiv

  21. [29]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148–166. Springer, 2024

    2024

  22. [30]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023

    arXiv

  23. [31]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  24. [32]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  25. [33]

    Unichart: A universal vision-language pretrained model for chart comprehension and reasoning

    Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761, 2023

    2023 arXiv

  26. [34]

    Deplot: One-shot visual language reasoning by plot-to-table translation

    Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. Deplot: One-shot visual language reasoning by plot-to-table translation. arXiv preprint arXiv:2212.10505, 2022

    2022 arXiv

  27. [35]

    Chartmoe: Mixture of expert connector for advanced chart understanding

    Zhengzhuo Xu, Bowen Qu, Yiyan Qi, Sinan Du, Chengjin Xu, Chun Yuan, and Jian Guo. Chartmoe: Mixture of expert connector for advanced chart understanding. arXiv preprint arXiv:2409.03277, 2024

    2024 arXiv

  28. [36]

    The opencv library

    Gary Bradski. The opencv library. Dr. Dobb’s Journal: Software Tools for the Professional Pro- grammer, 25(11):120–123, 2000

    2000

  29. [37]

    Context-aware chart element detection

    Pengyu Yan, Saleem Ahmed, and David Doermann. Context-aware chart element detection. In International conference on document analysis and recognition, pages 218–233. Springer, 2023

    2023

  30. [38]

    Chartocr: Data extraction from charts images via a deep hybrid framework

    Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin. Chartocr: Data extraction from charts images via a deep hybrid framework. In Proceedings of the IEEE/CVF winter conference on appli- cations of computer vision, pages 1917–1925, 2021. 14

    1917

  31. [39]

    Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning

    Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv:2401.02384, 2024

    2024 arXiv

  32. [40]

    Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning

    Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Peng Ye, Min Dou, Botian Shi, et al. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185, 2024

    2024 arXiv

  33. [41]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  34. [42]

    Diagram formalization enhanced multi-modal geometry problem solver

    Zeren Zhang, Jo-Ku Cheng, Jingyang Deng, Lu Tian, Jinwen Ma, Ziran Qin, Xiaokai Zhang, Na Zhu, and Tuo Leng. Diagram formalization enhanced multi-modal geometry problem solver. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP...

    2025

  35. [43]

    G-llava: Solving geometric problem with multi-modal large language model

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370, 2023

    2023 arXiv

  36. [44]

    Multimath: Bridging visual and mathematical reasoning for large language models

    Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multimath: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.00147, 2024

    2024 arXiv

  37. [45]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  38. [46]

    The claude 3 model family: Opus, sonnet, haiku, 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024

    2024

  39. [47]

    PaliGemma: A versatile 3B VLM for transfer

    Lucas Beyer, Andreas Steiner, Andr´ e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil...

    2024 arXiv

  40. [48]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  41. [49]

    Internvl2: Better than the best—expanding performance boundaries of open- source multimodal models with the progressive scaling strategy, July 2024

    OpenGVLab Team. Internvl2: Better than the best—expanding performance boundaries of open- source multimodal models with the progressive scaling strategy, July 2024

    2024

  42. [50]

    The llama 3 herd of models

    Meta AI. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  43. [51]

    Cambrian-1: A fully open, vision- centric exploration of multimodal LLMs

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision- centric exploration of multimodal LLMs. In NeurIPS, 2024

    2024

  44. [52]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am´ elie H´ eliou, Paul Jacob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024

    2024 arXiv

  45. [53]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  46. [54]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294, 2024

    2024 arXiv

  47. [55]

    Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025

    Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284, 2025

    2025 arXiv

  48. [56]

    Improved baselines with visual instruction 15 tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction 15 tuning. In CVPR, 2024

    2024

  49. [57]

    xGen-MM (BLIP-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xGen-MM (BLIP-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024

    2024

  50. [58]

    LLaV A-OneVision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  51. [59]

    Vision language models are blind

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza, Taesiri, and Anh Totti Nguyen. Vision language models are blind. In ACCV, 2024. 16

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.