Pith. sign in

REVIEW 4 major objections 6 minor 89 references

MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MMAT-1M claims to be the first million-scale multimodal agent-tuning dataset, and fine-tuning on it lifts open-source models' reasoning and tool use.

desk verdict A genuinely useful million-scale multimodal agent-tuning dataset with a plausible but unproven headline: train/eval overlap and a missing raw-QA baseline leave the central claim not fully established. read the letter →

arxiv 2507.21924 v1 pith:W2CG4GPB submitted 2025-07-29 cs.CV

Tianhong Gao , Yannian Fu , Weiqun Wu , Haixiao Yue , Shanshan Liu , Gang Zhang This is my paper
classification cs.CV
keywords multimodalagenttuningchain-of-thoughtreasoningtooluseretrieval-augmentedgenerationlargelanguagemodelsreflectionmechanismdatasetconstructionvisualquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MMAT-1M is a proposed answer to the lack of large-scale, high-quality agent-tuning data for multimodal models. The paper claims that by converting one million public visual QA pairs into structured multi-turn trajectories—with chain-of-thought rationales, a reflection step, and dynamic calls to OCR, captioning, detection, face recognition, and retrieval—it produces a dataset that can teach open-source multimodal models to reason and use tools. Fine-tuning on this dataset is reported to lift InternVL2.5-8B by 2.7 points on average across eight benchmarks and by 8.8 points on Dyn-VQA, with gains in all three model families tested. If the claim holds, MMAT-1M would give researchers a public training resource that brings small open models closer to proprietary agent behavior.

What carries the argument

The load-bearing machinery is the four-stage data engine that turns static QA pairs into executable trajectories. In the foundation stage it normalizes prompts and prepares five tools: image captioning built on scene graphs, open-vocabulary object detection, OCR, face detection, and search-based RAG. In the rationale stage GPT-4o iteratively emits a thought, chooses operator calls, and issues RAG queries until it reaches the answer. In the reflection stage a second pass repairs step-skipping and 'reasoning cheating' by injecting corrective thoughts. In the integration stage multi-turn dialogues are compressed into one-turn ORR samples with tool outputs prepended to the input. The RR and ORR formats are then used for LoRA fine-tuning, so the reasoning traces themselves are the training signal.

What would settle it

Compare every test question and image in MMStar, MMMU, MathVista, MathVision, AI2D, OCRBench, RealWorldQA, HallusionBench, and Dyn-VQA against the 1,090,263 MMAT-1M training pairs using exact-match and near-duplicate detection; if any benchmark items are present in training (the training source lists include AI2D, TabMWP, ChartQA, DocVQA, ST-VQA, and IconQA), retrain after removing those items and check whether the reported gains disappear.

Watch

Extended reading notes

Core claim

The central claim is that MMAT-1M, assembled by rewriting 1,090,263 question-answer pairs from five public multimodal sources into structured agent trajectories, is a workable training resource for multimodal agent tuning. The paper reports that a model fine-tuned on this data, InternVL2.5-8B-RR, improves from 60.7 to 63.4 average accuracy on an eight-benchmark suite and from 35.2 to 44.0 on Dyn-VQA Golden Query, and that similar gains appear across Llama-3.2-11B-Vision-Instruct, MiniCPM-V-2.6, and all InternVL2.5 sizes tested. The authors present this as evidence that combining chain-of-thought rationales, a reflection step that repairs faulty reasoning, and dynamic calls to OCR, captioning, detection, face recognition, and web search can teach open-source models to behave like tool-using agents.

Load-bearing premise

The paper's reported gains assume that the eight evaluation benchmarks share no test items with the five source datasets used to build MMAT-1M; if benchmark questions appear in training, the 2.7-point and 8.8-point improvements would reflect memorization rather than learned reasoning.

Editorial extensions

If this is right

  • Fine-tuning open vision-language models on MMAT-1M's RR trajectories is reported to improve accuracy on general, mathematical, diagram, OCR, spatial, and hallucination benchmarks, so the dataset is usable as a drop-in SFT corpus.
  • The reflection mechanism contributes directly: removing reflection lowers the average from 61.3 to 60.2, and removing API calls or RAG lowers it to 57.3 or 59.8, so the tool-use and self-correction components are what make the trajectory data useful.
  • The ORR format retains most of the benefit while adding less inference latency than RR, offering a practical speed-accuracy tradeoff for deployed agents.
  • Fine-tuned models can invoke a tool they were never trained on, such as a celebrity-recognition operator, when prompted, indicating that the training teaches a general tool-calling policy.
  • Gains on Dyn-VQA, such as InternVL2.5-8B rising from 27.0 to 36.8 on Query, show that the dataset transfers to multi-hop retrieval tasks that require planning a web search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an extension, the ORR compression recipe could be applied to any existing multi-step agent-trajectory corpus, turning slow multi-turn traces into fast one-pass training data for latency-constrained deployments.
  • The reflection stage is a generic repair mechanism: the same two-pass design, detecting step-skipping and detecting forced alignment to the answer, could clean other automatically generated reasoning datasets, not only visual ones.
  • The reported zero-shot invocation of an unseen celebrity-recognition operator suggests the model learns a general when-to-call-a-tool policy; a systematic test would hold out one operator during training and probe whether the fine-tuned model spontaneously calls it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces MMAT-1M, a dataset of about 1.09 million multimodal question-answer pairs synthesized from five public sources (Visual CoT, LLaVA-CoT, The Cauldron, TabMWP, InfoSeek). A four-stage data engine generates rationales with GPT-4o, interleaves API calls (image caption, OVD, OCR, face detection) and RAG searches, applies reflection to repair inconsistencies, and optionally compresses trajectories into a one-turn ORR format. The authors fine-tune Llama-3.2-11B-Vision-Instruct, MiniCPM-V-2.6, and InternVL2.5-2B/4B/8B with LoRA on the ORR and RR formats and report average gains across eight benchmarks (e.g., InternVL2.5-8B-RR improves by 2.7%) plus an 8.8% gain on Dyn-VQA. They also provide ablations of API, RAG, reflection, and SFT, and qualitative examples of tool invocation.

Significance. If the reported gains are genuine, MMAT-1M would be a useful public resource for multimodal agent tuning: it is the first dataset at the million scale in this niche, spans diverse tasks, combines CoT with reflection and dynamic tools, and shows consistent improvements across three model families. The paper's strengths include the public release, the detailed four-stage pipeline, the large-scale LLM-based quality audit, and the explicit trade-off between ORR and RR formats. The main caveats are that the training/evaluation overlap risk is left entirely unaddressed and that no raw-QA SFT control isolates the contribution of the synthesized rationales; these issues must be resolved before the effectiveness claim can be considered established.

major comments (4)
  1. [§4.2 and Supplementary Table 6] The central empirical claim requires that the evaluation benchmarks be disjoint from the training sources, but the paper provides no overlap analysis. Supplementary Table 6 lists AI2D, ChartQA, TabMWP, ST-VQA, DocVQA, IconQA, and InfographicsVQA among the training sources, while Section 4.2 evaluates on AI2D, MathVista, OCRBench, and MMStar, which are constructed in part from those same datasets. For example, the AI2D benchmark directly uses the AI2D dataset listed in Table 6 under LLaVA-CoT, and OCRBench contains material from ST-VQA and DocVQA. Because the training data contain the same images and questions in transformed form, the reported gains—for instance, InternVL2.5-8B-RR's 2.7% average improvement and 8.8% improvement on Dyn-VQA—could partly reflect memorization of training samples rather than transferable agent ability. Please report exact overlap statistics per benchmark, remove overlapping samples before evaluation, and re-run Table 3 and Table 4 on the filtered benchmarks; alternatively, evaluate on held-out subsets that are provably disjoint from the five training sources.
  2. [§4.2, Table 3, and Table 5] All training comparisons are against untuned base models; there is no SFT control on the original QA pairs without rationales, reflection, API calls, or RAG. Table 5 ablates API, RAG, reflection, and SFT, but always within the RR-style scaffold, so it cannot separate the contribution of the proposed four-stage synthesis from the contribution of adding large amounts of in-domain QA data to the base model. This is load-bearing because the paper's contribution is specifically the agent-style dataset transformation, not merely additional fine-tuning data. Please add SFT on the original QA pairs (with and without plain CoT) as a control and report the same benchmarks and the same model family.
  3. [§3.2 and Supplementary B] The claim that 'over 89% [of samples] demonstrate high-quality reasoning' rests entirely on scores produced by Doubao-1.5-Vision-Pro-32K. No human validation, inter-annotator agreement, or calibration against a human-labeled subset is reported. Since dataset quality is one of the main contributions, the automated judge should be validated before this number is used to support the construction pipeline; at minimum, report a sample size, the distribution of scores, and a human agreement study on a random subset.
  4. [Table 4 and §4.2] The Dyn-VQA evaluation, which supports a headline result, lacks sufficient protocol detail. The meanings of 'Query' and 'Golden Query' are not defined, and the retrieval setup is not specified beyond top-k=3 in Section 4.1: the Google Search API configuration, the query-generation prompt, the number of retrieval steps, and the Dyn-VQA version are all missing. This makes the reported 8.8% improvement difficult to reproduce. Please provide the exact evaluation prompts, retrieval configuration, and dataset version.
minor comments (6)
  1. [§3.3, Eq. (1)] The Frobenius-norm regularizer on LoRA weight updates is not discussed or ablated; either justify this term or remove it, and clarify how it is implemented in the LoRA update.
  2. [Table 5] The row label 'Baseline-RR (w/o SFT)' is ambiguous because the table has no SFT column; it is unclear whether this row is a prompting-only baseline or a trained model. Please add an explicit SFT column or rename the row.
  3. [Supplementary B, Figure 7] The sentence 'The content of "reason" is is why you think...' contains a duplicated 'is'.
  4. [Supplementary Table 6] The entry 'A-OKVQA [43]' cites the ChartQA paper; the reference appears to be incorrect and should be fixed.
  5. [§4.2] The paper states that MathVista and MathVision use the testmini set but does not report the corresponding sample sizes or the exact evaluation mode (e.g., with or without CoT); please provide these details for reproducibility.
  6. [Figure 3] The caption for Figure 3 is missing; the text '2B 4B 8B 2B 4B 8B...' appears to be placeholder content and should be replaced with a proper description of the efficiency-performance trade-off.

Circularity Check

0 steps flagged · score 0.0 of 10

No formal circularity: the central results are empirical fine-tuning measurements, not derivations from fitted parameters, self-cited theorems, or definitional equivalences.

full rationale

The paper's central claim is that fine-tuning open-source multimodal models on MMAT-1M yields large benchmark gains (e.g., InternVL2.5-8B-RR improves by 2.7% on average across eight benchmarks). This is an empirical measurement, not a derived prediction. The dataset is constructed from public QA sources with GPT-4o-generated rationales, API calls, and RAG trajectories, and the evaluation is conducted with the external VLMEvalKit suite plus Dyn-VQA. No fitted parameter is renamed as a prediction, no author-self-citation is load-bearing, and no uniqueness theorem is imported from prior work by the same authors. The closest concern is potential train/eval overlap: Supplementary Table 6 lists AI2D, DocVQA, TabMWP, IconQA, ST-VQA, and ChartQA inside MMAT-1M's training sources, while the evaluation suite in Section 4.2 includes AI2D and OCRBench, and MathVista/MathVision draw on overlapping chart and table material. That overlap could inflate measured gains through memorization, but it is a soundness/contamination issue rather than a logical circularity: the reported numbers are not equivalent to the dataset's construction by definition, and the paper does not define its contribution in terms of the benchmark scores. The dataset-quality check by Doubao-1.5-Vision-Pro-32K is also a model-based proxy rather than human validation, but again it is an evaluation-design limitation, not a circular derivation. Under the specified circularity patterns, the derivation chain is self-contained and the circularity burden is minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on synthetic data quality, source label correctness, benchmark disjointness, and an automated quality judge. No free parameters are fitted in a mathematical sense; training hyperparameters are standard. There are no invented physical or conceptual entities.

assumptions (4)
  • domain assumption GPT-4o-generated rationales and reflection corrections are accurate enough to serve as supervised training targets.
    The data engine relies entirely on synthetic reasoning traces from GPT-4o; Section 3.2 states rationales are 'accurate, interpretable, and well-documented' but the only validation is an automated judge (Doubao), and Section E admits reliance on 'high-quality rationale training data' as a limitation.
  • domain assumption Source datasets' question-answer pairs and answers are correct and suitable for training.
    Section 3.1 compiles Visual CoT, LLaVA-CoT, The Cauldron, TabMWP, and Infoseek without re-validating their labels; errors in these sources propagate into MMAT-1M.
  • domain assumption Evaluation benchmarks are disjoint from training sources.
    Section 4.2 evaluates on MMStar, MMMU, MathVista, MathVision, AI2D, OCRBench, RealWorldQA, and HallusionBench while training on datasets that include AI2D, ChartQA, TabMWP, ST-VQA, and DocVQA (Supplementary Table 6); no deduplication is reported, so this assumption is likely violated.
  • domain assumption The Doubao judge's quality scores correlate with human judgment of reasoning quality.
    Section 3.2 and Supplementary B use Doubao-1.5-Vision-Pro to claim over 89% of samples are high-quality; no human agreement study is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning." pith.science (2026). https://pith.science/paper/W2CG4GPB

@misc{pith2026250721924,
  author       = {Pith},
  title        = {Pith review of: MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W2CG4GPB}},
  note         = {Machine review of arXiv:2507.21924}
}
read the original abstract

Large Language Models (LLMs), enhanced through agent tuning, have demonstrated remarkable capabilities in Chain-of-Thought (CoT) and tool utilization, significantly surpassing the performance of standalone models. However, the multimodal domain still lacks a large-scale, high-quality agent tuning dataset to unlock the full potential of multimodal large language models. To bridge this gap, we introduce MMAT-1M, the first million-scale multimodal agent tuning dataset designed to support CoT, reflection, and dynamic tool usage. Our dataset is constructed through a novel four-stage data engine: 1) We first curate publicly available multimodal datasets containing question-answer pairs; 2) Then, leveraging GPT-4o, we generate rationales for the original question-answer pairs and dynamically integrate API calls and Retrieval Augmented Generation (RAG) information through a multi-turn paradigm; 3) Furthermore, we refine the rationales through reflection to ensure logical consistency and accuracy, creating a multi-turn dialogue dataset with both Rationale and Reflection (RR); 4) Finally, to enhance efficiency, we optionally compress multi-turn dialogues into a One-turn Rationale and Reflection (ORR) format. By fine-tuning open-source multimodal models on the MMAT-1M, we observe significant performance gains. For instance, the InternVL2.5-8B-RR model achieves an average improvement of 2.7% across eight public benchmarks and 8.8% on the RAG benchmark Dyn-VQA, demonstrating the dataset's effectiveness in enhancing multimodal reasoning and tool-based capabilities. The dataset is publicly available at https://github.com/VIS-MPU-Agent/MMAT-1M.

Figures

Figures reproduced from arXiv: 2507.21924 by the authors.

Figure 1
Figure 1. Performance comparison of multimodal large language [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The data engine pipeline follows four stages: foundation, rationale generation, reflection, and trajectory integration. It generates [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of inference efficiency and performance [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The zero-shot capability of invoking a celebrity recognition operator of InternVL2.5-8B-RR. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Scene graph and caption generation prompts. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 7
Figure 7. Figure 7: General reflection prompt. The math reflection prompt specifically targets com￾pleteness issues in mathematical reasoning. It instructs GPT-4o to carefully inspect its mathematical derivations, identifying instances where crucial calculation steps might be omitted. By …
Figure 6
Figure 6. Figure 6: Rationale generation prompt. For reflection, we designed two targeted prompts to en￾hance reasoning robustness. The general reflection prompt is intended to detect and correct reasoning cheating behav￾iors. Specifically, it prompts GPT-4o to critically exam￾ine cases w…
Figure 9
Figure 9. Figure 9: One-turn rationale and reflection (ORR) prompt. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Rationale and reflection (RR) prompt. during the rationale and reflection generation stages, we performed a large-scale quality evaluation of the entire MMAT-1M dataset. Specifically, we employed the Doubao￾1.5-Vision-Pro-32K model to evaluate all samples against five…
Figure 11
Figure 11. Figure 11: Reasoning evaluation prompt. ration and use the open-source framework ms-swift [77] for training. The specific parameters are shown in [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Example of iterative rationale result. The OCR recognition is erroneous, but the rationale process corrects the mistake. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Example of reflection result. The inclusion of the reflection makes the mathematical derivation more complete and accurate. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Example of one-turn rationale result. It correctly answers the question with the aid of image caption information. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 52 canonical work pages

  1. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arxiv

  2. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    arXiv 2025

  3. [3]

    Jawahar, Ernest Valveny, and Dimos- thenis Karatzas

    Ali Furkan Biten, Rub `en Tito, Andr ´es Mafla, Lluis Gomez, Marc ¸al Rusi˜nol, C.V . Jawahar, Ernest Valveny, and Dimos- thenis Karatzas. Scene text visual question answering. In 2019 IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 4290–4300, 2019. 13

    2019

  4. [4]

    An augmented benchmark dataset for geometric question answering through dual parallel text en- coding

    Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text en- coding. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1511–1520, Gyeongju, Republic of Korea, 2022. International Committee on Com- putational Linguistics. 13

    2022

  5. [5]

    Fireact: Toward lan- guage agent fine-tuning

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward lan- guage agent fine-tuning. arXiv preprint arXiv:2310.05915,

    arXiv

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision (ECCV), 2024. 13

    2024

  7. [7]

    Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024. 6

    2024

  8. [8]

    M 3 cot: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3 cot: A novel benchmark for multi- domain multi-step multi-modal chain-of-thought. arXiv preprint arXiv:2405.16473, 2024. 3

    arXiv 2024

Show all 89 references
  1. [9]

    Can pre-trained vision and language models answer vi- sual information-seeking questions? arXiv preprint arXiv:2302.11713, 2023

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, So- ravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre-trained vision and language models answer vi- sual information-seeking questions? arXiv preprint arXiv:2302.11713, 2023. 3, 13

    2023 arXiv

  2. [10]

    Agent- flan: Designing data and methods of effective agent tuning for large language models.arXiv preprint arXiv:2403.12881,

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent- flan: Designing data and methods of effective agent tuning for large language models.arXiv preprint arXiv:2403.12881,

    arXiv

  3. [11]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 1, 6, 15

    2024 arXiv

  4. [12]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024

    2024

  5. [13]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF conference on computer...

    2024

  6. [14]

    Clevr-math: A dataset for compositional language, visual and mathematical reasoning

    Adam Dahlgren Lindstr ¨om and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. In International Joint Confer- ence on Learning and Reasoning, 16th International Work- shop on Neural-Symbolic Learning and Reasoning (NeSy ...

    2022

  7. [15]

    Pp-ocr: A practical ultra lightweight ocr system

    Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941, 2020. 4

    2009 arXiv

  8. [16]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia...

    2024

  9. [17]

    Assistgpt: A gen- eral multi-modal assistant that can plan, execute, inspect, and learn

    Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A gen- eral multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.08640, 2023. 1, 2

    2023 arXiv

  10. [18]

    Multi-modal agent tuning: Building a vlm-driven agent for efficient tool usage

    Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, and Qing Li. Multi-modal agent tuning: Building a vlm-driven agent for efficient tool usage. InThe Thirteenth International Conference on Learning Representations, 2025. 2, 3, 4

    2025

  11. [19]

    Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnos- tic suite for entangled language hallucination and visual il- lusion in large vision-language models. In P...

    2024

  12. [20]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 5

    2022

  13. [21]

    Icdar2019 compe- tition on scanned receipt ocr and information extraction

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthe- nis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 compe- tition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019. 13

    2019

  14. [22]

    Gqa: A new dataset for real-world visual reasoning and compositional 9 question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional 9 question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 13

    2019

  15. [23]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 6

    2024 arXiv

  16. [24]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elemen- tary visual reasoning. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recog...

  17. [25]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  18. [26]

    Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CV...

    2017

  19. [27]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In Advances in Neural Infor- mation Processing Systems, pages 2611–2624. Curran Asso- cia...

    2020

  20. [28]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection...

    1956

  21. [29]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 3, 13

    2024 arXiv

  22. [30]

    Kankanhalli

    Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. People in social context (pisc) dataset, 2017. Data set. 13

    2017

  23. [31]

    Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self- adaptive planning agent

    Yangning Li, Yinghui Li, Xinyu Wang, Yong Jiang, Zhen Zhang, Xinran Zheng, Hui Wang, Hai-Tao Zheng, Pengjun Xie, Philip S Yu, et al. Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self- adaptive planning agent. arXiv preprint arXiv:2411.02937,

    arXiv

  24. [32]

    Visual spatial reasoning

    Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023. 13

    2023

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1

    2023

  26. [34]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1

    2024

  27. [35]

    Llava-plus: Learning to use tools for creating multi- modal agents

    Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multi- modal agents. In European Conference on Computer Vision, pages 126–142. Springer, 2024. 2, 3, 4

    2024

  28. [36]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision , pages 38–55. Springe...

    2024

  29. [37]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv e-prints, pages arXiv–2305, 2023. 6

    2023

  30. [38]

    Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning. In The Joint Conference of the 59th An- nual Meeting of the Association for Computationa...

    2021

  31. [39]

    Iconqa: A new benchmark for abstract diagram understand- ing and visual language reasoning

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understand- ing and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Tra...

    2021

  32. [40]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems ...

    2022

  33. [41]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610, 2022. 3, 13

    2022 arXiv

  34. [42]

    Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learni...

    2024

  35. [43]

    ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Ena- mul Hoque. ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning. In Find- ings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. ...

    2022

  36. [44]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 13 10

    2021

  37. [45]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 13

    2022

  38. [46]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models

    AI Meta. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Meta AI Blog. Retrieved December, 20:2024, 2024. 6, 15

    2024

  39. [47]

    Compositional chain-of-thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14420–14431, 2024. 4, 13

    2024

  40. [48]

    Compositional semantic parsing on semi-structured tables

    Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. InProceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Nat- ural Language Processing (Volume 1: Long P...

    2015

  41. [49]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Pro- ceedings of the IEEE international conference on computer ...

    2015

  42. [50]

    A benchmark of facial recognition pipelines and co-usability performances of mod- ules

    Sefik Serengil and Alper ¨Ozpınar. A benchmark of facial recognition pipelines and co-usability performances of mod- ules. Bilis ¸im Teknolojileri Dergisi, 17(2):95–107, 2024. 4

    2024

  43. [51]

    Putting gpt-4o to the sword: A comprehensive evaluation of language, vision, speech, and multimodal proficiency

    Sakib Shahriar, Brady D Lund, Nishith Reddy Man- nuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, and Laiba Batool. Putting gpt-4o to the sword: A comprehensive evaluation of language, vision, speech, and multimodal proficiency. Ap- plied ...

    2024

  44. [52]

    Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning. In The Thirty-eight Conference on Neural Informat...

    2024

  45. [53]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38154–38180,

  46. [54]

    Textcaps: a dataset for image caption- ing with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image caption- ing with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 742–758. Springer,

    2020

  47. [55]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 13

    2019

  48. [56]

    Trial and error: Exploration-based trajectory optimization for llm agents

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502, 2024. 2

    2024 arXiv

  49. [57]

    Tang, Angie Boggust, and Arvind Satyanarayan

    Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. VisText: A Benchmark for Semantically Rich Chart Cap- tioning. In The Annual Meeting of the Association for Com- putational Linguistics (ACL), 2023. 13

    2023

  50. [58]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    2023 arXiv

  51. [59]

    Document understanding dataset and evaluation (dude)

    Jordy Van Landeghem, Rub `en Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Ju- rkiewicz, Micka¨el Coustaty, Bertrand Anckaert, Ernest Val- veny, et al. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF Interna...

    2023

  52. [60]

    The caltech-ucsd birds-200-2011 dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Per- ona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 13

    2011

  53. [61]

    Screen2words: Automatic mobile ui summarization with multimodal learning

    Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In The 34th An- nual ACM Symposium on User Interface Software and Tech- nology, page 498–510, New York, NY , USA, 2021. Associa- ...

    2021

  54. [62]

    Llms in the imaginarium: tool learning through simulated trial and error

    Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. Llms in the imaginarium: tool learning through simulated trial and error. arXiv preprint arXiv:2403.04746,

    arXiv

  55. [63]

    Mllm-tool: A multimodal large language model for tool agent learning

    Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, XM Xuan, Zhengxin Li, Lin Ma, and Shenghua Gao. Mllm-tool: A multimodal large language model for tool agent learning. arXiv preprint arXiv:2401.10727, 4, 2024. 2

    2024 arXiv

  56. [64]

    Mea- suring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Mea- suring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Sys- tems, 37:95095–95169, 2025. 6

    2025

  57. [65]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  58. [66]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 1, 2

    2023 arXiv

  59. [67]

    Grok-1.5 vision preview

    X.AI. Grok-1.5 vision preview. https://x.ai/blog/ grok-1.5v, 2024. 6

    2024

  60. [68]

    Llava-o1: Let vision language models reason step- by-step

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step- by-step. arXiv preprint arXiv:2411.10440, 2024. 3 11

    2024 arXiv

  61. [69]

    Llava-cot: Let vision language models reason step- by-step, 2024

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step- by-step, 2024. 3, 4, 13

    2024

  62. [70]

    Gpt4tools: Teaching large language model to use tools via self-instruction

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36:71995–72007, 2023. 1, 2

    2023

  63. [71]

    React: Synergizing rea- soning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing rea- soning and acting in language models. In International Con- ference on Learning Representations (ICLR), 2023. 1, 2

    2023

  64. [72]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 6, 15

    2024 arXiv

  65. [73]

    Agent lumos: Unified and modular training for open-source language agents

    Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024

  66. [74]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  67. [75]

    Agenttuning: En- abling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: En- abling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023. 1, 2

    2023 arXiv

  68. [76]

    Raven: A dataset for relational and analogical visual reasoning

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 13

    2019

  69. [77]

    Swift:a scal- able lightweight infrastructure for fine-tuning, 2024

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yun- lin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift:a scal- able lightweight infrastructure for fine-tuning, 2024. 16

    2024

  70. [78]

    Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017. 13

    2017

  71. [79]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, pages 4995–5004, 2016. 13 12 MMAT-1M: A Large Reasoning Dataset for Multimodal Agen...

    2016

  72. [83]

    Objects that are relevant to answering the question

  73. [84]

    Object attributes that are relevant to answering the question

  74. [85]

    objects": [

    Object relationships that are relevant to answering the question Example of a valid JSON response: { "objects": ["object_name1", "object_name2", ...], "attributes": [ {"name": "object_name", "value": "attribute_value"}, ... ], "relationships": [ {"object_name1": {"relation_typ...

  75. [86]

    thought: A concise reasoning step with reflection on previous results

  76. [87]

    image_caption

    operators: An array of operators to use next (e.g., "image_caption", "label_bbox", "ocr", "face_detection"). Leave it empty if no operators are needed

  77. [88]

    needed": true,

    rag: An dictof RAG queries that can help resolve the issue. (e.g. {"needed": true, "query": "A query can help reasoning and solve the promblem."})

  78. [89]

    continue

    next_action: Either "continue" if further steps are needed, or "final_answer" if you have the complete solution. Example of a valid JSON response: { "thought": "Identifying key information and reflecting on prior results", "operators": ["image_caption"], "rag": {"needed": fals...

  79. [170]

    Technical University of Aachen, 2022. 13

    2022

  80. [251]

    Springer, 2016. 6, 13

    2016

  81. [2023]

    arXiv preprint arXiv:2308.12966, 1(8), 2023. 1

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.