{"id":"c5799500-355a-4d2f-b3da-a48a27cf7e1e","arxiv_id":"2507.21924","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A new one-million-sample multimodal agent tuning dataset with GPT-4o-generated rationales, reflection, and tool/RAG calls is shown to improve fine-tuned models, though training/eval benchmark overlap is not addressed.","lead":"The authors built MMAT-1M, a million-scale dataset that adds step-by-step reasoning, self-reflection, and tool calls to existing visual question-answer pairs. They report that fine-tuning open multimodal models on it improves average scores on eight benchmarks and on a retrieval-augmented question answering benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposed SFT baseline on a held-out benchmark subset (e.g., MathVista testmini) to verify whether reported gains reflect dataset effectiveness versus contamination or spurious improvements.","rationale":"The reader's weakest assumption (train/eval overlap) is indeed the most load-bearing and is supported by the paper's own Supp. Table 6. I agree the dataset likely has real value because the Dyn-VQA (disjoint, dynamic retrieval) gains are large and consistent across models, and the qualitative zero-shot tool-invocation result is suggestive. But the headline 2.7% average gain across eight benchmarks is the central evidence for 'significant performance gains,' and its magnitude cannot currently be separated from possible memorization of source datasets that appear both in training and in benchmarks such as MathVista, MathVision, AI2D, and OCRBench. The paper also lacks a raw-QA control, so the causal attribution to rationale/reflection/API/RAG is not proven; this is a correct-and-fixable experimental omission rather than an internal contradiction. I therefore recommend CONDITIONAL (matching the reader) rather than REJECT, with the conditions being (i) a reported overlap analysis and (ii) a raw-QA SFT baseline. My concern tracks the reader's weakest assumption, so agreement is 'agree.'","tokens_in":22261,"tokens_out":1825,"duration_ms":17540,"concrete_test":"Compute exact image/QA overlap between the released MMAT-1M and the official test splits or VLMEvalKit benchmark versions (MMStar, MMMU, MathVista testmini, MathVision testmini, AI2D, OCRBench, RealWorldQA, HallusionBench), matching by image URL/hash and question text. Then re-run the InternVL2.5-8B Baseline/ORR/RR evaluations only on benchmark items with zero overlap, and compare the average. Additionally, fine-tune InternVL2.5-8B for one epoch on the original QA pairs (raw answers, no rationale/API/RAG) as an SFT baseline; if the RR model's disjoint-subset average gain over raw-QA SFT is near zero or negative, the dataset's specific agent-tuning value is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The core empirical claim is that MMAT-1M fine-tuning yields significant, generalizable gains (InternVL2.5-8B-RR: +2.7% average, +8.8% Dyn-VQA). The most load-bearing premise is that gains on the eight benchmarks reflect improved reasoning and tool-use, not memorization or train/test overlap. The paper never analyzes overlap between MMAT-1M's five training sources (Visual CoT, LLaVA-CoT, The Cauldron, TabMWP, Infoseek; Supp. Table 6) and the evaluation benchmarks (MMStar, MMMU, MathVista, MathVision, AI2D, OCRBench, RealWorldQA, HallusionBench). This is not merely a formal gap: Supp. Table 6 explicitly lists AI2D, DocVQA, ST-VQA, TabMWP, IconQA, ChartQA, and InfographicsVQA inside training sources, and MathVista/MathVision/OCRBench/MMStar draw on overlapping chart, table, and OCR datasets. Thus the measured 2.7% average improvement could partly reflect verbatim QA pairs rather than transferable agent ability. The paper also lacks a raw-QA ablation: models are only compared against base models, not against SFT on the original QA pairs without rationales, API outputs, or tool calls. Without this comparison, the specific contribution of the agent-style synthesis (CoT/reflection/RAG) is untested, even if some gains are real. The paper's own honesty about reflection helping only slightly (+1.1 average in Table 5) and its stated limitations (including lack of generalization testing) reinforce that the central claim is supported but not yet established beyond reasonable doubt. A definitive test would compute exact sample overlap between the released MMAT-1M and the official evaluation splits, then re-run evaluations on the disjoint subset; if the gains persist (and a raw-QA SFT baseline is matched), the central claim stands.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces MMAT-1M, a dataset of about 1.09 million multimodal question-answer pairs synthesized from five public sources (Visual CoT, LLaVA-CoT, The Cauldron, TabMWP, InfoSeek). A four-stage data engine generates rationales with GPT-4o, interleaves API calls (image caption, OVD, OCR, face detection) and RAG searches, applies reflection to repair inconsistencies, and optionally compresses trajectories into a one-turn ORR format. The authors fine-tune Llama-3.2-11B-Vision-Instruct, MiniCPM-V-2.6, and InternVL2.5-2B/4B/8B with LoRA on the ORR and RR formats and report average gains across eight benchmarks (e.g., InternVL2.5-8B-RR improves by 2.7%) plus an 8.8% gain on Dyn-VQA. They also provide ablations of API, RAG, reflection, and SFT, and qualitative examples of tool invocation.","tokens_in":22576,"tokens_out":6020,"duration_ms":67254,"significance":"If the reported gains are genuine, MMAT-1M would be a useful public resource for multimodal agent tuning: it is the first dataset at the million scale in this niche, spans diverse tasks, combines CoT with reflection and dynamic tools, and shows consistent improvements across three model families. The paper's strengths include the public release, the detailed four-stage pipeline, the large-scale LLM-based quality audit, and the explicit trade-off between ORR and RR formats. The main caveats are that the training/evaluation overlap risk is left entirely unaddressed and that no raw-QA SFT control isolates the contribution of the synthesized rationales; these issues must be resolved before the effectiveness claim can be considered established.","major_comments":[{"comment":"The central empirical claim requires that the evaluation benchmarks be disjoint from the training sources, but the paper provides no overlap analysis. Supplementary Table 6 lists AI2D, ChartQA, TabMWP, ST-VQA, DocVQA, IconQA, and InfographicsVQA among the training sources, while Section 4.2 evaluates on AI2D, MathVista, OCRBench, and MMStar, which are constructed in part from those same datasets. For example, the AI2D benchmark directly uses the AI2D dataset listed in Table 6 under LLaVA-CoT, and OCRBench contains material from ST-VQA and DocVQA. Because the training data contain the same images and questions in transformed form, the reported gains—for instance, InternVL2.5-8B-RR's 2.7% average improvement and 8.8% improvement on Dyn-VQA—could partly reflect memorization of training samples rather than transferable agent ability. Please report exact overlap statistics per benchmark, remove overlapping samples before evaluation, and re-run Table 3 and Table 4 on the filtered benchmarks; alternatively, evaluate on held-out subsets that are provably disjoint from the five training sources.","section":"§4.2 and Supplementary Table 6"},{"comment":"All training comparisons are against untuned base models; there is no SFT control on the original QA pairs without rationales, reflection, API calls, or RAG. Table 5 ablates API, RAG, reflection, and SFT, but always within the RR-style scaffold, so it cannot separate the contribution of the proposed four-stage synthesis from the contribution of adding large amounts of in-domain QA data to the base model. This is load-bearing because the paper's contribution is specifically the agent-style dataset transformation, not merely additional fine-tuning data. Please add SFT on the original QA pairs (with and without plain CoT) as a control and report the same benchmarks and the same model family.","section":"§4.2, Table 3, and Table 5"},{"comment":"The claim that 'over 89% [of samples] demonstrate high-quality reasoning' rests entirely on scores produced by Doubao-1.5-Vision-Pro-32K. No human validation, inter-annotator agreement, or calibration against a human-labeled subset is reported. Since dataset quality is one of the main contributions, the automated judge should be validated before this number is used to support the construction pipeline; at minimum, report a sample size, the distribution of scores, and a human agreement study on a random subset.","section":"§3.2 and Supplementary B"},{"comment":"The Dyn-VQA evaluation, which supports a headline result, lacks sufficient protocol detail. The meanings of 'Query' and 'Golden Query' are not defined, and the retrieval setup is not specified beyond top-k=3 in Section 4.1: the Google Search API configuration, the query-generation prompt, the number of retrieval steps, and the Dyn-VQA version are all missing. This makes the reported 8.8% improvement difficult to reproduce. Please provide the exact evaluation prompts, retrieval configuration, and dataset version.","section":"Table 4 and §4.2"}],"minor_comments":[{"comment":"The Frobenius-norm regularizer on LoRA weight updates is not discussed or ablated; either justify this term or remove it, and clarify how it is implemented in the LoRA update.","section":"§3.3, Eq. (1)"},{"comment":"The row label 'Baseline-RR (w/o SFT)' is ambiguous because the table has no SFT column; it is unclear whether this row is a prompting-only baseline or a trained model. Please add an explicit SFT column or rename the row.","section":"Table 5"},{"comment":"The sentence 'The content of \"reason\" is is why you think...' contains a duplicated 'is'.","section":"Supplementary B, Figure 7"},{"comment":"The entry 'A-OKVQA [43]' cites the ChartQA paper; the reference appears to be incorrect and should be fixed.","section":"Supplementary Table 6"},{"comment":"The paper states that MathVista and MathVision use the testmini set but does not report the corresponding sample sizes or the exact evaluation mode (e.g., with or without CoT); please provide these details for reproducibility.","section":"§4.2"},{"comment":"The caption for Figure 3 is missing; the text '2B 4B 8B 2B 4B 8B...' appears to be placeholder content and should be replaced with a proper description of the efficiency-performance trade-off.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The overlap problem is serious enough that I would want a filtered or held-out evaluation before accepting the effectiveness claim. The raw-QA SFT control is equally important for attribution. The paper is otherwise promising and the requested experiments appear feasible within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on MMAT-1M. The dataset is the real thing: first million-scale multimodal agent SFT resource with CoT, reflection, and API/RAG trajectories, in both iterative and one-turn formats, and it's released. That alone fills a gap; LLaVA-Plus and MM-Traj are much smaller. The four-stage pipeline is a synthesis of known ideas, but the scale and the reflection stage with 57k corrections plus RR/ORR formats are a useful contribution. Training InternVL2.5/LLaMA/MiniCPM variants shows consistent gains across most of eight benchmarks and large gains on Dyn-VQA. The ablation separating API, RAG, reflection, and SFT is decent evidence that the components matter.\n\nThe soft spot is the one the reader flagged: train/eval overlap is never analyzed. Supp Table 6 lists AI2D, ChartQA, TabMWP, ST-VQA, DocVQA, and IconQA inside MMAT-1M, and several of those feed MathVista, MathVision, MMStar, AI2D, and OCRBench. So part of the 2.7% average could be memorization of verbatim QA pairs. The paper doesn't compute overlap with the official eval splits. That's load-bearing for the headline claim, not fatal, because gains on clearly disjoint benchmarks (MMMU, HallusionBench, RealWorldQA, Dyn-VQA) are positive. Still, a leakage analysis and a re-run on a held-out subset would settle it.\n\nSecond soft spot: no raw-QA SFT baseline. They compare only against base models, so we can't tell whether the gains come from the agent-style rationales and tool calls or just from more instruction data on a small base. A baseline fine-tuned on the original QA pairs without rationales would isolate the contribution of their synthesis. Minor third point: the dataset quality eval was done with Doubao, not humans; 89% high-quality is a soft number without inter-annotator agreement.\n\nThe math and released artifacts are fine, no formal circularity, and the citation pattern looks reasonable. The paper is honest enough to report that reflection helps only slightly and to list limitations about generalization.\n\nWho's this for: anyone doing multimodal agent tuning or SFT data synthesis. It deserves a serious referee; the dataset release is valuable even if the headline accuracy claim needs tightening. I'd conditionally accept with a request for leakage analysis and the raw-QA baseline.","headline":"A genuinely useful million-scale multimodal agent-tuning dataset with a plausible but unproven headline: train/eval overlap and a missing raw-QA baseline leave the central claim not fully established.","tokens_in":23238,"tokens_out":1802,"would_cite":true,"duration_ms":22049,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMAT-1M claims to be the first million-scale multimodal agent-tuning dataset, and fine-tuning on it lifts open-source models' reasoning and tool use.","keywords":["multimodal agent tuning","chain-of-thought reasoning","tool use","retrieval-augmented generation","multimodal large language models","reflection mechanism","dataset construction","visual question answering"],"falsifier":"Compare every test question and image in MMStar, MMMU, MathVista, MathVision, AI2D, OCRBench, RealWorldQA, HallusionBench, and Dyn-VQA against the 1,090,263 MMAT-1M training pairs using exact-match and near-duplicate detection; if any benchmark items are present in training (the training source lists include AI2D, TabMWP, ChartQA, DocVQA, ST-VQA, and IconQA), retrain after removing those items and check whether the reported gains disappear.","tokens_in":22040,"feed_emoji":"🤖","tokens_out":10238,"duration_ms":115586,"temperature":0.7,"pith_summary":"MMAT-1M is a proposed answer to the lack of large-scale, high-quality agent-tuning data for multimodal models. The paper claims that by converting one million public visual QA pairs into structured multi-turn trajectories—with chain-of-thought rationales, a reflection step, and dynamic calls to OCR, captioning, detection, face recognition, and retrieval—it produces a dataset that can teach open-source multimodal models to reason and use tools. Fine-tuning on this dataset is reported to lift InternVL2.5-8B by 2.7 points on average across eight benchmarks and by 8.8 points on Dyn-VQA, with gains in all three model families tested. If the claim holds, MMAT-1M would give researchers a public training resource that brings small open models closer to proprietary agent behavior.","feed_headline":"Million-scale dataset lifts multimodal agents' benchmark scores","feed_subtitle":"Fine-tuning on MMAT-1M raises InternVL2.5-8B by 2.7 points on eight benchmarks and 8.8 on Dyn-VQA.","key_machinery":"The load-bearing machinery is the four-stage data engine that turns static QA pairs into executable trajectories. In the foundation stage it normalizes prompts and prepares five tools: image captioning built on scene graphs, open-vocabulary object detection, OCR, face detection, and search-based RAG. In the rationale stage GPT-4o iteratively emits a thought, chooses operator calls, and issues RAG queries until it reaches the answer. In the reflection stage a second pass repairs step-skipping and 'reasoning cheating' by injecting corrective thoughts. In the integration stage multi-turn dialogues are compressed into one-turn ORR samples with tool outputs prepended to the input. The RR and ORR formats are then used for LoRA fine-tuning, so the reasoning traces themselves are the training signal.","core_discovery":"The central claim is that MMAT-1M, assembled by rewriting 1,090,263 question-answer pairs from five public multimodal sources into structured agent trajectories, is a workable training resource for multimodal agent tuning. The paper reports that a model fine-tuned on this data, InternVL2.5-8B-RR, improves from 60.7 to 63.4 average accuracy on an eight-benchmark suite and from 35.2 to 44.0 on Dyn-VQA Golden Query, and that similar gains appear across Llama-3.2-11B-Vision-Instruct, MiniCPM-V-2.6, and all InternVL2.5 sizes tested. The authors present this as evidence that combining chain-of-thought rationales, a reflection step that repairs faulty reasoning, and dynamic calls to OCR, captioning, detection, face recognition, and web search can teach open-source models to behave like tool-using agents.","pith_inferences":["As an extension, the ORR compression recipe could be applied to any existing multi-step agent-trajectory corpus, turning slow multi-turn traces into fast one-pass training data for latency-constrained deployments.","The reflection stage is a generic repair mechanism: the same two-pass design, detecting step-skipping and detecting forced alignment to the answer, could clean other automatically generated reasoning datasets, not only visual ones.","The reported zero-shot invocation of an unseen celebrity-recognition operator suggests the model learns a general when-to-call-a-tool policy; a systematic test would hold out one operator during training and probe whether the fine-tuned model spontaneously calls it."],"forward_implications":["Fine-tuning open vision-language models on MMAT-1M's RR trajectories is reported to improve accuracy on general, mathematical, diagram, OCR, spatial, and hallucination benchmarks, so the dataset is usable as a drop-in SFT corpus.","The reflection mechanism contributes directly: removing reflection lowers the average from 61.3 to 60.2, and removing API calls or RAG lowers it to 57.3 or 59.8, so the tool-use and self-correction components are what make the trajectory data useful.","The ORR format retains most of the benefit while adding less inference latency than RR, offering a practical speed-accuracy tradeoff for deployed agents.","Fine-tuned models can invoke a tool they were never trained on, such as a celebrity-recognition operator, when prompted, indicating that the training teaches a general tool-calling policy.","Gains on Dyn-VQA, such as InternVL2.5-8B rising from 27.0 to 36.8 on Query, show that the dataset transfers to multi-hop retrieval tasks that require planning a web search."],"supporting_citations":[{"why":"Supplies roughly 434k QA pairs from visual reasoning, document, chart, and relation tasks, forming the largest source component of MMAT-1M.","marker":"[52]"},{"why":"Supplies about 98k QA pairs aimed at complex reasoning and systematic thinking, contributing the LLaVA-CoT portion of the trajectory data.","marker":"[69]"},{"why":"Supplies about 215k examples spanning OCR documents, tables, charts, and interleaved text-image data through The Cauldron.","marker":"[29]"},{"why":"Supplies about 23k tabular math QA pairs that become the TabMWP portion of MMAT-1M.","marker":"[41]"},{"why":"Supplies about 131k visual information-seeking QA pairs that drive the RAG and knowledge-retrieval calls in the dataset.","marker":"[9]"},{"why":"Provides the Dyn-VQA benchmark and the F1-Recall metric used to measure multi-hop retrieval and RAG gains.","marker":"[31]"},{"why":"Supplies the evaluation harness used to run the eight public benchmarks, making the reported comparison numbers reproducible.","marker":"[16]"},{"why":"Provides the InternVL2.5 series, the primary base models fine-tuned with ORR and RR and reported across the benchmarks.","marker":"[11]"},{"why":"Provides Llama-3.2-11B-Vision-Instruct, one of the open model families used to show cross-family gains from MMAT-1M.","marker":"[46]"},{"why":"Provides MiniCPM-V-2.6, the second independent model family fine-tuned and evaluated for cross-family applicability.","marker":"[72]"}],"fun_headline_variants":["First million-scale multimodal agent dataset lifts scores by 2.7%","MMAT-1M: turning 1M QA pairs into agent tasks improves tool use","InternVL2.5-8B gains 8.8 points on Dyn-VQA after MMAT-1M tuning","1M multimodal agent trajectories improve reasoning and tool use across models","MMAT-1M dataset: 1M QA pairs rewritten as agent tasks, lifts benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's reported gains assume that the eight evaluation benchmarks share no test items with the five source datasets used to build MMAT-1M; if benchmark questions appear in training, the 2.7-point and 8.8-point improvements would reflect memorization rather than learned reasoning.","fun_headline_variants_meta":{"raw":{"variants":["First million-scale multimodal agent dataset lifts scores by 2.7%","MMAT-1M: turning 1M QA pairs into agent tasks improves tool use","InternVL2.5-8B gains 8.8 points on Dyn-VQA after MMAT-1M tuning","1M multimodal agent trajectories improve reasoning and tool use across models","MMAT-1M dataset: 1M QA pairs rewritten as agent tasks, lifts benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001118,"raw_usage":{"total_tokens":4712,"prompt_tokens":1062,"completion_tokens":3650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":3536}},"tokens_in":678,"tokens_out":3650,"duration_ms":31242,"temperature":1.0,"reasoning_tokens":3536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:12:51.863749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare every test question and image in MMStar, MMMU, MathVista, MathVision, AI2D, OCRBench, RealWorldQA, HallusionBench, and Dyn-VQA against the 1,090,263 MMAT-1M training pairs using exact-match and near-duplicate detection; if any benchmark items are present in training (the training source lists include AI2D, TabMWP, ChartQA, DocVQA, ST-VQA, and IconQA), retrain after removing those items and check whether the reported gains disappear.","supporting_citations":[{"cited_title":"Visual cot: Advancing multi-modal language models with a comprehen- sive dataset and benchmark for chain-of-thought reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies roughly 434k QA pairs from visual reasoning, document, chart, and relation tasks, forming the largest source component of MMAT-1M."},{"cited_title":"Llava-cot: Let vision language models reason step- by-step, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies about 98k QA pairs aimed at complex reasoning and systematic thinking, contributing the LLaVA-CoT portion of the trajectory data."},{"cited_title":"Vlmevalkit: An open-source toolkit for evaluating large multi-modality models","cited_arxiv_id":null,"evidence_quote":"Supplies the evaluation harness used to run the eight public benchmarks, making the reported comparison numbers reproducible."},{"cited_title":"Llama 3.2: Revolutionizing edge ai and vision with open, customizable models","cited_arxiv_id":null,"evidence_quote":"Provides Llama-3.2-11B-Vision-Instruct, one of the open model families used to show cross-family gains from MMAT-1M."}],"review_version":1}