{"id":"7d776596-4d48-491b-a809-452be2548a42","arxiv_id":"2412.05237","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fully open pipeline that rewrites multimodal instruction data into CoT-style rationales yields a 12M dataset and an 8B model with strong benchmark gains, though some evaluation benchmarks overlap the training data.","lead":"The authors built a 12 million sample visual instruction dataset by rewriting existing image Q&A into detailed step-by-step reasoning with open models, then trained an 8B vision-language model on it. The model posts large gains on math and reasoning benchmarks, suggesting that open-source data pipelines can compete with paid proprietary data generation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MathVerse +8.1% headline is not decontaminated: MathVerse is assembled from existing visual-math datasets, several of which appear in the MAmmoTH-VL training mix, so the gain may reflect train/eval overlap rather than CoT rewriting.","rationale":"The reader's verdict was CONDITIONAL, with the self-judge assumption as the weakest point. I partly agree: self-filtering is self-referential, and the human Kappa agreement on only 60 items is thin evidence. However, the most load-bearing condition for the central leaderboard claim is that the headline benchmark numbers measure held-out reasoning. The paper contains training data from exact evaluation benchmarks such as ChartQA, DocVQA, InfoVQA, and WildVision, and it does not report any near-duplicate removal for MathVerse, whose constituent source datasets overlap with the training collection. The +8.1% MathVerse number is therefore the least secure pillar of the paper's central claim. If overlap is high, the claim that CoT rewriting elicits reasoning is not supported; if overlap is low, this conditional can be lifted on that axis. This is a standard verification step for instruction-tuning papers that assemble 153 public datasets, not an accusation of bad faith. No change to the conditionality is needed, but the conditional should explicitly require a MathVerse-specific decontamination check before the headline gain is accepted.","tokens_in":43437,"tokens_out":8088,"duration_ms":88719,"concrete_test":"Run a decontamination audit: collect images and question text from the MAmmoTH-VL-Instruct training set (at least the math subset) and from MathVerse testmini; compute exact image hashes (pHash/SSIM) and normalized question similarity (e.g., BLEU > 0.8 or edit distance). Report overlap rate overall and by MathVerse subset. Then restrict MathVerse evaluation to items with no near-duplicate in training and recompute MAmmoTH-VL-8B versus LLaVA-OV-7B; if the +8.1-point delta collapses to less than about 2 points, the headline gain is attributable to overlap rather than the rewriting pipeline. A weaker but still informative check: retrain or re-evaluate after dropping overlapping training sources such as MathV360K and Geo170K from the mix.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest empirical evidence is the MathVerse +8.1% gain (Table 2), but no decontamination analysis is reported anywhere. This is not a hypothetical worry: MathVerse (Zhang et al., 2024c) is documented as curating existing visual-math problems, and the training mix in Figure 3 and Appendix A3 explicitly contains many of those source collections—Geometry3K, GeoQA+, TabMWP, UniGeo, CLEVR-Math, MapQA, IconQA, and MathVision—often in the MathV360K repackaging used as Group B rewrite inputs. Since MAmmoTH-VL-8B is trained on all 12M pairs without excluding benchmark-derived samples, exact or near-duplicate MathVerse items may be present in its training set. The self-filtering weakness identified by the reader is real, but it does not directly invalidate the benchmark claim; uncontrolled train/test overlap would. The paper also evaluates on ChartQA, DocVQA, InfoVQA, and WildVision, which are themselves training sources, showing that no general decontamination policy was applied. Until a MathVerse-specific overlap check is provided, the central claim that an open rewriting pipeline elicits reasoning is not fully established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MAmmoTH-VL-Instruct, a 12M-pair multimodal instruction-tuning dataset constructed by rewriting 153 open instruction datasets with open-weight models (InternVL2-Llama3-76B and Llama-3-70B-Instruct) and then filtering the rewritten data with the same InternVL2 model acting as a judge. The authors train MAmmoTH-VL-8B, a LLaVA-OneVision-style model with a Qwen2.5-7B language backbone, and report state-of-the-art open-source results on MathVerse (+8.1%), MMMU-Pro (+7%), and MuirBench (+13.3%), along with gains on chart and document benchmarks. Ablations with a 1.5B backbone examine filtering, rewritten/original mixing ratio, training-data scale, and rewrite-model size.","tokens_in":43619,"tokens_out":10272,"duration_ms":96016,"significance":"If the reported gains are clean, the paper makes a useful community contribution: it demonstrates a fully open pipeline for producing rationale-rich multimodal instruction data at scale, releases the resulting dataset, and provides detailed prompts and scaling curves. The use of open models only, the careful categorization of 153 sources, and the public evaluation protocol via LMMs-Eval are all strengths. However, the central empirical claim is currently not cleanly established because several evaluation benchmarks overlap with training sources, the self-filtering evaluation is partly circular, and some reported deltas are not reproducible from the paper's own tables. These issues are fixable and do not invalidate the dataset contribution, but they must be resolved before the performance claims can be accepted.","major_comments":[{"comment":"The evaluation is contaminated by training-source overlap. ChartQA, DocVQA, Infographic-VQA, and WildVision appear in the training mix (Figure A4 and Figure A3), and Group A includes Video-MME and EgoSchema, which are then used as evaluation benchmarks in Table 4. MathVerse is assembled from existing visual-math problems, and several of its source collections (Geometry3K, GeoQA+, TabMWP, UniGeo, CLEVR-Math, MapQA, IconQA, MathVision) appear in the Group B rewrite inputs via MathV360K. No decontamination analysis is reported anywhere in the paper. As a result, the headline deltas (MathVerse +8.1, MMMU-Pro +7, MuirBench +13.3) cannot be attributed cleanly to the rewriting pipeline. Please report exact and near-duplicate overlap statistics for every evaluation benchmark, and either retrain without overlapping sources or explicitly quantify the effect of overlap on the reported gains.","section":"Section 2.1, Figures 3/A3/A4, Tables 2-4"},{"comment":"The reported dataset size is internally inconsistent. Table A2 lists pre-filter counts summing to about 12.3M and post-filter counts to about 7.8M, while the paper claims a 12M final dataset and Section 4.1 trains Stage-2 on 10M single-image pairs. The per-category proportions in Figure 3 also do not match Table A2's per-category counts (e.g., Chart is 15.4% of 12M in Figure 3 but 7.3M pre-filter or 3.8M post-filter in Table A2). Please clarify whether Table A2 refers to the entire corpus, the Group B rewritten subset, or a sample; without this, the dataset-size and scaling claims in Figure 1 cannot be reproduced.","section":"Section 4.1 and Table A2"},{"comment":"The self-filtering and quality verification use the same model (InternVL2-Llama3-76B) both to generate and to judge the data. The only human-agreement check is on 60 items, and Figure 4's content and relevance scores are assigned by the same model that produced the rewrites. This leaves open the possibility that systematic errors of the generator are shared by the judge, so the quality demonstration is partly self-confirming. Please add an independent judge (a different large MLLM or human annotation on a substantially larger sample) and report agreement separately for content and relevance, and per data category.","section":"Sections 2.3, 3.1, 3.3"},{"comment":"The 'Over Best Open-Source (~10B Scale)' deltas are not reproducible from the numbers in the same tables. For example, in Table 2, Qwen2-VL-7B scores 26.9 on MMMU-Pro vision while MAmmoTH-VL-8B scores 25.3, yet the delta row is +7.1; in Table 3, Qwen2-VL-7B scores 76.5 on InfoVQA while MAmmoTH-VL-8B scores 73.1, yet the delta row is +2.2; and in Table 4, InternVL2-8B scores 59.4 on MuirBench, above MAmmoTH-VL-8B's 55.1, despite the +13.3 delta. Please specify the exact comparison set (which models are considered 'open-source' and what '~10B scale' means) and recompute the deltas; the current presentation overstates the state-of-the-art claims.","section":"Tables 2, 3, 4 (delta rows)"},{"comment":"All ablations use Qwen2.5-1.5B-Instruct as the backbone and report single runs without variance estimates, while the main model is 8B. The claims that self-filtering, rewriting, and the mixing ratio are crucial for the 8B results are therefore extrapolations from a much smaller model. Please either run the key ablations (filtering and rewriting ratio) at the 8B scale or add variance estimates and explicitly discuss the transfer limitations; without this, the component importance claims are not directly supported for the submitted model.","section":"Section 5 and Appendix A.1"}],"minor_comments":[{"comment":"The left panel plots content and relevance scores on a common axis but does not define how the two metrics are aggregated or whether the displayed value is a mean over samples; a legend or separate panels would improve clarity.","section":"Figure 4"},{"comment":"The Kappa analysis is based on only 60 items and no confidence intervals are reported; given the small sample, the difference between 0.55 and 0.64 should be interpreted cautiously rather than as strong evidence that the model agrees with humans better than humans agree with each other.","section":"Section 3.3 and Table A5"},{"comment":"The caption should state whether the counts are for the full corpus, the Group B rewritten subset, or a sample; as written, the table conflicts with the 'randomly sample 1M instances' description in Section A.1.","section":"Table A2 caption"},{"comment":"The reference list contains duplicate entries (e.g., ChartQA, DocVQA, UReader) and some entries lack venue information; please clean up the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the evaluation contamination and the unreproducible delta rows. If the overlap analysis shows that MathVerse and MMMU-Pro gains largely disappear after excluding near-duplicates, the paper's main claim would reduce to a data-release contribution with weaker benchmark evidence. I would ask the authors for a complete decontamination table and a clear baseline definition before any acceptance decision. The dataset itself and the open pipeline are valuable and within scope for the journal; the issues are fixable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful artifact here is the 12M open dataset and the fully open rewriting/filtering pipeline. The trained MAmmoTH-VL-8B looks strong, but the headline numbers rest on benchmark overlap that the paper never addresses. The MathVerse gain specifically is suspect: MathVerse is assembled from visual-math collections that sit in the training mix (Geometry3K, GeoQA+, TabMWP, UniGeo, CLEVR-Math, MapQA, IconQA, MathVision), often via MathV360K. Without a MathVerse-specific decontamination check, the +8.1% is not evidence of elicited reasoning.\n\nWhat is genuinely new: the dataset itself (12M instruction-response pairs with CoT rationales, built from only open models), the trained 8B model, and the careful ablations comparing rewrite model sizes, mixing ratios, filter effects, and scaling curves. The paper is honest about limitations: it states that the quality verification uses the same model as judge (Sections 2.3 and 3.1), and it includes a Kappa study against human evaluators. That is more than most such papers do. The scaling plots show consistent improvements with data size, which supports the data-centric story.\n\nThe soft spots are real and load-bearing for the headline claims. First, no decontamination analysis appears anywhere; the training list includes ChartQA, DocVQA, InfoVQA, and WildVision, and the paper evaluates on them, so those deltas are circular. Second, the MathVerse overlap is concrete, not hypothetical: the training list in Figure 3 and Appendix A4 contains several MathVerse source datasets via MathV360K. Third, all ablations use a 1.5B backbone, so they do not directly validate the 8B results. Fourth, evaluations are single-run without error bars. Fifth, the model-as-judge uses the same InternVL2-Llama3-76B that generated the data, making the quality scores partly self-confirming, though the human Kappa check partially mitigates this.\n\nThis paper is for researchers working on multimodal data construction and open-source MLLMs. They will get immediate value from the dataset and the training recipe, and the ablations are a useful reference. The benchmark-sweeping claims should be read with caution until decontamination is provided.\n\nRecommendation: worth a serious referee, with major revision. The dataset and pipeline deserve publication, but the empirical claims need a decontamination pass, ideally with an independently built MathVerse overlap analysis and results on held-out benchmarks. If the authors supply that, the core contribution stands.","headline":"Useful open data pipeline and strong 8B model, but the headline benchmark gains are not decontaminated; the MathVerse claim in particular can't be trusted until overlap is checked.","tokens_in":44264,"tokens_out":2210,"would_cite":true,"duration_ms":21528,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that an open, scalable pipeline of rewriting existing visual instruction data into chain-of-thought rationales and self-filtering the result can push an 8B multimodal model to top open-source benchmark results.","keywords":["multimodal large language models","instruction tuning","chain-of-thought reasoning","data rewriting","self-filtering","open-source models","visual reasoning","dataset scaling"],"falsifier":"Take a random sample of accepted OCR and chart pairs from the 12M dataset, have independent human annotators mark each answer as factually correct or not, and compute the false-acceptance rate of the model judge; if a substantial share of accepted pairs contain errors that humans catch, the self-filtering step does not remove hallucinations and the reasoning gains need another explanation.","tokens_in":43169,"feed_emoji":"🧠","tokens_out":8127,"duration_ms":76167,"temperature":0.7,"pith_summary":"The paper is trying to establish that the main bottleneck in open multimodal reasoning is instruction data, not architecture, and that a cheap, fully open data pipeline can remove it. The authors collect 153 public instruction datasets, rewrite 60 of them into 12 million question-answer pairs with detailed step-by-step rationales using open-weight models, and filter the result with the same model acting as judge. Training an 8B model on this MAmmoTH-VL-Instruct data yields reported gains of 8.1 points on MathVerse, 7 points on MMMU-Pro, and 13.3 points on MuirBench over prior open-source models, plus up to 4 points on non-reasoning benchmarks. If the results hold, high-quality reasoning data becomes reproducible without proprietary APIs, and dataset construction itself becomes a scalable lever for model improvement.","feed_headline":"Rewriting 12M examples lifts an 8B model's reasoning up to 13.3%","feed_subtitle":"A 12M instruction set rebuilt with chain-of-thought rationales puts an 8B open model ahead of larger rivals.","key_machinery":"The load-bearing machinery is the rewrite-and-filter pipeline rather than any single new architecture. First, manual screening assigns each of 153 datasets to Group A (kept as is), Group B (rewritten), or Group C (discarded). Second, task-specific prompts drive InternVL2-Llama3-76B to rewrite Group B image tasks into complex instructions with detailed rationales, while Llama-3-70B-Instruct rewrites caption data using text only. Third, the same InternVL2 model scores each rewritten pair with a binary Yes/No judgement on relevance, consistency, and factual accuracy, removing roughly half of OCR and chart data where hallucinations are most common. Training then mixes 70% rewritten with 30% original data and scales the mixture from 2M to 12M samples across ten categories.","core_discovery":"The central claim is that simple rewriting plus self-filtering, applied at 12-million-sample scale, converts phrase-level visual question answering data into chain-of-thought reasoning data, and that an 8B model trained on it reaches top results among open-source models on reasoning-heavy benchmarks. The authors would state this as: our three-step pipeline, curating 153 sources into keep, rewrite, and drop groups, rewriting task-aware prompts with InternVL2-Llama3-76B and Llama-3-70B-Instruct, then filtering hallucinated or inconsistent pairs with the same InternVL2 model as judge, produces a dataset whose faithfulness and diversity let an 8B LLaVA-OneVision model beat much larger open models. Ablations show that filtering matters most for chart and document understanding, that mixing 70% rewritten with 30% original data helps, and that performance improves steadily as training data grows from 2M to 12M samples.","pith_inferences":["A direct way to test the generalization claim is to evaluate MAmmoTH-VL-8B on reasoning benchmarks released after this dataset was assembled; if the edge shrinks, part of the reported gain may come from overlap with training sources.","Because the judge is the same model that wrote the answers, systematic blind spots could let confident hallucinations pass; an independent verification model or a targeted human audit of OCR and chart subsets would quantify how much filtering actually buys.","The observed parity between 7B and 76B rewrite models on knowledge tasks suggests the recipe may be cheap enough to reuse for video, audio, or non-English modalities where data is scarcer.","A practical extension would insert an external OCR or chart-parsing tool into the rewrite step, since those categories show the highest filter rates and the largest measured gains from filtering."],"forward_implications":["Open-source teams can build large reasoning-oriented multimodal datasets without paying for proprietary generation or human annotation at scale.","Ablations indicate that filtering is a necessary component: removing it hurts performance most on chart and document understanding, where hallucination rates are highest.","Data scale behaves like a reliable lever: gains track training-set size from 2M to 12M samples across nearly all reported benchmarks.","Rewrite-model size matters unevenly: 7B-scale rewrites match 76B-scale rewrites on knowledge and reasoning tasks, but larger models are needed for chart, document, and preference-style data."],"supporting_citations":[{"why":"Supplies the LLaVA-OneVision architecture and base training recipe that MAmmoTH-VL-8B is built on.","marker":"Li et al., 2024b"},{"why":"The InternVL2-Llama3-76B open model performs both the rewriting of image data and the self-filtering judge step.","marker":"Chen et al., 2023b"},{"why":"Llama-3-70B-Instruct rewrites caption data into task-oriented Q&A pairs using text only.","marker":"Meta, 2024a"},{"why":"Establishes chain-of-thought prompting as the reasoning mechanism the rewritten rationales are designed to elicit.","marker":"Wei et al., 2022"},{"why":"Provides MMMU-Pro, one of the three key benchmarks where the trained model reports the largest gains.","marker":"Yue et al., 2024b"},{"why":"Provides MathVerse, the benchmark with the reported 8.1-point improvement.","marker":"Zhang et al., 2024c"},{"why":"Provides MuirBench, the multi-image benchmark with the reported 13.3-point improvement.","marker":"Wang et al., 2024a"},{"why":"LMMs-Eval is the evaluation harness used to produce standardized benchmark numbers.","marker":"Zhang et al., 2024a"},{"why":"Supplies Cohen's Kappa method used to argue that the model judge agrees with human evaluators.","marker":"McHugh, 2012"}],"fun_headline_variants":["12M CoT rewrites lift 8B model's reasoning by 13.3%","Open-source pipeline yields 12M reasoning pairs, beats larger models","Scaling instruction data with rationales boosts multimodal reasoning by 13.3%","Rewriting 12M examples with CoT lifts 8B model to SOTA","Self-filtering key: 12M rationales raise 8B model's benchmark scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the open model that generated the rewritten answers, InternVL2-Llama3-76B, is also trustworthy enough to judge whether those answers are correct, so any blind spot the model has in reading images can quietly become the training set's standard.","fun_headline_variants_meta":{"raw":{"variants":["12M CoT rewrites lift 8B model's reasoning by 13.3%","Open-source pipeline yields 12M reasoning pairs, beats larger models","Scaling instruction data with rationales boosts multimodal reasoning by 13.3%","Rewriting 12M examples with CoT lifts 8B model to SOTA","Self-filtering key: 12M rationales raise 8B model's benchmark scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001169,"raw_usage":{"total_tokens":4843,"prompt_tokens":961,"completion_tokens":3882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":3773}},"tokens_in":577,"tokens_out":3882,"duration_ms":27530,"temperature":1.0,"reasoning_tokens":3773,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:49:27.226443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of accepted OCR and chart pairs from the 12M dataset, have independent human annotators mark each answer as factually correct or not, and compute the false-acceptance rate of the model judge; if a substantial share of accepted pairs contain errors that humans catch, the self-filtering step does not remove hallucinations and the reasoning gains need another explanation.","supporting_citations":[],"review_version":1}