{"id":"e1b25edb-4ef0-49f4-b7f0-12e017d606bf","arxiv_id":"2508.12680","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Vision-G1 trains a 7B VLM with multi-round GRPO on an influence-filtered, difficulty-curated 46-source dataset spanning eight visual reasoning domains.","lead":"The authors assemble a 46-source, 8-domain RL-ready visual reasoning dataset, filter it with influence functions, and train a 7B vision-language model, Vision-G1, with multi-round GRPO and a difficulty curriculum. The paper claims state-of-the-art results across 17 visual reasoning benchmarks, but several evaluation benchmarks also appear in the training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation benchmarks overlap training sources; the SOTA claim is unsupported without a deduplication or holdout report.","rationale":"The reader's weakest assumption, that evaluation benchmarks are disjoint from training data, is exactly the load-bearing condition for the paper's central claim. The paper's own Table 5 lists ChartQA, VQA-RAD, SLAKE, and Path-VQA as training sources, and Section 4.1 lists the same datasets as evaluation benchmarks. No deduplication, split version, or holdout procedure is reported, so the burden of proof is on the authors. This concern is more fundamental than the math-average overclaim, because it affects all the headline comparisons, not just one table. I agree with the reader's REJECT verdict: the central claim is not supportable as stated. If a contamination scan clears the benchmarks, the verdict could be revisited, but the current manuscript does not provide the necessary evidence. The ablation study in Table 4 is a useful internal control, but it does not validate the benchmark fairness, and the limitations section acknowledges resource constraints rather than addressing evaluation leakage.","tokens_in":23310,"tokens_out":5260,"duration_ms":54438,"concrete_test":"Run a contamination scan between the released 40k training set and every evaluation set in Tables 1-3: normalize question text (lowercase, strip punctuation and numbers) and compute image hashes or pHash/CLIP similarity. For any benchmark with one or more exact or near-duplicate matches, re-evaluate Vision-G1 on the deduplicated non-overlapping subset and report the score delta. If no overlap exists and the official train/test split documentation is released, the concern is resolved; if overlap exists and the scores drop materially, the SOTA claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Vision-G1-7B achieves state-of-the-art visual reasoning and outperforms proprietary models such as GPT-4o and Gemini-1.5 Flash. For that claim to hold, the evaluation benchmarks must measure generalization rather than memorization. Table 5 explicitly lists ChartQA, VQA-RAD, SLAKE, and Path-VQA as training data sources, while Section 4.1 and Tables 1 and 3 evaluate on ChartQA, VQA-RAD, SLAKE, and Path-VQA. The paper never states which official splits were used, whether benchmark images or questions were deduplicated against the 40k training set, or whether any holdout procedure was followed. Consequently, the reported gains, e.g., ChartQA 90.8 vs. 88.3 for Qwen2.5-VL-7B and Path-VQA/SLAKE improvements in Table 3, are consistent with the model having trained on the same or near-duplicate items. The issue is compounded by the use of MathVista-mini as a validation signal during training (Figure 5 and Section 4.1), making the MathVista result in Table 1 less clean as a held-out measurement. A secondary overclaim is Section 4.2's statement that Vision-G1 achieves the best math average; Table 2 shows Vision-R1 at 42.5 vs. Vision-G1 at 42.1. These problems do not disprove the proposed curation method, but they make the headline result unreliable as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Vision-G1 (arXiv:2508.12680) proposes a pipeline for training a 7B vision-language model for general visual reasoning. The authors assemble 46 datasets across 8 domains, convert them to a verifiable rule-reward format, filter them with an influence-function score (Eq. 2) and a difficulty filter that keeps instances with average rollout accuracy in [0.2, 0.8], and train Qwen2.5-VL-7B-Instruct with multi-round GRPO and a data curriculum. Results are reported on general, mathematical, and domain-specific benchmarks (Tables 1-3), with claims of state-of-the-art performance among similarly sized models and superiority over proprietary models such as GPT-4o. The ablations (Table 4) indicate that the proposed components contribute to the reported in-paper gains.","tokens_in":23544,"tokens_out":6860,"duration_ms":64734,"significance":"If the evaluation were clean, this would be a useful contribution: it addresses a real bottleneck—the scarcity of verifiable reward data outside mathematical reasoning—with a plausible, fairly general recipe, and the ablations suggest that the data-selection and curriculum components matter. The open release of model, code, and dataset is also a concrete strength. However, as submitted, the headline generalization claims are not supported because several evaluation benchmarks also appear as training sources, no deduplication or holdout procedure is reported, and one of the paper's own summary claims is contradicted by its own table. The contribution should be judged only after a re-run on properly held-out benchmarks.","major_comments":[{"comment":"Table 5 lists ChartQA, VQA-RAD, SLAKE, and Path-VQA as training data sources, and Section 4.1 lists the same four datasets among the evaluation benchmarks reported in Tables 1 and 3. The manuscript nowhere states which official splits were used, whether the training and evaluation items were deduplicated, or whether any evaluation split was held out of the final 40k-question training set. Given this overlap, the reported gains on these benchmarks (e.g., ChartQA 90.8 vs. 88.3 for Qwen2.5-VL-7B, and the SLAKE and Path-VQA improvements in Table 3) are consistent with the model having encountered the same or near-duplicate questions during RL, rather than with generalization. In addition, Figure 5 and Section 4.1 report that MathVista-mini is used as a validation signal during training, so the MathVista row in Table 1 is not a clean held-out measurement. These issues directly undercut the abstract's state-of-the-art claim and the Section 4.2 statement that the method achieves the best performance on most benchmarks.","section":"§3.1/Table 5; §4.1/Tables 1, 3"},{"comment":"The paragraph on math-related visual reasoning states that 'our Vision-G1 achieves the best performance' and later asserts that Vision-G1 achieves higher average accuracy than Vision-R1. Table 2's Avg. column shows Vision-R1-7B at 42.5 and Vision-G1 at 42.1, so both claims are contradicted by the paper's own numbers. The text needs to be corrected or the comparison and averaging procedure need to be recomputed.","section":"§4.2, Table 2"},{"comment":"The abstract's claim that the model 'outperforms ... proprietary models like GPT-4o' is not supported by the reported tables as a general statement. In Table 1, GPT-4o is ahead of Vision-G1 on MMMU-Val (69.1 vs. 53.4) and MMMU-Pro (51.9 vs. 41.2), and in Table 2 GPT-4o's average is 45.7 vs. 42.1. Vision-G1 does beat GPT-4o on several specific benchmarks (e.g., ChartQA 90.8 vs. 85.7 and MathVista 76.1 vs. 63.8), so the claim should be restricted to those benchmarks rather than stated as general superiority.","section":"Abstract; §4.2"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'there hav been some initial attempts' should be 'there have been some initial attempts'.","section":"§1"},{"comment":"The text says 'we also collect four comprehensive visual reasoning datasets' and then lists five datasets: MM-R1, VerMulti, ThinkLite, ViRL39K, and MMK12. The count should be corrected.","section":"Appendix A.2"},{"comment":"The number of evaluation benchmarks is inconsistent: the abstract and introduction say 17, Section 4.2 says 18, and the conclusion says 17.","section":"Abstract, §4.2, §5"},{"comment":"References [15] and [68] are the same MM-Eureka paper, and references [18] and [19] both carry the title 'Vision-R1' with different author lists; one of the two citations is likely to be incorrect.","section":"References"},{"comment":"The symbol Sim in Eq. (1) is not defined; the authors should state explicitly that it denotes cosine similarity.","section":"Eq. (1)"},{"comment":"The text spells 'InternVL2.5-78B' as 'InvernVL2.5-78B'; this should be corrected.","section":"§4.2"}],"recommendation":"reject","confidential_remarks":"The paper's method may be salvageable, but as submitted the central empirical claims are invalidated by training/evaluation overlap and by an internal contradiction in the reported math averages. A full re-run on properly held-out benchmarks would be needed before the state-of-the-art claim can be assessed; I therefore recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, the 8-domain, 46-source RLVR dataset and the released pipeline are genuinely useful, and the domain-transfer analysis is the kind of evidence this literature usually lacks. Second, the headline benchmark claims are not reliable as reported, and the math-average sentence is simply contradicted by the paper's own Table 2.\n\nWhat is actually new: the dataset, the influence-function plus difficulty-filter plus multi-round GRPO combination, and the heatmap showing which training domains help which evaluation tasks. The ablations support the internal logic of the pipeline, and the model, code, and data are public. That is real value.\n\nThe soft spots are concentrated in the evaluation. Table 5 lists ChartQA, VQA-RAD, SLAKE, and Path-VQA as training sources, and Tables 1 and 3 report them as evaluation benchmarks. The paper never states which official splits were used or whether any deduplication or holdout was done. The stress-test note is right: the gains on those benchmarks are consistent with the model having trained on the same or near-same items. Using MathVista-mini to monitor training (Figure 5) also makes the MathVista number less clean as a held-out measurement. Separately, Section 4.2's claim that Vision-G1 has the best math average is wrong: Table 2 shows Vision-R1 at 42.5 and Vision-G1 at 42.1. And the abstract's \"outperforming GPT-4o\" is selective, since GPT-4o is ahead on most math and several domain benchmarks. These are fixable, but they need fixing before the performance numbers can be quoted.\n\nThe method itself holds up better than the evaluation. The ablations show each component contributes, and the dataset release is a service to the community. I would send this to a serious referee, but with a clear request: recompute results on held-out splits or show deduplication, correct the overclaims, and be explicit about which benchmarks are diagnostic. This is a solid systems/dataset paper with a correctable evaluation, not a flawed idea.","headline":"A useful RLVR dataset and pipeline wrapped in benchmark claims that are partly contaminated and one internally contradicted; the evaluation needs a careful redo, but the method is worth refereeing.","tokens_in":24166,"tokens_out":4007,"would_cite":true,"duration_ms":38284,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-G1 claims a 7B visual-reasoning model can beat much larger systems through data curation and multi-round RL.","keywords":["visual reasoning","vision-language model","reinforcement learning","data curation","influence function","data curriculum","GRPO","multi-domain dataset"],"falsifier":"Run the released training set against the evaluation benchmarks with near-duplicate detection, then retrain on a strictly deduplicated subset; if the reported gains shrink or disappear, the central claim of generalization would be falsified.","tokens_in":23066,"feed_emoji":"🧠","tokens_out":8775,"duration_ms":77518,"temperature":0.7,"pith_summary":"The paper tries to establish that a vision-language model can be taught to reason across many visual domains, not just math, by curating a large verifiable-answer dataset and training with reinforcement learning. It assembles 46 datasets into 8 domains, filters them with an influence-function score and a difficulty band, then runs multi-round GRPO with a data curriculum. The resulting 7B model, Vision-G1, is reported to reach state-of-the-art scores on 17 visual-reasoning benchmarks, beating similar-sized RL-trained models and, on several benchmarks, bigger or proprietary systems such as GPT-4o and Gemini-1.5 Flash. If the result holds, it would show that general visual reasoning can be elicited at small scale through data curation rather than model size alone.","feed_headline":"Curated data plus RL lifts a 7B model past bigger rivals","feed_subtitle":"Influence-based filtering and a difficulty curriculum take a 7B model past GPT-4o on several visual-reasoning benchmarks.","key_machinery":"The load-bearing mechanism is a three-stage curation-and-training loop. First, influence-function filtering scores each training instance by gradient similarity to other instances inside and outside its domain (after LoRA fine-tuning on reject-sampled reasoning chains) and removes negative-influence samples. Second, difficulty filtering estimates each instance's rollout accuracy with k=16 samples and keeps only instances with average accuracy between 0.2 and 0.8. Third, multi-round GRPO RL rebuilds the dataset from the current checkpoint each round, re-selects moderately difficult questions, and trains on about 40k curated questions per round. The paper argues that the interaction of these three components is what lets a 7B model generalize across infographic, math, spatial, cross-image, GUI, medical, science, and common-sense reasoning.","core_discovery":"The central claim is that a general visual-reasoning VLM can be trained by assembling 46 verifiable-answer datasets across 8 domains, filtering out unhelpful samples with an influence-function gradient similarity score, and then running multi-round GRPO reinforcement learning with difficulty-based data selection. The paper reports that the resulting 7B model, Vision-G1, achieves the best average performance among 7B baselines across the benchmarks and, on several of them, matches or exceeds much larger open models and proprietary systems. It also reports that removing either the data selection or the multi-round curriculum lowers performance, and that mixing all domains beats training on any single domain.","pith_inferences":["We infer that the data-curation recipe, not RL alone, drives most of the reported gains: the no-data-selection ablation loses roughly 4-6 points on MathVista, MathVision, and LogicVista, a larger drop than removing multi-round training.","We infer that strict contamination checks on the released data, comparing known training sources like ChartQA, VQA-RAD, SLAKE, and Path-VQA against the evaluation sets, would clarify how much of the SOTA gap reflects genuine generalization rather than near-duplicate exposure.","We infer that the pipeline is likely transferable to other backbones and modalities such as video or 3D perception, but the paper only tests one 7B backbone, so that transfer remains untested."],"forward_implications":["A 7B-scale VLM trained with this recipe can beat much larger open models and some proprietary systems on several visual-reasoning benchmarks, so parameter count is not the only route to strong multimodal reasoning.","Curating data across many domains protects against overfitting to a single domain; the paper shows that single-domain training, especially medical-only, underperforms mixed training even on that same domain.","Influence-function filtering and difficulty-based selection are each necessary: removing either one lowers scores on the reported ablation benchmarks.","Reasoning skills transfer across domains, since the model improves on logical reasoning benchmarks without being explicitly trained on logical-reasoning datasets.","The public release of the model, code, and training dataset enables independent reproduction and extension of the pipeline."],"supporting_citations":[{"why":"Establishes the reinforcement-learning-from-verifiable-rewards paradigm that motivates the training approach.","marker":"[1]"},{"why":"Provides the Qwen2.5-VL base model whose weights initialize Vision-G1 and whose vision encoder is frozen during RL.","marker":"[9]"},{"why":"Contributes the MM-R1 and VerMulti training sources for STEM and geometry problems.","marker":"[14]"},{"why":"Contributes the MMK12 K-12 exam data source and serves as the MM-Eureka baseline.","marker":"[15]"},{"why":"Contributes the ThinkLite-VL data source and the thinking-prompt format used to unify questions.","marker":"[20]"},{"why":"Contributes the ViRL39K training source and the VL-Rethinker baseline.","marker":"[21]"},{"why":"Supplies the LESS gradient-similarity data-selection method that the influence-function filtering adapts.","marker":"[28]"},{"why":"Supplies the influence-function estimation formula behind Equation (1).","marker":"[29]"},{"why":"Provides the GRPO algorithm used for multi-round reinforcement learning.","marker":"[35]"}],"fun_headline_variants":["7B VLM beats GPT-4o with curated data and RL","Influence-based curation lifts small model to top","Multi-domain data plus RL pushes 7B past giants","Vision-G1: 46 datasets, RL curriculum, beats bigger","How a 7B model used data curation to outdo GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported benchmark gains rest on the assumption that evaluation questions were not seen, in identical or near-identical form, during RL training; the paper does not report deduplication or holdout of evaluation splits.","fun_headline_variants_meta":{"raw":{"variants":["7B VLM beats GPT-4o with curated data and RL","Influence-based curation lifts small model to top","Multi-domain data plus RL pushes 7B past giants","Vision-G1: 46 datasets, RL curriculum, beats bigger","How a 7B model used data curation to outdo GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1260,"prompt_tokens":911,"completion_tokens":349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":260}},"tokens_in":527,"tokens_out":349,"duration_ms":3921,"temperature":1.0,"reasoning_tokens":260,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:19:16.548222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released training set against the evaluation benchmarks with near-duplicate detection, then retrain on a strictly deduplicated subset; if the reported gains shrink or disappear, the central claim of generalization would be falsified.","supporting_citations":[],"review_version":2}