{"id":"194368e2-848b-498b-832d-85b0fca9dfc9","arxiv_id":"2506.22434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A self-supervised contrastive triplets plus weak-to-strong augmented GRPO training method improves multi-image reasoning in Qwen2.5-VL-7B.","lead":"Researchers trained a vision-language model to compare images using automatically generated same/different examples, with no hand-written questions and answers. The trained model then improved on multi-image reasoning benchmarks and slightly on general vision tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's Augmented GRPO objective is not a valid policy-gradient estimator: rollouts are sampled and rewarded under weak prompts while the importance ratio is evaluated on strong prompts, so the reported gains may not arise from the claimed mechanism.","rationale":"The paper's central claim is that a self-supervised same/different objective with Augmented GRPO transfers to broad visual reasoning. The most load-bearing condition is that the training objective actually optimizes the policy on the strong-augmentation prompts that the model will later face. Algorithm 1 violates this condition: rollouts are drawn from and rewarded under weak prompts, yet the policy ratio is evaluated on strong prompts. This is a formal, internal inconsistency, not a matter of consensus, and it is directly testable by re-derivation and a small-scale ablation. The reader's weakest assumption about augmentation corrupting the ground truth is also real, but it is a data-quality risk that might be mitigated by filtering or milder augmentation; the estimator flaw affects the core methodological novelty irrespective of data quality. If the controlled test shows the biased estimator is responsible for the Weak-Strong advantage in Table 2(c), then the paper overstates its claimed mechanism, and the conditional verdict should stand or be tightened; if the test shows the bias is benign, the concern clears. Either way, the test settles the question. The reader already flagged the invalid estimator in the rationale, so this stress-test agrees with the overall conditional verdict but prioritizes a different concern than the reader's stated weakest assumption.","tokens_in":17597,"tokens_out":9160,"duration_ms":107461,"concrete_test":"Derive the expected gradient of Eq. 1; show it is not equal to ∇_θ E_{o~π_θold(·|q_s)}[A(q_s)] unless q_w = q_s. Then run a controlled comparison over the first 100 training steps with three variants: (a) the paper's Algorithm 1 as written; (b) rollouts and rewards on q_s (standard GRPO on strong prompts); (c) an importance-corrected objective using ratio π_θ(o|q_s)/π_θold(o|q_w) with rewards on q_w. If (a) matches (b) in final VLM2-Bench accuracy, the bias is benign; if (a) differs substantially from both (b) and (c), the reported gains depend on the invalid estimator and the paper's description of Augmented GRPO is not what was implemented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Algorithm 1, trajectories o_i are sampled from π_θold(·|q_w) (line 7) and rewarded on the weak prompt (line 8), but the loss in line 11 uses the ratio π_θ(o_i|q_s)/π_θold(o_i|q_s) for the strong prompt. Standard PPO/GRPO requires rollouts from the same state that appears in the ratio. Without a correction such as π_θ(o|q_s)/π_θold(o|q_w), the expectation is taken over q_w while the ratio is for q_s, so the objective does not estimate the policy gradient under q_s. The supplementary importance-sampling attempt (Section B.2) reweights advantages rather than correcting the ratio and is reported as unhelpful, but that does not repair the estimator. The consequence is that advantages computed on weak prompts can increase the likelihood of responses that are inappropriate under strong prompts, so the model may learn to shortcut the strong-prompt content. Table 2(c) then cannot separate the intended curriculum from this bias, and the paper's central methodological claim—'optimize the policy on stronger augmented contexts'—is not supported by the written objective. Since code and data are not released, the empirical results cannot confirm which objective was actually optimized. This is an internal inconsistency in the method description, independent of how the augmentation assumption (the reader's weakest assumption) resolves.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MiCo, a self-supervised reinforcement-learning framework for multi-image reasoning in VLMs. Training triplets are built from two augmented views of the same image plus a visually similar but distinct image, sourced from video frames and image editing datasets. The VLM is prompted to output a chain of thought and answer same/different questions, with rule-based rewards. The authors introduce \"Augmented GRPO,\" in which rollouts are sampled under weak augmentations and the policy is optimized on strong augmentations. Experiments on VLM2-Bench report a +12.93 average improvement over Qwen2.5-VL-7B, surpassing GPT-4o, and additional gains on several multi-image and single-image benchmarks.","tokens_in":17934,"tokens_out":3893,"duration_ms":45250,"significance":"The work addresses an important problem: improving cross-image reasoning without human-annotated question-answer pairs. The central idea of using inherent image constraints as reward supervision is attractive and potentially scalable. The paper also merits credit for systematic ablations (Table 2) and for evaluating on a range of benchmarks, including task-level analyses. If the method is valid, MiCo would be a promising annotation-free route to multi-image reasoning. However, the central algorithmic contribution, Augmented GRPO, is described with a mathematically invalid objective, which undermines the claimed mechanism and the interpretation of the key ablation. The paper needs a major revision to clarify or correct the training objective before its empirical claims can be fully credited.","major_comments":[{"comment":"The Augmented GRPO objective as written is not a valid policy-gradient estimator. In Algorithm 1, rollouts o_i are sampled from π_θold(·|q_w) and rewarded on q_w, but the loss in line 11 uses the importance ratio π_θ(o_i|q_s)/π_θold(o_i|q_s) for the strong prompt q_s. Standard PPO/GRPO requires the rollouts to be sampled from the same distribution that appears in the ratio. Without a correction such as π_θ(o|q_s)/π_θold(o|q_w), the expectation is over q_w while the ratio is for q_s, so the gradient estimator is not justified. The importance-sampling attempt in Appendix B.2 reweights advantages instead of correcting the sampling distribution and is reported as unhelpful, which does not repair the derivation. Consequently, the central claim that the policy is \"optimized on stronger augmented contexts\" is not supported by the written objective, and Table 2(c) cannot separate the intended curriculum from this bias. Since code and data are not released, the empirical results cannot confirm which objective was actually optimized. Please either provide a correct derivation (e.g., a valid off-policy correction or a properly formulated weak-to-strong objective) or explicitly present the loss as a heuristic weighted regression and provide empirical justification for that heuristic without invoking policy-gradient estimation.","section":"Algorithm 1, Eq. (1), Section 3.3"},{"comment":"The method assumes that random cropping and resizing preserve the subtle visual differences that determine same/different, so that the rule-based reward is correct under strong augmentation. This assumption is load-bearing but untested. If a strong crop removes the only discriminative region, the reward labels the model's answer as incorrect even though the images are visually indistinguishable, corrupting the training signal. The ablations in Table 2(f) compare augmentation types on benchmark performance but do not measure label-preservation rates or reward noise under strong augmentation. Please include a diagnostic such as human agreement, a vision model's comparison accuracy, or a quantitative measure of discriminative-detail preservation under the chosen augmentation distributions.","section":"Section 3.2, Figure 2, Algorithm 1 lines 4-5"},{"comment":"Even if the objective were corrected, the comparison (Weak, Strong) versus (Strong, Strong) versus (Weak, Weak) confounds rollout quality, reward noise, and optimization difficulty. Strong augmentation may yield many zero-reward rollouts, while weak augmentation yields higher-quality but potentially less challenging trajectories. The current experiment does not isolate the curriculum effect from these confounds. An additional control, such as using weak rollouts with a properly reweighted objective on weak prompts, or filtering/re-labeling strong rollouts, would be needed to support the claim that the gains arise from the proposed weak-to-strong transfer.","section":"Table 2(c), Section 4.3"}],"minor_comments":[{"comment":"Typo: \"reasnoning prompt\" should be \"reasoning prompt\".","section":"Section 3.3"},{"comment":"The displayed objective is malformed: the expectation is written without an integrand, and the notation mixes q with q_s/q_w. Please rewrite the equation to match the algorithm.","section":"Eq. (1)"},{"comment":"Typo: \"image edting data\" should be \"image editing data\".","section":"Appendix A"},{"comment":"Typo: \"successfuly\" should be \"successfully\".","section":"Section 4.5"},{"comment":"The notation T^w(I_1,I_2,I_3) is defined earlier as T = {T_1(I_a), T_2(I_a), T_3(I_b)}; please clarify the relationship between the triplet notation and the per-image augmentation operators.","section":"Algorithm 1"},{"comment":"The paper says GPT-4o is used to expand user questions but does not provide the expanded prompt set; please release the prompt templates or include them in an appendix for reproducibility.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are interesting and the annotation-free training data is a valuable contribution. However, the invalid objective in Algorithm 1/Eq. (1) is a load-bearing issue: the paper's main methodological novelty is described in a mathematically incorrect way. I believe this is fixable within the scope of a revision if the authors can either provide a correct derivation or reframe the method as a heuristic and defend it empirically with additional controls. If the authors cannot clarify the objective, the empirical claims would need to be reinterpreted substantially. I also note that code is not released, which is particularly problematic here because the written algorithm is not implementable as a standard GRPO variant."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a genuinely new way to get supervision for multi-image reasoning—contrastive triplets lifted from video frames and image edits, with rule-based RL rewarding same/different judgments—and it reports a big jump on VLM2-Bench (+12.9 average over Qwen2.5-VL-7B). But the Augmented GRPO objective in Algorithm 1 is not a valid estimator as written, and that is a load-bearing problem, not a typo.\n\nThe novelty is real. NoisyRollout mixes noisy rollouts and optimizes on clean images; MiCo instead samples on weak augmentations and claims to optimize on strong ones. That is a different idea. The ablations are better than most: they test learning paradigm, data source, sample formulation, prompt diversity, and augmentation type, and they report unsuccessful attempts (confidence reweighting, importance sampling) honestly. The limitations section is candid about face verification and visual math.\n\nThe soft spot is the objective. In Algorithm 1, trajectories come from π_θold(·|q_w) and are rewarded on q_w, but the importance ratio in line 11 is π_θ(o_i|q_s)/π_θold(o_i|q_s). That ratio is for a distribution that did not produce the rollouts. Standard GRPO/PPO needs the ratio to be between the policy and the sampling distribution for the same state. The supplementary importance-sampling experiment reweights advantages, which is a different correction and doesn't repair the ratio. So as written, the loss does not estimate the policy gradient for q_s, and the (Weak, Strong) advantage in Table 2(c) cannot be attributed to the claimed mechanism. The authors may have implemented something else, but they don't release code or data, so that can't be checked. That is the main reason I'm not taking the empirical results at face value.\n\nThe weaker assumption—that random cropping preserves the discriminative detail—is plausible and worth testing, but it's minor compared to the estimator issue. Generalization gains on other benchmarks are real but modest, and some sub-tasks drop; the abstract overstates 'strong performance' but the limitations section is honest.\n\nBottom line: the paper deserves a serious referee. The idea is novel and the reported gains are large enough to matter if they hold up. But it needs a major revision: fix or clarify the objective, release code/data, and add an analysis that actually supports the weak-to-strong transfer. I'd put it in a reading group to work through the off-policy issue; I wouldn't cite it in its current form.","headline":"Genuinely new idea for annotation-free multi-image reasoning supervision, with large reported gains, but the Augmented GRPO objective as written is not a valid estimator and the empirical claims rest on an unreleased implementation.","tokens_in":18442,"tokens_out":3799,"would_cite":false,"duration_ms":38579,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MiCo shows that a 7B vision-language model trained only on same/different image triplets with rule-based reinforcement learning outperforms GPT-4o on VLM2-Bench without any human-annotated QA pairs.","keywords":["multi-image reasoning","vision-language models","reinforcement learning","chain-of-thought","self-supervised learning","visual comparison","rule-based RL","data augmentation"],"falsifier":"Take MiCo's strongly augmented training triplets and have human annotators judge same/different on each; if a substantial fraction (say, more than 10%) are judged visually indeterminate, then a portion of the reward signal is wrong, and training should degrade as that fraction grows. A cleaner test would be to train two copies of MiCo, one with strong augmentations that always crop inside the discriminative region and one that never does, and compare their VLM2-Bench scores.","tokens_in":17445,"feed_emoji":"🖼️","tokens_out":11185,"duration_ms":100822,"temperature":0.7,"pith_summary":"This paper tries to show that the main thing vision-language models lack for multi-image understanding is a general visual-comparison skill, and that this skill can be trained from images alone, without human-written question-answer pairs. The method builds training triplets from two augmented views of the same image plus a similar but distinct third image, asks the model to reason about which views are the same, and optimizes with rule-based reinforcement learning under a weak-to-strong augmentation scheme. The reported result is that this comparison-only training transfers to a wide range of multi-image and general vision benchmarks, including a new state of the art on VLM2-Bench with a 7B model that surpasses GPT-4o. If correct, it offers an annotation-free route to multi-image reasoning and sharpens the claim that visual comparison is the core bottleneck.","feed_headline":"Comparison-only RL training beats GPT-4o on multi-image reasoning","feed_subtitle":"A 7B VLM taught only to compare image triplets with rule-based RL reaches new state of the art on VLM2-Bench.","key_machinery":"The central object is the contrastive triplet, a triple of images $T = \\{T_1(I_a), T_2(I_a), T_3(I_b)\\}$ in which the first two are different random crops of the same image and the third is a visually similar but distinct image, each augmented by cropping and resizing. The mechanism that carries the argument is Augmented GRPO: chain-of-thought rollouts are sampled under weak augmentation, reward is computed by a rule-based check of format and of the correct same/different labels for all queried pairs, and the policy is updated under a stronger augmentation of the same triplets. This weak-to-strong optimization forces the model to produce reliable comparisons that survive harder visual transforms, and the contrastive structure discourages shortcut learning by making the discrimination depend on fine detail. Data sources are filtered so negatives are subtle: video frames separated by about two seconds with SSIM-based removal of near-identical pairs, and image-editing before/after pairs with pixel-level filtering of overly large differences.","core_discovery":"On the paper's own terms, the discovery is that visual comparison can be treated as a self-supervised meta-skill: by constructing contrastive triplets from video frames and image-editing pairs, and by using rule-based reinforcement learning with group-relative advantages, a vision-language model can be incentivized to produce chain-of-thought comparisons and attend to subtle cross-image differences. The paper reports an average of 61.06 on VLM2-Bench for MiCo-7B-CoT, an improvement of +12.93 over the Qwen2.5-VL-7B baseline, surpassing GPT-4o's 60.36. Generalization is documented on MuirBench, BLINK, MMStar, MMMU, HallusionBench, and MathVista, with the largest gains on correspondence-style tasks such as visual retrieval and semantic correspondence. The paper attributes the transfer to the learned comparison ability rather than to task-specific supervision, and it notes that face verification, visual math, and relative depth remain weak points.","pith_inferences":["If visual comparison is truly the bottleneck skill, then scaling triplet difficulty and diversity—harder negatives, longer temporal gaps, more editing operations—should keep lifting multi-image benchmarks, which the paper's data-source ablation suggests.","The weak-to-strong rollout trick is a general curriculum recipe: sample reasoning traces where the reward is easy to achieve, then reinforce them under harder conditions; it could be applied to other self-supervised reinforcement learning problems beyond same/different comparison.","The face-verification failure suggests that language-based chain-of-thought may not carry purely perceptual identity information; a parallel non-verbal objective, such as a learned similarity head, might complement chain-of-thought rather than replace it.","A direct stress test of the core assumption would be to replace random cropping with region-dropping augmentations and measure whether the reward remains correct; the paper does not report this ablation."],"forward_implications":["A 7B vision-language model trained only on comparison triplets can surpass closed-source models on VLM2-Bench, showing that annotation-free rule-based RL scales multi-image reasoning.","The learned comparison skill transfers without retraining to tasks such as object matching, tracking, semantic correspondence, spatial relations, and some single-image reasoning, which the paper documents across six benchmarks.","Combining two heterogeneous contrastive data sources (video frames and image-editing pairs) is better than either alone, and the method generalizes across different editing datasets.","Tasks that resist verbal description, such as face identity verification, relative depth, and visual math, are not fixed by this training, marking the boundary of the claimed skill."],"supporting_citations":[{"why":"supplies the base vision-language model (Qwen2.5-VL-7B) whose perception and commonsense the method builds on and that serves as the primary comparison baseline.","marker":"[2]"},{"why":"provides the rule-based reinforcement learning with group-relative advantages that Augmented GRPO extends by applying weak augmentation to rollouts and strong augmentation to optimization.","marker":"[28]"},{"why":"defines the main evaluation benchmark (VLM2-Bench) with the fine-grained cross-image comparison tasks where MiCo reports its state-of-the-art results.","marker":"[40]"},{"why":"supplies before/after edited image pairs from OmniEdit used as visually similar negative samples in the contrastive triplets.","marker":"[35]"},{"why":"supplies video frames from VidGen-1M used to sample similar-but-distinct pairs with a two-second temporal gap and SSIM filtering.","marker":"[30]"},{"why":"used as a high-capability closed-source baseline and to expand prompt phrasing through model-generated variants of the same user question.","marker":"[15]"},{"why":"a prior augmentation-based GRPO method that MiCo contrasts with to position its weak-to-strong rollout strategy as different from mixing noisy and clean trajectories.","marker":"[20]"}],"fun_headline_variants":["Self-supervised RL makes 7B VLM beat GPT-4o on multi-image tasks","Teaching VLMs to compare images with RL lifts reasoning to new SOTA","No QA pairs needed: contrastive RL training boosts VLM reasoning","Visual comparison as meta-skill: RL-trained VLM beats GPT-4o","MiCo: RL on image triplets yields SOTA multi-image reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Random cropping and resizing are assumed to preserve the subtle difference that determines whether two images are the same, so the rule-based reward labels remain correct even after strong augmentation; if a strong crop removes the only distinguishing detail, the model is rewarded for an impossible discrimination.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised RL makes 7B VLM beat GPT-4o on multi-image tasks","Teaching VLMs to compare images with RL lifts reasoning to new SOTA","No QA pairs needed: contrastive RL training boosts VLM reasoning","Visual comparison as meta-skill: RL-trained VLM beats GPT-4o","MiCo: RL on image triplets yields SOTA multi-image reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001162,"raw_usage":{"total_tokens":4823,"prompt_tokens":971,"completion_tokens":3852,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":3750}},"tokens_in":587,"tokens_out":3852,"duration_ms":25846,"temperature":1.0,"reasoning_tokens":3750,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:04:31.396643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take MiCo's strongly augmented training triplets and have human annotators judge same/different on each; if a substantial fraction (say, more than 10%) are judged visually indeterminate, then a portion of the reward signal is wrong, and training should degrade as that fraction grows. A cleaner test would be to train two copies of MiCo, one with strong augmentations that always crop inside the discriminative region and one that never does, and compare their VLM2-Bench scores.","supporting_citations":[{"cited_title":"Omniedit: Building image editing generalist models through specialist supervision","cited_arxiv_id":null,"evidence_quote":"supplies before/after edited image pairs from OmniEdit used as visually similar negative samples in the contrastive triplets."}],"review_version":1}