{"id":"9d8108b0-b138-4e9a-a9e0-aac402e30aba","arxiv_id":"2602.18746","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MIRROR trains VLMs to close the loop between drafting an answer, reflecting, visually marking image regions, and revising, reducing hallucinations on several VQA benchmarks.","lead":"This paper introduces MIRROR, a training framework that makes vision-language models verify their own answers by looking again at marked image regions before revising. It adds a new reflective dataset and reports gains on VQA and hallucination benchmarks, though the evaluation leaves open questions about contamination and reproducibility.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReflectV is built from several evaluation benchmarks without any stated split-exclusion; reported gains on TextVQA, ChartQA, and OCRBench may be inflated by train/test overlap.","rationale":"The paper's central claim is that closed-loop visual reflection improves VLM correctness and reduces hallucinations. The load-bearing premise is that ReflectV's construction does not overlap with the evaluation benchmarks, because the dataset is explicitly curated from public benchmarks whose evaluation splits are used in Section 5. The manuscript never asserts split disjointness, and without released artifacts there is no way to verify it. This is the most direct threat to the headline claim: if overlap exists, the reported improvements over Qwen2.5-VL and SOTA baselines are partly memorization, not evidence for the proposed mechanism. I agree with the reader's weakest_assumption and would not relax the conditional verdict. The paper does have independent support in the form of controlled ablations (MIRROR vs MIRROR-QA, MIRROR vs w/o tool) and detailed qualitative trajectory analysis, which suggest the mechanism itself is coherent; the concern is specifically about the external benchmark numbers, not the internal consistency of the method. The fix is concrete: release the dataset metadata and report overlap counts, or re-evaluate after excluding any overlapping samples. If the authors provide this, the conditional verdict can be revisited.","tokens_in":28382,"tokens_out":4219,"duration_ms":43047,"concrete_test":"Release ReflectV item metadata (image IDs/URLs, question text, source benchmark and split). Compute exact overlap between ReflectV and each evaluation split used in Tables 1-3: TextVQA-Val, ChartQA-Test, OCRBench, POPE, HallusionBench, and any benchmark whose images derive from COCO/LLaVA/GQA/TextVQA/OCRVQA. For every overlapping item, determine whether the trajectory was generated from the same question-answer pair. Then retrain MIRROR and MIRROR-QA after excluding all overlapping samples and re-run Tables 1-2. If the gains versus Qwen2.5-VL shrink substantially while the MIRROR-vs-w/o-tool gap persists, the external claim is inflated by contamination; if the gains persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MIRROR improves correctness and reduces hallucinations depends on ReflectV being disjoint from every evaluation set. Appendix A.1 lists COCO, LLaVA, GQA, TextVQA, OCRVQA, DOCVQA, DVQA, and ChartQA as data sources, while Section 5 evaluates on TextVQA, ChartQA, OCRBench, POPE, HallusionBench, and other benchmarks whose images or questions overlap with these sources. The paper never states that evaluation splits were excluded during dataset construction, nor does it report any overlap statistics. Because the multi-agent pipeline uses ground-truth annotations to synthesize teacher feedback and final answers, even a small number of overlapping samples could teach the model the exact test answer, materially inflating the reported gains (e.g., +8.8 on OCRBench, +13.36 on HallusionBench). Without released ReflectV metadata or an explicit disjointness statement, the headline external numbers cannot be interpreted as evidence for the proposed mechanism. The internal ablation comparing MIRROR to MIRROR-w/o-tool is less affected by this issue because both variants train on the same data, but the paper's central external claim against base models remains unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIRROR, a framework that equips a VLM with a closed-loop visual reflection process: after an initial answer, the model writes a reflective critique, optionally invokes a visual prompt generator (grounding text anchors with Molmo-7B and overlaying markers with SAM 2), and revises the answer based on the marked image. To train this behavior, the authors construct ReflectV, a ~24k-sample dataset built from public VQA/OCR/chart sources via a GPT-4V student–teacher pipeline, followed by filtering, keyword-to-region grounding, and rewriting teacher feedback as first-person self-reflection. They fine-tune Qwen2.5-VL-3B/7B with LoRA and report improvements over base models and prior reflection/'thinking-with-images' methods on general, OCR, hallucination, fine-grained perception, and math benchmarks.","tokens_in":28694,"tokens_out":4894,"duration_ms":44807,"significance":"If the empirical claims hold, MIRROR provides a reusable recipe for training VLMs to ground their self-corrections in visual evidence. The internal ablations are well designed: comparing MIRROR to a tool-free variant, to QA-only training, and across trajectory ratios isolates the proposed mechanism, and the data filtering analysis supports the dataset-quality argument. The ReflectV dataset itself could be a valuable community resource. However, the headline external numbers are not currently trustworthy because ReflectV is built from the same benchmark families used for evaluation, with no stated exclusion of evaluation splits and no overlap statistics, and the key trajectory ratio ρ=0.75 is selected on the evaluation metric itself. These issues must be resolved before the central claim that visual grounding, rather than textual reflection, drives the gains can be accepted.","major_comments":[{"comment":"ReflectV is constructed from COCO, LLaVA, GQA, TextVQA, OCRVQA, DOCVQA, DVQA, and ChartQA, while Section 5 evaluates on TextVQA, ChartQA, OCRBench, and related benchmarks. The manuscript never states that evaluation splits were excluded during dataset construction, nor does it report overlap statistics. Since the training pipeline uses ground-truth annotations to synthesize teacher feedback and final answers (Section 4.1), even a small number of overlapping test examples could teach the model the exact answer, materially inflating reported gains such as +8.8 on OCRBench and +13.36 on HallusionBench. Without a clear disjointness statement, per-source overlap numbers, or release of ReflectV metadata, the external benchmark results cannot be interpreted as evidence for the proposed mechanism. The internal ablation comparing MIRROR and MIRROR-w/o-tool is less affected because both variants t","section":"Appendix A.1 vs. Section 5, Tables 1-2"},{"comment":"The trajectory ratio ρ=0.75 is selected as the configuration that maximizes MM-Vet in Table 5 and is then reported as the final model. There is no held-out validation split or a pre-specified selection criterion; this constitutes test-set tuning. The claim that the balanced strategy is optimal cannot be supported by the same table that was used to choose it. Please either fix ρ based on a held-out set or present the results as a model-selection curve with a separate validation evaluation.","section":"Section 5.3.3, Table 5"},{"comment":"All experiments report single numbers without error bars, repeated runs, or significance tests. Some differences are small relative to likely run-to-run variability (e.g., SeedBench-2-Plus: MIRROR-w/o-tool 70.36 vs. Qwen2.5-VL-7B 70.88; TextVQA: 85.37 vs. 84.90). Since the central claim is that MIRROR improves correctness and reduces hallucinations, the reader needs confidence intervals or at least multiple seeds for the main comparisons. This is especially important for hallucination benchmarks such as POPE and HallusionBench, where evaluation protocols and judge variability can dominate small point estimates.","section":"Tables 1-5"},{"comment":"The comparison against VL-Rethinker, PixelReasoner, DeepEyes, and Adaptive-CoF states that all methods are fine-tuned on Qwen2.5-VL-7B, but it is not clear whether the reported numbers are taken from the original papers or re-run under the same VLMEvalKit version, decoding settings, and visual-tool environment. Differences in evaluation protocol can easily produce several-point swings (e.g., +3.90 on OCRBench, +5.12 on POPE). Please clarify the source of these numbers and, if possible, re-run the baselines under identical conditions.","section":"Section 5.2, Table 3"}],"minor_comments":[{"comment":"The MIRROR-Raw row in Table 7 reports an overall MM-Vet score of 55.87, which is exactly the same as the ρ=0.5 row in Table 5. Please check whether this is a copy-paste error or a genuine coincidence; if genuine, the individual component scores should be re-verified.","section":"Table 7 vs. Table 5"},{"comment":"Training details are incomplete. Please report learning rate, batch size, sequence length, LoRA dropout, warmup steps, and evaluation decoding parameters. The current text gives only LoRA rank/alpha, epochs, and hardware.","section":"Appendix B.1"},{"comment":"The paper would benefit from explicitly stating whether the baseline numbers in Table 3 are from the original publications or from new runs. This is important for a fair comparison of 'reasoning models' and should be stated in the table caption or the experimental setup.","section":"Section 5.2"},{"comment":"The efficiency comparison reports time and tokens per sample but does not specify whether the numbers include the full MIRROR pipeline (Molmo grounding + SAM 2 segmentation) and what hardware/decoding settings were used. Since MIRROR invokes external tools, a breakdown of where the time is spent would help readers assess the applicability of the efficiency claim.","section":"Appendix D.1, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is plausible and the ablations are thoughtfully constructed, but the current evaluation design is vulnerable to dataset contamination and test-set tuning. I strongly recommend that the editor require the authors to provide (a) a precise overlap analysis between ReflectV source data and every evaluation benchmark, or a re-evaluation on explicitly disjoint splits; (b) a held-out validation procedure for ρ; and (c) confidence intervals for the main reported gains. Without these, the paper's external claims cannot be verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the central idea is good: MIRROR trains a VLM to draft an answer, reflect, call a visual prompt generator that marks regions (via Molmo + SAM 2), and revise from the marked image. That is a clean way to turn reflection into an evidence-seeking loop rather than text-only guesswork. Second, the reported gains over baselines are not secure because ReflectV appears to be built partly from the same benchmark families that the paper evaluates on, and the authors never state that those splits were excluded. The internal ablations are much less affected, but the external claims need verification before this can be cited as a solid result.\n\nWhat is actually new is the integration: multi-turn reflective trajectories with explicit visual tool calls, the 24k-sample ReflectV dataset, and the SFT recipe on Qwen2.5-VL. The ablations are thoughtfully designed. Removing the visual tool drops POPE by 6.47 points and MME-RW by 5.48; training on the same data as single-turn QA (MIRROR-QA) gives less improvement; the data-filtering study shows that noisy trajectories hurt. Those comparisons are informative and support the mechanism. The paper also openly documents failure cases in Appendix E, which is more than most.\n\nThe soft spot is the contamination risk, and it is load-bearing. Appendix A.1 lists COCO, LLaVA, GQA, TextVQA, OCRVQA, DOCVQA, DVQA, and ChartQA as sources. The evaluation in Tables 1–2 uses TextVQA, ChartQA, OCRBench, and related benchmarks. The authors never say that evaluation instances or images were excluded during dataset construction, and no overlap statistics are provided. If overlap is present, the +8.8 on OCRBench and +13.36 on HallusionBench are inflated. This is not a minor caveat; it is the difference between \"the mechanism works\" and \"the model memorized parts of the test set.\"\n\nThere are smaller issues: all tables are single runs with no error bars or significance tests, and rho=0.75 is chosen after looking at MM-Vet and then presented as the optimal setting. Those are minor relative to the contamination question. No code, data, or checkpoints are released, so nothing can be independently checked today.\n\nWho this is for: anyone working on VLM reasoning, self-correction, or hallucination reduction. The idea is worth engaging. The paper deserves a serious referee, but the review should require the authors to release ReflectV metadata, demonstrate disjointness from every evaluation set, and report multi-seed variance. If they can do that, this is a useful contribution. As it stands, I would not cite the headline numbers.","headline":"MIRROR's closed-loop visual reflection is a good idea with solid internal ablations, but the headline gains are unsecured until the ReflectV/evaluation overlap is ruled out.","tokens_in":29155,"tokens_out":2576,"would_cite":false,"duration_ms":24179,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language self-correction only works when the model goes back to the image, marks the disputed region, and revises against that evidence rather than linguistic priors.","keywords":["vision-language models","multimodal hallucination","self-reflection","visual grounding","closed-loop verification","visual prompting","ReflectV","multi-turn reasoning"],"falsifier":"Compare the model's performance on a held-out set of images that were explicitly disjoint from ReflectV, or check the training set for image-level duplicates against the evaluation test splits; if the OCR, chart, and hallucination gains shrink to base-model levels, the reported improvement came from memorization rather than the closed-loop verification process.","tokens_in":28316,"feed_emoji":"🔍","tokens_out":8129,"duration_ms":77283,"temperature":0.7,"pith_summary":"MIRROR tries to establish that reflection in vision-language models fails when it stays purely textual, because the model then 'corrects' itself from language priors rather than from image content. The paper turns reflection into a closed loop: draft an answer, critique it, invoke a visual prompt generator that marks task-relevant regions on the image, and revise with the marked image in view. To train this behavior it builds ReflectV, a dataset of roughly 24,000 multi-turn trajectories with reflection triggers, region-verification actions, and revisions tied to ground truth. The trained model reports higher accuracy and lower hallucination than the base model and than text-reflection and open-loop visual-search baselines, and the paper interprets this as evidence that reflection should be trained as an evidence-seeking process. The paper also states a boundary: in abstract symbolic derivations and fine-grained attribute binding, the visual markers give little usable signal.","feed_headline":"Closed-loop visual reflection cuts vision-language hallucinations","feed_subtitle":"A 7B model that marks image regions before revising beats larger models on OCR, chart, and hallucination tests.","key_machinery":"The load-bearing machinery is a repeated four-stage cycle—draft, critique, region-based verification, revision—driven by a visual prompt generator. The generator grounds a textual anchor from the model's reflection into image coordinates, segments the corresponding object, and overlays a colored marker, producing a new image that the VLM sees at the next turn. A multi-turn autoregressive loss trains the VLM to emit answer, reflection, and tool-call tokens jointly, while the ReflectV dataset supplies supervision for when to invoke the tool and how to revise from the marked evidence. The loop terminates when the model's reflection declares the answer verified.","core_discovery":"The central claim is that a vision-language model's reflection steps should be coupled to a visual grounding tool. When the model suspects an error, it emits a structured request naming the anchor phrase from its reflection; the tool grounds that phrase to coordinates, renders an overlay (point, box, ellipse, or mask), and feeds the marked image back to the model before the next answer. The paper argues this closed-loop 'region-based verification' converts self-correction from open-loop text generation into pixel-level evidence checking, and that this, not the dataset content alone, drives the gains—shown by the tool-free ablation dropping sharply on grounding-heavy tasks. Training the model","pith_inferences":["The strongest unstated check is dataset hygiene: since ReflectV is built from public benchmarks that overlap with the evaluation suite, the reported gains should be re-measured on a provably disjoint held-out set before attributing them to the closed loop.","If the effect survives that check, the framework should compose naturally with reinforcement learning on tool-use reward, where the marker's contribution to the final answer can be measured directly.","The mark-verify-revise recipe generalizes to any perception task in which the model can name its uncertainty as a phrase—counting, referring expressions, spatial reasoning—so the paper's mechanism is likely a scaffold for agentic perception beyond VQA.","The failure cases suggest the next bottleneck is not reflection but the tool's grounding ability; extending the generator from static markers to zoom, crop, or editing could push the same loop into abstract and attribute-heavy domains."],"forward_implications":["Self-correction training for multimodal models should include explicit tool calls and marked-image inputs, not just textual critique and rewrite pairs.","Because the verification is targeted, the loop can be cheaper than verbose text-only reflection, resolving errors in fewer tokens and less time.","The mechanism transfers across model sizes, with a smaller trained model surpassing a larger base model on several benchmarks, suggesting it is a parameter-efficient route to stronger perception.","The benefit is concentrated where errors are spatially localizable; the paper documents that abstract math and compositional attribute binding receive little or no help from the visual markers.","A balanced mix of multi-turn reflective chains and single-turn QA is necessary—too many reflective chains induce a 'failure-first' bias that degrades otherwise correct answers."],"fun_headline_variants":["Visual grounding loop stops VLM hallucinations","Region-checked reflection reduces VLM errors","Closed-loop visual checks beat pure text reflection","MIRROR: visual verification improves VLMs","Pixel-level evidence cuts VLM hallucination"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline gains assume the images used to build ReflectV were not taken from the same splits used for evaluation, but the paper does not state that evaluation images were excluded.","fun_headline_variants_meta":{"raw":{"variants":["Visual grounding loop stops VLM hallucinations","Region-checked reflection reduces VLM errors","Closed-loop visual checks beat pure text reflection","MIRROR: visual verification improves VLMs","Pixel-level evidence cuts VLM hallucination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3098,"prompt_tokens":724,"completion_tokens":2374,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":2321}},"tokens_in":468,"tokens_out":2374,"duration_ms":15128,"temperature":1.0,"reasoning_tokens":2321,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:51:43.614798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the model's performance on a held-out set of images that were explicitly disjoint from ReflectV, or check the training set for image-level duplicates against the evaluation test splits; if the OCR, chart, and hallucination gains shrink to base-model levels, the reported improvement came from memorization rather than the closed-loop verification process.","supporting_citations":[],"review_version":1}