{"id":"edac2d46-9f8d-491b-a72e-bf7e1d3ed48e","arxiv_id":"2608.08021","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CED adds a counterfactual evidence margin to GRPO so that correct answers are rewarded only when their likelihood drops more when the evidence region is neutralized than when matched non-evidence regions are neutralized.","lead":"This paper introduces a training-time reward, Counterfactual Evidence Disentanglement (CED), that scores whether a vision-language model's answer depends on a specific image region rather than language priors or distractor context. The authors combine this evidence margin with GRPO and report gains across nine benchmarks and four backbones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evidence Region selection may be anchored to question-specific target/argument boxes in COCO metadata, making the 'weak proposals' / 'no question-specific annotations' claim unsupported and the transfer result conditional on that hidden supervision.","rationale":"The paper has real strengths: Table 3 is a controlled comparison, cross-backbone results are consistent, and the same-answer-different-reward diagnostic supports the claim that CED adds within-group signal. The formal evidence-closed analysis is coherent but close to a definitional statement and is not where the argument is most vulnerable. The weakest point is the construction of Ωev. The paper's own text says the box 'may or may not contain evidence depending on the question,' and the appendix's priority list includes target/argument bbox fields. If those fields are question-specific, then the method is not using weak proposals in the claimed sense; it is using a hidden pointer from the question to the evidence object. This does not invalidate the mechanism, but it changes what the benchmark results demonstrate: they would show that training with object-localized supervision helps, not that CED is a scalable evidence audit with weak proposals. The proposed detector-based rerun settles this directly. I therefore keep the reader's CONDITIONAL verdict; no further adjustment is needed, but the condition should explicitly include this experiment.","tokens_in":20712,"tokens_out":9973,"duration_ms":108982,"concrete_test":"Request the training-data generation code or metadata schema and check whether target_bbox/argument_bbox are set to the COCO object referenced by the question (and how often they are non-null). Then rerun the Table 3 controlled comparison (Qwen3.5-9B, 15,314 samples, Answer-CED) with Ωev taken from a generic detector such as GroundingDINO queried with the question's category nouns, and with no metadata bbox fields. If the Answer-CED minus correctness-only gain drops by more than half (from +12.58) or loses significance, the method depends on question-specific evidence boxes and the 'weak proposals' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines Ωev as 'a weak spatial proposal resolved from dataset metadata' and the abstract claims CED 'requires no question-specific evidence annotations.' However, Appendix A.2 gives the resolution priority as proposal_bbox > target_bbox > argument_bbox > metadata.bbox > center-box fallback. If target_bbox/argument_bbox are populated from the same COCO annotation that the question asks about, then the Evidence Region is not a generic object proposal but the answer-relevant object for that question. The reward then teaches the model to attend to a metadata-supplied target object, which is a much stronger, question-specific form of supervision. The random-object-box ablation (margin 0.268→0.015) is consistent with this: it shows the signal tracks the metadata-selected box, not that it would survive replacement by off-the-shelf proposals without question-specific fields. Because the main empirical claim is that CED transfers using only weak proposals, this unstated dependency is load-bearing: if the priority fields are question-specific, the benchmark gains may reflect object-localized supervision rather than a general evidence audit, and the claimed advantage over region-annotation-requiring methods is overstated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Counterfactual Evidence Disentanglement (CED), a training-time reward component for post-training vision-language models. For each sampled answer, CED compares the drop in answer log-likelihood when an object-centric Evidence Region's visual tokens are mean-replaced with the corresponding drop for matched non-evidence regions, yielding a contrastive evidence margin. This margin is combined multiplicatively with answer correctness inside GRPO, so that equally correct rollouts are ranked by visual-evidence dependence. The authors claim that CED requires only weak object-level proposals, needs no question-specific evidence annotations, adds no inference-time overhead, and improves performance across nine public benchmarks and four backbones. The paper includes signal-validity diagnostics, a controlled comparison on Qwen3.5-9B, cross-backbone results, and robustness ablations under proposal and intervention perturbations.","tokens_in":20962,"tokens_out":6839,"duration_ms":70440,"significance":"If the claims hold, CED would be a practical and appealing training signal: it injects a local, answer-conditioned visual-dependence audit into RL post-training without inference cost and without dense region annotations. The paper has several genuine strengths: the controlled comparison in Table 3 isolates the CED contribution from data, router, and compute; the proposal-robustness ablations in Figure 5 and Appendix A.7 probe the mechanism rather than only reporting aggregate accuracy; the cross-backbone validation in Table 4 shows consistent positive gains; and the appendix provides unusually detailed hyperparameter and implementation documentation. The main source of significance risk is the unresolved relationship between the 'weak proposal' claim and the actual proposal-resolution procedure, which appears to use question-specific target metadata. The empirical gains are conditional on this procedure, and the central novelty claim rests on resolving it. The missing uncertainty estimates for the headline 7B result further weaken the strength of the transfer claim as currently stated.","major_comments":[{"comment":"The claim that CED 'uses weak object-level proposals' and 'requires no question-specific evidence annotations' is contradicted by the proposal-resolution procedure described in Appendix A.2: the Evidence Region Ωev is 'resolved from sample metadata using the priority proposal_bbox > target_bbox > argument_bbox > metadata.bbox > center-box fallback.' Since the training data are auto-generated from COCO val2017, the target_bbox and argument_bbox fields are almost certainly the bounding boxes of the objects named by the question; under that reading, Ωev is not a generic weak proposal but the answer-relevant object for each question, which is question-specific supervision. The random-object-box ablation in Figure 5(a) is consistent with this reading: the signal tracks the metadata-selected box, not a generic proposal. This is load-bearing because the claimed advantage over methods that require region annotations depends on CED operating with weak, annotation-free proposals. Please clarify what populates proposal_bbox, target_bbox, and argument_bbox; if target_bbox/argument_bbox encode the objects referenced by the question, revise the 'weak proposal' claims and the comparison in Table A.2, and provide an experiment with an off-the-shelf object detector (e.g., Detic or GroundingDINO) in which the Evidence Region is selected without using the question's target object identity.","section":"Section 3.1 and Appendix A.2"},{"comment":"The headline nine-benchmark result for Qwen2.5-VL-7B is reported as single-run numbers with no error bars or significance tests, yet it supports strong claims ('the only RL method with non-negative Δ on all nine benchmarks', '+5.94 average improvement'). Because the controlled comparison in Table 3 shows that some baselines are highly unstable on this protocol (e.g., PAPO collapsing on Qwen3.5-9B), single-run point estimates in Table 2 are not sufficient to establish the transfer claim. The paper already provides three-seed standard deviations for Qwen3.5-9B in Table 4; please provide at least three seeds for the Qwen2.5-VL-7B block of Table 2, or a justified statement of why the main comparison does not require them.","section":"Table 2"},{"comment":"The sentence 'after within-group normalization Ai = (Rtrain,i − μ)/σ, correctness remains the dominant gradient driver' is not guaranteed by the reward definition. Since m ∈ [−1,1] and g(m) ∈ (0,1), a wrong answer with m close to +1 has Rtrain ≈ 0 + 0.1 = 0.1, while a correct answer with m close to −1 has Rtrain ≈ 0 − 0.1 = −0.1; after group normalization, the wrong answer can receive a positive advantage over the correct one. The 'correctness-anchored' property needs a formal condition—for example, a proof that for all m the raw reward of any correct answer exceeds that of any wrong answer—or the reward should be modified, e.g., Rtrain = Rans · g(m) with ε_tie scaled so that this ordering holds. As written, the method may reward incorrect but evidence-sensitive rollouts, contradicting the mechanism claimed in Section 4.3.1.","section":"Section 3.2, Eq. (4)"},{"comment":"The formalization of 'evidence-closed self-evolution' does not exactly match the informal statement. Section 5 states that a text-only reward satisfies St ⊥⊥ It | (Yhat_t, Qt) and therefore I(St; Gt | Yhat_t, Qt) = 0. Lemma 1 in Section A.12 instead proves a conditional independence under the larger conditioning set (Yhat_t, Qt, Ft), where Ft is the filtration; Proposition 1 then proves I(T_cl^t; Gt | Bt) = 0 for a Bt-measurable signal. The conditioning sets differ, and the stronger informal claim in Section 5 is not established by the appendix. Proposition 2 is close to tautological and does not sharpen the grounding-identifiability statement. Please restate the formal results so that the conditioning sets and assumptions align with the claims made in Section 5.","section":"Section A.12 and Section 5"}],"minor_comments":[{"comment":"There is a dangling comma in the sentence 'the gains transfer across backbones,;' which should be corrected.","section":"Section 4"},{"comment":"The baseline VLM-R1 is compared against Qwen2.5-VL-3B while all other RL baselines use Qwen2.5-VL-7B; the caption should state the base model for each row explicitly so that the Δ values are not misinterpreted as comparable across rows.","section":"Table 2"},{"comment":"The figure caption says 'with the random-region floor at 0.147' but the text in Section 4.3.3 refers to a random floor of 0.147; the values are consistent, but the caption should define whether the floor is computed on the same count-exclusion samples or is a separate aggregate.","section":"Figure 5"},{"comment":"The blank-image ScienceQA discussion is interesting but does not quantify how many items in the ScienceQA evaluation have blank images or how much the 'Cannot determine from image' behavior affects the net gain; please add a count or a sensitivity analysis.","section":"Section 5"},{"comment":"The notation St is used both for the reward and for the evidence sensitivity s(Ω) in Section 3.1; using different symbols would avoid confusion.","section":"Appendix A.12"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the gap between the 'weak proposals / no question-specific annotations' claim and the Appendix A.2 resolution priority involving target_bbox and argument_bbox. If these fields encode the objects being asked about, the method effectively uses question-specific target-object boxes during training, which is a much stronger form of supervision than advertised. This affects the framing of the novelty and the comparison with region-supervised methods. The paper would need either a convincing clarification or an additional experiment with a truly annotation-free proposal source. I do not think the paper should be rejected outright—the controlled experiments and ablations are valuable—but the claim structure needs significant revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the read. The core idea is genuinely new as far as the cited literature goes: CED scores each sampled answer by measuring the log-likelihood drop when the answer-relevant region is neutralized against matched non-evidence regions, and turns that contrast into a GRPO reward. The controlled Table 3 comparison is the strongest part of the paper — same backbone, same data, same compute, with correctness-only as a baseline — and the proposal-robustness ablations (IoU degradation, random-box collapse) show the signal isn't just masking magnitude. The Answer-vs-CoT analysis, with the chain collapse into object-cue shorthand, is a thoughtful warning about dense process rewards. Those are real contributions.\n\nThe soft spot is the proposal story. The abstract claims CED requires no question-specific evidence annotations and uses weak object-level proposals, but Appendix A.2 gives the priority proposal_bbox > target_bbox > argument_bbox > metadata.bbox. If the training questions are generated from COCO annotations — likely, given the attribute/counting/spatial task families — then target_bbox and argument_bbox are the very objects the question asks about. That would make the Evidence Region question-specific supervision, not a weak proposal. The random-box ablation (margin 0.268 to 0.015, relevant-beats-random dropping to chance) is consistent with the signal tracking a metadata-selected box and says nothing about whether an off-the-shelf detector would survive. The paper would need a detector-based proposal experiment, or a clear description of how the boxes are decoupled from question generation, to support the headline claim. As it stands, the benchmark gains may reflect object-localized supervision rather than a general evidence audit. That's load-bearing, because the method's advantage over region-annotation-requiring approaches is at stake.\n\nMinor issues: Table 2 has no error bars for the main 7B result, and no code is released. The formal evidence-closed result is correct but close to a definitional statement.\n\nWho this is for: anyone working on grounding-aware RL for VLMs. The contrastive margin idea is worth pursuing even if the proposal pipeline needs to change. My call: it deserves peer review, with a referee who will push hard on the data-generation pipeline and the proposal dependency. It should not be desk rejected, but it needs substantial revision before acceptance.","headline":"A well-ablated training-time evidence-audit reward, but the 'no question-specific annotations' claim is undermined by the proposal priority chain and needs a hard look before the benchmark gains can be taken at face value.","tokens_in":21517,"tokens_out":4022,"would_cite":false,"duration_ms":41165,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewarding answers that depend on the relevant pixels lifts nine vision-language benchmarks.","keywords":["vision-language models","visual grounding","counterfactual intervention","reinforcement learning","GRPO","evidence disentanglement","post-training","object proposals"],"falsifier":"Retrain the full pipeline with the Evidence Region replaced by a random object box (or by proposals that systematically miss the question-relevant object) and compare benchmark deltas: if the reported gains survive or the evidence margin does not collapse to chance (roughly 0.015 with a relevant-beats-random rate near 0.49 on their count-exclusion diagnostic), then the claimed mechanism is not the cause of the improvement.","tokens_in":20487,"feed_emoji":"👁️","tokens_out":8327,"duration_ms":74210,"temperature":0.7,"pith_summary":"The paper claims that vision-language models can be post-trained to answer from the concrete visual evidence that supports an answer, rather than from language priors or irrelevant context, by auditing each sampled response with a counterfactual intervention. Its central proposal, Counterfactual Evidence Disentanglement (CED), measures how much a candidate answer's log-likelihood drops when an object-centric Evidence Region is neutralized, relative to matched non-evidence regions, and folds that margin into the GRPO reward together with answer correctness. The authors argue that this is the missing training signal: a text-only reward cannot distinguish a correct answer caused by the image from one caused by a shortcut, a regime they call evidence-closed self-evolution. They report that the resulting pipeline, EVIDENCE-RL, improves over prior RL-based post-training methods on nine public benchmarks across four backbones, with the largest gains where language priors conflict with the image.","feed_headline":"Evidence-aware rewards lift VLM grounding on nine benchmarks","feed_subtitle":"It rewards answers that depend on the relevant pixels, not on language priors, and this transfers across four backbones.","key_machinery":"The central object is the Counterfactual Evidence Disentanglement (CED) margin, a contrastive counterfactual likelihood drop. For a sampled answer y, CED neutralizes the visual tokens of an Evidence Region Ωev by mean replacement, records s(Ωev)=log πθ(y|I,q)−log πθ(y|I\\Ωev,q), does the same to K matched non-evidence regions, and computes m=tanh((s(Ωev)−μ(snon))/(σ(snon)+ε)). The non-evidence regions act as a sample-local null for masking artifacts and irrelevant sensitivity, and the margin is gated by correctness in the GRPO reward Rtrain=Rans·g(m)+εtie·m. This makes the reward answer-specific, evidence-localized, and robust to generic perturbation sensitivity.","core_discovery":"On the paper's own terms, the central discovery is a specification of what 'grounded' should mean for reward design and an implementable training signal for it. A candidate answer y to an image-question pair is evidence-grounded when its support drops more when the proposed Evidence Region is neutralized than when area-matched non-evidence regions are neutralized; the contrastive margin m(I,q,y) = tanh((s(Ωev)−μ(snon))/(σ(snon)+ε)), with s(Ω) the counterfactual log-likelihood drop under feature-space mean replacement, is then combined with answer correctness inside GRPO so that equally correct rollouts are ranked by evidence dependence. The paper claims this turns the causal insight—that the same observed answer can arise from the evidence path, a language-prior shortcut path, or an irrelevant-context nuisance path—into a training objective, and that doing so transfers to nine benchmarks and four backbones without question-specific region annotations or inference-time overhead.","pith_inferences":["A testable extension: the same contrastive counterfactual recipe should transfer to attribute-level or relational evidence by changing the intervention, a direction the paper sketches but does not implement.","Because the evidence margin depends on proposal quality, deployment on domains without good object proposals would likely need a learned or question-conditioned proposer; the paper's own random-box ablation shows why that matters.","The framework predicts that training gains should concentrate on items where a strong language prior conflicts with the image, and that error analyses should show the model moving from prior-consistent to evidence-consistent answers.","Since CED adds no inference-time overhead, its practical ceiling is set by training-time proposal coverage rather than by compute at deployment."],"forward_implications":["A correct answer that would survive removal of the evidence region is rewarded less than a correct answer that collapses when the evidence is removed, so GRPO preferentially amplifies visually grounded trajectories.","Text-only self-evolution rewards are formally unable to separate grounded from shortcut trajectories (evidence-closed self-evolution); an image-conditional signal is structurally necessary, and CED supplies one.","Answer-level CED avoids the chain-length reward hacking observed with dense per-token evidence rewards, which collapsed CoT-CED's chains to roughly 3.6 tokens on counting tasks.","The same recipe transfers: positive mean gains on all four backbones and non-negative gains on all 36 benchmark–backbone cells, with no degradation beyond 1.5 percentage points on eight text-only benchmarks."],"supporting_citations":[{"why":"Supplies GRPO, the group-relative policy optimization used to combine correctness with the CED evidence margin.","marker":"[Shao et al., 2024]"},{"why":"Supplies the COCO object-level annotations from which the weak Evidence Region proposals are resolved.","marker":"[Lin et al., 2014]"},{"why":"The global image-perturbation reward baseline that CED contrasts against and improves on.","marker":"[Wang et al., 2025d]"},{"why":"The attention-based visual-dependency baseline showing that internal routing proxies differ from causal evidence dependence.","marker":"[Huang et al., 2025]"},{"why":"Provides the structural-causal view in which evidence, shortcut, and nuisance paths converge on the same observed answer.","marker":"[Pearl, 2009]"},{"why":"Provides the FREAK benchmark whose prior-conflict items expose language-prior shortcuts and where CED gains are largest.","marker":"[Yin et al., 2026]"},{"why":"Provides VLMsAreBlind, a perception-heavy benchmark used to measure fine-grained visual grounding transfer.","marker":"[Rahmanzadehgervi et al., 2024]"},{"why":"Provides HallusionBench, used to evaluate entanglement of language hallucination and visual illusion.","marker":"[Guan et al., 2024]"},{"why":"Documents the process-reward hacking failure that motivates Answer-CED over CoT-CED.","marker":"[Guo et al., 2025]"},{"why":"Provides the Qwen2.5-VL backbone family on which the main head-to-head comparison is run.","marker":"[Bai et al., 2025b]"}],"fun_headline_variants":["Counterfactual audit grounds VLM answers in image evidence","Reward evidence, not priors: CED sharpens VLM grounding","CED: Training-time check that answers depend on pixels","VLM grounding via counterfactual evidence disentanglement","Neutralize regions to teach VLMs to reason from evidence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on the COCO-derived object box chosen as the Evidence Region actually containing the evidence that supports the correct answer; the paper itself notes the same box may or may not contain evidence depending on the question, and replacing it with a random object box collapses the evidence margin from 0.268 to 0.015.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual audit grounds VLM answers in image evidence","Reward evidence, not priors: CED sharpens VLM grounding","CED: Training-time check that answers depend on pixels","VLM grounding via counterfactual evidence disentanglement","Neutralize regions to teach VLMs to reason from evidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1524,"prompt_tokens":901,"completion_tokens":623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":539}},"tokens_in":517,"tokens_out":623,"duration_ms":6175,"temperature":1.0,"reasoning_tokens":539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:32:57.152195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the full pipeline with the Evidence Region replaced by a random object box (or by proposals that systematically miss the question-relevant object) and compare benchmark deltas: if the reported gains survive or the evidence margin does not collapse to chance (roughly 0.015 with a relevant-beats-random rate near 0.49 on their count-exclusion diagnostic), then the claimed mechanism is not the cause of the improvement.","supporting_citations":[],"review_version":1}