{"id":"00612efb-a1bc-4471-92c7-292f03fc0623","arxiv_id":"2607.03647","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Hard-negative contrastive training (CORAL/CGO) lifts medical VQA accuracy +6.7 pp and cuts visual hallucination rate 8 pp versus the matched 7B base, while image substitution costs far less on medical than general benchmarks.","lead":"Medical vision-language models often answer correctly by exploiting text shortcuts rather than reading the image. The authors introduce counterfactual metrics that expose this gap and a hard-negative contrastive training method (CORAL) that raises accuracy while cutting visual hallucinations.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Train/eval overlap on PMC-VQA (and PathVQA) may inflate CORAL's headline accuracy and VHR gains, so the central claim is not cleanly out-of-distribution.","rationale":"The reader's weakest assumption (CLIP nearest-neighbour hardness) is real but secondary: even imperfect hard negatives can still penalise answer invariance. The more load-bearing threat to the strongest claim is the openly acknowledged train/eval overlap. Because the largest accuracy effect sits on the training-domain benchmark (PMC-VQA) and the macro pools it with the other three, leakage can manufacture both the accuracy and VHR signals without genuine generalisation of visual reliance. The paper already treats the result as conditional; confirming or removing the overlap would either solidify or further qualify that verdict. No change to REJECT is warranted without the overlap numbers, but the claim cannot be treated as cleanly OOD until they are reported.","tokens_in":13419,"tokens_out":587,"duration_ms":5471,"concrete_test":"Compute exact image-hash and (image, question) overlap between the 5,778 CGO training pairs and the 400 Phase-2 eval items (seed 42). Recompute Table 3 macro accuracy and VHR after excluding every overlapping or near-duplicate item (or re-sample a fully disjoint 100-item set per benchmark). If the accuracy CI no longer excludes zero or the VHR reduction falls below ~4 pp, the headline claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim rests on a paired n=400 macro improvement of CORAL over the matched Qwen2.5-VL-7B base (+6.7 pp accuracy, -8.0 pp VHR). CORAL's CGO hard-negative pools (Table 1) are mined from the same benchmark sources used in Phase-2 evaluation, including 2,000 PMC-VQA pairs and 3,362 PathVQA-closed pairs. Table 4 shows the largest accuracy lift is precisely on PMC-VQA (+15 pp, CI excludes zero), the richest training pool, while PathVQA shows an accuracy regression but a large VHR drop. The paper itself flags \"train/eval benchmark overlap\" as a limitation, yet the headline macro still pools these in-domain items with SLAKE/VQA-RAD. If a non-trivial fraction of the 100 fixed eval items per benchmark (seed 42) appear in the 5,778-pair training pool or share near-duplicate images/questions, the reported gains partly measure memorisation of contrastive pairs rather than improved visual reliance on held-out medical VQA. The CLIP-hard-negative assumption is secondary; the primary threat is that the statistical resolution of the central claim is contaminated by leakage.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that medical VLMs often exploit textual shortcuts rather than visual evidence, and proposes (i) a counterfactual evaluation suite that substitutes real images with blank, shuffled, absent, and CLIP hard-negative surrogates, yielding metrics such as VRS and VHR, and (ii) CORAL, a LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that rewards answer change under hard-negative image swaps. On a paired n=400 evaluation across PathVQA, PMC-VQA, SLAKE, and VQA-RAD, CORAL is reported to raise macro accuracy by +6.7 pp and cut VHR by 8.0 pp relative to the matched base, while two MedVLThinker RL checkpoints do not; cross-domain diagnostics further show much smaller image-substitution costs on medical than on general VQA. The authors release framework, code, and weights and openly list train/eval overlap and underpowered secondary metrics as limitations.","tokens_in":13836,"tokens_out":1061,"duration_ms":13245,"significance":"If the accuracy and VHR gains hold under clean held-out evaluation, the work supplies both a practical audit protocol for clinical VLMs and a concrete training-time intervention that targets language-prior shortcuts—directly relevant to reliability and SaMD-style failure-mode analysis. Strengths include carefully paired bootstrap/McNemar inference, an explicit multi-condition metric suite, open discussion of underpowered VRS/BD and format-dependent IS, and planned release of code and weights. The medical-vs-general grounding gap (≤6.5 pp vs 48–61 pp) is a useful diagnostic contribution even independent of CORAL.","major_comments":[{"comment":"Table 1 and §3.1/§3.5: CGO hard-negative pools are mined from the same benchmark sources used in Phase-2 evaluation (2,000 PMC-VQA and 3,362 PathVQA-closed pairs among 5,778 total). Table 4 shows the only accuracy CI that cleanly excludes zero is PMC-VQA (+15 pp), the richest training pool, while PathVQA accuracy regresses. The headline macro in Table 3 and the abstract therefore pools in-domain items with SLAKE/VQA-RAD. Because the paper does not report decontamination (overlap of the seed-42 100-item eval sets with the training pools, or near-duplicate image/question rates), the central +6.7 pp / −8.0 pp claims are not cleanly out-of-distribution. Please either (a) remove overlapping items and re-report macro CIs, or (b) present a fully held-out macro (SLAKE+VQA-RAD only, or an external set) as the primary result and demote the pooled macro.","section":"Table 1, Table 3, Table 4, §3.1, §3.5"},{"comment":"Table 3, ISpred column and §4.1 “IS caveat”: CORAL has the worst (highest) extracted-answer invariance under shuffle (ISpred=0.698 vs base 0.518). The paper’s operational definition of visual reliance (§1, §3.3) is that predictions should change when the image is made uninformative. A model that keeps the same multiple-choice letter ~70% of the time under shuffle is not more visually reliant on this primary behavioural criterion, even if VHR falls. The accuracy/VHR gains can still be real, but the framing that CGO produces a “visually-reliant” model is not supported by ISpred and should be revised; either strengthen the contrastive term (as the authors suggest for future work) until ISpred improves, or restate claims as “reduced VHR / higher accuracy under hard-negative training” without the stronger reliance claim.","section":"Table 3, §4.1, §3.3 Eq. (2)"},{"comment":"§3.1 Eq. (1) and Table 1: Hard negatives are defined as CLIP-ViT-L/14 nearest neighbours with differing ground-truth labels (mean cosine 0.96–0.98). There is no independent check that these pairs are clinically or visually confusable (e.g., human rating, shared pathology labels, or feature-level overlap beyond CLIP). If many “hard” negatives differ only in non-visual metadata or answer format while looking nearly identical, CGO may primarily penalise answer-format invariance rather than force use of diagnostic visual content. A short validation (sample of pairs with clinical feature annotation or human confusability) or an ablation with weaker/random negatives would make the mechanism claim load-bearing rather than assumed.","section":"§3.1 Eq. (1), Table 1"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this paper gives a clean counterfactual protocol for asking whether medical VLMs actually use the image, and a simple hard-negative contrastive objective (CGO) that improves macro accuracy by ~6.7 pp and cuts Visual Hallucination Rate by 8 pp over the matched Qwen2.5-VL-7B base on a paired n=400 design. Neither of the MedVLThinker RL variants does that.\n\nWhat is actually new is the medical packaging, not the high-level idea of image ablations or language-prior diagnostics. They run blank, shuffle, no-image, and CLIP-retrieved label-different hard negatives, define a small suite (VRS, VHR, VBR, IS_pred, etc.), and show the grounding gap is real: image substitution costs ≤6.5 pp on medical benches versus 48–61 pp on general-domain ones. CGO then mines those same CLIP hard-negatives at train time and rewards answer change under the swap. The RL(image) checkpoint peaking on no-image prompts is a useful empirical signature they surface cleanly.\n\nThey do several things well. Stats are paired bootstrap + McNemar, primary claims clear zero while secondary ones (VRS, BD) are flagged as under-powered. Limitations section is unusually frank about train/eval overlap, small n, and the fact that IS_pred is worse for CORAL. Code, framework, and weights are promised. Citation pattern is appropriate; they sit on Agrawal, Bitton-Guetta, Liang, etc. without pretending to invent counterfactuals.\n\nSoft spots in proportion: the stress-test concern lands. Hard-negative pools and Phase-2 eval share PMC-VQA and PathVQA sources; the biggest accuracy lift is exactly on PMC-VQA (+15 pp). That does not make the result fake, but it means the macro number is not cleanly OOD and some of the gain may be contrastive memorisation rather than pure visual reliance. CLIP nearest-neighbours with different labels are assumed hard enough; no human or clinical confusability check. n=100 per bench is thin for secondary metrics. None of this overturns the central accuracy/VHR story or the audit value of the framework.\n\nThis is for people building or auditing medical VLMs and for multimodal safety folks who care about shortcut learning. It deserves a serious referee. I would engage with the framework and the CGO recipe; the overlap issue is fixable with a proper held-out split or external medical set.","headline":"Usable medical-VLM grounding audit plus a contrastive fix that moves accuracy and VHR; train/eval overlap softens the headline numbers but does not erase the contribution.","tokens_in":14419,"tokens_out":631,"would_cite":true,"duration_ms":8559,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Medical vision-language models often answer from text shortcuts; a hard-negative contrastive objective raises accuracy and cuts visual hallucination.","keywords":["medical vision-language models","visual grounding","counterfactual evaluation","hard-negative contrastive learning","Visual Hallucination Rate","medical VQA","language priors"],"falsifier":"Re-run the same paired n=400 evaluation after replacing CLIP hard negatives with images that human clinicians rate as visually confusable on the same clinical feature; if the accuracy and VHR gains disappear, the claim that CGO induces visual reliance fails.","tokens_in":14353,"feed_emoji":"👁️","tokens_out":608,"duration_ms":5130,"temperature":0.7,"pith_summary":"Medical vision-language models can score well on visual question answering while barely looking at the image, relying instead on language patterns in the question. This paper builds a counterfactual test that replaces the real image with blank, shuffled, missing, or confusable substitutes and measures how much the answer actually depends on visual content. It then trains CORAL, a 7-billion-parameter model, with a Contrastive Grounding Objective that rewards the model only when its answer changes under a hard-negative image swap. On four medical VQA benchmarks the method lifts accuracy by 6.7 points and cuts the rate of confidently wrong visual answers by 8 points relative to the matched base model, while two reinforcement-learning variants do not. The work matters because high accuracy alone can hide a model that would fail when the image truly matters for diagnosis.","feed_headline":"Medical VLMs answer from text; hard-negative training cuts the shortcut","feed_subtitle":"CORAL lifts accuracy 6.7 points and halves confident visual errors on four medical VQA sets","key_machinery":"The Contrastive Grounding Objective (CGO): an additive reward term that is positive only when the model’s extracted answer changes between the real image and a CLIP-similar but label-different hard negative, thereby making language-prior shortcuts high-loss at training time.","core_discovery":"Across four closed-form medical VQA benchmarks, a LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective that penalises answer invariance under CLIP-retrieved hard-negative image swaps improves macro accuracy by +6.7 percentage points and reduces Visual Hallucination Rate by 8.0 points over the matched base; neither of two accuracy-only RL checkpoints achieves a significant gain on either metric.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Medical VLMs lean on text; CORAL hard-negatives force true visual use","CORAL CGO training lifts medical VQA accuracy 6.7 pp, cuts VHR 8 pp","Counterfactual swaps expose text shortcuts; CORAL restores visual reliance","Hard-negative swaps train Qwen2.5-VL to ground medical answers in images","Contrastive grounding cuts visual hallucinations 8 pp on four medical VQAs"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The method assumes that CLIP nearest-neighbour images with different labels are hard enough visual confusers to force genuine visual reliance rather than just answer-format sensitivity.","fun_headline_variants_meta":{"raw":{"variants":["Medical VLMs lean on text; CORAL hard-negatives force true visual use","CORAL CGO training lifts medical VQA accuracy 6.7 pp, cuts VHR 8 pp","Counterfactual swaps expose text shortcuts; CORAL restores visual reliance","Hard-negative swaps train Qwen2.5-VL to ground medical answers in images","Contrastive grounding cuts visual hallucinations 8 pp on four medical VQAs"]},"model":"grok-4.5","effort":"low","cost_usd":0.00521,"raw_usage":{"total_tokens":1537,"prompt_tokens":907,"num_sources_used":0,"completion_tokens":112,"cost_in_usd_ticks":52100000,"prompt_tokens_details":{"text_tokens":907,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":518,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":907,"tokens_out":112,"duration_ms":4509,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T00:56:35.039029+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same paired n=400 evaluation after replacing CLIP hard negatives with images that human clinicians rate as visually confusable on the same clinical feature; if the accuracy and VHR gains disappear, the claim that CGO induces visual reliance fails.","supporting_citations":[],"review_version":1}