Pith. sign in

REVIEW 2 major objections 4 minor 33 references

No evaluated medical vision-language model produces a heatmap that is both image-using and causally aligned with patch occlusion; chest X-ray classifiers pass the same protocol.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:57 UTC pith:VCAFVAQJ

load-bearing objection A rigorous audit of VLM heatmaps on chest X-rays, but the headline 'attention anti-correlates with cause' may be an artifact of single-patch occlusion. the 2 major comments →

arxiv 2607.18577 v1 pith:VCAFVAQJ submitted 2026-07-20 cs.CV

Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs

Binesh Sadanandan , Vahid Behzadan This is my paper
classification cs.CV
keywords visual explanationattention faithfulnessmedical vision-language modelschest X-raypatch occlusioncausal perturbationsaliency mapsgrounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether attention heatmaps shown by medical vision-language models on chest X-rays point to the image evidence that actually drives each answer, and it answers no for every model evaluated. Faithfulness is defined by two joint conditions: the model must use the image at all, and its attention must concentrate on regions whose occlusion changes the prediction. Using 16x16 patch-occlusion causal maps on PadChest, the paper finds negative attention-causal correlations for MedGemma variants and Qwen3-VL, while LLaVA-RAD's positive correlation comes from almost no image use. Two chest-X-ray classifiers pass the same three-axis protocol with p<1e-4, showing the failures are specific to the VLM heatmaps rather than the evaluation. The practical conclusion is that visual inspection of such heatmaps is not a reliable way to certify clinical explanation.

Core claim

Every evaluated medical vision-language model fails a two-part faithfulness test: the model must use the image, and its attention must concentrate on regions whose occlusion changes the answer. Patch-occlusion causal maps give negative Spearman correlations with raw attention for all three MedGemma-4B configurations (rho from -0.050 to -0.168, bootstrap CIs excluding zero) and for Qwen3-VL-8B (rho = -0.117). LLaVA-RAD's positive rho (+0.137) comes from near-total image neglect: 99.1% text-only agreement and causal attribution mass 0.005, so both signals are near zero. No VLM method places more than 22% of attribution mass inside radiologist masks, while DenseNet121 and ResNet50 classifier co

What carries the argument

The central object is a 16x16 patch-occlusion causal map. For each of the 256 image patches, the model's answer margin is recomputed with that patch mean-filled to gray; the map records the margin shift, where a positive shift means hiding that patch drops the margin. The Spearman correlation between raw attention and this map, together with attribution mass inside radiologist masks and overlap against shifted or random bounding boxes, forms the three-axis evaluation. The causal map is the ground truth against which attention is judged.

Load-bearing premise

The entire negative result depends on treating a 16x16 patch-occlusion map with mid-gray mean-fill as the true causal importance of image regions; if occlusion artifacts disturb VLMs more than classifiers, negative correlations could reflect sensitivity to gray patches rather than unfaithful attention.

What would settle it

Run the same protocol on any medical VLM with dense segmentations: the claim is falsified if a configuration produces true-region overlap above both shifted and random controls, AMiM above 0.22, and a positive attention-causal Spearman rho with 95% CI excluding zero while causal AMiM is non-trivial (>0.05). Replacing the mid-gray fill with Gaussian-blur or inpainting and obtaining positive rho would likewise undermine the causal-map interpretation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Clinical explanations generated by these VLMs should not be accepted on the basis of visual inspection; controlled overlap, mask-based, and causal-perturbation metrics are needed before using a heatmap to support a diagnosis.
  • Fine-tuning that improves paraphrase consistency does not produce grounded attention and can actually reduce image use, so consistency is not a proxy for grounding.
  • Because CXR-trained classifiers pass the identical protocol, the negative result is attributable to the VLM heatmaps rather than to the evaluation design.
  • CheXagent's dataset-dependent grounding shows that a model's causal grounding on one annotation style does not generalize, so a single positive heatmap-causal alignment on one dataset is not evidence of general grounding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: if the negative attention-causal correlation generalizes beyond the evaluated families, the visual features actually driving a medical VLM answer are not encoded in the cross-modal attention weights that heatmaps usually display; locating evidence will require gradient- or perturbation-based probes, not attention readout.
  • Our inference: the near-text-only failure of LLaVA-RAD suggests that consistency-oriented fine-tuning can inadvertently reward image neglect, so benchmarks that score only text-side robustness may miss the failure mode; a useful extension is to couple paraphrase-consistency scores with a causal image-use metric.
  • Our inference: a direct test of the paper's mechanism is to fine-tune a VLM with a loss that rewards attention mass inside occlusion-causal regions, then run the same protocol; a flipped positive rho would show the failure is a training objective issue, not an architectural necessity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper audits the faithfulness of attention/saliency heatmaps in medical vision-language models on chest X-rays. It evaluates three MedGemma-4B configurations, cross-family probes LLaVA-RAD and Qwen3-VL-8B, the specialist CheXagent, and two CXR-trained classifiers as positive controls. Three validation axes are used: bounding-box overlap on PadChest (n=637), pixel-mask attribution on CheXlocalize (n=643), and 16×16 patch-occlusion causal maps with 256 forward passes per case. The paper reports that no evaluated VLM is faithful on both required criteria: MedGemma variants and Qwen3-VL use the image but their attention negatively correlates with patch-occlusion causal importance (ρ from −0.050 to −0.168, 95% bootstrap CIs below zero); LLaVA-RAD's positive ρ is tied to near-zero causal mass (0.005); true-region overlap never beats shifted/random controls; and CheXagent is causally grounded only on VinDr-CXR, not on PadChest. The CXR-trained classifiers pass the localization controls, leading to the conclusion that these heatmaps are reassuring but not faithful.

Significance. This is a timely and potentially important negative result for interpretability claims in medical VLMs, provided the causal probe is valid. The evaluation is unusually broad: three independent validation axes, six attribution methods, controlled baselines on bounding boxes and shuffled masks, 95% bootstrap CIs, permutation tests with Holm-Bonferroni correction, replication on VinDr-CXR, a fill-sensitivity check, and public code and protocol. However, the causal ground truth based on single 16×16 patch occlusion is load-bearing. If that probe systematically penalizes broad attention on spatially extended lesions, the negative attention–causal correlations would be an artifact of probe scale rather than evidence of unfaithful attention. This concern is fixable with additional scale-controlled experiments, but without them the central conclusion is under-supported.

major comments (2)
  1. [§3.2 / Table 3] The causal map is constructed by occluding one 16×16 patch at a time and measuring δ_{r,c}=m_orig−m_occluded. For a spatially extended finding, occluding any single patch leaves the rest of the lesion visible, so the margin shift is mechanically small; attention that correctly covers the whole lesion is therefore assigned low causal importance. This redundancy mismatch can induce a negative attention–causal ρ even for a faithful heatmap. The fill-sensitivity check changes the fill value but not the occlusion scale, and the whole-mask ROI deletion test is applied only to CheXagent, not to the MedGemma/Qwen3 models that produce the negative ρ. Please add scale-controlled causal tests for all VLMs (e.g., multi-patch contiguous occlusion with a radius sweep, or whole-radiologist-mask occlusion) and report whether the negative ρ persists. Without this, the central claim may be an artifact of
  2. [Table 3 / §4.2] The CXR-trained classifier 'positive controls' are not evaluated on the same patch-occlusion causal Spearman ρ. Table 2 shows Grad-CAM on pointing game, AMiM, and pixel-AP, and DenseNet is tested on insertion/deletion AUC, but no classifier attention–causal correlation appears in Table 3 or Fig. 3. Since the abstract claims the classifiers 'pass all metrics' and that the failure is specific to VLM heatmaps rather than the metric, the same causal protocol should be run on DenseNet121 and ResNet50 Grad-CAM. This is particularly important because classifier Grad-CAM maps are sharper; the observed contrast may reflect breadth of the saliency map rather than grounding.
minor comments (4)
  1. [§3.2] The protocol text says 'each of the 256 patches' but Qwen3-VL is evaluated on a native 8×8 grid (Table 3). Specify which grid applies to which model in the protocol description.
  2. [Eq. (1) / Table 3] Define the margin m explicitly and clarify why Table 3 (top) uses the signed δ_{r,c} while Table 3 (bottom) uses |δ_{r,c}|; readers currently have to infer this from the text.
  3. [§5] The sentence 'A fill-sensitivity check yields no positive ρ under any fill, so the reported value is a lower bound' is unclear. If no fill yields a positive ρ, specify which fills were tested and why the reported value is a 'lower bound' in magnitude.
  4. [§1 / §3.1] The 99.1% text-only agreement for LLaVA-RAD is attributed to companion work. State explicitly whether this quantity is measured here or imported from [17], and if imported, provide the measurement details or confidence interval.

Circularity Check

0 steps flagged

No significant circularity; self-citations are contextual, not load-bearing.

full rationale

The central claims are measurement results, not reductions. The headline result — negative Spearman rho between raw attention and the patch-occlusion causal map for MedGemma variants (−0.098, −0.050, −0.168) and Qwen3-VL (−0.117), with bootstrap CIs below zero — is a direct comparison between two independently computed maps: the model's attention over image tokens and the margin-shift map δ_{r,c} = m_orig − m_occluded (§3.2). The faithfulness criterion ('attention concentrates on regions whose occlusion alters the prediction') is a stipulated operational definition; the paper's contribution is the measurement that VLMs fail it, not a derivation of the criterion from the conclusion. No fitted parameter is renamed as a prediction: the LoRA adapters come from prior work [18], but the evaluated metrics are computed on PadChest, CheXlocalize, and VinDr-CXR evaluation sets, and the base MedGemma model (no LoRA) already shows the negative rho, so the checkpoint reuse is not load-bearing. The self-citations [17,18,19] are contextual: [17] supplies text-only-agreement percentages used as motivation/interpretation, [18] supplies the checkpoint and layer selection, and [19] is background on calibration; none is invoked to establish the central negative correlation. The admitted 16×16 patch-grid limitation is a construct-validity caveat about the causal probe, not a circular step, and the paper includes fill-sensitivity and blur controls. There is no equation of the form 'prediction ≡ input', no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The score of 2 reflects the presence of self-citations, but they are minor and non-load-bearing; no significant circularity is present.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central claim rests on two main assumptions: (1) radiologist annotations define the clinically relevant image evidence, and (2) patch-occlusion with margin shift is a valid causal probe. No new entities are introduced. The free parameters listed are evaluation protocol choices, not fitted to the data, but they are hand-selected and could influence the absolute values of the metrics.

free parameters (4)
  • Attention threshold (90th percentile) = 0.90
    Hand-chosen threshold for binarizing attention when computing overlap with bounding boxes; not swept, but the paper does report a layer sweep.
  • Attention layer range (10-20) = layers 10-20
    Manual selection of layers to average attention over; the paper reports a nine-group layer sweep that preserves the pattern, so this is not a critical fitted value.
  • Patch size for occlusion (16x16 grid) = 256 patches
    Chosen to match the VLM token grid; sub-patch pathologies may be partially masked, acknowledged in the limitations.
  • Occlusion fill value (mid-gray) = 128
    Selected for occlusion; the paper reports a fill-sensitivity check showing no positive correlation under any fill.
axioms (3)
  • domain assumption Radiologist bounding boxes and pixel masks are ground truth for clinically relevant findings.
    Used to define 'true regions' for overlap and AMiM metrics; standard in the interpretability literature, but it presumes annotations capture all relevant evidence.
  • domain assumption Patch occlusion (with margin shift) is a valid causal counterfactual for the model's prediction.
    The paper defines faithfulness operationally via δ_{r,c}=m_orig−m_occluded; this is the load-bearing definition of 'causal importance.'
  • standard math Spearman correlation between attention and occlusion map is a valid measure of attention-causal alignment.
    A chosen statistical measure; its validity depends on the rank relationship being meaningful, which is a reasonable assumption but not proven.

pith-pipeline@v1.3.0-alltime-deepseek · 128 in / 5844 out tokens · 120473 ms · 2026-08-01T14:57:28.780935+00:00 · methodology

0 comments
read the original abstract

Attention and saliency heatmaps are widely used to explain medical Vision-Language Model (VLM) outputs on chest X-rays, yet whether they truly highlight the image evidence driving predictions has not been causally tested. We audit faithfulness via overlap with radiologist bounding boxes on PadChest (n=637), attribution mass within radiologist masks on CheXlocalize (n=643), and 16x16 patch-occlusion maps that record which regions, when hidden, change the answer. We study three MedGemma-4B variants, cross-family probes on LLaVA-RAD and Qwen3-VL-8B-Instruct, and the specialist CheXagent-2-3b, with two CXR-trained classifiers (DenseNet121, ResNet50) as positive controls. A heatmap is faithful only if the model uses the image and attention concentrates on regions whose occlusion alters the prediction. No evaluated VLM meets both criteria. MedGemma and Qwen3-VL use the image, but attention anti-correlates with patch-occlusion importance (rho < 0 with 95% bootstrap CIs below zero). LLaVA-RAD's attention correlates positively, but the model is almost text-only (99.1% text-only agreement, near-zero causal mass), so correlation ties two near-zero signals. Attention also misses annotated anatomy: overlap with true regions never beats shifted or random controls, and no method places more than 22% of its mass inside radiologist masks. The two CXR classifiers pass all metrics, indicating the failure is specific to VLM heatmaps, not the evaluation. These heatmaps are visually reassuring but not faithful; clinical explanations require controlled localization metrics and causal perturbation, not visual inspection alone.

Figures

Figures reproduced from arXiv: 2607.18577 by Binesh Sadanandan, Vahid Behzadan.

Figure 1
Figure 1. Figure 1: Explanation-validation protocol. A saliency map (pipeline, top) is tested along three independent axes: (i) controlled overlap against shifted, random-far, and random￾size bounding boxes on PadChest; (ii) pixel-mask metrics (pointing-game, attribution mass in mask, pixel-AP) against radiologist segmentations on CheXlocalize; (iii) causal perturbation via 16×16 patch occlusion (256 forward passes per case).… view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative attribution comparison on three CheXlocalize cases. Columns show the CXR, radiologist mask, MedGemma-4B base methods, DenseNet-CXR Grad-CAM, and Qwen3-VL raw attention. MedGemma maps are broad or near-empty and Qwen3- VL is nearly uniform; DenseNet-CXR is the only method whose peak reliably falls near the mask. CXR-specialist VLM (CheXagent): grounding is dataset-dependent. CheXagent-2-3b [29] … view at source ↗
Figure 3
Figure 3. Figure 3: Grounding needs both image use (horizontal: causal attribution mass) and attention-causal alignment (vertical: Spearman ρ). Points are exact [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 11 linked inside Pith

  1. [1]

    arXiv:2403.08002 (2024)

    Zambrano Chaves, J.M., et al.: Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation. arXiv:2403.08002 (2024)

    Pith/arXiv arXiv 2024

  2. [2]

    In: ICLR (2022)

    Hu, E.J., et al.: LoRA: low-rank adaptation of large language models. In: ICLR (2022)

    2022

  3. [3]

    In: ACL, pp

    Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: behavioral testing of NLP models with CheckList. In: ACL, pp. 4902–4912 (2020)

    2020

  4. [4]

    Bustos, A., Pertusa, A., Salinas, J.-M., de la Iglesia-Vayá, M.: PadChest: a large chest X-ray image dataset with multi-label annotated reports. Med. Image Anal.66, 101797 (2020)

    2020

  5. [5]

    Johnson, A.E.W., et al.: MIMIC-CXR, a de-identified publicly available database of chest ra- diographs with free-text reports. Sci. Data6(1), 317 (2019) 10 B. Sadanandan and V. Behzadan

    2019

  6. [6]

    Lieberum, T., et al.: Gemma Scope: open sparse autoencoders everywhere all at once on Gemma

  7. [7]

    arXiv:2408.05147 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    In: CVPR, pp

    Chefer, H., Gur, S., Wolf, L.: Transformer interpretability beyond attention visualization. In: CVPR, pp. 782–791 (2021)

    2021

  9. [9]

    In: NAACL, pp

    Jain, S., Wallace, B.C.: Attention is not explanation. In: NAACL, pp. 3543–3556 (2019)

    2019

  10. [10]

    In: EMNLP, pp

    Wiegreffe, S., Pinter, Y.: Attention is not not explanation. In: EMNLP, pp. 11–20 (2019)

    2019

  11. [11]

    arXiv:2406.06007 (2024)

    Xia, P., et al.: CARES: a comprehensive benchmark of trustworthiness in medical vision lan- guage models. arXiv:2406.06007 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    arXiv:2407.02730 (2024)

    Gu, Z., Yin, C., Liu, F., Zhang, P.: MedVH: towards systematic evaluation of hallucination for large vision language models in the medical context. arXiv:2407.02730 (2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Saporta, A., et al.: Benchmarking saliency methods for chest X-ray interpretation. Nat. Mach. Intell.4(10), 867–878 (2022)

    2022

  14. [14]

    In: ACL, pp

    Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. In: ACL, pp. 4190–4197 (2020)

    2020

  15. [15]

    In: MICCAI (2025)

    Wan, X., et al.: Eliminating language bias for medical visual question answering with counter- factual contrastive training. In: MICCAI (2025)

    2025

  16. [16]

    medRxiv 2023.12.21.23300146 (2023)

    Han, T., Adams, L.C., Nebelung, S., et al.: Multimodal large language models are generalist medical image interpreters. medRxiv 2023.12.21.23300146 (2023)

    2023

  17. [17]

    arXiv:2510.10011 (2025)

    Chen,Y.,Xu,D.,Huang,Y.,etal.:MIMO:amedicalvisionlanguagemodelwithvisualreferring multimodal input and pixel grounding multimodal output. arXiv:2510.10011 (2025)

    arXiv 2025

  18. [18]

    arXiv:2602.21428 (2026)

    Sadanandan, B., Behzadan, V.: PSF-Med: measuring and explaining paraphrase sensitivity in medical vision language models. arXiv:2602.21428 (2026)

    Pith/arXiv arXiv 2026

  19. [19]

    In: Proceedings of Machine Learning Research (CHIL 2026), vol

    Sadanandan, B., Behzadan, V.: Mechanistically guided LoRA improves paraphrase consistency in medical vision-language models. In: Proceedings of Machine Learning Research (CHIL 2026), vol. 333, pp. 703–720 (2026)

    2026

  20. [20]

    arXiv:2604.08941 (2026)

    Sadanandan, B., Behzadan, V.: Predictive entropy links calibration and paraphrase sensitivity in medical vision-language models. arXiv:2604.08941 (2026)

    Pith/arXiv arXiv 2026

  21. [21]

    In: Clinical NLP Workshop, NAACL, pp

    Alsentzer, E., et al.: Publicly available clinical BERT embeddings. In: Clinical NLP Workshop, NAACL, pp. 72–78 (2019)

    2019

  22. [22]

    arXiv:2507.05201 (2025)

    Sellergren, A., Kazemzadeh, S., Jaroensri, T., et al.: MedGemma technical report. arXiv:2507.05201 (2025)

    Pith/arXiv arXiv 2025

  23. [23]

    In: ICCV, pp

    Selvaraju, R.R., et al.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: ICCV, pp. 618–626 (2017)

    2017

  24. [24]

    In: MIDL (2022)

    Cohen, J.P., Viviano, J.D., Bertin, P., et al.: TorchXRayVision: a library of chest X-ray datasets and models. In: MIDL (2022)

    2022

  25. [25]

    arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., et al.: Qwen3-VL technical report. arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  26. [26]

    In: BMVC (2018)

    Petsiuk, V., Das, A., Saenko, K.: RISE: randomized input sampling for explanation of black-box models. In: BMVC (2018)

    2018

  27. [27]

    In: NeurIPS, vol

    Adebayo, J., et al.: Sanity checks for saliency maps. In: NeurIPS, vol. 31 (2018)

    2018

  28. [28]

    In: ICML, pp

    Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: ICML, pp. 3319–3328 (2017)

    2017

  29. [29]

    Nguyen, H.Q., et al.: VinDr-CXR: an open dataset of chest X-rays with radiologist’s annota- tions. Sci. Data9(1), 429 (2022)

    2022

  30. [30]

    arXiv:2401.12208 (2024)

    Chen, Z., Varma, M., Delbrouck, J.-B., et al.: CheXagent: towards a foundation model for chest X-ray interpretation. arXiv:2401.12208 (2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Lan, Z., Sun, L., Walter, M.R., Zhou, J.: Seeing without looking: do vision-language benchmarks really test vision? arXiv:2605.22903 (2026)

    Pith/arXiv arXiv 2026

  32. [32]

    arXiv:2603.22623 (2026)

    Aranya, O.F.M.R.R., Desai, K.: To agree or to be right? the grounding-sycophancy tradeoff in medical vision-language models. arXiv:2603.22623 (2026)

    arXiv 2026

  33. [33]

    arXiv:2512.04238 (2025)

    Mayer, L., et al.: 6 fingers, 1 kidney: natural adversarial medical images reveal critical weak- nesses of vision-language models. arXiv:2512.04238 (2025)

    Pith/arXiv arXiv 2025