Pith. sign in

REVIEW 4 major objections 4 minor 64 references

This paper introduces RoentMod, a counterfactual editing tool that adds specified synthetic pathology to real chest X-rays, and uses it to show that state-of-the-art multi-task and foundation models frequently rely on off-target pathology a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 20:15 UTC pith:PIDTYI5E

load-bearing objection RoentMod is a genuinely useful counterfactual CXR editor with credible radiologist validation, but the headline training gains rest on a missing same-split control and the shortcut diagnosis needs a no-op edit control before I'd trust the numbers. the 4 major comments →

arxiv 2509.08640 v1 pith:PIDTYI5E submitted 2025-09-10 eess.IV cs.AIcs.CV

RoentMod: A Synthetic Chest X-Ray Modification Model to Identify and Correct Image Interpretation Model Shortcuts

Lauren H. Cooke , Matthias Jung , Jan M. Brendel , Nora M. Kerkovits , Borek Foldyna , Michael T. Lu , Vineet K. Raghu This is my paper
classification eess.IV cs.AIcs.CV
keywords chest radiographcounterfactual image editingshortcut learningdiffusion modelsfoundation modelsdata augmentationmulti-task learningmodel robustness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that modern chest X-ray interpretation models, including large foundation models, often make predictions using off-target pathologies as shortcuts rather than the specific finding being diagnosed. To test this, the authors built RoentMod, which takes a real X-ray and generates a realistic version of the same scan with one user-specified pathology added, while preserving the rest of the anatomy. They show that adding a single synthetic finding increases model-predicted probabilities for all other tested pathologies, and that including these counterfactual images in training reduces the shortcut and improves AUC across internal and external datasets. If correct, RoentMod offers a generalizable strategy for probing and correcting shortcut learning in medical AI.

Core claim

The central claim is that counterfactual chest X-rays generated by RoentMod provide both a diagnostic test and a corrective training signal for shortcut learning. Applying RoentMod to real scans with no documented finding, the paper shows that adding one pathology (e.g., edema) raises the predicted probability for every other tested pathology (e.g., hernia) in four publicly available multitask and foundation models, indicating that these models use the presence of any pathology as a shortcut. Augmenting a multitask model's training set with RoentMod counterfactual images—labeled with radiologist-read co-occurrence rates for unprompted findings—removes much of this off-target sensitivity and

What carries the argument

RoentMod is a counterfactual image editing pipeline that requires no additional training: it takes the text encoder and denoising U-Net weights from RoentGen, a diffusion model fine-tuned on MIMIC-CXR for chest X-ray generation, and attaches them to the Stable Diffusion Image-to-Image architecture's variational autoencoder. A text prompt such as 'right upper lobe mass' plus two operating parameters (guidance scale 4, strength 0.4) convert a real CXR into a synthetic version with the requested finding. The same tool then serves as a stress test—comparing model predictions between baseline and edited scans—and as a data augmentation source, with unprompted co-occurrences labeled using radiolog

Load-bearing premise

RoentMod's edits are clean interventions: the generated image differs from the original only in the requested pathology (plus clinically sensible co-occurrences), with no subtle generator artifacts that models can exploit; the paper itself notes that invisible changes may affect the accuracy of shortcut testing.

What would settle it

Train a classifier to distinguish RoentMod-edited images from real unedited scans using image content alone; if it achieves high accuracy, then a diagnostic model's different predictions on counterfactual images could be driven by the edit artifact rather than the added pathology, and the shortcut claim would need to be re-tested on pairs of real scans where the same pathology developed naturally.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • RoentMod can quantitatively expose off-target pathology shortcuts in any CXR interpretation model, including foundation models, without retraining or manual heatmap review.
  • Incorporating counterfactual images during training reduces shortcut reliance and improves out-of-distribution generalization, with the largest gains on in-distribution and generator-training cohorts.
  • The training paradigm with single-finding edits can induce reverse shortcut learning (underpredicting co-occurring findings), suggesting that multi-finding counterfactuals or recalibration are needed for full correction.
  • Because RoentMod runs in under two seconds per image on consumer GPUs, the approach can be applied at scale to stress-test and fine-tune models across diverse clinical settings.
  • The same framework could be extended to fairness evaluations and segmentation models by editing demographic or anatomical features rather than pathology alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A reader should treat the shortcut diagnosis as conditional on RoentMod edits being clean interventions: if generated images contain subtle artifacts invisible to radiologists, models may respond to those artifacts rather than to the added pathology, and both the shortcut measurement and the training benefit would be confounded.
  • A natural extension is to verify the mechanism using paired real follow-up X-rays where the same pathology occurred naturally, testing whether models show the same off-target probability increases on real images as on counterfactuals.
  • The reverse-shortcut behavior observed after single-finding training suggests that multi-pathology counterfactual editing, or loss weighting informed by co-occurrence, could further close the gap between synthetic-augmented and fully naturally-trained models.
  • Because the same architecture is modality-agnostic, RoentMod-style editing could be adapted to other imaging modalities where diffusion generators exist, providing a general probe for shortcut learning beyond chest radiography.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces RoentMod, a counterfactual chest X-ray (CXR) editing tool built by combining the RoentGen CXR generator with the Stable Diffusion image-to-image architecture, requiring no additional training. The authors evaluate RoentMod through radiologist reads of 800 synthetic scans, reporting high realism and prompt adherence, and use it to stress-test four public CXR interpretation models. They report that adding a single pathology via RoentMod increases predicted probability percentiles for all other tested pathologies, which they interpret as evidence of shortcut learning. They then train a multi-task DenseNet-121 on real NIH CXR-14 images plus RoentMod-generated counterfactuals and report AUC improvements of 3-19% on internal cohorts and 1-11% on 5/6 pathologies in external cohorts, along with reduced off-target probability shifts on the RoentMod stress test.

Significance. If the central interpretation is correct, this is a practical and broadly applicable contribution: it provides an open, fast, radiologist-validated counterfactual editing tool, a concrete demonstration that multi-task and foundation CXR models use off-target pathology as shortcuts, and a training augmentation scheme that partially corrects this behavior. The use of four public models and four datasets, including external cohorts, strengthens the empirical scope. However, the paper's core claims rest on the assumption that RoentMod edits are clean interventions. The authors explicitly acknowledge in the Discussion that 'subtle changes may be present in counterfactually generated CXRs that are not visible to human readers,' and the manuscript lacks a neutral control edit to quantify what fraction of the observed off-target probability shifts is due to generator artifacts rather than clinical shortcut learning. Because the training-based mitigation is also evaluated on counterfactuals from the same generator, the evidence that the model has learned true causal features rather than RoentMod-specific invariance is incomplete. These issues are addressable with additional control experime

major comments (4)
  1. [Methods – RoentMod development and design; Discussion (Limitations)] The shortcut diagnosis in Figure 3A assumes that RoentMod edits are clean interventions. With strength=0.4, every edit involves a global denoising pass, and the authors themselves concede that subtle, human-invisible changes may be present. A null-prompt control—e.g., editing with a neutral prompt such as 'no change' or an anatomically irrelevant finding—is needed to measure the generator-artifact baseline. Without it, the uniform off-diagonal percentile increases in Figure 3A could reflect models detecting an editing artifact rather than clinically meaningful shortcut use. This is load-bearing for the paper's central claim.
  2. [Results – Using RoentMod scans during training reduces reliance on shortcuts; Figure 3B] The mitigation evidence is circular in a specific sense: the counterfactual stress test used to demonstrate shortcut correction in the RoentMod-trained model is generated by the same RoentMod pipeline used to augment training. A model trained on RoentMod images may learn to ignore RoentMod-specific artifacts while still relying on real-world pathological co-occurrences. Please validate on an independent counterfactual generator (e.g., RadEdit or PRISM) or on real longitudinal pairs with naturally occurring pathology, and report off-target probability changes there. Also show that the external AUC gains in Table 1 are not replicated by training on equivalent volumes of synthetic images from a null-prompt edit or unrelated synthetic CXRs, which would separate shortcut correction from generic data augmentation.
  3. [Methods – Model Training; Table 1] The internal/external AUC improvement claim compares the RoentMod-augmented model to the pretrained TorchXRayVision NIH model. The architecture, training split, and data volume differ. A matched control is required: train the same DenseNet-121 on only the 20,000 real NIH patients, without synthetic images, using the same hyperparameters and early stopping. Otherwise the 3-19% and 1-11% differences may be attributable to additional training data, longer training, or different augmentation rather than to shortcut mitigation. This is central to the claim that RoentMod training 'improves' discrimination.
  4. [Results – Using RoentMod to add pathology to CXRs...; Figure 3] The headline finding is summarized as median percentile changes with no confidence intervals and no reported number of baseline scans per condition. Because the claim is about systematic off-target shifts, bootstrap confidence intervals or paired effect sizes are needed. This is especially important given that the co-occurrence rates in synthetic images differ from real NIH rates (Supplemental Figure 2)—e.g., cardiomegaly appears in 92% of edema prompts—so the off-diagonal signal may partly reflect clinically realistic co-occurrence rather than a model shortcut. A quantitative comparison against the radiologist-read co-occurrence reference (Supplemental Figure 6) should be made explicit for every off-diagonal entry.
minor comments (4)
  1. [Methods – RoentMod development and design] The text states 'we selected RoentMod’s guidance scale and strength parameters to be 0.4 and 4 respectively.' This appears to be reversed; as written, guidance_scale=0.4 is atypical. Please confirm the correct assignment and update the text.
  2. [Abstract / Table 1] The abstract reports 'improving model discrimination across multiple pathologies by 3-19% AUC in internal validation and by 1-11% for 5 out of 6 tested pathologies in external testing.' Table 1 values should be aligned with these ranges, and it should be stated explicitly which external pathology did not improve.
  3. [Results – RoentMod Evaluation] The reader study lacks real-image negative controls; the limitation is acknowledged in the Discussion, but the Results section would benefit from stating that the 93% realism figure is from synthetic-only reads and may be optimistic.
  4. [Figure 3] The color scale for 'blue boxes' is not defined. Please specify whether the reported values are absolute or relative percentile changes and provide a color bar so that effect sizes can be read from the figure.

Circularity Check

1 steps flagged

Partial circularity: RoentMod-trained model's off-target 'shortcut correction' is validated against the same radiologist-read co-occurrence labels used as training targets; real-data AUC provides independent support.

specific steps
  1. fitted input called prediction [Results 'Using RoentMod scans during training...' and Methods 'Model Training']
    "We used co-occurrence values from the RoentMod radiologist reads (Supplemental Figure 2) as the ground truth for off-target pathology for RoentMod synthetic scans ... We found that our model largely corrects shortcut learning on counterfactual scans for all tested conditions ... when compared to the radiologist-read co-occurrence rates of these pathologies as ground truth."

    The model was trained with off-target pathology targets set equal to the radiologist-read co-occurrence rates on RoentMod counterfactuals. The subsequent claim that RoentMod training 'corrects shortcut learning on counterfactual scans' is then evaluated against those same radiologist-read co-occurrence rates as ground truth. The model's agreement with this reference is therefore a check of training-fit on the very labels used for supervision, not an independent demonstration that reliance on off-target pathology was removed. Independent support for mitigation exists (real-data AUC gains on PadChest/CheXpert), but the counterfactual ground-truth comparison itself is circular.

full rationale

The paper's central claims rest on RoentMod as a clean counterfactual intervention. The radiologist reader study and external AUC improvements (1-11% on 5/6 pathologies) provide independent evidence and are not circular. The one concrete circular loop is in the shortcut-mitigation validation: the off-target 'ground truth' for RoentMod counterfactuals is derived from the same radiologist reads used to set training labels for the RoentMod-augmented model, so the model's corrected off-target predictions on RoentMod scans are partly forced by construction. The authors' admitted limitation—'subtle changes may be present in counterfactually generated CXRs that are not visible to human readers, affecting our ability to accurately perform shortcut testing'—is a validity confound for the shortcut diagnosis rather than a derivation-level circularity; it does not by itself reduce a prediction to an input. Likewise, reliance on RoentGen as a component is normal external tool use, not load-bearing self-citation. Overall, partial circularity in the counterfactual evaluation loop, but independent real-data benchmarks keep the paper from being wholly self-referential.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities. The load-bearing assumptions are about the fidelity of synthetic counterfactual images, the transferability of frozen diffusion weights, and the validity of radiologist-derived labels. The free parameters are hand-selected hyperparameters and the small-sample co-occurrence labels applied at scale.

free parameters (5)
  • guidance scale = 4 (text says 0.4; likely typo)
    Selected by a board-certified radiologist reviewing outputs from 5 images across a grid; affects realism and prompt adherence.
  • strength = 0.4 (text says 4; likely typo)
    Same hand-selected hyperparameter controlling how much the image can change.
  • real-to-synthetic training ratio = 65,315 real scans + 136,000 synthetic scans
    Chosen via small-scale experiments on 2,000 and 5,000 synthetic patients; no formal optimization or independent validation.
  • learning rate / batch size / epochs = 0.0001 / 32 / 100 (early stopping after 50)
    Training hyperparameters chosen ad hoc in small-scale experiments.
  • off-target pathology labels for synthetic scans = radiologist co-occurrence rates from 800 reads
    Used as ground truth labels for un-prompted pathologies on 136,000 synthetic training images; estimated from a small reader sample and applied globally.
axioms (5)
  • domain assumption Stable Diffusion image-to-image combined with RoentGen weights can edit CXRs without additional training.
    RoentMod's central mechanism; no retraining is claimed, so the frozen weights are assumed to transfer.
  • domain assumption Radiologist labels on 800 synthetic scans provide an unbiased ground truth for pathology presence.
    Used to validate RoentMod and to label synthetic training images; reader study lacked real negative controls and only had two readers.
  • domain assumption A CXR with 'no finding' in the report is truly free of pathology for baseline and counterfactual purposes.
    Baseline real scans and RoentGen-generated 'no acute cardiopulmonary process' images are treated as normal.
  • domain assumption Changes in model prediction percentile between a baseline and a counterfactual image reflect reliance on the added pathology rather than other image changes.
    Core to the shortcut stress test; the paper itself flags subtle non-human-visible changes.
  • domain assumption The four tested models are representative state-of-the-art CXR interpretation models.
    Claims about 'state-of-the-art' models are based on torchxrayvision, ElixrB, and Ark+.

pith-pipeline@v1.3.0-alltime-deepseek · 15498 in / 13147 out tokens · 129112 ms · 2026-08-04T20:15:32.202309+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of RoentMod: A Synthetic Chest X-Ray Modification Model to Identify and Correct Image Interpretation Model Shortcuts." pith.science (2026). https://pith.science/paper/PIDTYI5E

@misc{pith2026250908640,
  author       = {Pith},
  title        = {Pith review of: RoentMod: A Synthetic Chest X-Ray Modification Model to Identify and Correct Image Interpretation Model Shortcuts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PIDTYI5E}},
  note         = {Machine review of arXiv:2509.08640}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Chest radiographs (CXRs) are among the most common tests in medicine. Automated image interpretation may reduce radiologists\' workload and expand access to diagnostic expertise. Deep learning multi-task and foundation models have shown strong performance for CXR interpretation but are vulnerable to shortcut learning, where models rely on spurious and off-target correlations rather than clinically relevant features to make decisions. We introduce RoentMod, a counterfactual image editing framework that generates anatomically realistic CXRs with user-specified, synthetic pathology while preserving unrelated anatomical features of the original scan. RoentMod combines an open-source medical image generator (RoentGen) with an image-to-image modification model without requiring retraining. In reader studies with board-certified radiologists and radiology residents, RoentMod-produced images appeared realistic in 93\% of cases, correctly incorporated the specified finding in 89-99\% of cases, and preserved native anatomy comparable to real follow-up CXRs. Using RoentMod, we demonstrate that state-of-the-art multi-task and foundation models frequently exploit off-target pathology as shortcuts, limiting their specificity. Incorporating RoentMod-generated counterfactual images during training mitigated this vulnerability, improving model discrimination across multiple pathologies by 3-19\% AUC in internal validation and by 1-11\% for 5 out of 6 tested pathologies in external testing. These findings establish RoentMod as a broadly applicable tool for probing and correcting shortcut learning in medical AI. By enabling controlled counterfactual interventions, RoentMod enhances the robustness and interpretability of CXR interpretation models and provides a generalizable strategy for improving foundation models in medical imaging.

Figures

Figures reproduced from arXiv: 2509.08640 by Borek Foldyna, Jan M. Brendel, Lauren H. Cooke, Matthias Jung, Michael T. Lu, Nora M. Kerkovits, Vineet K. Raghu.

Figure 1
Figure 1. Figure 1: Graphical Abstract [PITH_FULL_IMAGE:figures/full_fig_p022_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representative real and counterfactual annotated chest radiographs produced by [PITH_FULL_IMAGE:figures/full_fig_p023_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: EƯect of adding pathologies on predicted probabilities from existing multitask CXR interpretation models (Panel A) and our RoentMod-trained multitask CXR interpretation model (Panel B). Blue boxes indicate greater sensitivity to adding pathology. Blank boxes indicate no change between baseline and RoentMod-generated counterfactuals. Values reflect the change in predicted probability percentile after pathol… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 10 canonical work pages · 7 internal anchors

  1. [1]

    Rajpurkar, P . et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLOS Med. 15, e1002686 (2018)

    2018

  2. [2]

    Fan, W. et al. A deep-learning-based framework for identifying and localizing multiple abnormalities and assessing cardiomegaly in chest X-ray. Nat. Commun. 15, 1347 (2024)

    2024

  3. [3]

    Yala, A. et al. Multi-Institutional Validation of a Mammography-Based Breast Cancer Risk Model. J. Clin. Oncol. OƯ. J. Am. Soc. Clin. Oncol. 40, 1732–1740 (2022)

    2022

  4. [4]

    McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89–94 (2020)

    2020

  5. [5]

    Zech, J. R. et al. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Med. 15, e1002683 (2018)

    2018

  6. [6]

    J., Karthikesalingam, A., Suleyman, M., Corrado, G

    Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 195 (2019)

    2019

  7. [7]

    & Lungren, M

    Rajpurkar, P . & Lungren, M. P . The Current and Future State of AI Interpretation of Medical Images. N. Engl. J. Med. 388, 1981–1990 (2023)

    1981

  8. [8]

    Gaube, S. et al. Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays. Sci. Rep. 13, 1383 (2023)

    2023

  9. [9]

    Yu, F . et al. Heterogeneity and predictors of the eƯects of AI assistance on radiologists. Nat. Med. 30, 837–849 (2024)

    2024

  10. [10]

    P ., Senadeera, M., Jacobs, S., Coghlan, S

    Quinn, T. P ., Senadeera, M., Jacobs, S., Coghlan, S. & Le, V . Trust and medical AI: the challenges we face and the expertise needed to overcome them. J. Am. Med. Inform. Assoc. 28, 890–894 (2021)

    2021

  11. [11]

    Gefter, W. B. et al. Human-AI Symbiosis: A Path Forward to Improve Chest Radiography and the Role of Radiologists in Patient Care. Radiology 310, e232778 (2024)

    2024

  12. [12]

    Ong Ly, C. et al. Shortcut learning in medical AI hinders generalization: method for estimating AI model generalization without external data. Npj Digit. Med. 7, (2024)

    2024

  13. [13]

    J., Janizek, J

    DeGrave, A. J., Janizek, J. D. & Lee, S.-I. AI for radiographic COVID-19 detection selects shortcuts over signal. Nat. Mach. Intell. 3, 610–619 (2021)

    2021

  14. [14]

    Brown, A. et al. Detecting shortcut learning for fair medical AI using shortcut testing. Nat. Commun. 14, 4314 (2023)

    2023

  15. [15]

    Gichoya, J. W. et al. AI recognition of patient race in medical imaging: a modelling study. Lancet Digit. Health 4, e406–e414 (2022)

    2022

  16. [16]

    K., Weiss, J., HoƯmann, U., Aerts, H

    Raghu, V . K., Weiss, J., HoƯmann, U., Aerts, H. J. W. L. & Lu, M. T. Deep Learning to Estimate Biological Age From Chest Radiographs. JACC Cardiovasc. Imaging 14, 2226–2236 (2021)

    2021

  17. [17]

    Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation

    Weng, N., Pegios, P ., Petersen, E., Feragen, A. & Bigdeli, S. Fast DiƯusion-Based Counterfactuals for Shortcut Removal and Generation. Preprint at https://doi.org/10.48550/arXiv.2312.14223 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.14223 2024

  18. [18]

    Hu, Z. et al. Improving Multi-Task Generalization via Regularizing Spurious Correlation. Adv. Neural Inf. Process. Syst. 35, 11450–11466 (2022)

    2022

  19. [19]

    Rajpurkar, P . et al. CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. Preprint at https://doi.org/10.48550/arXiv.1711.05225 (2017)

    Pith/arXiv arXiv doi:10.48550/arxiv.1711.05225 2017

  20. [20]

    Ma, D., Pang, J., Gotway, M. B. & Liang, J. A fully open AI foundation model applied to chest radiography. Nature 643, 488–498 (2025)

    2025

  21. [21]

    Sellergren, A. B. et al. Simplified Transfer Learning for Chest Radiography Models Using Less Data. Radiology 305, 454–465 (2022)

    2022

  22. [22]

    Bommasani, R. et al. On the Opportunities and Risks of Foundation Models. Preprint at https://doi.org/10.48550/arXiv.2108.07258 (2022)

    Pith/arXiv arXiv doi:10.48550/arxiv.2108.07258 2022

  23. [23]

    Lu, Y . et al. Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification. Preprint at https://doi.org/10.48550/arXiv.1611.05377 (2016)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1611.05377 2016

  24. [24]

    & Wang, X

    Chai, J., Lu, S. & Wang, X. Identifying and Mitigating Spurious Correlation in Multi-Task Learning. in 25698–25707 (2025)

    2025

  25. [25]

    Arun, N. et al. Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging. Radiol. Artif. Intell. 3, (2021)

    2021

  26. [26]

    Counterfactual Image Editing

    Pan, Y . & Bareinboim, E. Counterfactual Image Editing. Preprint at https://doi.org/10.48550/arXiv.2403.09683 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2403.09683 2024

  27. [27]

    & Arbel, T

    Kumar, A., Kriz, A., Havaei, M. & Arbel, T. PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable DiƯusion. Preprint at https://doi.org/10.48550/arXiv.2503.00196 (2025)

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.00196 2025

  28. [28]

    Pérez-García, F . et al. RadEdit: Stress-Testing Biomedical Vision Models via DiƯusion Image Editing. in Computer Vision – ECCV 2024 (eds Leonardis, A. et al.) 358–376 (Springer Nature Switzerland, Cham, 2025). doi:10.1007/978-3-031-73254-6_21

    work page doi:10.1007/978-3-031-73254-6_21 2024

  29. [29]

    Gu, Y . et al. BiomedJourney: Counterfactual Biomedical Image Generation by Instruction- Learning from Multimodal Patient Journeys. Preprint at https://doi.org/10.48550/arXiv.2310.10765 (2023)

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.10765 2023

  30. [30]

    Chambon, P . et al. RoentGen: Vision-Language Foundation Model for Chest X-ray Generation. Preprint at https://doi.org/10.48550/arXiv.2211.12737 (2022)

    Pith/arXiv arXiv doi:10.48550/arxiv.2211.12737 2022

  31. [31]

    & Ommer, B

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P . & Ommer, B. High-Resolution Image Synthesis with Latent DiƯusion Models. in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 10674–10685 (IEEE, New Orleans, LA, USA, 2022). doi:10.1109/CVPR52688.2022.01042

    arXiv 2022

  32. [32]

    Bluethgen, C. et al. A vision–language foundation model for the generation of realistic chest X- ray images. Nat. Biomed. Eng. 9, 494–506 (2024)

    2024

  33. [33]

    Cohen, J. P . et al. TorchXRayVision: A library of chest X-ray datasets and models. in Proceedings of Machine Learning Research: Proceedings of The 5th International Conference on Medical Imaging with Deep Learning vol. 172 231–249 (2022)

    2022

  34. [34]

    Kiraly, A. P . et al. Health AI Developer Foundations. Preprint at https://doi.org/10.48550/arXiv.2411.15128 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2411.15128 2024

  35. [35]

    & Hochreiter, S

    Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. & Hochreiter, S. GANs trained by a two time-scale update rule converge to a local nash equilibrium. in Proceedings of the 31st International Conference on Neural Information Processing Systems 6629–6640 (Curran Associates, Inc, Long Beach, CA, USA, 2017)

    2017

  36. [36]

    & Wojna, Z

    Szegedy, C., Vanhoucke, V ., IoƯe, S., Shlens, J. & Wojna, Z. Rethinking the Inception Architecture for Computer Vision. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2818–2826 (IEEE, Las Vegas, NV , USA, 2016). doi:10.1109/CVPR.2016.308

    doi:10.1109/cvpr.2016.308 2016

  37. [37]

    & Sun, J

    He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 770–778 (IEEE, Las Vegas, NV , USA, 2016). doi:10.1109/CVPR.2016.90

    doi:10.1109/cvpr.2016.90 2016

  38. [38]

    Radford, A. et al. Learning Transferable Visual Models From Natural Language Supervision. in Proceedings of the 38th International Conference on Machine Learning vol. 139 8748–8763 (Proceedings of Machine Learning Research, 2021)

    2021

  39. [39]

    Song, L. et al. Investigating causal associations between pneumonia and lung cancer using a bidirectional mendelian randomization framework. BMC Cancer 24, 721 (2024)

    2024

  40. [40]

    Smith-Bindman, R. et al. Trends in Use of Medical Imaging in US Health Care Systems and in Ontario, Canada, 2000-2016. JAMA 322, 843–856 (2019)

    2000

  41. [41]

    Wang, X. et al. ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 3462–3471 (IEEE, Honolulu, HI, 2017). doi:10.1109/CVPR.2017.369

    doi:10.1109/cvpr.2017.369 2017

  42. [42]

    ChestX-Ray 14 Dataset

    NIH Clinical Center. ChestX-Ray 14 Dataset. (2018)

    2018

  43. [43]

    Johnson, A. E. W. et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci. Data 6, 317 (2019)

    2019

  44. [44]

    Irvin, J. et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. Proc. AAAI Conf. Artif. Intell. 33, 590–597 (2019)

    2019

  45. [45]

    & De La Iglesia-Vayá, M

    Bustos, A., Pertusa, A., Salinas, J.-M. & De La Iglesia-Vayá, M. PadChest: A large chest x-ray image dataset with multi-label annotated reports. Med. Image Anal. 66, 101797 (2020)

    2020

  46. [46]

    Zeiler, M. D. & Fergus, R. Visualizing and Understanding Convolutional Networks. in Computer Vision – ECCV 2014 (eds Fleet, D., Pajdla, T., Schiele, B. & Tuytelaars, T.) 818–833 (Springer International Publishing, Cham, 2014). doi:10.1007/978-3-319-10590-1_53

    doi:10.1007/978-3-319-10590-1_53 2014

  47. [47]

    Selvaraju, R. R. et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. in 2017 IEEE International Conference on Computer Vision (ICCV) 618–626 (IEEE, Venice, 2017). doi:10.1109/iccv.2017.74

    doi:10.1109/iccv.2017.74 2017

  48. [48]

    & Gross, M

    Ancona, M., Ceolini, E., Öztireli, C. & Gross, M. Towards better understanding of gradient-based attribution methods for Deep Neural Networks. Preprint at https://doi.org/10.48550/arXiv.1711.06104 (2018)

    Pith/arXiv arXiv doi:10.48550/arxiv.1711.06104 2018

  49. [49]

    & Wattenberg, M

    Smilkov, D., Thorat, N., Kim, B., Viégas, F . & Wattenberg, M. SmoothGrad: removing noise by adding noise. Preprint at https://doi.org/10.48550/arXiv.1706.03825 (2017)

    Pith/arXiv arXiv doi:10.48550/arxiv.1706.03825 2017

  50. [50]

    & Zisserman, A

    Simonyan, K., Vedaldi, A. & Zisserman, A. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. Preprint at https://doi.org/10.48550/arXiv.1312.6034 (2014)

    Pith/arXiv arXiv doi:10.48550/arxiv.1312.6034 2014

  51. [51]

    Adebayo, J. et al. Sanity Checks for Saliency Maps. in Advances in Neural Information Processing Systems vol. 31 (Curran Associates, Inc., 2018)

    2018

  52. [52]

    Revisiting Sanity Checks for Saliency Maps

    Yona, G. & Greenfeld, D. Revisiting Sanity Checks for Saliency Maps. Preprint at https://doi.org/10.48550/arXiv.2110.14297 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2110.14297 2021

  53. [53]

    A., Tsaftaris, S

    Boland, C., Goatman, K. A., Tsaftaris, S. A. & Dahdouh, S. There Are No Shortcuts to Anywhere Worth Going: Identifying Shortcuts in Deep Learning Models for Medical Image Analysis. in Proceedings of The 7nd International Conference on Medical Imaging with Deep Learning 131– 150 (PMLR, 2024)

    2024

  54. [54]

    Stanley, E. A. M., Souza, R., Wilms, M. & Forkert, N. D. Where, why, and how is bias learned in medical image analysis models? A study of bias encoding within convolutional networks using synthetic data. eBioMedicine 111, (2025)

    2025

  55. [55]

    Wang, R. et al. Drop the shortcuts: image augmentation improves fairness and decreases AI detection of race and other demographics from medical images. eBioMedicine 102, (2024)

    2024

  56. [56]

    Ktena, I. et al. Generative models improve fairness of medical classifiers under distribution shifts. Nat. Med. 30, 1166–1173 (2024)

    2024

  57. [57]

    Cohen, J. P . et al. Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays. in Proceedings of Machine Learning Research 143 vol. 143 74–104 (Proceedings of Machine Learning Research, 2021)

    2021

  58. [58]

    Generative Visual Rationales

    Seah, J., Tang, J., Kitchen, A. & Seah, J. Generative Visual Rationales. Preprint at https://doi.org/10.48550/arXiv.1804.04539 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1804.04539 2018

  59. [59]

    in Lecture Notes in Computer Science 343–350 (Springer International Publishing, Cham, 2018)

    ScarGAN: Chained Generative Adversarial Networks to Simulate Pathological Tissue on Cardiovascular MR Scans. in Lecture Notes in Computer Science 343–350 (Springer International Publishing, Cham, 2018). doi:10.1007/978-3-030-00889-5_39

    work page doi:10.1007/978-3-030-00889-5_39 2018

  60. [60]

    Seah, J. C. Y ., Tang, J. S. N., Kitchen, A., Gaillard, F . & Dixon, A. F . Chest Radiographs in Congestive Heart Failure: Visualizing Neural Network Learning. Radiology 290, 514–522 (2019)

    2019

  61. [61]

    Wiatrak, M., Albrecht, S. V . & Nystrom, A. Stabilizing Generative Adversarial Networks: A Survey. Preprint at https://doi.org/10.48550/arXiv.1910.00927 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1910.00927 1910

  62. [62]

    & Glocker, B

    Pawlowski, N., Coelho de Castro, D. & Glocker, B. Deep Structural Causal Models for Tractable Counterfactual Inference. in Advances in Neural Information Processing Systems vol. 33 857– 869 (Curran Associates, Inc., 2020)

    2020

  63. [63]

    C., Carass, A

    Reinhold, J. C., Carass, A. & Prince, J. L. A Structural Causal Model for MR Images of Multiple Sclerosis. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (eds de Bruijne, M. et al.) 782–792 (Springer International Publishing, Cham, 2021). doi:10.1007/978-3-030-87240-3_75

    work page doi:10.1007/978-3-030-87240-3_75 2021

  64. [64]

    & Weinberger, K

    Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q. Densely Connected Convolutional Networks. in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, Honolulu, HI, 2017). doi:10.1109/cvpr.2017.243. Acknowledgements We gratefully acknowledge the contributions of the teams behind several open-access datasets and tools that ma...

    doi:10.1109/cvpr.2017.243 2017