REVIEW 4 major objections 5 minor 35 references
CF-Seg: Counterfactuals meet Segmentation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that generating counterfactual pseudo-healthy chest X-rays before segmentation improves lung segmentation substantially in the presence of pleural effusion, without any change to the segmentation model.
desk verdict A clean empirical application of counterfactual pre-processing to lung segmentation, with real expert evaluation, but the core anatomy-preservation assumption of the CF generator is unverified and the large-scale proxy is too weak to carry the causal claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a deep structural causal model (DSCM), a causal graph whose structural equations are neural networks, implemented here as a hierarchical variational autoencoder. The generator abducts the latent noise from the observed image, performs the do-intervention that sets disease status to healthy, and predicts the hypothetical image that would result, thereby removing effusion while keeping the same anatomy. This component carries the argument because it is what supposedly produces a clearer anatomical representation; the U-Net segmenter is unchanged and is applied only to the counterfactual image.
What would settle it
Compare the pseudo-healthy counterfactual of a healthy patient with the original healthy image; if the generator changes lung boundaries or volumes even when no disease is present, then the counterfactual does not preserve anatomy as claimed and the segmentation improvement cannot be attributed to disease removal.
Extended reading notes
Core claim
The central discovery is that a deep structural causal model's counterfactual intervention, setting the disease variable to healthy, removes pleural effusion from a chest X-ray while preserving the underlying lung anatomy, and feeding this pseudo-healthy image to an existing U-Net yields lung segmentations closer to radiologist ground truth than segmenting the original image. The improvement is concentrated in the right lung and in cases where the silver-standard masks undersegment, consistent with effusion obscuring lung boundaries. The paper also reports population-level evidence: lung volume distributions for healthy and effusion patients overlap when measured from CF-Seg segmentations but not from direct segmentations, indicating that the counterfactual pipeline recovers anatomy rather than merely shifting volumes.
Load-bearing premise
The counterfactual generator removes pleural effusion while preserving the exact lung boundaries; if the pseudo-healthy image subtly changes lung shape or size, the reported segmentation gains could be an artifact of altered anatomy rather than clearer anatomy.
Editorial extensions
If this is right
- For images with pleural effusion, segmenting the counterfactual instead of the original image improves Dice against expert ground truth, with the largest gain on undersegmented right lungs.
- Radiologists prefer CF-Seg segmentations over silver-standard masks for more than 70% of effusion images, while preferences are roughly balanced for healthy images.
- Population-level lung volume distributions of healthy and effusion patients overlap when measured with CF-Seg, suggesting the method recovers true anatomy rather than randomly rescaling volumes.
- No retraining or architectural change is needed, so the counterfactual step is a drop-in preprocessing module that can be combined with any pretrained segmenter.
- The authors note that other opacity-causing conditions, such as edema, ARDS, or pneumonia, could in principle be handled the same way given a suitable counterfactual generator.
Reading between the lines
- If the anatomy-preservation assumption holds, counterfactual-first preprocessing could generalize beyond segmentation to any task where disease-related appearance variation confounds an otherwise well-trained model.
- The gains likely depend on the causal graph capturing confounding variables such as scanner and sex, so deployment to a site with different scanner or population characteristics could degrade the benefit.
- A direct extension would test the same pipeline on other structures and pathologies, since the paper evaluates only lungs and pleural effusion.
- The volume-density overlap evidence is indirect; a direct test would measure whether CF-Seg's improvement comes from boundary correction rather than systematic volume bias.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CF-Seg, an inference-time framework that first generates pseudo-healthy counterfactual (CF) images of chest X-rays with pleural effusion using a deep structural causal model (the HVAE of Ribeiro et al. [5]), and then feeds these images to a standard U-Net for lung segmentation. The central claim is that segmenting the counterfactual image improves anatomical lung segmentation without retraining the segmentation model. The authors evaluate CF-Seg on MIMIC-CXR and PadChest through (i) a radiologist preference study on 300 images per dataset, (ii) expert segmentations on 140 images (50 pleural-effusion + 20 no-finding per dataset) with Dice-score comparisons, and (iii) a large-scale proxy that compares lung-volume distributions between healthy and effusion populations.
Significance. If the anatomy-preservation property of the counterfactual generator holds, this is a simple and attractive contribution: it converts a diseased image into a form that a pre-trained segmentor handles better, with no model changes, making it compatible with arbitrary segmentation networks. The paper includes a radiologist preference study and expert-annotated Dice comparisons, which are valuable clinical evidence for the approach. The reliance on an existing generator and the restriction to one pathology limit the scope, but as a proof-of-concept the result is compelling. I also credit the authors for committing to release code and segmentation masks, which will aid reproducibility and follow-up work.
major comments (4)
- [Sec. 3.1-3.2, Fig. 5] The causal mechanism asserted in the central claim—that the counterfactual removes pleural effusion while preserving the underlying lung anatomy—is never directly verified. The paper does not measure lung-boundary displacement between the original and CF images, nor does it obtain expert segmentations on the CF images to confirm they agree with expert segmentations on the original images. The indirect evidence in Table 1 could in principle arise from a CF image that simply looks healthy: the U-Net was trained on CheXMask silver labels that undersegment PE lungs, so a healthier-looking image could produce a larger mask that scores higher against expert masks even if the CF boundary deviates from the true boundary of the effused lung. Please add a direct anatomy-preservation test, e.g., compare radiologist segmentations drawn on CF images against those on the original images, or compute a boundary-distance metric (such as Hausdorff distance) between segmentations on original and CF images to confirm that changes are confined to regions where effusion was present.
- [Sec. 4.4, Table 1] All Dice results are reported as means without error bars, confidence intervals, or paired significance tests. With n=50 PE images per dataset (and roughly 35 in the ΔV+ subgroup), the observed gains of 0.5–2.7 Dice points, while consistent, may not be statistically significant, especially for the 'All' columns. Please report per-image variability and run a paired test (e.g., Wilcoxon signed-rank) between U-Net and CF-Seg on the same images. Additionally, the ΔV+ selection criterion and the Dice computation use the same expert masks, so the subgroup result should be accompanied by an appropriate caveat or by a clear pre-specified definition.
- [Sec. 4.5, Fig. 6] The large-scale evaluation relies on the assumption that healthy (NF) and pleural-effusion (PE) populations have identical lung-volume densities. This assumption is not established and could be false: pleural effusion may compress or collapse lung tissue, shifting the true volume distribution. The KDE overlap in Fig. 6 is only a qualitative visual comparison; no quantitative divergence metric is reported. Please validate the equal-volume assumption on the expert-annotated subset (e.g., compare Expert NF vs Expert PE volumes), or reframe the claim to state only that CF-Seg makes the PE volume distribution resemble the NF distribution, without asserting that this is anatomically correct.
- [Sec. 4.3, Fig. 3] The preference study compares CF-Seg against CheXMask labels, not against the same U-Net applied to the original image. Since CF-Seg is literally the U-Net run on the counterfactual, the preference results could be driven by the U-Net's output characteristics rather than by the counterfactual intervention. Please include a U-Net-on-original arm in the user study (or replace the CheXMask baseline with it) so that the effect of the counterfactual is isolated.
minor comments (5)
- [Sec. 4.2] The name 'Riberio' appears in the text; the correct spelling is 'Ribeiro' (as in the reference list).
- [Fig. 2] The schematic is unclear on which components are frozen or pre-trained; please annotate the DSCM and U-Net blocks more explicitly for reproducibility.
- [Table 1] Define 'All' and 'ΔV+' in the caption; currently they appear without explanation in the table.
- [Sec. 4.5] The sentence 'We hypothesize that irrespective of the disease stage...' should be explicitly framed as an unverified assumption, and the argument should acknowledge that this hypothesis is not tested.
- [Sec. 4.3] Please clarify the exact number of radiologists per dataset and the total number of images each radiologist reviewed; the text says 'one for each dataset' and '300 test images for each dataset', but it would help to state this explicitly in the figure caption or text.
Circularity Check
No significant circularity: CF-Seg's segmentation gains are tested against external expert annotations and radiologist preference, not derived from the counterfactual generator's own outputs.
full rationale
The paper's central claim is empirical: applying pseudo-healthy counterfactuals before segmentation improves anatomical segmentation. This claim is validated in Sec. 4.3 by radiologist preference and in Sec. 4.4 against expert-drawn ground-truth segmentations, both of which are external to the counterfactual generator and to the U-Net's training labels. The only self-citation is the use of the HVAE/DSCM from Ribeiro et al. [5] as an off-the-shelf counterfactual generator. The paper does not fit any parameter of that generator to the expert segmentations, nor does it define its evaluation metric in terms of the generator's outputs. The assumption that counterfactuals preserve anatomy is inherited from [5] and is not independently verified here, but this is a validity assumption rather than a circular reduction: the segmentation improvement is measured against external masks and human judgment. The large-scale proxy in Sec. 4.5 is presented as an explicit hypothesis about population lung volumes, not as a restatement of the method's objective. No equation or fitted parameter reduces to the claimed result, and no load-bearing uniqueness theorem or ansatz is smuggled in via self-citation. Therefore the derivation chain is not circular; the observed improvements are independently testable against expert annotations.
Assumptions & free parameters
free parameters (1)
- PadChest causal graph variables =
hand-chosen (scanner, sex, disease)
assumptions (3)
- domain assumption Counterfactual images generated by the HVAE preserve underlying anatomy while removing disease appearance.
- domain assumption Lung volume density across the population is similar between no-finding and pleural-effusion groups.
- domain assumption The causal graph and exogenous noise model of the DSCM from [5] are correctly specified for chest X-rays.
Cite this review
Pith. "Pith review of CF-Seg: Counterfactuals meet Segmentation." pith.science (2026). https://pith.science/paper/JVRVBSKM
@misc{pith2026250616213,
author = {Pith},
title = {Pith review of: CF-Seg: Counterfactuals meet Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/JVRVBSKM}},
note = {Machine review of arXiv:2506.16213}
}
read the original abstract
Segmenting anatomical structures in medical images plays an important role in the quantitative assessment of various diseases. However, accurate segmentation becomes significantly more challenging in the presence of disease. Disease patterns can alter the appearance of surrounding healthy tissues, introduce ambiguous boundaries, or even obscure critical anatomical structures. As such, segmentation models trained on real-world datasets may struggle to provide good anatomical segmentation, leading to potential misdiagnosis. In this paper, we generate counterfactual (CF) images to simulate how the same anatomy would appear in the absence of disease without altering the underlying structure. We then use these CF images to segment structures of interest, without requiring any changes to the underlying segmentation model. Our experiments on two real-world clinical chest X-ray datasets show that the use of counterfactual images improves anatomical segmentation, thereby aiding downstream clinical decision-making.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[5]
De Sousa Ribeiro, F., Xia, T., Monteiro, M., Pawlowski, N., Glocker, B.: High fidelity image counterfactuals with probabilistic causal models. ICML (2023)
work page 2023
-
[1]
Baugh, M., Reynaud, H., Marimont, S.N., Cechnicka, S., Müller, J.P., Tarroni, G., Kainz, B.: Image-conditioned diffusion models for medical anomaly detection. In: International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. pp. 117–127. Springer (2024)
work page 2024
-
[2]
Baugh, M., Tan, J., Müller, J.P., Dombrowski, M., Batten, J., Kainz, B.: Many tasks make light work: Learning to localise medical anomalies from multiple syn- thetic tasks. In: MICCAI 2023. pp. 162–172. Springer (2023)
work page 2023
-
[3]
Medical image anal- ysis 66, 101797 (2020) 10 Mehta et al
Bustos, A., Pertusa, A., Salinas, J.M., De La Iglesia-Vaya, M.: Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image anal- ysis 66, 101797 (2020) 10 Mehta et al
work page 2020
- [4]
-
[6]
Fathi, N., Kumar, A., Nichyporuk, B., Havaei, M., Arbel, T.: Decodex: Confounder detector guidance for improved diffusion-based counterfactual explanations. MIDL (2024)
work page 2024
-
[7]
Scientific Data11(1), 511 (2024)
Gaggion, N., Mosquera, C., Mansilla, L., Saidman, J.M., Aineseder, M., Milone, D.H., Ferrante, E.: Chexmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images. Scientific Data11(1), 511 (2024)
2024
-
[8]
Machine Learning for Biomedical Imaging2, 611–656 (2024)
Hassanaly, R., Brianceau, C., Solal, M., Colliot, O., Burgos, N.: Evaluation of pseudo-healthy image reconstruction for anomaly detection with deep generative models: Application to brain fdg pet. Machine Learning for Biomedical Imaging2, 611–656 (2024)
work page 2024
Show all 35 references
-
[9]
IEEE Transactions on Medical Imaging40(12), 3519–3530 (2021)
Huang, H., Zheng, H., Lin, L., Cai, M., Hu, H., Zhang, Q., Chen, Q., Iwamoto, Y., Han, X., Chen, Y.W., et al.: Medical image segmentation with deep atlas prior. IEEE Transactions on Medical Imaging40(12), 3519–3530 (2021)
2021
-
[10]
In: ICML
Ilse, M., Tomczak, J.M., Forré, P.: Selecting data augmentation for simulating interventions. In: ICML. pp. 4555–4562. PMLR (2021)
2021
-
[11]
IEEE transactions on medical imaging33(2), 233–245 (2013)
Jaeger, S., Karargyris, A., Candemir, S., Folio, L., Siegelman, J., Callaghan, F., Xue, Z., Palaniappan, K., Singh, R.K., Antani, S., et al.: Automatic tuberculosis screening using chest radiographs. IEEE transactions on medical imaging33(2), 233–245 (2013)
2013
-
[12]
In: MICCAI 2024
Jaus, A., Seibold, C., Reiß, S., Heine, L., Schily, A., Kim, M., Bahnsen, F.H., Her- rmann, K., Stiefelhagen, R., Kleesiek, J.: Anatomy-guided pathology segmentation. In: MICCAI 2024. pp. 3–13. Springer (2024)
2024
-
[13]
Scientific data10(1), 1 (2023)
Johnson, A.E., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T.J., Hao, S., Moody, B., Gow, B., et al.: Mimic-iv, a freely accessible electronic health record dataset. Scientific data10(1), 1 (2023)
2023
-
[14]
Scientific Reports13(1), 20899 (2023)
Khomduean, P., Phuaudomcharoen, P., Boonchu, T., Taetragool, U., et al.: Seg- mentation of lung lobes and lesions in chest ct for the classification of covid-19 severity. Scientific Reports13(1), 20899 (2023)
2023
-
[15]
ICLR (2017)
Kocaoglu, M., Snyder, C., Dimakis, A.G., Vishwanath, S.: Causalgan: Learning causal implicit generative models with adversarial training. ICLR (2017)
2017
-
[16]
In: MICCAI Workshop on Clinical Image-Based Procedures
Kumar, A., Fathi, N., Mehta, R., Nichyporuk, B., Falet, J.P.R., Tsaftaris, S., Arbel, T.: Debiasing counterfactuals in the presence of spurious correlations. In: MICCAI Workshop on Clinical Image-Based Procedures. pp. 276–286. Springer (2023)
2023
-
[17]
IEEE Transactions on Medical Imaging39(12), 3813–3820 (2020)
Larrazabal, A.J., Martínez, C., Glocker, B., Ferrante, E.: Post-dae: anatomically plausible segmentation via post-processing with denoising autoencoders. IEEE Transactions on Medical Imaging39(12), 3813–3820 (2020)
2020
-
[18]
Scientific Reports12(1), 8649 (2022)
Liu, W., Luo, J., Yang, Y., Wang, W., Deng, J., Yu, L.: Automatic lung segmen- tation in chest x-ray images using improved u-net. Scientific Reports12(1), 8649 (2022)
2022
-
[19]
In: MICCAI
Müller, P., Meissen, F., Brandt, J., Kaissis, G., Rueckert, D.: Anatomy-driven pathology detection on chest x-rays. In: MICCAI. pp. 57–66. Springer (2023)
2023
-
[20]
In: Machine Learning in Medical Imaging
Navarro, F., Shit, S., Ezhov, I., Paetzold, J., Gafita, A., Peeken, J.C., Combs, S.E., Menze, B.H.: Shape-aware complementary-task learning for multi-organ segmen- tation. In: Machine Learning in Medical Imaging. pp. 620–627. Springer (2019) CF-Seg: Counterfactuals meet Segmen...
2019
-
[21]
IEEE Transactions on Medical Imaging37(2), 384–395 (2017)
Oktay, O., Ferrante, E., Kamnitsas, K., Heinrich, M., Bai, W., et al.: Anatomically constrained neural networks (acnns): application to cardiac image enhancement and segmentation. IEEE Transactions on Medical Imaging37(2), 384–395 (2017)
2017
-
[22]
NeurIPS33, 857–869 (2020)
Pawlowski, N., Coelho de Castro, D., Glocker, B.: Deep structural causal models for tractable counterfactual inference. NeurIPS33, 857–869 (2020)
2020
-
[23]
Cambridge university press (2009)
Pearl, J.: Causality. Cambridge university press (2009)
2009
-
[24]
Medical Image Analysis 84, 102723 (2023)
Pombo, G., Gray, R., Cardoso, M.J., Ourselin, S., Rees, G., Ashburner, J., Nachev, P.: Equitable modelling of brain imaging by counterfactual augmentation with morphologically constrained 3d deep generative models. Medical Image Analysis 84, 102723 (2023)
2023
-
[25]
In: MICCAI
Puglisi, L., Alexander, D.C., Ravì, D.: Enhancing spatiotemporal disease progres- sion models via latent diffusion and prior knowledge. In: MICCAI. pp. 173–183. Springer (2024)
2024
-
[26]
BMC medical imaging20, 1–13 (2020)
Reamaroon, N., Sjoding, M.W., Derksen, H., Sabeti, E., Gryak, J., Barbaro, R.P., Athey, B.D., Najarian, K.: Robust segmentation of lung in chest x-ray: applications in analysis of acute respiratory distress syndrome. BMC medical imaging20, 1–13 (2020)
2020
-
[27]
In: MICCAI
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
2015
-
[28]
In: MICCAI Workshop on Data Engineering in Medical Imaging
Roschewitz, M., de Sousa Ribeiro, F., Xia, T., Khara, G., Glocker, B.: Counter- factual contrastive learning: robust representations via causal image synthesis. In: MICCAI Workshop on Data Engineering in Medical Imaging. pp. 22–32 (2024)
2024
-
[29]
Conference on Causal Learning and Reasoning (2022)
Sanchez, P., Tsaftaris, S.A.: Diffusion causal models for counterfactual estimation. Conference on Causal Learning and Reasoning (2022)
2022
-
[30]
In: ICLR 2023 (2023)
Xia, K.M., Pan, Y., Bareinboim, E.: Neural causal models for counterfactual iden- tification and estimation. In: ICLR 2023 (2023)
2023
-
[31]
Medical Image Analysis64, 101719 (2020)
Xia,T.,Chartsias,A.,Tsaftaris,S.A.:Pseudo-healthysynthesiswithpathologydis- entanglement and adversarial learning. Medical Image Analysis64, 101719 (2020)
2020
-
[32]
In: CVPR
Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., Wang, J.: Causalvae: Disentangled representation learning via neural structural causal models. In: CVPR. pp. 9593– 9602 (2021)
2021
-
[33]
In: MICCAI 2019
Yao, J., Cai, J., Yang, D., Xu, D., Huang, J.: Integrating 3d geometry of organ for improving medical image segmentation. In: MICCAI 2019. pp. 318–326. Springer (2019)
2019
-
[34]
arXiv:2304.13431 (2023)
Zhou, X., Wu, O., Ng, M.K.: Implicit counterfactual data augmentation for robust learning. arXiv:2304.13431 (2023)
2023 arXiv
-
[35]
IEEE Transactions on Medical Imaging41(10), 2728–2738 (2022)
Zimmerer, D., Full, P.M., Isensee, F., Jäger, P., Adler, T., Petersen, J., Köhler, G., Ross, T., Reinke, A., Kascenas, A., et al.: Mood 2020: A public benchmark for out- of-distribution detection and localization on medical images. IEEE Transactions on Medical Imaging41(10), 2...
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.