REVIEW 4 major objections 4 minor 1 cited by
Explaining 3D Computed Tomography Classifiers with Counterfactuals
T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Counterfactual explanations are extended from 2D images to entire 3D CT scan classifiers using a slice-based autoencoder with gradient blocking.
desk verdict A genuinely useful engineering step that makes Latent Shift tractable for 3D CT, but the quantitative evaluation needs to be much stronger before the effectiveness claim is taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the 'Slice AE': a VQ-GAN (a vector-quantized autoencoder with a perceptual adversarial loss) trained on 1,447,551 CT slices from 12,686 scans, downscaled to 256×256 and cropped to 128×128. At inference, each slice of a volume is encoded independently, selected latent codes are modified by the counterfactual step, and the decoded slices are concatenated into a full volume that is fed to the classifier. Gradient blocking restricts the backpropagation from the classifier output to the latent embeddings of only a small chunk of slices, typically ten with 32 GB of GPU memory, which makes the computation tractable while preserving whole-volume context. The decoder acts as a regularizer, keeping the modified images within the learned distribution of CT scans.
What would settle it
Take a test volume, encode it, modify one chunk of latent codes along the classifier gradient, and decode the full volume. If the mean absolute pixel difference across adjacent-slice boundaries at the chunk edges is systematically larger than across boundaries elsewhere in the same volume, the slice-based latent space is not producing anatomically coherent counterfactuals, and the central assumption fails.
Extended reading notes
Core claim
The central claim is that Latent Shift, which subtracts a gradient-scaled step from an autoencoder's latent code, can be made tractable for whole CT volumes by replacing the 3D autoencoder with a slice-based 2D autoencoder and restricting backpropagation to a few slices at a time. The remaining slices still pass through the classifier and provide anatomical context, but they receive no gradient updates. On two tasks—a lung-size regression derived from a released lung-segmentation model, and a pleural-effusion phenotype classifier—the resulting counterfactual volumes localize to clinically meaningful regions and shift classifier scores toward the opposite class. The paper claims this is the first counterfactual generation method for 3D CT classifiers, and it releases the trained slice-level VQ-GAN along with the code so that other classifiers can be audited without retraining the latent model.
Load-bearing premise
The slice-based autoencoder, with decoded slices concatenated back into a volume, is assumed to produce anatomically coherent 3D volumes after a latent-space step, even though no inter-slice consistency regularization is applied.
Editorial extensions
If this is right
- Any 3D CT classifier that accepts a full volume as input can be explained with the released slice-level VQ-GAN, without retraining the autoencoder for each classifier.
- Chunking the volume and measuring which chunk most reduces the prediction identifies the slices and regions most responsible for a classifier's decision.
- Larger chunk sizes produce larger prediction reductions but weaker spatial localization, giving users a tunable trade-off between explanation scope and computational cost.
- The method reduces pleural-effusion predictions by about eight points on positive examples, moving their score distribution closer to negative examples, and the difference between positive and negative cases is statistically significant.
Reading between the lines
- A testable implication left implicit is that decoded slices may create inter-slice discontinuities at chunk boundaries; measuring adjacent-slice differences in generated counterfactuals versus unmodified volumes would quantify whether this undermines anatomical coherence.
- Because the counterfactual step is constrained by the VQ-GAN's latent space, features the autoencoder cannot represent well may be unmovable, predicting that prediction reduction will plateau even with large step sizes, which the paper already observes for lung size.
- The method could likely be ported to other 3D modalities such as MRI by retraining the slice autoencoder on that modality, but only if the slice-independence assumption holds there too.
- The 5% pixel-change stopping rule may cap how far a prediction can move; testing whether a perceptual-loss penalty instead of a hard cap allows larger prediction shifts while keeping images plausible would probe that limit.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper extends the Latent Shift counterfactual-generation method from 2D to 3D computed tomography (CT). To make gradient-based latent manipulation tractable for whole-volume 3D classifiers, the authors train a slice-based VQ-GAN on 1,447,551 CT slices from 12,686 scans, encode/decode the volume slice-by-slice, and block gradients except for a small chunk of slices. They evaluate the approach on a lung-size regression task and a pleural-effusion classifier, reporting prediction reduction on the PleThora/NSCLC-Radiomics dataset and showing qualitative slice-level visualizations. The paper claims to be the first to generate counterfactuals for 3D CT classifiers and to be memory-efficient and effective for high-resolution volumes, and it releases code and model weights.
Significance. If the effectiveness claim holds, the work has practical value: a public slice-level VQ-GAN plus gradient blocking could let others audit arbitrary 3D CT classifiers without the memory cost of a 3D autoencoder, and the released model is a reusable resource. The engineering contribution is real and well described: the chunked-gradient design is a reasonable response to the 32 GB memory constraint, and the reported per-chunk runtimes make the tractability claim credible. However, the quantitative evidence for "effectiveness" is currently weak because the main metric is prediction reduction, which is exactly the objective that Latent Shift's gradient update is designed to optimize. The significance of the paper therefore hinges on whether the generated volumes are independently shown to be anatomically coherent 3D counterfactuals, not merely gradient-descent successes.
major comments (4)
- [§5.2.2, Table 1] The central quantitative claim that the method is "effective" rests on the prediction reduction from 0.24 ± 0.03 to 0.15 ± 0.02 for positive examples. Because Latent Shift constructs its update by subtracting the gradient of the classifier output with respect to the latent code, a decrease in the predicted probability is the direct objective of the optimization, not an independent measure of counterfactual validity. Please add outcome measures that do not share this objective, such as changes in pleural-effusion or lung segmentation overlap, image-distance or reconstruction-consistency statistics between input and CF, a likelihood or realism score from a held-out 3D model, or clinician/reader ratings of the modified volumes.
- [§2, Slice AE description] The validity of the slice-concatenated volume as a 3D CT depends on inter-slice coherence, yet the manuscript explicitly states that "no special regularization is performed" for neighboring-slice relationships and provides no measurement of whether this assumption holds. Please quantify slice-boundary continuity between the modified chunk and the untouched adjacent slices (e.g., mean absolute difference at boundary slice pairs versus the original volume) and evaluate 3D plausibility of the resulting volume. Without such evidence, the reported prediction drop could be driven by slice-wise artifacts or locally atypical anatomy rather than by a coherent counterfactual change.
- [§5.1, Figure 2] The lung-size validation is anecdotal: Figure 2A shows a single case and Figure 2B plots the sum of segmentation pixels for that one case. To support the claim that the method reliably changes the clinically relevant feature, please report aggregate statistics over the PleThora or LUNA16 volumes, such as the mean reduction in predicted lung volume across the positive set, and ideally correlate the CF-induced prediction change with the segmentation-based lung-size change.
- [§5.2.1, Figure 4] The comparison with Input Gradient, Guided Backprop, and Grad-CAM is purely qualitative and implicitly assumes that the classifier is using pleural effusion in the way the authors expect. To substantiate the effectiveness claim, compare against a simple non-gradient control (e.g., random latent perturbation or blurring of randomly selected regions) and report a quantitative localization metric, using the PleThora effusion/thoracic-cavity masks if available, rather than relying only on visual inspection.
minor comments (4)
- [Throughout] The manuscript consistently misspells "pleural effusion" as "plural effusion" (e.g., §5.2 heading and §5.2.1); please correct this.
- [§5.2.1] There is a typo, "identifyed" should be "identified," in the sentence describing what Latent Shift and Input Gradients identify.
- [§5.2 and §5.2.2] Figure 3B describes processing the volume in chunks of five slices, whereas §5.2.2 and Table 1 use a chunk size of 12; please clarify whether these are separate experiments and justify the difference in chunk-size selection.
- [§5.2.2, Table 1] The statement that the difference between positive and negative examples is statistically significant (p < 0.001) would benefit from specifying the statistical test used and whether the comparison is paired or unpaired.
Circularity Check
Prediction-reduction evaluation repeats the Latent Shift optimization objective; independent validation is only qualitative.
-
fitted input called prediction
[Section 2 (Counterfactual Generation) and Section 5.2.2 (Quantitative Evaluation), Table 1]
"The coefficient λ is selected using an iterative search to ensure a predetermined change in the classifier’s output. ... To quantitatively assess the impact of CF images on the classifier, we measure the change in prediction across a dataset. ... We report the minimum prediction here. ... Table 1 shows a 17-point difference between positive and negative examples. We also observe when our CF method is applied to the positive examples the prediction is reduced by 8 points, indicating an effective counterfactual generation method."
The quantitative evidence for 'effectiveness' is the reduction in classifier prediction on the positive examples. But that reduction is exactly the quantity that Latent Shift is constructed to produce: λ is iteratively selected 'to ensure a predetermined change in the classifier’s output', and the reported number is the minimum prediction obtained after that search. Thus the headline quantitative result is the optimization objective restated as an evaluation metric. It would hold for any method that successfully minimizes the classifier score, regardless of whether the resulting image is a meaningful, interpretable counterfactual.
full rationale
The paper's core methodological contribution is an engineering extension of the authors' prior Latent Shift method to 3D CT via a slice-based autoencoder and gradient blocking. That extension is described concretely and is not made circular by self-citation: the prior Latent Shift method is summarized in the text, and the trained VQ-GAN is released and trained on public data. No load-bearing uniqueness theorem or ansatz is smuggled in via citation. The main circularity concern is the quantitative evaluation in Section 5.2.2: the method optimizes a reduction in classifier confidence, and the paper then reports that reduction as evidence that the counterfactuals are effective. This is a circular success criterion for the method's interpretability claim, although the case studies and the lung-size experiment provide some independent, qualitative grounding. The admitted lack of inter-slice consistency regularization is a validity limitation rather than a circularity, and the limitations section appropriately acknowledges that the decoder provides no formal guarantee of identity preservation. Overall, the central 'effectiveness' claim is partially circular because its strongest quantitative evidence reduces by construction to the optimization objective, but the paper also contains non-circular qualitative validation and a public model release, so the score reflects partial circularity rather than a fully forced derivation.
Assumptions & free parameters
free parameters (3)
- lambda (gradient scaling coefficient) =
per-sample, unpublished
- chunk_size =
10 slices (V100) and 12 slices (PleThora)
- pixel_change_threshold =
5%
assumptions (5)
- domain assumption A gradient step in the autoencoder latent space moves the generated image meaningfully along the classifier's decision boundary while the decoder keeps changes in-distribution.
- domain assumption Decoding 2D slices independently and concatenating them preserves enough 3D context for coherent counterfactual volumes.
- domain assumption The VQ-GAN trained on 1.45 million slices from LUNA16, TotalSegmenter, and DeepLesion adequately represents the CT domain.
- domain assumption Lung segmentation output sum can serve as a proxy for lung size in the lung-size validation.
- domain assumption Pleural effusion is visually identifiable as bright regions along the lung periphery.
Cite this review
Pith. "Pith review of Explaining 3D Computed Tomography Classifiers with Counterfactuals." pith.science (2026). https://pith.science/paper/TGNOJKS3
@misc{pith2026250207156,
author = {Pith},
title = {Pith review of: Explaining 3D Computed Tomography Classifiers with Counterfactuals},
year = {2026},
howpublished = {\url{https://pith.science/paper/TGNOJKS3}},
note = {Machine review of arXiv:2502.07156}
}
read the original abstract
Counterfactual explanations enhance the interpretability of deep learning models in medical imaging, yet adapting them to 3D CT scans poses challenges due to volumetric complexity and resource demands. We extend the Latent Shift counterfactual generation method from 2D applications to explain 3D computed tomography (CT) scans classifiers. We address the challenges associated with 3D classifiers, such as limited training samples and high memory demands, by implementing a slice-based autoencoder and gradient blocking except for specific chunks of slices. This method leverages a 2D encoder trained on CT slices, which are subsequently combined to maintain 3D context. We demonstrate this technique on two models for clinical phenotype prediction and lung segmentation. Our approach is both memory-efficient and effective for generating interpretable counterfactuals in high-resolution 3D medical imaging.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering
CT-Agent combines an LLM planner, region-specific LoRA adapters, and global/local token compression to improve 3D chest CT report generation and question answering on CT-RATE and RadGenome-ChestCT.
Reference graph
Works this paper leans on
-
[1]
Hugo J. W. L. Aerts, Emmanuel Rios Velazquez, Ralph T. H. Leijenaar, Chintan Parmar, Patrick Grossmann, Sara Carvalho, Johan Bussink, René Monshouwer, Benjamin Haibe-Kains, Derek Rietveld, Frank Hoebers, Michelle M. Rietbergen, C. René Leemans, Andre Dekker, John Quackenbush, Robert J. Gillies, and Philippe Lambin. Decoding tumour phenotype by noninvasive...
-
[2]
Shah, Andrew Johnston, Robert D
Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, Christian Bluethgen, Malte Engmann Kjeldskov Jensen, Sophie Ostmeier, Maya Varma, Jeya Maria Jose Valanarasu, Zhongnan Fang, Zepeng Huo, Zaid Nabulsi, Diego Ardila, Wei-Hung Weng...
arXiv 2024
-
[3]
Joseph Paul Cohen, Rupert Brooks, Sovann En, Evan Zucker, Anuj Pareek, Matthew P. Lungren, and Akshay Chaudhari. Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays . Medical Imaging with Deep Learning, 2021. URL https://openreview.net/forum?id=rnunjvgxAMt
work page 2021
-
[4]
Taming transformers for high-resolution image synthesis
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis . In Computer Vision and Pattern Recognition, 2021. ISBN 9781665445092. doi:10.1109/CVPR46437.2021.01268. URL http://arxiv.org/abs/2012.09841
arXiv 2021
-
[5]
Johannes Hofmanninger, Forian Prayer, Jeanny Pan, Sebastian R \" o hrich, Helmut Prosch, and Georg Langs. Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem . European Radiology Experimental, 2020. ISSN 2509-9280. doi:10.1186/s41747-020-00173-2. URL https://eurradiolexp.springeropen.com/articles...
-
[6]
Kendall J Kiser, Sara Ahmed, Sonja Stieb, Abdallah S R Mohamed, Hesham Elhalawani, Peter Y S Park, Nathan S Doyle, Brandon J Wang, Arko Barman, Zhao Li, W Jim Zheng, Clifton D Fuller, and Luca Giancardo. PleThora: Pleural effusion and thoracic cavity segmentations in diseased lungs for benchmarking chest CT processing pipelines. Medical physics, 11 2020. ...
-
[7]
Rachana Krishna, Marsha H. Antoine, Mohamed H. Alahmadi, and Mohan Rudrappa. Pleural Effusion . StatPearls Publishing, 2025. URL http://www.ncbi.nlm.nih.gov/pubmed/25077579
-
[8]
Counterfactual Image Synthesis for Discovery of Personalized Predictive Image Markers
Amar Kumar, Anjun Hu, Brennan Nichyporuk, Jean-Pierre R. Falet, Douglas L. Arnold, Sotirios Tsaftaris, and Tal Arbel. Counterfactual Image Synthesis for Discovery of Personalized Predictive Image Markers . 2022. URL https://arxiv.org/abs/2208.02311
work page Pith review arXiv 2022
Show all 23 references
-
[9]
CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Nets
Daniel Nemirovsky, Nicolas Thiebaut, Ye Xu, and Abhishek Gupta. CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Nets . 2020. URL http://arxiv.org/abs/2009.05199
2020 arXiv
-
[10]
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K \" o pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, L...
2019 arXiv
-
[11]
Wei Peng, Tian Xia, Fabio De Sousa Ribeiro, Tomas Bosschieter, Ehsan Adeli, Qingyu Zhao, Ben Glocker, and Kilian M. Pohl. Latent 3D Brain MRI Counterfactual . 9 2024. URL http://arxiv.org/abs/2409.05585
2024
-
[12]
High Fidelity Image Counterfactuals with Probabilistic Causal Models
Fabio De Sousa Ribeiro, Tian Xia, Miguel Monteiro, Nick Pawlowski, and Ben Glocker. High Fidelity Image Counterfactuals with Probabilistic Causal Models . 2023. URL http://arxiv.org/abs/2306.15764
2023 arXiv
-
[13]
Seah, Jennifer S.N
Jarrel C.Y. Seah, Jennifer S.N. Tang, Andy Kitchen, Frank Gaillard, and Andrew F. Dixon. Chest radiographs in congestive heart failure: Visualizing neural network learning . Radiology, 2019. ISSN 15271315. doi:10.1148/radiol.2018180887
2019 doi
-
[14]
Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization . Computer Vision and Pattern Recognition, 10 2017. URL http://arxiv.org/abs/1610.02391
2017 arXiv
-
[15]
Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas de Bel, Moira S. N. Berens, Cas van den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, Robbert van der Gugten, Pheng Ann Heng, Bart Jansen, Michael M. J. de Kaste, Valentin Kotov, Jack...
2016 arXiv
-
[16]
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps . In International Conference on Learning Representations, 2014. URL http://arxiv.org/abs/1312.6034
2014 arXiv
-
[17]
Explaining the black-box smoothly—A counterfactual approach
Sumedha Singla, Motahhare Eslami, Brian Pollack, Stephen Wallace, and Kayhan Batmanghelich. Explaining the black-box smoothly—A counterfactual approach . Medical Image Analysis, 2023. ISSN 13618423. doi:10.1016/j.media.2022.102721. URL https://arxiv.org/abs/2101.04230
2023
-
[18]
Striving for Simplicity: The All Convolutional Net
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for Simplicity: The All Convolutional Net . In International Conference on Learning Representations Workshop, 2015. URL http://arxiv.org/abs/1412.6806
2015 arXiv
-
[19]
A Review of Deep Learning Techniques for Lung Cancer Screening and Diagnosis Based on CT Images
Mohammad A Thanoon, Mohd Asyraf Zulkifley, Muhammad Ammirrul Atiqi Mohd Zainuri, and Siti Raihanah Abdani. A Review of Deep Learning Techniques for Lung Cancer Screening and Diagnosis Based on CT Images. Diagnostics, 2023. ISSN 2075-4418. doi:10.3390/diagnostics13162617. URL h...
2023
-
[20]
Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR . SSRN Electronic Journal, 2017. doi:10.2139/ssrn.3063289. URL https://arxiv.org/abs/1711.00399
2017 arXiv
-
[21]
Meyer, Maurice Pradella, Daniel Hinck, Alexander W
Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth. TotalSegmentator: robust segmentation of 104 anatomical structures in CT images...
2022 arXiv
-
[22]
Ke Yan, Xiaosong Wang, Le Lu, and Ronald M. Summers. DeepLesion: Automated Deep Mining, Categorization and Detection of Significant Radiology Image Findings using Large-Scale Clinical Lesion Annotations . 2017. URL http://arxiv.org/abs/1710.01766
2017 arXiv
-
[23]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.