REVIEW 4 major objections 5 minor 28 references
CycleGAN-generated counterfactuals localize retinal disease, and a new CCAS score ties them to classifier evidence.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A CycleGAN-based counterfactual framework translates diseased retinal images to healthy-looking counterparts, and a new CCAS metric scores spatial agreement between the translation difference maps and classifier saliency.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection CCAS is a plausible new composite metric, but the headline alignment numbers rest on post-hoc filtering and a self-selected saliency reference; the paper deserves review with required revisions. the 4 major comments →
Counterfactual Explainability Framework With CycleGAN And Counterfactual-Classifier Alignnment Score for Retinal Disease Classification
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
On its own terms, the paper establishes that CycleGAN-based disease-to-normal translation produces counterfactual difference maps that overlap with classifier-yielded saliency regions in retinal fundus images. The central evidence is the CCAS protocol: for each pathological image, the channel-averaged absolute difference between the original and its generated healthy counterpart is radially weighted and normalized, then compared against an EigenCAM saliency map. Aggregated over 50 samples per class, the maps show a Spearman rank correlation of 0.95, a binary IoU at threshold 0.3 of 0.563, and a pointing accuracy of 0.547. The authors also show that filtering synthetic counterfactuals by CCAS
What carries the argument
The load-bearing object is the CCAS metric, which quantifies spatial agreement between a counterfactual difference map and a classifier saliency map. It combines three components: Spearman rank correlation for monotonic intensity agreement, binary IoU at a threshold for region overlap, and pointing accuracy to compare the location of peak response. The counterfactual difference map itself is produced by a CycleGAN generator that translates diseased fundus images into normal-looking counterparts, followed by channel-averaged absolute differencing, a center-focused radial mask to suppress boundary artifacts, and min-max normalization. The classifier saliency reference is EigenCAM, a covariance
Load-bearing premise
The evaluation assumes EigenCAM saliency maps are a valid proxy for clinically meaningful disease regions, so if those saliency maps reflect spurious cues rather than true pathology, high CCAS values would not indicate clinically valid explanations.
What would settle it
Compute CCAS against expert lesion-level masks on a retinal dataset with pixel annotations; if high-CCAS counterfactual difference maps fail to overlap the expert-annotated lesions, the claim of spatial consistency with classifier-relevant retinal evidence would be refuted. A second test: retrain the classifier on images with deliberately injected non-retinal artifacts and check whether CCAS stays high while the difference maps localize the artifacts.
If this is right
- If counterfactual difference maps reliably localize disease, clinicians gain pixel-level visualization of disease-associated changes without requiring lesion annotations.
- CCAS provides a single evaluation protocol that can rank saliency methods by spatial agreement with counterfactual evidence, independent of global image fidelity.
- CCAS-filtered counterfactual augmentation improves classification accuracy by about 2.1 percentage points over baseline while using roughly a quarter fewer synthetic images.
- The near-symmetric confidence shift after translation—disease confidence dropping and normal confidence rising equally—indicates the generator modifies disease-discriminative structure rather than just superficial appearance.
- The transferable CCAS results on an independent fundus dataset suggest the alignment generalizes beyond the training domain without parameter updates.
Where Pith is reading between the lines
- An editorial extension: CCAS could be adopted as a generic alignment metric for other image modalities (OCT, dermoscopy, chest X-ray), but its validity in those domains would depend on the choice of reference saliency method.
- A testable extension: replace EigenCAM with expert lesion masks as the reference in the CCAS protocol; this would directly test whether high CCAS equals clinical truth rather than merely classifier truth.
- If the classifier relies on spurious correlations such as imaging artifacts or borders, CCAS may reward explanations that match those artifacts; clinician-in-the-loop or artifact-controlled evaluation would expose such cases.
- The IoU>40% filtering threshold hints at a label-free curation rule for synthetic data, which could be generalized to other generative augmentation pipelines.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CounterFundus, a framework that combines an EfficientNet-B5 retinal disease classifier with a CycleGAN-based disease-to-normal counterfactual generator. For each pathological image, the difference between the original and the generated healthy counterpart is used as a localization map. The authors introduce the Counterfactual-Classifier Alignment Score (CCAS), which measures agreement between these difference maps and EigenCAM saliency maps using Spearman correlation, binary IoU, and pointing accuracy. The paper reports high classification performance (95.38% test accuracy), high internal CCAS values (Spearman 0.95, IoU@0.3 0.563, PA 0.547), and an ablation showing that CCAS-filtered counterfactual augmentation improves downstream accuracy. An external validation on RFMiD reports Spearman 0.48 and IoU 0.49. The central claim is that counterfactual explanations are spatially consistent with classifier-relevant retinal evidence and that the framework is clinically grounded.
Significance. If the claims were fully supported, CounterFundus would be a useful contribution: it offers a counterfactual explainability pipeline for multi-class fundus images, introduces a quantitative three-part alignment metric, and includes both an ablation study and external validation. The use of a public dataset, 5-fold cross-validation, locked test set, statistical tests (Kruskal-Wallis), and a clear pipeline are strengths. However, the evidence for the central claim is currently weakened by a post-hoc exclusion step whose size is unreported, by the use of a single saliency method as the reference without independent clinical validation, and by an external-validation result that is substantially below the internal numbers. These issues need to be resolved before the manuscript's broader interpretability and clinical-grounding claims can be accepted.
major comments (4)
- [§V.E / Table VII] The headline CCAS metrics are computed after excluding samples with difference-map standard deviation >0.35. The number of excluded samples per class is not reported, and no analysis without the filter or with alternative thresholds is given. If the filter preferentially removes hard or diffuse cases, the reported 0.95 Spearman and 0.563 IoU reflect a selected subset, not the framework generally. Please report exclusion rates per class, provide results without exclusion or as a sensitivity analysis, and justify the threshold.
- [§IV.D / §V.F / Table VIII] CCAS defines 'classifier-relevant evidence' via EigenCAM saliency, and EigenCAM is selected post hoc because it gives the highest CCAS (Table VIII). This creates a self-referential loop: the metric is agreement with an arbitrarily chosen saliency map, so high CCAS cannot by itself establish that the counterfactual maps are clinically meaningful. The manuscript explicitly defers lesion-level validation to future work; until then, the abstract's 'clinically-grounded' and 'spatially consistent with classifier-relevant retinal evidence' claims are too strong. Pre-specifying the reference saliency method or validating against expert/lesion annotations would address this.
- [§V.H] External validation on RFMiD reports Spearman ρ=0.48 and IoU=0.49, a large drop from the internal Spearman of 0.95; pointing accuracy is not reported, so the statement that the framework generalizes 'across all CCAS dimensions' is not supported. No sample size, exclusion criteria, or per-class breakdown is given for RFMiD. The authors should contextualize the drop, report all CCAS components, and avoid claiming transferability without a more detailed analysis.
- [§V.G / §IV.D] The CCAS-filtered augmentation (A3) uses the same classifier's EigenCAM saliency to select synthetic counterfactuals, then reports improved classification with those samples. This is a form of self-consistent filtering that may reinforce the classifier's existing biases rather than adding clinically independent information. The paper should clarify whether the filtering criterion is independent of the downstream classifier and discuss this potential circularity. The current framing presents the A3 improvement as validation of clinical relevance, which is not yet established.
minor comments (5)
- [§IV.D / Table VII] The text states that CCAS results are summarized with 95% confidence intervals, but Table VII contains only point estimates. Please add the intervals or remove the claim.
- [References] Several citations do not match the cited claim: reference [4] (Sharma et al.) is cited for GradCAM, and reference [20] (Zhao et al., TransFSM) is cited for EfficientNet-based backbones. These should be corrected.
- [Table IV] The FID value appears only in the first data row and again in the 'Overall' row, while the Diab. Ret. and Glaucoma rows are blank. Since FID is computed globally, this formatting is confusing; make it explicit.
- [§V.E / Eq. (8)] The pointing-accuracy threshold is 0.2×H in Eq. (8), while the CCAS evaluation uses a circular ROI of radius 0.35H. The relationship between these parameters should be explained, and the threshold choices should be justified or studied.
- [Table VI] Row T5 reports 'IoU prediction AUC – –' with no values or description. Either complete the test with actual results or remove it from the table.
Circularity Check
No significant circularity; the CCAS alignment is an empirical, externally checkable measurement, though the EigenCAM reference choice and sample exclusion are validity concerns rather than circular reductions.
full rationale
The central chain is: (i) an independently trained EfficientNet-B5 classifier (Sec. IV.A/E), (ii) a CycleGAN whose loss (Eq. 2) contains no classifier term and whose difference map is computed by Eqs. (3)-(5), and (iii) CCAS, defined by Eqs. (6)-(8) as standard rank-correlation/IoU/pointing agreement between the counterfactual difference map D and a saliency map S. The reported CCAS values (Table VII) are empirical measurements of that agreement, not consequences of the definitions: they could have been low, and indeed the RFMiD external validation (Sec. V.H) yields Spearman 0.48 and IoU 0.49, far below the internal 0.95/0.563, which would be impossible if the 'alignment' were forced by construction. The A2/A3 ablation (Table IX) measures test-set accuracy after CCAS-filtered augmentation; because the test set is locked and independent of the filter, the reported improvement is an empirical result, not a tautology. There are no load-bearing self-citations: references [8], [13], [18], [21] are external prior work, and no uniqueness theorem is imported from the authors. Two legitimate validity threats -- the post-hoc exclusion of samples with difference-map standard deviation >0.35 (Sec. V.E) and the use of EigenCAM as the sole reference without lesion-level annotations -- could inflate or bias the internal CCAS numbers, but they are not circular reductions: the metric would still be computable and could disagree with the claim. Overall score 1 reflects the mild self-referential flavor of choosing the best-scoring saliency method as the reference standard, but no part of the derivation reduces to its own inputs.
Axiom & Free-Parameter Ledger
free parameters (7)
- lambda_cyc =
7
- lambda_idt =
0.3
- IoU threshold tau =
0.3
- Pointing accuracy distance threshold =
0.2H
- ROI radius =
0.35H
- Sample exclusion threshold =
0.35
- Augmentation filter threshold =
IoU > 40%
axioms (4)
- domain assumption EigenCAM saliency maps are a valid proxy for clinically relevant retinal regions
- domain assumption The CycleGAN difference map, after radial weighting and normalization, localizes disease-associated changes
- domain assumption The Kaggle dataset labels are correct and representative
- standard math Spearman correlation, binary IoU, and pointing accuracy are appropriate and sufficient for spatial alignment
Cite this review
Pith. "Pith review of Counterfactual Explainability Framework With CycleGAN And Counterfactual-Classifier Alignnment Score for Retinal Disease Classification." pith.science (2026). https://pith.science/paper/XY4FJMCI
@misc{pith2026260721068,
author = {Pith},
title = {Pith review of: Counterfactual Explainability Framework With CycleGAN And Counterfactual-Classifier Alignnment Score for Retinal Disease Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/XY4FJMCI}},
note = {Machine review of arXiv:2607.21068}
}
read the original abstract
Automated detection of vision impairing retina-based ocular conditions from fundus images is important for early screening, timely referral and reducing dependency on specialist-only assessment, for which neural network-based deep learning (DL) models have been widely utilized. However, explainability of the DL frameworks remains a major bottleneck for clinical adoption, particularly when model decisions are not linked to retinal regions that are clinically meaningful. To address this issue, this study presents CounterFundus, a novel CycleGAN-driven counterfactual explainability framework, integrating EfficientNet-B5-based retinal disease detection with visually interpretable disease-to-normal fundus image translation. For each pathological image, the counterfactual yielded by the CycleGAN generator represents an estimated healthy counterpart and the resultant difference map is utilized to localize disease-associated retinal changes. Unlike conventional post-hoc saliency methods, CounterFundus provides counterfactual explanations through visually plausible disease-to-normal retinal translation. Thereafter, to quantify the spatial agreement between counterfactual difference maps and classifier saliency, the Counterfactual-Classifier Alignment Score (CCAS) is introduced, embedding Spearman correlation, binary IoU and pointing accuracy into a single assessment protocol. To this end, EigenCAM-aligned evaluation demonstrates that the generated counterfactual explanations remain spatially consistent with classifier-relevant retinal evidence across all CCAS dimensions. Along with that, ablation studies further confirm that CCAS-filtered counterfactual augmentation improves the downstream classification performance in fundus images, establishing CounterFundus as a clinically-grounded, explainable artificially intelligence (XAI) framework for retinal disease detection.
Figures
Reference graph
Works this paper leans on
-
[1]
Attention-embedded quadratic network (qttention) for effective and interpretable bearing fault diagnosis,
J.-X. Liao, H.-C. Dong, Z.-Q. Sun, J. Sun, S. Zhang, and F.-L. Fan, “Attention-embedded quadratic network (qttention) for effective and interpretable bearing fault diagnosis,”IEEE Transactions on Instrumen- tation and Measurement, vol. 72, pp. 1–13, 2023. 8 IEEE Journal and Transactions Template, V ol. XX, No. XX, 2026
2023
-
[2]
A virtual soil moisture sensor for smart farming using deep learning,
G. Patrizi, A. Bartolini, L. Ciani, V . Gallo, P. Sommella, and M. Carrat `u, “A virtual soil moisture sensor for smart farming using deep learning,” IEEE Transactions on Instrumentation and Measurement, vol. 71, pp. 1–11, 2022
2022
-
[3]
Cnn-based rolling bearing fault diagnosis method with quantifiable interpretability,
K. Jiang, Z. Yang, T. Jin, C. Chen, Z. Liu, and B. Zhang, “Cnn-based rolling bearing fault diagnosis method with quantifiable interpretability,” IEEE Transactions on Instrumentation and Measurement, 2025
2025
-
[4]
Improved saliency maps for interpretability of acoustic signal classification in condition monitor- ing of power transformers,
A. Sharma, H. Devarajan, and T. B. Shanker, “Improved saliency maps for interpretability of acoustic signal classification in condition monitor- ing of power transformers,”IEEE Transactions on Instrumentation and Measurement, 2025
2025
-
[5]
An analysis method for interpretability of convolutional neural network in bearing fault diagnosis,
L. Guo, X. Gu, Y . Yu, A. Duan, and H. Gao, “An analysis method for interpretability of convolutional neural network in bearing fault diagnosis,”IEEE Transactions on Instrumentation and Measurement, vol. 73, pp. 1–12, 2023
2023
-
[6]
Comprehensive analysis on machine learning approaches for inter- pretable and stable soft sensors,
L. Cao, J. Wang, J. Su, Y . Luo, Y . Cao, R. D. Braatz, and B. Gopaluni, “Comprehensive analysis on machine learning approaches for inter- pretable and stable soft sensors,”IEEE Transactions on Instrumentation and Measurement, 2025
2025
-
[7]
Counterfactual explanations without opening the black box: Automated decisions and the gdpr,
S. Wachter, B. Mittelstadt, and C. Russell, “Counterfactual explanations without opening the black box: Automated decisions and the gdpr,”
-
[8]
Assessing the (un)trustworthiness of saliency maps for localizing abnormalities in medical imaging,
N. Arun, N. Gaw, P. Singh, K. Chang, M. Aggarwal, B. Chen, K. Hoebel, S. Gupta, J. Patel, M. Gidwani, J. Adebayo, M. D. Li, and J. Kalpathy-Cramer, “Assessing the (un)trustworthiness of saliency maps for localizing abnormalities in medical imaging,” 2021. [Online]. Available: https://arxiv.org/abs/2008.02766
Pith/arXiv arXiv 2021
-
[9]
S. Mertes, T. Huber, K. Weitz, A. Heimerl, and E. Andr ´e, “GANTerfactual—counterfactual explanations for medical non-experts using generative adversarial learning,”Frontiers in Artificial Intelligence, vol. 5, p. 825565, 2022. [Online]. Available: https://doi.org/10.3389/ frai.2022.825565
arXiv 2022
-
[10]
V . Narayanaswamy, J. J. Thiagarajan, D. Rajan, and A. Spanias, “Counterfactual explanation and instance-generation using cycle- consistent generative adversarial networks,” 2023. [Online]. Available: https://arxiv.org/abs/2301.08939
Pith/arXiv arXiv 2023
-
[11]
M. Atad, D. Schinz, H. Moeller, R. Graf, B. Wiestler, D. Rueckert, N. Navab, J. S. Kirschke, and M. Keicher, “Counterfactual explanations for medical image classification and regression using diffusion autoencoder,”Machine Learning for Biomedical Imaging, vol. 2, no. iMIMIC 2023, pp. 2103–2125, Sep. 2024. [Online]. Available: http://dx.doi.org/10.59275/j....
-
[12]
L. ˇZigutyt˙e, T. Lenz, T. Han, K. J. Hewitt, N. G. Reitsam, S. Foersch, Z. I. Carrero, M. Unger, A. T. Pearson, D. Truhn, and J. N. Kather, “Counterfactual diffusion models for interpretable morphology-based explanations of artificial intelligence models in pathology,”bioRxiv, 2025, preprint. [Online]. Available: https://doi.org/10.1101/2024.10.29. 620913
-
[13]
Unpaired image-to-image translation using cycle-consistent adversarial networks,
J.-Y . Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” 2020. [Online]. Available: https://arxiv.org/abs/1703.10593
Pith/arXiv arXiv 2020
-
[14]
Residual- CycleGAN based camera adaptation for robust diabetic retinopathy screening,
D. Yang, Y . Yang, T. Huang, B. Wu, L. Wang, and Y . Xu, “Residual- CycleGAN based camera adaptation for robust diabetic retinopathy screening,” 2020. [Online]. Available: https://arxiv.org/abs/2007.15874
Pith/arXiv arXiv 2020
-
[15]
A. Sindel, A. Maier, and V . Christlein,A Vesselsegmentation- based CycleGAN for Unpaired Multi-modal Retinal Image Synthesis. Springer Fachmedien Wiesbaden, 2023, pp. 32–37. [Online]. Available: http://dx.doi.org/10.1007/978-3-658-41657-7 11
-
[16]
Selective synthetic augmentation with quality assurance,
Y . Xue, J. Ye, R. Long, S. Antani, Z. Xue, and X. Huang, “Selective synthetic augmentation with quality assurance,” 2019. [Online]. Available: https://arxiv.org/abs/1912.03837
Pith/arXiv arXiv 2019
-
[17]
Training calibration-based counterfactual explainers for deep learning models in medical image analysis,
J. J. Thiagarajan, K. Thopalli, D. Rajan, and P. Turaga, “Training calibration-based counterfactual explainers for deep learning models in medical image analysis,”Scientific Reports, vol. 12, no. 1, p. 597,
-
[18]
Visual explanations for the detection of diabetic retinopathy from retinal fundus images,
V . Boreikoet al., “Visual explanations for the detection of diabetic retinopathy from retinal fundus images,” inMedical Image Computing and Computer Assisted Intervention – MICCAI 2022, ser. Lecture Notes in Computer Science, L. Wang, Q. Dou, P. T. Fletcher, S. Speidel, and S. Li, Eds., vol. 13432. Cham: Springer, 2022, pp. 538–548. [Online]. Available: ...
-
[19]
An inherently interpretable AI model improves screening speed and accuracy for early diabetic retinopathy,
K. Djoumessi, Z. Huang, L. K ¨uhlewein, A. Rickmann, N. Simon, L. M. Koch, and P. Berens, “An inherently interpretable AI model improves screening speed and accuracy for early diabetic retinopathy,”medRxiv,
-
[20]
Transfsm: Fetal anatomy segmentation and biometric measurement in ultrasound images using a hybrid transformer,
L. Zhao, G. Tan, B. Pu, Q. Wu, H. Ren, and K. Li, “Transfsm: Fetal anatomy segmentation and biometric measurement in ultrasound images using a hybrid transformer,”IEEE Journal of Biomedical and Health Informatics, vol. 28, no. 1, pp. 285–296, 2023
2023
-
[21]
M. N. Bajwa, M. I. Malik, S. A. Siddiqui, A. Dengel, F. Shafait, W. Neumeier, and S. Ahmed, “Two-stage framework for optic disc localization and glaucoma classification in retinal fundus images using deep learning,”BMC Medical Informatics and Decision Making, vol. 19, no. 1, Jul. 2019. [Online]. Available: http: //dx.doi.org/10.1186/s12911-019-0842-8
-
[22]
Fundus image-based eye disease detection using EfficientNetB3 architecture,
R. Alsohemi and S. Dardouri, “Fundus image-based eye disease detection using EfficientNetB3 architecture,”Journal of Imaging, vol. 11, no. 8, p. 279, 2025. [Online]. Available: https://www.mdpi. com/2313-433X/11/8/279
2025
-
[23]
Eye diseases classification dataset,
G. V . Doddi, “Eye diseases classification dataset,” https://www.kaggle. com/datasets/gunavenkatdoddi/eye-diseases-classification, 2022, kaggle dataset
2022
-
[24]
Self-supervised underwater image generation for underwater domain pre-training,
Z. Wu, Z. Wu, X. Chen, Y . Lu, and J. Yu, “Self-supervised underwater image generation for underwater domain pre-training,”IEEE Transac- tions on Instrumentation and Measurement, vol. 73, pp. 1–14, 2024
2024
-
[25]
Retinal fundus multi-disease image dataset (rfmid),
S. Pachade, P. Porwal, D. Thulkar, M. Kokare, G. Deshmukh, V . Sahasrabuddhe, L. Giancardo, G. Quellec, and F. M ´eriaudeau, “Retinal fundus multi-disease image dataset (rfmid),” 2020. [Online]. Available: https://dx.doi.org/10.21227/s3g7-st65
-
[2018]
Available: https://arxiv.org/abs/1711.00399
[Online]. Available: https://arxiv.org/abs/1711.00399
-
[2022]
Available: https://doi.org/10.1038/s41598-021-04529-5
[Online]. Available: https://doi.org/10.1038/s41598-021-04529-5
-
[2024]
Available: https://www.medrxiv.org/content/early/2024/ 06/27/2024.06.27.24309574
[Online]. Available: https://www.medrxiv.org/content/early/2024/ 06/27/2024.06.27.24309574
2024
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.