REVIEW 5 major objections 4 minor 29 references
Robust Training with Data Augmentation for Medical Imaging Classification
T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read RTDA claims one training objective can resist both adversarial attacks and distribution shifts across three medical imaging modalities without losing clean accuracy.
desk verdict One-line tweak to RobustAugMix that trades clean CE for adversarial CE; useful but the abstract overstates a mostly second-best result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the RTDA objective, $\min_\theta \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[\mathcal{L}_{\mathrm{CE}}(f_\theta(x+\delta^*),y)+\lambda\,\mathcal{L}_{\mathrm{JSD}}(f_\theta(x),f_\theta(g_{\mathrm{Aug}}(x)),f_\theta(x+\delta^*))\big]$, with $\delta^*=\arg\max_{\|\delta\|_p<\epsilon}\mathcal{L}_{\mathrm{CE}}(f_\theta(x+\delta),y)$ approximated by projected gradient descent, and $g_{\mathrm{Aug}}$ the AugMix stochastic augmentation chain. It does the work by putting the primary cross-entropy loss on adversarial examples, hardening the decision boundary directly against worst-case perturbations, while the Jensen-Shannon consistency term requires aligned predictions for clean, AugMix-augmented, and adversarial views of the same image. This combination is what lets one model carry both adversarial robustness and invariance to natural variations.
What would settle it
Evaluate RTDA and the same baselines on data from a different scanner or clinical site rather than contrast-modified copies of the training images, measuring Brier score under that shift and adversarial accuracy; if RTDA is no longer best or second-best on both, the central claim fails. Re-running the POCUS experiment with a video-level split would also show whether the ultrasound robustness gap survives when frames from the same video cannot appear in both train and test.
Extended reading notes
Core claim
The paper claims that replacing the clean cross-entropy term in RobustAugMix with cross-entropy on adversarial examples, while retaining the Jensen-Shannon consistency among clean, AugMix-augmented, and adversarial views, yields a model that is robust under PGD attacks and under contrast-based distribution shift. In their experiments, RTDA is consistently among the top methods: it tracks or exceeds adversarial training on adversarial accuracy, tracks or exceeds AugMix and DataAug on shifted-data Brier score, and keeps clean accuracy high on the X-ray and mammogram datasets. The ultrasound results show RTDA staying above 70% adversarial accuracy at large epsilon where RobustAugMix drops to about 30%, at the cost of a small clean-accuracy reduction. The paper states these results as evidence that robustness to intentional and natural perturbations can be combined rather than traded off.
Load-bearing premise
The distribution-shift conclusions rest on contrast changes standing in for real-world variation, and the ultrasound result rests on the train/test split not mixing frames from the same video; if either assumption fails, the measured generalization could be optimistic.
Editorial extensions
If this is right
- Deployment pipelines can train one model with RTDA and get both adversarial robustness and distribution-shift robustness at once, rather than ensembling separate defenses.
- RTDA can be adopted without changing the architecture or the test-time procedure; only the training loss changes.
- On the X-ray and mammogram datasets, the added robustness does not come out of normal performance, since clean accuracy stays at or near the standard-training level.
- Across three imaging modalities, the same objective works without task-specific architectural changes, suggesting a general medical-imaging training recipe.
- In the ultrasound setting, RTDA keeps adversarial accuracy above 70% at the largest tested perturbation, while the closest augmentation-based competitor drops to roughly 30%.
Reading between the lines
- If the contrast proxy holds, the RTDA recipe of putting the main loss on adversarial inputs plus a consistency term across clean, augmented, and adversarial views is a general template that could extend to other safety-sensitive vision tasks.
- The JSD term in RTDA is not limited to one shift type; the paper's POCUS variant already feeds low-contrast and high-contrast views into the same consistency loss, so adding blur, noise, or scanner-specific augmentations and testing whether the balance holds is a natural next experiment.
- A fairer test of the generalization claim would use a held-out site or device rather than contrast-modified copies of the same images, separating genuine domain transfer from augmentation robustness.
- Since only PGD-style attacks are measured, checking RTDA against adaptive attacks or attacks designed for its own loss would clarify whether the adversarial robustness generalizes beyond the attack family used in training.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RTDA, a training objective that modifies the authors' prior RobustAugMix method by applying the cross-entropy term to adversarial examples rather than clean inputs, while retaining a Jensen-Shannon divergence consistency loss among clean, augmented, and adversarial views. The method is evaluated on three medical imaging classification tasks (COVID-19 X-ray, mammogram breast cancer, POCUS ultrasound) against six baselines: ERM, adversarial training (AT), adversarial logit pairing-style AdvL, DataAug, AugMix, and RobustAugMix. Performance is reported as adversarial accuracy versus perturbation strength and as Brier score on clean and low/high-contrast shifted data. The abstract claims that RTDA achieves superior adversarial robustness and improved distribution-shift generalization in each task while maintaining high clean accuracy.
Significance. If the stated claim were fully supported, RTDA would be a practical, drop-in training objective for medical imaging classifiers that balances adversarial robustness, distribution-shift robustness, and clean accuracy. The paper has notable strengths: it evaluates on three distinct imaging modalities, includes six comparison methods, trains five models per method, and uses an external benchmark design rather than deriving results from the method itself. The central methodological idea is simple and clearly motivated by a known weakness of RobustAugMix. However, the evidence as reported does not support the abstract's universal superiority claim: the paper's own results text concedes that AT outperforms RTDA on adversarial accuracy for X-ray and POCUS, and that DataAug and AugMix often achieve lower Brier scores under contrast shift. The POCUS evaluation also rests on an unverified train/test split and an incompletely specified training objective. These issues are correctable but require substantive revision of both the claims and the experimental reporting.
major comments (5)
- [Abstract / §5.1 / §5.3 / §6] The abstract states that RTDA 'achieves superior robustness against adversarial attacks and improved generalization performance in the presence of distribution shift in each image classification task,' but the results sections contradict this wording. In §5.1, AT surpasses RTDA on the X-ray adversarial-accuracy curves by 3-5% (Fig. 2a); in §5.3, RTDA 'never outperforms AT on adversarial robustness' for POCUS (Fig. 4a); and in Figs. 2b-4b, DataAug or AugMix often achieve lower Brier scores on shifted data. The conclusion itself describes RTDA as 'the best or second-best performer' and 'often achieved second-best performance under distribution shift.' The abstract and conclusion therefore need to be rewritten to state the actual finding: RTDA offers a balance between adversarial robustness and distribution-shift robustness, not universal superiority on every task and metric.
- [§4.1, POCUS dataset description] The POCUS dataset is described as '3119 frames from 195 ultrasound videos, split similarly into 80% for training and 20% for testing.' The manuscript does not state whether the split was performed at the video level so that all frames from one video appear in only one partition. If the split was by frame, then near-duplicate frames from the same ultrasound video can appear in both training and test sets, which would inflate clean accuracy, adversarial accuracy, and shifted-data Brier scores in Fig. 4. Since the paper claims a result 'in each image classification task,' the authors must either confirm that the split was video-level or repeat the POCUS evaluation with such a split. The absence of code and data makes this unverifiable from the manuscript alone.
- [§5.3 vs. §3.7, Eq. (10)] The POCUS training objective is not fully specified. Section 5.3 states that 'we had to generate an additional augmented sample with low-contrast and add their probability distributions to the JSD loss,' but Eq. (10) defines the RTDA objective with a three-way JSD over clean, AugMix-augmented, and adversarial inputs. The manuscript does not provide the exact loss used for POCUS, including how the low-contrast sample enters the JSD term and whether the JSD is computed over four distributions or as additional separate consistency terms. This omission prevents reproduction of the POCUS experiments and weakens the support for the paper's 'each image classification task' claim.
- [§4.3, Eq. (12)] Equation (12) defines the Brier score only for binary outcomes, with p_i the 'predicted probability for the positive class' and y_i in {0,1}. Two of the three tasks (X-ray and POCUS) are three-class problems. No multiclass extension of the Brier score is given, so as written the metric is not defined for those datasets and the Brier-score comparisons in Figs. 2b and 4b may be invalid. The authors need to state the exact multiclass formulation used, for example averaging the squared error over all classes.
- [§4.1 / §5, distribution-shift evaluation] The only distribution shift tested is a change in image contrast (low- and high-contrast augmentation). The abstract and conclusion speak of robustness to 'natural variations' and 'distribution shift' more generally, but the experimental evidence covers a single shift type. If the intended claim is about natural variations in general, then additional shift types (e.g., Gaussian noise, blur, resolution changes, or protocol differences) are needed, or the claims must be restricted to contrast shift. This distinction matters because the paper's central contribution is framed around distribution-shift robustness, not just contrast robustness.
minor comments (4)
- [§3.5, Eq. (7)] In Eq. (7), the two augmented samples are both written as f_theta(g_Aug(x)); presumably these are independent stochastic draws from the augmentation distribution. Please clarify the notation so that the two terms are visibly distinct.
- [§4.2 and §3.7] The values of the JSD weight lambda, the contrast augmentation strength, and the PGD step size used during training and testing are not reported. Since the paper does not provide code or data, these hyperparameters are essential for reproducibility.
- [§4.2] The text says the best hyperparameters were found for the standard model and then held fixed for the robust methods. This is a reasonable protocol, but the manuscript should state explicitly whether lambda and epsilon were also held fixed rather than tuned per method, since several losses depend on these values.
- [Figures 2-4] The shaded regions and error bars are described as standard deviation, but the manuscript does not specify whether this is the standard deviation across the five cross-validation models or across random seeds. Please state this in the figure captions or in §4.3.
Circularity Check
No significant circularity: RTDA is an empirical modification of RobustAugMix, benchmarked against external baselines.
full rationale
The paper makes no first-principles derivation; its central claim is an empirical comparison. RTDA's objective in Eq. (10) is explicitly defined as a modification of RobustAugMix (Eq. 9), changing only the cross-entropy term to apply to adversarial samples while retaining the three-way JSD consistency term. This is a testable proposal, not a quantity fitted to the evaluation. RobustAugMix and DataAug are authors' prior work, but they appear as baselines and background, and the main result is supported by direct measurements on three public datasets against several independent methods (ERM, AT, AdvL, AugMix, DataAug). No uniqueness theorem, fitted parameter renamed as prediction, or ansatz smuggled through self-citation carries the argument. The overlap between the contrast augmentations used in training for augmentation-based methods and the contrast-shift test set is an evaluation-design limitation, not a circular derivation, because the clean and adversarial results are also measured and the comparison remains externally meaningful. Therefore no circularity step meets the evidentiary bar.
Assumptions & free parameters
free parameters (4)
- lambda (JSD loss weight)
- epsilon (PGD perturbation bound) =
2 (X-ray), 0.15 (mammogram), 0.3 (POCUS)
- PGD steps =
7 training, 20 testing
- Contrast augmentation strength
assumptions (3)
- domain assumption Contrast shift is a representative distribution shift for medical imaging.
- domain assumption Train/test splits are independent; in particular POCUS frames from the same video are not shared.
- domain assumption PGD with an l_p bound is a sufficient adversary for evaluating robustness.
Cite this review
Pith. "Pith review of Robust Training with Data Augmentation for Medical Imaging Classification." pith.science (2026). https://pith.science/paper/HICTR5JU
@misc{pith2026250617133,
author = {Pith},
title = {Pith review of: Robust Training with Data Augmentation for Medical Imaging Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/HICTR5JU}},
note = {Machine review of arXiv:2506.17133}
}
read the original abstract
Deep neural networks are increasingly being used to detect and diagnose medical conditions using medical imaging. Despite their utility, these models are highly vulnerable to adversarial attacks and distribution shifts, which can affect diagnostic reliability and undermine trust among healthcare professionals. In this study, we propose a robust training algorithm with data augmentation (RTDA) to mitigate these vulnerabilities in medical image classification. We benchmark classifier robustness against adversarial perturbations and natural variations of RTDA and six competing baseline techniques, including adversarial training and data augmentation approaches in isolation and combination, using experimental data sets with three different imaging technologies (mammograms, X-rays, and ultrasound). We demonstrate that RTDA achieves superior robustness against adversarial attacks and improved generalization performance in the presence of distribution shift in each image classification task while maintaining high clean accuracy.
Figures
Reference graph
Works this paper leans on
-
[1]
BMC bioinformatics 20, 1–20 (2019)
Abdelhafiz, D., Yang, C., Ammar, R., Nabavi, S.: Deep convolutional neural networks for mammography: advances, challenges and applications. BMC bioinformatics 20, 1–20 (2019)
work page 2019
-
[2]
Medical Physics 49(6), 3654–3669 (2022)
Bai, J., Jin, A., Wang, T., Yang, C., Nabavi, S.: Feature fusion siamese network for breast can- cer detection comparing current and prior mammograms. Medical Physics 49(6), 3654–3669 (2022). DOI https://doi.org/10.1002/mp.15598. URL https://aapm.onlinelibrary. wiley.com/doi/abs/10.1002/mp.15598
-
[3]
Applied Sciences 11(2), 672 (2021)
Born, J., Wiedemann, N., Cossio, M., Buhre, C., Br ¨andle, G., Leidermann, K., Aujayeb, A., Moor, M., Rieck, B., Borgwardt, K.: Accelerating detection of lung pathologies with explainable ultrasound image analysis. Applied Sciences 11(2), 672 (2021). DOI 10.3390/ app11020672. URL http://dx.doi.org/10.3390/app11020672
-
[4]
Monthly weather review 78(1), 1–3 (1950)
Brier, G.W.: Verification of forecasts expressed in terms of probability. Monthly weather review 78(1), 1–3 (1950)
work page 1950
-
[5]
Cohen, J.P., Morrison, P., Dao, L., Roth, K., Duong, T.Q., Ghassemi, M.: Covid-19 image data collection: Prospective predictions are the future. arXiv 2006.11988 (2020). URL https://github.com/ieee8023/covid-chestxray-dataset
arXiv 2020
-
[6]
Artificial Intelligence in Medicine 132, 102386 (2022)
Garrucho, L., Kushibar, K., Jouide, S., Diaz, O., Igual, L., Lekadir, K.: Domain generalization in deep learning based mass detection in mammography: A large-scale multi-center study. Artificial Intelligence in Medicine 132, 102386 (2022)
work page 2022
-
[7]
arXiv preprint arXiv:1412.6572 (2014)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
arXiv 2014
-
[8]
In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). DOI 10.1109/CVPR.2016.90
Show all 29 references
-
[9]
URL https://arxiv.org/abs/1903.12261
Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations (2019). URL https://arxiv.org/abs/1903.12261
2019 arXiv
-
[10]
arXiv preprint arXiv:1912.02781 (2019)
Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 (2019)
2019 arXiv
-
[11]
CVPR (2022)
Hendrycks, D., Zou, A., Mazeika, M., Tang, L., Li, B., Song, D., Steinhardt, J.: Pixmix: Dreamlike pictures comprehensively improve safety measures. CVPR (2022)
2022
-
[12]
JCO Clinical Cancer Informatics 6, e2100170 (2022)
Joel, M.Z., Umrao, S., Chang, E., Choi, R., Yang, D.X., Duncan, J.S., Omuro, A., Herbst, R., Krumholz, H.M., Aneja, S.: Using adversarial images to assess the robustness of deep learning models trained on diagnostic images in oncology. JCO Clinical Cancer Informatics 6, e21001...
2022
-
[13]
Kannan, H., Kurakin, A., Goodfellow, I.: Adversarial logit pairing (2018)
2018
-
[14]
Heliyon p
Kansal, K., Krishna, P.S., Jain, P.B., Surya, R., Honnavalli, P., Eswaran, S.: Defending against adversarial attacks on covid-19 classifier: A denoiser-based approach. Heliyon p. e11209 (2022)
2022
-
[15]
Korean journal of radiology 20(3), 405–410 (2019)
Kim, D.W., Jang, H.Y., Kim, K.W., Shin, Y., Park, S.H.: Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean journal of radiology 20(3), 405–4...
2019
-
[16]
CoRR abs/2109.09658 (2021)
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Auss ´o, S., Alberich, L.C., Marias, K., Tsiknakis, M., Colantonio, S., Papanikolaou, N., Salahuddin, Z., Woodruff, H.C., Lambin, P., Mart´ı-Bonmat´ı, L.: FUTURE-AI: guiding principles and consensus rec...
2021 arXiv
-
[17]
In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp
Li, X., Zhu, D.: Robust detection of adversarial attacks on medical images. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 1154–1158. IEEE (2020)
2020
-
[18]
arXiv preprint arXiv:2304.10226 (2023) 16 Josu ´e Mart´ınez-Mart´ınez et al
Li, Z., Cui, Z., Zhang, L., Wang, S., Lei, C., Ouyang, X., Chen, D., Zhuang, Z., Zhao, X., Gu, Y., et al.: Domain generalization for mammographic image analysis via contrastive learning. arXiv preprint arXiv:2304.10226 (2023) 16 Josu ´e Mart´ınez-Mart´ınez et al
2023 arXiv
-
[19]
Pattern Recognition 110, 107332 (2021)
Ma, X., Niu, Y., Gu, L., Wang, Y., Zhao, Y., Bailey, J., Lu, F.: Understanding adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition 110, 107332 (2021)
2021
-
[20]
arXiv preprint arXiv:1706.06083 (2017)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
2017 arXiv
-
[21]
In: NeurIPS ML Safety Workshop (2022)
Mart ´ınez-Mart´ınez, J., Brown, O.: Robustaugmix: Joint optimization of natural and adversarial robustness. In: NeurIPS ML Safety Workshop (2022). URL https://openreview.net/ forum?id=8MfPfECiFET
2022
-
[22]
Proceedings of the AAAI Conference on Arti- ficial Intelligence 38(21), 23579–23581 (2024)
Mart ´ınez-Mart´ınez, J., Brown, O., Caceres, R.: Towards robustness to natural variations and distribution shift (student abstract). Proceedings of the AAAI Conference on Arti- ficial Intelligence 38(21), 23579–23581 (2024). DOI 10.1609/aaai.v38i21.30481. URL https://ojs.aaai...
2024 doi
-
[23]
In: 2023 IEEE Conference on Artificial Intelligence (CAI), pp
Mart ´ınez-Mart´ınez, J., Nabavi, S.: Addressing vulnerability in medical deep learning through robust training. In: 2023 IEEE Conference on Artificial Intelligence (CAI), pp. 341–342 (2023). DOI 10.1109/CAI54212.2023.00150
2023
-
[24]
In: 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp
Pierce, T.T., Ottensmeyer, M.P., Som, A., Brattain, L.J., Werblin, J.S., Sutphin, P.D., Schoen, S., Johnson, M.R., Gjesteby, L., Telfer, B.A., Samir, A.E.: Individualized ultrasound-guided intervention phantom development, fabrication, and proof of concept*. In: 2023 45th Annu...
2023
-
[25]
URL https: //arxiv.org/abs/1609.04747
Ruder, S.: An overview of gradient descent optimization algorithms (2017). URL https: //arxiv.org/abs/1609.04747
2017 arXiv
-
[26]
CoRR abs/2201.05647 (2022)
Soklaski, R., Goodwin, J., Brown, O.M., Yee, M., Matterer, J.: Tools and practices for re- sponsible AI engineering. CoRR abs/2201.05647 (2022). URL https://arxiv.org/abs/ 2201.05647
2022 arXiv
-
[27]
IEEE transactions on neural networks 10(5), 988–999 (1999)
Vapnik, V.N.: An overview of statistical learning theory. IEEE transactions on neural networks 10(5), 988–999 (1999)
1999
- [28]
-
[29]
In: International Conference on Machine Learning (2019)
Zhang, H., Yu, Y., Jiao, J., Xing, E.P., Ghaoui, L.E., Jordan, M.I.: Theoretically principled trade-off between robustness and accuracy. In: International Conference on Machine Learning (2019)
2019
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.