REVIEW 4 major objections 5 minor 13 references
Analise de Desaprendizado de Maquina em Modelos de Classificacao de Imagens Medicas
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that SalUn-based machine unlearning removes selected data from medical image classifiers with accuracy close to full retraining, at about a twentieth of the runtime.
desk verdict SalUn on MedMNIST: a useful first benchmark, but the 'close to retraining' claim collapses on PathMNIST and the AG metric in Table 2 is internally inconsistent. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Saliency Unlearning (SalUn): a saliency-map-based unlearning procedure that identifies which model weights are most activated by the samples to be forgotten and selectively updates those weights so the model behaves as if it had never seen them. The paper applies it to a ResNet-18 trained for 200 epochs and runs unlearning for 10 epochs, evaluating with six metrics—UA, RA, TA, MIA, AG, and RTE—against the retrained model as the gold standard.
What would settle it
Run SalUn on the same three datasets with 20 independent random forget-set splits at the same 10% and 50% rates and record the distribution of Average GAP; if the spread across splits equals or exceeds the reported 0.5 to 4.9 point gap to retraining, the 'close to retraining' claim rests on an artifact of one random split rather than a stable effect.
Extended reading notes
Core claim
The paper's central finding, on its own terms, is that Saliency Unlearning (SalUn) behaves nearly like full retraining on PathMNIST, OrganAMNIST, and BloodMNIST. Average GAP—the mean absolute difference across the UA, RA, TA, and MIA metrics relative to the retrained model—was 0.53 to 3.95 at a 10% forget rate and 0.70 to 4.91 at a 50% forget rate, while SalUn ran in 1.1 to 8.9 minutes versus 22.2 to 160 minutes for retraining. The authors interpret this as an efficient solution for medical applications, with harder forgetting at higher removal rates and on the more complex pathology dataset, and with data augmentation helping in most configurations.
Load-bearing premise
The conclusion that SalUn is close to retraining assumes that the single random split of each dataset's training set into forget and keep sets—reported as point values with no repeated runs—is representative of unlearning quality for that dataset.
Editorial extensions
If this is right
- A trained medical-image classifier could honor data-removal requests by running SalUn instead of retraining, cutting runtime from tens of minutes to a few minutes.
- Unlearning quality degrades gracefully rather than catastrophically when the forget fraction goes from 10% to 50%; the worst average gap on PathMNIST stays below 5 percentage points.
- Data augmentation, especially adding RandomAug to the default crop-and-flip, lowers the average gap in most configurations, so augmentation is a practical lever for better unlearning.
- The same SalUn protocol transfers across three medical imaging modalities—histopathology, abdominal MRI, and blood microscopy—suggesting the method is not tied to one image type.
- Because remaining accuracy and test accuracy stay near retrained levels, medical classifiers can forget selected data without sacrificing overall classification performance.
- Higher forget rates and more complex datasets make unlearning harder, so deployment planning should budget for more unlearning epochs or additional augmentation when large fractions of data are removed.
Reading between the lines
- I would expect the method's advantage to shrink on full-resolution clinical images: the 64x64 inputs used here are far smaller than typical pathology or MRI slices, so saliency localization may be easier; a test on higher-resolution data would bound the effect.
- A natural extension is to combine SalUn with the label-smoothing and network-sparsity tricks the authors cite in related work, since those were shown to improve unlearning on general datasets and are orthogonal to saliency-based weight selection.
- The comparison to retraining uses a single random split with no variance reporting; before clinical deployment, repeated random forget splits and confidence intervals would tell whether the near-zero average gap is stable or an artifact of one split.
- If the near-retraining quality holds across splits, a practical certification pipeline could pair SalUn with membership-inference audits, letting hospitals demonstrate regulatory compliance without retraining.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates SalUn, a state-of-the-art approximate machine-unlearning method, on three MedMNIST medical image classification datasets (BloodMNIST, OrganAMNIST, PathMNIST) using a ResNet-18 backbone. It compares SalUn against full retraining using UA, RA, TA, MIA, AG, and RTE metrics at forget rates of 10% and 50%, and also studies the effect of data augmentation (none, default random crop+horizontal flip, and default plus RandomAug) on the AG metric. The central claim, stated in the abstract and conclusion, is that SalUn achieves performance close to full retraining and is an efficient solution for medical applications.
Significance. If the claim were substantiated, this would be a useful early evaluation of an existing unlearning method in a domain (medical imaging) where privacy compliance is important. The paper also raises the underexplored question of how data augmentation interacts with unlearning quality. However, the evidence provided is uneven: the paper's own Table 1 and Table 2 show a 10.28-point and 7.93-point testing-accuracy gap on PathMNIST, which is not 'close' to retraining for that dataset. Moreover, the AG metric that quantifies closeness is defined inconsistently with the numbers in the tables and is not reproducible from the reported data. Because the central claim depends directly on these tables and AG, the paper currently overstates its findings. The use of a recognized unlearning method and a standard benchmark, and the focus on RTE, are positive features, but the evaluation lacks repeated runs, error bars, and statistical testing.
major comments (4)
- [Abstract, §4, Tables 1 and 2, §5] The claim that SalUn 'achieves performance close to full retraining' is contradicted by the PathMNIST TA results: in Table 1 (δ=10%) TA is 77.49 vs 87.77 (10.28-point gap), and in Table 2 (δ=50%) TA is 83.87 vs 91.80 (7.93-point gap). For a 9-class medical classification task, an 8–10 point drop is not 'close'. The text should either remove the unqualified claim or clearly restrict it to BloodMNIST and OrganAMNIST and characterize PathMNIST as substantially degraded.
- [§3.3, Tables 1 and 2] The AG metric is not defined consistently with the reported values. §3.3 defines AG as the absolute value of the mean of the differences between UA, RA, TA, and MIA ('módulo da média das diferenças'), which would be |(d1+d2+d3+d4)/4|. But the table values for Table 1 match a mean of absolute differences: e.g., BloodMNIST (0.84+0.12+0.32+1.59)/4 = 0.72. In Table 2, the reported values match neither definition: BloodMNIST would be (1.25+0.07+0.29+3.16)/4 = 1.19, not 1.57; OrganAMNIST 0.54, not 0.70; PathMNIST 4.23, not 4.91. The definition in the text must be corrected and the tables recomputed, or a clear explicit formula for AG must be given and the numbers checked.
- [§4, Tables 1 and 2] All experiments appear to be run on a single random split of the training set into forget and remaining sets, with no repeated seeds, no error bars, and no significance tests. Given that the PathMNIST TA gap varies between 10.28 and 7.93 across forget rates, the reader cannot determine whether these differences are stable or artifacts of a particular split. At minimum, report mean ± std over at least 3–5 runs, or justify why a single split is representative for these datasets.
- [§4, Figure 2] The augmentation analysis is only presented as bar values of AG in Figure 2, with no numerical table, no standard deviation, and no description of how the unlearning was run under each augmentation scenario (e.g., whether the same forget set and same random seed were used). The statement that 'Default+RA showed improvement for most analyses' is therefore not verifiable. Please provide the full quantitative results for all three augmentation settings and the associated variance.
minor comments (5)
- [§3.2] Typo: 'RestNet-18' should be 'ResNet-18'. Also, sets are denoted inconsistently (D⟩, Di, Dr, Df); clean up notation.
- [§3.3] The metric list says AG uses 'MU' but this should be 'MIA'. Also the AG definition sentence is ambiguous and should be rewritten as a formula.
- [Throughout] The method name is inconsistently spelled 'Salun' and 'SalUn'; use one form.
- [Figure 1] Figure 1 is in Portuguese while the body is in English; this may be acceptable for the intended venue, but the caption and labels should at least be understandable or translated.
- [References] Some recent unlearning works on medical data are not discussed; if none exist, the claim that this is the first evaluation should be stated more carefully and supported by a more systematic search.
Circularity Check
No circularity: SalUn is evaluated against an independently retrained gold-standard model; no fitted parameter or definition is recycled as a prediction.
full rationale
The paper's claim ('SalUn achieves performance close to full retraining') is an empirical comparison. The unlearned model θu = SalUn(Df, Dr, θi) is produced by a published external method (Fan et al.), and the gold standard is a fresh ResNet-18 retrained for 200 epochs on Dr. The evaluation metrics (UA, RA, TA, MIA, AG, RTE) are all measured against that independent retrained model; AG is a summary of differences between the two models' metrics, so it is a comparison statistic, not an input to SalUn or to retraining. No parameter is fitted to a subset and then renamed a prediction, no equation reduces the conclusion to its definition, and the few citations (SalUn, MedMNIST, augmentation survey) are external prior work, not self-citations by the present authors. The internal inconsistency between the stated definition of AG ('módulo da média das diferenças') and the reported values (which behave like a mean of absolute differences) is a correctness/reproducibility concern, not circularity. Similarly, the large PathMNIST TA gap (10.28 and 7.93 points) suggests the abstract's 'close to full retraining' is overstated, but overclaiming is not the same as a circular derivation. The derivation chain is self-contained: input data and models are external, the comparison target is independently retrained, and the reported outcome is not assumed by construction.
Assumptions & free parameters
assumptions (6)
- domain assumption Machine unlearning has not been previously evaluated on medical image classification (literature gap assumption).
- domain assumption SalUn is a state-of-the-art unlearning method.
- domain assumption The metrics UA, RA, TA, MIA and the aggregate AG are valid and sufficient measures of unlearning quality.
- domain assumption Randomly selected forget subsets at 10% and 50% represent realistic right-to-be-forgotten requests.
- domain assumption The three MedMNIST datasets represent the difficulty spectrum of medical image classification.
- domain assumption Single-run measurements without variance estimates can support comparative claims about unlearning quality.
Cite this review
Pith. "Pith review of Analise de Desaprendizado de Maquina em Modelos de Classificacao de Imagens Medicas." pith.science (2026). https://pith.science/paper/D4W3TFY2
@misc{pith2026250818509,
author = {Pith},
title = {Pith review of: Analise de Desaprendizado de Maquina em Modelos de Classificacao de Imagens Medicas},
year = {2026},
howpublished = {\url{https://pith.science/paper/D4W3TFY2}},
note = {Machine review of arXiv:2508.18509}
}
read the original abstract
Machine unlearning aims to remove private or sensitive data from a pre-trained model while preserving the model's robustness. Despite recent advances, this technique has not been explored in medical image classification. This work evaluates the SalUn unlearning model by conducting experiments on the PathMNIST, OrganAMNIST, and BloodMNIST datasets. We also analyse the impact of data augmentation on the quality of unlearning. Results show that SalUn achieves performance close to full retraining, indicating an efficient solution for use in medical applications.
Reference graph
Works this paper leans on
-
[1]
Chan, H.-P., Samala, R. K., Hadjiiski, L. M., and Zhou, C. (2020). Deep learning in medical image analysis. Deep learning in medical image analysis: challenges and applications , pages 3--21
work page 2020
-
[2]
Dang, Q.-V. (2021). Right to be forgotten in the age of machine learning. In Advances in Digital Science: ICADS 2021 , pages 403--411. Springer
work page 2021
-
[3]
Di, Z., Zhu, Z., Jia, J., Liu, J., Takhirov, Z., Jiang, B., Yao, Y., Liu, S., and Liu, Y. (2024). Label smoothing improves machine unlearning. arXiv preprint arXiv:2406.07698
arXiv 2024
-
[4]
Fan, C., Liu, J., Zhang, Y., Wong, E., Wei, D., and Liu, S. (2024). Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations (ICLR)
work page 2024
-
[5]
Golatkar, A., Achille, A., and Soatto, S. (2020). Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9304--9312
work page 2020
-
[6]
Graves, L., Nagisetty, V., and Ganesh, V. (2021). Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11516--11524
work page 2021
-
[7]
J., Van Der Sloot, B., and Borgesius, F
Hoofnagle, C. J., Van Der Sloot, B., and Borgesius, F. Z. (2019). The european union general data protection regulation: what it is and what it means. Information & Communications Technology Law , 28(1):65--98
work page 2019
-
[8]
Jia, J., Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y., Sharma, P., and Liu, S. (2023). Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems , 36:51584--51605
work page 2023
Show all 13 references
-
[9]
and Mumuni, F
Mumuni, A. and Mumuni, F. (2022). Data augmentation: A comprehensive survey of modern approaches. Array , 16:100258
2022
-
[10]
Wu, Z., Shen, C., and Van Den Hengel, A. (2019). Wider or deeper: Revisiting the resnet model for visual recognition. Pattern recognition , 90:119--133
2019
-
[11]
Yang, J., Shi, R., Wei, D., Liu, Z., Wang, L., Zhou, Y., Zhou, S., Bian, C., Li, L., Wang, X., et al. (2021). Medmnist: A lightweight automl benchmark for medical image analysis. https://medmnist.com. Accessed: February 13, 2025
2021
-
[12]
Zhang, H., Nakamura, T., Isohara, T., and Sakurai, K. (2023). A review on machine unlearning. SN Computer Science , 4(4):337
2023
-
[13]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.