REVIEW 4 major objections 4 minor 1 cited by
An analysis of data variation and bias in image-based dermatological datasets for machine learning classification
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Fine-tuning on 30% of clinical data closes most of the dermoscopy-to-clinic gap.
desk verdict Useful confirmation of a known fine-tuning result, but Table 3's key comparison relies on an undocumented PAD split and DDI class mapping. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is transfer learning via fine-tuning: models are first pretrained on ImageNet, then trained on the dermoscopic ISIC18 dataset to produce FDerm, and then a subset is re-trained on clinical PAD-UFES-20 images to produce FineClinic. The comparison table across FDerm, FClinic and FineClinic evaluated on ISIC18, PAD and DDI is the load-bearing instrument, because it isolates the effect of adding a small amount of clinical data to a dermoscopic base. Data augmentation (random flips, rotations, colour jitter, crops and affine transforms) is applied during fine-tuning to counter class imbalance and image noise, which the paper treats as essential for the minority malignant classes.
What would settle it
Reproduce FDerm, FClinic and FineClinic on a single public split of PAD-UFES-20 with one explicit class mapping shared by ISIC18, PAD and DDI, and test all models on the same held-out partition; if FineClinic's clinical F1 no longer lands near FClinic's, or if the PAD-trained model's F1 on DDI departs from roughly 11.80, the central efficiency claim fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is an efficiency and a limit. FineClinic—a model initialized on dermoscopic ISIC18 and then fine-tuned on a 30% subset of PAD-UFES-20—reaches an F1-score of 67.18 on clinical PAD data, almost identical to the 68.95 achieved by FClinic, a model trained directly on 50% of the same clinical data, while a purely dermoscopic FDerm model collapses to 5.22 on clinical images. The same experiment also finds that FClinic generalizes poorly to a second clinical dataset, DDI, with F1=11.80, indicating that two 'clinical' collections can be as far apart as dermoscopy and clinic are. These results are averaged across CNN architectures (ConvNext, DenseNet, ResNet, EfficientNet and InceptionV3), and the paper argues they show that combining a large dermoscopic source with a small clinical fine-tuning set is a more data-efficient route to clinical classification than training on clinical data alone, while cautioning that no single clinical dataset yet guarantees transfer to another.
Load-bearing premise
The entire comparison rests on the unstated assumption that the PAD and DDI evaluation splits and the class-label mappings across ISIC18, PAD and DDI are identical across all models; the paper never specifies the held-out test set or the mapping, so if those differ the headline 30%-versus-50% and DDI numbers are not comparable.
Editorial extensions
If this is right
- If the claim is right, a deployment recipe for smartphone-based skin-lesion screening is: take a dermoscopic-trained model and fine-tune it on a few hundred carefully chosen clinical images, rather than collecting a large clinical dataset from scratch.
- Dermoscopic-only models should not be used directly in clinical workflows: the F1 of 5.22 on PAD shows they effectively fail on uncontrolled phone photos.
- Evaluation across multiple clinical datasets is necessary, because a model trained on one clinical set (PAD) can score only 11.80 F1 on another (DDI), so single-dataset clinical numbers are not trustworthy.
- Class-imbalance-aware augmentation during fine-tuning matters: without it, the model would simply predict the majority 'nevus' class, hiding poor malignant-class accuracy behind acceptable averages.
Reading between the lines
- A testable extension the paper leaves implicit: sweep the fine-tuning fraction (e.g., 10%, 20%, 30%, 40%, 50%) and plot clinical F1 against it to locate the knee of the curve; if the 30%-equals-50% finding holds, the knee is likely below 30%, which would make data collection even cheaper.
- The F1=11.80 cross-clinical result suggests 'clinical' is not one domain; capture conditions such as resolution, distance, illumination and reflectance may dominate over skin-tone diversity. A controlled experiment that varies only the capture device while holding skin-tone distribution fixed would separate these confounds.
- The paper's demographic-bias framing and acquisition-shift framing are entangled in its data; one could test the paper's implicit claim that dermoscopic models fail on clinical images largely due to domain shift by fine-tuning on a skin-tone-stratified 30% subset versus a random 30% subset and comparing fairness metrics.
- If replicated, the result implies that benchmark design should report both the dermoscopic-to-clinical transfer gain and clinical-to-clinical transfer loss, since either number alone can mislead about real-world readiness.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the domain gap between dermoscopic and clinical skin-lesion images for CNN classification. Using the public ISIC18, PAD-UFES-20, and DDI datasets, the authors train ImageNet-initialized models on dermoscopic data (FDerm), on clinical data (FClinic), and fine-tune dermoscopic-trained models on 30% of the clinical data (FineClinic). The central quantitative claims are that FDerm achieves only 5.22 F1 on PAD, FineClinic reaches 67.18 F1 on PAD with 30% of the training images, which is comparable to FClinic's 68.95 with 50% of the images, and that PAD-trained models transfer poorly to DDI, with F1 around 11.80. The paper also reports class-level accuracy and discusses dataset biases, data imbalance, and the limitations of clinical deployment.
Significance. If the quantitative results were fully auditable, the paper would provide useful evidence on a clinically relevant question: how much dermoscopic pretraining helps in low-resource clinical settings, and how far clinical models generalize across clinical datasets. The qualitative finding that dermoscopic-trained models fail on clinical images and that fine-tuning on a small clinical subset substantially improves performance is plausible and consistent with prior work. Strengths of the manuscript include the use of public datasets, the comparison of several architectures, and the explicit attention to class imbalance and per-class performance. However, the precise numerical claims in Table 3 are currently under-supported because the evaluation protocol is underspecified, no variance estimates are reported, and the class-label mapping between datasets is not defined. These issues prevent the paper, in its present form, from supporting the specific quantitative conclusions it advertises.
major comments (4)
- [§3.3, footnote 4, and Table 3] The PAD-UFES-20 train/test protocol is never fully specified. Footnote 4 states that PAD has no official split and that 50% of the data was used for training, but it does not define the held-out test partition, state whether the test partition is the same for FDerm, FClinic, and FineClinic, describe how stratification is performed, or specify whether a validation subset is carved out separately. Since FineClinic uses 30% and FClinic uses 50% of the same data, any difference in the test split or in class stratification could explain the 1.77 F1-point gap reported in Table 3. This is load-bearing for the paper's central claim that fine-tuning on fewer clinical images reaches performance comparable to training on more clinical images.
- [§4.2, Table 3, and §4.3] The DDI evaluation is not auditable because the class-label mapping is never described. PAD-UFES-20 and DDI use different label schemes, and the paper reports DDI F1 values without stating how lesion-type classes are mapped to DDI labels, whether some classes are filtered out, or how the metric is averaged. If the mapping is inconsistent, the DDI F1 values in Table 3, including the repeated 11.80 values, lose their meaning. The clinical-to-clinical generalization claim in §4.3 therefore rests on an unspecified evaluation protocol.
- [§3.3 and Table 3] All results appear to come from single training runs, with no error bars, no multiple seeds, and no measure of run-to-run variance. The training setup also raises concerns: a learning rate of 1e-1 with a cosine annealing scheduler that decreases every 10 epochs is unusual for fine-tuning large CNN models, and no justification or learning-rate sensitivity analysis is provided. Without variance estimates, the central comparison between FClinic's 68.95 and FineClinic's 67.18 cannot be distinguished from noise, especially given the small clinical training sets.
- [Abstract, §5, and Table 3] The paper promises to show how to 'combine' data from divergent distributions, but the experiments do not actually train on a combined dataset. The FineClinic setup is sequential: pretraining on ISIC18 followed by fine-tuning on PAD-UFES-20. The conclusion in §5 also refers to assessing 'the combination of clinical and dermoscopic data,' which is a different claim from the fine-tuning protocol actually used. This mismatch between the stated contribution and the experiments should be resolved, either by adding a true combined-training experiment or by reframing the contribution as an analysis of fine-tuning under domain shift.
minor comments (4)
- [Table 1] The column headers are confusing: the table appears to show both 'Test' and 'Val' columns, but the text does not explain how the validation set differs from the test set or why some architectures report a separate validation accuracy value.
- [Table 3] There are typographical errors in the table and text: 'FClinc' should be 'FClinic', and the class labels in Table 2 include 'Melanome' and 'Akinc. Keratosis', which should be corrected to 'Melanoma' and 'Actinic Keratosis'.
- [Footnote 4 and §4.2] The text refers to FClinic as being trained on 'complete clinic images' and 'full clinic' data, but footnote 4 says only 50% of PAD-UFES-20 was used for training. This terminology should be clarified so readers do not mistakenly believe the full dataset was used for training.
- [Reproducibility] No code, trained model weights, or exact dataset preprocessing steps are provided. Given that the PAD split and DDI class mapping are central to the results, releasing these artifacts or at least a detailed protocol would substantially improve reproducibility.
Circularity Check
No significant circularity: the paper's results are empirical benchmark numbers from held-out evaluations on public datasets, with no fitted parameter or derivation that reduces to its inputs.
full rationale
The paper's central claims are measured accuracies and F1-scores from training and evaluating neural networks on public datasets (ISIC18, PAD-UFES-20, DDI). There is no analytical derivation in which an output quantity is defined in terms of the quantity it is said to predict. FineClinic is initialized on dermoscopic data and fine-tuned on 30% of PAD-UFES-20, FClinic is trained on 50% of PAD-UFES-20, and FDerm is trained on ISIC18; their scores in Table 3 are then read off from classification on held-out evaluation sets. No parameter is fitted to the target F1 values, and no uniqueness theorem or self-citation is used to force a conclusion. The authors do cite external dataset papers, including PAD-UFES-20 and DDI, but these citations are sources of data, not private justifications of the paper's own results. The most substantive concern is that the evaluation protocol is underspecified: footnote 4 states only that PAD has no official split and that 50% of the data was used for training, and the DDI label mapping is not described. That is a reproducibility and auditability weakness, not circular reasoning, because the reported numbers are not constructed to equal any input by definition. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (5)
- learning_rate =
1e-1
- fine_tune_subset_fraction =
0.30
- full_clinic_train_fraction =
0.50
- training_epochs =
100
- augmentation_configuration =
not specified
assumptions (4)
- domain assumption ImageNet pretrained weights provide a useful initialization for skin lesion classification.
- domain assumption The class labels of ISIC18, PAD-UFES-20, and DDI can be mapped into a common evaluation scheme.
- domain assumption A single training run is representative of model performance.
- ad hoc to paper The held-out PAD partition is a fair test set.
Cite this review
Pith. "Pith review of An analysis of data variation and bias in image-based dermatological datasets for machine learning classification." pith.science (2026). https://pith.science/paper/X6B3VAJH
@misc{pith2026250108962,
author = {Pith},
title = {Pith review of: An analysis of data variation and bias in image-based dermatological datasets for machine learning classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/X6B3VAJH}},
note = {Machine review of arXiv:2501.08962}
}
read the original abstract
AI algorithms have become valuable in aiding professionals in healthcare. The increasing confidence obtained by these models is helpful in critical decision demands. In clinical dermatology, classification models can detect malignant lesions on patients' skin using only RGB images as input. However, most learning-based methods employ data acquired from dermoscopic datasets on training, which are large and validated by a gold standard. Clinical models aim to deal with classification on users' smartphone cameras that do not contain the corresponding resolution provided by dermoscopy. Also, clinical applications bring new challenges. It can contain captures from uncontrolled environments, skin tone variations, viewpoint changes, noises in data and labels, and unbalanced classes. A possible alternative would be to use transfer learning to deal with the clinical images. However, as the number of samples is low, it can cause degradations on the model's performance; the source distribution used in training differs from the test set. This work aims to evaluate the gap between dermoscopic and clinical samples and understand how the dataset variations impact training. It assesses the main differences between distributions that disturb the model's prediction. Finally, from experiments on different architectures, we argue how to combine the data from divergent distributions, decreasing the impact on the model's final accuracy.
Figures
Forward citations
Cited by 1 Pith paper
-
Contrastive meta-domain adaptation for robust skin lesion classification across clinical and acquisition conditions
Contrastive pre-training followed by meta-domain guided tuning improves skin lesion classification accuracy on PAD-UFES-20 and DDI when transferring from HAM10000 dermoscopic training.
Reference graph
Works this paper leans on
-
[1]
arXiv preprint arXiv:1908.02288 (2019) Title Suppressed Due to Excessive Length 9
Combalia, M., Codella, N.C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A.C., Puig, S., et al.: Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019) Title Suppressed Due to Excessive Length 9
arXiv 2019
-
[2]
JAMA dermatology157(11), 1362–1369 (2021)
Daneshjou, R., Smith, M.P., Sun, M.D., Rotemberg, V., Zou, J.: Lack of trans- parency and potential bias in artificial intelligence data sets and algorithms: a scoping review. JAMA dermatology157(11), 1362–1369 (2021)
work page 2021
-
[3]
Daneshjou, R., Vodrahalli, K., Novoa, R.A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S.M., Bailey, E.E., Gevaert, O., Mukherjee, P., Phung, M., Yekrang, K., Fong, B., Sahasrabudhe, R., Allerup, J.A.C., Okata-Karigane, U., Zou, J., Chiou, A.S.: Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci. Adv.8(32)...
work page 2022
-
[4]
In: 2009 IEEE conference on computer vision and pattern recognition
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
2009
-
[5]
Freeman, K., Dinnes, J., Chuchu, N., Takwoingi, Y., Bayliss, S.E., Matin, R.N., Jain, A., Walter, F.M., Williams, H.C., Deeks, J.J.: Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. bmj 368 (2020)
work page 2020
-
[6]
Expert systems with applications42(19), 6578–6585 (2015)
Giotis, I., Molders, N., Land, S., Biehl, M., Jonkman, M.F., Petkov, N.: Med-node: A computer-assisted melanoma diagnosis system using non-dermoscopic images. Expert systems with applications42(19), 6578–6585 (2015)
work page 2015
-
[7]
Ha, Q., Liu, B., Liu, F.: Identifying melanoma images using efficientnet ensemble: Winning solution to the siim-isic melanoma classification challenge (2020)
work page 2020
-
[8]
Computers in Biology and Medicine p
Hasan, M.K., Ahamad, M.A., Yap, C.H., Yang, G.: A survey, review, and future trends of skin lesion segmentation and classification. Computers in Biology and Medicine p. 106624 (2023)
work page 2023
Show all 21 references
-
[9]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022)
2022
-
[10]
Dermoscopy image analysis (2015)
Mendonça, T., Celebi, M., Mendonca, T., Marques, J.: Ph2: A public database for the analysis of dermoscopic images. Dermoscopy image analysis (2015)
2015
-
[11]
Topical formulations
Oakley, A.: Dermnet new zealand. Topical formulations. Updated February (2016)
2016
-
[12]
Data Brief32(106221), 106221 (Oct 2020)
Pacheco, A.G.C., Lima, G.R., Salomão, A.S., Krohling, B., Biral, I.P., de An- gelo, G.G., Alves, Jr, F.C.R., Esgario, J.G.M., Simora, A.C., Castro, P.B.C., Ro- drigues, F.B., Frasson, P.H.L., Krohling, R.A., Knidel, H., Santos, M.C.S., do Es- pírito Santo, R.B., Macedo, T.L.S....
2020
-
[13]
Pytorch, A.D.I.: Pytorch (2018)
2018
-
[14]
Healthcare Analytics 3, 100143 (2023)
Sadik, R., Majumder, A., Biswas, A.A., Ahammad, B., Rahman, M.M.: An in-depth analysis of convolutional neural network architectures with transfer learning for skin disease diagnosis. Healthcare Analytics 3, 100143 (2023). https://doi.org/https://doi.org/10.1016/j.health.2023....
2023
-
[15]
Computers in biology and medicine140, 105111 (2022)
Salahuddin, Z., Woodruff, H.C., Chatterjee, A., Lambin, P.: Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Computers in biology and medicine140, 105111 (2022)
2022
-
[16]
Biomedical Sig- nal Processing and Control 84, 104729 (2023)
Çağrı Suiçmez, Tolga Kahraman, H., Suiçmez, A., Yılmaz, C., Balcı, F.: Detection of melanoma with hybrid learning method by removing hair from dermoscopic images using image processing techniques and wavelet transform. Biomedical Sig- nal Processing and Control 84, 104729 (202...
2023
-
[17]
Journal of the American Academy of Dermatology55(1), 54–58 (2006)
Tsang, M.W., Resneck Jr, J.S.: Even patients with changing moles face long derma- tologyappointmentwait-times:astudyofsimulatedpatientcallstodermatologists. Journal of the American Academy of Dermatology55(1), 54–58 (2006)
2006
-
[18]
Scientific data 5(1), 1–9 (2018)
Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)
2018
-
[19]
Nature Medicine27(4), 582–584 (2021)
Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D.E., Zou, J.: How medical ai devices are evaluated: limitations and recommendations from an analysis of fda approvals. Nature Medicine27(4), 582–584 (2021)
2021
-
[20]
In: Information Processing in Medical Imaging: 26th International Confer- ence, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26
Yan, Y., Kawahara, J., Hamarneh, G.: Melanoma recognition via visual atten- tion. In: Information Processing in Medical Imaging: 26th International Confer- ence, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26. pp. 793–804. Springer (2019)
2019
-
[21]
IEEE Access7, 9872–9880 (2019)
Zhang, K., Guo, Y., Wang, X., Yuan, J., Ding, Q.: Multiple feature reweight densenet for image classification. IEEE Access7, 9872–9880 (2019)
2019
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.