Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

An analysis of data variation and bias in image-based dermatological datasets for machine learning classification

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Fine-tuning on 30% of clinical data closes most of the dermoscopy-to-clinic gap.

desk verdict Useful confirmation of a known fine-tuning result, but Table 3's key comparison relies on an undocumented PAD split and DDI class mapping. read the letter →

arxiv 2501.08962 v2 pith:X6B3VAJH submitted 2025-01-15 cs.CV cs.AI

classification cs.CVcs.AI
keywords skinlesionclassificationdermoscopicimagesclinicaltransferlearningfine-tuningdomainshiftdatasetbiasclassimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Skin-lesion classifiers are usually trained on dermoscopic images, but real users photograph lesions with smartphones. The paper asks whether a model that has never seen clinical photos can be adapted to them with very little clinical data, and answers yes. After training on the dermoscopic ISIC18 data, fine-tuning on only 30% of the clinical PAD-UFES-20 images lifts the clinical F1-score from 5.22 to 67.18, essentially matching the 68.95 F1 of a model trained on 50% of the clinical data. The same experiments show that trained-on-clinical models do not automatically transfer to a different clinical dataset: PAD-trained models score only F1=11.80 on the DDI dataset. The paper reads this as evidence that domain differences, not just data quantity, dominate performance and that small fine-tuning sets can be an efficient remedy.

What carries the argument

The mechanism is transfer learning via fine-tuning: models are first pretrained on ImageNet, then trained on the dermoscopic ISIC18 dataset to produce FDerm, and then a subset is re-trained on clinical PAD-UFES-20 images to produce FineClinic. The comparison table across FDerm, FClinic and FineClinic evaluated on ISIC18, PAD and DDI is the load-bearing instrument, because it isolates the effect of adding a small amount of clinical data to a dermoscopic base. Data augmentation (random flips, rotations, colour jitter, crops and affine transforms) is applied during fine-tuning to counter class imbalance and image noise, which the paper treats as essential for the minority malignant classes.

What would settle it

Reproduce FDerm, FClinic and FineClinic on a single public split of PAD-UFES-20 with one explicit class mapping shared by ISIC18, PAD and DDI, and test all models on the same held-out partition; if FineClinic's clinical F1 no longer lands near FClinic's, or if the PAD-trained model's F1 on DDI departs from roughly 11.80, the central efficiency claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an efficiency and a limit. FineClinic—a model initialized on dermoscopic ISIC18 and then fine-tuned on a 30% subset of PAD-UFES-20—reaches an F1-score of 67.18 on clinical PAD data, almost identical to the 68.95 achieved by FClinic, a model trained directly on 50% of the same clinical data, while a purely dermoscopic FDerm model collapses to 5.22 on clinical images. The same experiment also finds that FClinic generalizes poorly to a second clinical dataset, DDI, with F1=11.80, indicating that two 'clinical' collections can be as far apart as dermoscopy and clinic are. These results are averaged across CNN architectures (ConvNext, DenseNet, ResNet, EfficientNet and InceptionV3), and the paper argues they show that combining a large dermoscopic source with a small clinical fine-tuning set is a more data-efficient route to clinical classification than training on clinical data alone, while cautioning that no single clinical dataset yet guarantees transfer to another.

Load-bearing premise

The entire comparison rests on the unstated assumption that the PAD and DDI evaluation splits and the class-label mappings across ISIC18, PAD and DDI are identical across all models; the paper never specifies the held-out test set or the mapping, so if those differ the headline 30%-versus-50% and DDI numbers are not comparable.

Editorial extensions

If this is right

  • If the claim is right, a deployment recipe for smartphone-based skin-lesion screening is: take a dermoscopic-trained model and fine-tune it on a few hundred carefully chosen clinical images, rather than collecting a large clinical dataset from scratch.
  • Dermoscopic-only models should not be used directly in clinical workflows: the F1 of 5.22 on PAD shows they effectively fail on uncontrolled phone photos.
  • Evaluation across multiple clinical datasets is necessary, because a model trained on one clinical set (PAD) can score only 11.80 F1 on another (DDI), so single-dataset clinical numbers are not trustworthy.
  • Class-imbalance-aware augmentation during fine-tuning matters: without it, the model would simply predict the majority 'nevus' class, hiding poor malignant-class accuracy behind acceptable averages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit: sweep the fine-tuning fraction (e.g., 10%, 20%, 30%, 40%, 50%) and plot clinical F1 against it to locate the knee of the curve; if the 30%-equals-50% finding holds, the knee is likely below 30%, which would make data collection even cheaper.
  • The F1=11.80 cross-clinical result suggests 'clinical' is not one domain; capture conditions such as resolution, distance, illumination and reflectance may dominate over skin-tone diversity. A controlled experiment that varies only the capture device while holding skin-tone distribution fixed would separate these confounds.
  • The paper's demographic-bias framing and acquisition-shift framing are entangled in its data; one could test the paper's implicit claim that dermoscopic models fail on clinical images largely due to domain shift by fine-tuning on a skin-tone-stratified 30% subset versus a random 30% subset and comparing fairness metrics.
  • If replicated, the result implies that benchmark design should report both the dermoscopic-to-clinical transfer gain and clinical-to-clinical transfer loss, since either number alone can mislead about real-world readiness.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies the domain gap between dermoscopic and clinical skin-lesion images for CNN classification. Using the public ISIC18, PAD-UFES-20, and DDI datasets, the authors train ImageNet-initialized models on dermoscopic data (FDerm), on clinical data (FClinic), and fine-tune dermoscopic-trained models on 30% of the clinical data (FineClinic). The central quantitative claims are that FDerm achieves only 5.22 F1 on PAD, FineClinic reaches 67.18 F1 on PAD with 30% of the training images, which is comparable to FClinic's 68.95 with 50% of the images, and that PAD-trained models transfer poorly to DDI, with F1 around 11.80. The paper also reports class-level accuracy and discusses dataset biases, data imbalance, and the limitations of clinical deployment.

Significance. If the quantitative results were fully auditable, the paper would provide useful evidence on a clinically relevant question: how much dermoscopic pretraining helps in low-resource clinical settings, and how far clinical models generalize across clinical datasets. The qualitative finding that dermoscopic-trained models fail on clinical images and that fine-tuning on a small clinical subset substantially improves performance is plausible and consistent with prior work. Strengths of the manuscript include the use of public datasets, the comparison of several architectures, and the explicit attention to class imbalance and per-class performance. However, the precise numerical claims in Table 3 are currently under-supported because the evaluation protocol is underspecified, no variance estimates are reported, and the class-label mapping between datasets is not defined. These issues prevent the paper, in its present form, from supporting the specific quantitative conclusions it advertises.

major comments (4)
  1. [§3.3, footnote 4, and Table 3] The PAD-UFES-20 train/test protocol is never fully specified. Footnote 4 states that PAD has no official split and that 50% of the data was used for training, but it does not define the held-out test partition, state whether the test partition is the same for FDerm, FClinic, and FineClinic, describe how stratification is performed, or specify whether a validation subset is carved out separately. Since FineClinic uses 30% and FClinic uses 50% of the same data, any difference in the test split or in class stratification could explain the 1.77 F1-point gap reported in Table 3. This is load-bearing for the paper's central claim that fine-tuning on fewer clinical images reaches performance comparable to training on more clinical images.
  2. [§4.2, Table 3, and §4.3] The DDI evaluation is not auditable because the class-label mapping is never described. PAD-UFES-20 and DDI use different label schemes, and the paper reports DDI F1 values without stating how lesion-type classes are mapped to DDI labels, whether some classes are filtered out, or how the metric is averaged. If the mapping is inconsistent, the DDI F1 values in Table 3, including the repeated 11.80 values, lose their meaning. The clinical-to-clinical generalization claim in §4.3 therefore rests on an unspecified evaluation protocol.
  3. [§3.3 and Table 3] All results appear to come from single training runs, with no error bars, no multiple seeds, and no measure of run-to-run variance. The training setup also raises concerns: a learning rate of 1e-1 with a cosine annealing scheduler that decreases every 10 epochs is unusual for fine-tuning large CNN models, and no justification or learning-rate sensitivity analysis is provided. Without variance estimates, the central comparison between FClinic's 68.95 and FineClinic's 67.18 cannot be distinguished from noise, especially given the small clinical training sets.
  4. [Abstract, §5, and Table 3] The paper promises to show how to 'combine' data from divergent distributions, but the experiments do not actually train on a combined dataset. The FineClinic setup is sequential: pretraining on ISIC18 followed by fine-tuning on PAD-UFES-20. The conclusion in §5 also refers to assessing 'the combination of clinical and dermoscopic data,' which is a different claim from the fine-tuning protocol actually used. This mismatch between the stated contribution and the experiments should be resolved, either by adding a true combined-training experiment or by reframing the contribution as an analysis of fine-tuning under domain shift.
minor comments (4)
  1. [Table 1] The column headers are confusing: the table appears to show both 'Test' and 'Val' columns, but the text does not explain how the validation set differs from the test set or why some architectures report a separate validation accuracy value.
  2. [Table 3] There are typographical errors in the table and text: 'FClinc' should be 'FClinic', and the class labels in Table 2 include 'Melanome' and 'Akinc. Keratosis', which should be corrected to 'Melanoma' and 'Actinic Keratosis'.
  3. [Footnote 4 and §4.2] The text refers to FClinic as being trained on 'complete clinic images' and 'full clinic' data, but footnote 4 says only 50% of PAD-UFES-20 was used for training. This terminology should be clarified so readers do not mistakenly believe the full dataset was used for training.
  4. [Reproducibility] No code, trained model weights, or exact dataset preprocessing steps are provided. Given that the PAD split and DDI class mapping are central to the results, releasing these artifacts or at least a detailed protocol would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's results are empirical benchmark numbers from held-out evaluations on public datasets, with no fitted parameter or derivation that reduces to its inputs.

full rationale

The paper's central claims are measured accuracies and F1-scores from training and evaluating neural networks on public datasets (ISIC18, PAD-UFES-20, DDI). There is no analytical derivation in which an output quantity is defined in terms of the quantity it is said to predict. FineClinic is initialized on dermoscopic data and fine-tuned on 30% of PAD-UFES-20, FClinic is trained on 50% of PAD-UFES-20, and FDerm is trained on ISIC18; their scores in Table 3 are then read off from classification on held-out evaluation sets. No parameter is fitted to the target F1 values, and no uniqueness theorem or self-citation is used to force a conclusion. The authors do cite external dataset papers, including PAD-UFES-20 and DDI, but these citations are sources of data, not private justifications of the paper's own results. The most substantive concern is that the evaluation protocol is underspecified: footnote 4 states only that PAD has no official split and that 50% of the data was used for training, and the DDI label mapping is not described. That is a reproducibility and auditability weakness, not circular reasoning, because the reported numbers are not constructed to equal any input by definition. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The quantitative claims rest on several free choices: the learning rate, training epochs, augmentation settings, and the fractions of clinical data used for training (50%) and fine-tuning (30%). The cross-dataset evaluations additionally assume a class mapping and a fixed test protocol that are never described. No new entities are introduced.

free parameters (5)
  • learning_rate = 1e-1
    Global learning rate for ADAM, unusual and possibly a typo; central to all training runs.
  • fine_tune_subset_fraction = 0.30
    30% of PAD used for fine-tuning in FineClinic, chosen by the authors.
  • full_clinic_train_fraction = 0.50
    50% of PAD used for FClinic training because PAD has no official split.
  • training_epochs = 100
    Number of epochs for all models.
  • augmentation_configuration = not specified
    RandomHorizontalFlip, RandomVerticalFlip, RandomRotation, ColorJitter, RandomResizedCrop, and RandomAffine are listed, but the exact parameters are not given.
assumptions (4)
  • domain assumption ImageNet pretrained weights provide a useful initialization for skin lesion classification.
    All models start from ImageNet weights (Sec 3.3); this is standard but unproven for this domain in the paper.
  • domain assumption The class labels of ISIC18, PAD-UFES-20, and DDI can be mapped into a common evaluation scheme.
    Cross-dataset evaluations in Tables 2 and 3 require a class mapping, which is never described.
  • domain assumption A single training run is representative of model performance.
    No repeated runs and no confidence intervals are reported; the paper implicitly assumes the reported numbers are stable.
  • ad hoc to paper The held-out PAD partition is a fair test set.
    The paper states 50% of PAD is used to train (footnote 4) but does not describe how the test set is constructed or whether FineClinic and FClinic use the same test split.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An analysis of data variation and bias in image-based dermatological datasets for machine learning classification." pith.science (2026). https://pith.science/paper/X6B3VAJH

@misc{pith2026250108962,
  author       = {Pith},
  title        = {Pith review of: An analysis of data variation and bias in image-based dermatological datasets for machine learning classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X6B3VAJH}},
  note         = {Machine review of arXiv:2501.08962}
}
read the original abstract

AI algorithms have become valuable in aiding professionals in healthcare. The increasing confidence obtained by these models is helpful in critical decision demands. In clinical dermatology, classification models can detect malignant lesions on patients' skin using only RGB images as input. However, most learning-based methods employ data acquired from dermoscopic datasets on training, which are large and validated by a gold standard. Clinical models aim to deal with classification on users' smartphone cameras that do not contain the corresponding resolution provided by dermoscopy. Also, clinical applications bring new challenges. It can contain captures from uncontrolled environments, skin tone variations, viewpoint changes, noises in data and labels, and unbalanced classes. A possible alternative would be to use transfer learning to deal with the clinical images. However, as the number of samples is low, it can cause degradations on the model's performance; the source distribution used in training differs from the test set. This work aims to evaluate the gap between dermoscopic and clinical samples and understand how the dataset variations impact training. It assesses the main differences between distributions that disturb the model's prediction. Finally, from experiments on different architectures, we argue how to combine the data from divergent distributions, decreasing the impact on the model's final accuracy.

Figures

Figures reproduced from arXiv: 2501.08962 by the authors.

Figure 1
Figure 1. Examples of images in the dermoscopic ISIC18 dataset (top row) and PAD￾UFES-20 (bottom row). While clinical features impact model decisions, it is evident how pixel differences arise from intrinsic characteristics of each domain (e.g., capture device quality, lighting, noise, resolution). 3 Methodology 3.1 Motivation According to Daneshjou et al.[3], dermatology classification approaches trained on dermoscopic datas… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Contrastive meta-domain adaptation for robust skin lesion classification across clinical and acquisition conditions

    cs.CV 2026-02 conditional novelty 4.0 of 10

    Contrastive pre-training followed by meta-domain guided tuning improves skin lesion classification accuracy on PAD-UFES-20 and DDI when transferring from HAM10000 dermoscopic training.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:1908.02288 (2019) Title Suppressed Due to Excessive Length 9

    Combalia, M., Codella, N.C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A.C., Puig, S., et al.: Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019) Title Suppressed Due to Excessive Length 9

  2. [2]

    JAMA dermatology157(11), 1362–1369 (2021)

    Daneshjou, R., Smith, M.P., Sun, M.D., Rotemberg, V., Zou, J.: Lack of trans- parency and potential bias in artificial intelligence data sets and algorithms: a scoping review. JAMA dermatology157(11), 1362–1369 (2021)

  3. [3]

    Daneshjou, R., Vodrahalli, K., Novoa, R.A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S.M., Bailey, E.E., Gevaert, O., Mukherjee, P., Phung, M., Yekrang, K., Fong, B., Sahasrabudhe, R., Allerup, J.A.C., Okata-Karigane, U., Zou, J., Chiou, A.S.: Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci. Adv.8(32)...

  4. [4]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

  5. [5]

    bmj 368 (2020)

    Freeman, K., Dinnes, J., Chuchu, N., Takwoingi, Y., Bayliss, S.E., Matin, R.N., Jain, A., Walter, F.M., Williams, H.C., Deeks, J.J.: Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. bmj 368 (2020)

  6. [6]

    Expert systems with applications42(19), 6578–6585 (2015)

    Giotis, I., Molders, N., Land, S., Biehl, M., Jonkman, M.F., Petkov, N.: Med-node: A computer-assisted melanoma diagnosis system using non-dermoscopic images. Expert systems with applications42(19), 6578–6585 (2015)

  7. [7]

    Ha, Q., Liu, B., Liu, F.: Identifying melanoma images using efficientnet ensemble: Winning solution to the siim-isic melanoma classification challenge (2020)

  8. [8]

    Computers in Biology and Medicine p

    Hasan, M.K., Ahamad, M.A., Yap, C.H., Yang, G.: A survey, review, and future trends of skin lesion segmentation and classification. Computers in Biology and Medicine p. 106624 (2023)

Show all 21 references
  1. [9]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976–11986 (2022)

  2. [10]

    Dermoscopy image analysis (2015)

    Mendonça, T., Celebi, M., Mendonca, T., Marques, J.: Ph2: A public database for the analysis of dermoscopic images. Dermoscopy image analysis (2015)

  3. [11]

    Topical formulations

    Oakley, A.: Dermnet new zealand. Topical formulations. Updated February (2016)

  4. [12]

    Data Brief32(106221), 106221 (Oct 2020)

    Pacheco, A.G.C., Lima, G.R., Salomão, A.S., Krohling, B., Biral, I.P., de An- gelo, G.G., Alves, Jr, F.C.R., Esgario, J.G.M., Simora, A.C., Castro, P.B.C., Ro- drigues, F.B., Frasson, P.H.L., Krohling, R.A., Knidel, H., Santos, M.C.S., do Es- pírito Santo, R.B., Macedo, T.L.S....

  5. [13]

    Pytorch, A.D.I.: Pytorch (2018)

  6. [14]

    Healthcare Analytics 3, 100143 (2023)

    Sadik, R., Majumder, A., Biswas, A.A., Ahammad, B., Rahman, M.M.: An in-depth analysis of convolutional neural network architectures with transfer learning for skin disease diagnosis. Healthcare Analytics 3, 100143 (2023). https://doi.org/https://doi.org/10.1016/j.health.2023....

  7. [15]

    Computers in biology and medicine140, 105111 (2022)

    Salahuddin, Z., Woodruff, H.C., Chatterjee, A., Lambin, P.: Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Computers in biology and medicine140, 105111 (2022)

  8. [16]

    Biomedical Sig- nal Processing and Control 84, 104729 (2023)

    Çağrı Suiçmez, Tolga Kahraman, H., Suiçmez, A., Yılmaz, C., Balcı, F.: Detection of melanoma with hybrid learning method by removing hair from dermoscopic images using image processing techniques and wavelet transform. Biomedical Sig- nal Processing and Control 84, 104729 (202...

  9. [17]

    Journal of the American Academy of Dermatology55(1), 54–58 (2006)

    Tsang, M.W., Resneck Jr, J.S.: Even patients with changing moles face long derma- tologyappointmentwait-times:astudyofsimulatedpatientcallstodermatologists. Journal of the American Academy of Dermatology55(1), 54–58 (2006)

  10. [18]

    Scientific data 5(1), 1–9 (2018)

    Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)

  11. [19]

    Nature Medicine27(4), 582–584 (2021)

    Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D.E., Zou, J.: How medical ai devices are evaluated: limitations and recommendations from an analysis of fda approvals. Nature Medicine27(4), 582–584 (2021)

  12. [20]

    In: Information Processing in Medical Imaging: 26th International Confer- ence, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26

    Yan, Y., Kawahara, J., Hamarneh, G.: Melanoma recognition via visual atten- tion. In: Information Processing in Medical Imaging: 26th International Confer- ence, IPMI 2019, Hong Kong, China, June 2–7, 2019, Proceedings 26. pp. 793–804. Springer (2019)

  13. [21]

    IEEE Access7, 9872–9880 (2019)

    Zhang, K., Guo, Y., Wang, X., Yuan, J., Ding, Q.: Multiple feature reweight densenet for image classification. IEEE Access7, 9872–9880 (2019)

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.