REVIEW 3 major objections 5 minor 22 references
The Impact of Skin Tone Label Granularity on the Performance and Fairness of AI Based Dermatology Image Classification Models
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Skin tone label granularity changes both the performance and the fairness of AI dermatology classifiers, with coarser labels degrading results.
desk verdict Plausible question, confounded design: the experiments hold total training size fixed, not per-group size, so the claimed granularity/group-specificity effects are entangled with data-quantity effects. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central device is the Fitzpatrick Skin Tone (FST) binning used to define protected groups. The paper trains separate DenseNet-161 classifiers on images labelled by coarse FST bins (1/2, 3/4, 5/6), then merges two light-skin bins (1/2 and 3/4) into one to test granularity. The bins determine both which images a model sees and which test subset its performance is measured on; the fairness gap (best minus worst group metric) then quantifies how bin choice changes bias.
What would settle it
Retrain the balanced model with 165 images per FST group (or with all available images) and compare per-group AUC, BACC, and ECE to the FST-specific models; if the balanced model then matches or beats them, the paper's performance claim collapses.
Extended reading notes
Core claim
Using DenseNet-161 classifiers trained on combined DDI and Fitzpatrick 17k images, the authors compare three ways of defining protected skin-tone groups. First, a single model trained on FST-balanced data (165 images total) is evaluated against separate models each trained on 165 images from one coarse FST group (1/2, 3/4, 5/6). The group-specific models generally achieve higher AUC and balanced accuracy, better calibration, and a smaller fairness gap. Second, the authors coarsen the label by combining FST 1/2 and 3/4 into a single group; the resulting model performs worse on FST 1/2 and 3/4 test data than the original finer-grained models. The authors conclude that the granularity of FST la
Load-bearing premise
The comparison that drives the paper's main conclusion gives the FST-balanced model only about 55 training images per group while each FST-specific model gets 165 images from a single group; if per-group data volume rather than group-specific training or label granularity is what drives the observed differences, the conclusions weaken.
Editorial extensions
If this is right
- If the claims hold, dermatology AI evaluations that report a single number over FST 5/6 as one group may hide meaningful within-group variation, so fairness numbers depend on label granularity.
- Protected group-specific training should be considered a viable bias mitigation strategy for skin lesion classifiers, provided reliable group labels are available at inference time.
- Coarsening FST labels, whether by design or because annotators disagree, can degrade accuracy for lighter-skin groups as well as shifting fairness comparisons.
- The argument for replacing the FST scale with alternative, more evenly granular skin-tone scales gains direct empirical support from these experiments.
- Future algorithmic fairness methods should be tested under different label granularities, since the definition of the protected group changes the optimization target.
Reading between the lines
- The training-set-size control (165 images for every model) means the balanced model gets only about 55 images per FST group, so part of the group-specific advantage may be a data-efficiency effect; a balanced model trained on 165 images per group could narrow or eliminate the gap.
- The coarsening effect observed for lighter skin (1/2 vs 1/2/3/4) might be stronger or weaker for darker skin; the paper's design only coarsens light-skin bins, leaving open whether coarse bins always hurt or whether some convergence is beneficial.
- An analogous test with a skin-tone scale that has uniform granularity could separate label-granularity effects from FST-specific unevenness; if the drop disappears, the problem is specifically the FST grouping, not coarseness per se.
- The fairness metrics here use group means; a finer-grained analysis might reveal that the fairness gap within the 5/6 group is substantial and understated by the FST 5/6 bin.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper investigates whether the granularity of Fitzpatrick Skin Tone (FST) labels used to define protected groups affects the performance and fairness of binary malignant/benign skin-lesion classifiers. Using combined DDI and Fitzpatrick17k images, the authors train DenseNet-161 models on FST-balanced data (165 images total, roughly 55 per FST group) and on FST-specific data (165 images from a single FST group), evaluating AUC, BACC, and ECE on a held-out test set stratified by FST 1/2, 3/4, and 5/6. Experiment 1 compares FST-balanced versus FST-specific training; Experiment 2 compares models trained on FST 1/2, FST 3/4, and a coarsened FST 1/2/3/4 group. The paper concludes that FST-specific models generally outperform the balanced model, that reduced label granularity harms performance, and that these findings challenge the suitability of the FST scale for fair AI research.
Significance. If the claims were established, the paper would make a useful contribution to the debate on skin-tone representation in dermatology AI, and it is one of the first studies to directly address label granularity. Strengths include the use of two public datasets, multiple random seeds, three complementary metrics, and explicit reporting of fairness gaps. However, the experimental design controls total training-set size rather than per-group sample size, which confounds label granularity and group-specificity with the quantity of same-group training data. The observed differences in Experiments 1 and 2 are therefore compatible with a pure data-efficiency effect. The paper's strongest conclusions are not currently supported, although the confound is addressable with additional experiments.
major comments (3)
- [Section 3.1, Tables 1-3] The training-set-size control confounds group-specific training with per-group sample size. The FST-balanced model uses 165 total images (~55 per group), while each FST-specific model uses 165 images from a single FST group. The improved performance of FST-specific models on their own group could be due to seeing three times as many same-group examples, not to group-specific training per se. The claim in Section 4 that protected group-specific models 'can lead to fairer outcomes and better performance' is therefore not established. Recommend a design that holds per-group training data constant (e.g., 55 images per group for every model) or a balanced model trained on 165 images per group (495 total), with comparable compute.
- [Section 3.2, Tables 4-5] The same confound affects Experiment 2. The coarsened FST 1/2/3/4 model is trained on 165 images sampled from two groups (~82 per group), whereas the original FST 1/2 model is trained on 165 images from FST 1/2 only, and the original FST 3/4 model on 165 from FST 3/4. The lower FST 1/2 performance of the coarsened model may therefore reflect fewer FST 1/2 training examples, not reduced label granularity. The conclusion in Section 4 that 'reducing the FST label granularity ... can negatively affect performance' is confounded for the same reason.
- [Section 3.1, Tables 1-3] The statement that 'based on AUC, the use of FST group-specific models improves performance for all three groups' is contradicted by the FST 5/6 results: the FST-specific and FST-balanced models both achieve AUC 0.93 on FST 5/6, and the FST-specific model is worse on BACC (0.84 vs 0.85) and ECE (0.14 vs 0.13). Only FST 1/2 shows consistent, statistically significant improvement across all metrics; FST 3/4 gains are mixed. The abstract's 'generally better' is therefore an overstatement, and the significance tests would need multiple-comparison correction to support the broader claim.
minor comments (5)
- [Table 5 caption] The caption reads 'from Table 3 and 2'; it should be 'Tables 2 and 3'.
- [Section 3] The text says 'same number of subjects (165 images)' but 'subjects' is used to mean images. Please clarify the unit of analysis, especially if multiple images can come from the same patient.
- [Section 2.2] The manuscript does not state whether the DenseNet-161 model was initialized with pretrained weights. Please specify the initialization and any transfer-learning details, as this affects reproducibility.
- [Section 3] The fairness gap is reported as a point estimate without confidence intervals or a significance test. Consider adding uncertainty quantification for the FG comparisons.
- [General] No code or trained models are provided. Releasing the training/testing split code and trained checkpoints would strengthen reproducibility, especially given the small sample sizes.
Circularity Check
No significant circularity: the study's claims are empirical comparisons with an independent test set; the only self-citation is background and not load-bearing.
full rationale
The paper is an empirical investigation: it trains models on different FST-stratified subsets and evaluates them on a fixed test set. There is no derived quantity that is defined in terms of the target result, no parameter fitted to the outcome and then renamed as a prediction, and no uniqueness theorem or ansatz imported from self-citations. The central claims about FST granularity follow directly from measured AUC, BACC, and ECE values. The only self-cited reference (ref. 16, Puyol-Antón et al.) is used in the introduction to note that protected group-specific training has improved performance in cardiac MRI segmentation; this is background motivation, not a load-bearing premise, and the same point is also supported by refs. 5 and 22. A legitimate scientific concern is that the training-set-size control (165 images total) confounds group-specificity with per-group sample quantity: FST-specific models see 165 same-group images while the FST-balanced model sees only ~55 per group. However, this is a potential experimental confound, not a circularity: it does not make the conclusion equivalent to the input by construction. Accordingly, no circular step is identified.
Assumptions & free parameters
free parameters (1)
- Model hyperparameters (batch size, learning rate) =
not reported; selected separately per model via grid search (batch size in {16,32,64}, learning rate in {0.0005,0.0003,0
assumptions (4)
- domain assumption FST labels in DDI and Fitzpatrick17k are accurate and directly comparable across datasets
- domain assumption The filtered benign/malignant labels reflect clinically meaningful ground truth for lesion classification
- domain assumption AUC, BACC, and ECE computed on a small fixed test set (50 images per group, 1:1 class ratio) are valid estimates for comparing models
- domain assumption Equalizing total training set size to 165 images isolates the effect of group-specificity or coarseness
Cite this review
Pith. "Pith review of The Impact of Skin Tone Label Granularity on the Performance and Fairness of AI Based Dermatology Image Classification Models." pith.science (2026). https://pith.science/paper/3MFSNUWO
@misc{pith2026250911184,
author = {Pith},
title = {Pith review of: The Impact of Skin Tone Label Granularity on the Performance and Fairness of AI Based Dermatology Image Classification Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/3MFSNUWO}},
note = {Machine review of arXiv:2509.11184}
}
read the original abstract
Artificial intelligence (AI) models to automatically classify skin lesions from dermatology images have shown promising performance but also susceptibility to bias by skin tone. The most common way of representing skin tone information is the Fitzpatrick Skin Tone (FST) scale. The FST scale has been criticised for having greater granularity in its skin tone categories for lighter-skinned subjects. This paper conducts an investigation of the impact (on performance and bias) on AI classification models of granularity in the FST scale. By training multiple AI models to classify benign vs. malignant lesions using FST-specific data of differing granularity, we show that: (i) when training models using FST-specific data based on three groups (FST 1/2, 3/4 and 5/6), performance is generally better for models trained on FST-specific data compared to a general model trained on FST-balanced data; (ii) reducing the granularity of FST scale information (from 1/2 and 3/4 to 1/2/3/4) can have a detrimental effect on performance. Our results highlight the importance of the granularity of FST groups when training lesion classification models. Given the question marks over possible human biases in the choice of categories in the FST scale, this paper provides evidence for a move away from the FST scale in fair AI research and a transition to an alternative scale that better represents the diversity of human skin tones.
Figures
Reference graph
Works this paper leans on
-
[1]
Appl Sci14(2024)
Bello, A., Ng, S.C., Leung, M.F.: Skin cancer classification using fine-tuned transfer learning of DENSENET-121. Appl Sci14(2024)
2024
-
[2]
Dermatol Nurs21, 170–177 (2009)
Bradford, P.T.: Skin cancer in skin of color. Dermatol Nurs21, 170–177 (2009)
2009
-
[3]
Int J Cosmet Sci13(4), 191–208 (1991)
Chardon, A., Cretois, I., Hourseau, C.: Skin colour typology and suntanning path- ways. Int J Cosmet Sci13(4), 191–208 (1991)
1991
-
[4]
JAMA Dermatol157(11), 1362–1369 (2021)
Daneshjou, R., Smith, M.P., Sun, M.D., Rotemberg, V., Zou, J.: Lack of trans- parency and potential bias in artificial intelligence data sets and algorithms: A scoping review. JAMA Dermatol157(11), 1362–1369 (2021)
2021
-
[5]
Sci Adv8(32), eabq6147 (2022)
Daneshjou, R., Vodrahalli, K., Novoa, R.A., Jenkins, M., Liang, W., Rotemberg, V., Ko, J., Swetter, S.M., Bailey, E.E., Gevaert, O., Mukherjee, P., Phung, M., Yekrang, K., Fong, B., Sahasrabudhe, R., Allerup, J.A.C., Okata-Karigane, U., Zou, J., Chiou, A.S.: Disparities in dermatology AI performance on a diverse, curated clinical image set. Sci Adv8(32), ...
2022
-
[6]
Ko, S.M.S., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R.A., J. Ko, S.M.S., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. Na- ture542, 115–118 (2017)
2017
-
[7]
Arch Dermatol124(6), 869–871 (1988)
Fitzpatrick, T.B.: The validity and practicality of sun-reactive skin types I through VI. Arch Dermatol124(6), 869–871 (1988)
1988
-
[8]
In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., Badri, O.: Evaluating deep neural networks trained on clinical images in dermatology with the Fitzpatrick 17k dataset. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1820–1828 (2021)
2021
Show all 22 references
-
[9]
In: Proceedings of International Conference on Machine Learning (ICML) (2017)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neu- ral networks. In: Proceedings of International Conference on Machine Learning (ICML) (2017)
2017
-
[10]
In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) 10 P
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) 10 P. Shah et al
2017
-
[11]
Knowl Inf Syst33, 1–33 (2012)
Kamiran, F., Calders, T.: Data preprocessing techniques for classification without discrimination. Knowl Inf Syst33, 1–33 (2012)
2012
-
[12]
Monk, E.: The Monk skin tone scale (2023).https://doi.org/10.31235/osf.io/ pdf4c
2023 doi
-
[13]
In: Proceedings of Machine Learning for Healthcare (MLHC) (2023)
Movva, R., Shanmugam, D., Hou, K., Pathak, P., Guttag, J., Garg, N., Pierson, E.: Coarse race data conceals disparities in clinical risk score performance. In: Proceedings of Machine Learning for Healthcare (MLHC) (2023)
2023
-
[14]
Br J Dermatol185(1), 198–199 (2021)
Okoji, U.K., Taylor, S.C., Lipoff, J.B.: Equity in skin typing: Why it is time to replace the Fitzpatrick scale. Br J Dermatol185(1), 198–199 (2021)
2021
-
[15]
In: Proceedings of European Conference on Computer Vision (ECCV) (2022)
Pakzad, A., Abhishek, K., Hamarneh, G.: CIRCLe: Color invariant representation learning for unbiased classification of skin lesions. In: Proceedings of European Conference on Computer Vision (ECCV) (2022)
2022
-
[16]
In: Proceedings of Medical Image Computing and Computer Assisted Interventions (MICCAI)
Puyol-Antón, E., Ruijsink, B., Piechnik, S.K., Neubauer, S., Petersen, S.E., Razavi, R., King, A.P.: Fairness in cardiac MR image analysis: An investigation of bias due to data imbalance in deep learning based segmentation. In: Proceedings of Medical Image Computing and Comput...
2021
-
[17]
arXiv preprint arXiv:2309.05148 (2023),https://arxiv.org/abs/2309.05148
Renggli, C., Smith, D.B., Gola, H.M., Kindermans, P.J.: Do we need training data? towards fairness in computer vision with no-reference metrics. arXiv preprint arXiv:2309.05148 (2023),https://arxiv.org/abs/2309.05148
2023 arXiv
-
[18]
In: Proceedings of International Conference on Machine Learning (ICML) (2020)
Sagawa, S., Koh, P.W., Hashimoto, T.B., Liang, P.: Distributionally robust neu- ral networks for group shifts: On the importance of regularization for worst-case generalization. In: Proceedings of International Conference on Machine Learning (ICML) (2020)
2020
-
[19]
In: Proceedings of the 36th International Conference on Machine Learning (2019),https://proceedings.mlr.press/v97/ustun19a.html
Ustun, F.O., Laumann, J.R., Smith, A.D.: Fairness without demographics in re- peated decisions. In: Proceedings of the 36th International Conference on Machine Learning (2019),https://proceedings.mlr.press/v97/ustun19a.html
2019
-
[20]
Cutis105(2), 77–80 (2020)
Ware, O.R., Dawson, J.E., Shinohara, M.M., Taylor, S.C.: Racial limitations of Fitzpatrick skin type. Cutis105(2), 77–80 (2020)
2020
-
[21]
npj Digit Med7(191) (2024)
Weir, V.R., Dempsey, K., Gichoya, J.W., Rotemberg, V., Wong, A.K.I.: A survey of skin tone assessment in prospective research. npj Digit Med7(191) (2024)
2024
-
[22]
In: Proceedings of Conference on Health, Inference, and Learning
Zhang, H., Dullerud, N., Roth, K., Oakden-Rayner, L., Pfohl, S., Ghassemi, M.: Improving the fairness of chest X-ray classifiers. In: Proceedings of Conference on Health, Inference, and Learning. pp. 204–233 (2022)
2022
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.