REVIEW 4 major objections 8 minor 30 references
Incorporating Rather Than Eliminating: Achieving Fairness for Skin Disease Diagnosis Through Group-Specific Expert
T0 review · 4 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read FairMoE turns sensitive attributes into group-specific experts, raising skin-diagnosis accuracy while keeping fairness.
desk verdict FairMoE's group-specialized MoE is a promising way to use sensitive attributes in skin AI, but the missing capacity-matched baseline and unverified router mean the core claim isn't proven yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the layer-wise mixture-of-experts module: each convolutional layer of a ResNet-18 or VGG-11 backbone is replaced by several experts (one per group) and a small routing convolution. Two mechanisms make it work: a mutual-information loss $-I(C;E_Y)$ that establishes a strong correlation between group labels and expert choice, and a soft routing rule that weights the router's confidence score by inverse group size, so selection probability reflects a sample's proximity to each group rather than a hard label. The router's confidence score performs double duty: it is the training signal for specialization and the inference-time routing probability.
What would settle it
Measure the router's agreement with true skin-type or age labels on held-out test data; if that agreement is near chance while FairMoE still reproduces the reported F1 and equalized-odds numbers, then the group-specialization mechanism is not what produces the gains. Alternatively, setting the mutual-information loss term to zero and observing that the gains disappear would confirm the specialization term is doing the work.
Extended reading notes
Core claim
The central claim is that fairness in skin-disease diagnosis can be improved by incorporating sensitive attributes rather than eliminating them, through a layer-wise mixture-of-experts architecture with group-specific experts. FairMoE makes the router specialize by maximizing the mutual information $I(C;E)$ between group labels and expert assignments, then uses a soft probabilistic routing rule $P(E_k|x) = \alpha_k s_k(x) / \sum_j \alpha_j s_j(x)$, where $s_k(x)$ is the router's confidence that sample $x$ belongs to group $k$ and $\alpha_k$ rebalances group sizes. This lets out-of-group data train an expert when the sample is close to that group's distribution, which is the proposed mechanism for avoiding the distribution shift of hard group-specific training and for handling boundary samples. In the reported experiments, FairMoE attains the highest average F1 on both datasets and equalized-odds values at or below all baselines, so the paper's claim is that accuracy and fairness improve together.
Load-bearing premise
The entire mechanism rests on the router's confidence score being a good estimate of a sample's true group and of how close the sample is to each group's data distribution, and the paper reports no direct measurement of the router's accuracy.
Editorial extensions
If this is right
- Fairness methods that remove sensitive attributes are not the only viable route; incorporating those attributes can raise accuracy for both groups at once.
- On the reported results, the unprivileged group improves more than the privileged group, so group-specific experts narrow the gap by lifting the weaker side rather than dropping the stronger side.
- Because deeper MoE layers contribute more, practitioners with limited compute can replace only the deepest convolutional layers and retain most of the accuracy and fairness benefit.
- Soft routing gives borderline samples access to more than one expert, which is the proposed answer to the hard-boundary problem of fixed group splitting.
Reading between the lines
- Editorial inference: if the router's confidence score is a good proxy for group proximity, the same architecture should transfer to other diagnostic imaging tasks with clinically relevant demographic attributes, such as age-defined subgroups in retinal or chest imaging.
- Editorial inference: comparing FairMoE against a variant whose router is trained with explicit group labels would isolate whether the mutual-information objective or the soft routing itself drives the reported gains.
- Editorial inference: treating skin tone as a continuous rather than binary grouping could make the soft-routing design exploit intermediate Fitzpatrick types instead of forcing them into two buckets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FairMoE, a layer-wise mixture-of-experts (MoE) architecture for skin disease diagnosis in which each expert is encouraged, via a mutual-information loss, to specialize on one sensitive-attribute group (skin type or age), while a soft probabilistic router uses the router's confidence score to weight training data from other groups. On Fitzpatrick-17k and ISIC 2019, the authors report higher F1 scores and lower equalized-odds values than several fairness baselines, arguing that incorporating sensitive attributes improves accuracy without sacrificing fairness, in contrast to fairness-through-invariance methods.
Significance. If the results hold, FairMoE offers a useful counterpoint to the dominant fairness-through-invariance paradigm in medical imaging: rather than removing sensitive attributes, it exploits them through group-specialized experts and soft routing. The framework is conceptually simple, builds on a standard MoE architecture, and is evaluated on two public benchmarks against recent pruning and quantization baselines. The paper includes an ablation over the number of MoE layers and a case study showing routing-score behavior across layers, which are helpful. However, the strength of the empirical claim depends on several unverified assumptions: router reliability, capacity matching, and statistical stability. The current manuscript does not yet provide enough evidence to fully support the claim of simultaneous accuracy gain and fairness preservation.
major comments (4)
- [Section 4, Table 1] The central empirical claim of 'substantial accuracy improvements' is not supported by statistical evidence. No error bars, confidence intervals, or number of seeds are reported for any entry in Table 1. The F1 difference between FairMoE (0.502) and SCP-FairPrune (0.476) on Fitzpatrick-17k is 0.026, and the Eodd differences are even smaller (0.154 vs. 0.164), which may be within run-to-run variation. Please report mean and standard deviation over at least three independent runs and, where possible, a paired significance test for the accuracy and fairness metrics.
- [Section 4, Table 1; Section 3.2] FairMoE roughly doubles the number of parameters in each converted layer by introducing two experts per layer, while the baselines (Vanilla, FairAdaBN, SCP-FairPrune, FairQuantize) use a single weight set. The reported accuracy gains could therefore be partly attributable to added model capacity rather than to the proposed group-specialized routing mechanism. Please include a capacity-matched baseline, for example a standard MoE without the mutual-information loss or a single wide network with a comparable parameter count, so that the effect of the group-specialization mechanism is isolated.
- [Section 3.4, Eq. (2); Section 4, Fig. 2] The mechanism assumes that the router's confidence score s_k(x) is a reliable estimate of the true sensitive attribute and of the sample's proximity to each group's distribution, yet the paper does not report router accuracy, AUC, calibration, or the empirical mutual information between true group labels and selected experts on held-out data. Fig. 2(a) shows average routing scores across layers, but not prediction accuracy. For ISIC 2019, where the sensitive attribute is age (<=55 vs. >55), it is especially unclear whether the router can infer age from dermoscopic images. If the router is near chance, expert specialization may not occur at inference, and the claimed benefit of soft routing collapses. Please quantify router performance and, if necessary, analyze failure cases.
- [Section 3.3, Eq. (1); Section 3.4, Eq. (2)] The methodology for the mutual-information loss is underspecified. The text defines P(E_i|C_j) but Eq. (1) uses the joint P(C_i,E_j), and it is unclear how these probabilities are estimated (per batch or over the whole dataset) and how the loss is differentiated through the router's discrete assignments. In addition, Eq. (2) defines P(E_k|x) with alpha_k = 1/N_k; if s_k(x) is a softmax posterior probability, multiplying by 1/N_k changes the normalization in a way that is not justified, and it is unclear whether alpha is applied to logits or probabilities. Please clarify the notation and the exact computation.
minor comments (8)
- [Abstract] The phrase 'the lost of clinically relevant diagnostic cues' should be 'the loss of clinically relevant diagnostic cues'.
- [Section 2] The statement that 'malignant melanoma is 20 times more common in African Americans compared to other groups [16]' appears inconsistent with the cited source and with the general epidemiology of melanoma; please verify the direction of this claim.
- [Section 4, Fairness Metrics] The metrics Eopp0 and Eopp1 are used in Table 1 but are never defined; please define these subscripts (for example, whether they refer to specific classes or groups) and how they are computed.
- [Table 1] The caption should define 'Avg.' and 'Diff.'; in particular, 'Diff.' appears to be the absolute difference between groups, but this should be stated explicitly.
- [Table 1] For the Vanilla baseline, all FATE values are 0.000. Please explain how FATE is normalized and why Vanilla is defined as zero; otherwise the FATE comparisons are difficult to interpret.
- [Fig. 2(b)] The axes are unclear: F1 Score and Eodd appear to share the same y-axis label but have different scales. Please use separate axes or a clear legend.
- [Section 4, ISIC 2019 paragraph] There is a typo: 'group-specific modules for fairnessis' should be 'group-specific modules for fairness'.
- [Section 3.3] The phrase 'weighted by a parameterwM I' has a missing space and should read 'weighted by a parameter w_MI'.
Circularity Check
No circularity: the FairMoE derivation is an empirical training procedure evaluated against external baselines on public datasets.
full rationale
The paper's central claim is that a layer-wise mixture-of-experts architecture with mutual-information-based expert specialization and soft probabilistic routing improves accuracy while preserving fairness, compared with external baselines on Fitzpatrick-17k and ISIC 2019. The mutual information term in Eq. (1) is a training objective, not a derived prediction of the reported fairness metrics; the routing probability in Eq. (2) uses the router confidence as an assumed proxy for group proximity, which is an unverified modeling assumption rather than a definitional equivalence with the experimental outcome. The reported F1, Eopp, and Eodd values in Table 1 are computed from held-out evaluation and compared against baselines, so the central claim is not forced by construction. Self-citations such as FairPrune, FairQuantize, and the multi-exit fairness work appear as baselines or as related observation about bias in deeper layers, and the core method does not depend on a uniqueness theorem or a prior-work-derived ansatz. No fitted parameter is relabeled as a prediction, and no step of the derivation reduces to its own inputs. The possible weakness that router accuracy is unreported is a correctness or robustness concern, not circularity.
Assumptions & free parameters
free parameters (1)
- w_MI =
0.01
assumptions (4)
- domain assumption Sensitive attributes (skin type, age) encode clinically relevant diagnostic signals, so removing them harms accuracy.
- domain assumption The router confidence score s_k(x) is a reliable proxy for the sample's true group and for its distance to each group's distribution.
- ad hoc to paper Layer-wise MoE with one expert per group can be trained on the same data as the backbone without extra regularization beyond the MI loss and group-size balancing.
- standard math Mutual information as defined in Eq. (1) is differentiable and optimizable via standard estimators.
Cite this review
Pith. "Pith review of Incorporating Rather Than Eliminating: Achieving Fairness for Skin Disease Diagnosis Through Group-Specific Expert." pith.science (2026). https://pith.science/paper/I5JGXWO3
@misc{pith2026250617787,
author = {Pith},
title = {Pith review of: Incorporating Rather Than Eliminating: Achieving Fairness for Skin Disease Diagnosis Through Group-Specific Expert},
year = {2026},
howpublished = {\url{https://pith.science/paper/I5JGXWO3}},
note = {Machine review of arXiv:2506.17787}
}
read the original abstract
AI-based systems have achieved high accuracy in skin disease diagnostics but often exhibit biases across demographic groups, leading to inequitable healthcare outcomes and diminished patient trust. Most existing bias mitigation methods attempt to eliminate the correlation between sensitive attributes and diagnostic prediction, but those methods often degrade performance due to the lost of clinically relevant diagnostic cues. In this work, we propose an alternative approach that incorporates sensitive attributes to achieve fairness. We introduce FairMoE, a framework that employs layer-wise mixture-of-experts modules to serve as group-specific learners. Unlike traditional methods that rigidly assign data based on group labels, FairMoE dynamically routes data to the most suitable expert, making it particularly effective for handling cases near group boundaries. Experimental results show that, unlike previous fairness approaches that reduce performance, FairMoE achieves substantial accuracy improvements while preserving comparable fairness metrics.
Figures
Reference graph
Works this paper leans on
-
[1]
HealthcareAnalytics 2, 100122 (2022)
Ahammed, M., Al Mamun, M., Uddin, M.S.: A machine learning approach for skin diseasedetection andclassification usingimage segmentation. HealthcareAnalytics 2, 100122 (2022)
work page 2022
-
[2]
In: Proceedings of the European conference on computer vision (ECCV) workshops
Alvi, M., Zisserman, A., Nellåker, C.: Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In: Proceedings of the European conference on computer vision (ECCV) workshops. pp. 0–0 (2018)
work page 2018
-
[3]
Asuncion, A., Newman, D.: Uci machine learning repository (November 2007), https://archive.ics.uci.edu/ml/
work page 2007
-
[4]
Caini, S., Gandini, S., Sera, F., Raimondi, S., Fargnoli, M.C., Boniol, M., Arm- strong, B.K.: Meta-analysis of risk factors for cutaneous melanoma according to anatomicalsiteandclinico-pathologicalvariant.Europeanjournalofcancer 45(17), 3054–3063 (2009)
work page 2009
-
[5]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Chiu, C.H., Chung, H.W., Chen, Y.J., Shi, Y., Ho, T.Y.: Toward fairness through fair multi-exit framework for dermatological disease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 97–107. Springer (2023)
work page 2023
-
[6]
arXiv preprint arXiv:1908.02288 (2019)
Combalia, M., Codella, N.C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A.C., Puig, S., et al.: Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)
arXiv 2019
-
[7]
arXiv preprint arXiv:2405.17782 (2024)
Duan, Y., Tian, Y., Chawla, N., Lemmon, M.: Post-fair federated learning: Achiev- ing group and community fairness in federated learning via post-processing. arXiv preprint arXiv:2405.17782 (2024)
arXiv 2024
-
[8]
Federal Probation80, 38 (2016)
Flores, A.W., Bechtel, K., Lowenkamp, C.T.: False positives, false negatives, and falseanalyses:Arejoindertomachinebias:There’ssoftwareusedacrossthecountry to predict future criminals, and it’s biased against blacks. Federal Probation80, 38 (2016)
work page 2016
Show all 30 references
-
[9]
In: Seminars in oncology nursing
Gordon, R.: Skin cancer: an overview of epidemiology and risk factors. In: Seminars in oncology nursing. vol. 29, pp. 160–169. Elsevier (2013)
2013
-
[10]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., Badri, O.: Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...
2021
-
[11]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Guo, Y., Jia, Z., Hu, J., Shi, Y.: Fairquantize: Achieving fairness through weight quantization for dermatological disease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 329–338. Springer (2024)
2024
-
[12]
He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016) 10 Xu et al
2016
-
[13]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Jung, S., Lee, D., Park, T., Moon, T.: Fair feature distillation for visual recogni- tion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12115–12124 (2021)
2021
-
[14]
In: Medical Image ComputingandComputerAssistedIntervention–MICCAI2024.vol.LNCS15012, pp
Kampen, P.J.T., Christensen, A.N., Hannemose, M.R.: Is this hard for you? Per- sonalized human difficulty estimation for skin lesion diagnosis . In: Medical Image ComputingandComputerAssistedIntervention–MICCAI2024.vol.LNCS15012, pp. 175 – 184. Springer Nature Switzerland (Oct...
2024
-
[15]
In: Interna- tional Conference on Medical Image Computing and Computer-Assisted Interven- tion
Kong, Q., Chiu, C.H., Zeng, D., Chen, Y.J., Ho, T.Y., Hu, J., Shi, Y.: Achieving fairness through channel pruning for dermatological disease diagnosis. In: Interna- tional Conference on Medical Image Computing and Computer-Assisted Interven- tion. pp. 24–34. Springer (2024)
2024
-
[16]
International journal of dermatology49(9), 978–986 (2010)
Narayanan, D.L., Saladi, R.N., Fox, J.L.: Ultraviolet radiation and skin cancer. International journal of dermatology49(9), 978–986 (2010)
2010
-
[17]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Puyol-Antón, E., Ruijsink, B., Piechnik, S.K., Neubauer, S., Petersen, S.E., Razavi, R., King, A.P.: Fairness in cardiac mr image analysis: an investigation of bias due to data imbalance in deep learning based segmentation. In: International Conference on Medical Image Computi...
2021
-
[18]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Quadrianto, N., Sharmanska, V., Thomas, O.: Discovering fair representations in the data domain. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8227–8236 (2019)
2019
-
[19]
arXiv preprint arXiv:1701.06538 (2017)
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)
2017 arXiv
-
[20]
arXiv preprint arXiv:1409.1556 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
2014 arXiv
-
[21]
Scientific data 5(1), 1–9 (2018)
Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)
2018
-
[22]
In: International Conference on Machine Learning
Wang, J., Wang, X.E., Liu, Y.: Understanding instance-level impact of fairness constraints. In: International Conference on Machine Learning. pp. 23114–23130. PMLR (2022)
2022
-
[23]
Wang, M., Deng, W.: Mitigating bias in face recognition using skewness-aware reinforcementlearning.In:ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition. pp. 9322–9331. IEEE (2020)
2020
-
[24]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Wang,Z.,Qinami,K.,Karakozis,I.C.,Genova,K.,Nair,P.,Hata,K.,Russakovsky, O.: Towards fairness in visual recognition: Effective strategies for bias mitigation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8919–8928 (2020)
2020
-
[25]
In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention
Wu, Y., Zeng, D., Xu, X., Shi, Y., Hu, J.: Fairprune: Achieving fairness through pruning for dermatological disease diagnosis. In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention. pp. 743–753. Springer (2022)
2022
-
[26]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Xu, Z., Zhao, S., Quan, Q., Yao, Q., Zhou, S.K.: Fairadabn: Mitigating unfair- ness with adaptive batch normalization and its application to dermatological dis- ease classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp....
2023
-
[27]
Journal of Machine Learning Research 20(75), 1–42 (2019) Achieving Fairness Through Group-Specific Experts 11
Zafar, M.B., Valera, I., Gomez-Rodriguez, M., Gummadi, K.P.: Fairness con- straints: A flexible approach for fair classification. Journal of Machine Learning Research 20(75), 1–42 (2019) Achieving Fairness Through Group-Specific Experts 11
2019
-
[28]
In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society
Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adver- sarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. pp. 335–340 (2018)
2018
-
[29]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Zhang, Y., Cai, R., Chen, T., Zhang, G., Zhang, H., Chen, P.Y., Chang, S., Wang, Z., Liu, S.: Robust mixture-of-expert training for convolutional neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 90–101 (2023)
2023
-
[30]
Computers in Biology and Medicine149, 105966 (2022)
Zhou, J., Wu, Z., Jiang, Z., Huang, K., Guo, K., Zhao, S.: Background selection schema on deep learning-based classification of dermatological disease. Computers in Biology and Medicine149, 105966 (2022)
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.