Pith. sign in

REVIEW 4 major objections 8 minor 30 references

Incorporating Rather Than Eliminating: Achieving Fairness for Skin Disease Diagnosis Through Group-Specific Expert

T0 review · 4 major / 8 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read FairMoE turns sensitive attributes into group-specific experts, raising skin-diagnosis accuracy while keeping fairness.

desk verdict FairMoE's group-specialized MoE is a promising way to use sensitive attributes in skin AI, but the missing capacity-matched baseline and unverified router mean the core claim isn't proven yet. read the letter →

arxiv 2506.17787 v1 pith:I5JGXWO3 submitted 2025-06-21 cs.CV

classification cs.CV
keywords fairnessskindiseasediagnosismixture-of-expertsgroup-specificexpertssoftroutingmutualinformationFitzpatrick-17kISIC2019
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard strategy of removing sensitive attributes from skin-disease classifiers is self-defeating because traits like skin type and age carry clinically relevant diagnostic signal. It proposes FairMoE, a convolutional network in which each convolutional layer is replaced by a mixture-of-experts module with one expert per demographic group, trained so that routers prefer the matching expert while still allowing soft assignment of borderline samples. On Fitzpatrick-17k and ISIC 2019, the authors report that FairMoE raises the average F1 score over the best fairness baselines (0.502 vs 0.476, and 0.767 vs 0.737) while keeping equalized-odds gaps at or below the baselines. The significance, if these results hold, is that fairness need not be purchased by lowering accuracy: incorporating group information can help all groups, especially the unprivileged one.

What carries the argument

The load-bearing object is the layer-wise mixture-of-experts module: each convolutional layer of a ResNet-18 or VGG-11 backbone is replaced by several experts (one per group) and a small routing convolution. Two mechanisms make it work: a mutual-information loss $-I(C;E_Y)$ that establishes a strong correlation between group labels and expert choice, and a soft routing rule that weights the router's confidence score by inverse group size, so selection probability reflects a sample's proximity to each group rather than a hard label. The router's confidence score performs double duty: it is the training signal for specialization and the inference-time routing probability.

What would settle it

Measure the router's agreement with true skin-type or age labels on held-out test data; if that agreement is near chance while FairMoE still reproduces the reported F1 and equalized-odds numbers, then the group-specialization mechanism is not what produces the gains. Alternatively, setting the mutual-information loss term to zero and observing that the gains disappear would confirm the specialization term is doing the work.

Watch

Extended reading notes

Core claim

The central claim is that fairness in skin-disease diagnosis can be improved by incorporating sensitive attributes rather than eliminating them, through a layer-wise mixture-of-experts architecture with group-specific experts. FairMoE makes the router specialize by maximizing the mutual information $I(C;E)$ between group labels and expert assignments, then uses a soft probabilistic routing rule $P(E_k|x) = \alpha_k s_k(x) / \sum_j \alpha_j s_j(x)$, where $s_k(x)$ is the router's confidence that sample $x$ belongs to group $k$ and $\alpha_k$ rebalances group sizes. This lets out-of-group data train an expert when the sample is close to that group's distribution, which is the proposed mechanism for avoiding the distribution shift of hard group-specific training and for handling boundary samples. In the reported experiments, FairMoE attains the highest average F1 on both datasets and equalized-odds values at or below all baselines, so the paper's claim is that accuracy and fairness improve together.

Load-bearing premise

The entire mechanism rests on the router's confidence score being a good estimate of a sample's true group and of how close the sample is to each group's data distribution, and the paper reports no direct measurement of the router's accuracy.

Editorial extensions

If this is right

  • Fairness methods that remove sensitive attributes are not the only viable route; incorporating those attributes can raise accuracy for both groups at once.
  • On the reported results, the unprivileged group improves more than the privileged group, so group-specific experts narrow the gap by lifting the weaker side rather than dropping the stronger side.
  • Because deeper MoE layers contribute more, practitioners with limited compute can replace only the deepest convolutional layers and retain most of the accuracy and fairness benefit.
  • Soft routing gives borderline samples access to more than one expert, which is the proposed answer to the hard-boundary problem of fixed group splitting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the router's confidence score is a good proxy for group proximity, the same architecture should transfer to other diagnostic imaging tasks with clinically relevant demographic attributes, such as age-defined subgroups in retinal or chest imaging.
  • Editorial inference: comparing FairMoE against a variant whose router is trained with explicit group labels would isolate whether the mutual-information objective or the soft routing itself drives the reported gains.
  • Editorial inference: treating skin tone as a continuous rather than binary grouping could make the soft-routing design exploit intermediate Fitzpatrick types instead of forcing them into two buckets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes FairMoE, a layer-wise mixture-of-experts (MoE) architecture for skin disease diagnosis in which each expert is encouraged, via a mutual-information loss, to specialize on one sensitive-attribute group (skin type or age), while a soft probabilistic router uses the router's confidence score to weight training data from other groups. On Fitzpatrick-17k and ISIC 2019, the authors report higher F1 scores and lower equalized-odds values than several fairness baselines, arguing that incorporating sensitive attributes improves accuracy without sacrificing fairness, in contrast to fairness-through-invariance methods.

Significance. If the results hold, FairMoE offers a useful counterpoint to the dominant fairness-through-invariance paradigm in medical imaging: rather than removing sensitive attributes, it exploits them through group-specialized experts and soft routing. The framework is conceptually simple, builds on a standard MoE architecture, and is evaluated on two public benchmarks against recent pruning and quantization baselines. The paper includes an ablation over the number of MoE layers and a case study showing routing-score behavior across layers, which are helpful. However, the strength of the empirical claim depends on several unverified assumptions: router reliability, capacity matching, and statistical stability. The current manuscript does not yet provide enough evidence to fully support the claim of simultaneous accuracy gain and fairness preservation.

major comments (4)
  1. [Section 4, Table 1] The central empirical claim of 'substantial accuracy improvements' is not supported by statistical evidence. No error bars, confidence intervals, or number of seeds are reported for any entry in Table 1. The F1 difference between FairMoE (0.502) and SCP-FairPrune (0.476) on Fitzpatrick-17k is 0.026, and the Eodd differences are even smaller (0.154 vs. 0.164), which may be within run-to-run variation. Please report mean and standard deviation over at least three independent runs and, where possible, a paired significance test for the accuracy and fairness metrics.
  2. [Section 4, Table 1; Section 3.2] FairMoE roughly doubles the number of parameters in each converted layer by introducing two experts per layer, while the baselines (Vanilla, FairAdaBN, SCP-FairPrune, FairQuantize) use a single weight set. The reported accuracy gains could therefore be partly attributable to added model capacity rather than to the proposed group-specialized routing mechanism. Please include a capacity-matched baseline, for example a standard MoE without the mutual-information loss or a single wide network with a comparable parameter count, so that the effect of the group-specialization mechanism is isolated.
  3. [Section 3.4, Eq. (2); Section 4, Fig. 2] The mechanism assumes that the router's confidence score s_k(x) is a reliable estimate of the true sensitive attribute and of the sample's proximity to each group's distribution, yet the paper does not report router accuracy, AUC, calibration, or the empirical mutual information between true group labels and selected experts on held-out data. Fig. 2(a) shows average routing scores across layers, but not prediction accuracy. For ISIC 2019, where the sensitive attribute is age (<=55 vs. >55), it is especially unclear whether the router can infer age from dermoscopic images. If the router is near chance, expert specialization may not occur at inference, and the claimed benefit of soft routing collapses. Please quantify router performance and, if necessary, analyze failure cases.
  4. [Section 3.3, Eq. (1); Section 3.4, Eq. (2)] The methodology for the mutual-information loss is underspecified. The text defines P(E_i|C_j) but Eq. (1) uses the joint P(C_i,E_j), and it is unclear how these probabilities are estimated (per batch or over the whole dataset) and how the loss is differentiated through the router's discrete assignments. In addition, Eq. (2) defines P(E_k|x) with alpha_k = 1/N_k; if s_k(x) is a softmax posterior probability, multiplying by 1/N_k changes the normalization in a way that is not justified, and it is unclear whether alpha is applied to logits or probabilities. Please clarify the notation and the exact computation.
minor comments (8)
  1. [Abstract] The phrase 'the lost of clinically relevant diagnostic cues' should be 'the loss of clinically relevant diagnostic cues'.
  2. [Section 2] The statement that 'malignant melanoma is 20 times more common in African Americans compared to other groups [16]' appears inconsistent with the cited source and with the general epidemiology of melanoma; please verify the direction of this claim.
  3. [Section 4, Fairness Metrics] The metrics Eopp0 and Eopp1 are used in Table 1 but are never defined; please define these subscripts (for example, whether they refer to specific classes or groups) and how they are computed.
  4. [Table 1] The caption should define 'Avg.' and 'Diff.'; in particular, 'Diff.' appears to be the absolute difference between groups, but this should be stated explicitly.
  5. [Table 1] For the Vanilla baseline, all FATE values are 0.000. Please explain how FATE is normalized and why Vanilla is defined as zero; otherwise the FATE comparisons are difficult to interpret.
  6. [Fig. 2(b)] The axes are unclear: F1 Score and Eodd appear to share the same y-axis label but have different scales. Please use separate axes or a clear legend.
  7. [Section 4, ISIC 2019 paragraph] There is a typo: 'group-specific modules for fairnessis' should be 'group-specific modules for fairness'.
  8. [Section 3.3] The phrase 'weighted by a parameterwM I' has a missing space and should read 'weighted by a parameter w_MI'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the FairMoE derivation is an empirical training procedure evaluated against external baselines on public datasets.

full rationale

The paper's central claim is that a layer-wise mixture-of-experts architecture with mutual-information-based expert specialization and soft probabilistic routing improves accuracy while preserving fairness, compared with external baselines on Fitzpatrick-17k and ISIC 2019. The mutual information term in Eq. (1) is a training objective, not a derived prediction of the reported fairness metrics; the routing probability in Eq. (2) uses the router confidence as an assumed proxy for group proximity, which is an unverified modeling assumption rather than a definitional equivalence with the experimental outcome. The reported F1, Eopp, and Eodd values in Table 1 are computed from held-out evaluation and compared against baselines, so the central claim is not forced by construction. Self-citations such as FairPrune, FairQuantize, and the multi-exit fairness work appear as baselines or as related observation about bias in deeper layers, and the core method does not depend on a uniqueness theorem or a prior-work-derived ansatz. No fitted parameter is relabeled as a prediction, and no step of the derivation reduces to its own inputs. The possible weakness that router accuracy is unreported is a correctness or robustness concern, not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The key premises are domain assumptions about the clinical relevance of sensitive attributes and about the router's ability to infer them, plus an ad hoc design choice for group-size balancing. The only numeric hand-set parameter is w_MI = 0.01.

free parameters (1)
  • w_MI = 0.01
    Weight of the mutual information loss; chosen by hand, no ablation reported, so the reported accuracy-fairness balance may depend on this value.
assumptions (4)
  • domain assumption Sensitive attributes (skin type, age) encode clinically relevant diagnostic signals, so removing them harms accuracy.
    This is the paper's core motivation, argued with citations in Sections 1 and 2; if false, incorporating sensitive attributes would not be justified.
  • domain assumption The router confidence score s_k(x) is a reliable proxy for the sample's true group and for its distance to each group's distribution.
    Used in Eq. (2) for soft routing at training and inference; no analysis of router accuracy is provided.
  • ad hoc to paper Layer-wise MoE with one expert per group can be trained on the same data as the backbone without extra regularization beyond the MI loss and group-size balancing.
    The balanced routing with alpha_k = 1/N_k and the layer-wise stacking are design choices specific to this paper.
  • standard math Mutual information as defined in Eq. (1) is differentiable and optimizable via standard estimators.
    Standard definition, but the practical estimator is not given, so this is also a reproducibility gap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incorporating Rather Than Eliminating: Achieving Fairness for Skin Disease Diagnosis Through Group-Specific Expert." pith.science (2026). https://pith.science/paper/I5JGXWO3

@misc{pith2026250617787,
  author       = {Pith},
  title        = {Pith review of: Incorporating Rather Than Eliminating: Achieving Fairness for Skin Disease Diagnosis Through Group-Specific Expert},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5JGXWO3}},
  note         = {Machine review of arXiv:2506.17787}
}
read the original abstract

AI-based systems have achieved high accuracy in skin disease diagnostics but often exhibit biases across demographic groups, leading to inequitable healthcare outcomes and diminished patient trust. Most existing bias mitigation methods attempt to eliminate the correlation between sensitive attributes and diagnostic prediction, but those methods often degrade performance due to the lost of clinically relevant diagnostic cues. In this work, we propose an alternative approach that incorporates sensitive attributes to achieve fairness. We introduce FairMoE, a framework that employs layer-wise mixture-of-experts modules to serve as group-specific learners. Unlike traditional methods that rigidly assign data based on group labels, FairMoE dynamically routes data to the most suitable expert, making it particularly effective for handling cases near group boundaries. Experimental results show that, unlike previous fairness approaches that reduce performance, FairMoE achieves substantial accuracy improvements while preserving comparable fairness metrics.

Figures

Figures reproduced from arXiv: 2506.17787 by the authors.

Figure 1
Figure 1. Overview of FairMoE Framework: (a) Establishing correlations between [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. (a) Router scores for dark and light skin experts across model layers. (b) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 23 canonical work pages

  1. [1]

    HealthcareAnalytics 2, 100122 (2022)

    Ahammed, M., Al Mamun, M., Uddin, M.S.: A machine learning approach for skin diseasedetection andclassification usingimage segmentation. HealthcareAnalytics 2, 100122 (2022)

  2. [2]

    In: Proceedings of the European conference on computer vision (ECCV) workshops

    Alvi, M., Zisserman, A., Nellåker, C.: Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In: Proceedings of the European conference on computer vision (ECCV) workshops. pp. 0–0 (2018)

  3. [3]

    Asuncion, A., Newman, D.: Uci machine learning repository (November 2007), https://archive.ics.uci.edu/ml/

  4. [4]

    Caini, S., Gandini, S., Sera, F., Raimondi, S., Fargnoli, M.C., Boniol, M., Arm- strong, B.K.: Meta-analysis of risk factors for cutaneous melanoma according to anatomicalsiteandclinico-pathologicalvariant.Europeanjournalofcancer 45(17), 3054–3063 (2009)

  5. [5]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Chiu, C.H., Chung, H.W., Chen, Y.J., Shi, Y., Ho, T.Y.: Toward fairness through fair multi-exit framework for dermatological disease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 97–107. Springer (2023)

  6. [6]

    arXiv preprint arXiv:1908.02288 (2019)

    Combalia, M., Codella, N.C., Rotemberg, V., Helba, B., Vilaplana, V., Reiter, O., Carrera, C., Barreiro, A., Halpern, A.C., Puig, S., et al.: Bcn20000: Dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)

  7. [7]

    arXiv preprint arXiv:2405.17782 (2024)

    Duan, Y., Tian, Y., Chawla, N., Lemmon, M.: Post-fair federated learning: Achiev- ing group and community fairness in federated learning via post-processing. arXiv preprint arXiv:2405.17782 (2024)

  8. [8]

    Federal Probation80, 38 (2016)

    Flores, A.W., Bechtel, K., Lowenkamp, C.T.: False positives, false negatives, and falseanalyses:Arejoindertomachinebias:There’ssoftwareusedacrossthecountry to predict future criminals, and it’s biased against blacks. Federal Probation80, 38 (2016)

Show all 30 references
  1. [9]

    In: Seminars in oncology nursing

    Gordon, R.: Skin cancer: an overview of epidemiology and risk factors. In: Seminars in oncology nursing. vol. 29, pp. 160–169. Elsevier (2013)

  2. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., Badri, O.: Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

  3. [11]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Guo, Y., Jia, Z., Hu, J., Shi, Y.: Fairquantize: Achieving fairness through weight quantization for dermatological disease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 329–338. Springer (2024)

  4. [12]

    He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016) 10 Xu et al

  5. [13]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Jung, S., Lee, D., Park, T., Moon, T.: Fair feature distillation for visual recogni- tion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12115–12124 (2021)

  6. [14]

    In: Medical Image ComputingandComputerAssistedIntervention–MICCAI2024.vol.LNCS15012, pp

    Kampen, P.J.T., Christensen, A.N., Hannemose, M.R.: Is this hard for you? Per- sonalized human difficulty estimation for skin lesion diagnosis . In: Medical Image ComputingandComputerAssistedIntervention–MICCAI2024.vol.LNCS15012, pp. 175 – 184. Springer Nature Switzerland (Oct...

  7. [15]

    In: Interna- tional Conference on Medical Image Computing and Computer-Assisted Interven- tion

    Kong, Q., Chiu, C.H., Zeng, D., Chen, Y.J., Ho, T.Y., Hu, J., Shi, Y.: Achieving fairness through channel pruning for dermatological disease diagnosis. In: Interna- tional Conference on Medical Image Computing and Computer-Assisted Interven- tion. pp. 24–34. Springer (2024)

  8. [16]

    International journal of dermatology49(9), 978–986 (2010)

    Narayanan, D.L., Saladi, R.N., Fox, J.L.: Ultraviolet radiation and skin cancer. International journal of dermatology49(9), 978–986 (2010)

  9. [17]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Puyol-Antón, E., Ruijsink, B., Piechnik, S.K., Neubauer, S., Petersen, S.E., Razavi, R., King, A.P.: Fairness in cardiac mr image analysis: an investigation of bias due to data imbalance in deep learning based segmentation. In: International Conference on Medical Image Computi...

  10. [18]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Quadrianto, N., Sharmanska, V., Thomas, O.: Discovering fair representations in the data domain. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8227–8236 (2019)

  11. [19]

    arXiv preprint arXiv:1701.06538 (2017)

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)

  12. [20]

    arXiv preprint arXiv:1409.1556 (2014)

    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  13. [21]

    Scientific data 5(1), 1–9 (2018)

    Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5(1), 1–9 (2018)

  14. [22]

    In: International Conference on Machine Learning

    Wang, J., Wang, X.E., Liu, Y.: Understanding instance-level impact of fairness constraints. In: International Conference on Machine Learning. pp. 23114–23130. PMLR (2022)

  15. [23]

    Wang, M., Deng, W.: Mitigating bias in face recognition using skewness-aware reinforcementlearning.In:ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition. pp. 9322–9331. IEEE (2020)

  16. [24]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Wang,Z.,Qinami,K.,Karakozis,I.C.,Genova,K.,Nair,P.,Hata,K.,Russakovsky, O.: Towards fairness in visual recognition: Effective strategies for bias mitigation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8919–8928 (2020)

  17. [25]

    In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention

    Wu, Y., Zeng, D., Xu, X., Shi, Y., Hu, J.: Fairprune: Achieving fairness through pruning for dermatological disease diagnosis. In: International Conference on Med- ical Image Computing and Computer-Assisted Intervention. pp. 743–753. Springer (2022)

  18. [26]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Xu, Z., Zhao, S., Quan, Q., Yao, Q., Zhou, S.K.: Fairadabn: Mitigating unfair- ness with adaptive batch normalization and its application to dermatological dis- ease classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp....

  19. [27]

    Journal of Machine Learning Research 20(75), 1–42 (2019) Achieving Fairness Through Group-Specific Experts 11

    Zafar, M.B., Valera, I., Gomez-Rodriguez, M., Gummadi, K.P.: Fairness con- straints: A flexible approach for fair classification. Journal of Machine Learning Research 20(75), 1–42 (2019) Achieving Fairness Through Group-Specific Experts 11

  20. [28]

    In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society

    Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adver- sarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. pp. 335–340 (2018)

  21. [29]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, Y., Cai, R., Chen, T., Zhang, G., Zhang, H., Chen, P.Y., Chang, S., Wang, Z., Liu, S.: Robust mixture-of-expert training for convolutional neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 90–101 (2023)

  22. [30]

    Computers in Biology and Medicine149, 105966 (2022)

    Zhou, J., Wu, Z., Jiang, Z., Huang, K., Guo, K., Zhao, S.: Background selection schema on deep learning-based classification of dermatological disease. Computers in Biology and Medicine149, 105966 (2022)

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.