REVIEW 3 major objections 5 minor 28 references
Comparative assessment of fairness definitions and bias mitigation strategies in machine learning-based diagnosis of Alzheimer's disease from MR images
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper argues that equalized odds is the reliable fairness measure for Alzheimer's MRI diagnosis, and that Reject Option Classification best reduces race and gender bias while adversarial debiasing best handles age.
desk verdict A solid empirical fairness audit for MRI-based AD diagnosis, but the headline rankings rest on an equalized-odds ratio the authors themselves show is unstable at near-zero false-positive rates. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the ratio-based equalized odds metric: for each subgroup pair and each binary subproblem, the true-positive rate (TPR) and false-positive rate (FPR) are expressed as min/max ratios across subgroups and averaged. The proposed composite is the harmonic mean of this ratio and the weighted F1-score; because a harmonic mean is dominated by its smaller component, a model must score well on both diagnostic accuracy and subgroup parity to rank high. The classifier itself is a one-versus-one ensemble of six support vector machines with nested cross-validation; Reject Option Classification post-processing relabels instances whose predicted probability lies near the decision boundary, choosing among 100 candidate thresholds to maximize equalized odds, while adversarial debiasing adds an adversary trained to prevent prediction of the sensitive attribute. An explainability-based feature ranking identifies total brain volume as the main proxy feature for all three sensitive attributes.
What would settle it
Recompute all equalized odds values and harmonic-mean rankings on the same dataset using absolute differences in true-positive and false-positive rates instead of min/max ratios, or using a small smoothing count for zero false positives; if Reject Option Classification no longer yields the reported 46% and 57% improvements and adversarial debiasing no longer yields 40%, the ranking is an artifact of ratio instability.
Extended reading notes
Core claim
The paper's central claim is that the choice of fairness definition changes what a fairness audit concludes, and that equalized odds is the definition best matched to medical diagnosis. Applied to 9,155 participants with volumetric MRI features, the unmitigated SVM ensemble reaches 91% weighted F1-score, yet shows low equalized odds in the MCI/AD subproblem for all three sensitive attributes; race and age exhibit genuine disparities, while the low gender equalized odds traces to a zero false-positive rate in one subgroup rather than to real unfairness. Among the tested mitigation strategies, Reject Option Classification gives the largest equalized-odds gains for race (46%) and gender (57%) in MCI/AD, and adversarial debiasing gives the largest age gain (40%); the paper also finds total brain volume is the dominant proxy feature encoding age, gender, and race, and that correcting for it adds little. The headline methodological contribution is a composite score, the harmonic mean of weighted F1-score and equalized odds ratio, which quantifies the fairness-performance trade-off and drives the method rankings.
Load-bearing premise
The results depend on the assumption that comparing subgroups by the ratio of their true-positive and false-positive rates is a stable measure; that assumption is strained when a subgroup has zero or near-zero false positives, which the paper itself reports in the gender analysis.
Editorial extensions
If this is right
- Fairness audits of MRI-based diagnostic models should report equalized odds together with per-subgroup TPR, FPR, and FNR; accuracy parity and F1 parity near one do not mean the model is fair.
- Post-hoc relabeling of uncertain cases can remove much of the race and gender disparity in MCI/AD discrimination without retraining the classifier.
- For age bias, which stems partly from class imbalance across age groups, threshold relabeling is weaker and adversarial or linear-correction methods look preferable.
- The harmonic-mean composite gives an operational ordering of mitigation methods for medical diagnostics, where a low equalized odds score cannot be hidden by a high F1-score.
- Including total brain volume in covariate correction does not substantially change fairness, suggesting that simply stripping sensitive attributes from the feature space is not the main lever.
Reading between the lines
- A caution implied by the paper's own discussion: the equalized-odds ratio is unstable at zero false-positive counts, so the MCI/AD gender result should be rechecked with absolute differences or smoothed counts before relying on the 57% figure.
- A testable extension would regress out total brain volume before training to see whether demographic leakage flows through global brain size rather than regional volumes.
- A consequence the paper leaves implicit is that counterfactual consistency, which stayed near 1 for every attribute, behaves much like demographic parity and should not be the sole audit metric in screening.
- An intersectional extension, comparing older versus younger participants within each race and gender, would test whether the reported mitigation rankings hold at higher demographic granularity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a fairness analysis of SVM-based ensemble classifiers for CN/MCI/AD diagnosis from MRI-derived volumetric features, using the multi-cohort iSTAGING dataset (9155 participants). It evaluates group fairness metrics (demographic parity, equalized odds ratio, accuracy parity, F1-score parity) and counterfactual fairness, and compares three bias mitigation strategies (linear covariate correction, adversarial debiasing, and Reject Option Classification). A composite metric, defined as the harmonic mean of the weighted F1-score and the equalized odds ratio, is introduced to quantify the fairness–performance trade-off. The main reported findings are that age- and race-related biases exist, gender bias is not significant, total brain volume acts as a proxy feature for all sensitive attributes, and Reject Option Classification yields the largest equalized-odds improvements for race (46%) and gender (57%) in the MCI versus AD subproblem, while adversarial debiasing yields the largest improvement for age (40%).
Significance. If the results were robust, the paper would provide a useful large-scale comparison of fairness metrics and mitigation strategies in a medical diagnostic setting, using a relatively large multi-cohort dataset. The inclusion of multiple sensitive attributes, multiple fairness definitions, and a composite fairness-performance metric is a step toward unified evaluation. The authors are also transparent about limitations of specific metrics, which is commendable. However, the central claim that equalized odds is a reliable fairness measure and that the reported mitigation rankings are meaningful is undermined by the acknowledged instability of the equalized odds ratio when subgroup false positive rates are near zero. Since the headline rankings and the proposed composite metric are built on Eq. (2), the load-bearing part of the analysis needs rework. The paper does not ship code or reproducible experiments, but the data source (iSTAGING) and methods are described.
major comments (3)
- [§IV and Eq. (2), with Table II] The abstract's claim that Reject Option Classification improves gender equalized odds by 57% in the MCI/AD subproblem is based on a metric that the authors themselves state is driven by zero FPR values and does not reflect actual disparities. In Section IV you write that the low equalized odds ratio 'was driven by instances of zero values of the FPR ratio... no actual disparities existed.' Yet the same metric is used to rank mitigation methods and to compute the harmonic mean in Fig. 3 and the abstract. When one subgroup has zero FPR, the ratio is exactly zero; when FPRs are near zero, the ratio is hypersensitive to a handful of false positives. The gender result in Table II (0.47 ± 0.00 to 0.74 ± 0.11) therefore cannot support the claim that ROC is the best mitigation strategy for gender in MCI/AD. Please recompute the analysis with a stable measure, such as absolute TPR/FPR differences or a regularized ratio with a smoothing prior, and re-evaluate all rankings and harmonic-mean scores.
- [§III, Tables I–III] The headline improvements (e.g., race EO ratio from 0.43 ± 0.05 to 0.63 ± 0.18, age EO ratio from 0.42 ± 0.02 to 0.59 ± 0.10) are point estimates with large standard deviations and no significance tests. The standard deviations overlap for some of the key comparisons, so the claimed ranking of mitigation methods (ROC best for race/gender, adversarial best for age) is not statistically supported. Given that the metrics are averages over only five folds, the paper should present paired tests across folds, confidence intervals, or effect sizes before making comparative claims in the abstract and conclusions.
- [§II-A and §II-D] There is a direct contradiction about whether sensitive attributes were included in the model input for adversarial debiasing. Section II-A states that sensitive attributes were excluded from the input space 'except for the cases of adversarial debiasing and counterfactual fairness calculation, in which the respective sensitive attribute was included,' while Section II-D states that adversarial debiasing was excluded from counterfactual fairness evaluation 'since the sensitive attributes could not be included in the input space, due to the adversary's task of predicting them.' This inconsistency is load-bearing because if the sensitive attribute was literally included as a feature in adversarial debiasing, the comparison against other methods is not like-for-like, and the interpretation of the adversarial debiasing results changes substantially. Please clarify what the actual input features were for each method and correct the text.
minor comments (5)
- [Table II] The gender MCI/AD equalized odds ratio for the no-mitigation case is reported as 0.47 ± 0.00. A zero standard deviation across folds is suspicious and likely a rounding artifact; please report more decimals or explain why the value is identical across all folds.
- [§II-E] The abbreviation ROC is used for Reject Option Classification, but ROC is widely used for Receiver Operating Characteristic; please disambiguate at first use to avoid confusion.
- [Abstract and §III] The abstract states that 'no significant gender bias is observed' while also reporting a 57% equalized-odds improvement for gender in the MCI/AD subproblem. These statements appear contradictory; please align the phrasing with the revised metric analysis.
- [Figures 2–4] The figures would be more informative with error bars or a display of fold-level variability, especially given the small number of folds and the large standard deviations reported in the tables.
- [References] Some references are incomplete or informal, for example reference [4] ('ArXiv, pp. arXiv–2308, 2024') lacks a proper title and journal/venue details; please update all references to a consistent format.
Circularity Check
No significant circularity; the fairness metrics and composite measure are explicitly defined and the results are empirical comparisons, not fitted predictions.
full rationale
The paper's claims are empirical comparisons of fairness metrics and bias mitigation strategies; there is no derivation chain in which an output is defined in terms of the target claim. Eq. (1) and Eq. (2) define the harmonic-mean composite metric and the equalized-odds ratio as explicit functions of measured TPR, FPR, and F1 values; these are constructed evaluation metrics, not predictions or fitted parameters. The ranking of Reject Option Classification and adversarial debiasing is obtained by applying these metric definitions to cross-validated results, not by fitting the reported improvements. The choice of equalized odds as the most suitable fairness definition is justified by comparing its behavior with other metrics on the same data, and the paper explicitly discusses a known limitation (zero FPR ratios in the gender MCI/AD case) that the metric can indicate nonexistent disparities; this is a validity caveat, not circularity. The one self-citation (ref [21], for the SVM ensemble and nested-cross-validation setup) is methodological and non-load-bearing: it does not supply the fairness definitions, the composite metric, or the mitigation rankings. Therefore, no load-bearing circular step is present.
Assumptions & free parameters
free parameters (2)
- Age group threshold =
69 years
- Reject Option Classification thresholds =
Selected among 100 candidate thresholds per subproblem
assumptions (4)
- domain assumption Equalized odds is the most appropriate fairness definition for medical diagnostic tasks.
- domain assumption ComBAT harmonization removes site effects while preserving disease and demographic effects.
- domain assumption Total brain volume is the only relevant proxy feature for all three sensitive attributes.
- domain assumption A binary black/white race grouping is sufficient for race fairness analysis.
Cite this review
Pith. "Pith review of Comparative assessment of fairness definitions and bias mitigation strategies in machine learning-based diagnosis of Alzheimer's disease from MR images." pith.science (2026). https://pith.science/paper/TQRD57SH
@misc{pith2026250523528,
author = {Pith},
title = {Pith review of: Comparative assessment of fairness definitions and bias mitigation strategies in machine learning-based diagnosis of Alzheimer's disease from MR images},
year = {2026},
howpublished = {\url{https://pith.science/paper/TQRD57SH}},
note = {Machine review of arXiv:2505.23528}
}
read the original abstract
The present study performs a comprehensive fairness analysis of machine learning (ML) models for the diagnosis of Mild Cognitive Impairment (MCI) and Alzheimer's disease (AD) from MRI-derived neuroimaging features. Biases associated with age, race, and gender in a multi-cohort dataset, as well as the influence of proxy features encoding these sensitive attributes, are investigated. The reliability of various fairness definitions and metrics in the identification of such biases is also assessed. Based on the most appropriate fairness measures, a comparative analysis of widely used pre-processing, in-processing, and post-processing bias mitigation strategies is performed. Moreover, a novel composite measure is introduced to quantify the trade-off between fairness and performance by considering the F1-score and the equalized odds ratio, making it appropriate for medical diagnostic applications. The obtained results reveal the existence of biases related to age and race, while no significant gender bias is observed. The deployed mitigation strategies yield varying improvements in terms of fairness across the different sensitive attributes and studied subproblems. For race and gender, Reject Option Classification improves equalized odds by 46% and 57%, respectively, and achieves harmonic mean scores of 0.75 and 0.80 in the MCI versus AD subproblem, whereas for age, in the same subproblem, adversarial debiasing yields the highest equalized odds improvement of 40% with a harmonic mean score of 0.69. Insights are provided into how variations in AD neuropathology and risk factors, associated with demographic characteristics, influence model fairness.
Figures
Reference graph
Works this paper leans on
-
[1]
Diversity and disparity in dementia: the impact of ethnoracial differences in alzheimer disease,
A. L. Chin, S. Negash, and R. Hamilton, “Diversity and disparity in dementia: the impact of ethnoracial differences in alzheimer disease,” Alzheimer Disease & Associated Disorders, vol. 25, no. 3, pp. 187–195, 2011
work page 2011
-
[2]
Unraveling gender fairness analysis in deep learning prediction of alzheimer’s disease,
S. K. Hammonds, T. Eftestøl, K. Oppedal, and A. Fernandez-Quilez, “Unraveling gender fairness analysis in deep learning prediction of alzheimer’s disease,” in2024 4th International Conference on Applied Artificial Intelligence (ICAPAI), pp. 1–7, IEEE, 2024
work page 2024
-
[3]
Machine learning in neuroimaging: Progress and challenges,
C. Davatzikos, “Machine learning in neuroimaging: Progress and challenges,” NeuroImage, vol. 197, pp. 652–656, 2019
work page 2019
-
[4]
R. Wang, G. Erus, P. Chaudhari, and C. Davatzikos, “Adapting machine learning diagnostic models to new populations using a small amount of data: Results from clinical neuroscience,”ArXiv, pp. arXiv–2308, 2024
work page 2024
-
[5]
Addressing fairness issues in deep learning-based medical image analysis: a systematic review,
Z. Xu, J. Li, Q. Yao, H. Li, M. Zhao, and S. K. Zhou, “Addressing fairness issues in deep learning-based medical image analysis: a systematic review,”npj Digital Medicine , vol. 7, no. 1, p. 286, 2024
work page 2024
-
[6]
Sex and gender considerations in dementia: a call for global research,
C. Vila-Castelar, C. Udeh-Momoh, N. T. Aggarwal, and M. M. Mielke, “Sex and gender considerations in dementia: a call for global research,” Nature aging, vol. 3, no. 5, pp. 463–465, 2023
work page 2023
-
[7]
Medfair: Benchmarking fairness for medical imaging,
Y. Zong, Y. Yang, and T. Hospedales, “Medfair: Benchmarking fairness for medical imaging,” arXiv preprint arXiv:2210.01725, 2022
arXiv 2022
-
[8]
What’s fair about individual fairness?,
W. Fleisher, “What’s fair about individual fairness?,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pp. 480– 490, 2021
work page 2021
Show all 28 references
-
[9]
Counterfactual fairness,
M. J. Kusner, J. Loftus, C. Russell, and R. Silva, “Counterfactual fairness,” Advances in neural information processing systems , vol. 30, 2017
2017
-
[10]
Using explainability for bias mitigation: A case study for fair recruitment assessment,
G. Sogancioglu, H. Kaya, and A. A. Salah, “Using explainability for bias mitigation: A case study for fair recruitment assessment,” in Proceedings of the 25th International Conference on Multimodal Interaction, pp. 631–639, 2023
2023
-
[11]
Fair machine learning in healthcare: A survey,
Q. Feng, M. Du, N. Zou, and X. Hu, “Fair machine learning in healthcare: A survey,”IEEE Transactions on Artificial Intelligence , 2024
2024
-
[12]
Algorithmic fairness of machine learning models for alzheimer disease progression,
C. Yuan, K. A. Linn, and R. A. Hubbard, “Algorithmic fairness of machine learning models for alzheimer disease progression,” JAMA Network Open, vol. 6, no. 11, pp. e2342203–e2342203, 2023
2023
-
[13]
Feature robustness and sex differences in medical imaging: a case study in mri-based alzheimer’s disease detection,
E. Petersen, A. Feragen, M. L. da Costa Zemsch, A. Henriksen, O. E. Wiese Christensen, M. Ganz, and A. D. N. Initiative, “Feature robustness and sex differences in medical imaging: a case study in mri-based alzheimer’s disease detection,” inInternational Conference on Medical ...
2022
-
[14]
Auditing unfair biases in cnn-based diagnosis of alzheimer’s disease,
V. N. Dang, A. Casamitjana, M. P. Starmans, C. Mart ´ın-Isla, J. Hern ´andez-Gonz´alez, K. Lekadir, and f. t. A. Disease Neuroimag- ing Initiative, “Auditing unfair biases in cnn-based diagnosis of alzheimer’s disease,” in Workshop on Clinical Image-Based Proce- dures, pp. 172...
2023
-
[15]
Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing,
S. Dutta, D. Wei, H. Yueksel, P.-Y. Chen, S. Liu, and K. Varshney, “Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing,” inInternational conference on machine learning, pp. 2803–2813, PMLR, 2020
2020
-
[16]
Accurate fairness: Improving individual fairness without trading accuracy,
X. Li, P. Wu, and J. Su, “Accurate fairness: Improving individual fairness without trading accuracy,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 14312–14320, 2023
2023
-
[17]
Optimizing fairness and accuracy: a pareto optimal approach for decision-making,
R. Nagpal, R. Shahsavarifar, V. Goyal, and A. Gupta, “Optimizing fairness and accuracy: a pareto optimal approach for decision-making,” AI and Ethics , pp. 1–14, 2024
2024
-
[18]
Habes, R
M. Habes, R. Pomponio, H. Shou, J. Doshi, E. Mamourian, G. Erus, I. Nasrallah, L. J. Launer, T. Rashid, M. Bilgel, et al., “The brain chart of aging: machine-learning analytics reveals links between brain aging, white matter disease, amyloid burden, and cognition in the istagi...
2021
-
[19]
Muse: Multi-atlas region segmentation utilizing ensembles of registration algo- rithms and parameters, and locally optimal atlas selection,
J. Doshi, G. Erus, Y. Ou, S. M. Resnick, R. C. Gur, R. E. Gur, T. D. Satterthwaite, S. Furth, C. Davatzikos, A. N. Initiative, et al. , “Muse: Multi-atlas region segmentation utilizing ensembles of registration algo- rithms and parameters, and locally optimal atlas selection,”...
2016
-
[20]
Harmonization of large multi-site imaging datasets: Application to 10,232 mris for the analysis of imaging patterns of structural brain change throughout the lifespan,
R. Pomponio, G. Erus, M. Habes, J. Doshi, D. Srinivasan, E. Mamourian, V. Bashyam, Y. Fan, L. J. Launer, C. L. Masters,et al., “Harmonization of large multi-site imaging datasets: Application to 10,232 mris for the analysis of imaging patterns of structural brain change throug...
2019
-
[21]
A comprehensive interpretable machine learning framework for mild cognitive impairment and alzheimer’s disease diagnosis,
M. E. Vlontzou, M. Athanasiou, K. V. Dalakleidi, I. Skampardoni, C. Davatzikos, and K. Nikita, “A comprehensive interpretable machine learning framework for mild cognitive impairment and alzheimer’s disease diagnosis,” Scientific Reports, vol. 15, no. 1, p. 8410, 2025
2025
-
[22]
Mitigating unwanted biases with adversarial learning,
B. H. Zhang, B. Lemoine, and M. Mitchell, “Mitigating unwanted biases with adversarial learning,” in Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pp. 335–340, 2018
2018
-
[23]
Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias,
R. K. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilovi ´c, et al., “Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias,”IBM Journal of Research and Development , vol. 63, no. 4/5, pp. 4–1, 2019
2019
-
[24]
Decision theory for discrimination-aware classification,
F. Kamiran, A. Karim, and X. Zhang, “Decision theory for discrimination-aware classification,” in 2012 IEEE 12th international conference on data mining , pp. 924–929, IEEE, 2012
2012
-
[25]
Structural mri predictors of late-life cognition differ across african americans, hispanics, and whites,
L. B Zahodne, J. J Manly, A. Narkhede, E. Y Griffith, C. DeCarli, N. S Schupf, R. Mayeux, and A. M Brickman, “Structural mri predictors of late-life cognition differ across african americans, hispanics, and whites,” Current Alzheimer Research, vol. 12, no. 7, pp. 632–639, 2015
2015
-
[26]
Sex and gender differences in cognitive and brain reserve: Implications for alzheimer’s disease in women,
S. Subramaniapillai, A. Almey, M. N. Rajah, and G. Einstein, “Sex and gender differences in cognitive and brain reserve: Implications for alzheimer’s disease in women,” Frontiers in Neuroendocrinology , vol. 60, p. 100879, 2021
2021
-
[27]
Sex differences in brain mri using deep learning toward fairer healthcare outcomes,
M. Dibaji, J. Ospel, R. Souza, and M. Bento, “Sex differences in brain mri using deep learning toward fairer healthcare outcomes,” Frontiers in Computational Neuroscience , vol. 18, p. 1452457, 2024
2024
-
[28]
Counterfactual fairness is basically demographic parity,
L. Rosenblatt and R. T. Witter, “Counterfactual fairness is basically demographic parity,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 14461–14469, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.