REVIEW 3 major objections 5 minor 1 cited by
Invisible Attributes, Visible Biases: Exploring Demographic Shortcuts in MRI-based Alzheimer's Disease Classification
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Demographic shortcuts learned from brain MRI measurably bias Alzheimer's classification.
desk verdict Sex-based shortcut learning in AD classification is well supported; the race claim, and the abstract's 'both race and sex,' go beyond what the data show. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is the controlled construction of 'baseline' vs 'biased' datasets, in which the joint distribution of protected attribute A and diagnostic label Y is manipulated so that the training set has majority/minority groups opposite to the test set. Combined with this is a rank-based attribution analysis: GradCAM maps are registered to the Hammersmith atlas (a standard parcellation of the brain into anatomical regions), each region's attribution is ranked per sample, and the difference vectors B and P are correlated via Spearman's rho. This quantifies the overlap between features used for protected attribute classification and those changed by the induced bias
What would settle it
A model trained on a sex-balanced dataset that still shows the same high Spearman correlation between B and P (as in the sex-biased experiment) would contradict the claim that induced imbalance causes shortcut learning—if the correlation appears without any imbalance, the metric is not measuring what the paper claims. Alternatively, replacing GradCAM with a different attribution method and finding that the significant correlations vanish would cast doubt on the interpretability chain.
Extended reading notes
Core claim
The paper demonstrates the existence of both race- and sex-based shortcut learning in DL-based AD classification from 3D brain MRI. It first establishes that protected attributes are identifiable from structural MRI, reporting race classification from brain MRI for the first time. It then shows that curated group imbalance causes performance drops on minority groups. The central interpretability discovery is that the rank-based difference between biased and baseline diagnostic models' regional attributions (B = r_BI - r_BA) correlates significantly with the protected-attribute-vs-baseline difference (P = r_PA - r_BA), with Spearman correlations of 0.85/0.66 for the first sex experiment, 0.59
Load-bearing premise
The shortcut-learning conclusion rests on the assumption that GradCAM attributions, after registration to the atlas, faithfully isolate the features the models actually use; if these attributions are noisy or point to uninformative regions, the observed rank correlations do not prove that the biased models are exploiting demographic features.
Editorial extensions
If this is right
- Protected attributes like sex and race are recoverable from structural 3D brain MRI, meaning any downstream diagnostic model could inadvertently encode them.
- Training set imbalance by sex can cause significant drops in Alzheimer's classification performance for minority groups, even when overall class balance is maintained.
- The rank-correlation metric between biased-diagnosis attribution shifts and protected-attribute attribution shifts provides a generalizable tool to detect shortcut learning in medical imaging.
- The weaker race-related findings, driven by limited Black-subject data, indicate that shortcut detection is harder when the minority group is numerically scarce, so larger, more diverse datasets are needed to fully characterize racial bias.
- Models trained on imbalanced data may appear to perform well on the majority while silently failing on minority patients, which could translate to unequal diagnostic accuracy in real clinical deployment.
Reading between the lines
- The authors' rank-correlation approach could be ported to other interpretability methods (e.g., integrated gradients, SHAP) to test whether the shortcut regions identified are method-dependent or genuinely reflect the model's decision process; a robust shortcut should persist across methods.
- If race can be classified from brain MRI as claimed, this has privacy and consent implications for sharing medical imaging datasets—demographic information is not 'invisible' to models even when absent from labels.
- The finding that inverse probability weighting does not eliminate the bias suggests that reweighting alone is insufficient; future work might explore group-aware regularization or adversarial debiasing that directly penalizes demographic classification accuracy.
- A direct extension would be to apply the same biased-dataset construction to other diagnostic tasks (e.g., tumor detection, stroke segmentation) to see whether demographic shortcuts are a general phenomenon across neuroimaging models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether deep learning models for MRI-based Alzheimer's disease (AD) classification learn shortcuts based on protected demographic attributes, specifically sex and race. It comprises three experiments: (1) binary sex/race classification from structural 3D brain MRI on ADNI, OASIS-3, and HCP using ResNet50 and SwinTransformer; (2) CN vs. AD classification on carefully constructed baseline and 'biased' ADNI datasets with differing group imbalances, measuring performance drops; and (3) an interpretability analysis using GradCAM, registration to the Hammersmith atlas, and Spearman rank correlations between bias-related and protected-attribute-related region-rank difference vectors, including a permutation test to account for the shared baseline dependence. The paper concludes that both race- and sex-based shortcut learning and bias occur in DL-based AD classification. Code is provided.
Significance. If the sex-based results hold, the paper makes a solid contribution: it demonstrates, across multiple datasets and two architectural families, that sex can be recovered from brain MRI, that deliberately imbalanced training sets cause measurable performance drops on minority sex groups, and that the resulting feature attributions align with sex-classification attributions. The interpretability methodology, including the permutation test and stability analysis, is a useful addition. The code release is a clear strength for reproducibility. The race-based claims are more preliminary: the Black test samples are small (ADNI n=17), only one race-biased dataset was created, and one of the two models shows a non-significant Spearman correlation for Race 1. The paper's abstract overstates the race evidence, and this discrepancy is load-bearing for the central claim.
major comments (3)
- [Abstract and Section 3, Experiment 3] The abstract states 'we demonstrate the existence of both race and sex based shortcut learning and bias,' but the race evidence is not at the same standard. For Race 1, the Spearman correlation is 0.32 for ResNet50 and 0.08 (p=0.41) for SwinTransformer. Section 2 notes that no Race 2 biased dataset was created and that the race imbalance is less severe due to insufficient Black subjects. The Discussion itself concedes that 'a deeper analysis is required with more Black subjects to produce stronger evidence of shortcut learning and/or distributional shifts.' Thus the race-based demonstration is not established at the level claimed. Please temper the abstract and conclusions (e.g., 'preliminary evidence for race') and ensure the Discussion's limitation is reflected in the headline claims.
- [Section 2 and Table 1b (race classification)] The race classification results rely on very small Black test sets: ADNI n=17, OASIS-3 n=39, HCP n=33. While F1 scores are reported (0.77-0.93), no confidence intervals or bootstrap estimates are given. With n=17, the F1 estimate is highly unstable. This directly affects the race shortcut claims because the biased race dataset inherits this limited sample, and Figure 2 shows a less pronounced performance drop for Race 1. Please report uncertainty (e.g., bootstrap CIs) for the race classification and race-bias results, or explicitly frame the race findings as preliminary due to sample size.
- [Section 3, Experiment 3 (permutation test)] The manuscript reports only binary significance for the permutation test: 'We observed significant p-values (<0.05) for S1 using ResNet50 and S1, R1 using SwinTransformer.' This leaves unclear whether Sex 2 (both models) and Race 1 (ResNet50) were significant. Since the permutation test is the primary safeguard against the mechanical dependence of B and P on r_BA, the exact p-values for all experiments must be reported. If Sex 2 or Race 1 are not significant after the permutation, the interpretability-based shortcut claim for those settings is unsupported. In addition, the stability analysis reports a lower average stability for SwinTransformer (0.79 vs. 0.95), yet SwinTransformer rank correlations are still used; please state how this lower stability affects the interpretability conclusions.
minor comments (5)
- [Table 1 caption] Typo: 'repsectively' should be 'respectively'.
- [Section 2, Experiment 3] Typo: 'netween' should be 'between'.
- [Section 2, Experiment 1] The age threshold for 'younger' and 'older' is described as a 'single age threshold based on absolute age range' but the actual threshold values are not reported. Please state them for each dataset, as this affects the stratification and reproducibility.
- [Section 3, stability analysis] The stability analysis uses the 'top 12.5% patches' and 'radii (with steps) up to 90,' but the number of radii and the step size are unspecified. Please clarify the protocol.
- [Section 2, Experiment 1] The sentence 'One model was trained on each dataset' is ambiguous: does it mean one model per task (race and sex separately) per dataset, or a single multi-task model? Please clarify.
Circularity Check
No circularity: empirical study with external benchmarks, acknowledged limitations, and no fitted parameter renamed as prediction.
full rationale
This is an empirical study, not a derivation chain. The central claims—(1) race and sex can be classified from 3D brain MRI, (2) training-set imbalance can cause performance drops, and (3) the biased model's attribution changes overlap with protected-attribute attribution changes—are each measured on held-out test data drawn from external datasets (ADNI, OASIS-3, HCP). No parameter is fitted to a target and then called a prediction; the F1 scores and accuracy differences in Experiments 1 and 2 are direct observations. The only potentially mechanical step is the interpretability statistic in Experiment 3, where B = r_BI − r_BA and P = r_PA − r_BA share the baseline vector r_BA. The authors explicitly acknowledge this dependence and report a permutation test to account for it: 'We also performed a permutation test to account for dependence of P and B on r_BA. We observed significant p-values (<0.05) for S1 using ResNet50 and S1, R1 using SwinTransformer.' Thus the correlation between B and P is not presented as a raw forced identity; it is tested against a null that removes the shared-baseline artifact. The paper also discloses that the race evidence is weaker and requires more data: 'a deeper analysis is required with more Black subjects to produce stronger evidence of shortcut learning and/or distributional shifts.' That is a limitation on the strength of the claim, not circularity. The only self-citation is [2] (Bercea et al., including two of the present authors), which appears in the Related Work as background on prior sex-bias studies and is not used to justify any of the paper's novel steps. No uniqueness theorem, ansatz, or renaming is invoked. The derivation is self-contained against external benchmarks and the paper's own measurements; no circular step is present.
Assumptions & free parameters
free parameters (2)
- age threshold for younger/older stratification =
not reported
- top-patch fraction for stability analysis =
12.5%
assumptions (4)
- domain assumption Protected attribute labels (race, sex) in ADNI, OASIS-3, and HCP are accurate and reflect the groups the models learn.
- domain assumption GradCAM attributions faithfully reflect the features the models use for prediction.
- domain assumption Registration to the Hammersmith atlas accurately maps attributions to anatomical regions.
- domain assumption The permutation test correctly accounts for the statistical dependence of B and P on r_BA.
Cite this review
Pith. "Pith review of Invisible Attributes, Visible Biases: Exploring Demographic Shortcuts in MRI-based Alzheimer's Disease Classification." pith.science (2026). https://pith.science/paper/GLZEE7MD
@misc{pith2026250909558,
author = {Pith},
title = {Pith review of: Invisible Attributes, Visible Biases: Exploring Demographic Shortcuts in MRI-based Alzheimer's Disease Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/GLZEE7MD}},
note = {Machine review of arXiv:2509.09558}
}
read the original abstract
Magnetic resonance imaging (MRI) is the gold standard for brain imaging. Deep learning (DL) algorithms have been proposed to aid in the diagnosis of diseases such as Alzheimer's disease (AD) from MRI scans. However, DL algorithms can suffer from shortcut learning, in which spurious features, not directly related to the output label, are used for prediction. When these features are related to protected attributes, they can lead to performance bias against underrepresented protected groups, such as those defined by race and sex. In this work, we explore the potential for shortcut learning and demographic bias in DL based AD diagnosis from MRI. We first investigate if DL algorithms can identify race or sex from 3D brain MRI scans to establish the presence or otherwise of race and sex based distributional shifts. Next, we investigate whether training set imbalance by race or sex can cause a drop in model performance, indicating shortcut learning and bias. Finally, we conduct a quantitative and qualitative analysis of feature attributions in different brain regions for both the protected attribute and AD classification tasks. Through these experiments, and using multiple datasets and DL models (ResNet and SwinTransformer), we demonstrate the existence of both race and sex based shortcut learning and bias in DL based AD classification. Our work lays the foundation for fairer DL diagnostic tools in brain MRI. The code is provided at https://github.com/acharaakshit/ShortMR
Figures
Forward citations
Cited by 1 Pith paper
-
Understanding Sources of Demographic Predictability in Brain MRI via Disentangling Anatomy and Contrast
Demographic predictability in brain MRI is primarily carried by anatomical structure, with only a weaker, dataset-specific contribution from acquisition contrast.
Reference graph
Works this paper leans on
-
[1]
MIT press (2023)
Barocas, S., Hardt, M., Narayanan, A.: Fairness and machine learning: Limitations and opportunities. MIT press (2023)
2023
-
[2]
In: Workshop on Clin- ical Image-Based Procedures
Bercea, C.I., Puyol-Antón, E., Wiestler, B., Rueckert, D., Schnabel, J.A., King, A.P.: Bias in unsupervised anomaly detection in brain mri. In: Workshop on Clin- ical Image-Based Procedures. pp. 122–131. Springer (2023)
2023
-
[3]
In: Conference on fairness, accountability and transparency
Buolamwini, J., Gebru, T.: Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency. pp. 77–91. PMLR (2018)
2018
-
[4]
arXiv preprint arXiv:2211.02701 (2022)
Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., Zhao, C., Yang, D., et al.: Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 (2022)
arXiv 2022
-
[5]
In: 2009 IEEE conference on computer vision and pattern recognition
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
2009
-
[6]
Neuroimage62(2), 774–781 (2012)
Fischl, B.: Freesurfer. Neuroimage62(2), 774–781 (2012)
2012
-
[7]
AI and Ethics pp
Friedrich, F., Brack, M., Struppek, L., Hintersdorf, D., Schramowski, P., Luccioni, S., Kersting, K.: Auditing and instructing text-to-image generation models on fair- ness. AI and Ethics pp. 1–21 (2024)
2024
-
[8]
Nature Machine In- telligence2(11), 665–673 (2020) 10 Achara et al
Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine In- telligence2(11), 665–673 (2020) 10 Achara et al
2020
Show all 34 references
-
[9]
The Lancet Digital Health 4(6), e406–e414 (2022)
Gichoya, J.W., Banerjee, I., Bhimireddy, A.R., Burns, J.L., Celi, L.A., Chen, L.C., Correa, R., Dullerud, N., Ghassemi, M., Huang, S.C., et al.: Ai recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health 4(6), e406–e414 (2022)
2022
-
[10]
arXiv preprint arXiv:2503.23398 (2025)
Girrbach, L., Alaniz, S., Smith, G., Akata, Z.: A large scale analysis of gender biases in text-to-image generative models. arXiv preprint arXiv:2503.23398 (2025)
2025
-
[11]
Human brain mapping19(4), 224–247 (2003)
Hammers, A., Allom, R., Koepp, M.J., Free, S.L., Myers, R., Lemieux, L., Mitchell, T.N., Brooks, D.J., Duncan, J.S.: Three-dimensional maximum probability atlas of the human brain, with particular reference to the temporal lobe. Human brain mapping19(4), 224–247 (2003)
2003
-
[12]
He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
2016
-
[13]
In: Proceedings of the AAAI conference on artificial intelligence
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial in...
2019
-
[14]
Human brain mapping 40(17), 4952–4964 (2019)
Isensee, F., Schell, M., Pflueger, I., Brugnara, G., Bonekamp, D., Neuberger, U., Wick, A., Schlemmer, H.P., Heiland, S., Wick, W., et al.: Automated brain extrac- tion of multisequence mri using artificial neural networks. Human brain mapping 40(17), 4952–4964 (2019)
2019
-
[15]
arXiv preprint arXiv:2504.13787 (2025)
Jin, H., Xue, A., You, W., Goel, S., Wong, E.: Probabilistic stability guarantees for feature attributions. arXiv preprint arXiv:2504.13787 (2025)
2025 arXiv
-
[16]
Alzheimer’s Research & Therapy15(1), 84 (2023)
Klingenberg, M., Stark, D., Eitel, F., Budding, C., Habes, M., Ritter, K., Initiative, A.D.N.: Higher performance for women than men in mri-based alzheimer’s disease detection. Alzheimer’s Research & Therapy15(1), 84 (2023)
2023
-
[17]
medrxiv pp
LaMontagne, P.J., Benzinger, T.L., Morris, J.C., Keefe, S., Hornbeck, R., Xiong, C., Grant, E., Hassenstab, J., Moulder, K., Vlassenko, A.G., et al.: Oasis-3: longitu- dinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. medrxiv pp. 2019–...
2019
-
[18]
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:Hierarchical visiontransformerusingshiftedwindows.In:Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
2021
-
[19]
In: Proceedings of the IEEE international conference on computer vision
Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE international conference on computer vision. pp. 3730– 3738 (2015)
2015
-
[20]
arXiv preprint arXiv:1711.05101 (2017)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
2017 arXiv
-
[21]
ACM computing surveys (CSUR)54(6), 1–35 (2021)
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM computing surveys (CSUR)54(6), 1–35 (2021)
2021
-
[22]
Advances in Neural Information Processing Systems35, 10068–10077 (2022)
Moayeri, M., Singla, S., Feizi, S.: Hard imagenet: Segmentations for objects with strong spurious cues. Advances in Neural Information Processing Systems35, 10068–10077 (2022)
2022
-
[23]
In: Proceedings of the ACM conference on health, inference, and learning
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., Ré, C.: Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In: Proceedings of the ACM conference on health, inference, and learning. pp. 151–159 (2020) Invisible Attributes, Visible...
2020
-
[24]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Petersen, E., Feragen, A., da Costa Zemsch, M.L., Henriksen, A., Wiese Chris- tensen, O.E., Ganz, M., Initiative, A.D.N.: Feature robustness and sex differences in medical imaging: a case study in mri-based alzheimer’s disease detection. In: International Conference on Medical...
2022
-
[25]
Neurology74(3), 201–209 (2010)
Petersen, R.C., Aisen, P.S., Beckett, L.A., Donohue, M.C., Gamst, A.C., Harvey, D.J., Jack Jr, C., Jagust, W.J., Shaw, L.M., Toga, A.W., et al.: Alzheimer’s disease neuroimaging initiative (adni) clinical characterization. Neurology74(3), 201–209 (2010)
2010
-
[26]
arXiv preprint arXiv:1911.08731 (2019)
Sagawa, S., Koh, P.W., Hashimoto, T.B., Liang, P.: Distributionally robust neu- ral networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 (2019)
1911 arXiv
-
[27]
In: Proceedings of the IEEE international conference on computer vision
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
2017
-
[28]
The Bell system tech- nical journal27(3), 379–423 (1948)
Shannon, C.E.: A mathematical theory of communication. The Bell system tech- nical journal27(3), 379–423 (1948)
1948
-
[29]
EBioMedicine111(2025)
Stanley, E.A., Souza, R., Wilms, M., Forkert, N.D.: Where, why, and how is bias learned in medical image analysis models? a study of bias encoding within convo- lutional networks using synthetic data. EBioMedicine111(2025)
2025
-
[30]
Journal of Medical Imaging9(6), 061102–061102 (2022)
Stanley, E.A., Wilms, M., Mouches, P., Forkert, N.D.: Fairness-related performance and explainability effects in deep learning models for brain image analysis. Journal of Medical Imaging9(6), 061102–061102 (2022)
2022
-
[31]
Neuroimage80, 62–79 (2013)
Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K., Consortium, W.M.H., et al.: The wu-minn human connectome project: an overview. Neuroimage80, 62–79 (2013)
2013
-
[32]
Proceedings of the National Academy of Sciences120(6), e2211613120 (2023)
Wang, R., Chaudhari, P., Davatzikos, C.: Bias in machine learning models can be significantly mitigated by careful training: Evidence from neuroimaging studies. Proceedings of the National Academy of Sciences120(6), e2211613120 (2023)
2023
-
[33]
EBioMedicine102(2024)
Wang, R., Kuo, P.C., Chen, L.C., Seastedt, K.P., Gichoya, J.W., Celi, L.A.: Drop the shortcuts: image augmentation improves fairness and decreases ai detection of race and other demographics from medical images. EBioMedicine102(2024)
2024
-
[34]
JAMA dermatology 155(10), 1135–1141 (2019)
Winkler, J.K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., et al.: Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learn- ing convolutional neural networ...
2019
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.