REVIEW 4 major objections 5 minor 24 references
Normative Modeling for AD Diagnosis and Biomarker Identification
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that adding focal-loss weighting to an adversarial autoencoder's discriminator makes it a sharper normative model for detecting Alzheimer's disease, improving AUROC and sensitivity.
desk verdict A reasonable incremental idea—focal loss on the ACVAE discriminator—but the headline AUROC claim is contradicted on ADNI and the sample-level split risks subject leakage. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the adversarial focal-loss discriminator integrated into a conditional variational autoencoder (CVAE). The CVAE provides a probabilistic latent space and uses covariates (age, gender, intracranial volume) to condition reconstruction; the discriminator aligns the latent distribution with a prior. Focal loss, defined as $\mathrm{FL}(p) = -\alpha(1-p)^\gamma \log p$ for the positive class and $-(1-\alpha)p^\gamma \log(1-p)$ for the negative class, replaces the standard cross-entropy in the adversarial loss, with parameters $\alpha$ and $\gamma$ controlling how much weight is given to hard samples. This mechanism is what the paper credits for improving sensitivity and for surfacing subtle regional deviations that prior autoencoder models miss.
What would settle it
A subject-level split (grouping all intervals of each individual into either train or test) would settle it: if AUROC and sensitivity drop substantially under that split, the reported gains are partly leakage rather than true generalization.
Extended reading notes
Core claim
The paper's central claim is that embedding a focal-loss-weighted adversarial discriminator into a conditional variational autoencoder (CVAE) improves the performance of normative modeling for Alzheimer's disease. Trained only on healthy controls, the model learns a normative range of regional brain activity; at test time, the mean squared error between input and reconstructed features quantifies neuroanatomical deviation. The focal loss reweights the adversarial objective so the discriminator prioritizes difficult samples, which the authors argue mitigates the bias toward easy, healthy patterns and raises sensitivity to subtle disease-related deviations. Across 30 bootstrap runs on OASIS-3 and ADNI, FAAE attains the highest sensitivity among the compared methods (e.g., 70.00 ± 12.06 on OASIS-3 versus 64.76 ± 17.97 for ACVAE) and the highest or comparable AUROC, and its regional effect-size analysis flags additional prefrontal, precuneus, and somatosensory regions as candidate biomarkers.
Load-bearing premise
The evaluation assumes that each 100-day brain scan interval from the same subject is an independent sample, so an 80/20 random split of these samples can place the same person in both training and test sets without inflating the reported accuracy.
Editorial extensions
If this is right
- If FAAE's sensitivity gains hold in clinical validation, fewer Alzheimer's cases would be missed in screening, at a modest cost in specificity.
- The regional deviation maps produced by the method could serve as a data-driven shortlist of brain regions for biomarker studies, including novel candidates such as the precuneus/posterior cingulate and somatosensory cortex.
- The focal-loss discriminator is a generic component and could be attached to other autoencoder-based normative models or applied to other one-class medical imaging tasks beyond Alzheimer's disease.
- The reported AUROC improvement is modest on ADNI (66.15 vs 67.82 for ACVAE, actually lower) but sensitivity is substantially higher (72.20 vs 67.38), suggesting the main practical gain is recall, not overall discrimination.
- The parameter analysis shows that higher $\gamma$ values (around 15–17.5) with moderate $\alpha$ increase sensitivity, giving practitioners a concrete tuning direction for prioritizing detection rate.
Reading between the lines
- The paper's evaluation splits 100-day intervals of the same subject at the sample level, not the subject level; if a subject-level split were used and performance fell, the reported AUROC and sensitivity gains would partly reflect leakage rather than true generalization. This is a testable extension the authors do not report.
- The focal-loss mechanism could be interpreted as a form of hard-example mining for the discriminator; the same idea might transfer to conditional generative models in other small-sample medical domains where the healthy class dominates.
- The identified 'novel' regions (e.g., somatosensory cortex) could be validated against longitudinal cognitive decline data; if they predict progression, they would be stronger biomarker candidates than simple cross-sectional group differences.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes FAAE, a normative modeling method that combines a conditional variational autoencoder (CVAE) with an adversarial discriminator trained using focal loss, and applies it to Alzheimer's disease diagnosis and regional biomarker identification. The model is trained on healthy controls only, and patient-level deviation is measured by reconstruction error. The authors report experiments on OASIS-3 and ADNI, comparing FAAE with AE, VAE, CVAE, ACVAE, and AAE, and claim that FAAE significantly outperforms prior methods in AUROC and sensitivity. The paper also includes parameter sensitivity and sample-size analyses, and identifies AD-related brain regions via regional deviations.
Significance. If validated, the contribution is a modest but potentially useful modification to adversarial normative modeling: replacing the standard discriminator objective with focal loss could improve sensitivity to hard examples in one-class anomaly detection. The paper has positive elements: public code is provided, the comparison includes several standard deep normative models, the preprocessing is standard (fMRIPrep, Schaefer-100), and bootstrap resampling is used. However, the central claim of significant outperformance is not currently supported: the reported numbers are internally contradictory on ADNI, no statistical significance tests accompany the comparisons, and the evaluation design is vulnerable to subject-level leakage. The significance of the contribution therefore depends on whether a corrected evaluation still shows an advantage over ACVAE.
major comments (4)
- [§3, Table 1] The abstract and §4 claim that FAAE 'significantly outperforms previous state-of-the-art methods' in AUROC and sensitivity, but Table 1 does not support this. On ADNI, FAAE has AUROC 66.15±1.17, which is lower than ACVAE's 67.82±0.98; on OASIS-3, FAAE's AUROC 68.56±3.98 overlaps with ACVAE's 64.64±2.53 when uncertainty is considered. Sensitivity values also have wide standard deviations (e.g., 70.00±12.06 on OASIS-3). No significance test is reported anywhere in the paper. The authors should report paired bootstrap or DeLong tests for AUROC and paired permutation tests for sensitivity/specificity, and then revise the abstract and conclusions to match what the statistical analysis actually shows.
- [§2.1 and §3] The evaluation is vulnerable to subject-level data leakage. Section 2.1 states that 'we treated the data for each 100-day interval as a sample,' and Section 3 describes an 80/20 split of 'randomly selected HC samples' into training and test sets. Both OASIS-3 and ADNI are longitudinal, so multiple 100-day intervals from the same subject are likely present in both the training and test partitions. Because the model is trained only on HC data, HC test intervals from training subjects will tend to have artificially low reconstruction error, while AD test intervals were never seen during training; this mechanically widens the HC/AD deviation gap and inflates AUROC and sensitivity. The paper reports no subject-level split, no count of subjects shared across partitions, and no analysis of how many intervals per subject are used. The 30 bootstrap repetitions resample intervals, not subjects, and therefore do not address this leakage. A subject-level split (or at least a sensitivity analysis excluding shared subjects) is essential before the performance claims can be taken at face value.
- [§3, Fig. 2(c)] The hyperparameter selection procedure is not described, and Figure 2(c) suggests that the focal loss parameters α and γ were evaluated on the test set. The figure reports higher AUROC at γ=15 and γ=17.5, and the text says sensitivity is maximized at γ=15 and α=0.2. However, the exact α and γ used to produce Table 1 are not reported. If the reported results were selected as the maximum over a test-set AUROC surface, the claims are overfit and cannot be compared fairly with baselines that use fixed parameter settings. The authors should report the exact hyperparameters used in Table 1, and should either use a validation split for hyperparameter selection or perform nested cross-validation.
- [§3, Table 2] The sample-size analysis contains an inconsistency that undermines its interpretation. Section 2.1 reports ADNI as 579 total samples with 438 HC samples; an 80% HC split would give about 350 training samples, and the test set would contain the remaining HC samples plus all AD samples. Yet Table 2 reports training sample sizes of 600, 800, 1000, 1200, and 1400 for the ADNI dataset. Unless these are augmented or bootstrap-sampled intervals rather than distinct HC samples, those sample sizes exceed the available HC data. The paper should clarify how these training sets were constructed, and should ensure the description matches the actual experimental procedure.
minor comments (5)
- [§2.2, Eq. (3)] Equation (3) contains a typographical error: Qθ(x|z,c) should be Qθ(z|x,c), consistent with the encoder notation used elsewhere.
- [§3, Fig. 2(c)] The parameter sensitivity figure labels the y-axis as 'AUC' rather than 'AUROC', and the text refers to 'specific α values' without reporting them; please add a legend or table of the α and γ grid values used.
- [§3, Results] The sentence 'The reason behind this is that AAE applies adversarial learning on an imbalanced dataset, which leads to overfitting' is speculative; overfitting is not directly measured, and the argument should be either removed or supported with evidence.
- [§4] The paper has a 'Prospect of Application' section but no explicit Limitations section; given the data-leakage and statistical-testing concerns, a limitations discussion would improve the manuscript.
- [§1] The claim in the introduction that focal loss 'prevents easy examples from dominating the training process' is standard, but the paper does not provide evidence that this is what occurs in the discriminator; consider reporting discriminator loss or gradient statistics to support the mechanism.
Circularity Check
Test-set hyperparameter selection and sample-level splitting make FAAE's reported AUROC/sensitivity gains partially constructed rather than independent predictions.
-
fitted input called prediction
[Section 3, Experimental Settings and Parameter Analysis (Fig. 2(c)); Table 1]
"We split the data into a training set, comprising 80% of the randomly selected HC samples, and a test set, consisting of the remaining HC samples and all AD samples. ... Fig. 2(c) demonstrates the parameter sensitivity analysis of FAAE with respect to the focal loss parameters α and γ on OASIS-3. The results reveal stable AUROC performance across most settings, with a noticeable increase in AUROC values at higher γ levels, particularly γ = 15 and γ = 17.5 combined with specific α values."
The experimental protocol defines only an 80/20 HC train/test split with no validation set. The parameter sensitivity analysis of Fig. 2(c) therefore evaluates AUROC, sensitivity, and specificity directly on the test set. If the configuration used for Table 1 corresponds to the grid points identified as optimal in Fig. 2(c) — the paper does not report the α and γ values used for the headline results — then the reported 'significantly outperforms' claim is a test-set selection artifact. The focal-loss parameters are fitted to the same test labels used for evaluation, so the AUROC/sensitivity improvements over baselines are not independent predictions but consequences of tuning on the test set.
-
other
[Sections 2.1 and 3 (data sampling and split)]
"To handle data collected from multiple periods, we treated the data for each 100-day interval as a sample ... We split the data into a training set, comprising 80% of the randomly selected HC samples, and a test set, consisting of the remaining HC samples and all AD samples."
Both OASIS-3 and ADNI are longitudinal, so one subject contributes multiple 100-day interval samples. A sample-level 80/20 split can place intervals from the same HC subject in both the training and test partitions. Because the FAAE is trained only on HC, HC test intervals drawn from training subjects will have artificially low reconstruction error, while AD test samples — never seen in training — receive no such advantage. This mechanically widens the HC/AD deviation gap and inflates AUROC and sensitivity, exactly the metrics used for the central superiority claim in Table 1. The 30 bootstrap repetitions resample intervals rather than subjects, so the leakage persists across repetitions.
full rationale
The methodological core of the paper — training a CVAE with an adversarial focal-loss discriminator on HC data and scoring deviations by reconstruction MSE — is not circular in itself: the AD labels are used only for evaluation, and the deviation metric is an unsupervised reconstruction error. However, the empirical support for the paper's central claim that FAAE 'significantly outperforms previous state-of-the-art methods' is compromised by two evaluation choices. First, the only parameter analysis for the new focal-loss hyperparameters α and γ is performed on the OASIS-3 test set (Fig. 2(c)), with no validation split and no statement of the α/γ values used in Table 1; if the reported results are the selected best over that test grid, the AUROC/sensitivity advantage is a fitted quantity rather than a prediction. Second, treating each 100-day interval as an independent sample while performing an 80/20 split at the sample level allows the same HC subject to appear in both training and test sets, inflating the HC/AD separation by construction. These are not instances of self-citation or ansatz smuggling; they are evaluation-protocol circularities that directly bear on the headline contribution. The paper's normative-modeling derivation itself is self-contained, so the score reflects partial circularity in the reported performance evidence rather than circularity of the architecture or loss formulation.
Assumptions & free parameters
free parameters (2)
- focal loss alpha =
not reported
- focal loss gamma =
not reported
assumptions (4)
- domain assumption Samples from 100-day intervals of the same subject are independent for train/test splitting
- domain assumption MSE reconstruction error of an HC-trained CVAE is a valid normative deviation metric
- standard math The prior p(z|c) and encoder q(z|x,c) are standard Gaussian conditional distributions (CVAE reparameterization)
- ad hoc to paper Focal loss weighting (α,γ) improves discriminator focus on hard samples without distorting the normative distribution
Cite this review
Pith. "Pith review of Normative Modeling for AD Diagnosis and Biomarker Identification." pith.science (2026). https://pith.science/paper/Q6LXKW66
@misc{pith2026241110570,
author = {Pith},
title = {Pith review of: Normative Modeling for AD Diagnosis and Biomarker Identification},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q6LXKW66}},
note = {Machine review of arXiv:2411.10570}
}
read the original abstract
In this paper, we introduce a novel normative modeling approach that incorporates focal loss and adversarial autoencoders (FAAE) for Alzheimer's Disease (AD) diagnosis and biomarker identification. Our method is an end-to-end approach that embeds an adversarial focal loss discriminator within the autoencoder structure, specifically designed to effectively target and capture more complex and challenging cases. We first use the enhanced autoencoder to create a normative model based on data from healthy control (HC) individuals. We then apply this model to estimate total and regional neuroanatomical deviation in AD patients. Through extensive experiments on the OASIS-3 and ADNI datasets, our approach significantly outperforms previous state-of-the-art methods. This advancement not only streamlines the detection process but also provides a greater insight into the biomarker potential for AD. Our code can be found at \url{https://github.com/soz223/FAAE}.
Figures
Reference graph
Works this paper leans on
-
[1]
Nature computational science 1(9), 598–606 (2021)
Chamberland, M., Genc, S., Tax, C.M., Shastin, D., Koller, K., Raven, E.P., Cun- ningham, A., Doherty, J., van den Bree, M.B., Parker, G.D., et al.: Detecting mi- crostructural deviations in individuals with deep diffusion mri tractometry. Nature computational science 1(9), 598–606 (2021)
work page 2021
-
[2]
Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders
Chen, X., Konukoglu, E.: Unsupervised detection of lesions in brain mri using constrained adversarial auto-encoders. arXiv preprint arXiv:1806.04972 (2018)
work page Pith review arXiv 2018
-
[3]
Nature methods16(1), 111–116 (2019)
Esteban, O., Markiewicz, C.J., Blair, R.W., Moodie, C.A., Isik, A.I., Erramuzpe, A., Kent, J.D., Goncalves, M., DuPre, E., Snyder, M., et al.: fmriprep: a robust preprocessing pipeline for functional mri. Nature methods16(1), 111–116 (2019)
work page 2019
-
[4]
Information Processing & Management57(3), 102192 (2020)
Gao, F., Zhu, J., Jiang, H., Niu, Z., Han, W., Yu, J.: Incremental focal loss gans. Information Processing & Management57(3), 102192 (2020)
work page 2020
-
[5]
Journal of Neuroscience23(3), 986–993 (2003)
Grady, C.L., McIntosh, A.R., Beig, S., Keightley, M.L., Burian, H., Black, S.E.: Evidence from functional neuroimaging of a compensatory prefrontal network in alzheimer’s disease. Journal of Neuroscience23(3), 986–993 (2003)
work page 2003
-
[6]
Brain communications3(3), fcab125 (2021)
Jobson, D.D., Hase, Y., Clarkson, A.N., Kalaria, R.N.: The role of the medial prefrontal cortex in cognition, ageing and dementia. Brain communications3(3), fcab125 (2021)
work page 2021
-
[7]
arXiv preprint arXiv:1312.6114 (2013)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
arXiv 2013
-
[8]
In: International conference on machine learning
Kusner, M.J., Paige, B., Hernández-Lobato, J.M.: Grammar variational autoen- coder. In: International conference on machine learning. pp. 1945–1954. PMLR (2017)
work page 2017
Show all 24 references
-
[9]
MedRxiv pp
LaMontagne, P.J., Benzinger, T.L., Morris, J.C., Keefe, S., Hornbeck, R., Xiong, C., Grant, E., Hassenstab, J., Moulder, K., Vlassenko, A.G., et al.: Oasis-3: longitu- dinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. MedRxiv pp. 2019–...
2019
-
[10]
In: In- ternational Conference on Medical Image Computing and Computer-Assisted In- tervention
Lawry Aguila, A., Chapman, J., Janahi, M., Altmann, A.: Conditional vaes for confound removal and normative modelling of neurodegenerative diseases. In: In- ternational Conference on Medical Image Computing and Computer-Assisted In- tervention. pp. 430–440. Springer (2022)
2022
-
[11]
arXiv preprint arXiv:2207.07739 (2022) 10 S
Liu, C., Dong, X., Potter, M., Chang, H.M., Soni, R.: Adversarial focal loss: Asking your discriminator for hard examples. arXiv preprint arXiv:2207.07739 (2022) 10 S. Zhao et al
2022 arXiv
-
[12]
Biological psychiatry 80(7), 552–561 (2016)
Marquand, A.F., Rezek, I., Buitelaar, J., Beckmann, C.F.: Understanding hetero- geneity in clinical cohorts using normative models: beyond case-control studies. Biological psychiatry 80(7), 552–561 (2016)
2016
-
[13]
Neuroimaging Clinics15(4), 869–877 (2005)
Mueller, S.G., Weiner, M.W., Thal, L.J., Petersen, R.C., Jack, C., Jagust, W., Trojanowski, J.Q., Toga, A.W., Beckett, L.: The alzheimer’s disease neuroimaging initiative. Neuroimaging Clinics15(4), 869–877 (2005)
2005
-
[14]
Human brain mapping40(3), 944–954 (2019)
Pinaya, W.H., Mechelli, A., Sato, J.R.: Using deep autoencoders to identify abnor- mal brain structural patterns in neuropsychiatric disorders: A large-scale multi- sample study. Human brain mapping40(3), 944–954 (2019)
2019
-
[15]
Scientific reports11(1), 15746 (2021)
Pinaya, W.H., Scarpazza, C., Garcia-Dias, R., Vieira, S., Baecker, L., F da Costa, P., Redolfi, A., Frisoni, G.B., Pievani, M., Calhoun, V.D., et al.: Using norma- tive modelling to detect disease progression in mild cognitive impairment and alzheimer’s disease in a cross-sect...
2021
-
[16]
bioRxiv pp
Pinaya, W.H., Scarpazza, C., Garcia-Dias, R., Vieira, S., Baecker, L., da Costa, P.F., Redolfi, A., Frisoni, G.B., Pievani, M., Calhoun, V.D., et al.: Normative mod- elling using deep autoencoders: a multi-cohort study on mild cognitive impairment and alzheimer’s disease. bioR...
2020
-
[17]
Cerebral cortex28(9), 3095–3114 (2018)
Schaefer, A., Kong, R., Gordon, E.M., Laumann, T.O., Zuo, X.N., Holmes, A.J., Eickhoff, S.B., Yeo, B.T.: Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex28(9), 3095–3114 (2018)
2018
-
[18]
In: International conference on information processing in medical imag- ing
Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsu- pervised anomaly detection with generative adversarial networks to guide marker discovery. In: International conference on information processing in medical imag- ing. pp. 146–157. Springer (2017)
2017
-
[19]
Advances in neural information processing systems 28 (2015)
Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. Advances in neural information processing systems 28 (2015)
2015
-
[20]
Human brain mapping28(10), 967–978 (2007)
Wang, K., Liang, M., Wang, L., Tian, L., Zhang, X., Li, K., Jiang, T.: Altered functional connectivity in early alzheimer’s disease: A resting-state fmri study. Human brain mapping28(10), 967–978 (2007)
2007
-
[21]
In: 2023 IEEE 20th International Symposium on Biomedical Imaging
Wang, X., Zhou, R., Zhao, K., Leow, A., Zhang, Y., He, L.: Normative modeling via conditional variational autoencoder and adversarial learning to identify brain dysfunction in alzheimer’s disease. In: 2023 IEEE 20th International Symposium on Biomedical Imaging. pp. 1–4. IEEE (2023)
2023
-
[22]
Brain Structure and Function220, 745–762 (2015)
Wang, Z., Xia, M., Dai, Z., Liang, X., Song, H., He, Y., Li, K.: Differentially disrupted functional connectivity of the subregions of the inferior parietal lobule in alzheimer’s disease. Brain Structure and Function220, 745–762 (2015)
2015
-
[23]
arXiv preprint arXiv:2305.00738 (2023)
Wicaksana, J., Yan, Z., Cheng, K.T.: Fca: Taming long-tailed federated medical image classification by classifier anchoring. arXiv preprint arXiv:2305.00738 (2023)
2023 arXiv
-
[24]
In: International Conference on Medical image computing and computer-assisted intervention
Wolleb, J., Bieder, F., Sandkühler, R., Cattin, P.C.: Diffusion models for medical anomaly detection. In: International Conference on Medical image computing and computer-assisted intervention. pp. 35–45. Springer (2022)
2022
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.