{"id":"64af5380-41ff-4e60-aabd-fe384277f034","arxiv_id":"2411.10570","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Focal loss added to an adversarial autoencoder improves sensitivity of Alzheimer's disease detection via normative modeling, but AUROC gains are inconsistent across datasets.","lead":"This paper presents a new Alzheimer's disease detection method that combines focal loss with adversarial autoencoders to learn what healthy brains look like, then flags patients whose brain patterns deviate from that norm. The authors report improved sensitivity on the OASIS-3 and ADNI datasets, though AUROC gains are inconsistent and one dataset shows no improvement.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sample-level train/test splitting risks subject leakage in longitudinal OASIS-3/ADNI, potentially inflating FAAE's reported AUROC/sensitivity advantage; a subject-level split is needed.","rationale":"The strongest claim is empirical: FAAE 'significantly outperforms previous state-of-the-art methods.' The most load-bearing condition is the integrity of the train/test comparison. The paper's own description of the split, combined with the longitudinal nature of the datasets, creates a clear risk of subject-level leakage: the same subject's 100-day intervals can appear on both sides of the split. This would systematically lower HC reconstruction errors in the test set and inflate all separation metrics, making the FAAE's advantage over baselines unreliable. The reader's weakest assumption identifies exactly this issue; I agree. I also considered the undisclosed focal-loss hyperparameters and the lack of statistical significance tests, but those are secondary: if the leakage is real, no amount of hyperparameter reporting would salvage the comparison, and significance tests on leaked data are meaningless. The paper is otherwise well-specified—the loss equations are coherent, and the code release makes a subject-level re-run feasible. Since the concern is concrete and testable, and the reader already conditioned on it, the verdict remains CONDITIONAL.","tokens_in":7795,"tokens_out":4637,"duration_ms":42184,"concrete_test":"Use the released code to re-run the full pipeline (FAAE and the five baselines) with a strict subject-level split: group all 100-day intervals by subject ID, randomly assign 80% of HC subjects to training and the remaining HC subjects plus all AD subjects to testing, and keep the same hyperparameters and number of bootstrap iterations. Then compare FAAE vs. ACVAE and AAE on OASIS-3 and ADNI using AUROC and sensitivity with confidence intervals. If the FAAE advantage over the best baseline drops below the margin in Table 1 or is no longer statistically significant, the sample-level leakage is the cause and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 states that 'we treated the data for each 100-day interval as a sample' and Section 3 describes an 80/20 split of 'randomly selected HC samples' into training and test sets. Both OASIS-3 and ADNI are longitudinal, so the same subject contributes multiple 100-day samples. A sample-level split can place intervals from the same subject in both training and test partitions. Because the FAAE is trained exclusively on HC, any HC test samples drawn from training subjects will have artificially low reconstruction errors (the model has already seen that subject's brain patterns), while AD test samples—never seen during training—are not similarly favored. This mechanically widens the HC/AD deviation gap and inflates AUROC and sensitivity, exactly the metrics used to claim superiority in Table 1. The paper reports no subject IDs, no subject-level split, and no analysis of how many subjects are shared across partitions. The 30 bootstrap repetitions only resample the sample-level split and do not remove the leakage. This is load-bearing because the headline claim 'significantly outperforms previous state-of-the-art methods' rests on these numbers; if the gap shrinks or reverses under a proper subject-level split, the central contribution is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FAAE, a normative modeling method that combines a conditional variational autoencoder (CVAE) with an adversarial discriminator trained using focal loss, and applies it to Alzheimer's disease diagnosis and regional biomarker identification. The model is trained on healthy controls only, and patient-level deviation is measured by reconstruction error. The authors report experiments on OASIS-3 and ADNI, comparing FAAE with AE, VAE, CVAE, ACVAE, and AAE, and claim that FAAE significantly outperforms prior methods in AUROC and sensitivity. The paper also includes parameter sensitivity and sample-size analyses, and identifies AD-related brain regions via regional deviations.","tokens_in":8098,"tokens_out":3844,"duration_ms":39391,"significance":"If validated, the contribution is a modest but potentially useful modification to adversarial normative modeling: replacing the standard discriminator objective with focal loss could improve sensitivity to hard examples in one-class anomaly detection. The paper has positive elements: public code is provided, the comparison includes several standard deep normative models, the preprocessing is standard (fMRIPrep, Schaefer-100), and bootstrap resampling is used. However, the central claim of significant outperformance is not currently supported: the reported numbers are internally contradictory on ADNI, no statistical significance tests accompany the comparisons, and the evaluation design is vulnerable to subject-level leakage. The significance of the contribution therefore depends on whether a corrected evaluation still shows an advantage over ACVAE.","major_comments":[{"comment":"The abstract and §4 claim that FAAE 'significantly outperforms previous state-of-the-art methods' in AUROC and sensitivity, but Table 1 does not support this. On ADNI, FAAE has AUROC 66.15±1.17, which is lower than ACVAE's 67.82±0.98; on OASIS-3, FAAE's AUROC 68.56±3.98 overlaps with ACVAE's 64.64±2.53 when uncertainty is considered. Sensitivity values also have wide standard deviations (e.g., 70.00±12.06 on OASIS-3). No significance test is reported anywhere in the paper. The authors should report paired bootstrap or DeLong tests for AUROC and paired permutation tests for sensitivity/specificity, and then revise the abstract and conclusions to match what the statistical analysis actually shows.","section":"§3, Table 1"},{"comment":"The evaluation is vulnerable to subject-level data leakage. Section 2.1 states that 'we treated the data for each 100-day interval as a sample,' and Section 3 describes an 80/20 split of 'randomly selected HC samples' into training and test sets. Both OASIS-3 and ADNI are longitudinal, so multiple 100-day intervals from the same subject are likely present in both the training and test partitions. Because the model is trained only on HC data, HC test intervals from training subjects will tend to have artificially low reconstruction error, while AD test intervals were never seen during training; this mechanically widens the HC/AD deviation gap and inflates AUROC and sensitivity. The paper reports no subject-level split, no count of subjects shared across partitions, and no analysis of how many intervals per subject are used. The 30 bootstrap repetitions resample intervals, not subjects, and therefore do not address this leakage. A subject-level split (or at least a sensitivity analysis excluding shared subjects) is essential before the performance claims can be taken at face value.","section":"§2.1 and §3"},{"comment":"The hyperparameter selection procedure is not described, and Figure 2(c) suggests that the focal loss parameters α and γ were evaluated on the test set. The figure reports higher AUROC at γ=15 and γ=17.5, and the text says sensitivity is maximized at γ=15 and α=0.2. However, the exact α and γ used to produce Table 1 are not reported. If the reported results were selected as the maximum over a test-set AUROC surface, the claims are overfit and cannot be compared fairly with baselines that use fixed parameter settings. The authors should report the exact hyperparameters used in Table 1, and should either use a validation split for hyperparameter selection or perform nested cross-validation.","section":"§3, Fig. 2(c)"},{"comment":"The sample-size analysis contains an inconsistency that undermines its interpretation. Section 2.1 reports ADNI as 579 total samples with 438 HC samples; an 80% HC split would give about 350 training samples, and the test set would contain the remaining HC samples plus all AD samples. Yet Table 2 reports training sample sizes of 600, 800, 1000, 1200, and 1400 for the ADNI dataset. Unless these are augmented or bootstrap-sampled intervals rather than distinct HC samples, those sample sizes exceed the available HC data. The paper should clarify how these training sets were constructed, and should ensure the description matches the actual experimental procedure.","section":"§3, Table 2"}],"minor_comments":[{"comment":"Equation (3) contains a typographical error: Qθ(x|z,c) should be Qθ(z|x,c), consistent with the encoder notation used elsewhere.","section":"§2.2, Eq. (3)"},{"comment":"The parameter sensitivity figure labels the y-axis as 'AUC' rather than 'AUROC', and the text refers to 'specific α values' without reporting them; please add a legend or table of the α and γ grid values used.","section":"§3, Fig. 2(c)"},{"comment":"The sentence 'The reason behind this is that AAE applies adversarial learning on an imbalanced dataset, which leads to overfitting' is speculative; overfitting is not directly measured, and the argument should be either removed or supported with evidence.","section":"§3, Results"},{"comment":"The paper has a 'Prospect of Application' section but no explicit Limitations section; given the data-leakage and statistical-testing concerns, a limitations discussion would improve the manuscript.","section":"§4"},{"comment":"The claim in the introduction that focal loss 'prevents easy examples from dominating the training process' is standard, but the paper does not provide evidence that this is what occurs in the discriminator; consider reporting discriminator loss or gradient statistics to support the mechanism.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is currently not supported by its own reported numbers, and the longitudinal data handling raises a serious leakage risk. A revision that addresses the subject-level split, adds significance tests, and reports exact hyperparameters could make the contribution publishable, but as written the primary result is not established. I would not recommend rejection outright because the proposed architectural change is plausible and the identified issues are potentially fixable with additional experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward extension of the authors' own ACVAE normative model: replace the discriminator's cross-entropy loss with focal loss to focus on hard samples. That's the whole novelty, and it's a sensible idea worth a quick test. The writing is clear, the loss definitions are precise, and the code is public.\n\nWhat the paper does well: it compares against five autoencoder/VAE baselines on two datasets, reports AUROC, sensitivity, and specificity, and includes regional deviation maps that pick out known AD-related regions. On OASIS-3 the gain looks real: AUROC from 64.6 to 68.6, sensitivity from 64.8 to 70.0. That is the kind of incremental improvement that can be believable.\n\nSoft spots, in order of severity:\n\n1. The abstract says \"significantly outperforms previous state-of-the-art methods in terms of AUROC and sensitivity.\" On ADNI, FAAE's AUROC (66.15) is below ACVAE's (67.82). So the AUROC claim is simply false for one of the two datasets, and no significance tests are reported anywhere. The \"significantly\" is unsupported.\n\n2. The more serious problem is the train/test split. Section 2.1 says each 100-day interval is treated as a sample, and Section 3 describes an 80/20 split of HC samples. Both datasets are longitudinal, so the same subject can contribute intervals to both training and test. Since the model is trained only on HC, test HC samples from seen subjects will have artificially low reconstruction errors, widening the HC/AD gap. This is load-bearing because the paper's central claim rests on those AUROC and sensitivity numbers. A subject-level split, or at least a per-subject cross-validation, is needed before the results can be trusted.\n\n3. The focal-loss hyperparameters α and γ used in Table 1 are not reported. Figure 2(c) shows the OASIS-3 AUROC surface with distinctly better numbers at γ=15 and 17.5. It looks like they selected settings from the test set. They should state the exact values and tune on a validation fold.\n\n4. Minor: OASIS-3 has only 21 AD samples, so those numbers are fragile.\n\nThe core idea is plausible and the implementation is clean, but the empirical support as written is not there. A subject-level split, reported hyperparameters, and significance tests could turn this into a small, acceptable contribution for a medical imaging venue. As it stands, it needs major revision.\n\nMy recommendation: yes, send it to peer review, because the idea and the leakage issue are worth referees' attention. But expect the reviewers to demand the split correction. If you are looking for something to cite in your own work, this is not it right now.","headline":"A reasonable incremental idea—focal loss on the ACVAE discriminator—but the headline AUROC claim is contradicted on ADNI and the sample-level split risks subject leakage.","tokens_in":8593,"tokens_out":2418,"would_cite":false,"duration_ms":24564,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding focal-loss weighting to an adversarial autoencoder's discriminator makes it a sharper normative model for detecting Alzheimer's disease, improving AUROC and sensitivity.","keywords":["Normative modeling","Focal loss","Adversarial learning","Autoencoder","Alzheimer's Disease","fMRI","Biomarker discovery"],"falsifier":"A subject-level split (grouping all intervals of each individual into either train or test) would settle it: if AUROC and sensitivity drop substantially under that split, the reported gains are partly leakage rather than true generalization.","tokens_in":7621,"feed_emoji":"🧠","tokens_out":2878,"duration_ms":26192,"temperature":0.7,"pith_summary":"The paper introduces FAAE, a normative modeling method that combines a conditional variational autoencoder with an adversarial discriminator whose loss is weighted by focal loss. The goal is to build a model of healthy brain activity from healthy controls and then flag Alzheimer's patients by measuring reconstruction deviation, with the focal loss forcing the discriminator to attend to hard, easily confused cases. The authors report that on the OASIS-3 and ADNI fMRI datasets, FAAE achieves higher AUROC and sensitivity than five prior autoencoder normative models, and that it identifies both known and novel AD-related brain regions. If correct, this would give clinicians a more sensitive screening tool that also points to potential biomarkers.","feed_headline":"Focal-loss autoencoder sharpens Alzheimer's detection","feed_subtitle":"Adding focal-loss weighting to an adversarial autoencoder improves sensitivity and pins down AD-related brain regions on two fMRI datasets.","key_machinery":"The key machinery is the adversarial focal-loss discriminator integrated into a conditional variational autoencoder (CVAE). The CVAE provides a probabilistic latent space and uses covariates (age, gender, intracranial volume) to condition reconstruction; the discriminator aligns the latent distribution with a prior. Focal loss, defined as $\\mathrm{FL}(p) = -\\alpha(1-p)^\\gamma \\log p$ for the positive class and $-(1-\\alpha)p^\\gamma \\log(1-p)$ for the negative class, replaces the standard cross-entropy in the adversarial loss, with parameters $\\alpha$ and $\\gamma$ controlling how much weight is given to hard samples. This mechanism is what the paper credits for improving sensitivity and for surfacing subtle regional deviations that prior autoencoder models miss.","core_discovery":"The paper's central claim is that embedding a focal-loss-weighted adversarial discriminator into a conditional variational autoencoder (CVAE) improves the performance of normative modeling for Alzheimer's disease. Trained only on healthy controls, the model learns a normative range of regional brain activity; at test time, the mean squared error between input and reconstructed features quantifies neuroanatomical deviation. The focal loss reweights the adversarial objective so the discriminator prioritizes difficult samples, which the authors argue mitigates the bias toward easy, healthy patterns and raises sensitivity to subtle disease-related deviations. Across 30 bootstrap runs on OASIS-3 and ADNI, FAAE attains the highest sensitivity among the compared methods (e.g., 70.00 ± 12.06 on OASIS-3 versus 64.76 ± 17.97 for ACVAE) and the highest or comparable AUROC, and its regional effect-size analysis flags additional prefrontal, precuneus, and somatosensory regions as candidate biomarkers.","pith_inferences":["The paper's evaluation splits 100-day intervals of the same subject at the sample level, not the subject level; if a subject-level split were used and performance fell, the reported AUROC and sensitivity gains would partly reflect leakage rather than true generalization. This is a testable extension the authors do not report.","The focal-loss mechanism could be interpreted as a form of hard-example mining for the discriminator; the same idea might transfer to conditional generative models in other small-sample medical domains where the healthy class dominates.","The identified 'novel' regions (e.g., somatosensory cortex) could be validated against longitudinal cognitive decline data; if they predict progression, they would be stronger biomarker candidates than simple cross-sectional group differences."],"forward_implications":["If FAAE's sensitivity gains hold in clinical validation, fewer Alzheimer's cases would be missed in screening, at a modest cost in specificity.","The regional deviation maps produced by the method could serve as a data-driven shortlist of brain regions for biomarker studies, including novel candidates such as the precuneus/posterior cingulate and somatosensory cortex.","The focal-loss discriminator is a generic component and could be attached to other autoencoder-based normative models or applied to other one-class medical imaging tasks beyond Alzheimer's disease.","The reported AUROC improvement is modest on ADNI (66.15 vs 67.82 for ACVAE, actually lower) but sensitivity is substantially higher (72.20 vs 67.38), suggesting the main practical gain is recall, not overall discrimination.","The parameter analysis shows that higher $\\gamma$ values (around 15–17.5) with moderate $\\alpha$ increase sensitivity, giving practitioners a concrete tuning direction for prioritizing detection rate."],"supporting_citations":[{"why":"Defines the normative modeling framework that the paper builds upon, comparing individual deviations against a healthy reference range.","marker":"[12]"},{"why":"Provides a deep autoencoder normative modeling baseline and the cohort-based methodology that FAAE extends.","marker":"[15]"},{"why":"The ACVAE method and experimental setting (dataset split, normalization, architecture) that FAAE follows and directly compares against.","marker":"[21]"},{"why":"Supplies the incremental focal loss GAN formulation that the focal-loss discriminator is adapted from.","marker":"[4]"},{"why":"Argues that adversarial discriminators struggle with hard samples and motivates using adversarial focal loss for such cases.","marker":"[11]"},{"why":"The OASIS-3 dataset that provides the main validation data for the reported results.","marker":"[9]"},{"why":"The ADNI dataset that provides the second independent cohort for evaluation.","marker":"[13]"},{"why":"The Schaefer-100 atlas used to parcellate the brain into 100 ROIs, generating the regional features used throughout.","marker":"[17]"}],"fun_headline_variants":["Focal loss zeroes in on tough Alzheimer's cases","Adversarial autoencoder pinpoints Alzheimer's brain regions","Normative model with focal loss flags AD biomarkers","Autoencoder learns norm from healthy brain, spots AD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that each 100-day brain scan interval from the same subject is an independent sample, so an 80/20 random split of these samples can place the same person in both training and test sets without inflating the reported accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Focal loss zeroes in on tough Alzheimer's cases","Adversarial autoencoder pinpoints Alzheimer's brain regions","Normative model with focal loss flags AD biomarkers","Autoencoder learns norm from healthy brain, spots AD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00079,"raw_usage":{"total_tokens":3457,"prompt_tokens":896,"completion_tokens":2561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":2497}},"tokens_in":512,"tokens_out":2561,"duration_ms":22741,"temperature":1.0,"reasoning_tokens":2497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:32:40.931103+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A subject-level split (grouping all intervals of each individual into either train or test) would settle it: if AUROC and sensitivity drop substantially under that split, the reported gains are partly leakage rather than true generalization.","supporting_citations":[{"cited_title":"Biological psychiatry 80(7), 552–561 (2016)","cited_arxiv_id":null,"evidence_quote":"Defines the normative modeling framework that the paper builds upon, comparing individual deviations against a healthy reference range."},{"cited_title":"Scientific reports11(1), 15746 (2021)","cited_arxiv_id":null,"evidence_quote":"Provides a deep autoencoder normative modeling baseline and the cohort-based methodology that FAAE extends."},{"cited_title":"In: 2023 IEEE 20th International Symposium on Biomedical Imaging","cited_arxiv_id":null,"evidence_quote":"The ACVAE method and experimental setting (dataset split, normalization, architecture) that FAAE follows and directly compares against."},{"cited_title":"Information Processing & Management57(3), 102192 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the incremental focal loss GAN formulation that the focal-loss discriminator is adapted from."},{"cited_title":"Adversarial Focal Loss: Asking Your Discriminator for Hard Examples","cited_arxiv_id":"2207.07739","evidence_quote":"Argues that adversarial discriminators struggle with hard samples and motivates using adversarial focal loss for such cases."}],"review_version":1}