Pith. sign in

REVIEW 2 major objections 5 minor 3 references

Uncertainty-Aware Genomic Classification of Alzheimer's Disease: A Transformer-Based Ensemble Approach with Monte Carlo Dropout

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that filtering out high-variance genomic predictions improves Alzheimer's classification accuracy on the remaining, more certain subset.

desk verdict Solid but overstated: without a probability-margin baseline, the paper's central claim—that MC-dropout variance, not generic confidence, drives the gain—is untested. read the letter →

arxiv 2506.00662 v1 pith:PBSMQHCW submitted 2025-05-31 q-bio.GN cs.LG

Taeho Jo , Eun Hye Lee , Alzheimer's Disease Sequencing Project This is my paper
classification q-bio.GNcs.LG
keywords Alzheimer'sdiseasegenomicsdeeplearningtransformerMonteCarloDropoutuncertaintydementiaclassificationwhole-genomesequencing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a genomic classifier for Alzheimer's disease becomes more clinically useful when it reports not just a prediction but an uncertainty estimate. Using Monte Carlo Dropout, the authors measure how much each sample's predicted probability varies across many stochastic passes of a transformer model, then merge that variance with a random forest's variance. They find that excluding high-variance samples raises accuracy on the retained subset from 0.6263 to 0.7287 and F1 from 0.5843 to 0.8205, at the cost of labeling about 75% of the test set as uncertain. The claim is that this uncertainty-aware filtering identifies genuinely ambiguous cases that deserve extra clinical scrutiny.

What carries the argument

The central object is a 2x2 matrix $M = \begin{pmatrix} P_{\text{trans}} & V_{\text{trans}} \\ P_{\text{rf}} & V_{\text{rf}} \end{pmatrix}$ and a weight vector $w = (\alpha, 1-\alpha)$, whose product gives the ensemble probability and ensemble variance. The weight $\alpha = 0.6$ and a variance threshold of 0.0741 are tuned on out-of-fold training predictions. The transformer partitions each genome into windows of 100 SNPs, uses two encoder layers with four attention heads, and keeps dropout active at inference to produce $V_{\text{trans}}$; a random forest on flattened genotypes produces $V_{\text{rf}}$. A variance-penalty term $\lambda_{\text{unc}} \cdot \operatorname{mean}(\operatorname{Var}(\text{logits}))$ is added to the cross-entropy loss to encourage stable latent representations, and the resulting ensemble variance is what assigns each sample to the certain or uncertain group.

What would settle it

A direct comparison on the same test set between the proposed variance-based filtering and a simple probability-margin filter (e.g., excluding samples with predicted probability between 0.3 and 0.6) would settle whether the improvement is specific to MC-dropout uncertainty; if the margin filter achieves the same or better certain-group accuracy and F1, the paper's central claim about uncertainty estimation is not supported.

Watch

Extended reading notes

Core claim

The central discovery is that a per-sample predictive variance, derived from Monte Carlo Dropout and merged with random-forest variance, separates test samples into a low-variance group that the ensemble classifies substantially better and a high-variance group it classifies poorly. The certain subset reaches 0.7287 accuracy and 0.8205 F1, versus 0.6263 and 0.5843 for the uncertain subset, while the all-sample accuracy is 0.6514. The paper interprets this as evidence that uncertainty estimation has practical value: ambiguous cases can be flagged for additional evaluation rather than force-fitted into a binary label.

Load-bearing premise

The variance threshold and ensemble weight tuned on out-of-fold training predictions generalize to the held-out test set, and the Monte Carlo dropout variance captures genuine ambiguity rather than merely the distance of the predicted probability to the 0.5 decision boundary.

Editorial extensions

If this is right

  • Adopting this pipeline in a clinical workflow would let high-variance predictions be routed to additional testing, such as neuroimaging or biomarker assays, instead of being reported as a binary diagnosis.
  • The certain group's F1 of 0.8205 suggests that for a subset of patients the genomic model is reliable enough to serve as a screening tool, while the uncertain group's low F1 of 0.5843 marks it as needing caution.
  • The learned ensemble weight $\alpha = 0.6$ indicates that the transformer's probability contributes more than the random forest's, and that ignoring uncertainty entirely (setting $\alpha = 1$) lowers AUC from 0.6508 to 0.6283 on the validation folds.
  • The variance threshold at 0.0741 quantifies a tunable trade-off between completeness and accuracy, letting clinicians adjust how many samples to defer based on the relative cost of misdiagnosis versus additional testing.
  • Variance-based filtering can be extended to other genomic classification tasks where diagnostic ambiguity is costly, not only Alzheimer's disease.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains may be a generic confidence effect: Figure 3 shows uncertain samples cluster near predicted probabilities of 0.3 to 0.5, so a simple margin-based filter that drops samples near the decision boundary could reproduce most of the improvement without Monte Carlo Dropout. The paper does not test this baseline, so the distinctive contribution of its uncertainty measure remains unprove
  • Because the variance threshold and ensemble weight are selected on the same out-of-fold predictions that later define the certain and uncertain groups, the magnitude of the certain-group improvement is likely optimistically biased; an independent validation cohort or nested cross-validation would give a more honest estimate.
  • The 75% uncertain rate means the practical benefit is a selective reporting strategy rather than a better classifier overall; whether this is worthwhile depends on whether the cost of extra testing for most patients outweighs the cost of misdiagnosis.
  • A testable extension would apply the same variance-based triage to transcriptomic or proteomic Alzheimer's data to see whether the same individuals are flagged as ambiguous across modalities, which would suggest a shared biological ambiguity rather than a model artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes TrUE-Net, a transformer plus random forest ensemble with Monte Carlo Dropout for Alzheimer's disease classification from whole-genome sequencing, and evaluates it on 1,050 ADNI participants split into 525 training and 525 test samples. The overall test accuracy is 0.6514 and AUC is 0.6636. Using a variance threshold tuned on out-of-fold predictions, the test set is split into an uncertain group (396 samples, 75.4%) and a certain group (129 samples, 24.6%); the certain group achieves accuracy 0.7287 and F1 0.8205, compared with 0.6263 and 0.5843 for the uncertain group. The authors conclude that filtering high-variance samples improves classification reliability and demonstrates the practical benefit of an uncertainty-aware approach.

Significance. If it held, the central claim would be practically useful: a genomic classifier that can flag samples needing additional clinical review would be valuable for Alzheimer's disease risk assessment. The paper contributes a specific ensemble architecture, publicly available code, and a clear empirical evaluation protocol with held-out data. However, the significance is currently limited because the paper does not establish that Monte Carlo Dropout variance, rather than a generic confidence signal, is responsible for the reported improvement; the provided figures suggest the variance threshold is largely a proxy for distance from the decision boundary. The manuscript also lacks statistical uncertainty quantification for its headline performance differences.

major comments (2)
  1. [Results, Figure 3; Conclusion] The central claim that MC-dropout variance drives the reported gain is not established because no baseline using a simple probability-margin filter is provided. Figure 3 shows that samples assigned to the uncertain group have predicted probabilities clustered in 0.3–0.5, while certain samples are below 0.3 or above 0.6. For binary softmax outputs, dropout-induced variance is, to first order, proportional to (p(1−p))² Var(logit), so thresholding on variance is, in these data, almost equivalent to thresholding on distance from the decision boundary. A margin filter such as m=|p−0.5| or max-class probability, evaluated under the same out-of-fold threshold-selection protocol, could likely reproduce the certain-group accuracy (0.7287) and F1 (0.8205) without any MC-dropout component. Without this comparison, the Conclusion's statement that 'filtering out samples with elevated variance ... demonstrates the practical benefits of an uncertainty-aware approach' conflates a generic confidence effect with the specific MC-dropout mechanism named in the title.
  2. [Methods: Predictive Variance for Uncertainty Estimation; Results] The variance threshold (0.0741) and ensemble weight (α=0.6) are fitted on out-of-fold training predictions using criteria of the same family as the reported test metrics: the Methods state that α is selected to maximize OOF accuracy and the threshold maximizes the sum of subset AUCs, while the Results report α=0.6 as the value with highest AUC. Although this is not label leakage, the magnitude of the certain-versus-uncertain difference is likely optimistic because the threshold was chosen on training data with the same kind of objective that is later reported on the test set. No confidence intervals, bootstrap estimates, or significance tests are given for the differences in accuracy (0.7287 vs 0.6263) and F1 (0.8205 vs 0.5843) on sample sizes of 129 and 396. The authors should report confidence intervals for these differences and, ideally, validate the entire threshold-selection protocol in nested cross-validation or at least with a permutation-based null.
minor comments (5)
  1. [Methods: Predictive Variance for Uncertainty Estimation] The exact value of the uncertainty penalty λ_unc used in the final model is not reported; the text says 'λ_unc (for instance, 0.05)' but no sensitivity analysis or final value is given, even though this penalty directly shapes the variance estimates that drive the main result.
  2. [Methods: An Ensemble Approach] The matrix notation for the ensemble is unnecessarily confusing because the same weight α is applied to both the probability and the variance components; the authors should clarify whether a separate weighting for variance was considered and justify applying the same α to both quantities.
  3. [Methods: Predictive Variance for Uncertainty Estimation] The threshold-selection criterion is described as optimizing 'a chosen metric (e.g., the sum of AUCs in uncertain and more certain subsets)', but the exact metric and the formula used to combine the two subset AUCs are never specified; this should be stated precisely since the final threshold is a key result.
  4. [Results] The paper reports the variance threshold as 0.0741 without specifying the scale or normalization of the variance estimates; the authors should state what units this threshold is in and how the variance was computed across the Monte Carlo dropout passes and random forest trees.
  5. [References] There are duplicate and inconsistently formatted references for Gatz et al. (2006), and several citations in the text use nonstandard formats (e.g., 'Genomes Project et al., 2015'); the reference list should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ensemble weight and variance threshold are tuned on out-of-fold training predictions and then applied to a held-out test set, so the improved certain-group metrics are a genuine conditional evaluation rather than a refit of the test labels.

full rationale

The derivation chain is self-contained with respect to the central claim. The ensemble weight (α=0.6) and variance threshold (0.0741) were selected using five-fold cross-validation on the training set only, and the reported accuracy, AUC, and F1 for the uncertain and certain groups were computed on the held-out 50% test set (525 samples) after retraining on the full training set. No fitted parameter is renamed as a prediction: each test sample is assigned to the uncertain or certain group by comparing its ensemble variance to the pre-fixed threshold, and the group-wise metrics are then measured against the actual test labels. The possible concern that the variance filter is largely a proxy for distance to the decision boundary, as suggested by Figure 3, is a comparison/baseline completeness issue rather than an instance where the claimed result is equivalent to its inputs by construction. The self-citations (Jo et al. 2022, 2023, 2025) support prior methodology and related findings, but they are not load-bearing for the uncertainty-filtering claim, which is empirically evaluated within this paper on held-out data.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on two fitted hyperparameters (alpha and the variance threshold) and one hand-set penalty weight. No new physical or biological entities are introduced. The most fragile assumption is that the threshold tuned on training folds transfers to the test set and that the dropout variance provides information beyond a simple probability margin.

free parameters (3)
  • ensemble weight alpha = 0.6
    Selected on out-of-fold predictions by maximizing accuracy; ranged over 0 to 1 in 0.2 increments.
  • variance threshold tau = 0.0741
    Selected on out-of-fold predictions by maximizing the sum of AUCs in the uncertain and certain subsets; 10 candidate thresholds tested.
  • uncertainty penalty lambda_unc = 0.05
    Added to the cross-entropy loss to penalize logit variance; chosen by hand and not tuned.
assumptions (4)
  • domain assumption ADNI clinical diagnoses are accurate ground truth for AD status.
    The paper relies on ADNI diagnostic criteria to label cases and controls; any misdiagnosis biases the classification and the uncertainty filter.
  • domain assumption Variant calling and QC procedures produce reliable genotype calls.
    The pipeline removes low-quality variants and uses k-NN imputation for missing genotypes; this assumes residual genotypes are accurate.
  • ad hoc to paper The variance threshold selected on out-of-fold predictions generalizes to the test set.
    The reported improvement depends on the threshold chosen from training-fold predictions; if this threshold overfits the folds, the test-set gap would not replicate.
  • standard math Standard neural network training assumptions: softmax, cross-entropy, AdamW optimizer.
    Standard background, not questioned.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Uncertainty-Aware Genomic Classification of Alzheimer's Disease: A Transformer-Based Ensemble Approach with Monte Carlo Dropout." pith.science (2026). https://pith.science/paper/PBSMQHCW

@misc{pith2026250600662,
  author       = {Pith},
  title        = {Pith review of: Uncertainty-Aware Genomic Classification of Alzheimer's Disease: A Transformer-Based Ensemble Approach with Monte Carlo Dropout},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PBSMQHCW}},
  note         = {Machine review of arXiv:2506.00662}
}
read the original abstract

INTRODUCTION: Alzheimer's disease (AD) is genetically complex, complicating robust classification from genomic data. METHODS: We developed a transformer-based ensemble model (TrUE-Net) using Monte Carlo Dropout for uncertainty estimation in AD classification from whole-genome sequencing (WGS). We combined a transformer that preserves single-nucleotide polymorphism (SNP) sequence structure with a concurrent random forest using flattened genotypes. An uncertainty threshold separated samples into an uncertain (high-variance) group and a more certain (low-variance) group. RESULTS: We analyzed 1050 individuals, holding out half for testing. Overall accuracy and area under the receiver operating characteristic (ROC) curve (AUC) were 0.6514 and 0.6636, respectively. Excluding the uncertain group improved accuracy from 0.6263 to 0.7287 (10.24% increase) and F1 from 0.5843 to 0.8205 (23.62% increase). DISCUSSION: Monte Carlo Dropout-driven uncertainty helps identify ambiguous cases that may require further clinical evaluation, thus improving reliability in AD genomic classification.

Figures

Figures reproduced from arXiv: 2506.00662 by the authors.

Figure 4
Figure 4. (A) compares the distribution of final predicted probabilities for the cognitively normal (CN) and AD groups, showing that CN cases tend to peak near 0.4 while AD cases shift toward higher probabilities around 0.6. (B) shows the distribution of model uncertainty, where CN reaches its highest frequency near 0.10 and AD clusters slightly below that range, although substantial overlap remains between the two groups [P… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

  1. [1]

    J., Renton, A

    Andrews, S. J., Renton, A. E., Fulton -Howard, B., Podlesny -Drabiniok, A., Marcora, E., & Goate, A. M. (2023). The complex genetic architecture of Alzheimer's disease: novel insights and future directions. EBioMedicine, 90, 104511. https://doi.org/10.1016/j.ebiom.2023.104511 Bekris, L. M., Yu, C. E., Bird, T. D., & Tsuang, D. W. (2010). Genetics of Alzhe...

    arXiv 2023

  2. [22]

    J., Huentelman, M., & Hardy, J

    https://doi.org/10.1186/s13195-022-00962-4 Escott-Price, V., Baker, E., Shoai, M., Leonenko, G., Myers, A. J., Huentelman, M., & Hardy, J. (2019). Genetic analysis suggests high misassignment rates in clinical Alzheimer's cases and controls. Neurobiology of Aging , 77, 178 -182. https://doi.org/https://doi.org/10.1016/j.neurobiolaging.2018.12.002 Ferrante...

    work page doi:10.1186/s13195-022-00962-4 2019

  3. [232]

    https://doi.org/10.1038/s41398-023- 02531-1 Diagnosis Group Sample Size Mean Age (SD) % Males / Females APOE ε2/ε2 APOE ε2/ε3 APOE ε2/ε4 APOE ε3/ε3 APOE ε3/ε4 APOE ε4/ε4 APOE ε4 Carriers (%) Cognitive normal (CN) 443 72.63 (6.31) 53.05 / 46.95 2 57 7 257 110 10 27.09 Alzheimer’s disease (AD) 607 73.97 (7.25) 42.01 / 57.99 1 19 14 194 278 101 62.44 Table

    work page doi:10.1038/s41398-023-

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.