{"id":"b2e087e9-cc5d-4362-b86f-d841a26b0ecb","arxiv_id":"2412.09376","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An interpretable ML framework combining bagging with one-versus-one classification and multiple XAI methods achieves 87.5% balanced accuracy and 90.8% F1 on MCI/AD diagnosis in the ADNI cohort.","lead":"This paper presents an interpretable machine learning framework that combines brain MRI volumes and genetic markers to classify people as healthy, mildly cognitively impaired, or having Alzheimer's disease. It layers multiple explanation methods, including SHAP and counterfactual examples, to identify which brain regions and genes drive the model's decisions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The necessity/sufficiency unification is method-dependent, with one generator yielding zero necessity for every top feature, so the central robustness claim is not supported without sensitivity and baseline comparisons.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the necessity/sufficiency unification is only meaningful if the counterfactual generators reliably produce valid, diverse counterfactuals. The paper's own results show degeneracy for Permute Attack and strong method-dependence, which the Discussion acknowledges. My stress-test sharpens this into a concrete, testable deficiency: the central claim of 'robust interpretations' rests entirely on Fig. 6, and that figure is not accompanied by the sensitivity analyses needed to separate a real causal property of the features from an artifact of counterfactual generation in a high-dimensional space. The paper is honest about the limitation, and the public code makes the proposed check feasible, so this does not warrant rejection. A conditional acceptance with the robustness claim scoped or strengthened by sensitivity evidence is the appropriate disposition. I agree with the reader that the concern is real; no new concern beyond this is needed to justify the conditional verdict.","tokens_in":16437,"tokens_out":5227,"duration_ms":59233,"concrete_test":"Using the released GitHub code, rerun the SVM Bagging-OVO pipeline and recompute Fig. 6 for nCF = 1, 2, and 8, with 10 random seeds for DiCE and Permute Attack, and include a control set of 10 randomly selected non-top features. If the top-10 combined necessity/sufficiency changes by more than the between-seed confidence interval, or if the control features receive necessity/sufficiency values comparable to the SHAP top-10 features, then the unification result is not robust and the central claim should be restricted to 'consistency of feature rankings across attribution methods' rather than 'robust unified explanations.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the Abstract and Methods Overview, is that the proposed framework provides 'robust interpretable predictions' and that the unification of SHAP with counterfactual explanations assesses the robustness of interpretability techniques. The only quantitative support for that claim is the necessity/sufficiency analysis in Fig. 6. That analysis is method-dependent in a way the paper itself concedes: Permute Attack yields necessity equal to 0 for every top-10 feature individually, while DiCE yields individual necessity values of only 1% to 9%, and the Discussion states that the necessity/sufficiency results are 'highly dependent on the selected counterfactual generation method.' The paper also notes that low necessity is expected as the number of features increases, citing reference 18. If no single feature is necessary, and if the combined necessity of the top 10 features is roughly half that of the remaining features only under DiCE, then the unification does not demonstrate that the SHAP attributions are robust; it demonstrates that in a 208-feature space, individual features are largely redundant. Without a random-feature baseline, a stability check across counterfactual generation settings, or validation on data with known causal structure, the statement that the unification results 'further showcase their significance' is an over-reading of the evidence. The concern is not that the authors are wrong that the metrics are useful; it is that the central robustness claim rests on a single, acknowledged-unstable number.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a comprehensive interpretable machine learning framework for MCI and AD diagnosis using ADNI data, combining bagging with one-versus-one classification to handle multiclass imbalance, evaluating six classifiers under nested cross-validation, and applying SHAP, LIME, PDP, and counterfactual methods (Permute Attack and DiCE). The best classifier (SVM) achieves 87.5% balanced accuracy and 90.8% F1-score. A unification approach based on Mothilal et al. computes necessity and sufficiency of top-10 SHAP features to assess robustness of interpretations. The paper reports that Permute Attack yields zero necessity for individual top features while DiCE yields low values, and it discusses the method-dependence of these metrics. The main claim is that the framework provides robust interpretable predictions of MCI/AD diagnosis.","tokens_in":16667,"tokens_out":2172,"duration_ms":23905,"significance":"If the central claim holds, the paper would make a useful contribution to interpretable ML for dementia diagnosis: it applies a relatively recent unification framework (Mothilal et al.) to a clinically relevant problem, combines volumetric MRI and SNP features, reports results on a public dataset, and makes code available. The strengths include nested cross-validation for hyperparameter tuning and performance evaluation, explicit comparison with prior multiclass ADNI studies, and a multi-method XAI analysis whose qualitative findings (e.g., hippocampus, entorhinal cortex, lateral ventricles, APOE-related rs429358) align with established AD neuropathology. However, the paper's central claim about the robustness of the interpretations rests on the necessity/sufficiency analysis, and that analysis is method-dependent in a way the paper itself concedes; without sensitivity checks or baseline comparisons, the unification does not yet demonstrate robustness of SHAP attributions.","major_comments":[{"comment":"The central claim that the framework assesses the robustness of interpretability methods is not supported by the necessity/sufficiency results as presented. Fig. 6 shows that Permute Attack yields necessity 0 for every individual top-10 feature and for the top-10 combined, while DiCE yields individual necessity values of only 1% to 9%; the Discussion concedes that the calculation of necessity and sufficiency is \"highly dependent on the selected counterfactual generation method\" and that low necessity is expected with a large number of features. These results show that individual features are largely redundant in a 208-feature space, but they do not show that the SHAP attributions are robust. The authors should add a sensitivity analysis across counterfactual generation settings, a random-feature baseline, or validation on data with known causal structure before claiming that the unification \"assesses the interpretability techniques' robustness\" (Abstract) or that the results \"further showcase their significance\" (Abstract).","section":"Results: Unified explanations, Fig. 6, Discussion"},{"comment":"The headline performance (87.5% balanced accuracy, 90.8% F1) is the best of six classifiers selected after nested CV, yet the paper does not correct for multiple comparisons or for selecting the maximum over classifiers. The reported t-test comparisons are pairwise without adjustment for the multiple tests, and the MLP is the only classifier found significantly worse. Since the SVM is selected as the best model and then used for all interpretability analyses, the authors should justify that the selection does not inflate the reported performance, or present performance with appropriate correction, e.g., reporting all classifiers' cross-validated performance with confidence intervals and a multiple-comparison correction.","section":"Table 1 and Classification space"},{"comment":"The necessity and sufficiency metrics in Eqs. (1) and (2) depend critically on the ability of the counterfactual generator to produce valid counterfactuals under the allowed perturbations, but the paper does not report the fraction of instances for which valid counterfactuals were successfully generated for each method and each feature subset. Without this information, a zero necessity value could reflect generation failure rather than true non-necessity, and the sufficiency values could be artificially high or low depending on the generator's success rate. The authors should report generation success rates and, if they are not near 100%, discuss how this affects the interpretation of Figs. 6a-d.","section":"Methods: Unifying feature attribution-based methods and counterfactual explanations, Eqs. (1)-(2)"},{"comment":"The term \"robust interpretable predictions\" is used in the Abstract and the Methods Overview, but the paper does not define what robustness means in this context or how the unification framework measures robustness beyond the necessity/sufficiency metrics. The paper should either provide a precise definition of interpretation robustness and a justification that the necessity/sufficiency metrics capture it, or temper the claim to avoid overstating the evidence. As it stands, the claim that the unification \"assesses the interpretability techniques' robustness\" is an over-reading of the results, given the method-dependence acknowledged in the Discussion.","section":"Methods: Overview of the conceptual framework"}],"minor_comments":[{"comment":"The caption reads \"the Mild Cognitive Impairment (MCI) classes\" in one place; it should read \"MCI class\" to be consistent with the binary subproblem framing.","section":"Results, Fig. 2 caption"},{"comment":"The notation \"x31(CFi)\" appears corrupted and should be a proper indicator function, e.g., 1[CF_i is valid]. Please fix the typesetting and define all symbols.","section":"Methods, Eqs. (1)-(2)"},{"comment":"The table caption states that the optimal metrics are highlighted in bold, but the bold entries do not clearly correspond to a single stated selection rule (mean CV, max CV, or TTS). Please clarify which column is used for model selection and mark it explicitly.","section":"Results, Table 1"},{"comment":"The description of the bagging strategy says \"two different training subsets were created, both containing all CN and AD training samples and each of them a different half of the majority class samples.\" It is unclear whether the two halves are disjoint and how they are sampled; please specify the exact procedure and whether the CN and AD samples are also split or kept intact.","section":"Methods: Classification space"},{"comment":"The text mentions that LIME explanations were obtained for True Positive, True Negative, False Positive, and False Negative instances, but only one LIME example is shown in Fig. 3. Please state where the other examples are presented (or note that they appear in the Supplementary material).","section":"Results: Explanations from individual interpretability methods"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical contribution and the code availability is a plus. However, the central robustness claim is load-bearing and currently rests on necessity/sufficiency results that the authors themselves describe as highly dependent on the counterfactual generation method. I recommend major revision rather than rejection, as the issues are addressable with additional analyses (generation success rates, sensitivity checks, baseline comparisons) and with a more measured statement of what the unification can and cannot certify. I would also suggest the editor ask the authors to clarify the statistical treatment of selecting the best classifier across six candidates, as this affects the credibility of the headline performance figure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate applied paper, not a breakthrough. The genuinely new piece is the first deployment of the Mothilal et al. actual-causality unification (necessity/sufficiency) in the MCI/AD setting, and that is worth taking seriously. The authors also ran a careful six-classifier comparison under nested cross-validation, provided code, and their feature findings (hippocampus, entorhinal cortex, lateral ventricles, ApoE SNP) line up with the pathology literature. Credit where due: they are transparent about the main weak spot—the necessity/sufficiency numbers depend heavily on which counterfactual generator you use, and Permute Attack returns zero necessity for every top-10 feature individually. They did not hide that.\n\nThe soft spots are real, though not fatal to the whole paper. The abstract and methods say the framework 'ensures robustness' and the unification 'assesses the interpretability techniques' robustness,' but the results do not show that. What they show is that in a 208-feature space, single features are not necessary, which is expected, and they cite Mothilal et al. for exactly that. With one generator, necessity is 0 across the board; with DiCE, individual top-feature necessity ranges 1-9%. That is method-dependence, not robustness evidence. To support the robustness claim they would need a random-feature baseline, stability across generator settings, or a synthetic ground-truth check. Their own Discussion concedes this—'highly dependent on the selected counterfactual generation method'—so the overreach is in the framing, not in the data reporting.\n\nAlso, the headline 87.5% balanced accuracy is comparable to existing ADNI multiclass models; not a step change. And among six classifiers, SVM was selected as best without a multiple-comparisons correction, though the t-test shows most differences are not significant.\n\nOverall, this is a solid engineering contribution for researchers working on interpretable medical ML, particularly those wanting a worked example of necessity/sufficiency applied to MRI+SNP data. It deserves a serious referee—the method application and honest limitations make it publishable, but the robustness claim should be reworded or supported with additional analyses. If I were the editor, I would send it out with a request to either soften the 'robustness' language or add the missing baselines.","headline":"First real application of Mothilal's necessity/sufficiency metrics to MCI/AD, honest about its limits, but the robustness claim in the framing is stronger than the evidence supports.","tokens_in":17296,"tokens_out":2283,"would_cite":true,"duration_ms":23316,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that unifying SHAP with counterfactual necessity/sufficiency metrics makes interpretable MCI and Alzheimer's diagnoses robust enough for clinical decision support.","keywords":["interpretable machine learning","Alzheimer's disease diagnosis","mild cognitive impairment","SHAP","counterfactual explanations","necessity and sufficiency","ensemble learning","MRI volumetric features"],"falsifier":"Recompute DiCE necessity and sufficiency for the top-10 SHAP features on the same MCI-versus-AD data and compare them with ten randomly selected features; if the random features achieve comparable or higher scores, the unification does not actually show that SHAP's top features are special, and the robustness claim fails.","tokens_in":16217,"feed_emoji":"🧠","tokens_out":9779,"duration_ms":79881,"temperature":0.7,"pith_summary":"The paper sets out to make machine-learning diagnoses of mild cognitive impairment (MCI) and Alzheimer's disease (AD) trustworthy by pairing high-accuracy classification with explanations that are checked for robustness. Using MRI-derived brain-region volumes and genetic markers from a public cohort of 1,463 subjects, it trains six classifiers under a bagging one-versus-one decomposition; the best model reaches 87.5% balanced accuracy and a 90.8% F1-score. The central move is to unify SHAP attributions with counterfactual explanations through necessity and sufficiency metrics, so that a feature counts as important only if changing it (or fixing it) demonstrably changes (or preserves) the model's output. The authors argue that the resulting explanations—highlighting entorhinal areas, lateral ventricles, temporal gyri, and the ApoE-linked SNP rs429358—are consistent with known Alzheimer's neuropathology. If right, this gives clinicians a decision-support tool that is both accurate and transparent about why a patient was classified as MCI or AD.","feed_headline":"ML diagnosis hits 87.5% balanced accuracy for MCI, Alzheimer's","feed_subtitle":"It couples SHAP with counterfactual checks to tell clinicians which features truly drive the diagnosis.","key_machinery":"The central machinery is the unification of feature attribution with counterfactual explanations through the actual-causality notions of necessity and sufficiency. Necessity is computed by generating counterfactuals that change only the feature under test and measuring the fraction that produce a valid class flip; sufficiency fixes that feature and permutes all others, measuring one minus the fraction of valid counterfactuals produced. These two scalar metrics turn a ranked feature list from SHAP into statements about causal indispensability and causal adequacy, and the paper applies them to the top-10 SHAP features of the MCI-versus-AD classifier.","core_discovery":"The paper's central claim is that a single framework can deliver diagnostically competitive and robustly interpretable predictions for MCI and AD by decomposing the three-class problem into one-versus-one binary subproblems, resampling the majority MCI class via bagging, and combining attribution-based methods (SHAP, Gini, LIME, PDP) with counterfactual-based methods (Permute Attack, DiCE) inside a unification scheme grounded in actual causality. The unification computes necessity and sufficiency scores for the top-10 SHAP features, quantifying whether a feature is indispensable for the outcome and whether fixing it alone can guarantee the outcome. The authors show that individual top features have low necessity in a 208-feature space, but the combined top-10 features achieve a necessity of 29.8% with DiCE, nearly half the 63.7% necessity of the remaining 95% of features, with sufficiency scores of 33–38%; they interpret this as evidence that the SHAP ranking captures features that genuinely drive the model.","pith_inferences":["Inference: the necessity/sufficiency framework could be turned into a model- or feature-selection criterion—choose the classifier and feature set that maximize combined-top-feature necessity—rather than applying the unification only post hoc to a fixed model.","Inference: because one counterfactual generator (Permute Attack) produced zero necessity for every single feature, the choice of generator is not neutral; a fair robustness assessment should require multiple generators or validate counterfactual quality before scoring necessity.","Inference: the same unification could be applied to longitudinal data to test whether features necessary for conversion from MCI to AD differ from features necessary for cross-sectional diagnosis, a question the present cross-sectional design cannot answer."],"forward_implications":["Explanation robustness becomes a measurable quantity: a model whose top features score low on necessity and sufficiency can be flagged as untrustworthy before clinical adoption.","Combining MRI volumes with SNPs in one model shows that both anatomical (hippocampus, entorhinal cortex, ventricles) and genetic (rs429358/APOE) factors drive AD risk, supporting multimodal diagnostic inputs.","The bagging with one-versus-one decomposition offers a template for other imbalanced multiclass medical datasets where the majority class dominates but every class matters.","The finding that combined top-10 features have much higher necessity than any single feature suggests explanations should be read as feature sets rather than individual biomarkers."],"supporting_citations":[{"why":"Supplies the actual-causality necessity and sufficiency metrics that the unification framework is built on.","marker":"[18]"},{"why":"Provides the Permute Attack counterfactual generator used for one set of necessity and sufficiency calculations.","marker":"[24]"},{"why":"Provides the DiCE counterfactual generator used for the second set of necessity and sufficiency calculations.","marker":"[25]"},{"why":"Defines SHAP, the attribution method whose top-10 features are subjected to necessity and sufficiency testing.","marker":"[20]"},{"why":"Documents hippocampal atrophy and ventricular enlargement in MCI and AD, grounding the interpretation that the highlighted ROIs match known neuropathology.","marker":"[30]"},{"why":"Links APOE genetic variants to Alzheimer's susceptibility, supporting the interpretation of the rs429358 SNP finding.","marker":"[23]"},{"why":"Provides the neuropathological background on Alzheimer's diagnosis and progression used to validate the highlighted anatomical features.","marker":"[29]"}],"fun_headline_variants":["SHAP meets counterfactuals for trustworthy Alzheimer's diagnosis","Explainable AI framework hits 87.5% accuracy for MCI, Alzheimer's","Unifying SHAP and counterfactuals diagnoses MCI with 87.5% accuracy","Robust interpretations: ML framework for Alzheimer's hits 87.5%","Necessity and sufficiency: new AI explains Alzheimer's diagnosis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the counterfactual generators can produce valid, diverse alternative outcomes when individual features are changed; if they cannot, the necessity and sufficiency scores do not measure what they claim, and the central claim of robust explanations collapses.","fun_headline_variants_meta":{"raw":{"variants":["SHAP meets counterfactuals for trustworthy Alzheimer's diagnosis","Explainable AI framework hits 87.5% accuracy for MCI, Alzheimer's","Unifying SHAP and counterfactuals diagnoses MCI with 87.5% accuracy","Robust interpretations: ML framework for Alzheimer's hits 87.5%","Necessity and sufficiency: new AI explains Alzheimer's diagnosis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2677,"prompt_tokens":932,"completion_tokens":1745,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1645}},"tokens_in":548,"tokens_out":1745,"duration_ms":10889,"temperature":1.0,"reasoning_tokens":1645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:05:10.784834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute DiCE necessity and sufficiency for the top-10 SHAP features on the same MCI-versus-AD data and compare them with ten randomly selected features; if the random features achieve comparable or higher scores, the unification does not actually show that SHAP's top features are special, and the robustness claim fails.","supporting_citations":[{"cited_title":"& Fathi, A","cited_arxiv_id":null,"evidence_quote":"Provides the Permute Attack counterfactual generator used for one set of necessity and sufficiency calculations."},{"cited_title":"K., Sharma, A","cited_arxiv_id":null,"evidence_quote":"Provides the DiCE counterfactual generator used for the second set of necessity and sufficiency calculations."},{"cited_title":"SHAP (SHapley Additive exPlanations)","cited_arxiv_id":null,"evidence_quote":"Defines SHAP, the attribution method whose top-10 features are subjected to necessity and sufficiency testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents hippocampal atrophy and ventricular enlargement in MCI and AD, grounding the interpretation that the highlighted ROIs match known neuropathology."},{"cited_title":"& Juárez-Cedillo, T","cited_arxiv_id":null,"evidence_quote":"Links APOE genetic variants to Alzheimer's susceptibility, supporting the interpretation of the rs429358 SNP finding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the neuropathological background on Alzheimer's diagnosis and progression used to validate the highlighted anatomical features."}],"review_version":1}