{"id":"3b6d2058-d546-4e82-bcf2-473752404d4f","arxiv_id":"2501.18071","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An ensemble of Random Forest, XGBoost, and LightGBM with SMOTE preprocessing reaches 92.5% reported accuracy (92.8% in the table) on a BRFSS diabetes survey dataset, with SHAP and LIME used to identify BMI, age, and general health as key predictors.","lead":"This paper trains seven machine learning models, including an ensemble, on a 253,680-person U.S. health survey dataset to predict diabetes and then uses explainability tools to show which factors drive the predictions. The result is a typical benchmark application: the reported accuracy is strong, but the paper's numbers are internally inconsistent and no code is provided, so the headline should be read as plausible rather than confirmed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported accuracy depends on SMOTE being applied only to the training split; the paper's ambiguous pipeline description and a mismatch with its own Table II leave this unverified.","rationale":"The reader's weakest_assumption correctly identifies SMOTE leakage as the central threat to the accuracy claim. The paper's own wording in Section III-C ('balanced training set created from the original training set') hints that SMOTE was applied only to the training data, but the section ordering and lack of a pipeline diagram leave room for doubt. If synthetic examples entered the test set, the reported accuracy and ROC-AUC would be optimistically biased, directly falsifying the central claim as a measure of real-world performance. This is more load-bearing than the label-quality concern, because it concerns the internal validity of the experiment rather than dataset semantics. The abstract-to-table mismatch (92.50%/0.975 vs. 92.80%/0.97) further shows that the numeric claim is fragile and needs a clear correction. Given that the reader already conditioned acceptance on resolving this ambiguity, no change to the verdict is needed; the stress-test affirms the reader's focus and adds the Table II inconsistency as supporting evidence. The proposed test would settle the concern by re-running with both pipeline orders and checking for synthetic test samples, which is a concrete, feasible check even without the original code.","tokens_in":11980,"tokens_out":3997,"duration_ms":40853,"concrete_test":"Obtain the authors' implementation or re-implement the pipeline from Section III in two orders: (a) SMOTE before the train/validation/test split and (b) SMOTE after the split, using the same models and hyperparameters (or default hyperparameters where unspecified). Compare the test accuracy and ROC-AUC on the untouched original test set. If order (a) reproduces the reported 92.50%/0.975 while order (b) yields materially lower numbers, the headline claim is an artifact of leakage. Additionally, inspect the test set for synthetic minority instances (e.g., exact duplicates of SMOTE-generated points) to directly confirm contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a 92.50% test accuracy and 0.975 ROC-AUC is valid only if SMOTE was applied exclusively to the training split. Section III-A lists SMOTE as part of preprocessing, Section III-B describes the 70/15/15 split afterward, and Section III-C says models were 'trained on the balanced training set created from the original training set.' This leaves the ordering ambiguous. If SMOTE was applied before the split, synthetic minority instances would contaminate validation and test sets, inflating all metrics and invalidating the reported numbers as a fair estimate of generalization. No leakage check or explicit pipeline step showing SMOTE after splitting is provided. Compounding this, Table II lists the ensemble test accuracy as 92.80% and test ROC-AUC as 0.97, while the abstract and Section IV-A quote 92.50% and 0.975; these numbers do not match, creating an additional inconsistency that undermines confidence in the reported result. The feature importance claim ('BMI, Age, General Health, Income, and Physical Activity') is also not consistently supported across the SHAP, EBM, and permutation importance figures, which mention slightly different top predictors.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a machine-learning pipeline for binary diabetes prediction on the public Diabetes Binary Health Indicators dataset (derived from BRFSS). Preprocessing consists of median imputation, StandardScaler, and SMOTE; several base models (Random Forest, XGBoost, LightGBM, SVM, Decision Tree, Logistic Regression, Naive Bayes) are trained and compared with a soft-voting ensemble of RF, XGBoost, and LightGBM. The central claim is that the ensemble achieves 92.50% test accuracy and 0.975 ROC-AUC. The paper also applies SHAP, EBM, permutation importance, LIME, PDPs, anchors, and counterfactual explanations, and reports explainability metrics (fidelity, faithfulness, sparsity, stability, consistency) to address four research questions on accuracy, explanation usefulness, combination of methods, and evaluation of explanations.","tokens_in":12188,"tokens_out":2654,"duration_ms":28496,"significance":"If the reported results are correct, the paper provides a useful large-scale empirical comparison of ML and XAI methods on a widely used public dataset, and it explicitly attempts to assess explanation quality rather than only reporting feature rankings. The use of a public dataset, multiple XAI techniques, and a comparative performance table are strengths. However, the contribution is incremental: the methodology is standard, no code or data artifacts are released, and the headline accuracy claim is not yet fully supported because of a mismatch between the abstract and Table II and an ambiguity about whether SMOTE was applied before or after the train/validation/test split. The explainability findings are plausible post-hoc summaries but should not be read as independent external validation of the predictors.","major_comments":[{"comment":"The abstract and Section IV-A state that the ensemble achieved 92.50% test accuracy and 0.975 ROC-AUC, but Table II reports 92.80% test accuracy and 0.97 test ROC-AUC for the same ensemble. This is a direct numerical inconsistency in the central claim. The authors must correct the numbers, state which value is the final test result, and explain any discrepancy (for example, whether validation and test metrics were conflated or whether the abstract predates a re-run).","section":"Abstract / Section IV-A / Table II"},{"comment":"The SMOTE pipeline ordering is ambiguous and load-bearing. Section III-A lists SMOTE as part of preprocessing, Section III-B describes the 70/15/15 split afterward, and Section III-C says models were 'trained on the balanced training set created from the original training set.' If SMOTE was applied before the split, synthetic minority instances would appear in the validation and test sets, inflating all reported metrics and invalidating them as estimates of generalization. The authors must specify explicitly that SMOTE was fitted only on the training split, that validation and test sets contain only original instances, and preferably report a leakage check or the exact pipeline order.","section":"Section III-A, III-B, III-C"},{"comment":"The hyperparameter tuning procedure is described only as '3-fold cross-validation and Randomized SearchCV' with no search spaces, no fold construction details, no selected hyperparameters, and no random seeds. Since the accuracy claim depends on the tuned ensemble, this missing information prevents reproduction and makes it impossible to assess overfitting risk. The authors should provide the hyperparameter grids, final hyperparameter values, and the exact evaluation protocol, or release the code.","section":"Section III-C"},{"comment":"The feature-importance claim is not consistently supported by the reported explanations. The abstract and conclusion list BMI, Age, General Health, Income, and Physical Activity as the most influential predictors, but Section IV-B reports SHAP drivers as BMI, GenHlth, Age, and income; the EBM figure highlights BMI and GenHlth; and Fig. 4 (permutation importance) is described as highlighting BMI, general health, and physical health (PhysHlth), with no mention of Income. The authors should reconcile these lists and state which features are consistently top-ranked across methods, or soften the abstract claim accordingly.","section":"Section IV-B and Abstract"}],"minor_comments":[{"comment":"The counterfactual explanation table lists coefficient-like values (e.g., BMI 0.24, PhysActivity 0.57) but does not define the sign convention, the units, or how the 'cost' column should be interpreted; the cost values appear blank or truncated. Please add a legend or a worked example.","section":"Table III"},{"comment":"The explainability metrics are presented without definitions or formulas; for example, Sparsity 10.0 and Stability 5.35e-05 need units and direction of goodness so that a reader can interpret whether larger or smaller values are better.","section":"Section IV-B.6 / Table IV"},{"comment":"The sentence 'However, the ligand model cannot be applied technically in the clinical setting due to its large computational requirements' appears to contain a typo (likely 'LightGBM model' or 'the model') and should be corrected.","section":"Section V, Limitations and Future Directions"},{"comment":"The conclusion states that 'Interoperability with electronic health records' is essential; this appears to be a word-choice issue, since the intended concept is likely 'interpretability' or 'integration' rather than interoperability.","section":"Section VI, Conclusion"},{"comment":"The related-work table is dense and some entries lack dataset details or clear XAI-metric comparisons; adding a column for the evaluation measures used by prior studies would strengthen the positioning of the paper's explainability-metric analysis.","section":"Section II, Literature Review"}],"recommendation":"major_revision","confidential_remarks":"The paper is plausible and likely salvageable, but the editorial decision should hinge on the authors clarifying the SMOTE/split ordering and reconciling the abstract/Table II discrepancy. If SMOTE was applied before splitting, the central accuracy claim would be invalid and the paper would require a full re-analysis; if it was applied only to the training split, the authors need to state that explicitly and add the missing experimental details. I recommend major revision rather than rejection because the issues are verifiable and fixable within the manuscript's scope, provided the authors can confirm the pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked what I think of arXiv:2501.18071. Short version: it is a competent, incremental application of standard ML and XAI tools to a public diabetes dataset. The genuine strength is breadth—seven models plus an ensemble, and five explanation techniques (SHAP, LIME, EBM, anchors, counterfactuals) with explainability metrics. Most prior work in this space uses one or two XAI methods; this paper gives a more complete picture, and that has practical value for someone wanting a survey-in-one-paper of what works on BRFSS-style data.\n\nThe central accuracy claim, however, is not clean. The abstract says the ensemble hits 92.50% test accuracy and 0.975 ROC-AUC; Table II says 92.80% and 0.97. That is a direct internal contradiction, not a rounding difference, and it needs to be fixed. The related SMOTE concern is real but I read it as a reporting flaw rather than evidence of cheating: Section III-B describes splitting first, and Section III-C says models were trained on the balanced training set created from the original training set, which implies SMOTE after the split. But the preprocessing section lists SMOTE earlier, no diagram or code is provided, and no leakage check is reported. That ambiguity is enough to make the headline numbers unverifiable as submitted, and it is the first thing a referee should demand be clarified.\n\nThe feature-importance claim in the abstract—BMI, Age, General Health, Income, Physical Activity—is also oversimplified. The SHAP, EBM, and permutation importance figures agree on BMI and general health, but they do not all rank the same five predictors in the same order. That is normal for different explanation methods, but the abstract states the list as if it were a single consensus result. The discussion overreaches in other ways too: claiming the model performs \"similar to human practitioners\" or \"establishes a new standard\" goes beyond the evidence.\n\nWhat holds up? The methodology is standard, the dataset is public and large, and the reported accuracies are plausible for this dataset—others have gotten close to 90% with similar approaches. So the core result is believable once the inconsistencies are resolved. I would not call this a novel scientific contribution, but it is a useful comparative benchmark for the applied ML-for-health audience.\n\nShould it be peer reviewed? Yes, with major revisions. The paper deserves referee time because the XAI comparison is broader than typical and the methodological ambiguity is fixable. I would not cite it in its current form, but after a careful revision addressing the accuracy mismatch, the SMOTE pipeline, and the overclaiming, it could become a reasonable reference for applied XAI work.","headline":"A broad but incremental XAI comparison on a public diabetes dataset, undercut by an internal accuracy mismatch and ambiguous SMOTE ordering, but worth a careful referee.","tokens_in":12738,"tokens_out":2542,"would_cite":false,"duration_ms":25686,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that a soft-voting ensemble of Random Forest, XGBoost, and LightGBM predicts diabetes at 92.50% test accuracy with ROC-AUC 0.975, and that its explainability tools consistently rank BMI, age, general health, income, and…","keywords":["diabetes prediction","ensemble learning","soft voting","explainable AI","SHAP","LIME","SMOTE","BRFSS"],"falsifier":"Re-run the exact pipeline with a test split that is fixed before any SMOTE synthesis and contains only original BRFSS rows; if test accuracy and ROC-AUC fall materially below 92.50% and 0.975, the central claim is not supported. The re-run should also reconcile the abstract's 92.50% test accuracy with Table II's 92.80% test accuracy.","tokens_in":11777,"feed_emoji":"🩺","tokens_out":8557,"duration_ms":83566,"temperature":0.7,"pith_summary":"This paper sets out to show that diabetes screening does not have to choose between accuracy and interpretability. Using the 253,680-row Diabetes Binary Health Indicators dataset, it builds seven machine-learning models plus a soft-voting ensemble of Random Forest, XGBoost, and LightGBM and reports a test accuracy of 92.50% with ROC-AUC 0.975. It then applies a suite of explainability tools and claims that they consistently identify BMI, age, general health, income, and physical activity as the main drivers of diabetes risk. If these results hold, clinicians could receive a prediction tool that tells them not only who is at risk but also why, and can even suggest what changes would reduce that risk.","feed_headline":"Ensemble plus explainable AI hits 92.5 percent accuracy on diabetes","feed_subtitle":"The soft-voting model also names BMI, age, income, and physical activity as top risk factors, so clinicians see the why behind each…","key_machinery":"The load-bearing object is the soft-voting ensemble of Random Forest, XGBoost, and LightGBM, in which each model's predicted probabilities are averaged so that confident models contribute more than they would in a hard-vote majority. It is supported by SMOTE, which synthesizes minority-class rows to counter the dataset's 13.07% diabetes prevalence, and by StandardScaler normalization. The interpretability machinery is a suite rather than a single tool: SHAP supplies global and local additive attributions, LIME supplies instance-level explanations, Explainable Boosting Machines give an interpretable-by-design model, partial dependence plots expose pairwise feature interactions, anchors produce rule thresholds, and counterfactual explanations quantify required feature changes. The paper also defines five explanation-quality metrics, fidelity, faithfulness, sparsity, stability, and consistency, to measure whether the explanations are trustworthy.","core_discovery":"In the paper's own terms, the central result is that combining three tree-based learners, Random Forest, XGBoost, and LightGBM, into a soft-voting ensemble, after SMOTE balancing and standard scaling, yields a diabetes classifier with 92.50% test accuracy and an ROC-AUC of 0.975 on the Diabetes Binary Health Indicators dataset. The accompanying XAI analysis is a second result: SHAP, EBM, and permutation importance consistently rank BMI, general health, age, income, and physical activity as the most influential predictors, while LIME, SHAP force and waterfall plots, anchors, and counterfactual explanations convert those rankings into patient-level statements, such as BMI above 30 combined with low physical activity marking high risk. A third result is quantitative: explanation quality is measurable, with reported fidelity 0.744, faithfulness 0.709, sparsity 10, stability 5.35e-05, and consistency 0.73. The paper frames these together as evidence that a transparent, clinically usable diabetes prediction framework is achievable.","pith_inferences":["The paper's protocol description does not make explicit that SMOTE was applied only to the training split, so the safest reading is that 92.5% is an upper-bound estimate; an independent re-run with a strictly untouched test set would settle how much of the accuracy is real.","Because the outcome label is a self-reported diabetes diagnosis from BRFSS, the model may be predicting diagnosis-and-reporting behavior rather than biological diabetes; external validation with clinically confirmed cases would test this.","Income surfacing as a top predictor suggests that social determinants carry predictive signal in this dataset, but the paper does not disentangle income from healthcare access, a natural follow-up question.","The agreement across SHAP, EBM, and permutation importance could itself be used as a low-cost model-agnostic consistency check before applying black-box classifiers in other chronic-disease settings."],"forward_implications":["A soft-voting ensemble of Random Forest, XGBoost, and LightGBM can outperform each of its members, so ensemble construction is a plausible route to better diabetes screening.","If the reported explanations are representative, screening tools can focus data collection on cheap-to-measure factors such as BMI, age, general health, income, and physical activity.","Combining global and local explanation methods can produce actionable thresholds and what-if recommendations, such as targeting BMI reduction or increased physical activity for high-risk patients.","The reported explanation metrics give a concrete template for auditing whether an explanation method is faithful and stable before clinical deployment."],"supporting_citations":[{"why":"Supplies the soft-voting ensemble-with-XAI approach this study extends to the larger BRFSS dataset.","marker":"[17]"},{"why":"Defines SHAP, the main global and local attribution method used for the feature rankings.","marker":"[19]"},{"why":"Defines LIME, the instance-level explanation method used for per-patient heatmaps.","marker":"[21]"},{"why":"Provides the bagging and boosting background that motivates the Random Forest, XGBoost, LightGBM combination.","marker":"[12]"},{"why":"Compares bagging and boosting ensemble classifiers, further grounding the choice of tree ensembles.","marker":"[13]"},{"why":"Benchmarks LIME versus SHAP on BRFSS survey data for diabetes, giving a comparative baseline for the XAI results.","marker":"[1]"},{"why":"Represents earlier Pima-based ML diabetes classification whose 70-91% accuracy range the paper claims to surpass.","marker":"[4]"},{"why":"Combines diabetes prediction with SHAP and LIME, a prior integration the paper positions against its multi-tool framework.","marker":"[24]"},{"why":"Defines anchors, the rule-based high-precision explanations used for threshold rules such as BMI and physical activity.","marker":"[22]"}],"fun_headline_variants":["Explainable AI ensemble predicts diabetes with 92.5% accuracy","Soft-voting ML plus XAI: BMI and age top diabetes risk factors","Diabetes prediction hits 92.5% accuracy with transparent AI","ML ensemble and explainable AI reveal key diabetes predictors","XAI-backed model: 92.5% diabetes prediction, top risks named"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy rests on the assumptions that SMOTE-created synthetic rows never entered the validation or test sets and that the BRFSS self-reported diabetes label is an accurate ground truth; if either fails, the reported 92.50% would overstate performance on real, clinically confirmed patients.","fun_headline_variants_meta":{"raw":{"variants":["Explainable AI ensemble predicts diabetes with 92.5% accuracy","Soft-voting ML plus XAI: BMI and age top diabetes risk factors","Diabetes prediction hits 92.5% accuracy with transparent AI","ML ensemble and explainable AI reveal key diabetes predictors","XAI-backed model: 92.5% diabetes prediction, top risks named"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000544,"raw_usage":{"total_tokens":2597,"prompt_tokens":932,"completion_tokens":1665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1573}},"tokens_in":548,"tokens_out":1665,"duration_ms":13888,"temperature":1.0,"reasoning_tokens":1573,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:46:37.205499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact pipeline with a test split that is fixed before any SMOTE synthesis and contains only original BRFSS rows; if test accuracy and ROC-AUC fall materially below 92.50% and 0.975, the central claim is not supported. The re-run should also reconcile the abstract's 92.50% test accuracy with Table II's 92.80% test accuracy.","supporting_citations":[],"review_version":1}