{"id":"7be647be-0980-497d-bf0d-ed7bcdcc9ac0","arxiv_id":"2505.05683","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A LightGBM model trained on undersampled 2015 BRFSS data is wrapped in a Dash app with SHAP, LIME, and comorbidity insights for interactive diabetes risk prediction.","lead":"This paper builds a web-based diabetes risk calculator from CDC survey data, combining machine learning with SHAP and LIME explanations. It is an applied engineering effort rather than a new scientific discovery, and its main value is as a template for explainable public health tools.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Raw model probabilities from the undersampled LightGBM are presented as personal diabetes risk without recalibration; this is the load-bearing weakness.","rationale":"The reader's weakest assumption correctly identifies the undersampling-induced miscalibration of user-facing probabilities as the central issue. My read agrees: the model-selection story (recall maximization under undersampling) is internally coherent for ranking, but the deployed Dash application converts raw LightGBM scores into personal risk probabilities, and no recalibration step is described. The internal inconsistency in the SHAP base value (1.393 log-odds, implying about 0.80 baseline probability, rather than the 0.5 prior expected under random undersampling) strengthens the concern that the reported probability interpretation is not tied to the actual deployed model. Because the paper provides no numeric tables, code, or live application, the severity of the miscalibration cannot be estimated from the manuscript alone, which is exactly why the conditional verdict is appropriate: acceptance should require either a demonstrated recalibration or an explicit reframing of the output as a relative risk score rather than a probability. I find no reason to move away from the reader's CONDITIONAL verdict; the concern is real but testable and does not by itself invalidate the engineering prototype claim.","tokens_in":8543,"tokens_out":2411,"duration_ms":25769,"concrete_test":"On a held-out test set drawn from the original (unbalanced) BRFSS distribution, compute both the raw LightGBM probability output and a calibrated probability using Platt scaling or isotonic regression fit on a separate validation set. Then compare the raw and calibrated probabilities for the representative profile from Figure 10 (BMI 40, GenHlth 5, RiskFactorCount 2, HighBP present). If the calibrated probability differs from the displayed 92% by more than, say, 10 percentage points, the app's risk communication is materially misleading without recalibration. Also report the expected calibration error on the original-distribution test set to quantify the miscalibration systematically.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a recall-optimized LightGBM trained on random-undersampled 2015 BRFSS data is suitable for deployed diabetes risk detection. The ranking-oriented part of this claim can survive undersampling: with a 50/50 training prior, the model can still order individuals by risk. What does not survive is the user-facing presentation of raw classifier probabilities as personal diabetes risk. Undersampling changes the class prior: the training set is roughly 50/50, while the true BRFSS prevalence is approximately 15.59% diabetic (Section III.B). The raw LightGBM probability output therefore encodes the 50/50 prior, not the population prior. Without Platt scaling, isotonic regression, or a Bayes-factor adjustment, the probabilities displayed in the Dash app are systematically inflated. Section IV.D reports a representative prediction at 92% probability; this number should be understood as an undersampled-model score, not a posterior probability of diabetes. The paper never states that this adjustment was made, and the SHAP waterfall in Section IV.C even reports a base log-odds of 1.393 (about 0.80 probability), which is inconsistent with a 50/50 training prior and is never reconciled with the deployed model. Additionally, the key numeric results exist only in figures; no code, data, or live app is provided, so the recall advantage and the calibration state cannot be independently checked. Because the app communicates risk probabilities directly to end users, the missing recalibration is the most load-bearing problem for the central deployment claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an end-to-end system for diabetes risk prediction using the 2015 CDC BRFSS dataset, with models trained under original, SMOTE, and random-undersampling strategies. LightGBM trained on the undersampled data is selected as the final model based on recall-optimized cross-validation and ANOVA/Tukey HSD tests. The paper also introduces composite lifestyle and healthcare scores, adds SHAP and LIME explanations, and deploys the model as a multi-step Dash web application with personalized recommendations and comorbidity insights.","tokens_in":8962,"tokens_out":4021,"duration_ms":45562,"significance":"If the claims are taken at face value, the paper shows a practical integration of standard ML tools, class-imbalance handling, and explainability for a public-health screening scenario. The use of a large public dataset, comparison across three sampling strategies, statistical significance testing, and an interactive deployment are strengths that make the result relevant to applied ML and health-informatics audiences. However, the central claim that the deployed model provides reliable personal diabetes risk estimates is undermined by the absence of probability calibration after undersampling, by an unexplained SHAP base value, and by the lack of a numeric results table or reproducible artifacts. These issues are fixable, but they are load-bearing for the paper's central contribution.","major_comments":[{"comment":"The LightGBM model selected in Section IV.A is trained on data that random undersampling has made approximately class-balanced, while the true BRFSS prevalence is 15.59% diabetic as stated in Section III.B. The raw probabilistic output of such a model encodes the 50/50 training prior, not the population prior. In Section IV.D, the paper reports a '92% probability of diabetes' for a representative individual, but no calibration step (Platt scaling, isotonic regression, or Bayes-factor adjustment) is described anywhere. Because the Dash application presents these probabilities directly to end users as personal diabetes risk, the displayed numbers are systematically inflated unless a calibration adjustment is made. The paper must either add a described and validated calibration procedure or reframe the output as an uncalibrated risk score and clearly state the limitation.","section":"Sections III.D, IV.A, IV.D"},{"comment":"The SHAP waterfall plot in Figure 9 reports a base value of 1.393 in log-odds, which corresponds to a baseline probability of approximately 0.80. This is inconsistent with the claimed 50/50 training prior of the undersampled model and also inconsistent with the population prevalence of 15.59% reported in Section III.B. The paper never explains why the SHAP base value is so high or whether the SHAP explanation was computed on a different model, a different feature encoding, or a transformed target. Without this reconciliation, the interpretability claim in Section IV.C is not internally consistent with the deployment claim.","section":"Section IV.C, Figure 9"},{"comment":"The evaluation protocol is incompletely specified and impedes verification. Section III.E states that GridSearchCV was used for hyperparameter tuning, but it is not stated whether tuning was nested inside the cross-validation folds used to report recall. If hyperparameters were selected on the full dataset before cross-validation, the reported recall values and the ANOVA results in Section IV.B are optimistically biased. Additionally, the paper gives only figures for model performance and no numeric table of recall, precision, F1, or AUC means and standard deviations, and no effect sizes or confidence intervals for the pairwise Tukey HSD comparisons. The central selection claim of 'best recall' cannot be independently assessed from the text as written.","section":"Sections III.E and IV.B"},{"comment":"The paper presents the Dash application as suitable for real-world public health use, but it provides no external validation on a temporally or geographically distinct dataset and no comparison with established diabetes risk instruments. The conclusion itself lists 'external clinical validation' only as future work (Section V). Given that the deployed system communicates individual risk probabilities to lay users, a held-out validation set, or at minimum a clear explanation of why such validation is unnecessary for the paper's scope, is needed to support the deployment claim.","section":"Sections V and III.H"}],"minor_comments":[{"comment":"The list of models is inconsistent: Section III.E names Logistic Regression, Decision Tree, Random Forest, XGBoost, and LightGBM, while Section IV.A states that six classifiers were evaluated, including KNN and Neural Network. Please reconcile the lists.","section":"Section III.E versus Section IV.A"},{"comment":"Two subsections are labeled IV.D: one covers LIME-Based Local Explanations and the other covers the Interactive Web Interface. The section numbering should be corrected.","section":"Section IV"},{"comment":"The 'Lifestyle Score' and 'Healthcare Access Score' are presented to users as personalized scores, but the component weights and the normalization procedure are not defined anywhere. Please provide the exact formulas or, if the weights are arbitrary, state that explicitly.","section":"Section III.C"},{"comment":"The paper excludes prediabetic entries to form a binary target, but the deployed app is described as a general diabetes risk tool. It should be stated clearly in the interface and limitations that the model cannot distinguish prediabetes from non-diabetes and that prediabetic individuals are treated as non-diabetic in training.","section":"Section III.B"},{"comment":"The sentence 'No missing values were present in the dataset' is surprising for BRFSS data and should be clarified, since typical BRFSS extracts contain 'Don't know/Refused' categories that must be handled explicitly.","section":"Section III.A"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a readable, end-to-end machine learning pipeline for diabetes risk prediction on BRFSS 2015, with a sensible recall-focused model selection (LightGBM + undersampling), SHAP/LIME explanations, and a Dash web app. The thing to know before you read it: the app shows raw model probabilities as personal diabetes risk, but the model was trained on a 50/50 undersampled class distribution while the true prevalence is about 15.6%. Those probabilities are systematically inflated and are never recalibrated. The SHAP waterfall base value of 1.393 log-odds (~0.80) is inconsistent with both the 50/50 training prior and the population prior, and the paper does not reconcile this. That is not a minor bug; it undercuts the central deployment claim.\n\nWhat the paper does well: the pipeline is complete and clearly described. The authors compare six classifiers across three sampling strategies, run ANOVA and Tukey HSD on recall scores, and are explicit about their recall-first rationale. The feature engineering (lifestyle score, healthcare access score, risk factor count) is useful for interpretability even if hand-defined. The comorbidity correlations are clinically plausible and the Dash interface, with personalized suggestions, is a genuinely usable artifact.\n\nSoft spots, in proportion: the calibration issue is the most serious. Also, all key numeric results appear only in figures; there are no recall tables, effect sizes, or confidence intervals. No code, data, or live app link is provided, so the results cannot be independently checked. The ANOVA on cross-validation folds treats non-independent folds as independent, a common but real methodological slip. The related work is thin and leans on preprints, but it does cite the two most relevant prior works (Liu et al. on SHAP with BRFSS 2015, and Ahmed et al. on SHAP+LIME for diabetes), so the incremental contribution is honestly framed.\n\nOverall: the core ranking result is plausible, and the engineering is competent, but the user-facing risk communication is not valid as presented. This deserves a serious referee at an applied or engineering venue, not a desk reject, but only if the author supplies numeric tables, code/data/live app, and a recalibration step (or an explicit Bayes-factor adjustment). I would not cite it the next year, but I would bring it to a reading group as an example of real-world XAI pitfalls.","headline":"A competent but incomplete applied ML prototype: the recall-optimized LightGBM claim is plausible, yet the deployed app presents uncalibrated probabilities as personal risk, which is the load-bearing flaw.","tokens_in":9348,"tokens_out":1865,"would_cite":false,"duration_ms":20108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a recall-optimized LightGBM model trained on randomly undersampled 2015 BRFSS data provides the most reliable diabetes risk detection among six tested classifiers, and that embedding it in an explainable web app…","keywords":["diabetes risk prediction","BRFSS","LightGBM","random undersampling","class imbalance","SHAP","LIME","Dash application"],"falsifier":"Re-run the six classifiers on a second, untouched BRFSS year (for example, 2017 or 2019) using the same undersampling strategy, the same recall metric, and the same cross-validation folds; if LightGBM does not again achieve the highest recall with a statistically significant margin, the paper's central model-selection claim is refuted.","tokens_in":1442,"feed_emoji":"🩺","tokens_out":1614,"duration_ms":93697,"temperature":0.7,"pith_summary":"This paper tries to establish that a complete, explainable diabetes-risk screening pipeline can be built from a large self-reported public-health survey, and that the choice of class-balancing technique matters as much as the choice of classifier. It reports that LightGBM trained on randomly undersampled BRFSS data, with diabetic and non-diabetic cases forced into a roughly 50/50 split, achieves the best recall among six classifiers, and argues that maximizing recall is the right objective for screening because missed diabetes cases delay care. To make the model usable, the paper wraps it in a Dash web application that shows SHAP and LIME explanations for each prediction, computes composite lifestyle and healthcare-access scores, and flags correlated comorbidities such as high blood pressure and high cholesterol. A sympathetic reader would care because the paper offers a template for turning a black-box classifier into a public-facing risk communication tool.","feed_headline":"Undersampled LightGBM wins recall for diabetes screening","feed_subtitle":"A recall-first model catches more true diabetes cases, and SHAP plus LIME explain each risk score to users.","key_machinery":"The load-bearing mechanism is the combination of random undersampling and recall-based model selection: forcing the training set to a 50/50 balance pushes the model to treat diabetic cases as common, raising sensitivity at the expense of precision. LightGBM, a gradient-boosting tree ensemble, carries the prediction, and engineered features, especially the Risk Factor Count, anchor both the model's decisions and its SHAP explanations. SHAP supplies global feature-attribution and local waterfall explanations, LIME gives a second, locally approximated view of individual predictions, and Pearson correlation analysis provides comorbidity flags. The Dash application turns these outputs into a multi-step web form with real-time results, lifestyle and healthcare-access scores, and personalized recommendations.","core_discovery":"The paper reports that, across three resampling strategies (original, SMOTE, and random undersampling), random undersampling produced the highest recall for detecting diabetic cases, and LightGBM was the model that benefited most consistently from that strategy. ANOVA and Tukey HSD tests on cross-validated recall scores (p-values of $2.13\\times10^{-24}$ for SMOTE and $5.39\\times10^{-10}$ for undersampling) are presented as evidence that the model differences are statistically significant. The paper therefore selects a recall-optimized LightGBM as the deployed classifier and justifies this choice for health screening by arguing that false negatives are the most costly error. It also reports that an engineered Risk Factor Count, summing high blood pressure, high cholesterol, stroke, and heart disease indicators, is the strongest SHAP contributor to predictions, and that diabetes correlates most strongly with high blood pressure ($r=0.26$) and high cholesterol ($r=0.20$).","pith_inferences":["The raw probabilities shown to users are likely inflated because the model was trained on a 50/50 sample while real-world type 2 diabetes prevalence is roughly 10–15%; calibrating the output to population prevalence would keep the risk ranking intact while making the displayed percentages honest.","Because SHAP identifies Risk Factor Count as the dominant predictor, a simple additive risk score may reproduce much of the model's ranking; testing whether a paper-based score matches LightGBM would offer a low-tech screening alternative.","The same modular pipeline, including undersampling, recall selection, SHAP/LIME, and correlation flags, could be retargeted to other chronic conditions such as hypertension or heart-disease risk by changing the target variable and re-running the comorbidity analysis."],"forward_implications":["If the recall advantage is real, a screening deployment of this type will catch more true diabetic cases than an accuracy-tuned model, at the cost of more false positives.","The SHAP and LIME integration lets a user see which factors, such as Risk Factor Count, general health, or BMI, drove their personal risk, enabling targeted lifestyle changes rather than a bare probability.","Comorbidity flags for hypertension, cholesterol, heart disease, and stroke turn the diabetes prediction into a broader cardiovascular risk prompt.","The pipeline's modular design suggests it can be adapted to other chronic conditions or to more recent survey data without a full redesign.","The deployed Dash application demonstrates that explainable machine learning can be delivered to nontechnical users in real time."],"supporting_citations":[{"why":"Supplies the prior application of sampling techniques (SMOTE, SMOTE-Tomek, SMOTE-EN) to BRFSS diabetes data, providing the imbalance-handling baseline this paper extends.","marker":"[1]"},{"why":"Provides a comparative BRFSS 2015 study with SMOTE and SHAP, justifying the dataset choice and the SHAP-based feature-importance precedent.","marker":"[2]"},{"why":"Motivates the dual explainability approach by comparing LIME and SHAP interpreters for diabetes predictions.","marker":"[3]"},{"why":"Serves as the simpler interpretable-model baseline on BRFSS lifestyle factors that this paper contrasts with its full pipeline.","marker":"[4]"}],"fun_headline_variants":["Recall-optimized LightGBM chosen for diabetes screening tool","Explainable AI dashboard highlights diabetes comorbidity links","Interactive diabetes risk app uses SHAP and LIME for insights","Risk Factor Count strongest predictor in explainable diabetes model","Dash-based tool explains each diabetes risk score with SHAP"],"cache_read_input_tokens":11520,"weakest_assumption_plain":"The tool's raw probability output comes from a model trained on a 50/50 balanced sample, so the displayed risk percentages assume diabetes is far more common in the user population than it actually is; if that prevalence assumption is wrong, the numbers are systematically inflated even if the ranking of who is at higher risk remains useful.","fun_headline_variants_meta":{"raw":{"variants":["Recall-optimized LightGBM chosen for diabetes screening tool","Explainable AI dashboard highlights diabetes comorbidity links","Interactive diabetes risk app uses SHAP and LIME for insights","Risk Factor Count strongest predictor in explainable diabetes model","Dash-based tool explains each diabetes risk score with SHAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000458,"raw_usage":{"total_tokens":2253,"prompt_tokens":857,"completion_tokens":1396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":1316}},"tokens_in":473,"tokens_out":1396,"duration_ms":15782,"temperature":1.0,"reasoning_tokens":1316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:58:53.295579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six classifiers on a second, untouched BRFSS year (for example, 2017 or 2019) using the same undersampling strategy, the same recall metric, and the same cross-validation folds; if LightGBM does not again achieve the highest recall with a statistically significant margin, the paper's central model-selection claim is refuted.","supporting_citations":[{"cited_title":"Diabetes diagnosis through machine learning: Investigating algorithms and data augmentation for class imbalanced BRFSS dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior application of sampling techniques (SMOTE, SMOTE-Tomek, SMOTE-EN) to BRFSS diabetes data, providing the imbalance-handling baseline this paper extends."},{"cited_title":"Acomparativestudy ofmachinelearningapproachesfordiabetesriskprediction:Insightsfrom SHAP and feature importance,","cited_arxiv_id":null,"evidence_quote":"Provides a comparative BRFSS 2015 study with SMOTE and SHAP, justifying the dataset choice and the SHAP-based feature-importance precedent."},{"cited_title":"A comparative analysis of LIME and SHAP interpreters with explainable ML-baseddiabetespredictions,","cited_arxiv_id":null,"evidence_quote":"Motivates the dual explainability approach by comparing LIME and SHAP interpreters for diabetes predictions."},{"cited_title":"A comparative study of diabetes prediction based on lifestyle factors using machine learning,","cited_arxiv_id":null,"evidence_quote":"Serves as the simpler interpretable-model baseline on BRFSS lifestyle factors that this paper contrasts with its full pipeline."}],"review_version":1}