{"id":"a7df002e-3f7c-44ee-bda8-9911d1c01ec5","arxiv_id":"2506.17620","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper reports that SHAP-identified top features of deep learning models trained on BRFSS 2023 align with medical literature for 13 chronic diseases, and uses this alignment to argue the models are broadly trustworthy.","lead":"This paper trains deep learning models on U.S. survey answers to predict risk of 13 chronic diseases using only lifestyle and personal details. The authors claim the models can be trusted because the top influential features match published medical research.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation is unfalsifiable: post-hoc feature reinterpretation and lack of negative controls make the claimed cross-disease alignment nearly guaranteed, so the trustworthiness claim is unsupported.","rationale":"The central claim is that SHAP-identified top features align with medical literature across all 13 diseases, and that this alignment supports broad trustworthiness. For this claim to hold, the validation procedure must be able to fail: there must be a meaningful chance that a model could produce top features that do NOT align. The paper's procedure cannot fail. Features are selected after inspecting SHAP results, then post-hoc reinterpreted (employment and marital status as age, personal doctor and education as socioeconomic status), and general-health features are excluded because they are deemed consequences. A literature search then finds associations for the remaining features. Because each disease gets three features and each feature can be mapped to a broad concept like age, sex, smoking, or alcohol, supporting citations are nearly always available. No quantitative score, threshold, pre-registered hypothesis, or negative control is reported. The reader's weakest assumption identifies one important instance of this problem: employment and marital status are not established as age proxies, since the 'Unable to work' category is a plausible consequence of disease and is not analyzed separately. My concern is broader: even if the age proxy were established, the overall validation is unfalsifiable. The paper does provide some useful components: a public dataset, reasonable model training with close train/test loss, and transparent reporting of SHAP-based rankings. However, these do not rescue the central claim. A model with no medical knowledge could pass the reported validation. The proposed concrete test is directly implementable and would settle whether the alignment is real: compare the model's alignment score to a null distribution from permuted labels, using the same post-hoc scoring rules. If the real score is not distinguishable from random, the 'strong alignment' is an artifact of the evaluation protocol. Therefore the verdict remains REJECT, not because the models are necessarily untrustworthy, but because the paper's evidence for trustworthiness is not probative.","tokens_in":1017,"tokens_out":1044,"duration_ms":55284,"concrete_test":"Run a pre-registered validation protocol on the same BRFSS data: (1) Fix a mapping from each of 13 diseases to expected risk factors with direction before computing SHAP; define a scoring rule, e.g., fraction of top-3 features matching expected factors with correct sign. Then compute SHAP and score. (2) Generate a null distribution by training the same architecture on datasets with disease labels randomly permuted (preserving prevalence) and applying the identical literature-matching procedure, including the same post-hoc exclusion and reinterpretation rules. If the real model's score is not significantly above the null (e.g., above the 95th percentile), the validation has no discriminative power.","verdict_should_be":"REJECT","load_bearing_attack":"Section IV-B and IV-C present the paper's central evidence: the three SHAP top features per disease are matched to medical literature. This matching is not a test of the model; it is a post-hoc narrative. The authors first discard general-health features because they are 'consequences' (Sec. IV-B), but retain mental health as top predictor for depressive disorder without applying the same logic. They reinterpret employment and marital status as age proxies (Sec. IV-B1, IV-B3) despite the employment feature containing 'Unable to work', which they themselves call a likely consequence; Table V's 'Others' category conflates unable-to-work with several other statuses, so the age interpretation is not demonstrated. 'Personal doctor' is called a socioeconomic proxy, and 'Education' is also called a socioeconomic proxy, with no evidence that the model uses these features in the cited way. With three features per disease, flexible definitions, and a literature search after seeing the features, finding supporting citations is close to guaranteed; the paper reports no quantitative alignment metric, no pre-specified criteria, and no negative control. Therefore the observed 'strong alignment' is consistent with a model that has no genuine medical reasoning, and the abstract's conclusion that the approach 'can be broadly trusted' is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains deep ResNet models on the 2023 BRFSS dataset to predict 13 chronic diseases from 38 non-clinical personal and lifestyle features, using weighted cross-entropy loss to handle class imbalance. For each disease, SHAP is used to identify the three most influential features, and these are then paired with citations from the medical literature that associate the feature with the disease. The authors report that this alignment holds across all 13 diseases and conclude that the machine learning approach 'can be broadly trusted for chronic disease prediction.' The manuscript includes model training metrics (loss, accuracy, recall), SHAP-based feature rankings, disease-specific literature citations, and a short limitations section.","tokens_in":10734,"tokens_out":4348,"duration_ms":50635,"significance":"If the central claim were rigorously established, the paper would make a useful contribution by proposing that model explanations be validated against external medical knowledge as a trustworthiness check for self-directed risk tools. The use of a large, public dataset and the construction of 13 separate disease models are practical strengths, and the framing around self-directed preventive care addresses a real gap. However, the validation is presented as a post-hoc narrative rather than a falsifiable test: features are selected first, citations are chosen afterward, and no quantitative alignment metric, pre-registered criteria, or negative controls are reported. Because of this, the paper's main conclusion is currently unsupported, and the contribution is more of a proposal for a validation approach than a demonstration that the models are trustworthy. The manuscript also contains an unaddressed temporal mismatch between the cross-sectional BRFSS outcomes and the stated goal of predicting risk of 'developing' disease, which further limits the significance of the results as they stand.","major_comments":[{"comment":"The paper consistently describes the task as predicting the risk of 'developing' a chronic disease, but the BRFSS outcome variables (e.g., BPHIGH6, DIABETE4) are self-reported lifetime diagnoses, i.e., prevalent disease status, not incident disease. The models are therefore trained and evaluated on current or past disease status, and the results cannot be interpreted as prospective risk prediction without additional evidence. This distinction is central to the paper's stated purpose of self-directed preventive care, so the mismatch must be addressed by either reframing the claims or re-analyzing with an appropriate longitudinal or incident-disease design.","section":"Abstract, §I, §III.A"},{"comment":"The central validation is post hoc and unfalsifiable as presented. The authors first obtain the SHAP top-three features, then search the literature for citations linking each feature to the disease, without a pre-specified alignment criterion, a quantitative overlap metric, or any negative control. With only three features per disease and flexible narrative interpretation, finding supporting citations for common demographic and lifestyle variables is close to guaranteed; the reported 'strong alignment' is therefore consistent with a model that uses spurious or confounded associations. To support the claim, the authors should pre-register the feature-disease hypotheses, define a measurable alignment score with a threshold, compare the observed alignment to a null distribution (e.g., random features or models trained on shuffled labels), and report cases where the literature did not support a top feature.","section":"§IV-B, §IV-C"},{"comment":"The reinterpretation of employment and marital status as proxies for age is not demonstrated. For employment, Table V compares only students against retirees, but the employment variable includes 'Unable to work,' which the authors themselves describe as a likely consequence of chronic disease, and the 'Others' category conflates several heterogeneous statuses. The claim that the model uses these variables 'as an indicator of age' requires direct evidence (e.g., showing that the SHAP dependence on employment is mediated by age, or that the model's predictions track age-like patterns after controlling for employment status). Without such evidence, the cited age-risk literature does not validate the model's most influential features for the 8 of 13 diseases where employment is the top predictor.","section":"§IV-B1, §IV-B3, Table V, Table VII"},{"comment":"The exclusion of general-health features is applied inconsistently. The authors remove general health, physical health, and poor-health days because they are 'more likely to be consequences, rather than causes,' yet they retain mental health as the top predictor for depressive disorder and retain 'Unable to work' within the employment feature without applying the same consequence-based criterion. This selective application weakens the cause/consequence filter as a principled design choice and makes the resulting top-feature lists appear constructed to align with the medical literature. A consistent and pre-specified feature-inclusion rule is needed.","section":"§IV-B"},{"comment":"The limitations section states that 'a model's alignment with medical literature doesn't guarantee trustworthiness' and that alignment might reinforce existing biases, yet the Abstract and Conclusion assert that the approach 'can be broadly trusted.' This tension is not resolved in the manuscript. Either the conclusion should be weakened to match the acknowledged limitations, or the authors should provide evidence that the validation, despite its limitations, supports the specific trustworthiness claim being made.","section":"§V"}],"minor_comments":[{"comment":"There is a typo: 'the performance of the models isare comparable' should read 'the performance of the models is comparable' or 'the performances are comparable.'","section":"§IV-A"},{"comment":"The sentence 'We use SHAP to explain of our machine learning models' contains an extra 'of'; it should read 'to explain our machine learning models.'","section":"§IV-B"},{"comment":"The phrase 'females are more at risk forto asthma' should be corrected to 'females are more at risk for asthma.'","section":"§IV-B2"},{"comment":"The feature name 'Difficulty walking' in Table IV (stroke row) does not appear in the categories listed in Table I, where the disability items are described as 'difficulty making decisions, climbing stairs, dressing, doing errands.' The naming should be harmonized with the dataset's actual BRFSS variable.","section":"Table I, Table IV"},{"comment":"References [28] and [29] are cited out of order in the text; the Sasco et al. citation should appear before the Coughlin citation, or the numbering should be adjusted.","section":"§IV-C7"},{"comment":"The abbreviation 'NSMC' is introduced as 'Non-melanoma skin cancer (NSMC),' but the standard abbreviation used elsewhere is NMSC; please correct the typo.","section":"§IV-C6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not include a data or code availability statement, which is relevant for a paper whose central contribution is a validation methodology. In a revision, the authors should be encouraged to release code and detailed preprocessing steps, and to add a reproducible pipeline for the SHAP analysis and literature matching. Given the large number of cited references and the post-hoc matching approach, an independent assessment of the alignment metric would be valuable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a straightforward 13-disease prediction paper on BRFSS 2023 with a SHAP explainability add-on, and the add-on is the problem. The authors train a ResNet on 38 non-clinical features, report 65-75% accuracy and recall, and then for each disease take the top 3 SHAP features and find medical citations that those features are known risk factors. They call this 'strong alignment' and conclude the models 'can be broadly trusted.' That conclusion doesn't follow from the evidence.\n\nWhat's genuinely new is the 13-disease aggregation and the attempt to systematically check SHAP features against literature. That's a reasonable idea, and the authors handle messy BRFSS data carefully: unit conversions, skip-pattern handling, class-imbalance weighting. They also write an honest limitations section, including the risk of reinforcing gender biases in heart disease perception. Credit where it's due.\n\nThe soft spot is load-bearing. The validation procedure is unfalsifiable. Section IV-B discards general-health features because they're 'consequences,' yet mental health stays as the top predictor for depressive disorder. Employment and marital status are reinterpreted as age proxies after the fact, even though employment includes 'unable to work' (likely a consequence) and Table V's 'Others' category conflates statuses. Personal doctor and education are called socioeconomic proxies without evidence the model uses them that way. With three features per disease, flexible definitions, and a literature search after seeing the features, finding supporting citations is near-guaranteed. No quantitative alignment metric, no pre-specified criteria, no negative controls. So the observed alignment is consistent with a model with no medical reasoning.\n\nAlso, BRFSS is cross-sectional, so 'risk of developing' is the wrong framing; they're modeling prevalence.\n\nBottom line: the empirical foundation (data cleaning, training, basic evaluation) is solid, but the trustworthiness claim rests on a post-hoc narrative. A systematic protocol - pre-registered criteria, negative controls, a quantitative alignment score, external validation - would be needed to support the conclusion. As written, this is a working paper, not a finished result. I'd engage with it as a workshop contribution and encourage a rigorous revision. It deserves a serious referee if the authors commit to that extra work.\n\nRecommendation: don't publish in current form; invite revision.","headline":"Post-hoc literature matching makes the trustworthiness claim unfalsifiable; the data work is decent but the central conclusion needs a real validation protocol.","tokens_in":11286,"tokens_out":3544,"would_cite":false,"duration_ms":38703,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For all 13 chronic diseases, the model's most influential features match established medical risk factors.","keywords":["chronic disease risk prediction","SHAP explainability","medical literature validation","Behavioral Risk Factor Surveillance System","deep learning","self-directed preventive care","model trustworthiness","non-clinical risk factors"],"falsifier":"Retrain the model with an explicit age feature among the inputs: if age becomes a top-three feature and employment and marital status fall out of the top three, the paper's age-proxy interpretation is supported, whereas if employment and marital status stay on top even with age controlled, the model is not simply encoding age and the age-based medical literature used for 11 of the 13 diseases does not explain the model's behavior.","tokens_in":10272,"feed_emoji":"🩺","tokens_out":8422,"duration_ms":89660,"temperature":0.7,"pith_summary":"This paper tries to establish that a deep learning model can be trusted to predict chronic-disease risk from ordinary personal and lifestyle questions, because the model's most influential features agree with established medical knowledge. The authors train a ResNet on the 2023 Behavioral Risk Factor Surveillance System survey to score risk for 13 chronic diseases using 38 non-clinical inputs, then use SHAP to rank feature influence for each disease. Across all 13 diseases, the three most influential features match documented risk factors, most often age, weight, sex, smoking, alcohol use, and physical activity. If that alignment is accepted, the models provide a low-cost, transparent screening signal for self-directed prevention and a reusable recipe for auditing health-risk tools.","feed_headline":"13 chronic disease risk models pass a medical literature check","feed_subtitle":"A model using only lifestyle questions picks the same top risk factors doctors know for all 13 diseases.","key_machinery":"The load-bearing mechanism is an explanatory audit: SHAP (Shapley Additive exPlanations) with the Kernel Explainer is run on each trained ResNet, using 500 randomly sampled survey responses and a 100-point k-means background set, and the per-feature SHAP values are summed to produce a global ranking. For each disease, the top three ranked features are then checked against peer-reviewed medical studies. Two categorical inputs, employment and marital status, are recast as age proxies on the basis of their answer categories (student, retired, unable to work; divorced, widowed), which is what lets the age-risk literature cover most diseases.","core_discovery":"The central claim is that for every one of the 13 chronic diseases, the three inputs SHAP ranks highest for the deep model's risk score correspond to risk factors the medical literature identifies as significant, and this cross-disease agreement is evidence that the models can be broadly trusted for chronic disease prediction. The authors deliberately exclude general-health and disability-like fields because those are more likely consequences than causes, and they interpret employment and marital status as indicators of age based on survey answer categories. The result is framed as the first self-directed-care model whose explanations are validated against medical literature rather than merely reported.","pith_inferences":["The authors do not report how stable the top-three SHAP rankings are across retraining runs; that variance would matter for anyone relying on the trustworthiness claim.","Because employment and marital status are validated mainly as age stand-ins, adding age as an explicit feature would probably absorb their influence and could yield a simpler model with the same behavior.","A stricter validation would check not only whether the top features appear in the literature but also whether the direction of each SHAP effect matches the known risk direction (for example, smoking raising COPD risk), which the paper does not test.","The age-proxy reading implies the models are in part social-surrogate age predictors, so deploying them in populations where employment and marital patterns differ from the U.S. survey sample would require re-validation."],"forward_implications":["Anyone can get a first-pass chronic disease risk score from a short lifestyle questionnaire, because none of the 38 inputs requires a lab test or exam.","The same explainability-plus-literature audit can be applied to other health-risk models before public release, flagging models whose top features are medically implausible.","Because the pattern repeats across 13 diseases, the approach is presented as a general trustworthiness check rather than a one-off result for a single condition.","Classification accuracy and recall between 65% and 75%, comparable to earlier non-clinical models, mean the trustworthiness argument applies to models with ordinary predictive performance."],"supporting_citations":[{"why":"Supplies the 2023 BRFSS survey data: all 38 input features and the 13 disease labels used to train and evaluate the models.","marker":"[10]"},{"why":"Provides the obesity-high blood pressure link used to validate weight as the top predictor for high blood pressure.","marker":"[12]"},{"why":"Establishes aging as a driver of heart disease, used to validate employment-as-age for heart attack and coronary heart disease.","marker":"[17]"},{"why":"Documents age and physical activity effects on stroke risk, used to validate employment and exercising as top predictors.","marker":"[19]"},{"why":"Establishes age as the leading carcinogen, used to validate employment-as-age for the two cancer outcomes.","marker":"[27]"},{"why":"Provides smoking-chronic bronchitis relative risks, used to validate smoking and employment for chronic bronchitis.","marker":"[30]"},{"why":"Establishes gender differences in depression, used to validate sex as a top predictor for depressive disorder.","marker":"[33]"},{"why":"Documents age as a major contributor to kidney disease, used to validate employment as a top predictor for kidney disease.","marker":"[35]"},{"why":"Shows aging and sex shape rheumatoid arthritis risk, used to validate employment, marital status, and sex for arthritis.","marker":"[38]"},{"why":"Supports BMI as a diabetes risk factor, used to validate weight as the top predictor for diabetes.","marker":"[40]"}],"fun_headline_variants":["Lifestyle-only AI risk scores validate against medical literature for 13 diseases","Models using only lifestyle questions pick doctor-known risk factors for 13 diseases","SHAP explanations match established literature across all 13 chronic diseases","13 chronic disease models pass a medical literature check on risk factors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on treating employment status (and, for two diseases, marital status) as a reliable proxy for age in the survey data; if those variables do not track age, the age-related medical studies cited for 11 of the 13 diseases no longer validate the models' most influential features.","fun_headline_variants_meta":{"raw":{"variants":["Lifestyle-only AI risk scores validate against medical literature for 13 diseases","Models using only lifestyle questions pick doctor-known risk factors for 13 diseases","SHAP explanations match established literature across all 13 chronic diseases","13 chronic disease models pass a medical literature check on risk factors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000488,"raw_usage":{"total_tokens":2376,"prompt_tokens":893,"completion_tokens":1483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1408}},"tokens_in":509,"tokens_out":1483,"duration_ms":13156,"temperature":1.0,"reasoning_tokens":1408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:30:09.807155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the model with an explicit age feature among the inputs: if age becomes a top-three feature and employment and marital status fall out of the top three, the paper's age-proxy interpretation is supported, whereas if employment and marital status stay on top even with age controlled, the model is not simply encoding age and the age-based medical literature used for 11 of the 13 diseases does not explain the model's behavior.","supporting_citations":[{"cited_title":"CDC - 2023 BRFSS survey data and Documenta- tion,","cited_arxiv_id":null,"evidence_quote":"Supplies the 2023 BRFSS survey data: all 38 input features and the 13 disease labels used to train and evaluate the models."},{"cited_title":"Aging and Diseases of the Heart,","cited_arxiv_id":null,"evidence_quote":"Establishes aging as a driver of heart disease, used to validate employment-as-age for heart attack and coronary heart disease."},{"cited_title":"The age of cancer,","cited_arxiv_id":null,"evidence_quote":"Establishes age as the leading carcinogen, used to validate employment-as-age for the two cancer outcomes."},{"cited_title":"Systematic review with meta-analysis of the epidemiological evidence relating smoking to COPD, chronic bronchitis and emphysema,","cited_arxiv_id":null,"evidence_quote":"Provides smoking-chronic bronchitis relative risks, used to validate smoking and employment for chronic bronchitis."},{"cited_title":"Gender differences in depression: Critical review,","cited_arxiv_id":null,"evidence_quote":"Establishes gender differences in depression, used to validate sex as a top predictor for depressive disorder."},{"cited_title":"Chronic kidney disease in older people,","cited_arxiv_id":null,"evidence_quote":"Documents age as a major contributor to kidney disease, used to validate employment as a top predictor for kidney disease."},{"cited_title":"The influence of ageing on the development and management of rheumatoid arthritis,","cited_arxiv_id":null,"evidence_quote":"Shows aging and sex shape rheumatoid arthritis risk, used to validate employment, marital status, and sex for arthritis."},{"cited_title":"Diabetes educators,","cited_arxiv_id":null,"evidence_quote":"Supports BMI as a diabetes risk factor, used to validate weight as the top predictor for diabetes."}],"review_version":1}