{"id":"e6d050f6-1097-4b06-b549-9a450ae02327","arxiv_id":"2607.15721","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CardioMeta reports a calibrated multi-task model for three cardiometabolic diseases across NHANES and MIMIC-IV, with the main contribution being a leakage-reduced evaluation protocol.","lead":"CardioMeta predicts diabetes, hypertension, and heart disease together from survey and hospital data, with calibrated probabilities intended for screening use. The paper's main claim—that its evaluation removes cheating signals—is undercut by the architecture's shared encoder, which may still leak label-defining variables.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Shared encoder architecture makes the reported per-head leakage exclusion impossible; label-defining variables can reach every disease head.","rationale":"The reader's weakest assumption is exactly the one that the manuscript text fails to support: that label-defining variables can be excluded per disease head when the architecture uses a shared encoder over a single feature vector. My reading of §3.3, §3.5, and Eq (3) confirms this. There is no mathematical or textual mechanism for the claimed exclusion. The gating operation in Eq (4) multiplies h_i by a learned mask, but h_i already contains information from all inputs that entered the shared encoder; masking after the encoder cannot remove information that is already entangled throughout the representation. The only way to achieve the claimed leakage reduction is to supply different, masked inputs to each task before the encoder, which is not described and would contradict the stated shared-encoder design. Thus the primary leakage-reduced evaluation, and the modest-but-consistent-improvement claim built on it, is not established. The paper does have genuine virtues: it provides confidence intervals, a full-clinical sensitivity analysis, calibration reporting, and an honest discussion of domain shift. But those do not rescue the central methodological contribution. Since the reader's verdict was REJECT and this analysis confirms that rejection, the recommended verdict is unchanged. The concrete test proposed here would settle the concern by checking either the actual input feature vector and/or a median-imputation probe that removes excluded-variable information before the encoder; a material performance change would demonstrate leakage in the reported numbers.","tokens_in":11432,"tokens_out":4202,"duration_ms":34420,"concrete_test":"Inspect the released code and feature manifests to determine whether the feature vector passed to Eq (3) in the leakage-reduced setting still contains HbA1c, fasting glucose, blood-pressure variables, and medication indicators. Then run one controlled probe: on the temporal test split, set all excluded variables to their training-set medians while keeping every other feature unchanged, and recompute each disease's predictions. If diabetes and hypertension AUROC or Brier score shifts materially, the excluded variables are reaching the heads through the shared encoder, confirming that the leakage-reduced setting is not implemented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the primary setting removes label-defining variables for each disease. But the architecture in Eq (3) maps a single preprocessed feature vector x̃_i to a shared representation h_i, and Eqs (4)-(5) then feed h_i to every disease-specific gated head. If x̃_i contains HbA1c, fasting glucose, blood-pressure measurements, or medication indicators, the information from those variables is nonlinearly embedded in h_i and cannot be blocked by the heads' input gating; gating can only reweight already-entangled features. §3.3 says these variables are 'excluded from the corresponding disease head's leakage-reduced feature set,' but no per-task input masking is described anywhere. The only way the described exclusion could work is if each task received its own masked x̃_i^k before the shared encoder, which would contradict Eq (3) and turn the 'shared encoder' into three effectively separate encoders. Therefore, as written, the leakage-reduced setting does not implement leakage reduction. The reported AUROC/calibration numbers in the primary setting may still be inflated by circular diagnostic evidence. This is the load-bearing assumption of the paper: reproducible leakage control is the stated principal contribution. If this assumption fails, the central claim collapses, regardless of the paper's other strengths such as transparent reporting intentions and confidence intervals. A secondary but related issue is that the temporal validation cohort (2017-March 2020) overlaps with the 2017-2018 cycle used in development, compounding the concern, but the architecture flaw alone is decisive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CardioMeta, a shared-encoder multi-task model for joint prediction of diabetes, hypertension, and cardiovascular disease using NHANES for development and temporal validation and MIMIC-IV for external EHR evaluation. The primary setting is claimed to be leakage-reduced by excluding label-defining variables (HbA1c, fasting glucose, blood pressure, medications, CVD history) from the corresponding disease prediction head; a full-clinical setting is retained as sensitivity. The authors report macro-AUROC 0.839, macro-AUPRC 0.536, macro-F1 0.614, and ECE 0.024 on NHANES temporal validation, with modest gains over boosting and neural tabular baselines, degraded direct transfer to MIMIC-IV, partial recovery after fine-tuning, plus calibration, ablation, subgroup, and SHAP analyses. The stated principal contribution is reproducible leakage control and transparent reliability reporting.","tokens_in":11739,"tokens_out":7694,"duration_ms":64594,"significance":"If the architecture actually enforced the stated feature exclusions, this would be a useful benchmark study: the protocol covers temporal validation, external domain shift, bootstrap confidence intervals, calibration, decision curves, subgroup reliability, and a broad set of baselines. I credit the authors for explicitly distinguishing prevalent disease-status prediction from incident forecasting, for reporting calibration and uncertainty alongside discrimination, and for making the reproducibility protocol a central focus. However, the two load-bearing problems below mean the primary leakage-reduced results cannot currently be taken at face value; the paper's central contribution is contingent on a corrected implementation and a non-overlapping temporal split.","major_comments":[{"comment":"The leakage-reduced setting is not implemented by the architecture as written. §3.3 says label-defining variables are 'excluded from the corresponding disease head's leakage-reduced feature set,' but Eq. (3) maps a single preprocessed vector x̃_i through a single shared encoder to h_i, and Eqs. (4)–(5) feed h_i to every disease-specific gated head. The gate g_i^k in Eq. (4) is itself a nonlinear function of h_i, so it can reweight but cannot remove information about HbA1c, fasting glucose, blood pressure, or medication indicators once those values are embedded in the shared representation. No per-task masked input x̃_i^k is defined anywhere, and Figure 1's 'leakage-safe preprocessing' is not specified. As written, the primary setting does not achieve leakage reduction, and the Table 2/3 results may still be driven by circular diagnostic evidence. This is the paper's stated principal cont","section":"§3.5, Eqs. (3)–(5); §3.3"},{"comment":"The temporal validation split overlaps with the development cohort. §3.2 states that the development cohort uses NHANES 2011–2012, 2013–2014, 2015–2016, and 2017–2018 cycles, while the temporal validation cohort uses the 2017–March 2020 pre-pandemic release. The 2017–March 2020 release combines the 2017–2018 cycle with the partial 2019–2020 cycle, so the temporal validation set appears to include the same 2017–2018 participants used for development. §4.1 says records used for model selection were 'separated,' but no record-level de-duplication or exclusion of 2017–2018 participants is described. This makes the reported 'temporal validation' not a clean temporal split and can inflate the headline results. Please specify exactly how overlapping cycles were handled or use a non-overlapping split (e.g., 2011–2016 development, 2017–March 2020 validation).","section":"§3.2, §4.1"}],"minor_comments":[{"comment":"The DeLong p-value of 0.047 for the AUROC gain over CatBoost is borderline and no multiple-comparison correction is applied across the many baselines; the calibration and macro-F1 differences are more robust and should be emphasized as the primary evidence of improvement.","section":"§4.2, Table 2"},{"comment":"The paper states that SMOTE-ENN was evaluated as a training-fold sensitivity analysis and that layer normalization was evaluated as a sensitivity variant, but no results for either are reported. Please either provide the results or remove the claims.","section":"§3.7, §4.5"},{"comment":"The text mentions 'SHAP and counterfactual analyses,' but only SHAP group attributions are shown; no counterfactual results are reported. Please add them or revise the wording.","section":"§4.6"},{"comment":"The CVD false-negative rate in Table 8 and the threshold/utility definitions in Table 7 are not described in the metrics section. Please define the operational threshold used for FNR and the net-benefit calculation.","section":"§3.8, Table 7, Table 8"},{"comment":"The comparison between full-NHANES temporal validation and shared-core MIMIC transfer is not apples-to-apples; clarify whether the NHANES temporal row in Table 5 uses the full NHANES feature space or the shared-core space, and state which features are used after MIMIC fine-tuning.","section":"§3.4, §4.4"},{"comment":"The 'clinical scores' baseline row in Table 2 is not specified; please list which scores were combined and how, since these are not standard implementations of Framingham or FINDRISC for multi-label disease-status prediction.","section":"§2.1, Table 2"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: the architecture as written cannot implement the claimed per-head leakage exclusion, and the temporal split overlaps. These are load-bearing, and the current headline numbers are not interpretable. I choose major_revision rather than reject because the flaws are correctable in principle: per-task masked inputs and a non-overlapping split would require rerunning all experiments, but the paper's scope can accommodate that. If the authors cannot provide a corrected implementation and rerun the experiments, the manuscript should not be published."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the central methodological claim—that the primary analysis excludes label-defining variables per disease—does not hold up in the architecture actually described. Equation (3) maps the full feature vector through one shared encoder; the gated heads then operate on that shared representation. So HbA1c, blood pressure, and medication indicators can flow to every disease head. The per-head exclusions in §3.3 are stated as intent, not as an implemented mechanism. There is no description of per-task input masking anywhere in the text. The only way the exclusion could work would be to feed each head its own masked input before the shared encoder, which would contradict Eq (3) and make the 'shared encoder' effectively task-specific. As written, the leakage-reduced setting does not reduce leakage.\n\nThat is a shame, because the paper has real strengths. The reporting discipline is unusually good: the authors separate leakage-reduced and full-clinical settings, report bootstrap confidence intervals, run a broad and sensible baseline suite, and check calibration, subgroup reliability, decision-curve analysis, and external domain shift. The NHANES-to-MIMIC transfer experiment is a useful, honestly interpreted stress test—the degradation under direct transfer is a fair cautionary result. The calibration advantage (ECE 0.024 vs 0.035 for CatBoost) is the most plausible real finding, since it survives even if the discrimination numbers are inflated by leakage.\n\nThe architecture flaw is load-bearing, not cosmetic. The paper's stated principal contribution is 'reproducible leakage control,' and that is what fails. There is also a secondary concern about temporal validation: development uses NHANES 2011–2018, and the 2017–March 2020 validation release overlaps those years. The text says 'records used for model selection' were separated, but the cohort definitions are not precise enough to confirm this. That is a minor issue if the separation was clean; the architecture problem is decisive on its own.\n\nWho is this for? A reader interested in the domain-shift results or the calibration methodology might get value from the empirical sections. But this paper should not be cited as an example of how to do leakage-controlled evaluation—it is, unintentionally, a textbook case of the gap between stated intent and implemented architecture.\n\nMy recommendation: a serious editor should send this to peer review rather than desk-reject it. The empirical sweep and the careful reporting give the authors a fair chance to fix the implementation—e.g., explicit per-task input masking or a true shared-encoder design with feature selection before the encoder. If the response is 'the code matches the text,' then reject.","headline":"The paper's headline contribution—leakage-reduced evaluation—is not implemented by the architecture as written: a shared encoder over the full feature vector means label-defining variables reach every head.","tokens_in":12241,"tokens_out":3370,"would_cite":false,"duration_ms":27966,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CardioMeta predicts diabetes, hypertension, and CVD with a shared encoder, gated heads, and post-hoc calibration; its central claim is that calibrated, leakage-controlled probabilities matter more than inflated accuracy.","keywords":["multi-task learning","cardiometabolic prediction","label leakage","probability calibration","NHANES","MIMIC-IV","tabular deep learning","clinical risk prediction"],"falsifier":"Train CardioMeta with HbA1c, fasting glucose, and diabetes medication removed from the shared encoder's input entirely (not just from the diabetes head) and compare the diabetes AUROC to the reported 0.854; if it drops substantially or ECE worsens, the reported leakage-reduced performance depends on features the protocol claims to have excluded.","tokens_in":11313,"feed_emoji":"🫀","tokens_out":6090,"duration_ms":44389,"temperature":0.7,"pith_summary":"CardioMeta is a multi-task neural model that jointly predicts diabetes, hypertension, and cardiovascular disease from population-survey and EHR data. The paper's central claim is that honest cardiometabolic screening models should be evaluated under a leakage-reduced feature setting, where variables that directly define a disease label are excluded, and that the model's main contribution is calibrated probabilities and transparent reliability reporting rather than large accuracy gains. In temporal validation on NHANES, CardioMeta achieves macro-AUROC 0.839, macro-F1 0.614, and expected calibration error 0.024, with modest but consistent improvements over gradient-boosting and neural tabular baselines. External transfer to MIMIC-IV degrades substantially, but limited fine-tuning recovers much of the gap. The paper argues that reporting calibration, subgroup reliability, and leakage control is at least as important as reporting discrimination.","feed_headline":"One model handles diabetes, hypertension, and CVD with calibrated odds","feed_subtitle":"It reaches 0.839 AUROC with a 0.024 calibration error on out-of-time data, and shows how label leakage inflates results.","key_machinery":"The central object is a multi-task architecture combining a shared cardiometabolic encoder (a multilayer perceptron with batch normalization, GELU activation, dropout, and residual projection) with disease-specific gated heads and post-hoc probability calibration. Each head applies a feature-wise gate to the shared representation before producing a disease probability. The leakage-reduced protocol excludes label-defining variables—HbA1c, fasting glucose, diabetes medication, blood pressure, antihypertensive medication, and direct CVD history—from the corresponding prediction head, while a full-clinical setting is kept as sensitivity analysis. This two-setting design is the machinery that car","core_discovery":"The paper's central claim is that cardiometabolic disease-status prediction can be made credible if the evaluation protocol explicitly separates a leakage-reduced diagnosis setting from a full-clinical sensitivity setting. On temporally separated NHANES data, CardioMeta achieves macro-AUROC 0.839, macro-AUPRC 0.536, macro-F1 0.614, and expected calibration error 0.024, with the calibration advantage being the most stable result over strong tabular baselines. External MIMIC-IV evaluation shows substantial degradation under domain shift, with partial recovery after fine-tuning. The paper argues that the principal value of multi-task cardiometabolic modeling lies not in inflated accuracy but in","pith_inferences":["If the shared-encoder leakage concern is real, the reported 0.839 AUROC may partly reflect circular-diagnosis artifacts; a clean test would exclude label-defining features from the entire architecture, not just from the prediction head.","The multi-task structure could be made leakage-proof by enforcing head-specific feature masks inside the shared encoder, turning the paper's exclusion design into a hard architectural constraint.","The gap between full-clinical and leakage-reduced performance (AUROC 0.899 vs 0.839) offers a natural, dataset-agnostic 'leakage index' that other chronic-disease studies could report to flag circular predictors.","A testable extension is to apply the same two-setting protocol to incident disease endpoints—for example, future diabetes diagnosis from earlier survey waves—to see whether leakage-reduced signals generalize to true forecasting rather than status identification."],"forward_implications":["Future cardiometabolic prediction studies can benchmark against a leakage-controlled multi-task target rather than inflated full-clinical numbers.","Calibrated probabilities support threshold-based screening decisions, since decision-curve analysis shows higher net benefit than treat-all or CatBoost at low thresholds.","Subgroup reporting reveals that older and lower-income patients have worse calibration and higher CVD false-negative rates, so recalibration or threshold adjustment is needed before deployment in those groups.","The degradation on MIMIC-IV direct transfer indicates that population-survey models should not be deployed on hospital EHR populations without local fine-tuning.","The full-clinical sensitivity analysis quantifies how much apparent accuracy is driven by circular evidence, making the leakage-reduced gap an interpretable diagnostic for future studies."],"fun_headline_variants":["Calibrated multi-task model tackles diabetes, hypertension, and CVD","Leakage control and calibration: CardioMeta's real win over baselines","One model, three diseases: CardioMeta keeps probabilities honest","0.024 calibration error on NHANES—CardioMeta shows how to avoid leakage","From NHANES to MIMIC-IV: CardioMeta exposes domain-shift degradation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The leakage-reduced setting actually keeps label-defining variables out of the model's predictive path: the paper removes HbA1c, glucose, diabetes medication, blood pressure, and antihypertensive medication from the corresponding prediction head, but because one shared encoder feeds all three heads, those features can still influence the excluded head through the shared representation.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated multi-task model tackles diabetes, hypertension, and CVD","Leakage control and calibration: CardioMeta's real win over baselines","One model, three diseases: CardioMeta keeps probabilities honest","0.024 calibration error on NHANES—CardioMeta shows how to avoid leakage","From NHANES to MIMIC-IV: CardioMeta exposes domain-shift degradation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1329,"prompt_tokens":846,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":590,"tokens_out":483,"duration_ms":4563,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:29:19.305058+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CardioMeta with HbA1c, fasting glucose, and diabetes medication removed from the shared encoder's input entirely (not just from the diabetes head) and compare the diabetes AUROC to the reported 0.854; if it drops substantially or ECE worsens, the reported leakage-reduced performance depends on features the protocol claims to have excluded.","supporting_citations":[],"review_version":1}