{"id":"088217e9-6b0e-494f-bd55-0dd573b3b752","arxiv_id":"2607.18270","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A severity-aware knowledge-graph retrieval-augmented LLM pipeline reports large gains in mortality and readmission prediction on MIMIC-III/IV.","lead":"TRACER predicts whether hospital patients will die or be readmitted by combining severity-weighted disease-progression paths from a medical knowledge graph with the patient's clinical notes and similar past cases, all fed to a large language model. It reports large gains over prior models on MIMIC-III/IV, but the comparison rests on a disputed re-labeling of the data that needs independent verification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cohort statistics contradict stated two-visit minimum, so the label-corrected baseline comparison is not reproducible as described.","rationale":"The reader's weakest assumption is that Appendix A.1's label-correction may not have been applied identically to all baselines. I agree this is the critical area, but I think the more concrete and falsifiable problem is the cohort contradiction: Table 10's mean visit counts (<2) directly contradict the explicit exclusion of single-visit patients, independent of whether the label redefinition is correct. This contradiction makes the fairness of the baseline comparison unverifiable from the manuscript and could explain the headline gains if single-visit patients were handled differently across models. I also note the α/β and Table 1-vs-Appendix-E discrepancies, which reinforce that the reported TRACER numbers are not tied to one reproducible configuration. These concerns stay within the reader's conditional verdict: the paper should be accepted only after the released code/cohort confirms that all methods use the same filtered cohort and the exact hyperparameters. Thus I do not move the verdict; I reaffirm CONDITIONAL with additional concrete checks.","tokens_in":28490,"tokens_out":7791,"duration_ms":83730,"concrete_test":"Run the released TRACER preprocessing code on MIMIC-III and count patients by number of visits in each generated split. Check (a) whether any split contains patients with fewer than two visits, (b) whether mean visits match Table 10, and (c) whether mortality/readmission positive rates match A.1's corrected values (10.83%, 21.60%). Also rerun TRACER with α=0.8/β=0.2 and α=0.7/β=0.3 and check which setting reproduces the Table 1 numbers. If the cohort includes single-visit patients or neither configuration yields 0.8197/0.7128, the exclusion claim is false and/or the reported results are not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim depends on a fair, correctly preprocessed comparison against KARE/GraphCare/etc. The paper's own Appendix C.2 states that only patients with at least two visits are retained, and §4.3.1 repeats that single-visit patients are excluded. Yet Table 10 lists mean visits per patient of 1.86 (MIMIC-III mortality, train), 1.77 (test), and 1.84/2.11 across readmission splits—impossible if every patient has ≥2 visits. Either single-visit patients are in fact included, which contradicts the core trajectory mechanism and the exclusion footnote, or the reported dataset statistics are incorrect. Appendix A.1's label correction (e.g., MIMIC-III readmission positives 54.82%→21.60%) may be valid, but the cohort composition is the linchpin: if TRACER's cohort differs from the baselines' cohort, all reported gains are confounded. This is an internal inconsistency, not a matter of disagreeing with the community. Additionally, Table 1's TRACER row (0.8197/0.7128) differs from Appendix E's 'Ours' (0.8284/0.6831), and §5.1 sets α=0.8/β=0.2 while Appendix J reports optimal α=0.7/β=0.3. The headline 28.5% gain is not tied to a uniquely specified configuration, so the central claim cannot be verified from the text alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRACER, a retrieval-augmented LLM framework for in-hospital mortality and 15-day readmission prediction from EHRs. It constructs a severity-weighted medical knowledge graph (SMKG) by having an LLM assign severity scores to diagnoses using PubMed/Wikipedia passages; retrieves risk and protective 'trajectories' as severity-weighted paths across consecutive visits; augments the prompt with retrieved clinical-note passages and similar-patient contexts; and uses GPT-4o-mini as the backbone LLM for prediction. On MIMIC-III and MIMIC-IV, the paper reports large gains over state-of-the-art baselines, including up to 28.5% relative Macro F1 improvement for mortality and 19.7% for readmission, with extensive ablations and robustness analyses.","tokens_in":28927,"tokens_out":6453,"duration_ms":66562,"significance":"If the empirical claims hold, TRACER would be a substantive contribution: it directly addresses EHR data sparsity via severity-grounded trajectories, makes productive use of clinical notes, and provides path-level interpretability. Strengths include the public code release (stated in Resource Availability), the transfer experiment in which replacing UMLS with SMKG improves MedGraphRAG on three medical QA benchmarks, and a broad ablation/robustness suite. However, the paper as written contains internal numeric inconsistencies in the central result tables and cohort statistics, and the headline comparison depends on an externally unverifiable re-labeling of MIMIC outcomes. These are load-bearing issues: until they are resolved, the magnitude of the reported gains cannot be assessed from the manuscript alone.","major_comments":[{"comment":"The stated inclusion criterion is contradicted by the reported dataset statistics. Footnote 4 of §4.3.1 and Appendix C.2 say that only patients with at least two visits are retained and that single-visit patients are excluded from the main cohort. Table 10 reports mean visits per patient of 1.86 (MIMIC-III mortality train), 1.77 (MIMIC-III mortality test), and 1.32 (MIMIC-IV readmission train). A mean below 2 is impossible under that filter. Figure 1 also reports 68.6% single-visit patients in MIMIC-III, and Appendix F groups patients by exact visit count including 1. Since trajectory retrieval is defined only for adjacent visits (t from 2 to T), the cohort composition is essential: if single-visit patients are included, the method and Figure 1 need re-analysis; if excluded, Table 10 is incorrect. Either way, the baseline comparison is confounded unless the exact cohort is specified and","section":"Appendix C.2 / Table 10 / footnote 4"},{"comment":"TRACER's headline configuration is not uniquely specified. Table 1 reports mortality Macro F1 0.8197 and sensitivity 0.7128; Table 11 ('Ours') reports 0.8284 and 0.6831; Table 3 uses 0.8197/0.7128; Tables 13 and Appendix J report 0.8284/0.6831 as the tuned model. Section 5.1 sets α=0.8 and β=0.2, while Appendix J states that the optimal values are α=0.7 and β=0.3; Appendix H further shows that the best severity scale (1–100) gives the 0.8284/0.6831 numbers, whereas §4.2 specifies 1–20. Because the abstract's 28.5% gain is computed from one of these configurations, the paper must state the exact final configuration and report all main tables, ablations, and sensitivity analyses from that same configuration.","section":"Tables 1, 3, 11, 13 and §5.1/Appendix J"},{"comment":"The evaluation rests on a large re-labeling of the benchmark. Appendix A.1 states that PyHealth's preprocessing mis-sorted visits and miscomputed readmission intervals, changing MIMIC-III positive rates from 5.42% to 10.83% (mortality) and 54.82% to 21.60% (readmission). The text asserts that all baselines were rerun, retuned, and that 'all reported baseline results are based on the corrected label-definition protocol,' but no diff, cohort manifest, or preprocessing code is supplied, making this externally unverifiable. A label change of this magnitude changes the predictive task itself. To support the central claim, the authors should (i) release the corrected preprocessing and cohort construction; (ii) state which tables use which label definition; and (iii) also report results under the original PyHealth protocol to separate re-labeling effects from the methodological contribution.","section":"Appendix A.1"},{"comment":"Several quantitative statements are internally inconsistent. §5.2 says 'sensitivity and Macro F1 score improve by 28.5% and 19.7%, respectively'; from Table 1, sensitivity improves from 0.3711 to 0.7128 (about +92% relative), while Macro F1 improves from 0.6375 to 0.8197 (about +28.6%). The abstract's 28.5% and 19.7% appear to be Macro F1 improvements for the two tasks, so the sentence should be corrected. In Table 6, TRACER is described as 'approximately 33% faster' than KARE based on 117s vs 88s, which is a 24.8% reduction; the retrieval-time reduction (92s to 60s) is 34.8%. These numbers should be reconciled.","section":"§5.2 / Table 6"}],"minor_comments":[{"comment":"The severity-score range is specified as 1–20 in §4.2, but Appendix H's best-performing variant uses a 1–100 scale with a '5-level prompt.' Clarify whether the default SMKG uses 1–20 or 1–100, and whether the correlations in Appendix K are computed with the final deployed scores.","section":"Appendix H and §4.2"},{"comment":"The consistency of the 'visit length' grouping should be fixed. Figure 1 and Appendix F group patients by visit count including 1, but the analysis cohort is said to exclude single-visit patients. Define the group structure explicitly after the two-visit filter.","section":"Appendix F / Figure 1"},{"comment":"The dataset statistics show mean 'Visits / Patient' below 2 in several cells. Even after the cohort filtering is clarified, consider reporting the distribution of visit counts rather than only the mean, since the trajectory mechanism applies only to patients with at least one historical visit.","section":"Table 10"},{"comment":"The Cartesian product P2 × ... × PT could be combinatorial for patients with many visits. A sentence on pruning or worst-case size would help reproducibility, especially since patients with up to 10 visits are retained.","section":"§4.3.1"},{"comment":"Figure 5 and Appendix J report the same hyperparameter sweeps but the main text says 'Figure 5 shows' while Appendix J gives the detailed tables. Please cross-check the α/β values in the figure caption and text against Appendix J.","section":"Section 6.4 / Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper has a compelling central idea and a broad experimental package, but the internal numeric inconsistencies and the unverified re-labeling of MIMIC outcomes are currently too serious for acceptance. In my view, a major revision is appropriate: the authors must provide a single, consistent experimental configuration; correct the cohort statistics; and release the exact preprocessing code used for all baselines. If these are resolved, the paper could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the method is a real synthesis. Severity-scored diagnosis nodes in a KG, severity-weighted trajectory retrieval, clinical-note passages, and similar-patient context in one RAG prompt is a combination I haven't seen in the cited prior work. The authors describe the pipeline clearly, give prompts, ablations, and ship code. I also don't see a circularity problem: severity scores come from an LLM reading PubMed/Wikipedia, not from outcome labels, and are used as a retrieval prior only. That part is fine.\n\nThe problem is the empirical section. The paper states twice that single-visit patients are excluded, yet Table 10 lists mean visits per patient around 1.3–1.9 in nearly every split. If those are total visits, the exclusion can't be true. Maybe the table reports only historical context visits (since the last visit is the target), but the caption doesn't say that and Appendix F groups patients by visit count including 1. Either way, the cohort definition needs to be pinned down. If the cohort differs from the baselines' cohort, all gains are confounded.\n\nSecond, there are two TRACER results. Table 1 reports mortality Macro F1/sensitivity of 0.8197/0.7128 with α=0.8, β=0.2. Appendix E, Appendix G, and Table 6 use 0.8284/0.6831, and Appendix J says α=0.7, β=0.3 is optimal. The abstract's 28.5% gain is computed from one of these. The paper never reconciles them. That's a basic reproducibility issue.\n\nThird, Appendix A.1's label correction is a strong claim: PyHealth's sorting by ROW_ID and readmission gap from ADMITTIME only, changing MIMIC-III mortality positives from 5.42% to 10.83%. It might be correct, but it makes the baseline comparison dependent on a re-preprocessing that the authors assert without external verification. Not a defect per se, but combined with the cohort ambiguity it means the headline numbers are not yet trustworthy.\n\nWorth noting: the method is not overfit to labels, hyperparameters were tuned on validation, ablations are extensive, and the paper is transparent about its own changes. So this is not a hollow paper. It's an interesting framework with a messy evaluation.\n\nWho should read it: people working on LLM-based clinical prediction and on EHR benchmark preprocessing. It would be a good reading-group discussion piece, but mainly to talk about evaluation pitfalls.\n\nMy recommendation: send it to peer review. The framework is novel and clearly described; the contradictions are fixable in revision. But the referee needs to demand a single, consistent TRACER configuration and an unambiguous cohort definition before the empirical claims can be accepted.","headline":"TRACER is a genuinely assembled RAG pipeline with a plausible severity-grounded design, but the empirical section has internal inconsistencies (cohort stats vs. stated exclusion, two different TRACER configurations) that make the headline gains unverifiable as written.","tokens_in":29332,"tokens_out":6993,"would_cite":false,"duration_ms":66018,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TRACER claims that augmenting a patient's sparse visit history with severity-weighted knowledge-graph trajectories, filtered clinical-note passages, and similar-patient cases lets an LLM predict mortality and readmission substantially bette","keywords":["clinical risk prediction","electronic health records","mortality prediction","readmission prediction","knowledge graph","severity scoring","retrieval-augmented generation","LLM"],"falsifier":"Recompute both label sets on the MIMIC-III dataset using the original uncorrected preprocessing rules and run TRACER under those labels, or run the strongest baseline pipeline with the paper's corrected labels but no other change. If TRACER's Macro F1 advantage over that baseline collapses to within noise in either condition, the central claim fails.","tokens_in":28432,"feed_emoji":"🩺","tokens_out":5353,"duration_ms":50781,"temperature":0.7,"pith_summary":"TRACER attempts to show that clinical risk prediction improves when a patient's sparse visit history is augmented with severity-weighted paths through a medical knowledge graph, passages from the patient's clinical notes, and trajectories of similar patients, before an LLM reads the assembled context. On two standard critical-care EHR benchmarks, the paper reports large gains over the strongest prior method, up to 28.5% Macro F1 for mortality prediction and 19.7% for readmission prediction. The novel element is a severity-weighted knowledge graph where diagnosis nodes carry LLM-assigned severity scores, and the claim is that this signal helps retrieve clinically meaningful progression paths ('red-flag events') that flat concept lists miss. A sympathetic reader would care because the method also comes with path-level explanations and a public implementation, and because the gains are attributed to the retrieval design rather than to a larger model. The central premise that makes the comparison valid is a correction to how outcome labels were constructed in prior benchmark preprocessing; the paper's own results rest on that correction.","feed_headline":"Severity-aware trajectories lift EHR risk prediction F1 by up to 28.5%","feed_subtitle":"Adding severity-weighted disease paths, clinical notes, and similar-patient cases to an LLM beats prior methods on two standard ICU benchmar","key_machinery":"The load-bearing object is the severity-weighted medical knowledge graph (SMKG): a medical knowledge graph whose diagnosis nodes carry literature-grounded severity scores. Given a patient's visits, trajectory retrieval enumerates shortest paths (up to 3 hops) between concepts in adjacent visits, scores each trajectory with a linear combination of semantic relevance to a risk (or protective) query and the average severity of its diagnoses, and this ranking selects the 'key supporting trajectories' the LLM reasons over. The severity score acts as a retrieval prior, not as a standalone prognostic label.","core_discovery":"The paper claims that modeling a patient's history as a sequence of severity-weighted paths through a medical knowledge graph—rather than as a flat bag of diagnoses—substantially improves mortality and 15-day readmission prediction, and that adding retrieved clinical-note passages and similar-patient trajectories produces further complementary gains. TRACER constructs a severity-weighted medical knowledge graph (SMKG) by having an LLM assign each diagnosis a severity score from 1 to 20, retrieves risk and protective trajectories (shortest paths linking concepts across consecutive visits up to 3 hops), reranks them with maximal marginal relevance and an NLI-based stopping rule, retrieves rele","pith_inferences":["If the label-correction claim is independently verified, several previously published benchmark numbers on these EHR tasks were computed under different outcome definitions and are not directly comparable to newer results.","Because severity scoring is LLM-generated and validated only against a coarse four-level severity framework, a natural extension is a clinical-outcome calibration test on a held-out cohort with hard endpoints.","The same retrieval scaffold (weighted graph + trajectory paths + peer cases + note passages) could be lifted to other sequential clinical tasks such as deterioration prediction, length-of-stay forecasting, or treatment-response prediction.","The paper's use of a conservative decision regulator in the prompts suggests that the 'default to none' prior can be tuned; a testable extension is varying this prior explicitly to measure the sensitivity-precision trade-off rather than leaving it to prompt phrasing."],"forward_implications":["If the claim holds, patients with very few recorded visits can be risk-stratified using paths through a disease graph rather than relying on visit count alone.","Clinical notes can be filtered down to task-relevant passages, countering the noise that long unstructured notes introduce to LLM prompts.","Retrieving similar patients by recent-visit Jaccard overlap is a cheap and effective alternative to sequence-level embeddings for cohort context.","The severity-weighted graph transfers to other downstream tasks: swapping a standard medical terminology graph for the severity-weighted graph improved accuracy on medical question-answering benchmarks in the paper's transfer experiment.","Path-level explanations give clinicians a concrete chain of evidence per prediction, not just a score."],"fun_headline_variants":["Severity-aware trajectories lift risk prediction F1 by 28.5%","TRACER: Severity-weighted paths boost ICU risk F1 by 28.5%","RAG over severity-grounded knowledge graphs improves ICU mortality 28.5%","Trajectory-aware severity paths boost clinical risk F1 by 28.5%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison stands or falls on the claim that prior benchmark preprocessing mislabeled outcomes by sorting visits by an internal row id instead of discharge time and computing readmission gaps from admission time only; the paper's reported gains are all measured under its corrected labels.","fun_headline_variants_meta":{"raw":{"variants":["Severity-aware trajectories lift risk prediction F1 by 28.5%","TRACER: Severity-weighted paths boost ICU risk F1 by 28.5%","RAG over severity-grounded knowledge graphs improves ICU mortality 28.5%","Trajectory-aware severity paths boost clinical risk F1 by 28.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001503,"raw_usage":{"total_tokens":5853,"prompt_tokens":718,"completion_tokens":5135,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":5045}},"tokens_in":462,"tokens_out":5135,"duration_ms":33408,"temperature":1.0,"reasoning_tokens":5045,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:28:46.143108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute both label sets on the MIMIC-III dataset using the original uncorrected preprocessing rules and run TRACER under those labels, or run the strongest baseline pipeline with the paper's corrected labels but no other change. If TRACER's Macro F1 advantage over that baseline collapses to within noise in either condition, the central claim fails.","supporting_citations":[],"review_version":1}