{"id":"3bfdf39b-5b97-498b-91db-6e03bebc0ae5","arxiv_id":"2411.16818","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"LLM-generated expert summaries of clinical notes, fused with vital signs and raw notes in an LSTM plus ClinicalBERT model, improve MIMIC-III in-hospital mortality prediction over a time-series-only baseline.","lead":"This paper combines hourly ICU vital signs, clinical notes, and LLM-written patient summaries to predict in-hospital death. The best model beats a vital-signs-only baseline, but the gain over using raw notes is small and the evaluation has no confidence intervals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that Med42 summaries add complementary signal rests on unverified faithfulness; hallucinated or shortcut-laden summaries could explain the small AUROC/AUPRC gain.","rationale":"The reader identified hallucination and faithfulness as the weakest assumption, and I agree: it is the load-bearing condition for the central claim that LLM-generated expert summaries add complementary predictive signal. The manuscript itself flags this concern in the Limitations section, which makes it a self-acknowledged gap rather than an external objection. My proposed NLI/clinician review of summary faithfulness directly tests the mechanism, whereas the other possible concerns (no error bars, small demographic strata, missing code) are important but secondary reproducibility issues that would not, by themselves, invalidate the causal story. Since the concern is addressable and the reader's conditional verdict already requires stronger evidence, I would not change the disposition; the verdict remains conditional pending the faithfulness check.","tokens_in":8095,"tokens_out":4603,"duration_ms":47652,"concrete_test":"Evaluate faithfulness on a random sample of 200 test split patients: use a validated medical NLI model (or two blinded clinicians) to judge whether each sentence in the Med42v2-70B summary is entailed by the corresponding concatenated notes. Report the rate of unsupported or contradictory statements and whether that rate differs between survivors and non-survivors. If more than 10% of summaries contain outcome-relevant hallucinations, or if the model's performance gain disappears when such summaries are excluded, the claim that faithful expert summaries drive the improvement is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EBCRs from Med42v2-70B summaries add predictive signal beyond raw notes and vitals. The paper's own Implementation Details state that Med42-8B outputs were 'so pervasive' in hallucinations that they were unreliable, and Limitations and Future Work concedes that 'any biases or errors in the LLM-generated summaries could impact model performance.' Yet no quantitative faithfulness check is reported for the 70B summaries used in the experiments. The incremental gain of the multi-representational model over the joint model with clinical notes only is small (AUROC +1.15%, AUPRC +1.65%), and it could arise from a less benign mechanism than medical insight: the LLM may restate already-present mortality proxies (DNR orders, comfort-care plans, metastatic disease) as a compressed global vector that ClinicalBERT can separate more easily. Since Vt is a single summary embedding applied to every time step, while note embeddings are time-decayed, the summary can aggregate outcome-relevant cues over the full 48-hour window in a way the note stream cannot. If the summaries contain unsupported statements, the measured gain is an artifact of the LLM's prior or of shortcut features, and will not generalize across hospitals or note styles. This is the load-bearing assumption: without a faithfulness/control evaluation, the strongest claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-representational learning framework for in-hospital mortality prediction from MIMIC-III ICU data, integrating (i) 48-hour physiological time series encoded by an LSTM, (ii) clinical notes encoded by ClinicalBERT with exponential time decay, and (iii) LLM-generated 'expert summaries' (EBCRs) produced by Med42-v2 70B from the concatenated notes and encoded as a single ClinicalBERT embedding. The authors report that the joint model achieves AUROC 0.8955 and AUPRC 0.6156, outperforming time-series-only and notes-plus-time-series baselines, and they present subgroup analyses across racial groups. The central claim is that LLM-generated summaries add complementary predictive signal beyond raw notes and vitals without relying on direct LLM predictions.","tokens_in":8310,"tokens_out":3460,"duration_ms":55215,"significance":"If the central claim is established, the work would provide a practical template for using open-source, locally hosted medical LLMs to distill clinical notes into predictions, which is relevant given data-use restrictions on cloud LLMs for MIMIC. The authors deserve credit for being transparent about the hallucination problems of Med42-8B, for selecting Med42-v2 70B for that reason, and for framing EBCRs as auxiliary representations rather than as direct LLM predictions. The main scientific value depends on two things that are not yet demonstrated: the statistical robustness of the reported performance differences and the faithfulness of the summaries actually used. The reported improvements are plausible but, on the current evidence, not conclusively attributable to the summaries' medical content.","major_comments":[{"comment":"The central claim that expert summaries add complementary signal rests on a small absolute improvement: AUROC +1.15% and AUPRC +1.65% over the time-series-plus-notes model. The paper reports a single random split with no confidence intervals, no standard deviations, and no significance tests for any of the metric differences in Table 2. Given that the differences are small and the baselines already incorporate substantial textual information, the stated improvement cannot be distinguished from random split variation. Please report metrics across multiple seeds or bootstrap resamples, with CIs, and a paired significance test (e.g., DeLong for AUROC, bootstrap for AUPRC) for the comparison that supports the paper's main claim.","section":"Results, Table 2"},{"comment":"The paper's own text states that Med42-8B outputs had hallucinations 'so pervasive that the outputs were often unreliable' and concedes that 'any biases or errors in the LLM-generated summaries could impact model performance.' Yet the 70B summaries used in the experiments receive no faithfulness evaluation, manual or automated. Because the EBCR gain could arise from hallucinated or shortcut features rather than from faithful medical distillation, the load-bearing assumption is unverified. Please add (a) a quantitative faithfulness check on a sample of 70B summaries (e.g., clinical-entity overlap, manual review with inter-annotator agreement), and (b) a control experiment in which the 'expert summary' is replaced by an extractive or non-expert summarizer to see whether the gain is specific to expert medical knowledge.","section":"Implementation Details; Limitations and Future Work"},{"comment":"The summary embedding Vt is defined as a single FeatureExtractor(Nexpert) applied to the whole 48-hour summary and then concatenated with Ht and Ut at every time step in hconcat. In contrast, the note representation Ut uses an exponential time decay, so the summary stream can pool information from the entire 48-hour window and inject it at every LSTM time step, giving it a structural advantage unrelated to domain knowledge. This asymmetry could explain part of the observed gain. Please clarify whether Vt is time-invariant in the implementation, and if so, run an ablation that either applies the same whole-sequence pooling to the raw notes (e.g., a time-independent note embedding) or restricts the summary embedding to the final time step, so that the comparison is fair.","section":"LLM-Generated Summary Embedding; Joint Learning, Eq. (7) and Eq. (9)"},{"comment":"The subgroup analysis in Table 3 is used to claim that improvements 'were consistent across demographic groups' and that the framework has 'equitable application potential.' However, the full cohort sizes for Hispanic/Latino (456), Asian (362), and Declined to Answer (166) are small, and the test subset is a 20% split, yielding very few positive cases for mortality; the reported AUPRC values of 0.92 for Asian and 0.50 for Hispanic/Latino are therefore unstable. No confidence intervals or event counts are provided. Please report the number of patients and mortality events per subgroup, or aggregate small groups, and add uncertainty estimates before making equity claims.","section":"Results, Table 3"}],"minor_comments":[{"comment":"The abstract states improvements of +36.41% AUPRC and +7.64% AUROC over the time-series-only baseline. These are relative improvements; the absolute AUPRC gain is 0.1643 and the absolute AUROC gain is 0.0635. Please state both or clarify that the percentages are relative, as the current phrasing can mislead.","section":"Abstract and Results"},{"comment":"Table 1 reports 'Median Age [IQR] 66.96 [18.08, 90.0]'. The bracket appears to be a min-max range rather than an interquartile range, which typically spans the 25th to 75th percentile. Please verify the reported values or relabel the column.","section":"Materials and Methods, Table 1"},{"comment":"There is a typo in the section heading 'T extual Representations Embedding'; it should be 'Textual Representations Embedding'. Also, in the baseline equation, 'andbm' should be 'and b m'.","section":"T extual Representations Embedding"},{"comment":"The prompt used to generate the expert summaries is said to be in the supplementary materials, but no supplementary materials are included in the submission. Please include the exact prompt template, including any instructions about format and length, because summary quality and faithfulness depend critically on the prompt.","section":"Implementation Details"},{"comment":"The paper states code will be released 'upon the paper's acceptance'. For reproducibility of the preprocessing, prompt, and training pipeline, please make the code available at least in a public repository at revision time, or justify why it cannot be shared.","section":"Data Availability"},{"comment":"Figure 2's t-SNE plots are qualitative and not quantified. If they are used to support the claim that EBCRs are more separable, consider reporting a quantitative cluster-separation metric (e.g., silhouette score) computed on held-out data rather than on the same data used to fit the representations.","section":"Discussion, t-SNE visualizations"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and addresses a timely question. The four major comments are all fixable with additional experiments and analysis within the manuscript's scope: multi-split or bootstrap evaluation, a faithfulness/control study for the 70B summaries, a fair temporal-alignment ablation, and statistically grounded subgroup reporting. I do not see a load-bearing error that would require rejection, but the current evidence is not strong enough for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: this is a competent, clearly written empirical paper that tests an obvious but under-explored idea—using an LLM to compress the first 48 hours of ICU notes into a single summary and feeding that embedding alongside raw notes and vitals into a mortality classifier. The result is a modest AUROC/AUPRC gain over the joint notes-plus-vitals model (about +1.1/+1.7 points). The paper is honest about several limitations and does not overclaim.\n\nWhat's actually new: applying Med42-v2 70B as a summary generator for MIMIC-III IHM prediction, with the summary embedded via ClinicalBERT and concatenated at every time step. The ablations are useful: summaries alone beat raw notes alone by a considerable margin (AUROC 0.887 vs 0.849), which is the most interesting finding and deserves scrutiny. The architecture is standard, and the authors correctly avoid direct LLM prediction, using summaries as auxiliary features instead.\n\nThe soft spots are real but mostly addressable. No confidence intervals, no repeated splits, no significance testing; a single random 60/20/20 split carries the entire quantitative claim. The subgroup analysis in Table 3 has very small mortality counts—Asian and Hispanic/Latino test sets are tiny—so those AUPRC numbers (0.50, 0.92) should not be read as evidence of equity. The faithfulness of the 70B summaries is never checked, and the paper notes that the 8B model hallucinated badly; if the 70B summaries encode spurious correlations (e.g., restating DNR orders or comfort-care plans in a compressed form that the model latches onto), the small gain would not generalize. The code and prompt are not released, which makes the 'no cloud LLM due to MIMIC agreement' point hard to verify.\n\nNone of this makes the paper a waste of time. The direction is sound and the negative result (if it is one) would be useful. But the central claim—that summaries add complementary signal beyond raw notes—needs a faithfulness/control evaluation (e.g., parse summaries for unsupported statements, compare against shuffled or corrupted summaries) and at least a paired bootstrap across multiple seeds. The improvement over the joint baseline is small enough that I would want to see it hold under that scrutiny.\n\nMy verdict: this deserves a serious referee, because the idea is timely and the execution is clean enough that the flaws are fixable. I would not cite it as-is, and I'd bring it to a reading group mainly to discuss the shortcut risk.","headline":"A clean, small-scale empirical study: LLM summaries add a few points to ICU mortality prediction, but the gain is fragile without faithfulness checks and error bars.","tokens_in":8879,"tokens_out":2418,"would_cite":false,"duration_ms":21580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fusing LLM-generated expert summaries with vital-sign time series and raw clinical notes improves in-hospital mortality prediction, reaching an AUROC of 0.8955 and an AUPRC of 0.6156.","keywords":["in-hospital mortality prediction","multi-representational learning","large language models","clinical notes","ICU","expert summaries","joint fusion","electronic health records"],"falsifier":"Generate expert summaries with a deliberately corrupted or hallucination-prone LLM on the same cohort; if the joint-fusion model still shows the same AUPRC gains, the summaries' specific content is not the active ingredient. More directly, retrain the pipeline on an external ICU dataset from another institution; the central claim would be falsified if the AUPRC gain from adding summaries does not reproduce outside the original cohort.","tokens_in":7863,"feed_emoji":"🏥","tokens_out":5094,"duration_ms":43881,"temperature":0.7,"pith_summary":"This paper tries to show that in-hospital mortality prediction for ICU patients improves when a large language model's distilled 'expert summaries' of clinical notes are added as a third representation alongside vital-sign time series and raw note text. The authors argue that the summaries carry complementary signal that neither structured physiology nor raw notes alone provide. If right, the practical route to using LLMs in high-stakes prediction is not to ask them for a final risk score but to use them as a feature-engineering layer whose outputs feed a conventional, interpretable classifier. On a large public ICU cohort of 15,337 patients, the joint model reaches an AUROC of 0.8955 and an AUPRC of 0.6156, outperforming all single-modality baselines. The paper is explicit that gains depend on summary quality and need external validation.","feed_headline":"LLM summaries lift ICU mortality prediction 36% in AUPRC","feed_subtitle":"Fusing vitals, raw notes, and LLM expert opinions pushes AUROC to 0.8955 on 15,337 ICU patients.","key_machinery":"The load-bearing mechanism is the Expert-Based Clinical Representation (EBCR): each patient's notes are concatenated in chronological order, an LLM is prompted to produce a structured expert summary, and that summary is embedded with a clinically pretrained transformer. Raw-note embeddings are aggregated per hour with an exponential decay weighting by chart time, and the summary embedding is fused with the LSTM's final time-series hidden state by concatenation before a sigmoid output layer. The design deliberately keeps the LLM out of the prediction path, using it only to produce a distilled textual input.","core_discovery":"The central claim is that transforming a patient's chronologically concatenated 48-hour clinical notes into an expert summary with a medical LLM, then embedding that summary and fusing it with time-series physiology and note embeddings, yields the best in-hospital mortality prediction among the configurations tested. The authors report that the expert-summary-only representation beats both raw notes and time-series alone, and that adding the summary to the joint fusion improves AUROC by 1.15% and AUPRC by 1.65% over the same fusion without it. Performance gains are consistent across racial and ethnic subgroups in the test set. The authors frame the result as evidence that LLM-generated expert summaries, introduced as auxiliary representations rather than as direct predictors, capture complementary prognostic information and mitigate the uncertainty-quantification and interpretability problems of end-to-end LLM prediction.","pith_inferences":["We infer that a direct testable extension would swap the summary generator for a smaller or general-purpose LLM; if the gains shrink or vanish, the specific medical pretraining, not summarization per se, carries the value.","Because the summaries are generated from the full 48-hour window, the design is not yet a real-time early-warning system; adapting the LLM to generate summaries from a sliding partial window could extend the approach to continuous monitoring.","We infer that if summary embeddings capture the same signal as raw notes, the small AUROC gain over the note-fused model may be partly due to the LLM implicitly normalizing style or noise; an ablated version using a deterministic extractive summarizer would separate extraction from generation."],"forward_implications":["If replicated, LLM-generated summaries can be treated as a cheap auxiliary modality that improves mortality-risk stratification over physiology-only and note-only models.","Because the summary alone outperforms raw notes alone, distillation is doing work beyond simple compression of the note text.","The fusion gains mean that combining modalities is more useful than relying on any single representation, and simple concatenation is sufficient to realize most of the gain.","Consistent gains across demographic subgroups suggest the auxiliary summaries do not amplify baseline disparities in this cohort."],"supporting_citations":[{"why":"Supplies the clinically pretrained transformer feature extractor used to embed notes and summaries.","marker":"[1]"},{"why":"Provides the open-source medical LLM used to generate expert summaries.","marker":"[6]"},{"why":"Justifies AUPRC as the primary metric for imbalanced mortality prediction.","marker":"[7]"},{"why":"Defines the 48-hour cohort and the ten-variable time-series preprocessing the study follows.","marker":"[10]"},{"why":"Provides the LSTM architecture used as the time-series baseline and fusion backbone.","marker":"[11]"},{"why":"Is the critical care database supplying the 15,337-patient cohort.","marker":"[12]"},{"why":"Explains the data-use restriction that leads the authors to a local open-source LLM rather than a cloud API.","marker":"[16]"}],"fun_headline_variants":["LLM expert summaries boost ICU mortality prediction by 36%","Medical LLM summaries sharpen ICU death prediction AUPRC 36%","Fusing LLM summaries with vitals lifts ICU mortality AUPRC 36%","Med42-v2 summaries raise ICU mortality AUPRC 36% over vitals only","LLM-generated expert notes improve ICU mortality AUPRC by a third"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the medical LLM's expert summaries faithfully condense the clinical notes; if the summaries introduce hallucinations or spurious patterns that happen to correlate with mortality, the reported gains will not transfer to new data.","fun_headline_variants_meta":{"raw":{"variants":["LLM expert summaries boost ICU mortality prediction by 36%","Medical LLM summaries sharpen ICU death prediction AUPRC 36%","Fusing LLM summaries with vitals lifts ICU mortality AUPRC 36%","Med42-v2 summaries raise ICU mortality AUPRC 36% over vitals only","LLM-generated expert notes improve ICU mortality AUPRC by a third"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000512,"raw_usage":{"total_tokens":2514,"prompt_tokens":996,"completion_tokens":1518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":1417}},"tokens_in":612,"tokens_out":1518,"duration_ms":11353,"temperature":1.0,"reasoning_tokens":1417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:59:00.382054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate expert summaries with a deliberately corrupted or hallucination-prone LLM on the same cohort; if the joint-fusion model still shows the same AUPRC gains, the summaries' specific content is not the active ingredient. More directly, retrain the pipeline on an external ICU dataset from another institution; the central claim would be falsified if the AUPRC gain from adding summaries does not reproduce outside the original cohort.","supporting_citations":[{"cited_title":"The relationship between precision-recall and roc curves","cited_arxiv_id":null,"evidence_quote":"Justifies AUPRC as the primary metric for imbalanced mortality prediction."},{"cited_title":"Mimic-iii, a freely accessible critical care database","cited_arxiv_id":null,"evidence_quote":"Is the critical care database supplying the 15,337-patient cohort."},{"cited_title":"Gpt responsible use","cited_arxiv_id":null,"evidence_quote":"Explains the data-use restriction that leads the authors to a local open-source LLM rather than a cloud API."}],"review_version":1}