{"id":"9d1f43ef-713f-4456-b4a4-473b76fb1750","arxiv_id":"2507.00205","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An extension of the HAIM multimodal framework that uses LLM-based retrieval and summarization to improve clinical prediction AUC from 79.9% to 90.3% and to generate document-grounded explanations.","lead":"xHAIM adds generative AI to an existing medical AI framework, using LLMs to pull relevant patient information, write short summaries, and produce explanations for each prediction. On an ICU dataset, the authors report raising average classification AUC from 79.9% to 90.3% compared with the original HAIM framework.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 10.4-point AUC gain depends on a HAIM baseline that appears lifted from the original paper; if it was not re-run on the same 5 splits, the headline comparison is invalid.","rationale":"The paper's central quantitative claim is that xHAIM raises average AUC from 79.9% to 90.3% and significantly outperforms HAIM on all five tasks. That claim rests entirely on Table 2's comparison between the new xHAIM rows and the 'HAIM Baseline' row. The baseline values and standard errors are identical to the original HAIM publication, strongly suggesting they were not recomputed on the same splits. Without a matched baseline, the reported deltas are not a controlled comparison, and the paper's stated mechanism ('intelligent data curation') is confounded by simultaneous changes to the embedding model (frozen to fine-tuned) and the image modality (DenseNet features to Qwen2.5-VL text descriptions). This is the single most load-bearing assumption. The xHAIM design is otherwise clearly described, uses open-source models, and includes a human-calibrated LLM judge for the explanation component, which is a genuine strength. However, the missing appendices, absent code, and unreported split indices prevent verification of the main result. The reader's conditional verdict already captures this concern; I agree with it and do not see grounds to strengthen or weaken the verdict without the requested re-run and ablations.","tokens_in":15568,"tokens_out":4239,"duration_ms":46283,"concrete_test":"Reproduce Table 2 by re-running the original HAIM baseline (frozen ClinicalBERT note embeddings and CheXpert-fine-tuned DenseNet121 image features) on the exact 5 split indices used for xHAIM, with the same per-stay scan selection and preprocessing. If the reproduced HAIM AUC values differ from the listed baseline by more than the reported standard errors, the headline comparison is invalid. Additionally, run two ablations to separate the changes: (A) xHAIM's fine-tuned note/image-text embeddings on raw, unfiltered chunks (no retrieval/summarization), and (B) xHAIM's retrieval+summarization pipeline with frozen embeddings. If (A) reproduces most of the gain, the summarization step is not the driver; if (B) does, the gain is not from fine-tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 2, the 'HAIM Baseline' row reports AUC values and standard errors (84.8±0.5, 81.1±0.2, 76.3±0.4, 82.0±0.2, 75.5±0.4) that match the original HAIM paper's published numbers to the reported precision. The manuscript says xHAIM was run on 5 random 80/20 splits but never states that the HAIM baseline was re-run under those same splits, and the code and split indices are not yet available. If the baseline was carried over from the original publication, the comparison may use different train/test splits, different data preprocessing, or different per-stay scan selection than the xHAIM runs. In addition, the xHAIM pipeline changes at least three factors simultaneously: (1) frozen ClinicalBERT embeddings are replaced by fine-tuned embeddings, (2) DenseNet121 image features are replaced by Qwen2.5-VL-72B text descriptions, and (3) raw chunks are replaced by curated, summarized text. Therefore the 79.9→90.3 average AUC and the per-task deltas in Table 2 cannot be attributed to the proposed summarization/curation step unless the baseline is matched and ablations are provided. This is the load-bearing assumption behind the abstract, the Results section, and the Discussion's claim that 'intelligent data curation' drives the gain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces xHAIM, an extension of the HAIM multimodal framework for ICU prediction. The pipeline (1) retrieves task-relevant text chunks using a hybrid BM25/SBERT score, (2) generates patient summaries with LLMs (Llama-3.3-70B, Qwen3-32B, and Qwen2.5-VL-72B for images), (3) fine-tunes ClinicalBERT on the summaries and integrates the resulting embeddings with tabular and time-series features, and (4) generates document-grounded explanations. On the HAIM-MIMIC-MM dataset, the authors report an average AUC improvement from 79.9% (HAIM baseline) to 90.3% (xHAIM-FT-Llama) across pleural effusion, cardiomegaly, pneumonia, 48-hour mortality, and length of stay. They also report manual and LLM-as-a-judge evaluations of explanation quality for two of the five tasks.","tokens_in":15853,"tokens_out":4831,"duration_ms":51011,"significance":"If established, the central claim that intelligent data curation via summarization yields large AUC gains while adding explainability would be an important contribution to medical AI. The paper uses a well-known public dataset, provides a concrete four-step pipeline, and attempts to validate explanations with both human annotators and an LLM judge. The strengths are the hybrid generative/discriminative framing, the use of open-source models, and the detailed description of the preprocessing steps. However, the evidence currently presented does not isolate the contribution of the curation step, and the apparent reuse of the original HAIM baseline undermines the headline comparison. The significance of the claimed 10.4-point gain is real if the comparison is valid, but the manuscript does not yet establish that validity.","major_comments":[{"comment":"The HAIM Baseline row reports AUCs and standard errors (84.8±0.5, 81.1±0.2, 76.3±0.4, 82.0±0.2, 75.5±0.4) that match the original HAIM publication to the reported precision, and the manuscript never states that HAIM was re-run under the same five 80/20 splits used for xHAIM. Since split indices, per-stay scan selection, and preprocessing may differ, this baseline cannot support the claimed 79.9-to-90.3 improvement. The authors must re-run the original HAIM pipeline under the identical protocol and report the resulting AUCs and standard errors, or the headline comparison is not valid.","section":"Results, Table 2"},{"comment":"xHAIM simultaneously changes three components relative to HAIM: (i) ClinicalBERT embeddings are fine-tuned on summaries instead of frozen on raw text, (ii) DenseNet121 image features are replaced by Qwen2.5-VL-72B text descriptions, and (iii) raw chunks are replaced by curated, summarized text. The observed gains cannot be attributed to 'intelligent data curation' without ablations that vary one component at a time. For example, report HAIM with fine-tuned ClinicalBERT on raw, non-summarized text, and report xHAIM without the chunk-retrieval step, to isolate the effect of summarization.","section":"Methods, Multimodal Integration"},{"comment":"The anchor-sentence retrieval injects label-related vocabulary into the summaries: the anchors for pneumonia contain 'pneumonia,' 'consolidation,' and 'infiltrate,' and similar anchors are used for other outcomes. Because the selected chunks are then summarized and embedded by ClinicalBERT, the model's input features contain explicit mentions of the target condition, creating a circularity that can inflate AUC beyond genuine predictive signal. The authors should either use outcome-blind anchors (e.g., general clinical concepts) or demonstrate that classification performance does not rely on this lexical overlap, for instance by masking outcome terms in the retrieved chunks.","section":"Methods, Finding Relevant Chunks"},{"comment":"The explanation-quality evaluation is limited to two conditions (pleural effusion and 48-hour mortality) with N=50 manual annotations per condition and N=1000 LLM-judge evaluations; Tables 3 and 4 do not cover cardiomegaly, pneumonia, or length of stay. The general claim that xHAIM provides clinically meaningful explanations across all tasks is therefore only partially supported. In addition, the text promises that supplementary information includes ablation studies, but no supplementary material is provided with this submission, and the code repository is not yet available, so the promised ablations and split definitions cannot be inspected.","section":"Results, Explainability Evaluation"}],"minor_comments":[{"comment":"The manuscript contains unresolved placeholder references to 'Appendix ??' in Sections 2 and 4; these must be filled in.","section":"General"},{"comment":"No inter-annotator agreement metric is reported for the manual evaluations, so the claim of 'strong alignment with human judgment' by the LLM judge is not quantified.","section":"Results, Tables 3 and 4"},{"comment":"The abstract mentions 'operative tasks,' but the evaluated tasks are chest pathology, mortality, and length of stay; the terminology should be clarified.","section":"Abstract / Results"},{"comment":"The hyperparameter alpha=0.5 in the hybrid score is fixed without sensitivity analysis; the authors should report whether results are robust to this choice.","section":"Methods, Finding Relevant Chunks"},{"comment":"The description says one scan is randomly selected per ICU stay; the original HAIM scan-selection procedure should be stated explicitly so that comparability can be assessed.","section":"Methods, Modalities Preprocessing"},{"comment":"Mortality and length of stay have identical cohort sizes (16,888); if this is due to a shared ground-truth definition, it should be stated in the text.","section":"Results, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The exact match between the reported HAIM baseline and the original HAIM paper's published numbers raises the strong possibility that the baseline was not re-run under the xHAIM protocol. I would ask the editor to require the authors to provide raw result files and split indices during revision, since the central quantitative claim depends entirely on this. The promised supplementary ablations are also absent and should be made available before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result—a 10-point AUC jump from curating and summarizing multimodal ICU data—is real only if the HAIM baseline was actually re-run under the same protocol. On the evidence in the paper, it probably wasn't. Table 2's HAIM row matches the original HAIM paper's AUCs and standard errors to the decimal, and nothing in the methods says the baseline was re-run under the same 80/20 splits, random scan selection, or preprocessing. Since xHAIM changes several inputs at once—fine-tuned ClinicalBERT instead of frozen embeddings, Qwen2.5-VL text descriptions instead of DenseNet image features, and summarized text instead of raw chunks—the 79.9→90.3 claim can't be attributed to \"intelligent data curation\" until those factors are separated. That's load-bearing for the abstract and Discussion.\n\nThe framework itself is a sensible and useful idea. Retrieval + LLM summarization + fine-tuned embeddings + grounded explanations is a coherent way to combine generative models with discriminative classifiers, and the paper is candid that generative models alone still underperform. The explanation examples are genuinely readable, and the manual annotation effort (50 per task, two annotators) is a real attempt to validate the LLM-as-judge. The use of open-weight models makes it reproducible in principle.\n\nSoft spots beyond the baseline: the anchor sentences for chunk retrieval contain task keywords like \"pneumonia,\" so the retrieval step may be injecting label-related text directly into the input. That is a mild circularity concern and should be examined in revision. The appendix is missing (references appear as \"Appendix ??\"), and the code is only \"available upon publication,\" so split indices and prompts can't be checked. Ablations—fine-tuned embeddings without summarization, frozen embeddings with summarization—are essential but absent. Inter-annotator agreement isn't reported, and the N=50 calibration for the LLM judge is small. These are fixable.\n\nMy take: as submitted, the central empirical claim is not established, but the idea is worth taking seriously. This paper is for people working on multimodal clinical prediction, RAG/LLM preprocessing, and explainability. It deserves serious peer review, mainly because the problems are fixable and the payoff—if real—matters for clinical multimodal AI. I'd send it out, but the revision should include a matched baseline and ablations.","headline":"The 10-point AUC gain is likely an artifact of a baseline that was copied from the original HAIM paper rather than re-run; the framework idea is still worth refereeing.","tokens_in":16388,"tokens_out":2404,"would_cite":false,"duration_ms":28128,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative-AI preprocessing layer raises average clinical AUC from 79.9% to 90.3% in the xHAIM framework.","keywords":["explainable AI","multimodal clinical prediction","generative AI","patient summarization","HAIM","clinical decision support","retrieval-augmented prediction","LLM evaluation"],"falsifier":"Re-run the original HAIM pipeline on the same five 80/20 splits used for xHAIM; if its average AUC matches the published 79.9% rather than xHAIM's splits, the reported comparison is not controlled. Separately, feed the fine-tuned ClinicalBERT and Qwen2.5-VL features the full unfiltered notes instead of the generated summaries; if AUC does not fall, summarization is not the mechanism.","tokens_in":15371,"feed_emoji":"🩺","tokens_out":6960,"duration_ms":68101,"temperature":0.7,"pith_summary":"This paper sets out to establish that a generative-AI preprocessing layer can make an existing multimodal clinical prediction framework both more accurate and more explainable. Its proposed system, xHAIM, first selects patient-record chunks most relevant to a given task, summarizes them with an open-source LLM, then feeds the summaries through a fine-tuned ClinicalBERT into the HAIM predictor. On the public HAIM-MIMIC-MM dataset, the authors report that average AUC rises from 79.9% to 90.3% across chest pathology detection, 48-hour mortality, and length-of-stay prediction, with the largest gains in pathologies whose detection depends on clinical narrative. They also report that the generated explanations, which cite specific patient documents, score comparably under human and LLM-based evaluation. If correct, this would mean that filtering and summarizing clinical data before prediction is more effective than feeding models unfiltered multimodal records.","feed_headline":"xHAIM lifts clinical AI accuracy from 79.9% to 90.3%","feed_subtitle":"Summarizing patient records before prediction beats unfiltered multimodal models and adds cited, doctor-checkable explanations.","key_machinery":"The load-bearing object is the four-step xHAIM pipeline: (1) retrieval of task-relevant chunks via a hybrid score $\\text{Score}_{\\text{hybrid}} = \\alpha\\cdot\\text{BM25}_{\\text{normalized}} + (1-\\alpha)\\cdot\\text{SBERT}_{\\text{sim}}$ with $\\alpha=0.5$; (2) LLM-generated task-specific summaries of the selected chunks, using Llama-3.3-70B or Qwen3-32B, with chest X-ray images converted to text descriptions by Qwen2.5-VL; (3) per-modality fine-tuned ClinicalBERT embeddings concatenated with tabular and time-series features into $X = [x_{\\text{notes summary}}, x_{\\text{cxr summary}}, x_{\\text{tabular}}, x_{\\text{time series}}]$ and fed to XGBoost; and (4) explanation generation that grounds each prediction in patient summaries, calibrated outputs, and medical knowledge with document citations. The key mechanism claimed is that filtering and summarizing removes noise from long clinical notes, allowing fine-tuning to concentrate on signal.","core_discovery":"xHAIM outperforms the HAIM baseline on all five tasks it evaluates. On the HAIM-MIMIC-MM dataset, average ROC AUC goes from 79.9% with HAIM to 90.3% with the best xHAIM variant, with per-task improvements of +13.5% for pleural effusion, +16.3% for cardiomegaly, +19.4% for pneumonia, +2.7% for 48-hour mortality, and +1.9% for length of stay. The paper attributes the gain to replacing noisy, averaged embeddings of full patient records with concise task-specific summaries produced by generative LLMs, and to fine-tuning ClinicalBERT on those summaries. It further claims that the same summaries, combined with the calibrated prediction and relevant medical knowledge, yield explanations that cite the exact documents supporting each prediction, and that an LLM-as-a-Judge evaluation of these explanations aligns with human annotation.","pith_inferences":["Editorial inference: the paper does not isolate the summarization step from the other changes in the pipeline, so the +10.4 average AUC should be read as the joint effect of fine-tuned ClinicalBERT, Qwen2.5-VL image descriptions, and curated summaries; an ablation replacing summaries with full notes would separate these.","Editorial inference: if the baseline HAIM numbers were carried over from the original publication rather than re-run on the same splits, the comparison is weaker than the paper presents; re-running HAIM under the xHAIM protocol would settle the magnitude of the gain.","Editorial inference: the explanation evaluation was calibrated on 50 examples per task and then scaled to 1,000; a natural next test is whether LLM-judge agreement with clinicians degrades on rare or atypical presentations.","Editorial inference: the same retrieval-summarize-predict-explain recipe could be carried to other clinical settings, such as outpatient notes or longitudinal EHR data, where the 'more data is better' assumption is even more costly."],"forward_implications":["If xHAIM's results hold, generative summarization becomes a drop-in front end for existing discriminative clinical predictors, not a replacement for them.","The largest AUC gains appear on narrative-dependent chest pathology tasks, implying that data curation matters most where clinical notes carry the signal.","Cited, document-grounded explanations can be produced automatically at scale, with LLM judge scores comparable to human annotations, shortening verification time for clinicians.","Because the pipeline can run on open-source models inside a hospital, the performance and explainability gains do not require sending patient data to external APIs.","Summaries that standardize free text may make the downstream model more robust to differences in note-writing style across institutions."],"supporting_citations":[{"why":"The baseline HAIM framework that xHAIM extends and compares against.","marker":"[25]"},{"why":"Supplies the HAIM-MIMIC-MM dataset and task definitions used for all experiments.","marker":"[17]"},{"why":"ClinicalBERT that xHAIM fine-tunes to embed the generated summaries.","marker":"[23]"},{"why":"Qwen2.5-VL converts chest X-ray images into text descriptions for summarization.","marker":"[41]"},{"why":"BM25 component of the hybrid retrieval score that selects relevant chunks.","marker":"[42]"},{"why":"SBERT component of the hybrid retrieval score that measures semantic similarity.","marker":"[43]"},{"why":"Llama-3.3-70B, one of the open-source LLMs that generate task-specific summaries.","marker":"[44]"},{"why":"Qwen3-32B, the other summarization LLM tested in the pipeline.","marker":"[45]"},{"why":"Evidence that generative-only models underperform traditional ML on clinical prediction, motivating the hybrid design.","marker":"[29]"}],"fun_headline_variants":["xHAIM: explainable AI, AUC up to 90.3%","xHAIM: +10.4% AUC, plus explainable predictions","Explainable AI lifts clinical AUC by 10.4 points","Summaries boost AI predictions, add cited explanations","xHAIM: from 79.9% to 90.3% AUC, with explainability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes the HAIM baseline was evaluated under the same five-split protocol and that the entire improvement comes from the summarization step, not from the simultaneous switch to fine-tuned ClinicalBERT embeddings and Qwen2.5-VL image descriptions.","fun_headline_variants_meta":{"raw":{"variants":["xHAIM: explainable AI, AUC up to 90.3%","xHAIM: +10.4% AUC, plus explainable predictions","Explainable AI lifts clinical AUC by 10.4 points","Summaries boost AI predictions, add cited explanations","xHAIM: from 79.9% to 90.3% AUC, with explainability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000859,"raw_usage":{"total_tokens":3714,"prompt_tokens":919,"completion_tokens":2795,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":2696}},"tokens_in":535,"tokens_out":2795,"duration_ms":23067,"temperature":1.0,"reasoning_tokens":2696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:21:39.917732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the original HAIM pipeline on the same five 80/20 splits used for xHAIM; if its average AUC matches the published 79.9% rather than xHAIM's splits, the reported comparison is not controlled. Separately, feed the fine-tuned ClinicalBERT and Qwen2.5-VL features the full unfiltered notes instead of the generated summaries; if AUC does not fall, summarization is not the mechanism.","supporting_citations":[{"cited_title":"PhysioNet (2022)","cited_arxiv_id":null,"evidence_quote":"Supplies the HAIM-MIMIC-MM dataset and task definitions used for all experiments."},{"cited_title":"Foundations and Trends ® in Information Retrieval 3(4), 333–389 (2009)","cited_arxiv_id":null,"evidence_quote":"BM25 component of the hybrid retrieval score that selects relevant chunks."},{"cited_title":"medRxiv (2024)","cited_arxiv_id":null,"evidence_quote":"Evidence that generative-only models underperform traditional ML on clinical prediction, motivating the hybrid design."}],"review_version":1}