{"id":"e91529af-09bc-4a32-8a23-bb782a575949","arxiv_id":"2507.14824","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A standardized MIMIC-IV benchmark comparing eight unimodal and multimodal foundation models shows multimodal inputs improve predictive performance without adding bias, while medical LVLMs underperform on length-of-stay prediction.","lead":"This paper benchmarks eight foundation models on MIMIC-IV electronic health records, testing them as unimodal encoders and as multimodal learners for in-hospital mortality and length-of-stay prediction. It reports that adding modalities improves predictions without worsening fairness, but that large vision-language models lag behind modular pipelines on length of stay.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'consistent improvements' claim is not testable as reported: Tables II–III do not state whether each modality combination is evaluated on the same patient cohort, and missing CXRs/notes are not handled explicitly; adding image+text sometimes underperforms text alone, so the central claim may…","rationale":"I read the paper in good faith. The benchmark has genuine strengths: public data, code release, bootstrap confidence intervals, and fairness and interpretability analyses. My concern is not that the benchmark is useless, but that the headline claim is under-specified. The most load-bearing uncertainty is not the logistic-regression probe per se; it is whether the comparisons in Tables II and III are on the same patients. The manuscript says 'for each ICU stay, chest X-rays were individually processed,' but it never says what happens when a stay has no CXR or no note. The interpretability section explicitly acknowledges missing-modality sensitivity, and the 'modest decline' in combined image+text is attributed to 'missing modalities or ineffective multimodal fusion strategies'—so the authors themselves leave the cause unresolved. The fix is concrete: evaluate all modality combinations on the intersection cohort and state the missingness policy. This does not change the reader's CONDITIONAL verdict; it sharpens the condition that must be met before the abstract's claim can be accepted. I agree with the reader that the 'consistent' wording is too strong; I disagree slightly that the linear probe is the core issue, because any fusion head would suffer the same cohort-confounding problem unless missingness is controlled.","tokens_in":12259,"tokens_out":3312,"duration_ms":38718,"concrete_test":"Reproduce all rows of Tables II and III on a single fixed cohort containing only ICU stays with structured data, at least one CXR, and at least one radiology note; report AUROC, AUPRC, and accuracy for every modality combination on this identical cohort, and explicitly state the missing-modality rule used for any auxiliary analysis. If the image+text AUROC remains below text-alone on this fixed cohort, the abstract should be revised; if it does not, the discrepancy may be a cohort artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—'incorporating multiple data modalities leads to consistent improvements'—depends on how the cohort changes when modalities are added. Section II-C describes embedding extraction but never states the imputation or exclusion rule for patients lacking CXRs or notes. Since CXRs and notes are not available for every ICU stay, Table II and Table III can be computed on different subcohorts: structured baseline on all 73,181 stays, +Images only on stays with CXRs, +Notes only on stays with notes, and +Images&Notes only on the intersection. That would make every comparison confounded by documentation status and acuity, because patients with more workup are systematically different. The paper's own interpretability section says 'missing modalities could influence model behavior' and that imaging contribution 'increased substantially in the subset of patients without missing modalities,' confirming that missingness changes the measured signal. The abstract's wording also conflicts with Tables II and III: CXR-Foundation plus Text-Embedding-3-Large yields mortality AUROC 0.8846, lower than text-alone 0.8943, so 'consistent improvements' is an overstatement unless it means only 'every multimodal combination beats the structured baseline.' This should be stated precisely, and the analysis should be rerun on one fixed cohort.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a benchmark evaluating eight foundation models on multimodal MIMIC-IV EHR data for two ICU prediction tasks (in-hospital mortality and length of stay). For unimodal encoders, the authors use a modular two-stage framework: frozen embeddings from structured vital signs, chest X-ray images, and clinical text are concatenated and fed to a logistic regression head. They also evaluate several LVLMs as multimodal learners. The benchmark includes predictive performance, fairness metrics across demographic subgroups, and interpretability via SHAP and logistic-regression coefficients. The central claim is that incorporating multiple data modalities leads to consistent improvements in predictive performance without introducing additional bias. The code is released on GitHub.","tokens_in":12461,"tokens_out":4143,"duration_ms":41908,"significance":"If the claims are substantiated, the paper would provide a useful standardized pipeline and a systematic reference for comparing unimodal and multimodal foundation models on a publicly available critical-care dataset. Strengths include the clear description of the data-processing pipeline, the use of bootstrap confidence intervals for the unimodal results, and the inclusion of both domain-specific and general-purpose models. The code release is a practical contribution that supports reproducibility. However, the paper's headline claims are currently weakened by an unspecified missing-data handling rule and by an abstract that overstates the consistency of the multimodal improvements.","major_comments":[{"comment":"The manuscript never states how patients without chest X-rays or clinical notes are handled when constructing the unimodal encoder evaluations. Section II-A2 mentions customizable cohort selection by modality availability, but it does not say whether the results in Tables II and III are computed on a common cohort, on per-row subcohorts, or with imputation. Section III-A3 explicitly says that imaging contribution 'increased substantially in the subset of patients without missing modalities,' which confirms that missingness is present and changes the measured signal. Without a fixed cohort or an explicit imputation rule, the incremental-value comparisons between rows of Tables II and III are confounded by documentation status and patient acuity, making the 'consistent improvements' claim untestable as reported.","section":"Section II-C / II-A2 and Tables II-III"},{"comment":"The abstract's claim of 'consistent improvements in predictive performance' is contradicted by the paper's own results. In Table II, for in-hospital mortality, text alone (Text-Embedding-3-Large) achieves AUROC 0.8943 (95% CI 0.8863-0.9018), while adding CXR-Foundation image embeddings reduces it to 0.8846 (95% CI 0.8767-0.8931). Section III-A1 acknowledges 'a modest decline in performance was observed when both image and text modalities were combined, compared to using text alone.' The abstract should either qualify that the improvements are relative to the structured-data baseline only, or the analysis should be rerun on a single fixed cohort to support the stronger wording.","section":"Abstract and Section III-A1"},{"comment":"The LVLM evaluation lacks confidence intervals, even though the unimodal encoder evaluation uses 1,000 bootstrap samples and reports 95% CIs for all metrics. The comparison between GPT-4o mini and the modular framework in Figure 6 is also made across different metrics (accuracy/precision/recall/F1 for LVLMs versus AUROC/AUPRC/accuracy for the modular framework), so the statement that GPT-4o mini achieved 'comparable' performance is not well-defined. Reporting error bars or confidence intervals and aligning the metrics would be necessary to support the conclusions drawn in this section.","section":"Section III-B / Figure 6"},{"comment":"The claim that incorporating multiple modalities 'did not introduce further bias' is based on visual inspection of Figure 4b without statistical tests or confidence intervals for demographic parity and equalized odds. Given the small sizes of some demographic subgroups and the multiple modality combinations being compared, the stability of these fairness metrics should be assessed quantitatively, for example by bootstrap intervals or by testing for significant differences across modality combinations.","section":"Section III-A2 and Figure 4b"}],"minor_comments":[{"comment":"The paper refers to CXR-Foundation but cites ELIXR [28]; please clarify the model name and note that ELIXR was pretrained on MIMIC-CXR, which shares the same patient population as MIMIC-IV, and discuss the potential for inflated performance due to this overlap.","section":"Section II-C2"},{"comment":"The GRU model is trained on the same MIMIC-IV data used for evaluation, unlike the other foundation models which are frozen. This difference should be explicitly acknowledged in the comparison, since the GRU is not a foundation model in the same sense.","section":"Section II-C1"},{"comment":"Adding confidence intervals or error bars to Figure 6 would bring the LVLM evaluation in line with the rest of the benchmark and improve interpretability.","section":"Figure 6"},{"comment":"There is a typographical issue in the author list ('Rui Y ang' with an extra space); this should be corrected.","section":"Author list"}],"recommendation":"major_revision","confidential_remarks":"The paper is a benchmark study rather than a new method, and the central empirical claim hinges on the treatment of missing modalities. The authors should be asked to either specify the exact cohort/imputation procedure and rerun the tables on a common cohort, or explicitly qualify the abstract's wording. The fairness analysis also needs quantitative support. The scope is appropriate for a journal that publishes reproducible benchmark studies; the code release is a positive aspect."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a genuinely useful benchmark for multimodal EHR modeling, with a clean pipeline, released code, and bootstrap CIs for the unimodal probes. Read it for the resource, not for the abstract.\n\nWhat is actually new: the paper adapts the HAIM-style preprocessing and evaluates eight foundation models—domain-specific and general-purpose, unimodal and LVLM—on two standard ICU tasks (mortality and long stay), with fairness metrics and SHAP interpretability on top. That combination isn't available in prior benchmarks like MC-BEC or RadFusion. The pipeline is described well enough to reproduce, and the code is public. The finding that domain-specific models match larger general-purpose ones at lower cost is credible, and the fairness analysis is a bonus most benchmarks skip.\n\nThe soft spots are real but fixable. First, the abstract says 'consistent improvements' from adding modalities. The body qualifies this: every multimodal combination beats the structured-only GRU baseline, but image+text sometimes underperforms text alone (mortality AUROC 0.8846 vs 0.8943). The abstract should say 'compared to structured data alone.' Second, the paper never states how missing CXRs or notes are handled per row in Tables II–III. If the +Images rows run on the subset of stays with CXRs, the comparisons are confounded by documentation status and acuity. The interpretability section even shows missingness changes feature contributions, so this isn't a nitpick—it affects the central comparison. The authors should rerun on one fixed cohort or explicitly state the imputation. Third, the LVLM results in Figure 6 have no confidence intervals, unlike the unimodal tables.\n\nThe citation pattern is fine; prior benchmarks are cited and the differences are stated. No circular reasoning beyond the usual concern that CXR-Foundation was pretrained partly on MIMIC data, which the paper doesn't hide.\n\nWho should read it: anyone building or evaluating multimodal models on MIMIC-IV, or doing clinical prediction with foundation models. It's a useful reference point, not a breakthrough. With revisions—toning down the abstract and clarifying the cohort—it's a solid contribution.\n\nRecommendation: send to peer review. The methods are reproducible and the resource is valuable enough to be worth referee time, even though the conclusions need tightening.","headline":"Useful, reproducible multimodal EHR benchmark with a clean pipeline and public code, but the abstract oversells 'consistent improvements' and the handling of missing modalities needs to be pinned down.","tokens_in":13039,"tokens_out":2780,"would_cite":true,"duration_ms":29468,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing chest X-ray and clinical-note embeddings with structured ICU data improves mortality and length-of-stay predictions without worsening group fairness, while current vision-language models lag, the paper claims.","keywords":["foundation models","multimodal electronic health records","MIMIC-IV","in-hospital mortality","length of stay","fairness","interpretability","large vision-language models"],"falsifier":"Run the same benchmark on only complete-case ICU stays, namely patients with structured data, chest X-rays, and notes all present, and compare AUROC, AUPRC, and fairness metrics to the reported full-cohort results; if the complete-case numbers show that text-plus-image worsens mortality AUROC and that the model rankings change, the claim that multimodal fusion adds value without bias is not supported.","tokens_in":12028,"feed_emoji":"🏥","tokens_out":10416,"duration_ms":115641,"temperature":0.7,"pith_summary":"This paper builds a public benchmark from the MIMIC-IV ICU database and asks whether embeddings from foundation models improve two clinical predictions—in-hospital mortality and an ICU stay longer than three days—when added to structured demographics and vital signs. The authors compare eight models, including domain-specific and general-purpose encoders for time series, chest X-rays, and clinical notes, plus three large vision-language models, and they evaluate fairness across age, gender, and race as well as feature importance. Their central claim is that adding image and text representations to structured data consistently improves predictive performance without worsening group fairness, and that small domain-specific encoders can match much larger general-purpose encoders on unimodal tasks. They also report that current vision-language models generalize poorly, especially to length-of-stay prediction, so modular pipelines remain the more reliable option. The contribution is a reproducible, analysis-ready multimodal EHR pipeline and a multidimensional comparison to guide model selection.","feed_headline":"Images and notes boost ICU predictions without new bias","feed_subtitle":"A MIMIC-IV benchmark puts eight foundation models head-to-head on mortality and length of stay, with fairness checks.","key_machinery":"The engine is the frozen-embedding linear probe. Each modality is encoded once by a foundation model—fixed-interval aggregation, a GRU, or Moment for vital signs; CXR-Foundation or Swin Transformer for chest X-rays; RadBERT or Text-Embedding-3-Large for notes—and the resulting vectors are concatenated and fed to a logistic-regression classifier, so any performance difference reflects the representation rather than the classifier. Fairness is measured with demographic parity and equalized odds computed across age, gender, and race subgroups, and interpretability uses SHAP values and logistic-regression coefficients. For the multimodal-learner arm, the machinery is a prompted evaluation that converts non-image data to text and asks closed-ended yes/no questions, with the percentage of answerable questions tracked.","core_discovery":"The central discovery the paper argues for is that a modular two-stage design—frozen embeddings per modality concatenated into a single vector and classified by logistic regression—lets a hospital exploit multiple EHR modalities without retraining the encoders. On MIMIC-IV v2.2, the authors report that this design beats structured-data-only baselines on both tasks: the best mortality AUROC is 0.8943 with text embeddings from Text-Embedding-3-Large, and the best length-of-stay AUROC is 0.8532 with the same text encoder; adding CXR embeddings on top of text yields slightly lower numbers, which the authors attribute to missing modalities and the limits of simple concatenation. They further claim that multimodal integration does not harm group fairness, that domain-specific encoders such as CXR-Foundation and RadBERT stay competitive with general-purpose models trained at far larger scale, and that GRU-based vital-sign embeddings outperform the time-series foundation model Moment. For vision-language models, the paper claims that GPT-4o mini reaches comparable accuracy to the modular framework on mortality, but all tested LVLMs, including the medical LLaVA-Med, perform poorly on length-of-stay prediction, indicating limited task generalizability.","pith_inferences":["Read carefully, the paper's numbers support 'adding a single modality to structured data helps' more strongly than 'adding every modality helps': in its mortality table, AUROC drops from 0.8943 with text alone to 0.8846 with text plus chest X-rays, so the consistency claim should be scoped to multimodal-versus-structured comparisons.","The fairness conclusion applies to the linear head used in the benchmark, not to arbitrary fusion architectures; expressive end-to-end models could redistribute errors across groups, so fairness should be re-audited whenever the fusion method changes.","Because the LVLM arm relies on a single prompt template and closed-ended answers, small template changes could shift results; a natural extension is to vary prompt wording and answer parsing and see whether the ranking of GPT-4o mini versus the modular framework survives.","The interpretability finding that image features become more important in the complete-case subset implies that missing-modality imputation drives apparent feature importance; reporting all headline results on the complete-case cohort would separate fusion effects from imputation effects."],"forward_implications":["Hospitals can evaluate ICU risk models by combining frozen embeddings with a simple logistic head, since this setup exposes the value of each modality without encoder fine-tuning.","Text embeddings carry most of the predictive signal beyond structured vital signs; the reported tables show that stacking averaged CXR embeddings on top of text can slightly reduce AUROC, so modality additions should be validated per task.","Small, domain-specific encoders such as CXR-Foundation and RadBERT are a cost-effective substitute for much larger general-purpose encoders, at least for these two ICU tasks.","Current large vision-language models are not yet reliable for length-of-stay prediction; modular frameworks outperform them and should be preferred for deployment until LVLMs improve.","The released pipeline can be extended to additional MIMIC-IV modalities such as ICD codes and ECG signals, making the benchmark a starting point for broader multimodal ICU research."],"supporting_citations":[{"why":"It supplies the MIMIC-IV v2.2 cohort, all ICU stays, and the hospital, ICU, chest X-ray, and notes data modules used in every experiment.","marker":"[5]"},{"why":"It defines the four-module data collection and multimodal integration scheme that the pipeline and two-stage framework adapt.","marker":"[24]"},{"why":"It provides the GRU vital-sign encoder that beats fixed-interval aggregation and Moment as the structured-data baseline.","marker":"[26]"},{"why":"It supplies the general-purpose time-series foundation model used as a zero-shot structured-data comparator.","marker":"[27]"},{"why":"It provides the domain-specific chest X-ray encoder whose embeddings are compared against the general-purpose Swin Transformer.","marker":"[28]"},{"why":"It gives the general-purpose image encoder pretrained on ImageNet-22K used as the visual comparator.","marker":"[29]"},{"why":"It supplies the domain-specific radiology-report encoder compared with Text-Embedding-3-Large.","marker":"[30]"},{"why":"It provides the medical vision-language model used to test whether domain specialization helps multimodal learners.","marker":"[32]"},{"why":"It supplies the general-purpose vision-language baseline for the LVLM comparisons.","marker":"[33]"},{"why":"It provides the Shapley-value method used to attribute predictions to modalities in the interpretability analysis.","marker":"[34]"}],"fun_headline_variants":["No retraining needed: multimodal EHR beats baselines","Specialists match big generalists on ICU EHR","Frozen embeddings, simple fuse: multimodal ICU win","Multimodal fusion: no extra bias, better ICU prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on the assumption that a logistic-regression head on concatenated frozen embeddings is a fair way to compare foundation models, and that the handling of patients who lack images or notes does not distort the comparison.","fun_headline_variants_meta":{"raw":{"variants":["No retraining needed: multimodal EHR beats baselines","Specialists match big generalists on ICU EHR","Frozen embeddings, simple fuse: multimodal ICU win","Multimodal fusion: no extra bias, better ICU prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3552,"prompt_tokens":954,"completion_tokens":2598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":2534}},"tokens_in":570,"tokens_out":2598,"duration_ms":24191,"temperature":1.0,"reasoning_tokens":2534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:45:37.599767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same benchmark on only complete-case ICU stays, namely patients with structured data, chest X-rays, and notes all present, and compare AUROC, AUPRC, and fairness metrics to the reported full-cohort results; if the complete-case numbers show that text-plus-image worsens mortality AUROC and that the model rankings change, the claim that multimodal fusion adds value without bias is not supported.","supporting_citations":[{"cited_title":"Mimic- iv, a freely accessible electronic health record dataset,","cited_arxiv_id":null,"evidence_quote":"It supplies the MIMIC-IV v2.2 cohort, all ICU stays, and the hospital, ICU, chest X-ray, and notes data modules used in every experiment."},{"cited_title":"Integrated multimodal artificial intelligence framework for healthcare applications,","cited_arxiv_id":null,"evidence_quote":"It defines the four-module data collection and multimodal integration scheme that the pipeline and two-stage framework adapt."},{"cited_title":"Improved fine-tuning of in-domain transformer model for inferring covid-19 presence in multi- institutional radiology reports,","cited_arxiv_id":null,"evidence_quote":"It supplies the domain-specific radiology-report encoder compared with Text-Embedding-3-Large."}],"review_version":1}