{"id":"388781cf-bd60-4391-939a-c6c8aa6ca663","arxiv_id":"2507.03165","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PiCME shows contrastive learning peaks at three modalities in MIMIC, and a Modality-Gated LSTM with contrastively learned weights improves five-modality mortality prediction over supervised baselines.","lead":"The paper introduces PiCME, a pipeline that pre-trains contrastive models on all 26 combinations of five clinical data types from MIMIC (discharge notes, radiology reports, chest X-rays, demographics, time-series) and evaluates them on in-hospital mortality and phenotype prediction. A smart generalist would read it for a systematic guide to which modality combinations help clinical prediction, plus a new gated fusion method that improves five-modality performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified disjointness of contrastive pre-training and fine-tuning test sets could invalidate the central performance comparison.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing issue: the paper's central quantitative claims rest on a clean pre-training/fine-tuning split, and Appendix 8.3 leaves that split ambiguous and incomplete. I agree that this is the most consequential concern. The principal reported result—mLSTM achieving 76.93% AUROC and 62.26% AUPRC in the five-modality IHM setting, and contrastive peak 75.94%—is only meaningful if no test patients were seen during contrastive pre-training. The ambiguous wording and placeholder sample sizes mean the current manuscript does not support verification. This is not an accusation of misconduct; it is an identifiable reproducibility gap that a concrete patient-ID overlap check or a rerun with a disjoint pre-training split would settle. The reader's CONDITIONAL verdict remains appropriate: the concern is addressable but unresolved, and the numerical headline should be read with caution until the split protocol is confirmed. I did not find a stronger alternative concern; the missing ungated-LSTM baseline and selective reporting of modality combinations are real weaknesses but secondary to the leakage question, because leakage alone would invalidate the core comparison.","tokens_in":12882,"tokens_out":2852,"duration_ms":33802,"concrete_test":"From the released repository, extract the subject_id/hadm_id sets used for contrastive pre-training, fine-tuning validation, and fine-tuning test. Check for any overlap between the pre-training set and the fine-tuning test set. If overlap exists, retrain all contrastive models and the Modality-Gated LSTM with pre-training restricted to the 80% fine-tuning training split only, then compare Table 1. If the five-modality IHM AUROC drops materially (e.g., below the 73.19% supervised baseline or within its standard error), the central claim of mLSTM gains is invalidated; if results are unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pivotal comparison in Table 1 is between contrastive pre-training plus fine-tuning and fully supervised training on the same task split. Appendix 8.3 says contrastive learning 'uses the full original training set' while fine-tuning is done on an '80/10/10 split of the remaining data', with sample counts given as placeholders ('X training samples'). If the 'full original training set' includes patients who later fall into the fine-tuning test split, then the contrastive encoders and the λ weights used by the Modality-Gated LSTM have been exposed to test patients. In that case the reported five-modality IHM improvements (contrastive 75.94% AUROC; mLSTM 76.93% AUROC and 62.26% AUPRC versus the 73.19% supervised baseline) would reflect transductive leakage rather than the benefit of contrastive representation learning. The paper provides no patient-ID overlap check, no explicit statement that pre-training and test sets are disjoint, and the placeholder dataset sizes make the split unverifiable from the text. The Appendix 8.3 wording is ambiguous, so this is a reporting gap rather than an established error, but it is the single condition on which the headline result depends.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PiCME proposes a systematic pipeline for contrastive representation learning across all 26 modality combinations of five clinical modalities in MIMIC-IV/MIMIC-CXR. After contrastive pre-training with InfoNCE for two modalities and a weighted One-Versus-Others loss for three or more, the authors fine-tune on in-hospital mortality and phenotyping, and they introduce a Modality-Gated LSTM that injects contrastively learned λ weights into the LSTM cell update for five-modality supervised fusion. The reported results show that three-modality contrastive models are competitive with or better than fully supervised baselines, that performance degrades with more modalities, and that mLSTM achieves the best five-modality results (IHM AUROC 76.93%, AUPRC 62.26%; phenotyping AUROC 74.36%). The paper also evaluates alignment via t-SNE and top-5 cosine similarity, demographic subgroup performance, and Integrated Gradients attribution, and it reports 10-seed averages with code.","tokens_in":13116,"tokens_out":9654,"duration_ms":104967,"significance":"If the central results hold, PiCME would provide a useful reference point for modality selection in MIMIC and a demonstration that contrastively learned weights can be transferred into a supervised fusion mechanism. The systematic all-combination evaluation is a genuine contribution, and the multi-seed evaluation with released code exceeds common practice in this area. The Modality-Gated LSTM idea is interesting but needs a complete specification. The significance of the paper hinges on split hygiene and on statistical support for the headline improvements, both of which require attention before the claims can be accepted.","major_comments":[{"comment":"The data-split description is ambiguous and the sample counts are placeholders ('X training samples'). The paper states that 'Contrastive learning uses the full original training set, while downstream fine-tuning is performed on a 80/10/10 split of the remaining data,' but it does not define what 'remaining data' excludes. Because the headline comparison in Table 1 is between contrastive pre-training plus fine-tuning and fully supervised training, the result is valid only if the fine-tuning test patients were never seen during contrastive pre-training. Please provide the exact MedFuse split definitions, the number of patients in each split, and a patient-ID intersection check between the contrastive pre-training set and the fine-tuning test split.","section":"Appendix 8.3"},{"comment":"The paper states that supervised baselines were trained only on the 'top three most effective' combinations, yet Table 1 draws general conclusions such as 'contrastive models remain competitive with supervised baselines' across the modality-combination sweep. The selection rule for the supervised subset is not defined, and the untested combinations cannot support those conclusions. Please either report supervised baselines for all 26 combinations or explicitly restrict the comparison claims to the combinations actually tested.","section":"Section 5.2"},{"comment":"The Modality-Gated LSTM is not fully specified. The equations for the input gate I_t, forget gate F_t, output gate O_t, and candidate memory C̃_t are omitted, and the mapping from modalities to LSTM timesteps is not described. As written, λ_t appears in the cell update, but it is unclear how a scalar λ_i per modality is broadcast over the hidden dimension and over time. Please provide the complete forward pass and state explicitly how each modality is assigned to a timestep.","section":"Section 3.3.4"},{"comment":"It is not stated which model is used to compute Integrated Gradients. If the attributions are computed on the concatenated classifier rather than on the mLSTM that consumes λ, then Table 2 does not validate the gating mechanism. In addition, because both the λ weights and the attributions are estimated from the same data and model family, their agreement needs a control condition (e.g., uniform λ or randomly permuted λ) before it can be interpreted as validation.","section":"Section 4.3.2, Table 2"},{"comment":"The main claims of improvement are based on mean differences over 10 seeds without significance tests, and several headline differences have overlapping standard deviations (e.g., IHM AUPRC 51.27±6.65 vs 62.26±2.79; phenotyping AUROC 73.43±0.52 vs 74.36±0.71). Please report paired tests or confidence intervals over the 10 seeds for the key comparisons: contrastive versus supervised, and mLSTM versus supervised.","section":"Table 1"}],"minor_comments":[{"comment":"The statement that performance declines beyond three modalities is not supported for the supervised IHM baseline in Table 1, where the five-modality AUROC (73.19) exceeds the three-modality TD+I+D AUROC (65.50). Please adjust the wording to distinguish the IHM and phenotyping tasks.","section":"Abstract and Section 4.2"},{"comment":"Replace the placeholders 'X training samples' with actual counts for both tasks and specify the sizes of the training, validation, and test splits.","section":"Appendix 8.3"},{"comment":"'NVIDIA GeForce RYX 3090' should be 'RTX 3090', and Table 5 still contains 'hrs' placeholders for the four- and five-modality runtime rows.","section":"Appendix 8.6"},{"comment":"The citation 'BERT [ devlin2018bert]' is malformed and should be completed; the paper also uses the phrase 'train all modality encoders from scratch' for the supervised baselines while initializing ClinicalBERT from pretrained weights, so the wording should be clarified.","section":"Section 3.1"},{"comment":"The y-axis of the top-5 cosine similarity accuracy plot should be labeled, and the metric definition should be repeated in the caption for readability.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"I am not aware of any novelty disclosure issue. The central risk is the unverified disjointness of the contrastive pre-training and fine-tuning test sets; if the authors confirm the split and fill in the missing methodological details, the paper could become a solid contribution. I would not reject on the current evidence, but the load-bearing gaps need to be resolved in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a genuinely useful paper. It is the first to train contrastive models on all 26 modality combinations in MIMIC, reports 10-seed means, and releases code. The central observation—contrastive alignment and performance peak at three modalities and decline beyond—is a useful empirical data point, and the Modality-Gated LSTM (mLSTM) that transfers contrastively learned weights into an LSTM is a plausible, small but non-obvious idea.\n\nThe weighted OvO loss with learnable lambdas is a reasonable extension, and the mLSTM does beat the concatenation baseline on five-modality IHM (76.93 vs 73.19 AUROC). The demographic subgroup reporting is a nice addition.\n\nThe big problem is Appendix 8.3. The text says contrastive learning 'uses the full original training set' while fine-tuning uses 'a 80/10/10 split of the remaining data', with sample sizes given as 'X'. On one reading, contrastive pre-training covers the same patients used for fine-tuning test, which would make the comparison transductive and invalid. On another reading, it is a disjoint but unusual protocol (splitting the original test set for fine-tuning). The paper does not provide a patient-ID overlap check or clear statement of disjointness. This is a reporting gap, but it is load-bearing: the entire comparison in Table 1 depends on it. The authors must clarify this before the numbers can be trusted.\n\nAlso, the mLSTM is only compared against concatenation, not an ungated LSTM. So part of the gain could be the recurrent architecture rather than the lambda gating. A plain LSTM baseline is needed. The supervised baselines also cover only a subset of top-performing combinations, which weakens the 'map' aspect for supervised learning, though this is acknowledged in limitations.\n\nOther minor issues: lambda is validated with Integrated Gradients on the same model/data, which is circular but not damning. The appendix has placeholder 'X training samples' and a few typos.\n\nWho is this for? Anyone working on multimodal fusion in clinical data, especially with MIMIC. The sweep is a useful benchmark reference, even if the absolute numbers shift after clarification. It deserves a serious referee: the novelty is real, the methodology is mostly sound, and the issues are addressable with additional experiments and clarification.\n\nRecommendation: send it to review, with the expectation that the authors resolve the split question and add an ungated LSTM baseline. If the split is fine, it is a solid paper; if not, the main claim collapses.","headline":"Useful systematic contrastive sweep in MIMIC with a promising gated fusion, but Appendix 8.3's split description is ambiguous enough to put the headline numbers on hold until clarified.","tokens_in":13667,"tokens_out":5845,"would_cite":true,"duration_ms":62416,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Contrastive learning peaks at three clinical modalities, and a modality-gated LSTM then beats the five-modality plateau.","keywords":["multimodal learning","contrastive learning","MIMIC-IV","Modality-Gated LSTM","One-Versus-Others loss","in-hospital mortality prediction","phenotyping","clinical prediction"],"falsifier":"Open the data-split implementation and check whether any patient in the fine-tuning test split contributed a modality view to contrastive pre-training; if any do, rerun contrastive pre-training on only the 80% subset and compare the five-modality mLSTM results (reported 76.93% AUROC and 62.26% AUPRC against a 73.19% AUROC and 51.27% AUPRC supervised baseline). If the margin collapses, leakage is confirmed; if it persists, the claim survives.","tokens_in":12671,"feed_emoji":"🏥","tokens_out":7386,"duration_ms":73014,"temperature":0.7,"pith_summary":"This paper proposes PiCME, a pipeline that pre-trains contrastive models on all 26 combinations of two to five clinical modalities from MIMIC (discharge summaries, radiology reports, chest X-rays, demographics, and time-series) and evaluates them on in-hospital mortality and phenotyping. It finds that contrastive representations are competitive with, and sometimes better than, fully supervised training when only two or three modalities are used, with the best three-modality combination reaching 75.94% AUROC on mortality. Performance degrades as more modalities are added, for both contrastive and supervised fusion. To recover from that plateau, the paper introduces a Modality-Gated LSTM that weights each modality by a contrastively learned importance parameter, achieving the best five-modality results: 76.93% AUROC and 62.26% AUPRC for mortality and 74.36% AUROC for phenotyping. It also reports that these learned weights align with Integrated Gradients attributions and that the gated model generalizes more consistently across demographic subgroups.","feed_headline":"Gated LSTM lifts five-modality mortality AUROC to 76.93","feed_subtitle":"Contrastive pre-training peaks at three modalities; the new gate pushes five-modality mortality prediction past the plateau.","key_machinery":"The load-bearing mechanism is the weighted One-Versus-Others (OvO) contrastive loss with learnable per-modality weights $\\lambda_i$, combined with the Modality-Gated LSTM (mLSTM). In the OvO loss, each modality is pulled toward the averaged embedding of all other modalities, and each loss term is scaled by a softmax-normalized $\\lambda_i$ that is trained jointly with the encoders; these $\\lambda$ values become the modality-importance signal. The mLSTM then injects those frozen $\\lambda$ values into the LSTM cell update as $C_t = F_t \\odot C_{t-1} + (I_t \\odot \\tilde{C}_t) \\odot \\vec{\\lambda}_t$, so that modalities judged important during pre-training contribute more to the hidden state during supervised fine-tuning.","core_discovery":"The central claim is that systematically scaling contrastive pre-training in a clinical multimodal setting reveals a three-modality peak and a subsequent plateau, and that this plateau can be overcome by transferring contrastively learned modality importances into a supervised fusion model. Concretely, the paper asserts that across all 26 modality subsets of MIMIC, the contrastive model peaks at discharge text plus chest X-ray plus demographics (75.94% AUROC for in-hospital mortality), then declines to 61.44% AUROC with all five modalities, while a fully supervised concatenation baseline saturates around 73% AUROC. The Modality-Gated LSTM, which scales the candidate memory by modality-specific $\\lambda$ values learned during contrastive pre-training, is claimed to be the fusion method that exceeds the plateau, reaching 76.93% AUROC and 62.26% AUPRC for mortality and 74.36% AUROC for phenotyping with all five modalities. The paper further claims that contrastively learned $\\lambda$ values are validated by Integrated Gradients attributions and that the gated model is the most consistent across gender, ethnicity, and age subgroups.","pith_inferences":["The paper leaves implicit that its three-modality peak may be partly shaped by encoder choice: both text modalities share ClinicalBERT, which likely inflates text-text alignment and could make text-heavy triples look stronger than a different encoder suite would.","A natural extension not tested here is to use the contrastive $\\lambda$ values as priors in attention-based fusion or as a guide for dropping low-weight modalities at inference, which could cut compute without repeating the full 26-combination sweep.","Because the evaluation keeps only complete-case patients, the reported AUROC numbers describe a selected population; the paper's own future-work plan of handling missing modalities suggests the method should be re-tested when entire modalities are absent at inference.","If the contrastive gains survive a leakage-free split, a practical implication the authors do not spell out is that PiCME's per-modality $\\lambda$ could serve as a cheap modality-selection screen before training expensive supervised models."],"forward_implications":["For two- or three-modality clinical tasks, contrastive pre-training with frozen encoders can match or beat end-to-end supervised training, so a single reusable embedding model can support multiple downstream tasks without task-specific labeled data.","Adding a fourth or fifth modality by simple concatenation hurts both contrastive and supervised models, so raw modality count is not a reliable proxy for predictive value.","Contrastively learned modality weights, normally a byproduct of pre-training, can act as a regularizer in supervised fusion and recover the performance lost to integration noise.","Modality importance estimated without labels (the $\\lambda$ values from pre-training) tracks task-specific attribution (Integrated Gradients), suggesting that the same pre-training can inform interpretability and model selection.","The gated fusion's consistency across age, gender, and ethnicity subgroups suggests that dynamic modality weighting can reduce demographic performance gaps relative to fixed fusion."],"supporting_citations":[{"why":"Supplies the in-hospital mortality and phenotyping tasks, the 48-hour observation window, and the data-split scheme PiCME inherits.","marker":"[14]"},{"why":"Provides the One-Versus-Others (OvO) loss that the paper extends with learnable $\\lambda$ weights for three-or-more-modality contrastive pre-training.","marker":"[17]"},{"why":"Provides the HAIM multimodal fusion baseline that the concatenated-embedding and supervised comparisons build on.","marker":"[21]"},{"why":"Supplies ClinicalBERT, the text encoder used for discharge summaries and radiology reports.","marker":"[22]"},{"why":"Supplies Integrated Gradients, the attribution method used to validate that contrastive $\\lambda$ values match task-specific modality importance.","marker":"[31]"},{"why":"Provides the CLIP vision-language contrastive paradigm that motivates treating all modality pairs as alignable views.","marker":"[16]"}],"fun_headline_variants":["Contrastive peak at 3 modalities; gate beats 5-modality plateau","Modality-gated LSTM breaks five-modality mortality AUROC plateau","PiCME: scaling contrastive learning across 26 MIMIC modalities","Three modalities suffice until gated LSTM recovers the drop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central comparison assumes that no patient used to fine-tune and test the models ever appears in contrastive pre-training, yet the appendix states that contrastive learning uses the full original training set while fine-tuning uses an 80/10/10 split of the remaining data, with sample sizes left as placeholders.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive peak at 3 modalities; gate beats 5-modality plateau","Modality-gated LSTM breaks five-modality mortality AUROC plateau","PiCME: scaling contrastive learning across 26 MIMIC modalities","Three modalities suffice until gated LSTM recovers the drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1542,"prompt_tokens":1093,"completion_tokens":449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":709,"tokens_out":449,"duration_ms":5506,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:16:49.803721+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the data-split implementation and check whether any patient in the fine-tuning test split contributed a modality view to contrastive pre-training; if any do, rerun contrastive pre-training on only the 80% subset and compare the five-modality mLSTM results (reported 76.93% AUROC and 62.26% AUPRC against a 73.19% AUROC and 51.27% AUPRC supervised baseline). If the margin collapses, leakage is confirmed; if it persists, the claim survives.","supporting_citations":[{"cited_title":"Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images,","cited_arxiv_id":null,"evidence_quote":"Supplies the in-hospital mortality and phenotyping tasks, the 48-hour observation window, and the data-split scheme PiCME inherits."},{"cited_title":"Fastsiam: Resource-efficient self-supervised learn- ing on a single gpu,","cited_arxiv_id":null,"evidence_quote":"Provides the One-Versus-Others (OvO) loss that the paper extends with learnable $\\lambda$ weights for three-or-more-modality contrastive pre-training."},{"cited_title":"Integrated multimodal artificial intelligence framework for healthcare appli- cations,","cited_arxiv_id":null,"evidence_quote":"Provides the HAIM multimodal fusion baseline that the concatenated-embedding and supervised comparisons build on."},{"cited_title":"Axiomatic attribution for deep networks,","cited_arxiv_id":null,"evidence_quote":"Supplies Integrated Gradients, the attribution method used to validate that contrastive $\\lambda$ values match task-specific modality importance."}],"review_version":1}