Pith. sign in

REVIEW 3 major objections 5 minor 31 references

Explicit multimodal routes for labs, notes, and X-rays make ICU predictions both stronger and auditable by showing which evidence pathways drive each decision.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 01:15 UTC pith:KMTH65Y3

load-bearing objection Clean multimodal routing with real audit machinery; interpretability claims rest on indirect evidence but the engineering and experiments are solid enough for peer review. the 3 major comments →

arxiv 2607.09982 v1 pith:KMTH65Y3 submitted 2026-07-10 cs.LG

Multimodal Routing for Interpretable, Robust, and Auditable Clinical Prediction

Nikkie Hooman , Zhongjie Wu , Eric C. Larson , Mehak Gupta This is my paper
classification cs.LG
keywords Multimodal learningEHRrouting modulemissing modalitiesICU outcome predictioninterpretabilityclinical noteschest X-rays
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most multimodal clinical models fuse labs, notes, and images into one opaque representation, so clinicians cannot tell which source supported a prediction. This paper argues that building ten discrete routes—unimodal, directional bimodal, and one trimodal—then learning patient- and label-specific weights for each route yields both higher accuracy and transparent reasoning on MIMIC-IV ICU mortality and 25-phenotype tasks. Inference-time route masking lets the model drop any modality without retraining and redistributes weight over the remaining routes, turning missing-data scenarios into an audit of how the system reweights evidence. The resulting routing patterns differ systematically by disease group: metabolic phenotypes lean on structured labs, respiratory phenotypes lean on imaging-plus-notes, and some low-prevalence conditions reveal shortcut reliance on notes. If the framework holds, hospitals gain a practical way to demand explanations and robustness checks for multimodal AI before deployment.

Core claim

An explicit set of ten routes (three unimodal, six directional bimodal via cross-attention, one hierarchical trimodal) together with patient-specific activations and label-specific routing coefficients, plus inference-time route masking, produces higher AUROC and F1 than joint fusion, late fusion, TriMF and MoE-Health on trimodal MIMIC-IV stays while exposing clinically coherent modality-reliance patterns across phenotypes and mortality.

What carries the argument

The multimodal routing layer: each of the ten routes produces an embedding; route activations measure expression strength and routing coefficients measure task relevance; their product forms an effective weight that is used both for prediction and for post-hoc audit under route masking.

Load-bearing premise

The directional routes and learned coefficients isolate genuine clinical interactions rather than architectural artifacts or dataset shortcuts that only look meaningful after the fact.

What would settle it

On a held-out trimodal ICU cohort, ablate or reverse the directional cross-attention (or replace learned coefficients with uniform weights) and check whether the claimed accuracy gains and clinically coherent missing-modality weight shifts both disappear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Clinicians can inspect, for any patient and phenotype, which unimodal or cross-modal route carried the decision.
  • Missing-modality performance can be audited without retraining by simply zeroing routes that involve the absent source.
  • Disease groups that collapse when one modality is removed become candidates for more data, regularization, or architectural fixes.
  • The same route set can be reused for other multi-label ICU tasks that share the same three modalities.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The notes–image shortcut observed for cerebrovascular disease suggests that route-level auditing could become a standard pre-deployment safety check for any multimodal clinical model.
  • Quadratic growth of directional bimodal routes implies sparse or dynamic routing will be needed before the method scales past three modalities.
  • Because structured data is treated as the always-available clinical anchor, the framework may understate fragility in settings where vitals themselves are missing or delayed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an explicit multimodal routing architecture for EHR prediction that constructs ten discrete routes—three unimodal, six directional bimodal (via cross-attention), and one hierarchical trimodal—over structured longitudinal data (L), clinical notes (N), and chest X-rays (I). Patient-specific route activations α and label-specific routing coefficients R yield effective weights W=α·R that drive prediction and are intended to support local, global, and group-level interpretability. Robustness and auditability are assessed by inference-time route masking (no retraining) that disables routes involving a missing modality and renormalizes the rest. On trimodal MIMIC-IV ICU stays the method reports higher AUROC/F1 than joint fusion, late fusion, TriMF, and MoE-Health for binary mortality and 25-phenotype multi-label prediction, with routing heatmaps and missing-modality breakdowns used to characterize modality reliance by clinical group.

Significance. If the directional routes and learned weights isolate clinically meaningful asymmetric evidence rather than encoder-capacity or dataset artifacts, the work would be a practical contribution to trustworthy multimodal clinical ML: it couples competitive accuracy with an auditable pathway structure and a no-retrain missing-modality protocol that can surface shortcut reliance (as the authors themselves note for cerebrovascular disease). Strengths that should be credited include matched-encoder baselines and ablations (Tables 1–2), open-source code, systematic per-phenotype and group-level analyses (Table 3, Figures 2–4), and the explicit separation of activations from coefficients. Absolute F1 remains modest under class imbalance, so the primary value is the interpretability/audit framing rather than a large accuracy leap.

major comments (3)
  1. The central interpretability/audit claim rests on the premise that the six directional bimodal routes (Eq. 3; R2 in §3.3) and effective weights W=α·R isolate clinically meaningful asymmetric interactions. Support is only indirect (missing-modality drops and heatmaps in §5.2–5.4). The paper itself flags a notes–image shortcut for acute cerebrovascular disease (§5.4: high I←N/N←I weight yet near-collapse when N is masked). There is no capacity-matched control that replaces directional cross-attention with symmetric attention (or undirected concatenation) while retaining the same routing layer, nor any external check (e.g., clinician-annotated evidence sources or route-level ablation of reverse directions only). Without such a control, the claim that routes are clinically asymmetric rather than architectural or shortcut artifacts remains under-supported for a paper whose title and abstract
  2. Robustness is evaluated exclusively by training on fully trimodal stays and masking routes only at inference (§3.5, §4.1, §4.3.3; Table 4). Structured data L is never masked because it is ‘always retained.’ This design is a valid stress test of post-deployment missingness for a trimodal-trained model, but it does not establish robustness under the more realistic regime in which the model is trained with missing modalities (or with a missing-modality bank). The manuscript should either (i) add a train-with-missing comparison against methods designed for that setting, or (ii) clearly scope the robustness claim to inference-time masking of a trimodal-trained model and avoid broader deployment language.
  3. Table 1 reports gains of roughly 3–6% over MoE-Health, but no statistical significance tests, confidence intervals beyond ±std, or multiple-run protocol details are given for the primary comparison. Given overlapping stds on some metrics and modest absolute F1 (0.45 mortality, 0.50 phenotypes), the performance superiority claim needs either paired significance testing across seeds or a clearer statement that gains are small and secondary to the routing analysis.
minor comments (5)
  1. Figure 1 is dense; route labels (e.g., L⟶N vs N⟶L) are hard to parse at print size. Consider a simplified schematic plus a table enumerating the ten routes.
  2. Notation mixes R for the route set and R_{p,r,c} for routing coefficients; a distinct symbol for coefficients would reduce confusion in §3.4.
  3. Appendix routing-coefficient heatmaps (Figures 5–6) largely duplicate effective-weight patterns; a short quantitative comparison (e.g., rank correlation of R vs W) would justify relegating them to the appendix.
  4. Related-work placement of FAME and DrFuse is brief; a one-sentence contrast on directional vs scalar/fairness weighting would help readers place the contribution.
  5. Typos/clarity: ‘dimen-sion’ line break in §4.2; ‘primary activation (α_r)’ in figure captions should match α_{p,r}; ACM reference placeholder URLs should be cleaned for camera-ready.

Circularity Check

0 steps flagged

No circularity: purely empirical architecture + held-out evaluation; routing weights and missing-modality audits are learned/post-hoc, not definitional reductions.

full rationale

The paper introduces an explicit 10-route multimodal architecture (unimodal + directional bimodal via cross-attention + hierarchical trimodal) with learnable patient/label-specific activations α and coefficients R, then evaluates predictive AUROC/F1 against joint/late fusion, TriMF and MoE-Health on held-out MIMIC-IV trimodal stays, plus ablations and inference-time route masking. There is no first-principles derivation, uniqueness theorem, or closed-form prediction that reduces by construction to a fitted constant or self-defined quantity. Route embeddings (Eq. 3), effective weights W=α·R, and reweighted remaining routes under masking are all free parameters optimized on training data and inspected post-hoc; performance numbers are empirical comparisons, not tautologies. Self-citations (e.g., the authors’ prior FAME work) appear only as related-work context and are not load-bearing for any claim. The work is therefore self-contained against external benchmarks and exhibits none of the six circularity patterns.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The central empirical claims rest on standard deep-learning practice plus a small set of architectural choices and domain conventions; no new physical entities are postulated. Free parameters are the usual hyper-parameters and the learned routing matrices; axioms are the usual assumptions of the MIMIC phenotype definitions and the chosen pre-trained encoders.

free parameters (4)
  • shared embedding dimension d = 256
    Set to 256 by hand; controls capacity of every route embedding and projection.
  • route-activation dimension a = 128
    Set to 128; size of the content vector before the sigmoid gate.
  • learning rate / batch size / max epochs = 1e-4 / 16 / 50
    Adam 1e-4, batch 16, early-stop on validation loss up to 50 epochs; standard but free choices that affect final weights.
  • routing coefficient matrices R_p,r,c
    Learned per patient and per label; the quantities whose values are later interpreted as modality importance.
axioms (3)
  • domain assumption The 25 phenotypes defined by ICD-9/10 groupings of Harutyunyan et al. (2019) are clinically meaningful prediction targets.
    Used without re-validation; all multi-label results rest on this labeling.
  • domain assumption BEHRT, BioClinicalBERT and ResNet-34 produce adequate unimodal embeddings for the subsequent routing stage.
    Encoders are frozen or lightly fine-tuned; their representational quality is taken as given.
  • ad hoc to paper Training exclusively on fully trimodal stays and evaluating missing-modality only at inference is a valid robustness test.
    Stated in §4.3.3; the audit claims depend on this protocol.
invented entities (2)
  • ten discrete multimodal routes (R1∪R2∪R3) no independent evidence
    purpose: Keep unimodal, directed bimodal and trimodal evidence pathways separate until the final weighted sum so each can be inspected or masked.
    The routes are architectural constructs introduced by the paper; their clinical meaning is inferred post-hoc from weight patterns.
  • route activations α_p,r and routing coefficients R_p,r,c no independent evidence
    purpose: Quantify patient-specific expression and label-specific importance of each route.
    Learned scalars whose product is treated as an “effective weight”; no external measurement validates them as true causal contributions.

pith-pipeline@v1.1.0-grok45 · 22326 in / 2888 out tokens · 27627 ms · 2026-07-14T01:15:51.418484+00:00 · methodology

0 comments
read the original abstract

Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.

Figures

Figures reproduced from arXiv: 2607.09982 by Eric C. Larson, Mehak Gupta, Nikkie Hooman, Zhongjie Wu.

Figure 1
Figure 1. Figure 1: Architecture of the proposed multimodal routing framework. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Routing activation and effective weight across all phenotypes. The top heatmap shows the average primary activation [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Routing activation and effective weight for mortality. The top heatmap shows the average primary activation ( [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The figure illustrates local interpretability for three patients, showing how routing weights are distributed across the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Routing coefficient heatmaps across all phenotypes. Three heatmaps correspond to the FULL setting and missing [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Routing coefficient heatmaps for mortality. Three heatmaps correspond to the FULL setting and missing-modality [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 2 linked inside Pith

  1. [1]

    Abdulrahman Al-Dailami, Hulin Kuang, and Jianxin Wang. 2025. Multimodal Representation Learning Based on Personalized Graph-Based Fusion for Mortality Prediction Using Electronic Medical Records.Big Data Mining and Analytics8, 4 (2025), 933–950

    2025

  2. [2]

    Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT em- beddings. InProceedings of the 2nd clinical natural language processing workshop. 72–78

    2019

  3. [3]

    Hejie Cui, Xinyu Fang, Ran Xu, Xuan Kan, Joyce C Ho, and Carl Yang. 2024. Multimodal fusion of ehr in structures and semantics: Integrating clinical records and notes with hypergraph and llm.arXiv preprint arXiv:2403.08818(2024)

    Pith/arXiv arXiv 2024

  4. [4]

    Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. 2024. Fuse- moe: Mixture-of-experts transformers for fleximodal fusion.Advances in Neural Information Processing Systems37 (2024), 67850–67900

    2024

  5. [5]

    Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. 2019. Multitask learning and benchmarking with clinical time series data.Scientific data6, 1 (2019), 96

    2019

  6. [6]

    Nasir Hayat, Krzysztof J Geras, and Farah E Shamout. 2022. MedFuse: Multi- modal fusion with clinical time-series data and chest X-ray images. InMachine Learning for Healthcare Conference. PMLR, 479–503

    2022

  7. [7]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition. 770–778

    2016

  8. [8]

    Nikkie Hooman, Zhongjie Wu, Eric C Larson, and Mehak Gupta. 2025. Equitable Electronic Health Record Prediction with FAME: Fairness-Aware Multimodal Embedding.arXiv preprint arXiv:2506.13104(2025)

    arXiv 2025

  9. [9]

    Shih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and Matthew P Lungren. 2020. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines.NPJ digital medicine3, 1 (2020), 136

    2020

  10. [10]

    Alistair Johnson, Matthew Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. 2024. MIMIC-CXR-JPG - chest radiographs with structured labels.PhysioNet(March 2024). doi:10.13026/jsn5-t979 Version 2.1.0

    doi:10.13026/jsn5-t979 2024

  11. [11]

    Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark

  12. [12]

    MIMIC-IV-Note: Deidentified free-text clinical notes.PhysioNet(Jan. 2023). doi:10.13026/1n74-ne17 Version 2.2

    doi:10.13026/1n74-ne17 2023

  13. [13]

    Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al

  14. [14]

    MIMIC-IV, a freely accessible electronic health record dataset.Scientific data10, 1 (2023), 1

    2023

  15. [15]

    Kwanhyung Lee, Soojeong Lee, Sangchul Hahn, Heejung Hyun, Edward Choi, Byungeun Ahn, and Joohyung Lee. 2023. Learning missing modal electronic health records with unified multi-modal data embedding and modality-aware attention. InMachine Learning for Healthcare Conference. PMLR, 423–442

    2023

  16. [16]

    Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi- Khorshidi. 2020. BEHRT: transformer for electronic health records.Scientific reports10, 1 (2020), 7155

    2020

  17. [17]

    Weimin Lyu, Xinyu Dong, Rachel Wong, Songzhu Zheng, Kayley Abell-Hart, Fusheng Wang, and Chao Chen. 2023. A multimodal transformer: Fusing clinical notes with structured ehr data for interpretable in-hospital mortality prediction. InAMIA Annual Symposium Proceedings, Vol. 2022. 719

    2023

  18. [18]

    Aniket Patil, Viraj Patil, Sangram Sankpal, Tanuja S Patankar, and Harsha Bhute

  19. [19]

    Multimodal Decision Support System for Improved Diagnosis and Health- care Decision Making.Journal of Biology and Health Science(2025)

    2025

  20. [20]

    Yucheng Ruan, Daniel J Tan, See-Kiong Ng, Ling Huang, and Mengling Feng. 2025. Towards accurate and reliable ICU outcome prediction: a multimodal learning framework based on belief function theory using structured EHRs and free-text notes.Journal of Healthcare Informatics Research(2025), 1–42

    2025

  21. [21]

    Di Sun, Lubomir Hadjiiski, John Gormley, Heang-Ping Chan, Elaine Caoili, Richard Cohan, Ajjai Alva, Grace Bruno, Rada Mihalcea, Chuan Zhou, et al

  22. [22]

    Outcome prediction using multi-modal information: integrating large language model-extracted clinical information and image analysis.Cancers16, 13 (2024), 2402

    2024

  23. [23]

    Yao-Hung Hubert Tsai, Martin Ma, Muqiao Yang, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Multimodal routing: Improving local and global interpretability of multimodal language analysis. InProceedings of the 2020 con- ference on empirical methods in natural language processing (emnlp). 1823–1833

    2020

  24. [24]

    Muyu Wang, Shiyu Fan, Yichen Li, Zhongrang Xie, and Hui Chen. 2025. Missing- modality enabled multi-modal fusion architecture for medical data.Journal of Biomedical Informatics164 (2025), 104796

    2025

  25. [25]

    Xinkun Wang, Yifang Wang, Senwei Liang, Feilong Tang, Chengzhi Liu, Ming Hu, Chao Hu, Junjun He, Zongyuan Ge, and Imran Razzak. 2025. Robust Multimodal Learning for Ophthalmic Disease Grading via Disentangled Representation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 447–456

    2025

  26. [26]

    Xiaoyang Wang and Christopher Yang. 2025. MoE-Health: A Mixture of Experts Framework for Robust Multimodal Healthcare Prediction. InProceedings of the 16th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics. 1–9

    2025

  27. [27]

    Yi Wang and Weihua Li. 2025. Integrating Multimodal EHR Data for Mortality Prediction in ICU Sepsis Patients.Statistics in Medicine44, 10-12 (2025), e70060

    2025

  28. [28]

    Torsten Wörtwein, Lisa Sheeber, Nicholas Allen, Jeffrey Cohn, and Louis-Philippe Morency. 2022. Beyond additive fusion: Learning non-additive multimodal inter- actions. InFindings of the Association for Computational Linguistics: EMNLP 2022. 4681–4696

    2022

  29. [29]

    Wenfang Yao, Kejing Yin, William K Cheung, Jia Liu, and Jing Qin. 2024. Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. InProceedings of the AAAI conference on artificial intelligence, Vol. 38. 16416–16424

    2024

  30. [30]

    Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. 2024. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts.Advances in Neural Information Processing Systems37 (2024), 98782–98805

    2024

  31. [31]

    Yuyin Zhou, Shih-Cheng Huang, Jason Alan Fries, Alaa Youssef, Timothy J Amrhein, Marcello Chang, Imon Banerjee, Daniel Rubin, Lei Xing, Nigam Shah, et al. 2021. Radfusion: Benchmarking performance and fairness for multimodal pulmonary embolism detection from ct and ehr.arXiv preprint arXiv:2111.11665 (2021). CHASE, Month DD–DD, 2026, City, Country Nikkie ...

    Pith/arXiv arXiv 2021