Pith. sign in

REVIEW 3 major objections 6 minor 4 references

MANGO: Multimodal Acuity traNsformer for intelliGent ICU Outcomes

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fusing EHR data with accelerometry, facial action units, and ambient sensors via masked self-attention improves ICU acuity prediction over EHR-only models, reaching AUROC 0.76 for transitions and 0.82 for acuity…

desk verdict MANGO builds a genuinely new multimodal ICU dataset and a sensible masked-fusion model, but the headline AUROC gains are likely confounded by non-ignorable missingness concentrated in the first seven days. read the letter →

arxiv 2412.17832 v1 pith:GGWZEX6W submitted 2024-12-13 eess.SP cs.AIcs.LG

Jiaqing Zhang , Miguel Contreras , Sabyasachi Bandyopadhyay , Andrea Davidson , Jessica Sena , Yuanfang Ren , Ziyuan Guan , Tezcan Ozrazgat-Baslanti
show 4 more authors
Tyler J. Loftus Subhash Nerella Azra Bihorac Parisa Rashidi
This is my paper · ORCID
classification eess.SPcs.AIcs.LG
keywords IntensivecareunitMultimodalfusionPatientacuityTransformerMaskedself-attentionAccelerometryFacialactionunitsAmbientsensors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that adding three non-EHR sensing modalities to structured electronic health records improves prediction of ICU patient acuity. It introduces MANGO, a transformer whose masked self-attention fuses four 128-dimensional modality embeddings, and a new dataset with EHR, wrist/ankle accelerometry, facial action units from video, and room light and sound. In the paper's experiments the four-modality model raised the area under the ROC curve from 0.71 to 0.76 for transitions and the need for life-sustaining therapy, and from 0.70 to 0.82 for acuity status classification. The implied payoff is that bedside monitoring of movement, facial cues, and environment can complement chart data, and that the model still works when some modalities are absent.

What carries the argument

The load-bearing mechanism is masked multi-head self-attention over a short sequence of modality embeddings. Each four-hour observation window yields one 128-dimensional vector per available modality; a binary mask turns missing modalities into $-$\infty attention logits, so the transformer learns interactions only from the streams actually present. The EHR embedding is produced by a pretrained EHR-only acuity encoder, and the other three modalities by small 1D convolutional networks, after which a shared backbone and ten classification heads output transition and status predictions.

What would settle it

Re-run the modality comparisons on only the windows where all four modalities are present, or after weighting by the propensity of sensor availability; if the all-modality AUROC advantage over EHR-only shrinks to within the confidence-interval overlap, the claim that the extra modalities carry predictive signal is not supported. A cleaner test would be a prospective collection with protocolized continuous sensors on every enrolled patient.

Watch

Extended reading notes

Core claim

The central claim is that multimodal integration, not any single non-EHR stream, drives the best acuity predictions. The all-modality MANGO outperformed the EHR-only baseline in most transition sub-tasks and in overall acuity status, with the reported best AUROC of 0.76 (95% CI 0.72-0.79) for transitions and the need for life-sustaining therapy and 0.82 (95% CI 0.69-0.89) for acuity status. The paper also found that EHR features dominate the integrated-gradient attributions, with accelerometer movement orientation and facial action units AU43 (eyes close) and AU26 (jaw drop) among the strongest non-EHR contributors, while environmental features contributed little. It positions MANGO as the first model to combine these four modalities for ICU outcome prediction.

Load-bearing premise

The load-bearing assumption is that observation windows with sensor data are representative of the whole ICU population, so that better AUROC comes from the modalities themselves rather than from patient characteristics tied to sensor availability.

Editorial extensions

If this is right

  • The EHR-plus-accelerometer and all-modality models both reached 0.82 AUROC for acuity status, showing that wearable motion can substitute for some chart-derived signal when it is present.
  • The masking design allows the same model to run with any subset of modalities, which matches real ICU conditions where facial video and sensors are often unavailable.
  • Integrated gradients point to movement orientation and specific facial actions as interpretable bedside signals, not just black-box correlates.
  • Environmental light and noise did not meaningfully improve acuity prediction, suggesting room sensors may be more useful for delirium risk than for acuity.
  • The gains over the EHR baseline were statistically significant for several transition classes, indicating the non-EHR streams are not redundant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because face, accelerometer, and environmental data appear in only about 11%, 7%, and 14% of observation windows respectively, the apparent AUROC gains may be confounded by which patients have sensors; a missingness-aware or matched analysis would test whether the modality signal is causal.
  • The four-hour aggregation into statistical features likely washes out rapid facial or motion dynamics, so raw-signal encoders are a natural next step and might yield larger gains.
  • The optional-modality architecture suggests a pragmatic deployment path: start with EHR-only predictions and upgrade to multimodal whenever streams are available, with the same model weights.
  • The same masked-fusion approach could transfer to other sparse clinical monitoring contexts, such as step-down units or post-discharge home monitoring, where data availability varies across patients and time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents MANGO, a multimodal transformer with masked self-attention for predicting ICU acuity status, transitions in acuity status, and the need for life-sustaining therapies. The authors collected a new dataset, ICU-Multimodal, from 310 patients at a single center, combining EHR data, wearable accelerometer data, facial action units extracted from video, and ambient light/sound data. They encode each modality into a 128-dimensional embedding, apply masked multi-head self-attention to handle missing modalities, and compare six modality combinations against an EHR-only baseline. The headline results are AUROCs of 0.76 for transition prediction and 0.82 for acuity status with the all-modality model, versus 0.71 and 0.70 for the EHR baseline.

Significance. If the central claim holds, the paper makes a useful contribution: it is among the first to integrate these four modalities for ICU acuity prediction, and the masking-based fusion approach is a practical way to handle incomplete modality data at inference time. The dataset itself, with its detailed modality and label distributions, is potentially valuable to the community, and the integrated-gradient analysis provides some interpretability. The main weakness is that the reported improvements over the EHR baseline may be confounded by non-ignorable, time-structured modality missingness, and the statistical significance testing as described is not valid. These issues are load-bearing because the abstract's quantitative claims rest on the comparison between the all-modality and EHR-only models.

major comments (3)
  1. [II.A.4 and Discussion (Limitations)] The paper does not establish that the apparent benefit of the All-modality model over the EHR baseline is due to modality signal rather than to non-ignorable, time-structured missingness. Table I shows facial AU, accelerometer, and environmental features are present in only 11%, 6.8% (reported), and 13.9% of development observation windows, and the Discussion states these modalities were collected for only the first seven days or until transfer/discharge. Because the masked transformer can use sensors when present and fall back to EHR otherwise, the All model is evaluated on a mixture of early windows (with extra inputs) and later windows (EHR only), whereas the EHR baseline is evaluated on all windows uniformly. If acuity transitions are easier to predict early in the stay, the reported AUROC gains of 0.05-0.12 over baseline are confounded by time since admission and by patient factors associated with sensor availability (e.g., shorter stays, ability to wear devices). Please stratify results by day of stay and by modality presence, and add a sensitivity analysis restricted to windows in which all modalities are present, to demonstrate that the gain is not an artifact of window selection.
  2. [II.D.2] The significance testing is not valid as reported. The paper states that 'we compared all metric values using the Wilcoxon rank-sum test' after computing a 100-iteration bootstrap for each metric. If the 100 bootstrap values per model are treated as the samples for the rank-sum test, the test is invalid because bootstrap resamples are not independent replicates and the effective sample size is an artifact of the resampling procedure. The p-values reported in Tables V and VI therefore do not support the asterisk claims of statistical improvement. Use a paired test on the same test set (e.g., the DeLong test for AUROC, or a bootstrap/permutation test of the AUROC difference) and correct for multiple comparisons across the ten sub-tasks and five model variants.
  3. [III.B.1 and Tables V-VI] The reported 'Overall' AUROC values (e.g., 0.76 and 0.82 in Tables V and VI) are used in the abstract as headline results but are never defined. It is unclear whether they are macro-averages of the per-class AUROCs, averages over tasks, or computed on pooled predictions. Without a definition, the main quantitative claims are not reproducible. Please define the aggregation rule and report per-class values consistently. Relatedly, the model selection rule is underspecified: for each experimental setup, 'the optimal model was selected based on the highest AUROC across three critical tasks,' but the paper does not state whether selection was made on the validation set, nor whether the maximum, average, or some other function of the three AUROCs was used. This can introduce selection bias and makes the per-task test results in Tables V and VI difficult to interpret.
minor comments (6)
  1. [Table I] The accelerometer feature count in the development set appears inconsistent: 3,634 out of 33,779 observation windows is 10.8%, not 6.8% as reported. Please verify the counts and percentages for all modalities, since these figures are central to assessing missingness.
  2. [Abstract and Section I] There is a typo in the Abstract and Introduction: 'all four mobilities' should be 'all four modalities'.
  3. [II.B (EHR data)] The text refers to 'APRICOT-T' in two places and to 'APRICOT-Mamba' in references [3] and [14] with different arXiv identifiers. Please clarify the relationship between APRICOT-T, APRICOT-Mamba, and the EHR encoder used in the baseline model.
  4. [III.B.2 and Table VI] The statement that the models showed 'robust classification performance in mortality prediction' is overstated given the test set contains only 3 deceased patients; the confidence intervals for the 'Deceased' row are extremely wide (e.g., 0.05-1.00). Please qualify or remove this claim.
  5. [II.D.2] The paper does not include a data availability or code availability statement. For a study whose main contribution includes a new dataset, a statement about whether the ICU-Multimodal data can be shared (or under what conditions) is needed for reproducibility.
  6. [References] Reference [18] contains a typo: 'AXective Computing' should be 'Affective Computing'. Also, Figure 2's subplot labels (A(a), B(a), etc.) are hard to follow; please label panels more explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: multimodal gains are empirical; self-citations are not load-bearing.

full rationale

The paper's central claim is empirical and self-contained: Table V reports that the All-modality model reaches AUROC 0.76 (0.72-0.79) versus 0.71 (0.67-0.75) for the EHR baseline on transition prediction, and Table VI reports 0.82 (0.69-0.89) versus 0.70 (0.61-0.85) for status prediction. There is no equation in which the multimodal output is defined in terms of the target labels, and no fitted parameter is later relabeled as a prediction. The APRICOT-Mamba encoder (refs [3] and [14]) and the Ren et al. computable phenotypes (ref [13]) are self-citations, but they serve as a feature extractor and label definition, respectively; the modality-fusion comparison is measured on a held-out test split, so the central claim does not reduce to those citations. The Discussion explicitly acknowledges that non-EHR modalities were collected for only seven days or until transfer/discharge (Section IV), which is a genuine missingness and confounding risk because sensor-bearing windows are concentrated early in the ICU stay, but that is a validity threat rather than a circular derivation. Table I also contains a numerical inconsistency (development accelerometer count 3,634/33,779 is 10.8%, not the reported 6.8%), but this is a reporting error, not circularity. No load-bearing step reduces by construction to its own input.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical and depends mainly on the availability and informativeness of the four modalities, the validity of the acuity phenotype labels, and the transferability of prior models from the same group. No new physical or conceptual entities are introduced. The free parameters are standard architectural and preprocessing choices rather than scientifically derived constants.

free parameters (5)
  • Observation window length = 4 hours
    The 4-hour window is chosen by hand for feature aggregation and label generation; the paper acknowledges in the Discussion that large windows may obscure temporal dynamics.
  • Embedding dimension = 128
    All modality encoders produce 128-dimensional embeddings; this is an architectural choice, not fitted to data.
  • MMSA blocks and shared FC layers = 2 blocks, 3 layers
    The number of transformer blocks and fully connected layers is chosen without systematic tuning; results may depend on these choices.
  • Early stopping patience = 10 epochs
    A standard regularization hyperparameter chosen by hand; affects which model checkpoint is selected.
  • Model selection rule = highest AUROC across three critical tasks
    The optimal model is selected based on AUROC for stable-to-unstable, no-MV-to-MV, and no-VP-to-VP; this selection rule is applied across six configurations, introducing multiple-testing concerns.
assumptions (5)
  • domain assumption Facial action units, accelerometer features, and environmental factors contain signal relevant to ICU acuity.
    This is asserted from prior literature (refs [6,7,22]) and supported by feature importance analysis, but the direct relationship to acuity is not established independently in this paper. Invoked in Section II.B and the Discussion.
  • domain assumption The computable phenotypes of acuity (stable/unstable, MV, VP, BT, CRRT) defined by Ren et al. are valid proxies for clinical acuity.
    The labels are generated using criteria from a prior paper by the same group (ref [13]); the validity of these phenotypes is not independently verified here. Invoked in Section II.A.5.
  • domain assumption The APRICOT-Mamba model provides a suitable and transferable EHR representation.
    The EHR encoder is initialized with weights from APRICOT-Mamba (refs [3,14]), a model developed by the same group on possibly overlapping data. The assumption that this representation is appropriate for the new task is not tested against other EHR encoders. Invoked in Section II.B.
  • standard math Masked multi-head self-attention (MMSA) can effectively combine modality embeddings when some modalities are missing.
    The MMSA mechanism (Eq. 1) is a standard extension of transformer attention with additive masks; this is a reasonable architectural assumption, not a substantive domain claim.
  • domain assumption The four-hour observation window is an appropriate temporal resolution for capturing acuity-relevant changes.
    The paper itself notes in the Discussion that the 4-hour window may average out important dynamics, so this assumption is recognized as potentially limiting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MANGO: Multimodal Acuity traNsformer for intelliGent ICU Outcomes." pith.science (2026). https://pith.science/paper/GGWZEX6W

@misc{pith2026241217832,
  author       = {Pith},
  title        = {Pith review of: MANGO: Multimodal Acuity traNsformer for intelliGent ICU Outcomes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGWZEX6W}},
  note         = {Machine review of arXiv:2412.17832}
}
read the original abstract

Estimation of patient acuity in the Intensive Care Unit (ICU) is vital to ensure timely and appropriate interventions. Advances in artificial intelligence (AI) technologies have significantly improved the accuracy of acuity predictions. However, prior studies using machine learning for acuity prediction have predominantly relied on electronic health records (EHR) data, often overlooking other critical aspects of ICU stay, such as patient mobility, environmental factors, and facial cues indicating pain or agitation. To address this gap, we present MANGO: the Multimodal Acuity traNsformer for intelliGent ICU Outcomes, designed to enhance the prediction of patient acuity states, transitions, and the need for life-sustaining therapy. We collected a multimodal dataset ICU-Multimodal, incorporating four key modalities, EHR data, wearable sensor data, video of patient's facial cues, and ambient sensor data, which we utilized to train MANGO. The MANGO model employs a multimodal feature fusion network powered by Transformer masked self-attention method, enabling it to capture and learn complex interactions across these diverse data modalities even when some modalities are absent. Our results demonstrated that integrating multiple modalities significantly improved the model's ability to predict acuity status, transitions, and the need for life-sustaining therapy. The best-performing models achieved an area under the receiver operating characteristic curve (AUROC) of 0.76 (95% CI: 0.72-0.79) for predicting transitions in acuity status and the need for life-sustaining therapy, while 0.82 (95% CI: 0.69-0.89) for acuity status prediction...

Figures

Figures reproduced from arXiv: 2412.17832 by the authors.

Figure 1
Figure 1. The workflow of the proposed model. We employed different models to encode the respective modalities into four 𝐷 = 128 vectors. While training, the Swin Transformer weights were frozen, and other blocks were trainable. These vectors were then stacked into an 4 × 128 sequence, where the sequence length is 4, and the dimension of the embeddings is 128. Along with the mask, this sequence was fed into two multi-head sel… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 3 canonical work pages

  1. [2]

    #$!,𝑋%&&'(!,𝑋)*&'!,𝑋

    We designed a novel feature fusion strategy that makes each modality optional so that our pipeline can be easily translated to clinical care and be helpful in analyzing incomplete datasets directly. II. Methodology Notations Definition. The multimodal dataset ICU-Multimodal 𝐷 consists of 𝑁=310 patients. For each patient 𝑝!, we ensured the presence of EHR ...

    work page

  2. [3]

    Evaluation metrics: We used AUROC to evaluate the classification performance. To determine if the classification performance difference between the baseline model (only EHR features) and other configurations that included at least one additional modality during training was statistically significant, we compared all metric values using the Wilcoxon rank-s...

    work page

  3. [4]

    No VP to VP

    Experiment results: We evaluated the performance across six different experimental setups of the different combinations of modalities (EHR baseline, EHR + Accel, EHR+ Face, EHR + Env, EHR + Accel + Env, and All), as shown in Table V and Table VI. For predicting transitions in acuity status and life-sustaining therapies, the model trained on all four modal...

    arXiv 2015

  4. [17]

    Predicting risk of delirium from ambient noise and light information in the ICU

    X. Zhang et al., "A high-resolution spontaneous 3d dynamic facial expression database, " in 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), 2013: IEEE, pp. 1-6. [18] S. M. Mavadati, M. H. Mahoor, K. Bartlett, P . Trinh, and J. F . Cohn, "Disfa: A spontaneous facial action intensity database, " IEEE Tra...

    work page Pith review arXiv 2013

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.