{"id":"4fcb6ae9-912f-47d0-a75f-fd3fcf7c8bb5","arxiv_id":"2412.17832","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multimodal transformer trained on EHR, accelerometer, facial action unit, and environmental data improves ICU acuity transition and status prediction compared with an EHR-only baseline in a single-center study.","lead":"MANGO fuses four data streams, electronic health records, wearable motion sensors, facial expressions, and room environment data, to predict how ICU patients' conditions change over four-hour windows. The model claims higher accuracy than an EHR-only system, but the supporting evidence is weakened by sparse sensor coverage and a private dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Non-ignorable modality missingness concentrated in the first 7 ICU days may confound the reported AUROC gains; the paper does not stratify by modality presence or day of stay.","rationale":"The reader's weakest_assumption correctly identifies missingness as the central threat. I agree because the paper's own Discussion confirms sensor data were collected for at most 7 days, so windows with sensor data are a time-selected, patient-selected subset. The masked-attention design means the All model exploits sensors exactly on that subset, making the comparison with the EHR baseline, which sees all windows, unfair unless the subset is equally predictable. A concrete, low-cost check is to repeat the evaluation restricted to windows where all modalities are present and to the complement; if the gain does not survive, the headline claim is not established. The Table I accelerometer percentage inconsistency (3,634 divided by 33,779 equals 10.8%, not the reported 6.8%) further suggests the missingness description needs verification. Thus the CONDITIONAL verdict is appropriate; no change is needed.","tokens_in":11737,"tokens_out":6245,"duration_ms":56432,"concrete_test":"Restrict the test set to observation windows where all four modalities are present (mask equals 1111) and compare the EHR baseline versus All-modality AUROC on exactly this subset, and also on the complement (sensor-absent windows). If the multimodal gain disappears or reverses on the sensor-present subset, or if the gain is fully explained by day of stay, the central claim of modality-driven improvement is not supported. In addition, recompute the Table I accelerometer percentage and verify the window-level modality counts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that modality availability is non-ignorable and time-structured, so the reported AUROC gain of the All-modality model over the EHR baseline may reflect selection of easier windows rather than true modality signal. Table I shows facial, accelerometer, and environmental features are present in only 11%, roughly 7–11%, and 14% of development windows, respectively, and the Discussion states these modalities were collected for only the first 7 days or until transfer or discharge. Consequently, sensor-bearing windows are concentrated early in the ICU stay, exactly when acuity transitions are most frequent. Because the masked transformer can use sensors when present and fall back to EHR otherwise, the All model is effectively evaluated on a mixture: windows with extra information (early stay) and windows without. If early-stay windows are easier to classify, the apparent improvement over the EHR-only model, which must handle all windows, is confounded by time since admission and by patient factors such as shorter stays or ability to wear devices. The paper does not stratify results by modality presence or by day of stay, nor does it perform a missing-data sensitivity analysis. Additionally, Table I contains a numerical inconsistency: the development-set accelerometer count 3,634 out of 33,779 is 10.8%, not the reported 6.8%, while the test-set figure 634 out of 8,349 is 7.6% and consistent, suggesting the missingness statistics require verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MANGO, a multimodal transformer with masked self-attention for predicting ICU acuity status, transitions in acuity status, and the need for life-sustaining therapies. The authors collected a new dataset, ICU-Multimodal, from 310 patients at a single center, combining EHR data, wearable accelerometer data, facial action units extracted from video, and ambient light/sound data. They encode each modality into a 128-dimensional embedding, apply masked multi-head self-attention to handle missing modalities, and compare six modality combinations against an EHR-only baseline. The headline results are AUROCs of 0.76 for transition prediction and 0.82 for acuity status with the all-modality model, versus 0.71 and 0.70 for the EHR baseline.","tokens_in":11998,"tokens_out":7032,"duration_ms":61452,"significance":"If the central claim holds, the paper makes a useful contribution: it is among the first to integrate these four modalities for ICU acuity prediction, and the masking-based fusion approach is a practical way to handle incomplete modality data at inference time. The dataset itself, with its detailed modality and label distributions, is potentially valuable to the community, and the integrated-gradient analysis provides some interpretability. The main weakness is that the reported improvements over the EHR baseline may be confounded by non-ignorable, time-structured modality missingness, and the statistical significance testing as described is not valid. These issues are load-bearing because the abstract's quantitative claims rest on the comparison between the all-modality and EHR-only models.","major_comments":[{"comment":"The paper does not establish that the apparent benefit of the All-modality model over the EHR baseline is due to modality signal rather than to non-ignorable, time-structured missingness. Table I shows facial AU, accelerometer, and environmental features are present in only 11%, 6.8% (reported), and 13.9% of development observation windows, and the Discussion states these modalities were collected for only the first seven days or until transfer/discharge. Because the masked transformer can use sensors when present and fall back to EHR otherwise, the All model is evaluated on a mixture of early windows (with extra inputs) and later windows (EHR only), whereas the EHR baseline is evaluated on all windows uniformly. If acuity transitions are easier to predict early in the stay, the reported AUROC gains of 0.05-0.12 over baseline are confounded by time since admission and by patient factors associated with sensor availability (e.g., shorter stays, ability to wear devices). Please stratify results by day of stay and by modality presence, and add a sensitivity analysis restricted to windows in which all modalities are present, to demonstrate that the gain is not an artifact of window selection.","section":"II.A.4 and Discussion (Limitations)"},{"comment":"The significance testing is not valid as reported. The paper states that 'we compared all metric values using the Wilcoxon rank-sum test' after computing a 100-iteration bootstrap for each metric. If the 100 bootstrap values per model are treated as the samples for the rank-sum test, the test is invalid because bootstrap resamples are not independent replicates and the effective sample size is an artifact of the resampling procedure. The p-values reported in Tables V and VI therefore do not support the asterisk claims of statistical improvement. Use a paired test on the same test set (e.g., the DeLong test for AUROC, or a bootstrap/permutation test of the AUROC difference) and correct for multiple comparisons across the ten sub-tasks and five model variants.","section":"II.D.2"},{"comment":"The reported 'Overall' AUROC values (e.g., 0.76 and 0.82 in Tables V and VI) are used in the abstract as headline results but are never defined. It is unclear whether they are macro-averages of the per-class AUROCs, averages over tasks, or computed on pooled predictions. Without a definition, the main quantitative claims are not reproducible. Please define the aggregation rule and report per-class values consistently. Relatedly, the model selection rule is underspecified: for each experimental setup, 'the optimal model was selected based on the highest AUROC across three critical tasks,' but the paper does not state whether selection was made on the validation set, nor whether the maximum, average, or some other function of the three AUROCs was used. This can introduce selection bias and makes the per-task test results in Tables V and VI difficult to interpret.","section":"III.B.1 and Tables V-VI"}],"minor_comments":[{"comment":"The accelerometer feature count in the development set appears inconsistent: 3,634 out of 33,779 observation windows is 10.8%, not 6.8% as reported. Please verify the counts and percentages for all modalities, since these figures are central to assessing missingness.","section":"Table I"},{"comment":"There is a typo in the Abstract and Introduction: 'all four mobilities' should be 'all four modalities'.","section":"Abstract and Section I"},{"comment":"The text refers to 'APRICOT-T' in two places and to 'APRICOT-Mamba' in references [3] and [14] with different arXiv identifiers. Please clarify the relationship between APRICOT-T, APRICOT-Mamba, and the EHR encoder used in the baseline model.","section":"II.B (EHR data)"},{"comment":"The statement that the models showed 'robust classification performance in mortality prediction' is overstated given the test set contains only 3 deceased patients; the confidence intervals for the 'Deceased' row are extremely wide (e.g., 0.05-1.00). Please qualify or remove this claim.","section":"III.B.2 and Table VI"},{"comment":"The paper does not include a data availability or code availability statement. For a study whose main contribution includes a new dataset, a statement about whether the ICU-Multimodal data can be shared (or under what conditions) is needed for reproducibility.","section":"II.D.2"},{"comment":"Reference [18] contains a typo: 'AXective Computing' should be 'Affective Computing'. Also, Figure 2's subplot labels (A(a), B(a), etc.) are hard to follow; please label panels more explicitly.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is empirically plausible but currently rests on a comparison that is confounded by modality missingness, and the statistical testing appears invalid. The authors should be asked for stratified analyses and corrected testing. The reliance on in-house baseline and feature-extraction models (APRICOT-Mamba, AU extraction) is not a reason for rejection, but the editors may want to require external validation or a data-sharing statement given the single-center design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, with a caveat. MANGO builds the first ICU dataset that pairs EHR with accelerometer, facial action unit, and ambient sound/light data, and its masked self-attention fusion handles missing modalities cleanly. That part is real. The collection effort is substantial: 310 consented patients, multiple devices, manual FACS annotation, IRB approval, and a clear train/test split. The reported gains (All vs EHR baseline: 0.76 vs 0.71 for transitions; 0.82 vs 0.70 for acuity status) look plausible, and the integrated gradient analysis gives a useful sanity check. Credit is due for the engineering and the honest limitations section.\n\nThe soft spots are mostly about inference. The biggest one is non-ignorable missingness. The authors admit non-EHR data were collected for only the first seven days or until transfer/discharge. That means sensor-bearing windows are concentrated early in the stay, and the masked transformer can use the presence of sensors as a proxy for 'early stay.' If early windows are easier to classify—likely, since acuity transitions cluster there—the AUROC advantage over the EHR-only model is confounded. The paper does not stratify by day of stay or modality presence, nor does it do a missing-data sensitivity analysis. That undermines the central claim.\n\nSecond, the baseline is their own APRICOT-Mamba model, not an independent state-of-the-art comparator. That is fine as a starting point, but 'significantly improved' needs a stronger reference.\n\nMinor but telling: Table I lists accelerometer features as 6.8% of development windows, yet 3,634/33,779 is 10.8%. The test set figure is consistent, so it looks like a typo, but it signals a sloppy verification pass.\n\nNo code or data release, and multiple Wilcoxon tests without adjustment, are also noted, though privacy and clinical data partly excuse the former.\n\nBottom line: as a dataset-and-method paper, it deserves a serious referee. The problem is real and the fusion idea is reusable. But the headline performance claim should be treated as provisional until the authors demonstrate that the gain survives adjustment for when sensors are present and how long the patient has been in the ICU. If they add that analysis, it becomes a strong contribution; if they can't, the multimodal advantage may largely vanish.","headline":"MANGO builds a genuinely new multimodal ICU dataset and a sensible masked-fusion model, but the headline AUROC gains are likely confounded by non-ignorable missingness concentrated in the first seven days.","tokens_in":12585,"tokens_out":2328,"would_cite":true,"duration_ms":20890,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fusing EHR data with accelerometry, facial action units, and ambient sensors via masked self-attention improves ICU acuity prediction over EHR-only models, reaching AUROC 0.76 for transitions and 0.82 for acuity…","keywords":["Intensive care unit","Multimodal fusion","Patient acuity","Transformer","Masked self-attention","Accelerometry","Facial action units","Ambient sensors"],"falsifier":"Re-run the modality comparisons on only the windows where all four modalities are present, or after weighting by the propensity of sensor availability; if the all-modality AUROC advantage over EHR-only shrinks to within the confidence-interval overlap, the claim that the extra modalities carry predictive signal is not supported. A cleaner test would be a prospective collection with protocolized continuous sensors on every enrolled patient.","tokens_in":11534,"feed_emoji":"🏥","tokens_out":7508,"duration_ms":65678,"temperature":0.7,"pith_summary":"The paper aims to establish that adding three non-EHR sensing modalities to structured electronic health records improves prediction of ICU patient acuity. It introduces MANGO, a transformer whose masked self-attention fuses four 128-dimensional modality embeddings, and a new dataset with EHR, wrist/ankle accelerometry, facial action units from video, and room light and sound. In the paper's experiments the four-modality model raised the area under the ROC curve from 0.71 to 0.76 for transitions and the need for life-sustaining therapy, and from 0.70 to 0.82 for acuity status classification. The implied payoff is that bedside monitoring of movement, facial cues, and environment can complement chart data, and that the model still works when some modalities are absent.","feed_headline":"Four-stream ICU model lifts acuity transition AUROC to 0.76","feed_subtitle":"Adding wearables, facial cues, and room sensors to EHR data improved forecasts of who will destabilize.","key_machinery":"The load-bearing mechanism is masked multi-head self-attention over a short sequence of modality embeddings. Each four-hour observation window yields one 128-dimensional vector per available modality; a binary mask turns missing modalities into $-$\\infty attention logits, so the transformer learns interactions only from the streams actually present. The EHR embedding is produced by a pretrained EHR-only acuity encoder, and the other three modalities by small 1D convolutional networks, after which a shared backbone and ten classification heads output transition and status predictions.","core_discovery":"The central claim is that multimodal integration, not any single non-EHR stream, drives the best acuity predictions. The all-modality MANGO outperformed the EHR-only baseline in most transition sub-tasks and in overall acuity status, with the reported best AUROC of 0.76 (95% CI 0.72-0.79) for transitions and the need for life-sustaining therapy and 0.82 (95% CI 0.69-0.89) for acuity status. The paper also found that EHR features dominate the integrated-gradient attributions, with accelerometer movement orientation and facial action units AU43 (eyes close) and AU26 (jaw drop) among the strongest non-EHR contributors, while environmental features contributed little. It positions MANGO as the first model to combine these four modalities for ICU outcome prediction.","pith_inferences":["Because face, accelerometer, and environmental data appear in only about 11%, 7%, and 14% of observation windows respectively, the apparent AUROC gains may be confounded by which patients have sensors; a missingness-aware or matched analysis would test whether the modality signal is causal.","The four-hour aggregation into statistical features likely washes out rapid facial or motion dynamics, so raw-signal encoders are a natural next step and might yield larger gains.","The optional-modality architecture suggests a pragmatic deployment path: start with EHR-only predictions and upgrade to multimodal whenever streams are available, with the same model weights.","The same masked-fusion approach could transfer to other sparse clinical monitoring contexts, such as step-down units or post-discharge home monitoring, where data availability varies across patients and time."],"forward_implications":["The EHR-plus-accelerometer and all-modality models both reached 0.82 AUROC for acuity status, showing that wearable motion can substitute for some chart-derived signal when it is present.","The masking design allows the same model to run with any subset of modalities, which matches real ICU conditions where facial video and sensors are often unavailable.","Integrated gradients point to movement orientation and specific facial actions as interpretable bedside signals, not just black-box correlates.","Environmental light and noise did not meaningfully improve acuity prediction, suggesting room sensors may be more useful for delirium risk than for acuity.","The gains over the EHR baseline were statistically significant for several transition classes, indicating the non-EHR streams are not redundant."],"supporting_citations":[{"why":"supplies the EHR encoder and the temporal and static feature preprocessing that MANGO reuses for the EHR modality","marker":"[14]"},{"why":"defines the computable acuity phenotypes used to create stable/unstable and life-sustaining-therapy transition labels","marker":"[13]"},{"why":"supports the association of facial cues with patient clinical status and motivates the facial-action-unit modality","marker":"[7]"},{"why":"provides the accelerometer feature set, including vector magnitude, angle, and dominant frequency","marker":"[15]"},{"why":"provides the ambient light and sound pressure features used for the environmental modality","marker":"[22]"},{"why":"introduces the masked self-attention mechanism that lets the model ignore missing modalities","marker":"[24]"},{"why":"gives the integrated-gradients method used to rank feature importance across modalities","marker":"[25]"}],"fun_headline_variants":["Four data streams improve ICU acuity forecasts over EHR alone","MANGO fuses EHR, wearables, face video, and sensors for ICU acuity","Wearables and facial cues boost ICU acuity prediction accuracy","Multimodal AI for ICU acuity achieves AUROC 0.82 on status","Adding video and wearables to EHR improves ICU acuity AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that observation windows with sensor data are representative of the whole ICU population, so that better AUROC comes from the modalities themselves rather than from patient characteristics tied to sensor availability.","fun_headline_variants_meta":{"raw":{"variants":["Four data streams improve ICU acuity forecasts over EHR alone","MANGO fuses EHR, wearables, face video, and sensors for ICU acuity","Wearables and facial cues boost ICU acuity prediction accuracy","Multimodal AI for ICU acuity achieves AUROC 0.82 on status","Adding video and wearables to EHR improves ICU acuity AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001134,"raw_usage":{"total_tokens":4744,"prompt_tokens":1010,"completion_tokens":3734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":3643}},"tokens_in":626,"tokens_out":3734,"duration_ms":22505,"temperature":1.0,"reasoning_tokens":3643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:45:59.759695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the modality comparisons on only the windows where all four modalities are present, or after weighting by the propensity of sensor availability; if the all-modality AUROC advantage over EHR-only shrinks to within the confidence-interval overlap, the claim that the extra modalities carry predictive signal is not supported. A cleaner test would be a prospective collection with protocolized continuous sensors on every enrolled patient.","supporting_citations":[],"review_version":1}