REVIEW 3 major objections 5 minor 18 references
Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper presents a reusable, model-agnostic harness that, from any mask-aware predictor and labels, attributes clinical AI errors per modality and separates loud from silent failures when a modality is dropped.
desk verdict A useful, honestly-scoped evaluation harness for multimodal clinical models; the main gap is that the signature per-example outputs on real data aren't there yet and the loud/silent split needs threshold sensitivity. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the mask-aware predictor abstraction: a function $f(S)$ that returns a prediction when only the subset $S$ of modalities is present and the others are masked at inference. On top of it sit three derived statistics: the per-example failure taxonomy using the gate $g = \mathbb{1}[y \le \tau]$ with $\tau = 40$ and a tolerance $\delta$; the leave-one-out complementarity value $\Delta_m = \mathrm{MAE}(f(M \setminus \{m\})) - \mathrm{MAE}(f(M))$ with per-example wins; and the loud-vs-silent dropout profile, where an induced critical failure is silent if $|f - \tau| \ge \kappa$ and loud if $|f - \tau| < \kappa$, using margin from the decision boundary as a proxy for monitorabi
What would settle it
Directly compare the masked fused checkpoint against a model trained without the missing modality on the same test patients. If the masked model's per-example error pattern differs substantially from the unimodal model's, the framework's attributed modality failures are artifacts of training-distribution shift; additionally, pre-register κ and δ and see whether the loud-vs-silent labels match a calibrated confidence head.
Extended reading notes
Core claim
The paper's central claim is that per-modality failure analysis can be reduced to a model-agnostic, deployment-observable procedure. The released harness takes N modality embeddings, any prediction function that accepts a subset of modalities, and labels, and returns three views: a failure taxonomy labeling each example correct, imprecise, or critical (the clinical gate flips); a complementarity matrix built from leave-one-out mean absolute error and per-example 'win' attribution; and a loud-vs-silent dropout profile that labels an induced gate failure silent when the wrong prediction sits at or beyond a margin κ from the decision boundary. On planted synthetic ground truth, the harness reco
Load-bearing premise
The load-bearing premise is that zeroing a trained fusion branch at inference faithfully reproduces what deployment does when a modality is absent; if masking instead creates out-of-distribution inputs, the measured degradation reflects the probe's brittleness rather than the missing modality.
Editorial extensions
If this is right
- A hospital can reuse the same failure-analysis harness across successive model generations without retraining, so the evaluation outlives any single backbone or fusion architecture.
- When the expensive-to-acquire modality is missing, the framework quantifies not only the aggregate accuracy drop but how many errors are clinically critical (gate-flipping) and how many are silent and hence unmonitorable.
- The real-cohort integration test shows that an ECG-only deployment of a fused echo/ECG model nearly doubles LVEF mean absolute error (10.28 to 18.57) and lowers the EF ≤ 40% AUROC from 0.766 to 0.693.
- Because value and detectability are separate axes, a modality that is most valuable overall need not be the most dangerous to lose; the framework measures both rather than assuming they coincide.
- The narrow temporal overlap between echo and ECG studies is itself a deployment finding: the modality that is expensive to acquire at inference is also the one with the most expensive preprocessing infrastructure.
Reading between the lines
- If the branch-masking assumption holds, the same instrument could be used pre-deployment to decide which modalities are worth keeping online, not just to audit failures after a model is shipped.
- The framework's taxonomy and silent rate depend on un-fixed user thresholds κ and δ; a natural next test is to report how sensitive the loud-vs-silent split is to those thresholds, or to calibrate them against an independent confidence model.
- A head-to-head comparison with a model genuinely trained without the missing modality would separate 'the shipped model is brittle to distribution shift' from 'the modality truly carries the signal' — a distinction the paper leaves unexplored.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a model-agnostic framework for analyzing how a multimodal clinical model fails when one or more modalities are unavailable at inference. Given N modality embeddings, a mask-aware prediction function, labels, and a clinical gate, it returns (i) a per-example failure taxonomy (correct/imprecise/critical), (ii) a per-modality complementarity matrix with leave-one-out attribution, and (iii) a loud-vs-silent dropout profile that separates gate-flipping errors near the decision boundary from those far from it. The authors instantiate the framework on frozen EchoJEPA and HuBERT-ECG embeddings for LVEF regression and the EF≤40% HFrEF gate in MIMIC-IV, validate the harness against synthetic planted structure, exercise a three-modality case, and report a real-data aggregate degradation in which dropping echo approximately doubles MAE. A unit-tested harness is released. The paper candidly notes in §6.1 and §7 that the synthetic validation confirms harness mechanics rather than clinical performance and that the real-data per-example views are not yet produced.
Significance. The framework addresses a real gap: evaluation of multimodal medical models under missing modalities is usually reported as a single accuracy drop, whereas deployment decisions need to know which modality is responsible and whether failures are monitorable. The definitions in §3 are clear and the released, unit-tested harness with planted-structure tests is a concrete reusable artifact. The paper is unusually honest about the limits of its validation. However, as it stands, the only fully realized real-data output is an aggregate MAE/AUROC table from a single held-out split; the per-example taxonomy, complementarity, and loud/silent profile on real data are deferred, and the loud/silent split is parameterized by an unjustified threshold. If the threshold sensitivity and real-data per-example results are added, this could be a useful methods contribution.
major comments (3)
- [§3, failure taxonomy and loud-vs-silent split] The taxonomy and loud-vs-silent profile depend on two free thresholds that are never fixed: δ in the correct/imprecise split (|f−y|≤δ) and κ in the loud/silent split (|f−τ|<κ). No values, clinical justification, or selection rule are given, and no sensitivity analysis is reported. Consequently the reported silent rates in §6.1 (0.31±0.07 vs 0.32±0.10) and any real-data profile are not well-defined measurements: κ→0 labels everything loud and κ→∞ labels everything silent, and δ similarly controls the critical/imprecise mix. This is load-bearing because the paper's signature contribution is separating monitorable from unflagged failures. Please fix κ and δ (or provide a threshold-selection rule) and report sensitivity of the taxonomy and silent rate to them.
- [§6.3 and §7] The abstract and title promise per-example failure taxonomy, complementarity, and loud-vs-silent profile as the framework's output, but the real-data instantiation reports only aggregate MAE/AUROC for a single held-out split (Table 4), and §7 explicitly states that the per-example views on real predictions, bootstrap CIs, the unimodal-probe sweep, and fairness stratification 'are not yet included.' The headline 'dropping echo nearly doubles MAE (10.28 to 18.57)' is thus an integration test, not a validation of the paper's central per-example claims on real data. This gap should be closed by running the harness on per-example prediction dumps and reporting uncertainty, or the paper should be reframed as a framework/mechanics contribution with only an aggregate real-data illustration.
- [§6.1] The synthetic validation is circular by the authors' own admission: because the modality structure is planted, recovering it with the same leave-one-out logic used to define attribution confirms the implementation's mechanics, not that the attribution reflects true causal modality importance in real settings. Since no per-example ground truth exists in real data, the paper currently provides no external evidence for the meaningfulness of the per-example attribution. I do not count the circularity itself as an error—the authors flag it—but it means the central real-data claim rests entirely on the deferred per-example run in §7. Please make this limitation more prominent (e.g., in the abstract) or provide an additional non-circular check, such as agreement with known physiological priors or intervention-style experiments on a subset.
minor comments (5)
- [§5.2 / Table 4] The evaluation protocol says bootstrap confidence intervals are reported where per-example predictions are available, but Table 4 shows no CIs. Please clarify whether per-example predictions were available and, if so, why CIs are omitted; if they were not available, state this explicitly.
- [§3, complementarity and attribution] The per-example attribution of a 'win' is described informally ('the modality whose removal hurts that example most'). A short formal definition, e.g., based on the per-example absolute error difference under each leave-one-out mask, would make the framework reproducible without relying on the released code.
- [§6.2] The statement that 'non-trivial attribution of this kind requires N≥3' appears only in a figure caption or table note. Since this is a structural property of leave-one-out attribution, it deserves a brief formal explanation in §3 or §6.2.
- [§5.2] Zeroing a branch at inference makes the input out-of-distribution for a probe trained only on complete data. This is a reasonable way to test the deployed checkpoint, but the paper should state explicitly that the reported degradation includes this distributional brittleness, and consider a random-masking training baseline when the goal is to model realistic missing-modality deployment.
- [Throughout] Minor presentational issues: the abstract uses 'L VEF' with inconsistent spacing; Figure 1's y-axis label 'test examples' is redundant; and the statement in the abstract that 'This validates recovery of per-example attribution rather than clinical performance' should be aligned with the §6.1 language about confirming harness mechanics.
Circularity Check
Acknowledged circular synthetic validation; real-data degradation result is independent.
-
other
[Section 6.1 'Validation: the harness recovers planted structure'; echoed in the Abstract]
"This validation is circular by construction: because we plant the modality structure ourselves, checking that the harness recovers it confirms the harness’s mechanics, not that either modality is clinically superior in real patients; this is nonetheless the only place a per-example attribution ground truth exists, since no such label is available in real data."
The planted ground truth (echo strong, ECG weak, an ≈18% echo-uninformative complementary subset) is defined by exactly the quantities the harness reports (leave-one-out MAE differences, per-example wins, complementarity). A correctly implemented harness must 'recover' this structure; the exercise is a unit test of the code, not an empirical confirmation of the framework's diagnostic value. The paper explicitly says it validates mechanics only and does not use it to support real-data clinical conclusions, so the circularity is disclosed and bounded.
full rationale
The central real-data result—dropping echo raises LVEF MAE from 10.28 to 18.57 on the held-out n=245 split—is a direct measurement independent of any planted ground truth and does not reduce to framework inputs. The loud/silent split depends on unstated thresholds κ and δ, and no sensitivity analysis is given, but that is an indeterminacy/reproducibility concern, not a circular derivation. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no fitted parameter renamed as a prediction. The only identified circular step is the synthetic validation, which the paper itself flags as 'circular by construction' and scopes to validating mechanics; it is therefore a minor, non-load-bearing circularity rather than a flaw in the real-data claim. Score 2 reflects that one disclosed circular validation while the main empirical claim stands independently.
Assumptions & free parameters
free parameters (3)
- delta (imprecision tolerance)
- kappa (loud/silent margin threshold)
- Planted structure fractions (synthetic validation) =
18% complementary subset; echo strong, ECG weak
assumptions (5)
- domain assumption A mask-aware predictor f(S) produces usable predictions for every subset S of modalities
- domain assumption Zeroing a branch at inference approximates real modality dropout
- domain assumption Distance from the EF≤40% threshold (margin) proxies monitorability
- ad hoc to paper Planted synthetic structure recovers per-example attribution ground truth
- domain assumption Frozen EchoJEPA and HuBERT-ECG embeddings encode clinically relevant LVEF information
Cite this review
Pith. "Pith review of Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI." pith.science (2026). https://pith.science/paper/BESSIXXJ
@misc{pith2026260801462,
author = {Pith},
title = {Pith review of: Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI},
year = {2026},
howpublished = {\url{https://pith.science/paper/BESSIXXJ}},
note = {Machine review of arXiv:2608.01462}
}
read the original abstract
Multimodal clinical models are usually judged on accuracy with every modality present, but deployment removes modalities; an echocardiogram is often unavailable where an ECG is routine. Two questions then matter beyond the size of the accuracy loss: which modality was responsible, and whether the model fails loudly or silently once that modality is dropped. The distinction is per-example and modality-level, and is separate from post-hoc feature attribution (e.g. SHAP). Models are replaced often; the evaluation that answers these questions is reused. We present a model-agnostic modality-failure framework: given N modality embeddings, any mask-aware probe, and labels, it returns a per-example failure taxonomy, a per-modality complementarity matrix that attributes error to modalities, and a loud-vs-silent dropout profile separating monitorable failures from those that pass unflagged far from the decision boundary, using only deployment-observable signals. We release it as a small, unit-tested harness and validate it against planted ground truth. Across seeds it recovers that planted modality dominance and complementary subset, reports per-modality loud-vs-silent rates, and scales to a three-modality complementarity matrix; because the planted structure is known by construction, this validates recovery of per-example attribution rather than clinical performance. We then instantiate the framework on frozen EchoJEPA and HuBERT-ECG embeddings for LVEF and the EF <= 40% HFrEF gate over a paired MIMIC-IV cohort, where on the held-out test split (n = 245) dropping echo nearly doubles error. The narrow echo-to-ECG overlap that bounds cohort size is itself a deployment finding for cardiac foundation models. All of our work can be found at https://github.com/criticaldata/PRIMED-AI.
Figures
Reference graph
Works this paper leans on
-
[6]
doi: 10.1007/978-3-032-04971-1
Springer. doi: 10.1007/978-3-032-04971-1
-
[9]
URLhttps://doi.org/10.13026/nrjh-5r77
doi: 10.13026/nrjh-5r77. URLhttps://doi.org/10.13026/nrjh-5r77. Version 1.0. Paul A. Heidenreich, Biykem Bozkurt, David Aguilar, Larry A. Allen, Joni J. Byun, Monica M. Colvin, Anita Deswal, Mark H. Drazner, Shannon M. Dunlay, Linda R. Evers, James C. Fang, Savitri E. Fedson, Gregg C. Fonarow, Salim S. Hayek, Adrian F. Hernandez, Prateeti Khazanie, Michel...
-
[12]
URL https://doi.org/10.13026/ kpb9-mt58
doi: 10.13026/kpb9-mt58. URL https://doi.org/10.13026/ kpb9-mt58. Version 3.1. Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo Anthony Celi, and Roger G. Mark. MIMIC-IV, a freely accessible electronic health record dataset.Scientific...
-
[15]
URL https://ojs.aaai.org/index.php/AAAI/article/ view/16330
doi: 10.1609/aaai.v35i3.16330. URL https://ojs.aaai.org/index.php/AAAI/article/ view/16330. Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18156–18165,
-
[16]
URLhttps://arxiv.org/abs/2204.05454
doi: 10.1109/CVPR52688.2022.01764. URLhttps://arxiv.org/abs/2204.05454. MIT Laboratory for Computational Physiology. MIMIC on BigQuery. MIMIC documentation,
arXiv 2022
-
[17]
URLhttps://doi.org/10.1007/978-3-032-04971-1_17. Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals.Circulation, 10...
-
[18]
URLhttps://arxiv.org/abs/2602.02603. Tom Pollard, Benjamin Moody, Li-wei Lehman, Brian Gow, Chrystinne Fernandes, Chen Xie, Alistair Johnson, Roger Mark, and Thomas Heldt. Physionet as a global platform for biomedical research.Nature Health, 03
-
[19]
URLhttps://doi.org/10.1038/s44360-026-00096-z
doi: 10.1038/s44360-026-00096-z. URLhttps://doi.org/10.1038/s44360-026-00096-z. 10
Show all 18 references
-
[2000]
URLhttps://doi.org/10.1161/01.CIR.101.23.e215
doi: 10.1161/01.CIR.101.23.e215. URLhttps://doi.org/10.1161/01.CIR.101.23.e215. Brian Gow, Tom Pollard, Nathaniel Greenbaum, Benjamin Moody, Alistair Johnson, Elizabeth Herbst, Jonathan W. Waks, Parastou Eslami, Ashish Chaudhari, Tanner Carbonati, Seth Berkowitz, Roger Mark, a...
-
[2015]
URL https://doi.org/10.1016/j.echo.2014.10.003
doi: 10.1016/j.echo.2014.10.003. URL https://doi.org/10.1016/j.echo.2014.10.003. Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. SMIL: Multimodal learning with severely missing modality. InProceedings of the AAAI Conference on Artificial Intelligence ...
2014 doi
-
[2019]
URL https: //doi.org/10.1038/s41591-018-0240-2
doi: 10.1038/s41591-018-0240-2. URL https: //doi.org/10.1038/s41591-018-0240-2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self- supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. B...
-
[2020]
URL https://proceedings.neurips.cc/paper_files/paper/2020/hash/ 92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html. Sebasti´ an Andr´ es Cajas Ord´ o˜ nez, Rowell Castro, Leo Anthony Celi, Roben Delos Reyes, Justin Engelmann, Ari Ercole, Almog Hilel, Mahima Kalla, Leo Kinyera, Max...
2020
-
[2021]
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Brian Gow, Benjamin Moody, Steven Horng, Leo Anthony Celi, and Roger Mark
doi: 10.1109/TASLP.2021.3122291. Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Brian Gow, Benjamin Moody, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV.PhysioNet, October
2021
-
[2022]
doi: https://doi.org/10.1016/j.jacc.2021.12.012
ISSN 0735-1097. doi: https://doi.org/10.1016/j.jacc.2021.12.012. URLhttps://www.sciencedirect.com/science/article/pii/S0735109721083959. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised ...
2021 doi
-
[2023]
URLhttps://doi.org/10.1038/s41597-022-01899-x
doi: 10.1038/s41597-022-01899-x. URLhttps://doi.org/10.1038/s41597-022-01899-x. Roberto M. Lang, Luigi P. Badano, Victor Mor-Avi, Jonathan Afilalo, Andrea Armstrong, Laura Ernande, Frank A. Flachskampf, Elyse Foster, Steven A. Goldstein, Tatiana Kuznetsova, Patrizio Lancellott...
-
[2024]
URLhttps://doi.org/10.1101/2024.11.14.24317328
doi: 10.1101/2024.11.14.24317328. URLhttps://doi.org/10.1101/2024.11.14.24317328. Preprint. Yuan Gao, Sangwook Kim, and Chris McIntosh. EchoingECG: An electrocardiogram cross-modal model for echocar- diogram tasks. InMedical Image Computing and Computer Assisted Intervention (...
2024 doi
-
[2025]
URLhttps://arxiv.org/abs/2506.09985. 8 Zachi Attia, Suraj Kapa, Francisco Lopez-Jimenez, Paul McKie, Dorothy Ladewig, Gaurav Satam, Patricia Pellikka, Maurice Enriquez-Sarano, Peter Noseworthy, Thomas Munger, Samuel Asirvatham, Christopher Scott, Rickey Carter, and Paul Friedm...
-
[2026]
URL https://doi.org/10.1371/journal.pdig.0001013
doi: 10.1371/journal.pdig.0001013. URL https://doi.org/10.1371/journal.pdig.0001013. Edoardo Coppola, Mattia Savardi, Mauro Massussi, Marianna Adamo, Marco Metra, and Alberto Signoroni. HuBERT- ECG as a self-supervised foundation model for broad and scalable cardiac applicatio...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.