{"id":"c2c295cf-4d89-414f-8a80-7c2189d187a8","arxiv_id":"2507.12945","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A linear uncertainty-propagation model fitted on cardiac MRI plus health-record text is shown to transfer across prediction tasks and data distributions, enabling cheaper uncertainty estimates.","lead":"This paper fits a simple linear model that predicts a medical multimodal AI's overall uncertainty from the uncertainty of its image inputs and text inputs separately. If the model's coefficients stay stable across tasks and data, then clinicians could estimate predictive uncertainty cheaply on new cardiac datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transferability of MUPM is asserted from ANOVA null results, but no cross-task transfer experiment is performed; the invariance claim is not demonstrated.","rationale":"The reader's verdict is CONDITIONAL, and the reader's rationale already notes that the invariance claims rely on null ANOVA results without effect sizes or power analysis. My stress test agrees with that point and sharpens it: the paper's headline claim is not merely under-powered, it is missing the actual operation that the claim describes. Generalisability without refitting means using a fitted model on unseen data; Table 2 compares re-fit models, so it tests coefficient stability, not transfer. A direct held-out transfer experiment would settle the matter. I do not see an internal inconsistency in the derivation, and the code release and reproducible dataset description are positive, but the central claim is currently supported only by a non-informative null result. The paper should not be accepted without either a transfer experiment or a substantially weakened claim. This keeps the CONDITIONAL verdict; it does not change it to reject, because the missing experiment is clearly feasible and the proposed framework may well survive it.","tokens_in":8281,"tokens_out":5418,"duration_ms":64835,"concrete_test":"Perform a held-out transfer experiment: fit MUPM on the 1-year task (or w.o. distribution), freeze beta_1, beta_2, beta_3, and on held-out 5-year task samples compute s2_FI, s2_FT, and s_FI s_FT; predict s2_FI,T with fixed coefficients. Compare predictive R2, ECE, and MAE against (a) the n=100 benchmark overall uncertainty and (b) a MUPM refitted on the held-out task, over five folds. Also report equivalence bounds (e.g. +/-0.1 on coefficients and +/-0.05 on ECE) and a paired bootstrap test. If fixed-coefficient predictions are outside the equivalence bound or significantly worse than refitting, the generalisability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that a fitted MUPM generalizes across input data distributions and downstream tasks, so a single fitted linear model can estimate overall uncertainty for novel tasks without refitting. The evidence in Tables 1 and 2 is ANOVA p-values (0.41, 0.33, 0.89; 0.70, 0.95, 0.98) comparing separately fitted coefficients. This does not establish transferability. First, failure to reject the null is non-informative when coefficients have large standard errors (e.g. beta_2 = 0.91 +/- 0.30 with 5 folds); no equivalence bounds, effect sizes, or power analysis are given. Second, the paper never actually transfers a fitted MUPM: it never fits on one distribution/task and uses those fixed coefficients to predict overall uncertainty on another. The closest experiment (Fig. 2a) uses a benchmark overall uncertainty computed with the same augmentation protocol and does not compare fixed-coefficient prediction to refitting. Because all experiments share the same MLLM, the same augmentation recipe, the same one-hot output encoding, and largely overlapping patient cohorts, the setting is maximally favorable; cross-task invariance could still fail when the model, prompt template, or augmentation strength changes, since the fitted coefficients are known to absorb those choices (Eq. 4 is a free linear fit, not the derived Eq. 3). Thus the headline generalisability claim is currently under-supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multimodal uncertainty propagation model (MUPM), a linear regression model in Eq. (4) motivated by first-order Taylor uncertainty propagation in Eqs. (2)-(3), to relate image-only uncertainty, text-only uncertainty, and their covariance to the overall output uncertainty of a multimodal large language model (MLLM). Using UK Biobank cardiac MR images and digital health records, the authors fit MUPM for closed-ended cardiac disease prediction tasks at 1-, 3-, and 5-year horizons, and claim that the fitted MUPM is robust across input data distributions and downstream tasks, supports efficient estimation of overall uncertainty with small resample sizes, and can identify redundant factors. The paper includes code release and uses a real clinical dataset.","tokens_in":8575,"tokens_out":4062,"duration_ms":49163,"significance":"If transferability were convincingly established, MUPM would be a low-cost post hoc estimator of overall uncertainty for multimodal clinical predictions, and the paper addresses an underexplored and practically relevant question. The first-order propagation derivation is standard, the use of a fitted linear model to avoid unstable derivative computation is sensible, and the clinical setting (cardiac MR plus electronic health records) is meaningful. Credit is also due for releasing code. However, the central claim that fitted MUPMs generalize across data distributions and tasks is currently supported only by non-significant ANOVA results with small per-group sample sizes and no direct transfer experiment, so the practical significance of the contribution is not yet established.","major_comments":[{"comment":"The headline claim that fitted MUPMs are generalisable across data distributions and downstream tasks (Abstract) is never directly tested. Tables 1 and 2 report ANOVA p-values for separately fitted regression coefficients, but failure to reject the null hypothesis is not evidence of invariance, especially with standard errors on the coefficients as large as ±0.40 and only five fitted models per group. A direct test would fit MUPM on one distribution or task, then use those fixed coefficients to predict overall uncertainty on a different distribution or task, and compare the resulting error with refitting. As written, the closest experiment, Fig. 2(a), uses the same augmentation protocol and does not perform such a fixed-coefficient transfer test.","section":"Section 3, 'Robustness of MUPM across various input data distributions' and 'Robustness of MUPM in different cardiac…"},{"comment":"The text states that 'improving the quality of health record text alone resulted in a more significant reduction in overall uncertainty as shown in Table 1,' but Table 1 contains no column for overall uncertainty, such as the average L2 norm of s^2_FI,T. The table only reports averaged regression coefficients, R², ECE, and ANOVA p-values. The claimed reduction in overall uncertainty is therefore not visible in the cited evidence; either remove this claim or add the relevant overall uncertainty metrics to the table.","section":"Table 1 and accompanying text"},{"comment":"Because Eq. (4) is an unconstrained least-squares fit rather than the derived Eq. (3), the fitted coefficients absorb all experiment-specific choices: the MLLM, the one-hot output encoding, the augmentation ranges, the resampling protocol, and the prompt template. Consequently, the ANOVA invariance reported in Tables 1 and 2 is conditional on all these choices being shared across the compared settings. The abstract's broader claim of transferability to novel tasks and differing input distributions requires either an explicit transfer experiment that varies these factors or a clearly stated limitation of the invariance claim to the current M3D-based setup.","section":"Section 2.2, Eq. (4)"},{"comment":"The redundancy criterion is not precisely defined: the text says a completely redundant factor is 'more likely' to have a very low regression coefficient and large covariance with other modalities, but it does not give a decision rule or demonstrate the criterion on a known-redundant input. The validation experiment (removing one modality and measuring accuracy drop) confirms that both image and text contribute to accuracy, but it does not test the paper's proposed redundancy criterion. Please specify the criterion and evaluate it, for example on synthetic inputs with controlled redundancy or by comparison with a baseline method.","section":"Section 3, 'Identifying Redundant Factors'"}],"minor_comments":[{"comment":"Equation (1) omits the increments in the Taylor expansion; it should read F(x,y) ≈ F(x0,y0) + ∂F/∂x (x - x0) + ∂F/∂y (y - y0) unless x0 and y0 are assumed to be zero. Please clarify.","section":"Section 2.1, Eq. (1)"},{"comment":"The horizontal and vertical axes of Fig. 2(a) are not described in the caption; please add axis labels and units, including the definition of the L2-norm of overall uncertainty used.","section":"Figure 2(a)"},{"comment":"Please correct typographical errors, including 'Insti-tute' in the affiliation line and 'predication tasks' in the Section 3 heading.","section":"Throughout"},{"comment":"The calibration measure ECE is computed using a log-sigmoid mapping from uncertainty L2-norms to confidence, but the mapping is not defined precisely and no confidence intervals are given. Please define the mapping and report calibration with uncertainty, e.g., bootstrap intervals.","section":"Section 3, ECE description"},{"comment":"The benchmark for overall uncertainty is computed as the average of 100 augmentation runs; the text does not justify why this average is the ground truth or how the benchmark's own sampling variability compares with the MUPM estimates. Please clarify or provide error bars for the benchmark.","section":"Section 3, benchmark computation"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound derivation and a useful clinical application, but the central transferability claim is overstated relative to the evidence. I would ask the authors to perform a direct fixed-coefficient transfer experiment and to report effect sizes or equivalence bounds rather than relying solely on ANOVA null results. The mismatch between the Table 1 content and the text's 'reduction in overall uncertainty' claim should also be fixed. This is a major revision rather than a rejection because the missing experiments are within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my honest read. The paper proposes MUPM, a linear model relating image-only, text-only, and joint input uncertainty to overall output uncertainty in an MLLM, fitted by resampling/augmentation rather than direct derivative computation. That is a sensible, low-cost idea, and the empirical setting—UK Biobank cardiac MRI plus health records, three prediction horizons—is real. The efficiency experiment, showing a fitted MUPM stabilizes with a few resamples on unseen pairs, is the most convincing part; the redundancy ablation is acceptable.\n\nThe soft spot is the transferability claim. The abstract says fitted MUPMs are generalisable across input distributions and downstream tasks. But the paper never performs a transfer experiment. It fits MUPM separately for each distribution and task, then runs ANOVA on the coefficient vectors. Non-significant p-values (0.41, 0.33, 0.89; 0.70, 0.95, 0.98) do not show invariance, especially with standard errors like beta2 = 0.91 ± 0.30. No effect sizes, equivalence bounds, or power analysis. So \"generalisable\" overstates what the data show. The stress-test note has it right: the closest thing to transfer, the efficiency experiment with unseen pairs, is within the same task, not across tasks. Cross-task transfer remains untested.\n\nSecond, the text claims that improving text quality reduced overall uncertainty, but Table 1 does not report overall uncertainty values—only coefficients, R², and ECE. That claim is not supported by the displayed numbers. Also, the redundancy identification section leans on the fitted covariance term plus an ablation; the ablation is more direct evidence, and the framing overreaches.\n\nThat said, the derivation is standard, the regression is honest, and the data work is genuine. The paper is not circular. It deserves serious refereeing, but the referee should push for a direct cross-task/cross-distribution transfer experiment, effect sizes, and proper reporting of the overall uncertainty values. I'd engage with it but with the headline claim held in suspense.","headline":"A sensible uncertainty-propagation framework for medical MLLMs whose headline transferability claim rests on null ANOVA results rather than a direct transfer experiment.","tokens_in":9099,"tokens_out":3717,"would_cite":true,"duration_ms":39931,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the overall uncertainty of a multimodal large language model's prediction can be written as a linear combination of image-only uncertainty, text-only uncertainty, and their covariance, and that a single fitted…","keywords":["uncertainty propagation","multimodal large language models","cardiac MRI","test-time augmentation","visual question answering","uncertainty calibration","transferability"],"falsifier":"Fit the MUPM regression on one cardiac prediction task, then use the fitted coefficients to predict overall uncertainty on a different task or data distribution where the benchmark uncertainty is computed directly from a large number of resamples; if the predicted variances deviate systematically from the benchmark beyond the calibration error reported for the fitted setting, the transferability claim breaks. A more targeted test: apply the same procedure to a different multimodal architecture and see whether the fitted coefficients remain the same, since the paper's explanation of transferability invokes shared pretraining and light fine-tuning.","tokens_in":1736,"feed_emoji":"🫀","tokens_out":3045,"duration_ms":94659,"temperature":0.7,"pith_summary":"This paper proposes and tests a simple linear law for how input uncertainties combine in a multimodal large language model (MLLM) that reads cardiac MR images together with text health records. The law, which the authors call the multimodal uncertainty propagation model (MUPM), states that the overall variance of the model's output is approximately a weighted sum of the image-only output variance, the text-only output variance, and their covariance. Because computing the partial derivatives behind this law is numerically unstable, the authors fit the three weights as a least-squares regression from augmented samples. Their main evidence is that the fitted weights stay statistically the same across altered data-quality conditions and across 1-, 3-, and 5-year cardiac disease prediction tasks, so a model fit once can estimate uncertainty for novel tasks without refitting. If this transferability holds, uncertainty estimation for new clinical prediction tasks becomes a matter of collecting a small number of resampled inputs rather than recomputing full model uncertainty.","feed_headline":"One fitted law estimates uncertainty for new cardiac tasks","feed_subtitle":"Multimodal image-text uncertainty in an LLM is a simple linear mix, and one fit transfers across MR tasks.","key_machinery":"The load-bearing object is the MUPM regression equation $s^2_{F_{I,T}} = \\beta_1 s^2_{F_I} + \\beta_2 s^2_{F_T} + \\beta_3 s_{F_I}s_{F_T} + \\epsilon$, a linear model whose independent variables are sample variances of MLLM outputs under image-only augmentation, text-only augmentation, and a covariance term, and whose coefficients are fitted by least squares. It is derived from a first-order Taylor expansion of the model output $F(x,y)$, which yields the variance-propagation formula $\\sigma^2_{F_{I,T}} \\approx (a/a_I)^2 \\sigma^2_{F_I} + (b/b_T)^2 \\sigma^2_{F_T} + 2(ab/(a_I b_T))\\rho\\sigma_{F_I}\\sigma_{F_T}$. The machinery is model-agnostic: uncertainties are estimated by resampling and augmentation (affine transforms and Gaussian noise for images, synonym replacement for text) and computing sample variances of one-hot outputs, and the linear fit sidesteps the unstable derivative computation.","core_discovery":"The central discovery on the paper's own terms is that the variance of an MLLM's one-hot-encoded output under joint image-text resampling can be modelled as a linear function of three measurable quantities: the variance under image-only resampling, the variance under text-only resampling, and their cross-covariance, where the coefficients are fitted rather than computed from derivatives. The authors derive this relation from a first-order Taylor expansion of the model output, replacing the unstable partial derivatives with regression coefficients estimated by least squares. They report that the fitted coefficients are statistically indistinguishable across four data-distribution settings and across three prediction horizons, with all pairwise ANOVA p-values above 0.05, and that the resulting model reproduces the benchmark overall uncertainty to $R^2 \\approx 0.8$. They also show that the text-only coefficient dominates, that overall uncertainty rises with prediction horizon, and that removing either modality substantially lowers accuracy.","pith_inferences":["The transferability claim suggests the fitted coefficients are a property of the MLLM's pretraining and fine-tuning rather than of the specific task; an out-of-domain test on non-cardiac medical image-text pairs would show whether the law is a general feature of such models.","The paper's text augmentation uses a generative language model for synonym replacement, which introduces sampling randomness; a controlled comparison with deterministic lexical substitution would test how much the fitted coefficients depend on the augmentation distribution.","Because the covariance coefficient is consistently negative in the reported fits, a natural extension would examine whether this reflects complementary information between modalities and whether it strengthens or weakens for tasks where the modalities are more or less aligned.","The first-order derivation is local, so the linear law should degrade for very large input perturbations; measuring output variance under progressively larger image rotations or text rewrites would map the regime in which the approximation holds."],"forward_implications":["A fitted MUPM can estimate overall prediction uncertainty for novel cardiac disease prediction tasks without refitting, as long as the output is one-hot encoded in the same token format.","Uncertainty estimates stabilize with roughly 20 resamples per input, so overall uncertainty can be computed efficiently on unseen downstream data instead of running hundreds of augmentations.","The relative sizes of the fitted coefficients reveal which modality drives overall uncertainty—here text dominates—so data-improvement efforts can be directed to the more influential modality.","The covariance term and coefficient magnitudes can flag redundant factors, although in the tested cardiac tasks both image and text proved essential.","Transferability across data distributions implies that quality perturbations such as image-view changes, signal-to-noise changes, and text-style changes do not require refitting the uncertainty model within the tested range."],"supporting_citations":[{"why":"supplies the first-order uncertainty propagation theory used to derive the variance law.","marker":"[15]"},{"why":"supplies the resampling-and-augmentation approach used to estimate input variances.","marker":"[25]"},{"why":"the 3D-medical-image multimodal LLM whose output uncertainties are modeled.","marker":"[3]"},{"why":"supplies the language model used for synonym-replacement text augmentation.","marker":"[1]"},{"why":"the least-squares method used to fit the linear MUPM coefficients.","marker":"[4]"},{"why":"the analysis-of-variance test used to show coefficient stability across tasks and distributions.","marker":"[24]"},{"why":"the expected-calibration-error metric used to validate uncertainty quality.","marker":"[21]"},{"why":"defines the Pearson correlation coefficient used for the covariance term.","marker":"[7]"}],"fun_headline_variants":["A simple linear law predicts LLM uncertainty across tasks","One fitted equation transfers uncertainty across cardiac tasks","Text dominates uncertainty in multimodal LLMs for heart scans","Linear mix of unimodal variances estimates LLM doubt","Uncertainty law for multimodal LLMs holds across MR tasks"],"cache_read_input_tokens":11264,"weakest_assumption_plain":"The validity of the whole construction rests on the assumption that the resampling-and-augmentation procedure produces output variances that faithfully capture the clinically relevant input uncertainty, and that the first-order Taylor approximation is accurate enough for the linear law to hold for the nonlinear MLLM.","fun_headline_variants_meta":{"raw":{"variants":["A simple linear law predicts LLM uncertainty across tasks","One fitted equation transfers uncertainty across cardiac tasks","Text dominates uncertainty in multimodal LLMs for heart scans","Linear mix of unimodal variances estimates LLM doubt","Uncertainty law for multimodal LLMs holds across MR tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2315,"prompt_tokens":1012,"completion_tokens":1303,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":1227}},"tokens_in":628,"tokens_out":1303,"duration_ms":10759,"temperature":1.0,"reasoning_tokens":1227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:34:37.978709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit the MUPM regression on one cardiac prediction task, then use the fitted coefficients to predict overall uncertainty on a different task or data distribution where the benchmark uncertainty is computed directly from a large number of resamples; if the predicted variances deviate systematically from the benchmark beyond the calibration error reported for the fitted setting, the transferability claim breaks. A more targeted test: apply the same procedure to a different multimodal architecture and see whether the fitted coefficients remain the same, since the paper's explanation of transferability invokes shared pretraining and light fine-tuning.","supporting_citations":[{"cited_title":"Structural and multidisciplinary optimization37, 239– 253 (2009)","cited_arxiv_id":null,"evidence_quote":"supplies the first-order uncertainty propagation theory used to derive the variance law."},{"cited_title":"Neurocomputing338, 34–45 (2019)","cited_arxiv_id":null,"evidence_quote":"supplies the resampling-and-augmentation approach used to estimate input variances."},{"cited_title":"Handbook of numerical analysis1, 465–652 (1990)","cited_arxiv_id":null,"evidence_quote":"the least-squares method used to fit the linear MUPM coefficients."},{"cited_title":"Chemometrics and intelligent laboratory systems 6(4), 259–272 (1989)","cited_arxiv_id":null,"evidence_quote":"the analysis-of-variance test used to show coefficient stability across tasks and distributions."},{"cited_title":"In: CVPR workshops","cited_arxiv_id":null,"evidence_quote":"the expected-calibration-error metric used to validate uncertainty quality."},{"cited_title":"Noise reduction in speech processing pp","cited_arxiv_id":null,"evidence_quote":"defines the Pearson correlation coefficient used for the covariance term."}],"review_version":1}