{"id":"8f93cea0-ef02-4b90-88e7-69e03e0d9fe0","arxiv_id":"2504.13465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SURE reconstructs missing modalities in the latent space of frozen pretrained multimodal models and uses a Pearson-correlation loss plus error propagation to output uncertainty estimates that track prediction error.","lead":"This paper introduces SURE, a training pipeline that lets pretrained multimodal models cope with missing data by reconstructing absent modalities in latent space and estimating how uncertain those reconstructions and the final predictions are. It is worth a read if you want a concrete way to add calibrated 'I don't know' signals to multimodal systems that must keep running on incomplete inputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LPCC loss is affine-invariant, so the absolute scale of all learned uncertainty heads is unidentifiable; the paper's 'precise/reliable' uncertainty claim is unsupported and no UCE numbers are reported.","rationale":"The paper's central claim is that SURE 'delivers reliable uncertainty estimates' and provides 'precise quantification' through error propagation. The weakest link is not primarily the first-order expansion in Proposition 2.2 but the objective used to train the uncertainty heads. LPCC = 1 - r(sigma^2, epsilon^2) is invariant to affine transformations of sigma^2 with positive slope, so the loss surface has a flat direction: the constants a and b in sigma^2_i -> a + b*sigma^2_i do not affect the loss. The reconstruction uncertainty head is trained with L_rec, whose MSE term acts on the reconstructed latent, not on sigma^2_z; the classifier uncertainty head is trained only with LPCC. Therefore neither sigma^2_z nor sigma^2_omega has a data-determined scale, and the propagated sigma^2_input in Eq. (8) and final sigma^2_Y in Eq. (10) inherit this arbitrariness. This directly weakens the 'precise quantification' claim and makes the promised UCE evaluation the decisive evidence; yet Tables 1-3 report only PCC. The author-acknowledged tendency to overestimate uncertainties in Section 4.2 is consistent with miscalibration. The Taylor-expansion assumption flagged by the reader is a genuine secondary issue, but even an exact Jacobian cannot rescue the scale non-identifiability. The appropriate response is to keep the conditional verdict and require that UCE/reliability diagrams be reported, and either add a calibration term or restrict the claims to ranking-based uncertainty rather than absolute precision.","tokens_in":20777,"tokens_out":11151,"duration_ms":108385,"concrete_test":"","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that SURE's uncertainty values are reliable and precisely quantified. But the proposed LPCC loss in Eq. (4)-(5) is invariant to replacing every sigma^2_i in a batch by a + b*sigma^2_i (b>0), because Pearson correlation is unchanged under affine transformations. The reconstruction uncertainty head is trained with L_rec in Eq. (7), whose MSE term is on the reconstructed latent z_i, not on sigma^2_z; the classifier uncertainty head sigma^2_omega is trained only with LPCC. Hence neither sigma^2_z nor sigma^2_omega has a data-determined scale or offset. The propagated input uncertainty in Eq. (8) and the final output uncertainty in Eq. (10) inherit this arbitrariness, so 'precise quantification' is not delivered by the method. The paper states in Section 3.1 that uncertainty quality is evaluated with UCE and PCC, yet Tables 1-3 report only Pearson Corr, not UCE. Section 4.2 also admits a 'tendency toward overestimating both reconstruction and output uncertainties,' an author-acknowledged calibration failure. The Taylor-expansion worry in Proposition 2.2 is secondary: even with an exact Jacobian, LPCC cannot set the absolute scale of the uncertainty estimates. Thus the headline claim of reliable uncertainty estimates is currently unsupported by the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SURE, a framework for adapting pretrained multimodal models to small-scale datasets with missing modalities. Missing latent representations are reconstructed by lightweight per-modality modules, and two types of uncertainty are estimated: reconstruction uncertainty for the imputed latents and output uncertainty for the final prediction. The output uncertainty combines a model-intrinsic term learned with a Pearson-correlation loss (LPCC) and an input-induced term computed by first-order error propagation through the frozen fusion network. The authors report state-of-the-art results on sentiment analysis (CMU-MOSI), book genre classification, and human action recognition (UTD-MHAD), and claim reliable, precisely quantified uncertainty estimates.","tokens_in":21075,"tokens_out":4557,"duration_ms":43497,"significance":"If the central claims held, the contribution would be practically useful: an architecture-agnostic way to keep pretrained multimodal models usable when modalities are missing, with an uncertainty signal that tracks prediction errors and supports deferral decisions. The paper has genuine strengths: Theorem 2.1 is correctly derived (up to a finite-sample factor of (N-1)/N), the experimental coverage is broad with three different backbone architectures and multiple missing-modality configurations, ablations isolate the contributions of reconstruction, uncertainty estimation, and pretraining, and the decision-making experiment is a concrete demonstration of the intended use. The code is promised for release. However, the headline claim of 'precise quantification of uncertainties' is undermined by an affine-invariance property of the proposed loss and by the absence of any calibration metric (UCE) in the reported tables, as detailed below.","major_comments":[{"comment":"The LPCC loss in Eq. (4)-(5) is invariant under the affine reparameterization sigma^2_i -> a*sigma^2_i + b with a>0, because Pearson correlation is unaffected by affine transformations. The reconstruction loss L_rec in Eq. (7) contains an MSE term, but that term supervises the reconstructed latent z_i, not sigma^2_{z_i}; the output-head uncertainty sigma^2_omega is trained only through LPCC. Hence neither sigma^2_z nor sigma^2_omega has a data-determined scale or offset, and the propagated sigma^2_input in Eq. (8) and the total sigma^2_Y in Eq. (10) inherit arbitrary units. The abstract's claim of 'precise quantification of uncertainties from missing data and model predictions' is therefore not supported by the method as written. I request either a calibration mechanism that fixes the scale/offset (e.g., a held-out temperature/affine calibration before reporting UCE) or an explicit restatement that only rank-based (monotone) uncertainty is claimed.","section":"Section 2.2, Eqs. (4)-(7)"},{"comment":"Section 3.1 states that uncertainty quality is evaluated with both Uncertainty Calibration Error (UCE) and Pearson Correlation Coefficient (PCC), but Tables 1-3 report only Pearson correlations. UCE is the standard metric for assessing whether uncertainty values are calibrated in absolute terms, and it is precisely the metric affected by the affine-invariance issue. The absence of UCE numbers means the 'reliable uncertainty estimates' claim is not evidenced. Additionally, Section 4.2 explicitly acknowledges a 'tendency toward overestimating both reconstruction and output uncertainties,' which is an author-acknowledged calibration failure. Please add UCE results or explicitly drop the calibration claim.","section":"Section 3.1 and Tables 1-3"},{"comment":"The first-order error-propagation formula in Eq. (8) assumes that the frozen fusion network omega is sufficiently smooth that a first-order Taylor expansion is accurate over the range of reconstruction errors. This is asserted rather than validated. For deep nonlinear fusion networks (transformers, attention-based fusion), reconstruction residuals are not necessarily small, and the Jacobian approximation can be badly biased. The paper should provide evidence of local linearity, such as a finite-difference or Monte Carlo comparison on the frozen network, or a bound on the reconstruction-error magnitude; otherwise the propagated input uncertainty may misstate the true contribution of missing modalities.","section":"Proposition 2.2 and Eq. (8)"}],"minor_comments":[{"comment":"The phrase 'applies statistical error propagation in deep networks for the first time' is a strong novelty claim; the literature review does not provide evidence that no prior work has used error propagation in deep networks. Please soften or provide a more careful search-based justification.","section":"Abstract and Section 1"},{"comment":"The column groupings in Tables 1-3 are hard to parse: the T/A/F headings for reconstruction and output uncertainty are ambiguous, and the number of entries per row is inconsistent (e.g., the OutputUncertainty Corr row for SURE in Table 1 appears to have two values under a three-column header). Please reformat and ensure each row has the same number of entries as the header.","section":"Tables 1-3"},{"comment":"'SolfPlus' is a typo for 'SoftPlus'.","section":"Appendix A.2.2, Eq. (11)"},{"comment":"In Algorithm 1, line 27 uses L_PCC(sigma^2_y; l_downstream), but l_downstream is the downstream loss value, not the per-sample error; for classification tasks the error used in LPCC should be defined explicitly (cross-entropy per sample) rather than relying on the mini-batch scalar.","section":"Algorithm 1"},{"comment":"The paper reports averages over three runs but does not report standard deviations; given the small-scale datasets and missing-modality masks, error bars would materially strengthen the SOTA claims.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The affine-invariance objection is the key issue: as written, the LPCC loss cannot identify the absolute scale of the uncertainty estimates, and the paper's own Section 4.2 admits overestimation. This is fixable—either by adding a calibration step and reporting UCE, or by reframing the contribution as rank-based uncertainty—but it must be addressed before publication. I would also encourage the editors to ensure the 'first adaptation of error propagation' novelty claim is checked, since the cited references are classical statistics texts and the claim seems under-supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my quick read of SURE. The genuinely new piece is the Pearson-correlation loss for training uncertainty heads, plus the idea of propagating reconstructed-input uncertainty through a frozen fusion network via a first-order Taylor expansion. Theorem 2.1 is correct, and the pipeline is genuinely architecture-agnostic. On the experimental side, the paper covers three tasks with a sensible set of baselines, and the downstream performance numbers are competitive.\n\nThe soft spot is the uncertainty claim, and it's load-bearing. The LPCC loss is invariant to affine transformations of sigma^2: any a + b*sigma^2 with b>0 gives the same correlation. Since the reconstruction head's sigma^2 is trained only with MSE on z_i and LPCC on the error, and the classifier's sigma_omega^2 is trained only with LPCC, the absolute scale and offset of the uncertainty estimates are never pinned down by the data. That means the 'precise quantification' advertised in the abstract is not delivered. The numbers fed into Eq. (8) and (10) are arbitrary in scale, so the propagated 'output uncertainty' cannot be interpreted as a calibrated variance. The paper claims to evaluate with UCE in Section 3.1 but never reports a single UCE number; the tables show Pearson correlations, which is precisely what LPCC optimizes, so that doesn't tell us about calibration. And the authors themselves note in Section 4.2 a 'tendency toward overestimating both reconstruction and output uncertainties.' That is an honest admission, but it directly contradicts 'reliable uncertainty estimates.'\n\nThe Taylor-expansion worry in Proposition 2.2 is secondary; even with an exact Jacobian, the scale problem remains. I also want to flag that the ablations in Table 4 show removing the uncertainty modules (2a, 2b) has almost no effect on F1/Acc, so the performance gains come from reconstruction, not from uncertainty. That's fine, but the paper frames uncertainty as the core contribution.\n\nOther issues are more mundane: no error bars or significance tests, code only promised for after publication, and a few typos (e.g., 'SolfPlus').\n\nOverall, the reconstruction idea is useful, the Pearson loss is a clever alternative to NLL, and the experimental setup is reasonable. But the central calibration claim needs major rework: either add a scale-fixing term (e.g., an NLL-like component on the variance) or explicitly reframe the method as producing a ranking/ordering for deferral rather than precise uncertainty quantification. If the authors do that and report UCE with error bars, the paper would be a solid contribution.\n\nI'd send this to peer review rather than desk-reject it, because the core idea is novel and the flaws are fixable, but I wouldn't accept it as-is. It needs a serious referee and probably a major revision. If you have a student working on missing modalities, it's worth a look.","headline":"The Pearson-loss trick is neat and the downstream results are solid, but the paper's central claim of precise, reliable uncertainty is not supported: the loss cannot determine the scale of the uncertainties, UCE is never reported, and the authors admit overestimation.","tokens_in":21547,"tokens_out":3634,"would_cite":false,"duration_ms":32590,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SURE reconstructs missing modalities in frozen multimodal models and learns uncertainty scores—via a Pearson-correlation loss and error propagation—that track both reconstruction and prediction error.","keywords":["missing modalities","multimodal learning","uncertainty estimation","pretrained models","latent space reconstruction","error propagation","Pearson correlation","selective prediction"],"falsifier":"Inject a known-noise perturbation of increasing magnitude into a reconstructed latent input $\\tilde{Z}_i$ at test time, and compare the output uncertainty predicted by $\\tilde{\\sigma}^2_{\\mathrm{input}} = \\sum_{i\\in J}(\\partial\\omega/\\partial\\tilde{Z}_i)^2\\tilde{\\sigma}^2_{\\tilde{Z}_i}$ with the empirically observed variance of the network's outputs under many perturbations. If the propagated uncertainty diverges from the observed spread as the perturbation grows, the first-order Taylor approximation is the point of failure.","tokens_in":20612,"feed_emoji":"🧩","tokens_out":10133,"duration_ms":85634,"temperature":0.7,"pith_summary":"SURE is a framework for adapting pretrained multimodal models to data with missing modalities: small trainable modules reconstruct missing latent representations, and each reconstruction comes with an uncertainty estimate. The paper's central claim is that a distribution-free loss based on Pearson correlation can train these uncertainties to track actual errors, and that statistical error propagation—applied to a frozen fusion network for the first time—carries reconstruction uncertainty all the way to the final prediction. If the claim is right, practitioners can fine-tune a pretrained multimodal backbone on small, incompletely labeled datasets and, just as important, obtain a reliable signal for when the model should abstain or defer. SURE is evaluated on sentiment analysis, book genre classification, and human action recognition, where it reports better downstream accuracy than reconstruction-only baselines and better uncertainty-error correlation than standard uncertainty baselines.","feed_headline":"Missing modalities get rebuilt with uncertainty that tracks error","feed_subtitle":"A correlation loss plus error propagation lets frozen multimodal models flag unreliable predictions.","key_machinery":"The load-bearing machinery is three pieces working together. (1) Reconstruction modules $r_i$ are inserted after each frozen unimodal projector $f_i$; a module maps an available modality's latent $Z_j$ to an estimate $\\tilde{Z}_i$ of the missing modality plus a reconstruction uncertainty $\\tilde{\\sigma}^2_{\\tilde{Z}_i}$. (2) The Pearson-correlation loss $L_{\\mathrm{PCC}} = 1 - r(\\tilde{\\sigma}^2,\\tilde{\\epsilon}^2)$ aligns uncertainty with squared error while remaining well defined when errors approach zero, because it standardizes both quantities rather than dividing by the error. (3) Error propagation through the frozen fusion network, $\\tilde{\\sigma}^2_{\\mathrm{input}} = \\sum_{i\\in J}(\\partial\\omega/\\partial\\tilde{Z}_i)^2 \\tilde{\\sigma}^2_{\\tilde{Z}_i}$, turns reconstruction uncertainty into an input-induced output uncertainty, which is added to the classifier head's intrinsic uncertainty $\\tilde{\\sigma}^2_\\omega$ to form the final $\\tilde{\\sigma}^2_Y$. The propagation step is what makes the framework's uncertainty signal depend explicitly on missing data.","core_discovery":"The central discovery is that uncertainty about a missing input does not have to be learned indirectly at the output; it can be computed from the reconstruction module and then propagated through the frozen pretrained fusion function $\\omega$ using $\\tilde{\\sigma}^2_{\\mathrm{input}} = \\sum_{i \\in J}(\\partial \\omega / \\partial \\tilde{Z}_i)^2 \\tilde{\\sigma}^2_{\\tilde{Z}_i}$. Adding this to the model's intrinsic output uncertainty, $\\tilde{\\sigma}^2_Y = \\tilde{\\sigma}^2_{\\mathrm{input}} + \\tilde{\\sigma}^2_\\omega$, gives a final uncertainty that is trained, via the Pearson-correlation loss $L_{\\mathrm{PCC}} = 1 - r(\\tilde{\\sigma}^2, \\tilde{\\epsilon}^2)$, to match the pattern of actual prediction errors. The paper demonstrates that this combined signal outperforms Gaussian negative-log-likelihood uncertainty, Monte Carlo dropout, and deep ensembles on three tasks, and that thresholding the uncertainty supports a sensible defer-or-decide rule.","pith_inferences":["Because the Pearson-correlation loss is invariant to scaling, SURE's uncertainty values are only a relative confidence ordering; adopting the method in a safety-critical setting would still require fitting a per-dataset threshold or a separate recalibration step.","The error-propagation formula treats each reconstructed modality as an independent source of variance and omits higher-order terms, so correlated reconstruction errors could be systematically misstated; propagating a full covariance matrix is a natural next step.","When several modalities are available, SURE averages their reconstructions of a missing modality; weighting each reconstruction by its own uncertainty would be a direct, testable improvement.","The same two-phase recipe could be tested with all pretrained weights frozen except the reconstruction heads, which would show whether the uncertainty signal transfers to a completely unmodified backbone under distribution shift."],"forward_implications":["Any frozen pretrained multimodal model can be adapted to incomplete data by training only small reconstruction heads and a classifier head, leaving the bulk of the network untouched.","Reconstructed modalities carry a per-input uncertainty that rises with reconstruction error, so an imputed value can be labeled as trustworthy or not before it is used.","Final predictions carry a combined uncertainty that correlates with actual error, which enables selective prediction: deferring high-uncertainty samples improves reliability at a tunable cost.","Because reconstruction happens during training, samples with missing modalities are fully used rather than discarded or zero-padded, which is why the paper reports accuracy gains on small datasets.","The framework is not tied to one architecture: the same recipe is applied to three different pretrained backbones across sentiment, genre, and action-recognition tasks."],"supporting_citations":[{"why":"It supplies the pretrained multimodal fusion backbone used for the sentiment-analysis experiments and at the center of the framework diagram.","marker":"Wu et al. (2024)"},{"why":"It provides the sentiment-analysis benchmark and evaluation splits used to measure performance under missing modalities.","marker":"Zadeh et al. (2016)"},{"why":"It provides the larger pretraining corpus from which the sentiment backbone's weights are taken.","marker":"Zadeh et al. (2018)"},{"why":"It supplies the pretrained multimodal bitransformer backbone used for book genre classification.","marker":"Kiela et al. (2019)"},{"why":"It supplies the pretrained action-recognition backbone used in the human action recognition experiments.","marker":"Islam & Iqbal (2020)"},{"why":"It is the stated source of the error-propagation formula adapted in Proposition 2.2.","marker":"Arras (1998)"},{"why":"It supplies the statistical error-propagation derivation used to justify Equation (8).","marker":"Tellinghuisen (2001)"},{"why":"It is the heteroscedastic Gaussian negative-log-likelihood baseline that SURE replaces and compares against.","marker":"Kendall & Gal (2017a)"},{"why":"It is the deep-ensemble uncertainty baseline used in the comparison tables.","marker":"Lakshminarayanan et al. (2017)"},{"why":"It is a latent-space reconstruction baseline for missing modalities that SURE is compared with.","marker":"Woo et al. (2023)"}],"fun_headline_variants":["Pearson loss and error propagation boost missing-modality accuracy","Uncertainty-aware reconstruction beats ensembles on incomplete data","SURE: propagating uncertainty through frozen fusion for missing inputs","Correlation loss sharpens uncertainty and predicts errors on missing data","Error propagation in deep nets gives reliable missing-modality uncertainty"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen pretrained fusion network is locally linear enough for a first-order Taylor expansion to convert reconstruction uncertainty into output uncertainty; if reconstructed latent values are far from the true ones, the propagated uncertainty can be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["Pearson loss and error propagation boost missing-modality accuracy","Uncertainty-aware reconstruction beats ensembles on incomplete data","SURE: propagating uncertainty through frozen fusion for missing inputs","Correlation loss sharpens uncertainty and predicts errors on missing data","Error propagation in deep nets gives reliable missing-modality uncertainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1273,"prompt_tokens":953,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":239}},"tokens_in":569,"tokens_out":320,"duration_ms":4110,"temperature":1.0,"reasoning_tokens":239,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:07:15.101935+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inject a known-noise perturbation of increasing magnitude into a reconstructed latent input $\\tilde{Z}_i$ at test time, and compare the output uncertainty predicted by $\\tilde{\\sigma}^2_{\\mathrm{input}} = \\sum_{i\\in J}(\\partial\\omega/\\partial\\tilde{Z}_i)^2\\tilde{\\sigma}^2_{\\tilde{Z}_i}$ with the empirically observed variance of the network's outputs under many perturbations. If the propagated uncertainty diverges from the observed spread as the perturbation grows, the first-order Taylor approximation is the point of failure.","supporting_citations":[{"cited_title":"Hamlet: A hierarchical multimodal attention-based human activ- ity recognition algorithm","cited_arxiv_id":null,"evidence_quote":"It supplies the pretrained action-recognition backbone used in the human action recognition experiments."}],"review_version":1}