Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

Duy A. Nguyen; Khoa D. Doan; Minh N. Do; Quan Huu Do

REVIEW 3 major objections 5 minor 14 references

Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SURE reconstructs missing modalities in frozen multimodal models and learns uncertainty scores—via a Pearson-correlation loss and error propagation—that track both reconstruction and prediction error.

desk verdict The Pearson-loss trick is neat and the downstream results are solid, but the paper's central claim of precise, reliable uncertainty is not supported: the loss cannot determine the scale of the uncertainties, UCE is never reported, and the authors admit overestimation. read the letter →

arxiv 2504.13465 v1 pith:B2ADVJ4F submitted 2025-04-18 cs.LG

Duy A. Nguyen , Quan Huu Do , Khoa D. Doan , Minh N. Do This is my paper · ORCID

classification cs.LG

keywords missingmodalitiesmultimodallearninguncertaintyestimationpretrainedmodelslatentspacereconstructionerrorpropagationPearsoncorrelationselectiveprediction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SURE is a framework for adapting pretrained multimodal models to data with missing modalities: small trainable modules reconstruct missing latent representations, and each reconstruction comes with an uncertainty estimate. The paper's central claim is that a distribution-free loss based on Pearson correlation can train these uncertainties to track actual errors, and that statistical error propagation—applied to a frozen fusion network for the first time—carries reconstruction uncertainty all the way to the final prediction. If the claim is right, practitioners can fine-tune a pretrained multimodal backbone on small, incompletely labeled datasets and, just as important, obtain a reliable signal for when the model should abstain or defer. SURE is evaluated on sentiment analysis, book genre classification, and human action recognition, where it reports better downstream accuracy than reconstruction-only baselines and better uncertainty-error correlation than standard uncertainty baselines.

What carries the argument

The load-bearing machinery is three pieces working together. (1) Reconstruction modules $r_i$ are inserted after each frozen unimodal projector $f_i$; a module maps an available modality's latent $Z_j$ to an estimate $\tilde{Z}_i$ of the missing modality plus a reconstruction uncertainty $\tilde{\sigma}^2_{\tilde{Z}_i}$. (2) The Pearson-correlation loss $L_{\mathrm{PCC}} = 1 - r(\tilde{\sigma}^2,\tilde{\epsilon}^2)$ aligns uncertainty with squared error while remaining well defined when errors approach zero, because it standardizes both quantities rather than dividing by the error. (3) Error propagation through the frozen fusion network, $\tilde{\sigma}^2_{\mathrm{input}} = \sum_{i\in J}(\partial\omega/\partial\tilde{Z}_i)^2 \tilde{\sigma}^2_{\tilde{Z}_i}$, turns reconstruction uncertainty into an input-induced output uncertainty, which is added to the classifier head's intrinsic uncertainty $\tilde{\sigma}^2_\omega$ to form the final $\tilde{\sigma}^2_Y$. The propagation step is what makes the framework's uncertainty signal depend explicitly on missing data.

What would settle it

Inject a known-noise perturbation of increasing magnitude into a reconstructed latent input $\tilde{Z}_i$ at test time, and compare the output uncertainty predicted by $\tilde{\sigma}^2_{\mathrm{input}} = \sum_{i\in J}(\partial\omega/\partial\tilde{Z}_i)^2\tilde{\sigma}^2_{\tilde{Z}_i}$ with the empirically observed variance of the network's outputs under many perturbations. If the propagated uncertainty diverges from the observed spread as the perturbation grows, the first-order Taylor approximation is the point of failure.

Watch

Extended reading notes

Core claim

The central discovery is that uncertainty about a missing input does not have to be learned indirectly at the output; it can be computed from the reconstruction module and then propagated through the frozen pretrained fusion function $\omega$ using $\tilde{\sigma}^2_{\mathrm{input}} = \sum_{i \in J}(\partial \omega / \partial \tilde{Z}_i)^2 \tilde{\sigma}^2_{\tilde{Z}_i}$. Adding this to the model's intrinsic output uncertainty, $\tilde{\sigma}^2_Y = \tilde{\sigma}^2_{\mathrm{input}} + \tilde{\sigma}^2_\omega$, gives a final uncertainty that is trained, via the Pearson-correlation loss $L_{\mathrm{PCC}} = 1 - r(\tilde{\sigma}^2, \tilde{\epsilon}^2)$, to match the pattern of actual prediction errors. The paper demonstrates that this combined signal outperforms Gaussian negative-log-likelihood uncertainty, Monte Carlo dropout, and deep ensembles on three tasks, and that thresholding the uncertainty supports a sensible defer-or-decide rule.

Load-bearing premise

The load-bearing premise is that the frozen pretrained fusion network is locally linear enough for a first-order Taylor expansion to convert reconstruction uncertainty into output uncertainty; if reconstructed latent values are far from the true ones, the propagated uncertainty can be miscalibrated.

Editorial extensions

If this is right

Any frozen pretrained multimodal model can be adapted to incomplete data by training only small reconstruction heads and a classifier head, leaving the bulk of the network untouched.
Reconstructed modalities carry a per-input uncertainty that rises with reconstruction error, so an imputed value can be labeled as trustworthy or not before it is used.
Final predictions carry a combined uncertainty that correlates with actual error, which enables selective prediction: deferring high-uncertainty samples improves reliability at a tunable cost.
Because reconstruction happens during training, samples with missing modalities are fully used rather than discarded or zero-padded, which is why the paper reports accuracy gains on small datasets.
The framework is not tied to one architecture: the same recipe is applied to three different pretrained backbones across sentiment, genre, and action-recognition tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the Pearson-correlation loss is invariant to scaling, SURE's uncertainty values are only a relative confidence ordering; adopting the method in a safety-critical setting would still require fitting a per-dataset threshold or a separate recalibration step.
The error-propagation formula treats each reconstructed modality as an independent source of variance and omits higher-order terms, so correlated reconstruction errors could be systematically misstated; propagating a full covariance matrix is a natural next step.
When several modalities are available, SURE averages their reconstructions of a missing modality; weighting each reconstruction by its own uncertainty would be a direct, testable improvement.
The same two-phase recipe could be tested with all pretrained weights frozen except the reconstruction heads, which would show whether the uncertainty signal transfers to a completely unmodified backbone under distribution shift.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my quick read of SURE. The genuinely new piece is the Pearson-correlation loss for training uncertainty heads, plus the idea of propagating reconstructed-input uncertainty through a frozen fusion network via a first-order Taylor expansion. Theorem 2.1 is correct, and the pipeline is genuinely architecture-agnostic. On the experimental side, the paper covers three tasks with a sensible set of baselines, and the downstream performance numbers are competitive.

The soft spot is the uncertainty claim, and it's load-bearing. The LPCC loss is invariant to affine transformations of sigma^2: any a + b*sigma^2 with b>0 gives the same correlation. Since the reconstruction head's sigma^2 is trained only with MSE on z_i and LPCC on the error, and the classifier's sigma_omega^2 is trained only with LPCC, the absolute scale and offset of the uncertainty estimates are never pinned down by the data. That means the 'precise quantification' advertised in the abstract is not delivered. The numbers fed into Eq. (8) and (10) are arbitrary in scale, so the propagated 'output uncertainty' cannot be interpreted as a calibrated variance. The paper claims to evaluate with UCE in Section 3.1 but never reports a single UCE number; the tables show Pearson correlations, which is precisely what LPCC optimizes, so that doesn't tell us about calibration. And the authors themselves note in Section 4.2 a 'tendency toward overestimating both reconstruction and output uncertainties.' That is an honest admission, but it directly contradicts 'reliable uncertainty estimates.'

The Taylor-expansion worry in Proposition 2.2 is secondary; even with an exact Jacobian, the scale problem remains. I also want to flag that the ablations in Table 4 show removing the uncertainty modules (2a, 2b) has almost no effect on F1/Acc, so the performance gains come from reconstruction, not from uncertainty. That's fine, but the paper frames uncertainty as the core contribution.

Other issues are more mundane: no error bars or significance tests, code only promised for after publication, and a few typos (e.g., 'SolfPlus').

Overall, the reconstruction idea is useful, the Pearson loss is a clever alternative to NLL, and the experimental setup is reasonable. But the central calibration claim needs major rework: either add a scale-fixing term (e.g., an NLL-like component on the variance) or explicitly reframe the method as producing a ranking/ordering for deferral rather than precise uncertainty quantification. If the authors do that and report UCE with error bars, the paper would be a solid contribution.

I'd send this to peer review rather than desk-reject it, because the core idea is novel and the flaws are fixable, but I wouldn't accept it as-is. It needs a serious referee and probably a major revision. If you have a student working on missing modalities, it's worth a look.

Referee Report

3 major / 5 minor

Summary. The paper proposes SURE, a framework for adapting pretrained multimodal models to small-scale datasets with missing modalities. Missing latent representations are reconstructed by lightweight per-modality modules, and two types of uncertainty are estimated: reconstruction uncertainty for the imputed latents and output uncertainty for the final prediction. The output uncertainty combines a model-intrinsic term learned with a Pearson-correlation loss (LPCC) and an input-induced term computed by first-order error propagation through the frozen fusion network. The authors report state-of-the-art results on sentiment analysis (CMU-MOSI), book genre classification, and human action recognition (UTD-MHAD), and claim reliable, precisely quantified uncertainty estimates.

Significance. If the central claims held, the contribution would be practically useful: an architecture-agnostic way to keep pretrained multimodal models usable when modalities are missing, with an uncertainty signal that tracks prediction errors and supports deferral decisions. The paper has genuine strengths: Theorem 2.1 is correctly derived (up to a finite-sample factor of (N-1)/N), the experimental coverage is broad with three different backbone architectures and multiple missing-modality configurations, ablations isolate the contributions of reconstruction, uncertainty estimation, and pretraining, and the decision-making experiment is a concrete demonstration of the intended use. The code is promised for release. However, the headline claim of 'precise quantification of uncertainties' is undermined by an affine-invariance property of the proposed loss and by the absence of any calibration metric (UCE) in the reported tables, as detailed below.

major comments (3)

[Section 2.2, Eqs. (4)-(7)] The LPCC loss in Eq. (4)-(5) is invariant under the affine reparameterization sigma^2_i -> a*sigma^2_i + b with a>0, because Pearson correlation is unaffected by affine transformations. The reconstruction loss L_rec in Eq. (7) contains an MSE term, but that term supervises the reconstructed latent z_i, not sigma^2_{z_i}; the output-head uncertainty sigma^2_omega is trained only through LPCC. Hence neither sigma^2_z nor sigma^2_omega has a data-determined scale or offset, and the propagated sigma^2_input in Eq. (8) and the total sigma^2_Y in Eq. (10) inherit arbitrary units. The abstract's claim of 'precise quantification of uncertainties from missing data and model predictions' is therefore not supported by the method as written. I request either a calibration mechanism that fixes the scale/offset (e.g., a held-out temperature/affine calibration before reporting UCE) or an explicit restatement that only rank-based (monotone) uncertainty is claimed.
[Section 3.1 and Tables 1-3] Section 3.1 states that uncertainty quality is evaluated with both Uncertainty Calibration Error (UCE) and Pearson Correlation Coefficient (PCC), but Tables 1-3 report only Pearson correlations. UCE is the standard metric for assessing whether uncertainty values are calibrated in absolute terms, and it is precisely the metric affected by the affine-invariance issue. The absence of UCE numbers means the 'reliable uncertainty estimates' claim is not evidenced. Additionally, Section 4.2 explicitly acknowledges a 'tendency toward overestimating both reconstruction and output uncertainties,' which is an author-acknowledged calibration failure. Please add UCE results or explicitly drop the calibration claim.
[Proposition 2.2 and Eq. (8)] The first-order error-propagation formula in Eq. (8) assumes that the frozen fusion network omega is sufficiently smooth that a first-order Taylor expansion is accurate over the range of reconstruction errors. This is asserted rather than validated. For deep nonlinear fusion networks (transformers, attention-based fusion), reconstruction residuals are not necessarily small, and the Jacobian approximation can be badly biased. The paper should provide evidence of local linearity, such as a finite-difference or Monte Carlo comparison on the frozen network, or a bound on the reconstruction-error magnitude; otherwise the propagated input uncertainty may misstate the true contribution of missing modalities.

minor comments (5)

[Abstract and Section 1] The phrase 'applies statistical error propagation in deep networks for the first time' is a strong novelty claim; the literature review does not provide evidence that no prior work has used error propagation in deep networks. Please soften or provide a more careful search-based justification.
[Tables 1-3] The column groupings in Tables 1-3 are hard to parse: the T/A/F headings for reconstruction and output uncertainty are ambiguous, and the number of entries per row is inconsistent (e.g., the OutputUncertainty Corr row for SURE in Table 1 appears to have two values under a three-column header). Please reformat and ensure each row has the same number of entries as the header.
[Appendix A.2.2, Eq. (11)] 'SolfPlus' is a typo for 'SoftPlus'.
[Algorithm 1] In Algorithm 1, line 27 uses L_PCC(sigma^2_y; l_downstream), but l_downstream is the downstream loss value, not the per-sample error; for classification tasks the error used in LPCC should be defined explicitly (cross-entropy per sample) rather than relying on the mini-batch scalar.
[Section 3.2] The paper reports averages over three runs but does not report standard deviations; given the small-scale datasets and missing-modality masks, error bars would materially strengthen the SOTA claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the uncertainty estimates are trained on one split and reported on held-out data, and the error-propagation formula is imported from external references.

full rationale

I walked the derivation chain from Eq. (4)-(5) through Eq. (10) and the experimental tables. The LPCC loss is defined as 1 minus the Pearson correlation between the estimated uncertainty and the observed squared error; this is a training objective. The paper reports uncertainty-error correlations on evaluation/test data (Tables 1-3, and Figures 5-6 described as UTD-MHAD test dataset), so the reported alignment is not forced by construction: it is a supervised calibration check on held-out samples. Theorem 2.1 is a mathematical equivalence between LPCC and the MSE of standardized variables; it identifies the loss objective but is not a derivation of an empirical conclusion. The reconstruction uncertainty in Eq. (7) receives no scale from a data term: the MSE term applies to the reconstructed latent, not to sigma^2, and LPCC is affine-invariant, so the absolute scale of sigma^2 is not identifiable; the paper also omits the promised UCE metric and admits in Section 4.2 a 'tendency toward overestimating both reconstruction and output uncertainties.' These are calibration and identifiability concerns about the strength of the 'precise quantification' claim, not circularity: no prediction is equivalent to its inputs by definition, no parameter is renamed as a prediction, and no load-bearing argument reduces to a self-citation (there are no author self-citations). The error propagation Proposition 2.2 is taken from Arras (1998) and Tellinghuisen (2001), external sources, and its local-linearity assumption is an unverified modeling assumption rather than a circular step. The finding is therefore a normal honest non-finding: score 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

SURE introduces no new physical entities. It relies on two assumptions imported from measurement theory and a newly proposed correlation loss.

free parameters (1)

lambda (weighting in L_rec)
Introduced in Equation (7) as the balance between reconstruction MSE and Pearson-correlation uncertainty loss; the paper does not report its value or sensitivity.

assumptions (3)

domain assumption First-order error propagation formula applies to the frozen pretrained fusion function.
Used in Section 2.3 Proposition 2.2 to derive sigma^2_input; requires differentiability and local linearity of the deep network over the range of reconstruction errors.
domain assumption Input-induced and model-intrinsic uncertainties are independent, so variances add.
Equation (10) combines sigma^2_input and sigma^2_omega; if the two sources are correlated, the sum overestimates total uncertainty.
ad hoc to paper Pearson-correlation loss is a suitable objective for downstream uncertainty calibration.
Section 2.2 introduces L_PCC as the core uncertainty loss; it normalizes away scale, so the resulting uncertainty only carries ordering information, not absolute calibration.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation." pith.science (2026). https://pith.science/paper/B2ADVJ4F

@misc{pith2026250413465,
  author       = {Pith},
  title        = {Pith review of: Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2ADVJ4F}},
  note         = {Machine review of arXiv:2504.13465}
}

read the original abstract

Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present SURE (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks. Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.

Figures

Figures reproduced from arXiv: 2504.13465 by the authors.

**Figure 2.** SURE reconstructs missing modalities for final predictions, reporting average errors and uncertainties for both reconstruction (R.) and output (O.). the larger CMU-MOSEI dataset Zadeh et al. (2018) and another trained from scratch (i.e. vanilla). Performance was assessed across varying dataset sizes ( [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Overview of SURE. Reconstruction modules [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Correlation of estimated uncertainty with prediction error on UTD-MHAD dataset. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Figures from the paper (7 more)

**Figure 5.** Figure 5: Relationship between estimated output uncertainty and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 7.** Figure 7: Decision Making Process with Uncertainty on UTD-MHAD Dataset. ties across all test samples in the UTD-MHAD dataset, with different modalities combinations where each modality is missing. Ideally, the points should cluster along the bottom-left to top-right diagonal,…

**Figure 8.** Figure 8: Inter-relationship between estimated output uncertainty, reconstruction uncertainty and [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]

**Figure 9.** Figure 9: Decision Making with uncertainty when Video is missing [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

**Figure 10.** Figure 10: Decision Making with uncertainty when Accel is missing [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

**Figure 11.** Figure 11: Decision Making with uncertainty when Gyro is missing [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

**Figure 12.** Figure 12: Decision making analysis with different input modalities combinations on UTD-MHAD [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages

[1]

Space Complexity

=O(M∗L∗d2) By utilizing matrix product and GPU acceleration, d2 operations can in fact be performed inO(1) time, make the whole time complexity for individual branches beO(M∗L), which is linearly scaled withM. Space Complexity. Regarding the space complexity, within ith layer, beside the need for storing parameter matrix of size (di−1 + 1)×di, output afte...

work page 2024
[5]

Deep ordinal regression network for monocular depth estimation

Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2002–2011,

work page 2002
[7]

Hamlet: A hierarchical multimodal attention-based human activ- ity recognition algorithm

12 Md Mofijul Islam and Tariq Iqbal. Hamlet: A hierarchical multimodal attention-based human activ- ity recognition algorithm. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10285–10292. IEEE,

work page 2020
[8]

Towards robust multimodal prompting with missing modalities

Jaehyuk Jang, Yooseung Wang, and Changick Kim. Towards robust multimodal prompting with missing modalities. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8070–8074. IEEE,

work page 2024
[10]

Well-calibrated model uncertainty with temperature scaling for dropout variational inference

Max-Heinrich Laves, Sontje Ihler, Karl-Philipp Kortmann, and Tobias Ortmaier. Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv:1909.13550,

arXiv 1909
[11]

Dropout: a simple way to prevent neural networks from overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,

1929
[30]

URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf

Curran Associates, Inc., 2017a. URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf. Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017b. Alex Kendall and Yarin Gal. What uncertainties do ...

arXiv 2017
[2007]

Mixtures of laplace approximations for improved post-hoc uncertainty in deep learning

Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of laplace approximations for improved post-hoc uncertainty in deep learning. arXiv preprint arXiv:2111.03577,

arXiv

Show all 14 references

[2017]

Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition

Zirun Guo, Tao Jin, and Zhou Zhao. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. arXiv preprint arXiv:2407.05374,

arXiv
[2018]

Multimodal multi-loss fusion network for sentiment analysis

Zehui Wu, Ziwei Gong, Jaywon Koo, and Julia Hirschberg. Multimodal multi-loss fusion network for sentiment analysis. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Lon...

2024
[2020]

doi: 10.1007/s00521-019-04559-1

ISSN 0941-0643. doi: 10.1007/s00521-019-04559-1. URL https://doi.org/10.1007/ s00521-019-04559-1 . Kai O Arras. An introduction to error propagation: derivation, meaning and examples of equation cy= fx cx fxt. Technical report, ETH Zurich,

doi
[2021]

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

arXiv
[2022]

Utd-mhad: A multimodal dataset for hu- man action recognition utilizing a depth camera and a wearable inertial sensor

Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz. Utd-mhad: A multimodal dataset for hu- man action recognition utilizing a depth camera and a wearable inertial sensor. In 2015 IEEE International conference on image processing (ICIP), pp. 168–172. IEEE,

2015
[2023]

Recent research has focused on developing models resilient to missing modalities Ma et al

15 A A PPENDIX A.1 R ELATED WORKS Multimodal missing modalities. Recent research has focused on developing models resilient to missing modalities Ma et al. (2021; 2022); Poklukar et al. (2022); Woo et al. (2023); Lee et al. (2023); Li et al. (2024b); Lian et al. (2023); Li et ...

2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Space Complexity

=O(M∗L∗d2) By utilizing matrix product and GPU acceleration, d2 operations can in fact be performed inO(1) time, make the whole time complexity for individual branches beO(M∗L), which is linearly scaled withM. Space Complexity. Regarding the space complexity, within ith layer, beside the need for storing parameter matrix of size (di−1 + 1)×di, output afte...

work page 2024

[2] [5]

Deep ordinal regression network for monocular depth estimation

Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2002–2011,

work page 2002

[3] [7]

Hamlet: A hierarchical multimodal attention-based human activ- ity recognition algorithm

12 Md Mofijul Islam and Tariq Iqbal. Hamlet: A hierarchical multimodal attention-based human activ- ity recognition algorithm. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10285–10292. IEEE,

work page 2020

[4] [8]

Towards robust multimodal prompting with missing modalities

Jaehyuk Jang, Yooseung Wang, and Changick Kim. Towards robust multimodal prompting with missing modalities. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8070–8074. IEEE,

work page 2024

[5] [10]

Well-calibrated model uncertainty with temperature scaling for dropout variational inference

Max-Heinrich Laves, Sontje Ihler, Karl-Philipp Kortmann, and Tobias Ortmaier. Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv:1909.13550,

arXiv 1909

[6] [11]

Dropout: a simple way to prevent neural networks from overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,

1929

[7] [30]

URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf

Curran Associates, Inc., 2017a. URL https://proceedings.neurips.cc/paper_files/ paper/2017/file/2650d6089a6d640c5e85b2b88265dc2b-Paper.pdf. Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017b. Alex Kendall and Yarin Gal. What uncertainties do ...

arXiv 2017

[8] [2007]

Mixtures of laplace approximations for improved post-hoc uncertainty in deep learning

Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustinus Kristiadi. Mixtures of laplace approximations for improved post-hoc uncertainty in deep learning. arXiv preprint arXiv:2111.03577,

arXiv