Pith. sign in

REVIEW 4 major objections 43 references

When a modality is unreliable, its fusion weight should fall—and that weight can be learned from how much the prediction worsens without it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-07-14 10:33 UTC pith:VP4R44DX

load-bearing objection Solid reliability-aware fusion on DMD with consistent but modest MOSI/MOSEI gains; LOO teacher is the load-bearing assumption and is only weakly stress-tested. the 4 major comments →

arxiv 2607.10599 v1 pith:VP4R44DX submitted 2026-07-12 cs.AI eess.SP

MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis

Haoran Ma , Yinfeng Yu , Liejun Wang This is my paper
classification cs.AI eess.SP
keywords multimodal sentiment analysisreliability-aware fusionuncertainty-aware fusionmodality routingcontrastive alignmentleave-one-out supervisionCMU-MOSICMU-MOSEI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal sentiment systems often fuse language, vision, and audio as if every channel is equally trustworthy on every utterance. In practice one channel may be occluded, noisy, or mis-transcribed, so ordinary fusion over-trusts the bad evidence. This paper claims that reliability can be made explicit: first estimate each modality’s contribution by measuring how much prediction error rises when that modality is left out, then shrink its gate when the model also predicts high uncertainty for it. Built on a decoupling backbone, the method routes at both subspace and modality levels, adds contrastive alignment of the shared space, and keeps a single forward pass at test time. On the standard MOSI and MOSEI benchmarks, under both aligned and unaligned inputs, the resulting model improves classification and regression metrics over strong baselines, and a scatter analysis shows that higher predicted uncertainty tracks lower final weights. The practical stake is simple: systems that must read human affect under imperfect sensors need a way to down-weight the bad channel without throwing the utterance away.

Core claim

The paper establishes that utterance-level modality importance for sentiment fusion can be supervised by leave-one-out error increases and then calibrated by predicted modality uncertainty via inverse-variance reweighting; the resulting multi-granularity, reliability-aware fusion (MRUF) consistently outperforms strong baselines including its decoupling-distillation backbone on CMU-MOSI and CMU-MOSEI, while modalities with higher predicted uncertainty receive lower fusion weights.

What carries the argument

MRUF: multi-granularity routing (subspace gates plus modality gates) supervised by leave-one-out error increases, refined by inverse-variance uncertainty weights, and regularized by modality-invariant contrastive alignment.

Load-bearing premise

The method assumes that how much the model’s own prediction worsens when a modality is masked during training is a trustworthy teacher of that modality’s real importance at test time, when no masking is used.

What would settle it

On held-out utterances with controlled, ground-truth quality labels (e.g., known occlusion or noise levels), check whether the single-pass modality gates still rank modalities by true contribution; if high-uncertainty or leave-one-out-trained gates systematically promote the degraded channel, the reliability claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Final fusion no longer needs to treat every modality as equally trustworthy on every utterance; gates can shrink for occluded faces, noisy audio, or bad transcripts.
  • Training can use expensive leave-one-out teachers while inference stays a single forward pass, keeping deployment cost comparable to the backbone.
  • The same leave-one-out-plus-uncertainty pattern can be stacked on other decoupling or distillation backbones without redesigning the whole architecture.
  • Mechanism plots of uncertainty versus gate weight become a standard sanity check that the model is actually behaving as reliability-aware.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same LOO-error teacher could transfer to other asynchronous multimodal tasks (navigation, source separation) where one sensor routinely fails without a clean missing-modality flag.
  • If LOO teachers prove noisy on small data, cheaper proxies such as reconstruction residual or entropy of the modality encoder might approximate the same importance signal at lower training cost.
  • The residual MAE degradation under heavy masking suggests reliability-aware gates protect polarity better than intensity; intensity-specific heads may need a separate calibration path.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper proposes MRUF, a reliability-aware fusion method for multimodal sentiment analysis built on the DMD decoupling-distillation backbone. It adds (i) invariant-branch temporal summarization, (ii) multi-granularity routing at subspace and modality levels, with modality routing supervised by leave-one-out absolute-error increases (Eqs. 7–8), (iii) uncertainty-aware inverse-variance reweighting of modality gates (Eqs. 9–10), and (iv) modality-invariant contrastive alignment (MCA). On CMU-MOSI and CMU-MOSEI under aligned and unaligned settings, MRUF reports consistent gains over strong baselines including DMD, Self-MM, and EMOE (Tables I–II). Ablations (Tables III–IV), controlled robustness tests (Table V), and a mechanism plot of predicted uncertainty versus final gates (Fig. 2) are used to support the reliability-aware design.

Significance. If the reliability-aware story holds, the paper offers a practical and interpretable way to reduce over-trust of degraded language, visual, or acoustic cues in utterance-level sentiment fusion—an important failure mode for real multimodal systems. Strengths include evaluation under both aligned and unaligned protocols, comparison against recent competitive baselines with BERT language features, component ablations, a robustness suite, and an explicit mechanism check that higher predicted uncertainty yields lower fusion weights. The LOO-supervised router plus inverse-variance calibration is a coherent combination of known ideas (task contribution, uncertainty weighting, contrastive alignment) rather than a purely incremental fusion tweak. The contribution is solid applied multimodal learning work, not a foundational theoretical advance; its value depends on whether the LOO teacher truly induces generalizable reliability-aware routing at single-pass inference.

major comments (4)
  1. §III.C.3, Eqs. (7)–(8): The central “task-aware” claim treats leave-one-out absolute-error increases from the model’s own masked forward passes as a faithful teacher of utterance-level modality importance. Because the teacher is generated by the same backbone under optimization, it can encode current fusion biases rather than intrinsic contribution. Table IV only shows that LOO-drop+MSE wins among in-architecture supervision variants; it does not show that the single-pass router ˆπ at inference recovers the correct degraded-modality ranking when the teacher is unavailable. Please add a direct validation: under controlled modality drop/noise, report correlation or ranking agreement between inference-time ˆπ (or g_mod) and the true degraded modality, separate from the uncertainty–gate scatter in Fig. 2.
  2. Fig. 2 vs. §III.C–D: Mechanism analysis validates the uncertainty branch (higher log σ²_m → lower g_m) under synthetic drop/Gaussian noise, but does not isolate whether LOO-trained multi-granularity routing itself down-weights the truly unreliable modality. Given that final gates are g_m = g_mod_m · ω_m, the paper should disentangle g_mod and ω under the same perturbation conditions (e.g., ablate UAF vs. LOO routing, or plot g_mod alone). Without this, the reliability narrative attributes gains to “task-aware routing” more strongly than the evidence supports.
  3. Tables I–II and §IV.C: Results are averages over only three seeds with no standard deviations, confidence intervals, or significance tests. Gains over DMD/EMOE are modest on MOSEI (e.g., ACC7 52.3 vs. 51.6/52.2 aligned). For a reliability claim that rests on small metric improvements, please report mean±std over more seeds (or bootstrap CIs) and, where possible, paired significance tests against the strongest baselines. Otherwise it is hard to judge whether the improvements are stable.
  4. Table V and the accompanying discussion: Under severe text masking, MRUF improves ACC2/F1/ACC7 but can worsen MAE relative to DMD. The explanation that uncertainty-aware fusion protects discrete boundaries at the expense of intensity estimation is plausible but incomplete. Please quantify how often the model correctly down-weights the masked modality versus simply relying more on remaining modalities, and discuss whether LOO absolute-error supervision (Eq. 7) is well matched to classification metrics when regression quality degrades.

Circularity Check

1 steps flagged

Main MOSI/MOSEI gains are non-circular external evaluation; only mild self-definitional content is the Fig. 2 claim that higher uncertainty yields lower gates, which Eq. 10 largely enforces by construction.

specific steps
  1. self definitional [Sec. III-D Eqs. 9–10; Sec. IV-H / Fig. 2]
    "MRUF defines the final modality gate as g_m = g_mod_m ω_m, where ω_m is derived from predicted uncertainty. ... As shown in Fig. 2, the learned gates decrease as predicted uncertainty increases, especially under degraded inputs. This supports the intended reliability-aware behavior of MRUF."

    ω_m is defined as the normalized inverse of (σ²_m + ε). Therefore g_m falls when predicted uncertainty rises unless g_mod_m strongly compensates. Presenting the observed negative scatter as independent mechanism verification of reliability-aware fusion is partly tautological: the claimed relation is built into the gate formula rather than an external prediction checked against an independent reliability label.

full rationale

MRUF is an empirical multimodal fusion method, not a first-principles derivation. Its load-bearing performance claims are ACC7/ACC2/F1/MAE improvements on public CMU-MOSI and CMU-MOSEI splits against independent baselines (TFN, LMF, MulT, Self-MM, EMOE, DMD, etc.) under aligned and unaligned settings. Those metrics are not fitted inputs renamed as predictions, and the backbone DMD is external work (Li et al.), not a self-citation uniqueness theorem. Leave-one-out router supervision (Eqs. 7–8) is self-referential during training—the teacher ˜I_m is built from the model’s own masked absolute-error increases—but that is standard teacher-style training loss, used only at train time and removed at inference; it does not make the held-out test metrics true by construction. The only clear circular step is the mechanism claim that higher predicted uncertainty receives lower fusion weights: the final gate is defined as g_m = g_mod_m · ω_m with ω_m the normalized inverse variance of the predicted σ²_m, so a negative uncertainty–gate relationship is largely enforced by Eq. 10 rather than independently discovered. That affects only the interpretive Fig. 2 story, not the external benchmark results. Overall circularity is therefore minor (score 2).

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central empirical claim rests on standard multimodal-sentiment assumptions, the DMD backbone as a fixed representation substrate, hand-chosen loss weights, and the modeling premise that LOO error increases plus predicted log-variance are valid reliability signals. No physical constants are fitted; free parameters are training hyperparameters and architectural choices. Invented entities are named method modules rather than new natural kinds.

free parameters (4)
  • λ_r (router supervision weight)
    Set to 0.3 in implementation details; scales LOO router loss in the joint objective (Eq. 13) and directly affects how strongly modality gates track LOO teachers.
  • λ_m (MCA loss weight)
    Set to 0.07; controls strength of modality-invariant contrastive alignment relative to task and distillation losses.
  • contrastive temperature τ
    Described as learnable in Eq. 2; free scalar that shapes the MCA similarity distribution.
  • training hyperparameters (batch size 16, early-stopping patience 10, optimizer/schedule not fully specified)
    Chosen for reported runs; affect checkpoint selection and final metrics averaged over three seeds.
axioms (5)
  • domain assumption Utterance-level modality quality varies and can cause conventional fusion to over-trust unreliable modalities.
    Stated in abstract and Introduction as the motivating premise for reliability-aware fusion.
  • domain assumption DMD-style decoupled modality-invariant/specific representations plus graph distillation are a suitable backbone for final reliability-aware fusion.
    Section III builds MRUF explicitly on DMD representations and losses rather than deriving a new backbone.
  • ad hoc to paper Leave-one-out error increase Δ_m is a valid proxy for task contribution of modality m (Eqs. 7–8).
    Core supervision design of multi-granularity routing; not independently proven, only ablated against alternative teachers in Table IV.
  • domain assumption Predicted modality log-variance can be converted to inverse-variance weights that correctly down-weight unreliable modalities (Eqs. 9–10).
    Inspired by Kendall & Gal uncertainty weighting; adopted as the UAF calibration rule and checked via Fig. 2 correlation plots.
  • domain assumption Standard MOSI/MOSEI splits, ACC7/ACC2/F1/MAE definitions, and BERT/FACET/COVAREP features are valid evaluation protocol.
    Section IV follows prior work for datasets, metrics, and features.
invented entities (3)
  • MRUF multi-granularity router (subspace + modality gates with LOO teacher) no independent evidence
    purpose: Estimate interpretable utterance-level modality importance and balance specific vs invariant features.
    Named architectural module introduced in §III.C; independent_evidence false because its validity is only shown via this paper’s ablations and benchmarks.
  • Uncertainty-Aware Fusion (UAF) inverse-variance modality gates no independent evidence
    purpose: Calibrate content-based modality gates so higher predicted uncertainty yields lower fusion weight.
    Defined in §III.D as g_m = g_mod_m · ω_m; mechanism plots support intended behavior only within this study.
  • Modality-Invariant Contrastive Alignment (MCA) no independent evidence
    purpose: Stabilize the shared representation space used by routing via six-direction contrastive loss.
    Auxiliary regularizer in §III.B; standard contrastive form, but the named module and its role in MRUF are paper-specific.

reviewed 2026-07-14 · how reviews work

0 comments
Cite this review

Pith. "Pith review of MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis." pith.science (2026). https://pith.science/paper/VP4R44DX

@misc{pith2026260710599,
  author       = {Pith},
  title        = {Pith review of: MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VP4R44DX}},
  note         = {Machine review of arXiv:2607.10599}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal sentiment analysis relies on language, visual, and acoustic cues, but utterance-level modality quality may vary due to occlusion, background noise, motion blur, or imperfect transcripts, causing conventional fusion to over-trust unreliable modalities. We propose MRUF, a reliability-aware fusion method that combines multi-granularity routing with uncertainty-aware calibration. MRUF summarizes sentiment-relevant representations, performs subspace- and modality-level routing, and supervises modality routing with leave-one-out error increases to estimate utterance-level modality importance. It further predicts modality-wise uncertainty and refines modality gates through inverse-variance reweighting, while modality-invariant contrastive alignment stabilizes the shared representation space. Experiments on CMU-MOSI and CMU-MOSEI under aligned and unaligned settings show consistent improvements over strong baselines, and mechanism analysis verifies that modalities with higher predicted uncertainty receive lower fusion weights.

Figures

Figures reproduced from arXiv: 2607.10599 by Haoran Ma, Liejun Wang, Yinfeng Yu.

Figure 1
Figure 1. Figure 1: Overview of MRUF. MRUF extends DMD representations with invariant-branch summarization, modality-invariant contrastive alignment, task [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mechanism analysis of MRUF. Scatter plots of predicted uncertainty [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 3 linked inside Pith

  1. [1]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltru ˇsaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2019

    2019

  2. [2]

    Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages,

    A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages,”IEEE Intelligent Systems, vol. 31, no. 6, pp. 82–88, 2016

    2016

  3. [3]

    Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,

    A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2236–2246

    2018

  4. [4]

    Tensor fusion network for multimodal sentiment analysis,

    A. Zadeh, M. Chen, S. Poria, E. Cambria, and L.-P. Morency, “Tensor fusion network for multimodal sentiment analysis,”arXiv preprint arXiv:1707.07250, 2017

    Pith/arXiv arXiv 2017

  5. [5]

    Efficient low-rank multimodal fusion with modality-specific factors,

    Z. Liu, Y . Shen, V . B. Lakshminarasimhan, P. P. Liang, A. B. Zadeh, and L.-P. Morency, “Efficient low-rank multimodal fusion with modality-specific factors,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2247–2256

    2018

  6. [6]

    Multimodal transformer for unaligned multimodal language sequences,

    Y .-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, “Multimodal transformer for unaligned multimodal language sequences,” inProceedings of the Conference. Association for Computational Linguistics. Meeting, vol. 2019, 2019, p. 6558

    2019

  7. [7]

    Mtag: Modal-temporal attention graph for unaligned human multimodal language sequences,

    J. Yang, Y . Wang, R. Yi, Y . Zhu, A. Rehman, A. Zadeh, S. Poria, and L.-P. Morency, “Mtag: Modal-temporal attention graph for unaligned human multimodal language sequences,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 1009–1021

    2021

  8. [8]

    Progressive modal- ity reinforcement for human multimodal emotion recognition from unaligned multimodal sequences,

    F. Lv, X. Chen, Y . Huang, L. Duan, and G. Lin, “Progressive modal- ity reinforcement for human multimodal emotion recognition from unaligned multimodal sequences,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2554–2562

    2021

  9. [9]

    Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion,

    T. Liang, G. Lin, L. Feng, Y . Zhang, and F. Lv, “Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion,” inProceedings of the IEEE/CVF Inter- national Conference on Computer Vision, 2021, pp. 8148–8156

    2021

  10. [10]

    Multimodal emotion recognition calibration in conversations,

    G. Tu, F. Xiong, B. Liang, H. Wang, X. Zeng, and R. Xu, “Multimodal emotion recognition calibration in conversations,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 9621–9630

    2024

  11. [11]

    Contrastive learn- ing based modality-invariant feature acquisition for robust multimodal emotion recognition with missing modalities,

    R. Liu, H. Zuo, Z. Lian, B. W. Schuller, and H. Li, “Contrastive learn- ing based modality-invariant feature acquisition for robust multimodal emotion recognition with missing modalities,”IEEE Transactions on Affective Computing, vol. 15, no. 4, pp. 1856–1873, 2024

    2024

  12. [12]

    Carat: Contrastive feature reconstruction and aggregation for multi-modal multi-label emotion recognition,

    C. Peng, K. Chen, L. Shou, and G. Chen, “Carat: Contrastive feature reconstruction and aggregation for multi-modal multi-label emotion recognition,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 13, 2024, pp. 14 581–14 589

    2024

  13. [13]

    Emoe: Modality- specific enhanced dynamic emotion experts,

    Y . Fang, W. Huang, G. Wan, K. Su, and M. Ye, “Emoe: Modality- specific enhanced dynamic emotion experts,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 14 314–14 324

    2025

  14. [14]

    Modality-invariant bidirec- tional temporal representation distillation network for missing multi- modal sentiment analysis,

    X. Wang, L. Wang, Y . Yu, and X. Jiao, “Modality-invariant bidirec- tional temporal representation distillation network for missing multi- modal sentiment analysis,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  15. [15]

    Sound adversarial audio-visual navigation,

    Y . Yu, W. Huang, F. Sun, C. Chen, Y . Wang, and X. Liu, “Sound adversarial audio-visual navigation,” inThe Tenth International Con- ference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022

    2022

  16. [16]

    Measuring acoustics with collaborative multiple agents,

    Y . Yu, C. Chen, L. Cao, F. Yang, and F. Sun, “Measuring acoustics with collaborative multiple agents,” inProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023, pp. 335–343

    2023

  17. [17]

    Audio-guided dynamic modality fusion with stereo-aware attention for audio-visual naviga- tion,

    J. Li, Y . Yu, L. Wang, F. Sun, and W. Zheng, “Audio-guided dynamic modality fusion with stereo-aware attention for audio-visual naviga- tion,” inInternational Conference on Neural Information Processing. Springer, 2025, pp. 346–359

    2025

  18. [18]

    Beyond textual knowledge: Leverag- ing multimodal knowledge bases for enhancing vision-and-language navigation,

    D. Yang, Y . Yu, and L. Wang, “Beyond textual knowledge: Leverag- ing multimodal knowledge bases for enhancing vision-and-language navigation,”Information Processing & Management, vol. 63, no. 6, p. 104766, 2026

    2026

  19. [19]

    Dope: Dual object perception-enhancement network for vision-and-language navigation,

    Y . Yu and D. Yang, “Dope: Dual object perception-enhancement network for vision-and-language navigation,” inProceedings of the 2025 International Conference on Multimedia Retrieval, 2025, pp. 1739–1748

    2025

  20. [20]

    Dgfnet: End-to-end audio-visual source separation based on dynamic gating fusion,

    Y . Yu and S. Sun, “Dgfnet: End-to-end audio-visual source separation based on dynamic gating fusion,” inProceedings of the 2025 Interna- tional Conference on Multimedia Retrieval, 2025, pp. 1730–1738

    2025

  21. [21]

    Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,

    D. Hazarika, R. Zimmermann, and S. Poria, “Misa: Modality-invariant and-specific representations for multimodal sentiment analysis,” in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 1122–1131

    2020

  22. [22]

    Decoupled multimodal distilling for emotion recognition,

    Y . Li, Y . Wang, and Z. Cui, “Decoupled multimodal distilling for emotion recognition,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6631–6640

    2023

  23. [23]

    Co-space representation interaction network for multimodal sentiment analysis,

    H. Shi, Y . Pu, Z. Zhao, J. Huang, D. Zhou, D. Xu, and J. Cao, “Co-space representation interaction network for multimodal sentiment analysis,”Knowledge-Based Systems, vol. 283, p. 111149, 2024

    2024

  24. [24]

    Weavenet: End- to-end audiovisual sentiment analysis,

    Y . Yu, Z. Jia, F. Shi, M. Zhu, W. Wang, and X. Li, “Weavenet: End- to-end audiovisual sentiment analysis,” inInternational Conference on Cognitive Systems and Signal Processing, 2021, pp. 3–16

    2021

  25. [25]

    Words can shift: Dynamically adjusting word representations using nonverbal behaviors,

    Y . Wang, Y . Shen, Z. Liu, P. P. Liang, A. Zadeh, and L.-P. Morency, “Words can shift: Dynamically adjusting word representations using nonverbal behaviors,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 7216–7223

    2019

  26. [26]

    Target and source modality co- reinforcement for emotion understanding from asynchronous multi- modal sequences,

    D. Yang, Y . Liu, C. Huang, M. Li, X. Zhao, Y . Wang, K. Yang, Y . Wang, P. Zhai, and L. Zhang, “Target and source modality co- reinforcement for emotion understanding from asynchronous multi- modal sequences,”Knowledge-Based Systems, vol. 265, p. 110370, 2023

    2023

  27. [27]

    Iterative residual cross-attention mechanism: An integrated approach for audio-visual navigation tasks,

    H. Zhang, Y . Yu, L. Wang, F. Sun, and W. Zheng, “Iterative residual cross-attention mechanism: An integrated approach for audio-visual navigation tasks,”arXiv preprint arXiv:2509.25652, 2025

    arXiv 2025

  28. [28]

    Advancing audio-visual navigation through multi-agent collab- oration in 3d environments,

    ——, “Advancing audio-visual navigation through multi-agent collab- oration in 3d environments,” inInternational Conference on Neural Information Processing. Springer, 2025, pp. 502–516

    2025

  29. [29]

    Dynamic multi-target fusion for efficient audio-visual navigation,

    Y . Yu, H. Zhang, and M. Zhu, “Dynamic multi-target fusion for efficient audio-visual navigation,”arXiv preprint arXiv:2509.21377, 2025

    arXiv 2025

  30. [30]

    Bss-cffma: cross-domain feature fusion and multi-attention speech enhancement network based on self- supervised embedding,

    A. Mattursun, L. Wang, and Y . Yu, “Bss-cffma: cross-domain feature fusion and multi-attention speech enhancement network based on self- supervised embedding,” in2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2024, pp. 3589–3594

    2024

  31. [31]

    Nonlinear regularization decoding method for speech recognition,

    J. Zhang, L. Wang, Y . Yu, and M. Xu, “Nonlinear regularization decoding method for speech recognition,”Sensors, vol. 24, no. 12, p. 3846, 2024

    2024

  32. [32]

    Vnet: A gan-based multi-tier discriminator network for speech synthesis vocoders,

    Y . Cao, Y . Li, L. Wang, and Y . Yu, “Vnet: A gan-based multi-tier discriminator network for speech synthesis vocoders,” in2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2024, pp. 4384–4389

    2024

  33. [33]

    Fsdenet: A frequency and spatial do- mains based detail enhancement network for remote sensing semantic segmentation,

    J. Fu, Y . Yu, and L. Wang, “Fsdenet: A frequency and spatial do- mains based detail enhancement network for remote sensing semantic segmentation,”IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2025

    2025

  34. [34]

    Moddrop: Adaptive multi-modal gesture recognition,

    N. Neverova, C. Wolf, G. Taylor, and F. Nebout, “Moddrop: Adaptive multi-modal gesture recognition,”IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, vol. 38, no. 8, pp. 1692–1706, 2015

    2015

  35. [35]

    Gated multimodal units for information fusion,

    J. Arevalo, T. Solorio, M. Montes-yG ´omez, and F. A. Gonz ´alez, “Gated multimodal units for information fusion,”arXiv preprint arXiv:1702.01992, 2017

    Pith/arXiv arXiv 2017

  36. [36]

    Memory fusion network for multi-view sequential learn- ing,

    A. Zadeh, P. P. Liang, N. Mazumder, S. Poria, E. Cambria, and L.-P. Morency, “Memory fusion network for multi-view sequential learn- ing,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018

    2018

  37. [37]

    Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis,

    L. Sun, Z. Lian, B. Liu, and J. Tao, “Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis,”IEEE Transactions on Affective Computing, vol. 15, no. 1, pp. 309–325, 2023

    2023

  38. [38]

    Learning factorized multimodal representations,

    Y .-H. H. Tsai, P. P. Liang, A. Zadeh, L.-P. Morency, and R. Salakhutdi- nov, “Learning factorized multimodal representations,”arXiv preprint arXiv:1806.06176, 2018

    Pith/arXiv arXiv 2018

  39. [39]

    Disentangled representation learning for multimodal emotion recognition,

    D. Yang, S. Huang, H. Kuang, Y . Du, and L. Zhang, “Disentangled representation learning for multimodal emotion recognition,” inPro- ceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 1642–1651

    2022

  40. [40]

    Found in translation: Learning robust joint representations by cyclic translations between modalities,

    H. Pham, P. P. Liang, T. Manzini, L.-P. Morency, and B. P ´oczos, “Found in translation: Learning robust joint representations by cyclic translations between modalities,” inProceedings of the AAAI Confer- ence on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 6892–6899

    2019

  41. [41]

    What uncertainties do we need in bayesian deep learning for computer vision?

    A. Kendall and Y . Gal, “What uncertainties do we need in bayesian deep learning for computer vision?”Advances in Neural Information Processing Systems, vol. 30, 2017

    2017

  42. [42]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171– 4186

    2019

  43. [43]

    Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis,

    W. Yu, H. Xu, Z. Yuan, and J. Wu, “Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis,” inProceedings of the AAAI Conference on Arti- ficial Intelligence, vol. 35, no. 12, 2021, pp. 10 790–10 797

    2021

This paper was first reviewed by grok-4.5 on July 14, 2026.