Pith. sign in

REVIEW 4 major objections 6 minor 41 references

Improving Multimodal Learning via Imbalanced Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that optimal multimodal learning is imbalanced: each modality should contribute in inverse proportion to its prediction variance, and proposes the ARL training rule to realize that ratio.

desk verdict A genuinely useful empirical training plugin whose central proof doesn't survive contact with the covariance term; referee it for the experiments, but don't let the theory slide. read the letter →

arxiv 2507.10203 v2 pith:QOMJKHT3 submitted 2025-07-14 cs.CV

Shicai Wei , Chunbo Luo , Yang Luo This is my paper
classification cs.CV
keywords multimodallearningimbalancedbias-variancedecompositiongradientmodulationasymmetricrepresentationmodalityvarianceunder-optimized
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal models that fuse several input streams are commonly trained with gradient-balancing tricks meant to equalize how much each modality influences learning. This paper argues that equalizing is the wrong target: by a bias-variance decomposition of the fused predictor, the optimal contribution of each modality is inversely proportional to that modality's prediction variance, so the model should rely more on the low-variance modality. To realize that ratio in practice, the paper introduces Asymmetric Representation Learning (ARL), a training plugin that estimates each branch's variance from its softmax logits and rescales per-modality gradients to track the inverse-variance ratio, with no extra parameters and no dependence on the fusion architecture. If the claim holds, balanced multimodal learning is not just unnecessary but suboptimal, and a simple gradient-rescaling rule can improve a wide range of multimodal systems.

What carries the argument

The load-bearing object is the optimization dependency ratio $d_{m_0}/d_{m_1}$, defined as the ratio of softmax-normalized logit probabilities of the two modality branches; ARL's goal is to make this ratio track the inverse variance ratio $q_{m_0}/q_{m_1}$, where each $q$ is estimated from the self-information entropy of a branch's logits. The mechanism is a per-modality gradient modulation: after the backward pass, the gradient arriving at each encoder is rescaled by a softmax over the two ratios (with temperature $T$) to push the dependency ratio toward the inverse-variance ratio, plus the original gradient retained as a residual so a strongly suppressed branch still updates. A unimodal bias regularizer, the cross-entropy of each branch's own logits, is added to the multimodal loss with weight $\gamma$ to reduce the bias term that cannot be minimized by any convex mixture of modalities.

What would settle it

Train a two-modality model, record the unimodal logits at convergence, and compute the sample covariance between $s_{m_0}$ and $s_{m_1}$; if the covariance term is non-negligible, compare validation accuracy when weights are set by the paper's inverse-variance ratio versus weights that minimize the full variance including $2 w_0 w_1 \mathrm{Cov}(s_{m_0}, s_{m_1})$ — if the full-variance weights win, the claimed optimality fails in that regime.

Watch

Extended reading notes

Core claim

The paper's central claim is that the under-optimized behavior of multimodal networks should not be blamed on imbalance between modalities; the real optimum is an imbalanced allocation. Writing the fused logit as $f = w_0 s_{m_0} + w_1 s_{m_1}$ and minimizing the bias-variance decomposition of the generalization error, the paper derives $w_0/w_1 = (1/\mathrm{Var}(s_{m_0}))/(1/\mathrm{Var}(s_{m_1}))$, so the optimal dependency ratio equals the inverse variance ratio. Since balanced learning is optimal only in the special case of equal variances, the paper re-frames existing gradient-balancing methods as enforcing a suboptimal target. ARL operationalizes the inverse-variance rule by computing each modality's variance through the self-information entropy of its own logits, reweighting the gradients from the shared fusion module, and adding a residual gradient to keep the suppressed branch updating; a unimodal cross-entropy term controls prediction bias. The paper reports consistent gains over balanced baselines across five datasets, including cases where balancing methods hurt performance.

Load-bearing premise

The proof that inverse-variance weights are optimal treats the two modality predictions as statistically independent, so the variance of their weighted sum is just the sum of the weighted variances; if the modality logits are strongly correlated in a jointly trained network, the derived ratio may no longer be the exact minimum.

Editorial extensions

If this is right

  • Balanced gradient methods are optimal only when modality variances coincide; whenever variances differ, the paper's theory predicts they underperform a variance-aware allocation, and the experiments show balancing methods can even degrade accuracy.
  • ARL can be applied without architectural changes: it introduces no extra parameters, works with two or three modalities, and improves both CNN and Transformer backbones and both late and intermediate fusion.
  • The optimal dependency ratio is not generally 1; on the A VE dataset the tracked ratio converges to about 3, giving a direct visual demonstration that imbalanced reliance is the target.
  • Jointly minimizing each unimodal bias, as ARL does, is needed because no nonnegative weighting of the fused branches can reduce the bias term once the branch biases are fixed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the inverse-variance rule transfers beyond multimodal networks, it predicts that ensemble methods should also weight predictors by reciprocal variance; this is a testable extension the paper does not discuss.
  • The derivation drops the covariance between the two modality logits, so the exact optimum under correlated branches likely includes a covariance term; testing whether the inverse-variance rule still wins when covariance is large would separate the heuristic from the proof.
  • The paper's limitation note that single-inference entropy 'may be imperfect' suggests a concrete follow-up: estimate variance over multiple stochastic forward passes and check whether more accurate coefficients improve the gains.
  • Because ARL leaves the fusion module untouched, it could be combined with learned fusion weights or attention-based fusion to see whether the inverse-variance target and learnable fusion are complementary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper challenges the prevailing view that multimodal learning suffers from imbalanced gradient updates that should be balanced. The authors claim to prove that the optimal dependency ratio between modalities is the inverse ratio of their prediction variances, so balanced learning is optimal only when variances are equal. They propose Asymmetric Representation Learning (ARL), which estimates each modality's variance via the entropy of its softmax logits, modulates per-modality gradients by a softmax over temperature-scaled variance and dependency ratios, and adds unimodal cross-entropy losses to reduce bias. Experiments on CREMA-D, AVE, Kinetics-Sounds, MOSI, and UCF-101 show consistent improvements over existing gradient modulation and regularization baselines across CNN and Transformer backbones.

Significance. The paper has the potential to reframe the discussion of multimodal optimization from balancing to variance-aware asymmetry. The proposed ARL is simple, introduces no additional network parameters, and is agnostic to fusion architectures; the code is released and the experiments cover five datasets, multiple backbones, and several fusion strategies. If the theoretical claim were rigorously established, the inverse-variance law would be a useful design principle. However, the proof has a load-bearing gap (omission of covariance), and the link from the theoretical optimization problem to the implemented gradient rule is not made. The empirical results are strong and suggest the heuristic is valuable, but the central theoretical claim as stated is not yet supported.

major comments (4)
  1. [Section 3.1, Eq. (13)] The derivation of the optimal weighting ratio implicitly assumes Var(w0 s_m0 + w1 s_m1) = w0^2 Var(s_m0) + w1^2 Var(s_m1), dropping the covariance term 2 w0 w1 Cov(s_m0, s_m1). In a jointly trained multimodal network with shared labels and fused representations, the unimodal logits are not independent, so the variance-minimizing weights are w0 = (V1 - C)/(V0 + V1 - 2C) and w1 = (V0 - C)/(V0 + V1 - 2C) where C is the covariance. The asserted inverse-variance ratio (Eq. 16) follows only when C=0, a condition never stated or justified. Since this is the foundation of the paper's central claim, the proof is incomplete; the authors should either state and justify an independence assumption, or generalize the result to account for nonzero covariance (e.g., via a bound or a modified ratio).
  2. [Section 3.2, Eqs. (18)-(20) vs. Eqs. (5)-(6)] The bias-variance analysis optimizes fixed combination weights w0 and w1 in f = w0 s_m0 + w1 s_m1, yielding a statement about the final logit weighting. ARL, however, modulates gradients to steer the softmax probability ratio d_m0/d_m1 (Eq. 4) toward the variance ratio q_m0/q_m1. No argument is given that aligning the softmax ratio to the variance ratio realizes the optimal weighting of the bias-variance problem; the softmax is nonlinear and gradient modulation does not directly set the combination weights. The theoretical support for the specific ARL rule is therefore missing, even if the inverse-variance result were correct. Please provide a formal derivation connecting the optimized quantity (d) to the optimal weights, or explicitly present ARL as a heuristic motivated by the variance analysis.
  3. [Section 3.1, Eqs. (11)-(12) and surrounding text] The claim that 'one of w0 or w1 must be smaller than 0' when minimizing Bias(f,y)^2 is false when the two unimodal biases have opposite signs. For instance, with Bias(s_m0,y)=1 and Bias(s_m1,y)=-1, Eqs. (11)-(12) give w0=0.5 and w1=0.5, both positive and yielding zero bias. The conclusion that bias can only be reduced by minimizing the individual unimodal biases is therefore not generally valid. This error undermines the justification for the unimodal bias regularization term in Eq. (22), even though that component shows empirical gains. Please correct the analysis or qualify the claim.
  4. [Section 3.2, Eq. (17), and Section 5 (Limitation)] The variance ratio q_m0/q_m1 is computed from the model's own softmax entropy, and the modulation target is defined with respect to this self-measured quantity. The paper's statement that imbalanced learning 'obeying the inverse of their variances' contributes to 'optimal performance' is thus not validated against an independent ground-truth variance or an externally defined optimal dependence ratio; it is a self-consistency condition on the model's predictions. This does not invalidate the empirical gains, but it should be acknowledged that the theoretical optimality claim is conditional on the model's own variance estimates, and the claim of 'optimal performance' in the abstract is stronger than what the experiments and analysis support.
minor comments (6)
  1. [Eqs. (14) and (15)] Both denominators read 'V ar(s_m1) + V ar(s_m1)' but should be 'V ar(s_m0) + V ar(s_m1)'; please correct the typographical error.
  2. [Paragraph after Eq. (12)] The phrase 'conflicting with w1 > 0 and w1 > 0' repeats w1; it should be 'w0 > 0 and w1 > 0'.
  3. [Section 4.2] The temperature T is set to different values per dataset (8,4,4,4,4) and gamma is fixed at 4; the paper does not report sensitivity to these hyperparameters, and a short study would clarify how robust the method is to their choice.
  4. [Tables 1 and 2] Results are reported without standard deviations or significance tests; given the large reported gains, reporting mean and standard deviation over multiple seeds would strengthen the empirical claims.
  5. [Figure 1(c)] The caption introduces d_a/d_v and q_a/q_v but does not define these symbols; please define them in the caption or in the text.
  6. [Section 4.1] CREMA-D input size is given as 224x244, which appears to be a typo for 224x224; if the non-square size is intentional, please clarify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central inverse-variance claim is derived from an explicit bias-variance decomposition, and ARL's entropy-based target is an approximation rather than a fitted prediction.

full rationale

The paper's main theoretical step, Eq. 16, is derived by minimizing Var(f) with respect to w0 and w1 under w0 + w1 = 1, and the result w0/w1 = (1/Var(sm0))/(1/Var(sm1)) is presented as a self-contained algebraic consequence of that minimization. The subsequent ARL target q is defined as the inverse self-information entropy of each modality's softmax output, which is an approximation of variance adopted from external references [12,39], not a quantity fitted to the final accuracy and then relabeled as a prediction. The fact that q is measured from the model's own logits and then used to modulate that same model's gradients creates a self-referential training loop, but it does not make the claimed optimality proof circular, because the optimality claim rests on the bias-variance derivation rather than on matching q to any performance datum. The per-dataset temperature T and bias weight gamma are hyperparameters; tuning hyperparameters and then reporting held-out accuracy is standard practice and is not shown to encode the evaluated metric. Self-citations in the related work are contextual and are not used to justify the central derivation or to import an unverified uniqueness theorem. The omitted covariance term in Eq. 13 is a genuine mathematical correctness risk, since Var(w0 sm0 + w1 sm1) should include 2 w0 w1 Cov(sm0, sm1), but this is an error in the derivation, not a circular reduction of the result to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The derivation rests on two unstated assumptions: independence of modality logits in Eq 13, and the equivalence between static ensemble weights and train-time gradient multipliers. The only fitted scalars are T and gamma. No invented entities are introduced.

free parameters (2)
  • temperature T = 8, 4, 4, 4, 4 for CREMA-D, AVE, KS, MOSI, UCF101
    Controls softmax sharpness in Eq 18; tuned per dataset with no sensitivity analysis in the main text.
  • gamma = 4 for all datasets
    Weight for unimodal bias regularization in Eq 22; set by hand.
assumptions (4)
  • domain assumption Bias-variance decomposition applies to classification with entropy as a variance surrogate.
    Eq 17 uses self-information entropy to stand in for regression variance, following references [12, 39]; no derivation for classification is given.
  • ad hoc to paper Modality logits are independent, so Var(f) = w0^2 Var(s0) + w1^2 Var(s1).
    Eq 13 omits the covariance term; this is not stated or justified, and jointly trained fused logits are typically correlated.
  • ad hoc to paper The ratio d of softmax probabilities measures optimization dependency and can be steered by gradient multipliers.
    Eq 4 defines d, and Eqs 18-20 assert the steering mechanism; no proof connects the two.
  • standard math Lagrange multiplier optimization with w0 + w1 = 1 is valid for the stated objective.
    Used to compute w0 and w1 in Eqs 11-15; this is standard constrained optimization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Multimodal Learning via Imbalanced Learning." pith.science (2026). https://pith.science/paper/QOMJKHT3

@misc{pith2026250710203,
  author       = {Pith},
  title        = {Pith review of: Improving Multimodal Learning via Imbalanced Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOMJKHT3}},
  note         = {Machine review of arXiv:2507.10203}
}
read the original abstract

Multimodal learning often encounters the under-optimized problem and may perform worse than unimodal learning. Existing approaches attribute this issue to imbalanced learning across modalities and tend to address it through gradient balancing. However, this paper argues that balanced learning is not the optimal setting for multimodal learning. With bias-variance analysis, we prove that imbalanced dependency on each modality obeying the inverse ratio of their variances contributes to optimal performance. To this end, we propose the Asymmetric Representation Learning(ARL) strategy to assist multimodal learning via imbalanced optimization. ARL introduces auxiliary regularizers for each modality encoder to calculate their prediction variance. ARL then calculates coefficients via the unimodal variance to re-weight the optimization of each modality, forcing the modality dependence ratio to be inversely proportional to the modality variance ratio. Moreover, to minimize the generalization error, ARL further introduces the prediction bias of each modality and jointly optimizes them with multimodal loss. Notably, all auxiliary regularizers share parameters with the multimodal model and rely only on the modality representation. Thus the proposed ARL strategy introduces no extra parameters and is independent of the structures and fusion methods of the multimodal model. Finally, extensive experiments on various datasets validate the effectiveness and versatility of ARL. Code is available at \href{https://github.com/shicaiwei123/ICCV2025-ARL}{https://github.com/shicaiwei123/ICCV2025-ARL}

Figures

Figures reproduced from arXiv: 2507.10203 by the authors.

Figure 1
Figure 1. The visualization on the audio-visual dataset CREMA-D. (a) presents the performance of the unimodal branch in the vanilla [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of the multimodal model with ARL strategy. It consists of three components: the modality analysis to measure the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the optimization dependency and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 33 canonical work pages

  1. [1]

    Deep audio-visual speech recognition

    Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 44(12):8717–8727, 2018. 1, 2

    work page 2018

  2. [2]

    Learnable irrelevant modality dropout for multimodal action recogni- tion on modality-specific annotated videos

    Saghir Alfasly, Jian Lu, Chen Xu, and Yuru Zou. Learnable irrelevant modality dropout for multimodal action recogni- tion on modality-specific annotated videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 20208–20217, 2022. 1, 2

    work page 2022

  3. [3]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pages 609–617, 2017. 6

    work page 2017

  4. [4]

    Mmnet: A model-based multimodal net- work for human action recognition in rgb-d videos

    XB Bruce, Yan Liu, Xiang Zhang, Sheng-hua Zhong, and Keith CC Chan. Mmnet: A model-based multimodal net- work for human action recognition in rgb-d videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3522–3538, 2022. 2

    work page 2022

  5. [5]

    Crema-d: Crowd-sourced emotional multimodal actors dataset

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377–390, 2014. 6

    work page 2014

  6. [6]

    Shapeconv: Shape- aware convolutional layer for indoor rgb-d semantic segmen- tation

    Jinming Cao, Hanchao Leng, Dani Lischinski, Daniel Cohen-Or, Changhe Tu, and Yangyan Li. Shapeconv: Shape- aware convolutional layer for indoor rgb-d semantic segmen- tation. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 7088–7097, 2021. 2

    work page 2021

  7. [7]

    Mars: Motion-augmented rgb stream for action recognition

    Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari, and Cordelia Schmid. Mars: Motion-augmented rgb stream for action recognition. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7882–7891, 2019. 2, 3

    work page 2019

  8. [8]

    Rfnet: Region-aware fusion network for incomplete multi-modal brain tumor seg- mentation

    Yuhang Ding, Xin Yu, and Yi Yang. Rfnet: Region-aware fusion network for incomplete multi-modal brain tumor seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3975–3984, 2021. 2

    work page 2021

Show all 41 references
  1. [9]

    Improving multi- modal learning with uni-modal teachers

    Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Improving multi- modal learning with uni-modal teachers. arXiv preprint arXiv:2106.11059, 2021. 1, 3

    2021 arXiv

  2. [10]

    Pmr: Prototypical modal rebalance for multi- modal learning

    Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multi- modal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20029– 20038, 2023. 1, 3, 6, 7

    2023

  3. [11]

    Modal- ity distillation with multiple stream networks for action recognition

    Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modal- ity distillation with multiple stream networks for action recognition. In Proceedings of the European Conference on Computer Vision, pages 103–118, 2018. 2

    2018

  4. [12]

    Ensembles of classifiers: a bias-variance perspective

    Neha Gupta, Jamie Smith, Ben Adlam, and Zelda E Mariet. Ensembles of classifiers: a bias-variance perspective. Trans- actions on Machine Learning Research, 2022. 5

    2022

  5. [13]

    Deep multimodal multilinear fusion with high- order polynomial pooling

    Ming Hou, Jiajia Tang, Jianhai Zhang, Wanzeng Kong, and Qibin Zhao. Deep multimodal multilinear fusion with high- order polynomial pooling. Advances in Neural Information Processing Systems, 32, 2019. 3

    2019

  6. [14]

    Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation

    Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang. Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1440–1444. IEEE, 2019. 2

    2019

  7. [15]

    Reconboost: Boosting can achieve modal- ity reconcilement

    Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modal- ity reconcilement. arXiv preprint arXiv:2405.09321, 2024. 1, 3

    2024 arXiv

  8. [16]

    Cdnet: Complementary depth network for rgb-d salient object detection

    Wen-Da Jin, Jun Xu, Qi Han, Yi Zhang, and Ming-Ming Cheng. Cdnet: Complementary depth network for rgb-d salient object detection. IEEE Transactions on Image Pro- cessing, 30:3376–3390, 2021. 1, 2

    2021

  9. [17]

    Mmtm: Multimodal transfer module for cnn fusion

    Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuz- zolino, and Kazuhito Koishida. Mmtm: Multimodal transfer module for cnn fusion. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 13289–13299, 2020. 7

    2020

  10. [18]

    Boosting multi-modal model performance with adaptive gradient modulation

    Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting multi-modal model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22214–22224, 2023. 1, 3, 6, 7

    2023

  11. [19]

    Dynamic- hierarchical attention distillation with synergetic instance se- lection for land cover classification using missing hetero- geneity images

    Xiao Li, Lin Lei, Yuli Sun, and Gangyao Kuang. Dynamic- hierarchical attention distillation with synergetic instance se- lection for land cover classification using missing hetero- geneity images. IEEE Transactions on Geoscience and Re- mote Sensing, 60:1–16, 2021. 2

    2021

  12. [20]

    librosa: Audio and music signal analysis in python

    Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, pages 18–25, 2015. 6

    2015

  13. [21]

    Balanced multimodal learning via on-the-fly gradient modulation

    Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8238– 8247, 2022. 1, 3, 6, 7

    2022

  14. [22]

    Efficient rgb-d seman- tic segmentation for indoor scene analysis

    Daniel Seichter, Mona K ¨ohler, Benjamin Lewandowski, Tim Wengefeld, and Horst-Michael Gross. Efficient rgb-d seman- tic segmentation for indoor scene analysis. In 2021 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 13525–13531. IEEE, 2021. 2

    2021

  15. [23]

    Opti- mizing ensemble weights and hyperparameters of machine learning models for regression problems

    Mohsen Shahhosseini, Guiping Hu, and Hieu Pham. Opti- mizing ensemble weights and hyperparameters of machine learning models for regression problems. Machine Learning with Applications, 7:100251, 2022. 4

    2022

  16. [24]

    Multimodal sparse transformer network for audio-visual speech recognition

    Qiya Song, Bin Sun, and Shutao Li. Multimodal sparse transformer network for audio-visual speech recognition. IEEE Transactions on Neural Networks and Learning Sys- tems, 2022. 1, 2

    2022

  17. [25]

    Soomro, A

    K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 6

    2012 arXiv

  18. [26]

    Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion

    Peng Sun, Wenhu Zhang, Huanyu Wang, Songyuan Li, and Xi Li. Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1407–1417, 2021. 1, 2

    2021

  19. [27]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European conference on com- puter vision (ECCV), pages 247–263, 2018. 6

    2018

  20. [28]

    What makes train- ing multi-modal classification networks hard? In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695–12705, 2020

    Weiyao Wang, Du Tran, and Matt Feiszli. What makes train- ing multi-modal classification networks hard? In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695–12705, 2020. 1, 2, 3, 6

    2020

  21. [29]

    Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning

    Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20039–20049, 2023. 3

    2023

  22. [30]

    Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning

    Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20039–20049, 2023. 2

    2023

  23. [31]

    Privi- leged modality learning via multimodal hallucination

    Shicai Wei, Chunbo Luo, Yang Luo, and Jialang Xu. Privi- leged modality learning via multimodal hallucination. IEEE Transactions on Multimedia, 2023. 1

    2023

  24. [32]

    Msh- net: Modality-shared hallucination with joint adaptation dis- tillation for remote sensing image classification using miss- ing modalities

    Shicai Wei, Yang Luo, Xiaoguang Ma, and Ren Peng. Msh- net: Modality-shared hallucination with joint adaptation dis- tillation for remote sensing image classification using miss- ing modalities. IEEE Transactions on Geoscience and Re- mote Sensing, 61:1–15, 2023. 2

    2023

  25. [33]

    Ro- bust multimodal learning via representation decoupling

    Shicai Wei, Yang Luo, Yuji Wang, and Chunbo Luo. Ro- bust multimodal learning via representation decoupling. In European Conference on Computer Vision , pages 38–54. Springer, 2025. 2

    2025

  26. [34]

    Mmpareto: Boosting multimodal learning with innocent unimodal assistance

    Yake Wei and Di Hu. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. arXiv preprint arXiv:2405.17730, 2024. 1, 3, 6

    2024 arXiv

  27. [35]

    Enhanc- ing multi-modal cooperation via fine-grained modality valu- ation

    Yake Wei, Ruoxuan Feng, Zihe Wang, and Di Hu. Enhanc- ing multi-modal cooperation via fine-grained modality valu- ation. arXiv preprint arXiv:2309.06255, 2023. 1, 3

    2023 arXiv

  28. [36]

    Diagnos- ing and re-learning for balanced multimodal learning

    Yake Wei, Siwei Li, Ruoxuan Feng, and Di Hu. Diagnos- ing and re-learning for balanced multimodal learning. In European Conference on Computer Vision , pages 71–86. Springer, 2025. 1, 3, 6

    2025

  29. [37]

    Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning

    Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, and Di Hu. Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 3

    2023

  30. [38]

    Multi-attention recurrent network for human communication comprehen- sion

    Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehen- sion. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 6

    2018

  31. [39]

    Multimodal fusion on low-quality data: A comprehen- sive survey

    Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, et al. Multimodal fusion on low-quality data: A comprehen- sive survey. arXiv preprint arXiv:2404.18947, 2024. 5

    2024 arXiv

  32. [40]

    Multimodal representation learning by alternating uni- modal adaptation

    Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating uni- modal adaptation. arXiv preprint arXiv:2311.10707, 2023. 1, 3, 6

    2023 arXiv

  33. [41]

    mmformer: Multimodal medical transformer for in- complete multimodal learning of brain tumor segmentation

    Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, and Yefeng Zheng. mmformer: Multimodal medical transformer for in- complete multimodal learning of brain tumor segmentation. arXiv preprint arXiv:2206.02425, 2022. 7

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.