Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Boosting Multimodal Learning via Disentangled Gradient Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that cross-modal fusion shrinks the gradient returned to every modality encoder, and introduces DGL to restore unimodal-strength learning.

desk verdict A useful empirical training trick with solid results, but the paper's proof that fusion suppresses encoder gradients is not mathematically sound. read the letter →

arxiv 2507.10213 v1 pith:6OFOZD7R submitted 2025-07-14 cs.CV

Shicai Wei , Chunbo Luo , Yang Luo This is my paper
classification cs.CV
keywords multimodallearninggradientsuppressionunder-optimizationdisentangledmodalitydropoutcross-modalfusiondominantaudio-visual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal models are supposed to beat their unimodal parts, but in practice each branch often learns worse than the same branch trained alone. This paper claims to explain why: when a shared classifier sits on top of fused features, the cross-entropy terms contributed by the other modality shrink the gradient that flows back to each encoder. The paper proves the shrinkage, $\mathrm{abs}(g^{\mathrm{Uni}}_{\theta_1}) > \mathrm{abs}(g^{\mathrm{Multi}}_{\theta_1})$, and shows that the gap widens as training proceeds, so even the dominant, better-performing modality is under-optimized. It then introduces a gradient surgery, DGL, that truncates the fusion-to-encoder gradient and substitutes a unimodal gradient computed with the other modalities zeroed, so encoder and fusion updates no longer interfere. If the proof holds, this reframes multimodal underperformance as an encoder–fusion conflict rather than merely a weak-modality problem.

What carries the argument

The load-bearing object is the gradient-suppression factor $s_k = e^{(W^{m_2}_k - W^{m_2}_y)\cdot z^{m_2}_i}$ for the non-target class $k \neq y$. Because the paper assumes converged embeddings sit near the classifier's class-center rows $W_y$, each $s_k$ lies in $(0,1)$, which makes the multimodal encoder gradient strictly smaller than the unimodal one. The repair mechanism is DGL's gradient surgery: it uses a detach operation so the multimodal loss updates only the fusion module and classifier, while a modality-dropout unimodal loss (other modality set to 0) updates each encoder, with its gradient removed before it reaches the fusion module.

What would settle it

Measure the actual gradient norm delivered to one encoder in a matched pair of unimodal and multimodal models over the full training run: if at any epoch the multimodal encoder's gradient norm is not consistently below its unimodal counterpart, or if the learned embedding $z^{m_2}$ does not satisfy $(W^{m_2}_k - W^{m_2}_y)\cdot z^{m_2} < 0$ for all $k \neq y$, then the Eq. (8) inequality fails and the DGL rationale collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is an inequality: in a concatenation-fusion multimodal classifier with cross-entropy loss, the gradient received by any modality encoder is strictly smaller in magnitude than the gradient the same encoder would receive in the matching unimodal model, $\mathrm{abs}(g^{\mathrm{Uni}}_{\theta_1}) > \mathrm{abs}(g^{\mathrm{Multi}}_{\theta_1})$. The reason is that the shared softmax contains factors $e^{(W^{m_2}_k - W^{m_2}_y)\cdot z^{m_2}_i}$ from the other modality; once $z^{m_2}$ converges toward its class center $W^{m_2}_y$, every wrong-class factor is below 1, so the back-propagated gradient shrinks. The proposed remedy, disentangled gradient learning (DGL), detaches the fused representation before the multimodal loss is computed, gives each encoder its own unimodal loss by zeroing the other modality's input (modality dropout), and blocks the unimodal gradient from reaching the fusion module. Experiments on audio-visual classification, three-modality sentiment, and RGB-D segmentation show that DGL improves both fused and unimodal branch performance over gradient-modulation and alternating-optimization baselines.

Load-bearing premise

The proof rests on the assumption that the second modality's embedding aligns with its class-center vector so that every wrong-class softmax term for that modality is less than 1; the paper asserts this geometric condition rather than deriving it.

Editorial extensions

If this is right

  • Every modality branch in a fusion-based multimodal model, including the strongest one, learns more slowly than the same branch trained unimodally, and the gap grows with training.
  • Rebalancing methods that only boost weak modalities leave the dominant modality under-optimized; DGL claims to fix both branches at once.
  • DGL should transfer across fusion methods, modality types, and tasks, because it only depends on detaching and rerouting gradients, not on the fusion architecture.
  • With DGL, unimodal branch performance inside a multimodal model can exceed that of the unimodal baseline, a property prior methods do not consistently show.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The class-center assumption suggests the same suppression should appear in any shared-classifier setting where features converge near the classifier rows, including metric-learning and contrastive heads; DGL-style detaching may transfer there.
  • Always detaching the fusion path means the fusion module receives no encoder-learning signal, so the fused representation could drift stale relative to fresh encoder features; a testable extension is to re-attach gradients at a later epoch or with a learned schedule.
  • Because DGL's unimodal loss is computed through the shared fusion module with other modalities zeroed, the unimodal gradient still depends on the fusion parameters even though its update is blocked; an ablation that also stops the unimodal loss from touching fusion-normalization layers could reveal whether that residual coupling matters.
  • The paper's Eq. (8) is derived for concatenation fusion; attention- or tensor-based fusion may induce suppression with a different quantitative form, so the proof's exact inequality is a testable special case, not a universal law.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Disentangled Gradient Learning (DGL), a training strategy for multimodal models in which the gradient of the multimodal loss is detached from the modality encoders, and each encoder is instead optimized by gradients from unimodal losses computed via modality dropout. The gradient of these unimodal losses is prevented from updating the fusion module. The authors motivate this design with a theoretical claim in Section 3.1: that cross-modal fusion suppresses the gradient backpropagated to every modality encoder relative to the unimodal case, formalized as |g_Uni| > |g_Multi| in Eq. (8). Experiments on audio-visual classification (CREMA-D, Kinetics-Sounds, VGGSound), three-modality sentiment analysis (MOSI), and RGB-D semantic segmentation (NYUv2) show consistent improvements over existing gradient-modulation and alternating-optimization baselines, including under dense fusion methods such as MMTM and MLP-Mixer.

Significance. If the theoretical claim were established, the paper would provide a unified explanation for a known but poorly understood phenomenon: the dominant modality in a multimodal model underperforms its unimodal counterpart. The empirical contribution is substantial and carefully executed: results are reported on held-out test sets, multiple fusion architectures are tested, the method extends to three modalities and to dense prediction, and the code is promised. The proposal is structure-agnostic and adds no extra classifiers, which is a practical strength. However, the theoretical derivation that anchors the paper's central claim is currently not mathematically sound; the claimed inequality in Eq. (8) is derived from an incomplete gradient formula. This makes the main explanatory contribution unproven, even though the empirical method may remain effective.

major comments (4)
  1. [Section 3.1, Eq. (4)] Eq. (4) is not the full cross-entropy gradient. With logits f_k = W_k^{m1} z_{m1} + W_k^{m2} z_{m2} + b_k and softmax probabilities p_k, the true gradient of the cross-entropy loss with respect to z_{m1} is sum_{k=1}^K (p_k - 1_{k=y}) W_k^{m1}. Eq. (4) keeps only the k=y term (p_y - 1) W_y^{m1} and drops the sum over k != y of p_k W_k^{m1}. Eq. (7) has the same defect. Consequently, Eq. (8) does not compare the full gradient norms as the notation |g_Uni| > |g_Multi| claims; it compares only one component of each gradient vector, and the dropped terms may dominate the norm.
  2. [Section 3.1, Eq. (5)] The geometric condition e^{(W_k^{m2} - W_y^{m2}) z_{m2}} < 1 for k != y is asserted rather than derived. The text states that the y-th row of W is the class center because of Refs. [34,35], but those references concern center loss, which is not used here; a softmax classifier does not in general make W_y a class center, nor does it ensure z_{m2} converges to W_y while moving away from all other W_k. Even if the condition held, it only bounds the exponential factor in the dropped terms of the true gradient; it does not control the sum_{k != y} p_k W_k^{m1} contribution, so it cannot by itself establish Eq. (8).
  3. [Section 3.1, Eq. (3)] Eq. (3) conflates the gradient with respect to the encoder output with the gradient with respect to the encoder parameters. The chain rule displayed is ∂L/∂f · ∂f/∂z_τ · ∂z_τ/∂z_{m1}, but the quantity g^{Multi}_{θ1} is described as the gradient passed back to the encoder φ1(θ1, ·). The Jacobian ∂z_{m1}/∂θ1 is missing. If the subsequent inequality is meant to compare parameter-gradient norms, the encoder Jacobian can rescale and rotate the components and cannot be ignored; if it is meant to compare output-gradients only, the notation g_{θ1} is misleading.
  4. [Section 3.1, Eq. (8)] Because Eqs. (4) and (7) omit the non-target-class gradient terms, the claimed inequality |g_Uni| > |g_Multi| is unproven even under the geometric assumption in Eq. (5). This inequality is the only formal support for the abstract claim that cross-modal fusion decreases the gradient passed back to each modality encoder. The authors should either provide a corrected full-gradient derivation, state explicitly the additional assumptions under which the componentwise comparison implies a norm comparison, or reframe the claim as an empirical observation rather than a theorem.
minor comments (5)
  1. [Section 3.1, Eqs. (5)-(6)] There are typographical issues in Eqs. (5) and (6): the exponential notation is inconsistent and a closing parenthesis is missing in the sentence following Eq. (6) ("eW m2 k −W m2yi )zm2 i = 1is").
  2. [Section 3.1, paragraph after Eq. (8)] The statement that "W_m2_y z_m2 and W_m1_y z_m1 become increasingly close to 1" is not justified: logits are not probabilities and need not approach 1. The subsequent claim that the absolute value of g^{Multi}_{θ1} becomes increasingly smaller than that of g^{Uni}_{θ1} inherits the problems of Eq. (8).
  3. [Section 3.2, Eq. (15)] The notation LDGL = [Ld, α(Lm1 + Lm2)] is explained as "concatenation instead of summation," but this is unconventional; it would be clearer to state that the two gradient terms are computed and applied via separate backward passes or that one gradient is zeroed before the other update.
  4. [Algorithm 1] The line "Eliminate the gradient passed back from the unimodal loss to the fusion module" is implemented only verbally; the pseudocode would benefit from specifying how this is done (e.g., by zeroing grads with respect to θτ after the unimodal backward pass).
  5. [Tables 1 and 4] The tables report accuracy, but no variance or number of runs is given; given that the gains over the second-best method are sometimes small (1.34% on VGGSound in Table 1), reporting standard deviations or multiple seeds would strengthen the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claim is an attempted (though flawed) mathematical derivation, and DGL is validated on held-out test sets rather than fitted to the target result.

full rationale

The paper's main explanatory claim is that cross-modal fusion suppresses encoder gradients, formalized as abs(g_Uni) > abs(g_Multi) in Eq. (8). This is a derivation claim, not a fitted prediction. The derivation has a genuine correctness problem: Eq. (4) omits the non-target-class gradient terms sum_{k != y} p_k W_k^{m1}, so Eq. (8) does not follow from Eq. (4) even under the geometric condition in Eq. (5). However, that is a mathematical flaw, not circularity: the inequality is not equivalent to any training statistic, hyperparameter, or fitted quantity, and it is not assumed as an input. The DGL method is then evaluated on held-out test sets across CREMA-D, KS, VGGSound, MOSI, and NYUv2, and the training objective (Eq. 15) is not calibrated to those test results. The self-citations for modality dropout ([29,31]) provide a component technique, but the success of DGL is independently demonstrated by external benchmarks and ablations, so those citations are not load-bearing in a circular sense. No step reduces, by construction or by self-citation, to its own inputs. Therefore the appropriate circularity score is 0; the concerns raised about the proof belong under correctness risk, not circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central theoretical claim rests on several unstated geometric and architectural assumptions. The classifier rows being class centers and the embedding approaching its class center are not guaranteed by cross-entropy training. The chain rule comparison also omits the encoder Jacobian. These assumptions are load-bearing for the proof, though the empirical method itself does not depend on them.

free parameters (1)
  • alpha = 4 (best on CREMA-D; values 1-5 tested)
    Hyper-parameter scaling the unimodal losses in Eq. (15). Tuned on CREMA-D (Table 4); value for other datasets is not reported, so it is a free parameter of the method.
assumptions (5)
  • domain assumption Rows of the final linear classifier W serve as class centers of the learned representations.
    Invoked in Section 3.1 before Eq. (5); cross-entropy training does not by itself guarantee this property.
  • domain assumption The embedding z_m2 converges to its class-center row W_y and moves away from other rows, so (W_k - W_y) dot z_m2 < 0 for all k not equal y.
    Used to derive Eq. (5); no proof is given and the condition can fail depending on the geometry of learned features.
  • domain assumption The gradient w.r.t. the encoder output can be treated as the gradient w.r.t. the encoder parameters when comparing unimodal and multimodal models.
    The chain rule in Eq. (3) omits the encoder Jacobian, which may differ between unimodal and multimodal training.
  • domain assumption Fusion module is concatenation for the theoretical analysis.
    Section 3.1 explicitly restricts the proof to concatenation, but the method is claimed to be fusion-agnostic.
  • domain assumption Modality dropout with zero-padding produces outputs comparable to unimodal logits.
    Used in Eq. (12) and (14) to define unimodal losses; relies on the fusion module treating zero input as missing modality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Boosting Multimodal Learning via Disentangled Gradient Learning." pith.science (2026). https://pith.science/paper/6OFOZD7R

@misc{pith2026250710213,
  author       = {Pith},
  title        = {Pith review of: Boosting Multimodal Learning via Disentangled Gradient Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6OFOZD7R}},
  note         = {Machine review of arXiv:2507.10213}
}
read the original abstract

Multimodal learning often encounters the under-optimized problem and may have worse performance than unimodal learning. Existing methods attribute this problem to the imbalanced learning between modalities and rebalance them through gradient modulation. However, they fail to explain why the dominant modality in multimodal models also underperforms that in unimodal learning. In this work, we reveal the optimization conflict between the modality encoder and modality fusion module in multimodal models. Specifically, we prove that the cross-modal fusion in multimodal models decreases the gradient passed back to each modality encoder compared with unimodal models. Consequently, the performance of each modality in the multimodal model is inferior to that in the unimodal model. To this end, we propose a disentangled gradient learning (DGL) framework to decouple the optimization of the modality encoder and modality fusion module in the multimodal model. DGL truncates the gradient back-propagated from the multimodal loss to the modality encoder and replaces it with the gradient from unimodal loss. Besides, DGL removes the gradient back-propagated from the unimodal loss to the modality fusion module. This helps eliminate the gradient interference between the modality encoder and modality fusion module while ensuring their respective optimization processes. Finally, extensive experiments on multiple types of modalities, tasks, and frameworks with dense cross-modal interaction demonstrate the effectiveness and versatility of the proposed DGL. Code is available at \href{https://github.com/shicaiwei123/ICCV2025-GDL}{https://github.com/shicaiwei123/ICCV2025-GDL}

Figures

Figures reproduced from arXiv: 2507.10213 by the authors.

Figure 1
Figure 1. Visualization on the CREMA-D dataset. (a) illustrates the performance of each unimodal branch and their fusion in the multi [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of vanilla multimodal model (a) and multimodal model with DGL (b). Compared with the vanilla model, DGL [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization on the CREMA-D dataset. (a) illustrates [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Balancing Multimodal Learning through Label Space Reshaping

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    BMLR reshapes the cross-modal label space to equalize mapping difficulty and balance optimization across modalities in multimodal learning.

  2. No Data? No Problem: Robust Vision-Tabular Learning with Missing Values

    cs.CV 2025-12 conditional novelty 5.0 of 10

    Missingness-aware contrastive pretraining plus a 'tabular more vs fewer' ranking loss lets a vision-tabular model perform across the full 0%–100% range of tabular attribute availability.

Reference graph

Works this paper leans on

41 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    Deep audio-visual speech recognition

    Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 44(12):8717–8727, 2018. 2

    work page 2018

  2. [2]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pages 609–617, 2017. 6

    2017

  3. [3]

    Crema-d: Crowd-sourced emotional multimodal actors dataset

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377–390, 2014. 6

    2014

  4. [4]

    Shapeconv: Shape- aware convolutional layer for indoor rgb-d semantic segmen- tation

    Jinming Cao, Hanchao Leng, Dani Lischinski, Daniel Cohen-Or, Changhe Tu, and Yangyan Li. Shapeconv: Shape- aware convolutional layer for indoor rgb-d semantic segmen- tation. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 7088–7097, 2021. 1, 2

    work page 2021

  5. [5]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721–725. IEEE, 2020. 6

    work page 2020

  6. [6]

    Mars: Motion-augmented rgb stream for action recognition

    Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari, and Cordelia Schmid. Mars: Motion-augmented rgb stream for action recognition. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 7882–7891, 2019. 2

    work page 2019

  7. [7]

    Rfnet: Region-aware fusion network for incomplete multi-modal brain tumor seg- mentation

    Yuhang Ding, Xin Yu, and Yi Yang. Rfnet: Region-aware fusion network for incomplete multi-modal brain tumor seg- mentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3975–3984, 2021. 1, 2

    work page 2021

  8. [8]

    Improving multi- modal learning with uni-modal teachers

    Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Improving multi- modal learning with uni-modal teachers. arXiv preprint arXiv:2106.11059, 2021. 1, 2

    arXiv 2021

Show all 41 references
  1. [9]

    Pmr: Prototypical modal rebalance for multi- modal learning

    Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multi- modal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20029– 20038, 2023. 1, 2, 5, 6, 8

    2023

  2. [10]

    Modal- ity distillation with multiple stream networks for action recognition

    Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modal- ity distillation with multiple stream networks for action recognition. In Proceedings of the European Conference on Computer Vision, pages 103–118, 2018. 2

    2018

  3. [11]

    Multimodal remote sensing benchmark datasets for land cover classification with a shared and spe- cific feature learning model

    Danfeng Hong, Jingliang Hu, Jing Yao, Jocelyn Chanussot, and Xiao Xiang Zhu. Multimodal remote sensing benchmark datasets for land cover classification with a shared and spe- cific feature learning model. ISPRS Journal of Photogram- metry and Remote Sensing, 178:68–80, 2021. 1

    2021

  4. [12]

    Deep multimodal multilinear fusion with high- order polynomial pooling

    Ming Hou, Jiajia Tang, Jianhai Zhang, Wanzeng Kong, and Qibin Zhao. Deep multimodal multilinear fusion with high- order polynomial pooling. Advances in Neural Information Processing Systems, 32, 2019. 1, 2

    2019

  5. [13]

    Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation

    Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang. Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation. In 2019 IEEE International Conference on Image Processing (ICIP), pages 1440–1444. IEEE, 2019. 1, 2

    2019

  6. [14]

    Reconboost: Boosting can achieve modal- ity reconcilement

    Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modal- ity reconcilement. arXiv preprint arXiv:2405.09321, 2024. 1, 3

    2024 arXiv

  7. [15]

    Cdnet: Complementary depth network for rgb-d salient object detection

    Wen-Da Jin, Jun Xu, Qi Han, Yi Zhang, and Ming-Ming Cheng. Cdnet: Complementary depth network for rgb-d salient object detection. IEEE Transactions on Image Pro- cessing, 30:3376–3390, 2021. 1

    2021

  8. [16]

    Mmtm: Multimodal transfer module for cnn fusion

    Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuz- zolino, and Kazuhito Koishida. Mmtm: Multimodal transfer module for cnn fusion. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 13289–13299, 2020. 7

    2020

  9. [17]

    Boosting multi-modal model performance with adaptive gradient modulation

    Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting multi-modal model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22214–22224, 2023. 1, 2, 3, 6, 8

    2023

  10. [18]

    Multi-modal face anti-spoofing at- tack detection challenge at cvpr2019

    Ajian Liu, Jun Wan, Sergio Escalera, Hugo Jair Escalante, Zichang Tan, Qi Yuan, Kai Wang, Chi Lin, Guodong Guo, Isabelle Guyon, et al. Multi-modal face anti-spoofing at- tack detection challenge at cvpr2019. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2019

  11. [19]

    Recognizing human ac- tions as the evolution of pose estimation maps

    Mengyuan Liu and Junsong Yuan. Recognizing human ac- tions as the evolution of pose estimation maps. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1159–1168, 2018. 2

    2018

  12. [20]

    librosa: Audio and music signal analysis in python

    Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, pages 18–25, 2015. 6

    2015

  13. [21]

    Balanced multimodal learning via on-the-fly gradient modulation

    Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8238– 8247, 2022. 1, 2, 5, 6, 8

    2022

  14. [22]

    End-to- end audiovisual speech recognition

    Stavros Petridis, Themos Stafylakis, Pingehuan Ma, Feipeng Cai, Georgios Tzimiropoulos, and Maja Pantic. End-to- end audiovisual speech recognition. In 2018 IEEE interna- tional conference on acoustics, speech and signal processing (ICASSP), pages 6548–6552. IEEE, 2018. 2

    2018

  15. [23]

    Efficient rgb-d seman- tic segmentation for indoor scene analysis

    Daniel Seichter, Mona K ¨ohler, Benjamin Lewandowski, Tim Wengefeld, and Horst-Michael Gross. Efficient rgb-d seman- tic segmentation for indoor scene analysis. In 2021 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 13525–13531. IEEE, 2021. 1, 2

    2021

  16. [24]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference on computer vision , pages 746–760. Springer, 2012. 6

    2012

  17. [25]

    Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation

    Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, and Lanfen Lin. Cubemlp: An mlp-based model for multimodal sentiment analysis and depression estimation. In Proceed- ings of the 30th ACM international conference on multime- dia, pages 3722–3729, 2022. 7

    2022

  18. [26]

    Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion

    Peng Sun, Wenhu Zhang, Huanyu Wang, Songyuan Li, and Xi Li. Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1407–1417, 2021. 1

    2021

  19. [27]

    Aligning audiovisual features for audiovisual speech recognition

    Fei Tao and Carlos Busso. Aligning audiovisual features for audiovisual speech recognition. In 2018 IEEE Interna- tional Conference on Multimedia and Expo (ICME) , pages 1–6. IEEE, 2018. 2

    2018

  20. [28]

    What makes train- ing multi-modal classification networks hard? In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695–12705, 2020

    Weiyao Wang, Du Tran, and Matt Feiszli. What makes train- ing multi-modal classification networks hard? In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12695–12705, 2020. 2, 5, 6, 7

    2020

  21. [29]

    Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning

    Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin- aware distillation and modality-aware regularization for in- complete multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20039–20049, 2023. 5

    2023

  22. [30]

    Msh- net: Modality-shared hallucination with joint adaptation dis- tillation for remote sensing image classification using miss- ing modalities

    Shicai Wei, Yang Luo, Xiaoguang Ma, and Ren Peng. Msh- net: Modality-shared hallucination with joint adaptation dis- tillation for remote sensing image classification using miss- ing modalities. IEEE Transactions on Geoscience and Re- mote Sensing, 61:1–15, 2023. 1

    2023

  23. [31]

    Ro- bust multimodal learning via representation decoupling

    Shicai Wei, Yang Luo, Yuji Wang, and Chunbo Luo. Ro- bust multimodal learning via representation decoupling. In European Conference on Computer Vision , pages 38–54. Springer, 2025. 5

    2025

  24. [32]

    Mmpareto: Boosting multimodal learning with innocent unimodal assistance

    Yake Wei and Di Hu. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. arXiv preprint arXiv:2405.17730, 2024. 1, 3, 5, 6, 7

    2024 arXiv

  25. [33]

    Diagnos- ing and re-learning for balanced multimodal learning

    Yake Wei, Siwei Li, Ruoxuan Feng, and Di Hu. Diagnos- ing and re-learning for balanced multimodal learning. In European Conference on Computer Vision , pages 71–86. Springer, 2025. 1, 3, 6

    2025

  26. [34]

    A discriminative feature learning approach for deep face recog- nition

    Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recog- nition. In Computer vision–ECCV 2016: 14th European con- ference, amsterdam, the netherlands, October 11–14, 2016, proceedings, part VII 14, pages 499–515. Spri...

    2016

  27. [35]

    A comprehensive study on center loss for deep face recogni- tion

    Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A comprehensive study on center loss for deep face recogni- tion. International Journal of Computer Vision , 127:668– 683, 2019. 3

    2019

  28. [36]

    Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning

    Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, and Di Hu. Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 1, 2

    2023

  29. [37]

    Tensor fusion net- work for multimodal sentiment analysis

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cam- bria, and Louis-Philippe Morency. Tensor fusion net- work for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250, 2017. 1, 2

    2017 arXiv

  30. [38]

    Multi-attention recurrent network for human communication comprehen- sion

    Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehen- sion. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 6

    2018

  31. [39]

    Multimodal representation learning by alternating uni- modal adaptation

    Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating uni- modal adaptation. arXiv preprint arXiv:2311.10707, 2023. 1, 2, 3, 6, 8

    2023 arXiv

  32. [40]

    mmformer: Multimodal medical transformer for in- complete multimodal learning of brain tumor segmentation

    Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, and Yefeng Zheng. mmformer: Multimodal medical transformer for in- complete multimodal learning of brain tumor segmentation. arXiv preprint arXiv:2206.02425, 2022. 1, 2

    2022 arXiv

  33. [41]

    Rgb-d salient object detection: A sur- vey

    Tao Zhou, Deng-Ping Fan, Ming-Ming Cheng, Jianbing Shen, and Ling Shao. Rgb-d salient object detection: A sur- vey. Computational Visual Media, 7(1):37–69, 2021. 1

    2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.