Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Improving Multimodal Learning Balance and Sufficiency through Data Remixing

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Batch-level interference, not just slow modalities, causes weak multimodal learning; Data Remixing trains each batch on one modality and lifts accuracy by up to 6.5 percentage points.

desk verdict A plausible training recipe with consistent empirical gains, but the paper's interference-removal story doesn't hold up under scrutiny. read the letter →

arxiv 2506.11550 v2 pith:YKYRP2DZ submitted 2025-06-13 cs.LG cs.AI

Xiaoyu Ma , Hao Chen , Yongjian Deng This is my paper
classification cs.LGcs.AI
keywords multimodallearningmodalitylazinessclashdataremixingbatch-levelinterferenceunimodalsufficiencyaudiovisualclassificationgradientalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that two failures of multimodal training—modality laziness, where the strong modality dominates and the weak one learns too little, and modality clash, where inconsistent gradient directions from different modalities interfere—can both be traced to how data is grouped into batches, and that fixing the grouping fixes both. The proposed Data Remixing strategy first decouples each training sample so only the weaker modality remains, allocating more and harder samples to the weaker modality, then reassembles samples so that every batch contains only one modality. On the CREMAD and Kinetic-Sounds benchmarks, adding this training strategy to existing fusion methods improves accuracy by roughly 6.50 and 3.41 percentage points, respectively, without expanding the dataset or adding inference-time cost. The paper claims this is the first batch-level analysis and solution for these problems. In the conclusion, the authors note a limitation: when one modality is auxiliary and carries limited information, the unimodal evaluation and allocation strategy may require refinement.

What carries the argument

The machinery is a two-stage data-remixing schedule. Stage one is a sample-level unimodal capability score: for each sample, compute the KL divergence $D_{\mathrm{KL}}(p^k_i \| U)$ between each modality's prediction and the uniform distribution; the modality with the smaller value is judged less separable, i.e. weaker on that sample, and is the only modality kept. Stage two is a batch-composition constraint: after partitioning the data into subsets $\mathcal{D}_{m_k}$ by retained modality, every mini-batch is drawn entirely from one subset, so the batch gradient mixes gradients of only one modality. The paper argues that this algebraically removes the cross-modal interference terms from the batch cross-entropy loss, making each modality's update direction consistent with a unimodal objective.

What would settle it

Measure, on a held-out dataset, the angle between a batch's gradient and the gradient of the same samples with only one modality active; the paper predicts this discrepancy is smaller after reassembling. If mixed-modality batches show no larger discrepancy than remixed batches, or if single-modality batches reduce accuracy on a task where cross-modal complementarity is essential, the batch-level story is incomplete.

Watch

Extended reading notes

Core claim

The paper's central claim is that insufficient and imbalanced multimodal learning has a batch-level cause: even after decoupling multimodal data into unimodal samples, a batch containing both modalities still produces gradient updates whose directions interfere, so each modality deviates from its ideal optimization path. Data Remixing therefore consists of two linked operations: decoupling, in which each sample is assigned to the modality whose unimodal prediction has the smaller KL divergence to the uniform distribution (the weaker modality), with other modalities masked to zero; and reassembling, in which the decoupled subsets are sampled so that each mini-batch $B_i$ satisfies $B_i \subseteq \mathcal{D}_{m_k}$ for one modality $k$. The paper shows that with such batches, the cross-entropy loss no longer mixes $W^{a}z^{a}$ and $W^{v}z^{v}$ terms for different modalities within a batch, which it identifies as the source of modality clash. After a warm-up stage on the full multimodal data, training alternates between modality-specific batches, exposing the weak modality to more samples and aligning each modality's gradient direction, yielding the reported gains.

Load-bearing premise

The central assumption is that interference between modalities arises from putting different modalities in the same training batch, and that removing that mixing by training each batch on a single modality improves multimodal learning without sacrificing the cross-modal features that joint training is meant to provide.

Editorial extensions

If this is right

  • Data Remixing can be layered on top of existing balance methods: combining it with MLA gives 74.19% on CREMAD and with Resample gives 58.40% on Kinetic-Sounds, beating either method alone.
  • The strategy generalizes across fusion choices—concatenation, summation, decision fusion—and across intermediate-fusion architectures such as MMTM and CentralNet, so it is not tied to one model design.
  • Because the training set is not expanded, Data Remixing converges faster in wall-clock time than Resample and MLA under the same hardware and batch size.
  • Reassembling single-modality batches aligns each modality's gradient direction with its ideal direction, which the paper demonstrates by measuring gradient-angle discrepancies during training on CREMAD.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: if batch-level interference is the true bottleneck, the same remixing schedule should transfer to other paired-modality objectives such as audiovisual retrieval or video captioning, where batch composition is equally controllable.
  • Extension: the KL-to-uniform score is a cheap proxy for sample-level weakness; a testable variant would use per-sample holdout validation or true-label confidence to assign samples, and check whether the accuracy gains change.
  • Extension: the mechanism predicts a measurable signature—the cosine angle between a batch update and a pure unimodal update should be consistently smaller for remixed batches; tracking this for both modalities, not just the strong one, would sharpen the evidence.
  • Extension: the paper's stated auxiliary-modality limitation suggests a boundary: when one channel carries almost no task information, over-allocating training to it could waste capacity, and the allocation rule may need a floor on how many samples each modality receives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a training strategy called Data Remixing for multimodal models, targeting the two problems of modality laziness (weak modalities learning too slowly) and modality clash (cross-modal gradient interference). The method first decouples multimodal inputs by evaluating per-sample unimodal separability using KL divergence to a uniform distribution, retaining only the weaker modality for each sample. It then reassembles training batches so that each batch contains samples from only one such modality subset, with the non-selected modalities masked to zero. The authors claim this batch-level control eliminates cross-modal interference, and demonstrate on CREMAD and Kinetic-Sounds consistent accuracy improvements over conventional fusion baselines and several prior imbalance-aware methods, without dataset expansion or inference overhead. The paper includes ablations of the two components, analyses of sample allocation and gradient alignment, and comparisons of unimodal prediction strategies.

Significance. If the empirical results hold, the method is attractive for its simplicity, architecture-agnosticism, and consistent gains across two audio-visual benchmarks and multiple fusion architectures (Concat, Summation, Decision Fusion, FiLM, Bi-Gated, MMTM, CentralNet). The authors provide the source code, and the ablations give some support for both design choices. However, the paper's central mechanistic claim—that mixing modalities within a batch causes interference and that single-modality batches with zeroed other modalities remove that interference—is undermined by an incorrect simplification in the derivation of Eq. (10). The claimed batch-level novelty therefore needs either a corrected mechanism with supporting evidence or a revised, more modest framing. The empirical gains are still potentially valuable, but the current explanation does not reliably attribute them to the stated cause.

major comments (4)
  1. [Section 3.2, Eq. (10)] The simplification of the batch loss to depend only on the retained modality assumes that masking a modality input to zero yields a zero encoder output (z_a = 0). Under the architecture used in Section 4.1 (ResNet-18 encoders), this is false: BatchNorm layers apply an affine transformation to a zero input using running statistics, and residual connections propagate nonzero signals, so a zeroed input still produces a nonzero feature vector. Consequently, the zeroed modality still contributes to f(x_i) in Eq. (5) and still receives nonzero gradients from the batch loss. The claimed elimination of cross-modal interference at the batch level is therefore not actually achieved by the described procedure. Please provide empirical evidence that the contribution of the zeroed branch is negligible for the trained encoders (e.g., gradient norms of the masked branch), or modify the method to explicitly zero the encoder output (or detach the gradient) for the non-selected modality and re-run the experiments. This is load-bearing because the paper's central novelty is the batch-level interference-removal mechanism.
  2. [Section 4.5.1, Table 5] The 'Reassemble' ablation is described as reassembling batches without modality masking ('without modality masking'). However, the formal definition of the modality subset in Eq. (7) requires that non-selected modalities are zeroed: D_mk = {x_i | x_j_i = 0 for all j ≠ k}. Without masking, each sample still contains both modalities, so the batch does not satisfy the condition in Eq. (9) as formalized. The ablation therefore cannot isolate the effect of batch-level reassembling under the proposed mechanism; the improvement could be due to a change in sampling distribution (e.g., over-sampling the weak modality) rather than to the removal of cross-modal interference via zeroed inputs. Please include an ablation that reassembles batches while masking the non-selected modality (the full Eqs. 7-9 setup) and report the individual contributions of masking and batch composition separately.
  3. [Section 4.3, Table 3] The text states 'our approach neither expands the dataset nor reduces the model's training efficiency,' but Table 3 shows that Remix training time to convergence is 2537s on CREMAD and 4946s on Kinetic-Sounds, compared to 1536s and 3849s for the baseline—an increase of about 65% and 29%, respectively. This directly contradicts the stated claim about training efficiency. Please revise the claim to state that the method does not expand the dataset, that it is more efficient than Resample and MLA, and acknowledge a non-negligible increase in training time relative to the plain baseline.
  4. [Abstract and Section 1] The headline numbers 'approximately 6.50% up on CREMAD and 3.41% up on Kinetic-Sounds' do not match any comparison reported in Table 2. The improvement of Remix over the Concatenation baseline is 8.20% and 5.40%; the improvement of MLA+Remix over MLA is 6.18% and 3.09%; Resample+Remix over Resample is 5.64% and 3.23%. Please clarify the reference baseline for the claimed 6.50% and 3.41% figures, or correct the numbers and the associated text.
minor comments (6)
  1. [Section 3.1, Eq. (1)] The weights w_k in the auxiliary losses are not specified; please state how they are chosen in the experiments (e.g., all equal to 1) and whether they are tuned per dataset.
  2. [Section 4.1] The paper reports averages over three random seeds but no standard deviations or confidence intervals; please include variance information in the tables or at least in an appendix.
  3. [Section 4.2] The 'Decision fusion' method (Gunes & Piccardi, 2005) is not defined; please specify the exact aggregation rule (e.g., averaging softmax probabilities) to ensure reproducibility.
  4. [Section 4.5.3] The reported unimodal accuracy improvements (audio +1.75%, video +2.96% on CREMAD) are not tied to a clearly defined baseline; please specify whether these are relative to the baseline model's unimodal classification heads and at which training stage they are measured.
  5. [Figure 3(c)] The 'ideal guidance direction' used to compute gradient alignment is not defined; please specify how it is computed and which gradient (e.g., of the audio branch) is being measured.
  6. [Algorithm 1] The warm-up epoch count E_r is a free parameter; the text mentions a 10-epoch warm-up, but please state explicitly whether the same value is used for both datasets and whether it was tuned.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Data Remixing gains are measured against external baselines, and Eq. 10 is a motivational tautology rather than a fitted prediction.

full rationale

The paper's central claim is an empirical accuracy improvement, and every reported result is obtained by training on fixed external benchmarks (CREMA-D, Kinetic-Sounds) and comparing against standard baselines. No parameter is fitted to the target results, and no reported number is derived by plugging the paper's own output back into a formula. The closest thing to a self-referential step is Eq. (10), where masking one modality to zero and restricting each batch to one modality subset removes that modality's term from the loss expression. That simplification is definitional and tautological, but the paper uses it only as motivation for the batch-level reassembling hypothesis; the ablations in Table 5 and the gradient-direction measurements in Fig. 3(c) are independent empirical checks. The authors cite prior work for baselines and standard techniques, but these are not self-citations carrying a load-bearing uniqueness theorem or a forbidden-alternative argument. Concerns about BatchNorm making zeroed inputs produce nonzero features bear on the technical validity of Eq. (10)'s simplification, but that is an empirical correctness issue, not a circular reduction. The derivation chain therefore does not reduce to its own inputs, and the central claim is externally testable.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. Free parameters are limited to the warm-up schedule and unspecified loss weights. The main assumptions are the KL-based capability proxy and the hypothesis that single-modality batches reduce harmful interference; these are domain assumptions that the paper tests empirically.

free parameters (2)
  • Warm-up epochs Er = 10
    The method is applied after a 10-epoch warm-up stage (Section 4.5.2). This is chosen by hand; no sensitivity analysis is reported, and the central method depends on the model having reasonable unimodal predictions before decoupling.
  • Unimodal loss weights w_k in Eq. (1)
    The auxiliary classification heads are weighted by w_k in the combined loss (Eq. 1), but the values are not reported. If these weights are tuned, they could affect results; their omission makes the experimental setup incompletely specified.
assumptions (3)
  • domain assumption KL divergence of unimodal prediction probabilities to a uniform distribution is a valid measure of per-sample modality capability.
    Used in Eq. (2) to decide which modality to mask for each sample. No independent validation is provided that this proxy is optimal or even monotonic with true modality sufficiency.
  • domain assumption Cross-modal gradient interference at the batch level is harmful, and eliminating it by training on single-modality batches improves learning.
    This is the core hypothesis of the paper, introduced in Section 3.2. It is supported only by the empirical results and a single gradient-angle curve, not by a rigorous optimization argument.
  • domain assumption The multimodal model's fused output benefits from separately trained unimodal branches that are later combined during inference.
    The method alternates single-modality batches, which prevents the model from seeing cross-modal interactions during most of training. The assumption that this does not harm the final fusion is implicit and tested only indirectly through the final accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Multimodal Learning Balance and Sufficiency through Data Remixing." pith.science (2026). https://pith.science/paper/YKYRP2DZ

@misc{pith2026250611550,
  author       = {Pith},
  title        = {Pith review of: Improving Multimodal Learning Balance and Sufficiency through Data Remixing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKYRP2DZ}},
  note         = {Machine review of arXiv:2506.11550}
}
abstract

Different modalities hold considerable gaps in optimization trajectories, including speeds and paths, which lead to modality laziness and modality clash when jointly training multimodal models, resulting in insufficient and imbalanced multimodal learning. Existing methods focus on enforcing the weak modality by adding modality-specific optimization objectives, aligning their optimization speeds, or decomposing multimodal learning to enhance unimodal learning. These methods fail to achieve both unimodal sufficiency and multimodal balance. In this paper, we, for the first time, address both concerns by proposing multimodal Data Remixing, including decoupling multimodal data and filtering hard samples for each modality to mitigate modality imbalance; and then batch-level reassembling to align the gradient directions and avoid cross-modal interference, thus enhancing unimodal learning sufficiency. Experimental results demonstrate that our method can be seamlessly integrated with existing approaches, improving accuracy by approximately 6.50%$\uparrow$ on CREMAD and 3.41%$\uparrow$ on Kinetic-Sounds, without training set expansion or additional computational overhead during inference. The source code is available at https://github.com/MatthewMaxy/Remix_ICML2025.

Figures

Figures reproduced from arXiv: 2506.11550 by the authors.

Figure 1
Figure 1. We decouple the multimodal data to assign samples to each modality’s training and then reassemble the inputs to control the consistency of modalities within the batch. By regulating the number of samples, we mitigate modality laziness, and by adjusting the batch composition, we alleviate modality clash. perception and decision-making capabilities of models by integrating data from diverse modalities, including visio… view at source ↗
Figure 2
Figure 2. The pipeline of Data Remixing method. In step (a), the complete dataset is used for training to ensure the model develops the basic representational capability. In step (b), the multimodal data is decoupled based on unimodal separability (calculated using KL-divergence), and the original dataset is reassembled into non-overlapping subsets. In step (c), the subsets obtained in step (b) are used to train on the specif… view at source ↗
Figure 3
Figure 3. Proof of effectiveness of Decouple and Reassemble Methods. The results are obtained on CREMAD. (a) Statistics of the number of samples used for training specific modalities. (b) The change of the imbalance ratio ρ. (c) Comparison of gradient direction discrepancies [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 21 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Zisserman, A

    Arandjelovic, R. and Zisserman, A. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp.\ 609--617, 2017

    2017

  3. [3]

    G., Keutmann, M

    Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5 0 (4): 0 377--390, 2014

    2014

  4. [4]

    Improving multi-modal learning with uni-modal teachers

    Du, C., Li, T., Liu, Y., Wen, Z., Hua, T., Wang, Y., and Zhao, H. Improving multi-modal learning with uni-modal teachers. arXiv preprint arXiv:2106.11059, 2021

    arXiv 2021

  5. [5]

    On uni-modal feature learning in supervised multi-modal learning

    Du, C., Teng, J., Li, T., Liu, Y., Yuan, T., Wang, Y., Yuan, Y., and Zhao, H. On uni-modal feature learning in supervised multi-modal learning. In International Conference on Machine Learning, pp.\ 8632--8656. PMLR, 2023

    work page 2023

  6. [6]

    Pmr: Prototypical modal rebalance for multimodal learning

    Fan, Y., Xu, W., Wang, H., Wang, J., and Guo, S. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 20029--20038, 2023

    2023

  7. [7]

    Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing

    Fu, J., Gao, J., Bao, B.-K., and Xu, C. Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing. IEEE Transactions on Circuits and Systems for Video Technology, 2023

    work page 2023

  8. [8]

    and Zou, J

    Ghorbani, A. and Zou, J. Y. Neuron shapley: Discovering the responsible neurons. Advances in neural information processing systems, 33: 0 5922--5932, 2020

    work page 2020

Show all 40 references
  1. [9]

    and Piccardi, M

    Gunes, H. and Piccardi, M. Affect recognition from face and body: early fusion vs. late fusion. In 2005 IEEE international conference on systems, man and cybernetics, volume 4, pp.\ 3437--3443. IEEE, 2005

    2005

  2. [10]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  3. [11]

    Multimodal temporal attention in sentiment analysis

    He, Y., Sun, L., Lian, Z., Liu, B., Tao, J., Wang, M., and Cheng, Y. Multimodal temporal attention in sentiment analysis. In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pp.\ 61--66, 2022

    2022

  4. [12]

    Improving neural networks by preventing co-adaptation of feature detectors

    Hinton, G. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012

    2012 arXiv

  5. [13]

    Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably)

    Huang, Y., Lin, J., Zhou, C., Yang, H., and Huang, L. Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably). In International conference on machine learning, pp.\ 9226--9259. PMLR, 2022

    2022

  6. [14]

    Joze, H. R. V., Shaban, A., Iuzzolino, M. L., and Koishida, K. Mmtm: Multimodal transfer module for cnn fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 13289--13299, 2020

    2020

  7. [15]

    The kinetics human action video dataset

    Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  8. [16]

    Efficient large-scale multi-modal classification

    Kiela, D., Grave, E., Joulin, A., and Mikolov, T. Efficient large-scale multi-modal classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  9. [17]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  10. [18]

    Improving multimodal learning with multi-loss gradient modulation

    Kontras, K., Chatzichristos, C., Blaschko, M., and De Vos, M. Improving multimodal learning with multi-loss gradient modulation. arXiv preprint arXiv:2405.07930, 2024

    2024 arXiv

  11. [19]

    Boosting multi-modal model performance with adaptive gradient modulation

    Li, H., Li, X., Hu, P., Lei, Y., Li, C., and Zhou, Y. Boosting multi-modal model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 22214--22224, 2023

    2023

  12. [20]

    Variational probabilistic fusion network for rgb-t semantic segmentation

    Lin, B., Lin, Z., Guo, Y., Zhang, Y., Zou, J., and Fan, S. Variational probabilistic fusion network for rgb-t semantic segmentation. arXiv preprint arXiv:2307.08536, 2023

    2023 arXiv

  13. [21]

    Umap: Uniform manifold approximation and projection for dimension reduction

    McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  14. [22]

    Y., et al

    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A. Y., et al. Multimodal deep learning. In ICML, volume 11, pp.\ 689--696, 2011

    2011

  15. [23]

    and Efros, A

    Owens, A. and Efros, A. A. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European conference on computer vision (ECCV), pp.\ 631--648, 2018

    2018

  16. [24]

    Balanced multimodal learning via on-the-fly gradient modulation

    Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8238--8247, 2022

    2022

  17. [25]

    Film: Visual reasoning with a general conditioning layer

    Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  18. [26]

    Graph interactive network with adaptive gradient for multi-modal rumor detection

    Sun, T., Qian, Z., Li, P., and Zhu, Q. Graph interactive network with adaptive gradient for multi-modal rumor detection. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, pp.\ 316--324, 2023

    2023

  19. [27]

    Learning to balance the learning rates between various modalities via adaptive tracking factor

    Sun, Y., Mai, S., and Hu, H. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 28: 0 1650--1654, 2021

    2021

  20. [28]

    and Hinton, G

    Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9 0 (11), 2008

    2008

  21. [29]

    Centralnet: a multilayer approach for multimodal fusion

    Vielzeuf, V., Lechervy, A., Pateux, S., and Jurie, F. Centralnet: a multilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp.\ 0--0, 2018

    2018

  22. [30]

    What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12695--12705, 2020

    Wang, W., Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12695--12705, 2020

    2020

  23. [31]

    and Hu, D

    Wei, Y. and Hu, D. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. In International Conference on Machine Learning, pp.\ 52559--52572. PMLR, 2024

    2024

  24. [32]

    Enhancing multimodal cooperation via sample-level modality valuation

    Wei, Y., Feng, R., Wang, Z., and Hu, D. Enhancing multimodal cooperation via sample-level modality valuation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 27338--27347, 2024

    2024

  25. [33]

    Diagnosing and re-learning for balanced multimodal learning

    Wei, Y., Li, S., Feng, R., and Hu, D. Diagnosing and re-learning for balanced multimodal learning. In European Conference on Computer Vision, pp.\ 71--86. Springer, 2025

    2025

  26. [34]

    Wu, N., Jastrzebski, S., Cho, K., and Geras, K. J. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning, pp.\ 24043--24055. PMLR, 2022

    2022

  27. [35]

    Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning

    Xu, R., Feng, R., Zhang, S.-X., and Hu, D. Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  28. [36]

    Mcl: A contrastive learning method for multimodal data fusion in violence detection

    Yang, L., Wu, Z., Hong, J., and Long, J. Mcl: A contrastive learning method for multimodal data fusion in violence detection. IEEE Signal Processing Letters, 30: 0 408--412, 2022

    2022

  29. [37]

    Multimodal fusion on low-quality data: A comprehensive survey

    Zhang, Q., Wei, Y., Han, Z., Fu, H., Peng, X., Deng, C., Hu, Q., Xu, C., Wen, J., Hu, D., et al. Multimodal fusion on low-quality data: A comprehensive survey. arXiv preprint arXiv:2404.18947, 2024 a

    2024 arXiv

  30. [38]

    Multimodal representation learning by alternating unimodal adaptation

    Zhang, X., Yoon, J., Bansal, M., and Yao, H. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 27456--27466, 2024 b

    2024

  31. [39]

    Adaptive mask co-optimization for modal dependence in multimodal learning

    Zhou, Y., Liang, X., Zheng, S., Xuan, H., and Kumada, T. Adaptive mask co-optimization for modal dependence in multimodal learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  32. [40]

    Vision+ x: A survey on multimodal learning in the light of data

    Zhu, Y., Wu, Y., Sebe, N., and Yan, Y. Vision+ x: A survey on multimodal learning in the light of data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.