Balance-aware Sequence Sampling Makes Multi-modal Learning Better

Zhi-Hao Guan

REVIEW 4 major objections 4 minor 38 references

Balance-aware Sequence Sampling Makes Multi-modal Learning Better

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that ordering multimodal training samples from balanced to imbalanced improves multi-modal learning and outperforms existing rebalancing baselines.

desk verdict The core claim about balance-aware ordering is not yet supported because Algorithm 1 ranks samples before training from an unspecified initialization, but the paper has enough merit to warrant a serious referee. read the letter →

arxiv 2501.01470 v1 pith:MXR65Z45 submitted 2025-01-01 cs.LG cs.AI

Zhi-Hao Guan This is my paper · ORCID

classification cs.LGcs.AI

keywords multi-modallearningmodalityimbalancecurriculumsamplesequencebalancescoretrainingschedulerclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the order in which multimodal training samples are presented matters as much as the optimization objective: training a model on well-balanced samples first and progressively adding imbalanced ones reduces the gap between strong and weak modalities. It defines a balance score for each sample from two criteria — how similar the two modalities' predictions are, and how well the sample is learned according to training loss — and feeds this score into either a fixed curriculum or a dynamically updated probabilistic sampler. Across six audio-video, text-image, and three-modality datasets, the approach outperforms existing rebalancing methods without changing the network architecture. The practical payoff is a plug-in training strategy for any multimodal classifier.

What carries the argument

The load-bearing object is the balance score $s(x_i)$ of Equation 6, defined as the normalized prediction similarity between modalities minus the normalized total training loss. Prediction similarity (Equation 4) tells how much the two unimodal classifiers agree; the loss term (Equation 5) tells whether the sample is actually learned, guarding against the case where both modalities confidently agree on a wrong answer. This score ranks every training sample once before training (heuristic scheduler) or is updated every $E$ epochs and converted into a softmax sampling probability (learning-based scheduler). The root pacing function of Equation 7 decides what fraction of the most balanced samples is exposed at each epoch, implementing the curriculum from balanced to imbalanced.

What would settle it

Train the same model twice with the same scheduler and random seed but with the ranked order reversed (or with rankings produced from two different random initializations); if performance does not drop, or if the two rankings give different gains, the balance-aware order itself is not what drives the improvement.

Watch

Extended reading notes

Core claim

The central claim is that multi-modal learning can be improved by replacing random sample shuffling with a balanced-to-imbalanced sequence, and that this alone can set new state-of-the-art accuracy on CREMA-D, Kinetics-Sounds, VGGSound, Twitter2015, Sarcasm, and NVGesture. The paper's toy experiment shows curriculum ordering (balanced first) outperforms vanilla training while anti-curriculum ordering suppresses it, and the full method—ranking samples by the balance score of Equation 6, then scheduling them with a root pacing function or a learned sampling probability—consistently beats eleven multi-modal rebalance baselines. The authors further show the gain is not an artifact of a particular fusion architecture: BSS works with late fusion, on three modalities, and on top of CLIP-pretrained encoders.

Load-bearing premise

The balance ranking is computed once from a randomly initialized model, and the method depends on those initial predictions and losses being informative enough to sort samples from balanced to imbalanced.

Editorial extensions

If this is right

If the sequence claim holds, any existing multimodal training pipeline can be upgraded by pre-ranking samples with Equation 6; no architecture change or auxiliary module is required.
The learning-based scheduler, which refreshes balance scores every $E$ epochs, yields larger gains than the fixed heuristic scheduler on every dataset reported, so dynamic re-ranking is the stronger form of the method.
The method transfers to three-modality inputs (RGB, optical flow, depth on NVGesture), where several gradient-modulation baselines do not apply.
Using CLIP-pretrained encoders, sequence sampling still improves over the CLIP baseline and over MLA, indicating the benefit is complementary to strong pretrained features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural test the paper does not run: compare BSS against the same sampling probabilities applied in reverse (imbalanced-to-balanced) under identical compute; the toy experiment suggests the ordering itself, not the non-uniform sampling, is the active ingredient.
Because the balance score depends on initial random parameters, the ranking may be partly seed-dependent; if gains vanish under different initializations, the reported margin would shrink — an easily checkable robustness probe.
The same balanced-to-imbalanced principle could be applied to other training signals, such as per-modality gradient norms or confidence, and could be combined with gradient-modulation methods rather than only replacing them.
The heuristic scheduler's pacing hyperparameters ($\lambda_0$, $T_{grow}$) are the least principled part; a testable extension is an adaptive pace that responds to the current modality gap instead of a fixed root schedule.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper is a mixed bag: the idea is clean, the experiments are broad, and the writing is honest about what it does, but the central mechanism is not actually established as stated. The balance score in Equation 6 combines prediction similarity and training loss, and the learning-based scheduler that updates the scores every E epochs is a reasonable twist on curriculum learning. That part is new. The paper also tests on six datasets, including a tri-modal one, and the reported gains over strong baselines like MLA are substantial. Credit where due: the authors connect modality imbalance to sample ordering in a way I had not seen explicitly, and the toy experiment in Figure 1 at least tries to show that CL helps while anti-CL hurts.

Now the soft spots. Algorithm 1 computes the balance scores before any training, with only "INIT initialize parameters θ0." The manuscript never says whether θ0 is random or pretrained. If the encoders are pretrained but the uni-modal classifiers are randomly initialized, the softmax outputs are near-uniform, so prediction similarity is roughly 1 for every sample and the loss is roughly log(c). That makes the initial ranking essentially noise. The learning-based scheduler updates scores every 5 epochs, so it becomes informative later, but the initial sampling probabilities and all of BSS-H rely on that first untrusted ranking. This is a real hole, not manufactured. The paper needs to state what θ0 is, add a warm-up, and ablate with an intentionally random first ranking.

The second soft spot is a confound: Equation 5 adds the uni-modal cross-entropy losses to the objective. Baselines like OGM and MLA do not use that loss. So the reported gains could come from the auxiliary loss rather than the sequence. The ablation in Table 3 compares with/without PreSim and Loss, but never with random sampling under the same loss. That one missing cell would settle it.

Minor: no error bars, the supplement is absent, no code release. The single author and early version suggest this is a work in progress. Still, I would send it to peer review. The question it asks is worth asking, and the method is simple enough that a careful reviewer can quickly test whether the effect is real. Desk rejecting it would be a waste. My own verdict would be skeptical acceptance at best, but it deserves a serious referee.

Referee Report

4 major / 4 minor

Summary. This paper proposes Balance-aware Sequence Sampling (BSS) for multimodal learning, defined by a balance score that combines uni-modal prediction similarity and training loss (Equation 6). The method orders training samples from balanced to imbalanced and supplies them through either a heuristic curriculum scheduler (BSS-H) or a learning-based probabilistic scheduler (BSS-L) that updates scores every E epochs. Experiments on CREMA-D, Kinetics-Sounds, VGGSound, Twitter2015, Sarcasm, and NVGesture report accuracy/F1/MAP improvements over vanilla fusion and state-of-the-art multimodal rebalancing methods.

Significance. If the central claim holds, the paper identifies a previously underexploited dimension in multimodal imbalance research: the order in which balanced vs imbalanced samples are presented during training. The method is simple, model-agnostic, and tested across audio-video, text-image, and tri-modal datasets, which is a genuine strength. However, the manuscript provides no code, no supplement (despite referencing one), and no error bars, and a key technical detail about model initialization is missing. The contribution is potentially useful, but the current evidence does not yet establish the proposed causal mechanism.

major comments (4)

[Algorithm 1, Section 3.3] The manuscript never states whether the initial parameters θ0 used to compute the Equation 6 balance scores are randomly initialized, pretrained, or warmed up. Since Equation 6 depends on uni-modal softmax predictions and cross-entropy losses, a randomly initialized classifier will produce near-uniform predictions and near-constant losses across samples, making the one-time ranking X_rank essentially arbitrary. This is load-bearing because BSS-H relies entirely on this pre-training ranking, and BSS-L’s initial sampling probabilities also depend on it. Please specify θ0 precisely and add ablations that vary initialization (random, pretrained, and pretrained with a warmed-up classifier) to demonstrate that the reported gains are attributable to the balance score rather than to initialization noise.
[Section 3.2, Equation 6] The normalization operation Norm(·) in Equation 6 is never defined in the main text, and the sentence “More definitions of the criteria are presented in the supplementary materials” cannot be checked because no supplement is included with the submission. The balance score is computed as a difference between a similarity term bounded in [0,1] and a cross-entropy loss that is unbounded; without an explicit normalization, the relative ranking of samples is not well-defined. Please define Norm(·) in the main text and provide the supplementary material in the revision.
[Tables 1 and 2, Section 4.2] All experimental results are reported from single runs with no error bars or significance tests, and several margins over the strongest baseline are very small (e.g., Sarcasm ACC 84.41 for BSS-H vs 84.37 for ReconBoost in Table 1). Given the number of hyperparameters in BSS (α, β, λ0, T_grow, E), the claim of state-of-the-art performance needs multiple seeds with means and standard deviations to rule out run-to-run noise.
[Section 1, Figure 1(c)] The motivating experiment comparing curriculum learning with anti-curriculum learning is the only direct evidence in the paper that the sample ordering itself, rather than the pacing schedule, is responsible for the gain. The caption does not say how “learning difficulty” is measured in that toy experiment or whether it corresponds to Equation 6. Please add a control experiment in the main framework that keeps the same balanced-to-imbalanced pacing but randomizes the internal order of the ranked samples, and also tests the reverse order, so that the specific balance-aware ordering mechanism can be separated from a generic curriculum effect.

minor comments (4)

[Section 3.3 (before Algorithm 1)] There are several typos in this section: “Disscssion” should be “Discussion”, “liner function” should be “linear function”, and “pugin” should be “plugin”.
[Section 3.3, Equation 10] The softmax in Equation 10 has no temperature parameter; after arbitrary normalization of scores, the resulting sampling probabilities may be either too flat or too peaked. Please state how the normalization affects the effective curriculum and whether sampling in Equation 11 is done with or without replacement.
[Section 4.3, Table 3] The ablation study is reported on only one dataset (Kinetics-Sounds). To support the general claim that both prediction similarity and training loss contribute to the balance score, similar ablations on at least one text-image dataset and one tri-modal dataset would be helpful.
[Section 4.4] Hyperparameter sensitivity is shown only for α and β on the CREMA-D dataset; the curriculum parameters λ0, T_grow, and E, which directly control the BSS-H and BSS-L schedulers, are not analyzed. Adding a sensitivity plot for these parameters would strengthen the robustness claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BSS's balance score is an operational training-ordering heuristic, not a fitted parameter that predicts the evaluation metric, and the only self-citation is an introductory example.

full rationale

The paper's central claim is that ordering multimodal training samples from high to low balance score (Eq. 6) improves downstream accuracy. The balance score is defined as a normalized combination of uni-modal prediction similarity (Eq. 4) and the total classification loss (Eq. 5). Although these quantities come from the model's own predictions and losses, they are used as a curriculum-ordering signal, not as a fitted parameter that is later reported as a prediction of held-out accuracy. There is no equation that reduces the final evaluation metric to the balance score; the method is tested against external benchmarks (CREMA-D, Kinetics-Sounds, VGGSound, Twitter2015, Sarcasm, NVGesture) and compared with SOTA baselines. The only self-citation ([Wan et al., 2024], which includes the author) appears in the introduction as an example of multimodal information-retrieval applications and carries no load-bearing argument. No uniqueness theorem or prior-work-by-authors constraint is invoked to force the method's choices. A real concern, noted in Algorithm 1, is that the heuristic scheduler ranks samples once before training after only 'INIT initialize parameters θ0'; if θ0 is random, Eq. 4's similarities and Eq. 5's losses would be near-constant across samples, making the ranking dominated by noise. That is an assumption about the informativeness of the initial model, and potentially a reproducibility risk, but it is not a definitional reduction of the paper's result to its inputs. Consequently, no circular step is established under the required standard.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method rests on several hand-set hyperparameters (alpha, beta, lambda_0, T_grow, E) and three unproven assumptions about the validity of the balance score, the meaningfulness of scores at initialization, and the general benefit of balanced-to-imbalanced curricula. No new physical or algorithmic entities are introduced.

free parameters (5)

alpha (Equation 5) = 0.2
Weight mixing multi-modal and uni-modal losses; tuned in the sensitivity analysis and fixed across datasets.
beta (Equation 9) = 0.6
Update weight for the exponential moving average of balance scores; chosen by hand.
lambda_0 (Equation 7) = 0.1
Initial proportion of training samples in the heuristic scheduler.
T_grow (Equation 7) = 40
Epoch at which the pacing function first reaches 1; a curriculum hyperparameter.
E (epoch interval) = 5
Interval between balance-score updates in the learning-based scheduler.

assumptions (3)

domain assumption Randomly initialized model produces balance scores that reflect true modality balance.
Algorithm 1 evaluates Equation 6 before any training; no evidence is given that initial prediction similarity and loss are meaningful.
domain assumption Prediction similarity and training loss are sufficient and valid criteria for modality balance.
Equation 6 combines them without external validation or theoretical motivation.
domain assumption Learning from balanced to imbalanced samples improves multi-modal learning.
Supported only by a toy experiment on Twitter2015 (Figure 1), not by a general proof; this is the main premise of the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Balance-aware Sequence Sampling Makes Multi-modal Learning Better." pith.science (2026). https://pith.science/paper/MXR65Z45

@misc{pith2026250101470,
  author       = {Pith},
  title        = {Pith review of: Balance-aware Sequence Sampling Makes Multi-modal Learning Better},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXR65Z45}},
  note         = {Machine review of arXiv:2501.01470}
}

read the original abstract

To address the modality imbalance caused by data heterogeneity, existing multi-modal learning (MML) approaches primarily focus on balancing this difference from the perspective of optimization objectives. However, almost all existing methods ignore the impact of sample sequences, i.e., an inappropriate training order tends to trigger learning bias in the model, further exacerbating modality imbalance. In this paper, we propose Balance-aware Sequence Sampling (BSS) to enhance the robustness of MML. Specifically, we first define a multi-perspective measurer to evaluate the balance degree of each sample. Via the evaluation, we employ a heuristic scheduler based on curriculum learning (CL) that incrementally provides training subsets, progressing from balanced to imbalanced samples to rebalance MML. Moreover, considering that sample balance may evolve as the model capability increases, we propose a learning-based probabilistic sampling method to dynamically update the training sequences at the epoch level, further improving MML performance. Extensive experiments on widely used datasets demonstrate the superiority of our method compared with state-of-the-art (SOTA) MML approaches.

Figures

Figures reproduced from arXiv: 2501.01470 by the authors.

**Figure 2.** Illustration of BSS method. (a). Multi-modal training framework for learning multi-modal representations. (b1) and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** (a) and (b). Comparison with hyper-parameters [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Qualitative results of sample evaluation. We show [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 34 canonical work pages

[1]

Look, listen and learn

[Arandjelovic and Zisserman, 2017] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. InICCV, pages 609–617,

work page 2017
[5]

Cooper, Michael K

[Cao et al., 2014] Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. CREMA-D: crowd-sourced emotional multimodal actors dataset. TAC, 5(4):377–390,

work page 2014
[7]

Vggsound: A large-scale audio-visual dataset

[Chen et al., 2020] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, pages 721–725,

work page 2020
[9]

Im- proving multi-modal learning with uni-modal teachers

[Du et al., 2021] Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Im- proving multi-modal learning with uni-modal teachers. CoRR, abs/2106.11059,

arXiv 2021
[13]

On the power of curriculum learning in training deep networks

[Hacohen and Weinshall, 2019] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In ICML, volume 97, pages 2535–2544,

work page 2019
[14]

Deep residual learning for image recog- nition

[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770–778,

2016
[16]

Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably)

[Huang et al., 2022] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably). In ICML, volume 162, pages 9226–9259,

work page 2022
[18]

Dolphins: Multimodal language model for driving

[Ma et al., 2024] Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In ECCV, pages 403–420,

work page 2024

Show all 38 references

[19]

Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural net- works

[Molchanov et al., 2016] Pavlo Molchanov, Xiaodong Yang, Shalini Gupta, Kihwan Kim, Stephen Tyree, and Jan Kautz. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural net- works. In CVPR, pages 4207–4215,

2016
[20]

Taylor, and Peter Stone

[Narvekar et al., 2020] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning do- mains: A framework and survey.JMLR, 21:181:1–181:50,

2020
[21]

Balanced multimodal learning via on-the-fly gradient modulation

[Peng et al., 2022] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In CVPR, pages 8228–8237,

2022
[22]

Mitchell

[Platanios et al., 2019] Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnab ´as P ´oczos, and Tom M. Mitchell. Competence-based curriculum learning for neural machine translation. In NAACL-HLT, pages 1162–1172,

2019
[23]

Learning transferable visual models from natural language supervi- sion

[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi-...

2021
[24]

Sentiment and emotion- aware multi-modal complaint identification

[Singh et al., 2022] Apoorva Singh, Soumyodeep Dey, Ana- mitra Singha, and Sriparna Saha. Sentiment and emotion- aware multi-modal complaint identification. In AAAI, pages 12163–12171,

2022
[25]

Curriculum learning: A sur- vey

[Soviany et al., 2022] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A sur- vey. IJCV, 130(6):1526–1565,

2022
[26]

Utilizing greedy nature for multimodal conditional image synthesis in transformers

[Su et al., 2024] Sitong Su, Junchen Zhu, Lianli Gao, and Jingkuan Song. Utilizing greedy nature for multimodal conditional image synthesis in transformers. TMM, 26:2354–2366,

2024
[27]

Semi-supervised action recognition from tem- poral augmentation using curriculum learning

[Tong et al., 2023] Anyang Tong, Chao Tang, and Wenjian Wang. Semi-supervised action recognition from tem- poral augmentation using curriculum learning. TCSVT, 33(3):1305–1319,

2023
[28]

Covlr: Coordinating cross-modal consistency and intra-modal relations for vision-language retrieval

[Wan et al., 2024] Fengqiang Wan, Xiangyu Wu, Zhihao Guan, and Yang Yang. Covlr: Coordinating cross-modal consistency and intra-modal relations for vision-language retrieval. In ICME, pages 1–6,

2024
[29]

What makes training multi-modal classification net- works hard? In CVPR, pages 12692–12702,

[Wang et al., 2020] Weiyao Wang, Du Tran, and Matt Feis- zli. What makes training multi-modal classification net- works hard? In CVPR, pages 12692–12702,

2020
[30]

A survey on curriculum learning

[Wang et al., 2022] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. TPAMI, 44(9):4555–4576,

2022
[31]

Curriculum learning for multimedia in the era of large language models

[Wang et al., 2024] Xin Wang, Yuwei Zhou, Hong Chen, and Wenwu Zhu. Curriculum learning for multimedia in the era of large language models. In ACMMM, pages 11296–11297,

2024
[32]

[Wu et al., 2022] Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J. Geras. Character- izing and overcoming the greedy nature of learning in multi-modal deep neural networks. In ICML, volume 162, pages 24043–24055,

2022
[33]

Au- diovisual slowfast networks for video recognition

[Xiao et al., 2020] Fanyi Xiao, Yong Jae Lee, Kristen Grau- man, Jitendra Malik, and Christoph Feichtenhofer. Au- diovisual slowfast networks for video recognition. CoRR, abs/2001.08740,

2020 arXiv
[34]

[Xu et al., 2023] Peng Xu, Xiatian Zhu, and David A. Clifton. Multimodal learning with transformers: A survey. TPAMI, 45(10):12113–12132,

2023
[35]

Modality-specific learning rates for effective multimodal additive late-fusion

[Yao and Mihalcea, 2022] Yiqun Yao and Rada Mihalcea. Modality-specific learning rates for effective multimodal additive late-fusion. In ACL, pages 1824–1834,

2022
[36]

Adapting BERT for target-oriented multimodal sentiment classifica- tion

[Yu and Jiang, 2019] Jianfei Yu and Jing Jiang. Adapting BERT for target-oriented multimodal sentiment classifica- tion. In IJCAI, pages 5408–5414,

2019
[37]

Multi- modal fusion on low-quality data: A comprehensive sur- vey

[Zhang et al., 2024a] Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, and Changqing Zhang. Multi- modal fusion on low-quality data: A comprehensive sur- vey. CoRR, abs/2404.18947,

arXiv
[38]

Unis-mmc: Multimodal classification via unimodality- supervised multimodal contrastive learning

[Zou et al., 2023] Heqing Zou, Meng Shen, Chen Chen, Yuchen Hu, Deepu Rajan, and Eng Siong Chng. Unis-mmc: Multimodal classification via unimodality- supervised multimodal contrastive learning. InACL, pages 659–672, 2023

2023
[2009]

Multi-modal sarcasm detection in twitter with hierarchical fusion model

[Cai et al., 2019] Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sarcasm detection in twitter with hierarchical fusion model. In ACL, pages 2506–2515,

2019
[2014]

Quo vadis, action recognition? A new model and the kinetics dataset

[Carreira and Zisserman, 2017] Jo˜ao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724–4733,

2017
[2016]

Reconboost: Boost- ing can achieve modality reconcilement

[Hua et al., 2024] Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boost- ing can achieve modality reconcilement. In ICML,

2024
[2017]

Multimodal machine learning: A survey and taxonomy.TPAMI, 41(2):423–443,

[Baltrusaitis et al., 2019] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy.TPAMI, 41(2):423–443,

2019
[2019]

Curriculum learning

[Bengio et al., 2009] Yoshua Bengio, J ´erˆome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, volume 382, pages 41–48,

2009
[2020]

BERT: pre-training of deep bidirectional transformers for language understand- ing

[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171–4186,

2019
[2021]

PMR: prototypical modal rebalance for multimodal learning

[Fan et al., 2023] Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. PMR: prototypical modal rebalance for multimodal learning. In CVPR, pages 20029–20038,

2023
[2022]

Multimodal classification via modal-aware in- teractive enhancement

[Jiang et al., 2024] Qing-Yuan Jiang, Zhouyang Chi, and Yang Yang. Multimodal classification via modal-aware in- teractive enhancement. CoRR, abs/2407.04587,

2024 arXiv
[2023]

Detached and interac- tive multimodal learning

[Fan et al., 2024] Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junhong Liu, and Song Guo. Detached and interac- tive multimodal learning. In ACMMM, pages 5470–5478,

2024
[2024]

Maybank, and Dacheng Tao

[Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. IJCV, 129(6):1789–1819,

2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Look, listen and learn

[Arandjelovic and Zisserman, 2017] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. InICCV, pages 609–617,

work page 2017

[2] [5]

Cooper, Michael K

[Cao et al., 2014] Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. CREMA-D: crowd-sourced emotional multimodal actors dataset. TAC, 5(4):377–390,

work page 2014

[3] [7]

Vggsound: A large-scale audio-visual dataset

[Chen et al., 2020] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, pages 721–725,

work page 2020

[4] [9]

Im- proving multi-modal learning with uni-modal teachers

[Du et al., 2021] Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Im- proving multi-modal learning with uni-modal teachers. CoRR, abs/2106.11059,

arXiv 2021

[5] [13]

On the power of curriculum learning in training deep networks

[Hacohen and Weinshall, 2019] Guy Hacohen and Daphna Weinshall. On the power of curriculum learning in training deep networks. In ICML, volume 97, pages 2535–2544,

work page 2019

[6] [14]

Deep residual learning for image recog- nition

[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770–778,

2016

[7] [16]

Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably)

[Huang et al., 2022] Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably). In ICML, volume 162, pages 9226–9259,

work page 2022

[8] [18]

Dolphins: Multimodal language model for driving

[Ma et al., 2024] Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In ECCV, pages 403–420,

work page 2024