Pith. sign in

REVIEW 3 major objections 5 minor 61 references

RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that dynamic attention fusion in multimodal Transformers collapses during training through a self-reinforcing bias loop, and that rotating the class-token query toward a rebalanced anchor restores it.

desk verdict RollingQ is a solid empirical paper with a cheap fix for a real attention-bias problem, but its theoretical story conflates pre-softmax logits with attention mass, so treat the diagnosis as suggestive. read the letter →

arxiv 2506.11465 v1 pith:D4XXRMWY submitted 2025-06-13 cs.LG cs.AIcs.CV

Haotian Ni , Yake Wei , Hang Liu , Gong Chen , Chong Peng , Hao Lin , Di Hu This is my paper
classification cs.LGcs.AIcs.CV
keywords multimodallearningattentionmechanismdynamicfusionmodalitybiasself-reinforcingcyclequeryrotationaudio-visualimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal Transformers are supposed to fuse audio, visual, and text information by paying more attention to whichever modality is most informative for a given sample. The paper reports that this dynamic adaptability dies during training: the model locks onto one modality and keeps favoring it even when that modality is replaced by pure noise. The cause, it argues, is a self-reinforcing cycle in which the favored modality receives higher attention, its encoder receives more gradient, and the gap between modalities' attention-key distributions widens. The proposed fix, Rolling Query, rotates the class-token query toward a rebalancing anchor computed from the two modalities' average keys, interrupting the cycle. In experiments on audio-visual and vision-text benchmarks, this rotation restores attention's sensitivity to input quality and improves accuracy at negligible extra cost.

What carries the argument

The load-bearing object is the query rotation used by RollingQ. After each training epoch the method freezes the model, computes each modality's average key $\hat{k}_m$, the Attention Imbalance Rate $\mathrm{AIR} = \mathbb{E}[\cos\theta_a - \cos\theta_v]$, and a balanced anchor $q_b = (\alpha \mathbb{E}[\hat{K}_a]/\lVert\mathbb{E}[\hat{K}_a]\rVert_2 + (1-\alpha)\mathbb{E}[\hat{K}_v]/\lVert\mathbb{E}[\hat{K}_v]\rVert_2)\,\lVert\mathbb{E}[Q]\rVert_2$ with $\alpha = \tfrac{1}{2}[1+\mathrm{Tanh}(-\rho\,\mathrm{AIR})]$. It then forms the rotation matrix $R_b = \mathrm{SVD}([\mathbb{E}[Q], q_b])$ satisfying $q_b = \mathbb{E}[Q]R_b$ and replaces each query by $q_r = qR_b$ (Equation 13). The rotation preserves the query's norm, so it changes only which keys the query points toward; this redirects optimization momentum to the underused modality without adding a trainable module.

What would settle it

Take a trained multimodal Transformer that exhibits the reported modality bias, compute the true softmax attention score for each modality, and compare it with the surrogate $L_m \|q\| \|\hat{k}_m\| \cos\theta_m$ that RollingQ's diagnosis uses. If key norms or sequence lengths differ substantially between modalities, or if the surrogate ranks the modalities differently from the true attention on even a small fraction of held-out samples, the anchor and rotation target in Equations (9)-(13) are miscalibrated and RollingQ should be expected to fail.

Watch

Extended reading notes

Core claim

The paper's central claim is that the loss of dynamic fusion in multimodal Transformers is a train-time pathology, not an architectural limitation. At initialization both modalities receive similar attention, but once one modality's features become slightly better, a feedback loop amplifies the advantage: feed-forward attention favors the biased modality, and backpropagation gives that modality's encoder more gradient, so its features improve further. The measurable signature of this loop is the Attention Imbalance Rate (AIR), the expected difference in cosine similarity between the query and the two modalities' average keys. RollingQ detects when the absolute AIR passes a threshold, builds a balanced anchor from the normalized average keys weighted by a Tanh function of AIR, and rotates the query by the singular-value-decomposition rotation matrix that maps the expected query onto the anchor. The rotated query then learns in a region where the previously neglected modality receives more attention, narrowing the key distribution gap and restoring the model's ability to lower attention when the biased modality is corrupted.

Load-bearing premise

The argument assumes that a modality's attention score is essentially set by how similar the query is to that modality's average key, because the keys' sizes and sequence lengths are roughly equal across the two modalities.

Editorial extensions

If this is right

  • On the CREMA-D, Kinetic-Sound, and CMU-MOSEI benchmarks, RollingQ improves accuracy by 0.1 to 3.1 points over the corresponding vanilla or specialized multimodal Transformer baselines (Table 1).
  • When the biased modality is replaced with Gaussian noise, RollingQ sharply raises the correlation between attention scores and whether the input is noise (from about 0.44-0.52 to 0.76-0.78), showing that the attention mechanism has become sensitive to data quality again.
  • At every tested noise level, RollingQ suffers a smaller accuracy drop than the vanilla model, indicating better robustness to unreliable modalities.
  • The method adds roughly 1% parameters and 0.1% GFLOPs and improves performance across ViT depths, batch sizes, and a ResNet18 backbone, so the fix is cheap and portable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the AIR and anchor construction only use average keys and query expectations, the same rotation recipe should transfer to any attention-based fusion of two or more token sequences, such as vision-language or text-audio models, whenever key norms and sequence lengths are comparable.
  • Tracking AIR during training could serve as a standalone diagnostic for emerging modality collapse, since the paper's analysis links a widening gap to degraded adaptation even before accuracy suffers.
  • The rotation could plausibly be applied as a post-hoc correction on a frozen pretrained model, although the paper only reports train-time application; testing that variant would separate the training-dynamics effect from the inference-time redistribution.
  • Because RollingQ acts on the query rather than on encoder features, combining it with encoder-level rebalancing methods may yield further gains; the paper itself notes it does not directly equalize unimodal feature quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies why the attention mechanism in multimodal Transformers loses its dynamic adaptability, leading the model to favor one modality regardless of input quality. The authors argue that this bias is caused by a self-reinforcing cycle: the favored modality receives more attention, which amplifies its encoder gradients, which in turn further improves that modality's features and widens the gap between the attention-key distributions of different modalities. They propose Rolling Query (RollingQ), which detects imbalance with an Attention Imbalance Rate (AIR) indicator based on cosine similarity between the query and average keys, and then rotates the query toward a rebalance anchor that gives more weight to the under-attended modality. Experiments on CREMA-D, Kinetic-Sound, and CMU-MOSEI, plus noise-perturbation, correlation, and OOD tests, are presented as evidence that RollingQ restores dynamic attention and improves accuracy over vanilla and several specialized baselines.

Significance. If the central claim holds, the paper identifies a plausible and practically important failure mode of attention-based multimodal fusion and offers a remarkably simple intervention. The strength of the paper is that the proposed method is lightweight, the empirical demonstrations include a useful battery of diagnostic tests (noise corruption, Pearson correlation of attention with input quality, QUAG-style ablations, and OOD benchmarks), and the source code is promised. The claim that attention in multimodal Transformers becomes locked by a training feedback loop, and that a single query rotation can unlock it, is interesting and falsifiable. However, the paper's theoretical derivation is informal, the main accuracy numbers are reported without variance or multiple seeds, and the AIR indicator rests on a proxy that is not rigorously validated. The strongest baseline, MBT, still outperforms RollingQ on Kinetic-Sound and ties it on the other two datasets, so the practical edge over sophisticated baselines is not established even if the mechanism is correct.

major comments (3)
  1. [Table 1 and Section 4.2] The derivation equates the attention score for a modality with the sum of raw logits. This is not the post-softmax attention mass: the actual mass allocated to modality m is sum_{j in m} exp(q.k_j/sqrt(d)) divided by the sum over all tokens. Equation (4) rewrites the sum of raw logits, not the softmax allocation. Consequently, the AIR indicator in Eq. (9), the rebalance anchor in Eq. (10), and the rotation target all depend on the assumption that average-key cosine similarity tracks true softmax allocation. Even if ||k^a|| approx ||k^v|| and L_a approx L_v, this can fail: one modality with a few very large logits can dominate softmax while having a comparable sum of raw logits to a modality with many moderate logits. The noise-correlation experiments in Section 4.3 are suggestive but do not establish that the proxy remains faithful across the sample distribution or across different architectures. This is a load-bearing correctness risk; the paper should either reformulate AIR directly in terms of actual attention weights, or empirically validate the proxy, for example by reporting the correlation between the average-key cosine and the true softmax attention mass per sample.
  2. [Table 1 and Section 4.2] The reported accuracy improvements are not accompanied by standard deviations, confidence intervals, or multiple seeds. Some of the claimed gains are very small, e.g., +0.1 on MulT+RollingQ and +0.5 on Vanilla MT+RollingQ for CMU-MOSEI, and it is possible these are within run-to-run noise. The hyperparameters beta, rho, and the maximum rotation count are dataset-dependent, and the maximum rotation count is explicitly set to 1 or 3 per dataset in Appendix B. Without a sensitivity analysis or repeated-run statistics, the 'consistent accuracy gains' claim is not yet supported at the level the paper asserts. At minimum, the authors should provide mean and standard deviation over three or more seeds for the main comparison table and a sensitivity analysis for beta, rho, and rotation count.
  3. [Section 3.2, Eq. (8)] The self-reinforcing-cycle argument is presented as a theoretical analysis, but Eq. (8) does not rigorously imply that the modality receiving higher attention always obtains larger encoder gradients. The derivative of the softmax term through the attention weights can be negative for some tokens, and the gradient for modality m also includes the value-path term whose magnitude depends on the full attention distribution, not only on the total attention mass of that modality. The visualizations of gradient norms in Appendix A.2 are consistent with the proposed mechanism, but they are also consistent with other explanations such as differing learning dynamics of the two encoders. The paper should either formalize the cycle more carefully or explicitly label it as an intuitive mechanism supported only by empirical monitoring.
minor comments (5)
  1. [Abstract and Introduction] There are several typos and awkward phrasings: 'aims to to address', 'we has E[Q]=0', 'the the gradient', 'statisfied', and an inconsistent reference to the code repository as 'Github' without a URL in the abstract. These should be cleaned up.
  2. [Section 3.1] Equation (4) includes the sequence length L_m and the average key, but the text does not explain why the softmax denominator is ignored in the definition of 'attention score for modality m'. A sentence clarifying that this is a proxy, or a redefinition, would improve readability and prevent the misconception that it gives exact softmax mass.
  3. [Appendix B] The multi-layer extension trains the first attention block while 'temporarily dropping the second block' for several epochs, then switches to the second block. This is an unusual training schedule and the description is too brief to be reproducible; more detail on the number of epochs, how the dropped block is reintroduced, and how the AIR indicator is computed for the second block after switching is needed.
  4. [Table 4] The complexity analysis reports parameters and GFLOPs for CREMA-D, but the '1% increase in parameters' appears inconsistent with the table, which shows 59.87M to 60.46M, an increase of about 1% that checks out, while the GFLOPs increase is 0.07, about 0.005%. The text should state these numbers precisely rather than rounding to 0.1%, and the source of the GFLOPs value should be clarified.
  5. [Section 4.2] The paper claims RollingQ achieves 'comparable results' to static fusion imbalance techniques and 'strong performance' relative to MBT and JMT, but MBT outperforms RollingQ on Kinetic-Sound and ties on CREMA-D and MOSEI. The discussion should acknowledge this directly rather than implying superiority.

Circularity Check

1 steps flagged · score 6.0 of 10

Query rotation reduces the same AIR used to define imbalance, making part of the claimed 'revival' constructional; independent accuracy and noise tests prevent full circularity.

  1. self definitional [Section 3.3, Equations 9-13 (and Algorithm 1)]
    "To quantify the distribution gap, we define the AttentionImbalanceRate (AIR) indicator: AIR=E[cosθ a −cosθ v]∈[−2,2].(9) ... qb = (α E[ ˆK a] ||E[ ˆK a]||2 + (1−α) E[ ˆK v] ||E[ ˆK v]||2 )||E[Q]||2,(10) while the weight α is derived, indicating AIR: α= 1 2 [1 +T anh(−ρAIR)],(11) ... qr =qR b.(13)"

    The quantity RollingQ claims to restore is the same AIR that its update is constructed to reduce. AIR (Eq. 9) is the expected cosine difference between the query and the average keys. Eq. 11 makes the anchor weight α a decreasing function of AIR, Eq. 10 builds the anchor as the corresponding weighted average of the two normalized average keys, and Eqs. 12-13 rotate the query so that in expectation it becomes that anchor (q_b = E[Q]R_b, q_r = qR_b). Thus the immediate post-rotation decrease in AIR and the 'rebalancing' shown in Figures 4(a)-4(b) follow from the definition of the update rather than from a learned or independently verified training dynamic. The reported accuracy gains and noise-correlation experiments are separate results, so the circularity is partial rather than total.

full rationale

The paper's central diagnosis of a self-reinforcing cycle is an empirical/observational claim supported by visualizations and gradient monitoring; the softmax-versus-logit concern about Eq. 4 is a correctness risk, not a circularity. The circular step is in the intervention. AIR is defined as the expected cosine gap, and the rotation anchor is explicitly chosen to shrink that same gap: Eq. 11 sets α from AIR, Eq. 10 places the anchor as a weighted average of the normalized average keys, and Eqs. 12-13 enforce q_r = qR_b with E[Q]R_b = q_b. Hence 'rebalancing' after rotation is a mathematical identity of the update, not an independent empirical finding. However, the paper's most important validations—Table 1 accuracy gains, Table 3 noise-perturbation robustness, Table 2 Pearson correlations between attention and input quality, and Table 10 OOD results—do not follow from the construction and provide independent support. The self-citations in the related-work section (e.g., Peng et al. 2022, Huang et al. 2025) are baselines and background, not load-bearing justifications for the main claim. The authors' own limitation that the theoretical analysis focuses on a single attention layer is a scope restriction rather than a circularity. Overall, the central method is partially circular by construction, but the empirical evaluation keeps the paper from being fully circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method depends on a small set of hyperparameters (beta, rho, rotation limit) that are tuned per dataset, and on an informal geometric model of attention (cosine similarity with the average key). No new physical or architectural entities are introduced. The central theoretical mechanism, the self-reinforcing cycle, is asserted narratively rather than derived.

free parameters (3)
  • beta (AIR rotation threshold)
    Hyperparameter in Algorithm 1 and Equation 9 that decides when to rotate; the paper reports no default value and Appendix B notes smaller beta triggers more rotations.
  • rho (anchor weight sharpness)
    Positive hyperparameter in Equation 11 controlling how strongly the anchor down-weights the biased modality; value not reported.
  • maximum rotation count = 1 for CREMA-D and MOSEI, 3 for Kinetic-Sound
    Appendix B sets per-dataset rotation limits to stabilize training; this is a tuned hyperparameter that affects final accuracy.
assumptions (4)
  • domain assumption At initialization, the query and average keys are independent with E[Q]=0, so both modalities receive similar attention scores.
    Used in Proposition 3.1 to establish the starting point of the self-reinforcing cycle; follows from truncated normal initialization of the class token.
  • domain assumption The attention score for a modality is dominated by cosine similarity between the query and the modality's average key.
    Used to define AIR and the anchor; relies on approximately equal key norms and sequence lengths across modalities, stated in Section 3.3.
  • ad hoc to paper Higher attention to a modality amplifies its encoder gradients, which further improves that modality and entrenches the bias.
    The self-reinforcing cycle described in Section 3.2; Equation 8 shows gradient scaling but does not prove the feedback loop.
  • domain assumption Multimodal deep networks are greedy and favor the modality that first provides higher quality features.
    Borrowed from Wu et al. (2022) to justify why a modality becomes biased in the first place.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer." pith.science (2026). https://pith.science/paper/D4XXRMWY

@misc{pith2026250611465,
  author       = {Pith},
  title        = {Pith review of: RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D4XXRMWY}},
  note         = {Machine review of arXiv:2506.11465}
}
read the original abstract

Multimodal learning faces challenges in effectively fusing information from diverse modalities, especially when modality quality varies across samples. Dynamic fusion strategies, such as attention mechanism in Transformers, aim to address such challenge by adaptively emphasizing modalities based on the characteristics of input data. However, through amounts of carefully designed experiments, we surprisingly observed that the dynamic adaptability of widely-used self-attention models diminishes. Model tends to prefer one modality regardless of data characteristics. This bias triggers a self-reinforcing cycle that progressively overemphasizes the favored modality, widening the distribution gap in attention keys across modalities and deactivating attention mechanism's dynamic properties. To revive adaptability, we propose a simple yet effective method Rolling Query (RollingQ), which balances attention allocation by rotating the query to break the self-reinforcing cycle and mitigate the key distribution gap. Extensive experiments on various multimodal scenarios validate the effectiveness of RollingQ and the restoration of cooperation dynamics is pivotal for enhancing the broader capabilities of widely deployed multimodal Transformers. The source code is available at https://github.com/GeWu-Lab/RollingQ_ICML2025.

Figures

Figures reproduced from arXiv: 2506.11465 by the authors.

Figure 1
Figure 1. (a): Illustration of multimodal static fusion, where the modality weights ω a and ω v are fixed values after learned. (b): Illustration of multimodal dynamic fusion, where the modality weights ω a and ω v are determined by the input data x a and x v . (c): Attention scores assigned to each modality on the Kinetic-Sound dataset(Arandjelovic & Zisserman, 2017). The left half shows normal samples, while the right half … view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The illustration of RollingQ algorithm. encoder parameters θ v . Consequently, this dynamic further exacerbates the inequality between feature qualities. To verify this analysis, we provide visualization on gradients of unimodal encoders in Appendix A.2. In summary, the modality bias triggers a self-reinforcing cycle: during the feed-forward stage, biased modality ac￾cumulates more attention score due to its more in… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: (a): The distribution of two modalities before Rolling Query on the Kinetic-Sound dataset. (b):The distribution of two modalities after Rolling Query on Kinetic-Sound dataset. (c): The variation of attention score on each modality during the training process on Kinetic…
Figure 5
Figure 5. Figure 5: The distribution of two modalities is key when imple￾menting the RollingQ algorithm. The red dots denote the audio modality with noise interference [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The visualization of attention score and average key distribution on CREMA-D, Kinetic-Sound, CMU-MOSEI(V+T), CMU￾MOSEI(A+T), UCF-101, HMDB51 datasets, including audio (A), RGB (V), text (T) and optical flow (O) modalities. The left figure for each dataset denotes the a…
Figure 7
Figure 7. Figure 7: The L2-Norm of gradient of the audio encoder and the visual encoder on CREMA-D and Kinetic-Sound datasets. B. Implement Details of The RollingQ Algorithm Rotation time limits for training stability. In Algorithm 1, we calculate and update the rotation matrix whenever t…
Figure 8
Figure 8. Figure 8: The L2-Norm of gradient of the audio encoder and the visual encoder and attention score across modalities on CREMA-D and Kinetic-Sound datasets [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The L2-Norm of gradient of the audio encoder and the visual encoder and attention score across modalities on CREMA-D and Kinetic-Sound datasets when implement RollingQ [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 40 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Zisserman, A

    Arandjelovic, R. and Zisserman, A. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision, pp.\ 609--617, 2017

    work page 2017

  3. [3]

    Singular value decomposition tutorial

    Baker, K. Singular value decomposition tutorial. The Ohio State University, 24: 0 22, 2005

    work page 2005

  4. [4]

    Multimodal machine learning: A survey and taxonomy

    Baltru s aitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41 0 (2): 0 423--443, 2018

    work page 2018

  5. [5]

    Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters

    Bridle, J. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances in neural information processing systems, 2, 1989

    work page 1989

  6. [6]

    G., Keutmann, M

    Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5 0 (4): 0 377--390, 2014

    work page 2014

  7. [7]

    End-to-end object detection with transformers

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In European conference on computer vision, pp.\ 213--229. Springer, 2020

    2020

  8. [8]

    Vggsound: A large-scale audio-visual dataset

    Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 721--725. IEEE, 2020

    work page 2020

Show all 61 references
  1. [9]

    T., Rubanova, Y., Bettencourt, J., and Duvenaud, D

    Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  2. [10]

    Self-attention fusion for audiovisual emotion recognition with incomplete data

    Chumachenko, K., Iosifidis, A., and Gabbouj, M. Self-attention fusion for audiovisual emotion recognition with incomplete data. In 2022 26th International Conference on Pattern Recognition (ICPR), pp.\ 2822--2828. IEEE, 2022

    2022

  3. [11]

    What does bert look at? an analysis of bert’s attention

    Clark, K. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019

    1906 arXiv

  4. [12]

    Addressing failure prediction by learning model confidence

    Corbi \`e re, C., Thome, N., Bar-Hen, A., Cord, M., and P \'e rez, P. Addressing failure prediction by learning model confidence. Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [13]

    Multiood: Scaling out-of-distribution detection for multiple modalities

    Dong, H., Zhao, Y., Chatzi, E., and Fink, O. Multiood: Scaling out-of-distribution detection for multiple modalities. arXiv preprint arXiv:2405.17419, 2024

    2024 arXiv

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  7. [15]

    Pmr: Prototypical modal rebalance for multimodal learning

    Fan, Y., Xu, W., Wang, H., Wang, J., and Guo, S. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 20029--20038, 2023

    2023

  8. [16]

    A survey on deep learning for multimodal data fusion

    Gao, J., Li, P., Chen, Z., and Zhang, J. A survey on deep learning for multimodal data fusion. Neural Computation, 32 0 (5): 0 829--864, 2020

    2020

  9. [17]

    C., Wang, X., and Li, H

    Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C., Wang, X., and Li, H. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 6639--6648, 2019

    2019

  10. [18]

    A mathematical perspective on transformers

    Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. A mathematical perspective on transformers. arXiv preprint arXiv:2312.10794, 2023

    2023 arXiv

  11. [19]

    Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification

    Han, Z., Yang, F., Huang, J., Zhang, C., and Yao, J. Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 20707--20717, 2022

    2022

  12. [20]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  13. [21]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

    2022

  14. [22]

    Reconboost: Boosting can achieve modality reconcilement

    Hua, C., Xu, Q., Bao, S., Yang, Z., and Huang, Q. Reconboost: Boosting can achieve modality reconcilement. arXiv preprint arXiv:2405.09321, 2024

    2024 arXiv

  15. [23]

    Adaptive unimodal regulation for balanced multimodal information acquisition

    Huang, C., Wei, Y., Yang, Z., and Hu, D. Adaptive unimodal regulation for balanced multimodal information acquisition. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 25854--25863, 2025

    2025

  16. [24]

    Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably)

    Huang, Y., Lin, J., Zhou, C., Yang, H., and Huang, L. Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably). In International conference on machine learning, pp.\ 9226--9259. PMLR, 2022

    2022

  17. [25]

    Pixel-bert: Aligning image pixels with text by deep multi-modal transformers

    Huang, Z., Zeng, Z., Liu, B., Fu, D., and Fu, J. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020

    2004 arXiv

  18. [26]

    Joze, H. R. V., Shaban, A., Iuzzolino, M. L., and Koishida, K. Mmtm: Multimodal transfer module for cnn fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 13289--13299, 2020

    2020

  19. [27]

    Vilt: Vision-and-language transformer without convolution or region supervision

    Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning, pp.\ 5583--5594. PMLR, 2021

    2021

  20. [28]

    Revealing the dark secrets of bert

    Kovaleva, O. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.08593, 2019

    1908 arXiv

  21. [29]

    Hmdb: a large video database for human motion recognition

    Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pp.\ 2556--2563. IEEE, 2011

    2011

  22. [30]

    H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W

    Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., and Chang, K.-W. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  23. [31]

    P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M

    Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 2021 0 (DB1): 0 1, 2021

    2021

  24. [32]

    P., Zadeh, A., and Morency, L.-P

    Liang, P. P., Zadeh, A., and Morency, L.-P. Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430, 2022

    2022 arXiv

  25. [33]

    B., Liang, P

    Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Zadeh, A., and Morency, L.-P. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064, 2018

    2018 arXiv

  26. [34]

    Attention bottlenecks for multimodal fusion

    Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., and Sun, C. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34: 0 14200--14213, 2021

    2021

  27. [35]

    Balanced multimodal learning via on-the-fly gradient modulation

    Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8238--8247, 2022

    2022

  28. [36]

    S., Matyasko, A., Jaiswal, S., Fernando, B., and Tan, C

    Rawal, I. S., Matyasko, A., Jaiswal, S., Fernando, B., and Tan, C. Dissecting multimodality in videoqa transformer models by impairing modality fusion. arXiv preprint arXiv:2306.08889, 2023

    2023 arXiv

  29. [37]

    Zorro: the masked multimodal transformer

    Recasens, A., Lin, J., Carreira, J., Jaegle, D., Wang, L., Alayrac, J.-b., Luc, P., Miech, A., Smaira, L., Hemsley, R., et al. Zorro: the masked multimodal transformer. arXiv preprint arXiv:2301.09595, 2023

    2023 arXiv

  30. [38]

    Imagenet-21k pretraining for the masses

    Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik-Manor, L. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972, 2021

    2021 arXiv

  31. [39]

    and Monro, S

    Robbins, H. and Monro, S. A stochastic approximation method. The annals of mathematical statistics, pp.\ 400--407, 1951

    1951

  32. [40]

    R., and Shah, M

    Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  33. [41]

    L., Tickoo, O., and Huang, J

    Subedar, M., Krishnan, R., Meyer, P. L., Tickoo, O., and Huang, J. Uncertainty-aware audiovisual activity recognition using deep bayesian variational inference. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 6301--6310, 2019

    2019

  34. [42]

    H., Bai, S., Liang, P

    Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, pp.\ 6558. NIH Pub...

    2019

  35. [43]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  36. [44]

    H., Zeeshan, M

    Waligora, P., Aslam, M. H., Zeeshan, M. O., Belharbi, S., Koerich, A. L., Pedersoli, M., Bacon, S., and Granger, E. Joint multimodal transformer for emotion recognition in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4625...

    2024

  37. [45]

    What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12695--12705, 2020 a

    Wang, W., Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12695--12705, 2020 a

    2020

  38. [46]

    Deep multimodal fusion by channel exchanging

    Wang, Y., Huang, W., Sun, F., Xu, T., Rong, Y., and Huang, J. Deep multimodal fusion by channel exchanging. Advances in neural information processing systems, 33: 0 4835--4845, 2020 b

    2020

  39. [47]

    Multimodal token fusion for vision transformers

    Wang, Y., Chen, X., Cao, L., Huang, W., Sun, F., and Wang, Y. Multimodal token fusion for vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12186--12195, 2022

    2022

  40. [48]

    Wankhade, M., Rao, A. C. S., and Kulkarni, C. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55 0 (7): 0 5731--5780, 2022

    2022

  41. [49]

    Enhancing multimodal cooperation via sample-level modality valuation

    Wei, Y., Feng, R., Wang, Z., and Hu, D. Enhancing multimodal cooperation via sample-level modality valuation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 27338--27347, 2024

    2024

  42. [50]

    Wu, N., Jastrzebski, S., Cho, K., and Geras, K. J. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning, pp.\ 24043--24055. PMLR, 2022

    2022

  43. [51]

    Multimodal fusion with co-attention networks for fake news detection

    Wu, Y., Zhan, P., Zhang, Y., Wang, L., and Xu, Z. Multimodal fusion with co-attention networks for fake news detection. In Findings of the association for computational linguistics: ACL-IJCNLP 2021, pp.\ 2560--2569, 2021

    2021

  44. [52]

    Multimodal multi-loss fusion network for sentiment analysis

    Wu, Z., Gong, Z., Koo, J., and Hirschberg, J. Multimodal multi-loss fusion network for sentiment analysis. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers),...

    2024

  45. [53]

    Balanced audiovisual dataset for imbalance analysis

    Xia, W., Zhao, X., Pang, X., Zhang, C., and Hu, D. Balanced audiovisual dataset for imbalance analysis. arXiv preprint arXiv:2302.10912, 2023

    2023 arXiv

  46. [54]

    Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (10): 0 12113--12132, 2023

    2023

  47. [55]

    Facilitating multimodal classification via dynamically learning modality gap

    Yang, Y., Wan, F., Jiang, Q., and Xu, Y. Facilitating multimodal classification via dynamically learning modality gap. In Advances in Neural Information Processing Systems 38, 2024

    2024

  48. [56]

    Learning to rebalance multi-modal optimization by adaptively masking subnetworks

    Yang, Y., Pan, H., Jiang, Q., Xu, Y., and Tang, J. Learning to rebalance multi-modal optimization by adaptively masking subnetworks. IEEE Trans. Pattern Anal. Mach. Intell. , 2025

    2025

  49. [57]

    Deep modular co-attention networks for visual question answering

    Yu, Z., Yu, J., Cui, Y., Tao, D., and Tian, Q. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 6281--6290, 2019

    2019

  50. [58]

    Tensor fusion network for multimodal sentiment analysis

    Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250, 2017

    2017 arXiv

  51. [59]

    B., Liang, P

    Zadeh, A. B., Liang, P. P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2018

  52. [60]

    T., and Peng, X

    Zhang, Q., Wu, H., Zhang, C., Hu, Q., Fu, H., Zhou, J. T., and Peng, X. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pp.\ 41753--41769. PMLR, 2023

    2023

  53. [61]

    Multimodal fusion on low-quality data: A comprehensive survey

    Zhang, Q., Wei, Y., Han, Z., Fu, H., Peng, X., Deng, C., Hu, Q., Xu, C., Wen, J., Hu, D., et al. Multimodal fusion on low-quality data: A comprehensive survey. arXiv preprint arXiv:2404.18947, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.