Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that DICCAE, an encoder that measures inter-class confusion from the overlap of class feature circles and reweights a confusion loss accordingly, reaches 65.5% top-1 audio-visual accuracy on VGGSound, matching the…

desk verdict The confusion-weighting idea is worth a second look, but the headline 65.5% number is not an honest measurement because the method uses test-set labels during training. read the letter →

arxiv 2507.09323 v1 pith:FCXIMANM submitted 2025-07-12 cs.CV

Kaixuan Cong , Yifan Wang , Rongkun Xue , Yuyang Jiang , Yiming Feng , Jing Yang This is my paper
classification cs.CV
keywords audio-visualfusionhumanactivityrecognitioninter-classconfusiondynamiclossweightingself-supervisedpretrainingK-meansclusteringVGGSound
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes DICCAE, an encoder for human activity recognition that aligns audio and video at the category level rather than only aligning whole modalities. The core idea is to measure how much each pair of activity classes overlaps in the model's feature space, then use that overlap as a per-pair weight on a confusion loss, so training focuses on classes the model finds hard to tell apart. On VGGSound the method reports 60.1% top-1 with audio only and 65.5% with audio-video, matching the strongest published audio-visual result on that benchmark. The paper also introduces a cluster-guided self-supervised pretraining stage that uses K-means labels on roughly 2.3 million unlabeled audio-video samples, and ablations show each component contributes to the final accuracy. A sympathetic reader would take the contribution to be a practical, dynamic way to make multimodal models spend capacity where categories are confusable.

What carries the argument

The central object is the inter-class confusion degree matrix $M$, whose entry for classes $i,j$ is $M_{ij} = \max(0, r_i + r_j - d_{ij}) / d_{ij}$, where $r_i, r_j$ are radii of circles covering 95% of each class's feature points and $d_{ij}$ is the distance between class centroids. The matrix is recomputed after each training epoch on an update set, normalized to $[0,2]$, and used to weight the confusion loss $L_{\mathrm{DICCAE}} = \sum_{i,j} \hat{M}_{ij} L_{\mathrm{CL}}(i,j)$, where $L_{\mathrm{CL}}$ is a binary cross-entropy on feature similarity. The second load-bearing mechanism is cluster-guided pretraining: K-means with $K=300$ and 20 random initializations supplies category labels from roughly 2.3 million unlabeled audio-video samples, with clusters refreshed every 10 epochs.

What would settle it

Inspect the split logic used in the code accompanying the paper: if the update-set samples are included in the test subset evaluated for Table 1, then re-running evaluation after excluding them should change the reported accuracy, and if they are already excluded, the claim survives. A reproduction that draws the update set only from the training or validation split would settle whether the dynamic weighting gain is genuine.

Watch

Extended reading notes

Core claim

DICCAE's central claim is that inter-class confusion can be quantified during training and used to automatically reweight a contrastive loss. For each class, the model fits a circle covering 95% of its feature points, computes the overlap between class circles as a confusion degree, normalizes those degrees, and trains with a weighted confusion loss that pushes same-class features together and different-class features apart. The paper reports that this yields 60.1% top-1 audio-only and 65.5% audio-visual accuracy on VGGSound, matching the best existing audio-visual method on that dataset. Ablations show that removing the confusion loss, the dynamic weighting, the contrastive learning framework, the cluster refinement, or the K-means guidance each lowers accuracy, with the largest drop coming from removing the contrastive framework. The paper positions DICCAE as the first audio-video fusion framework to incorporate category-level induction and comparison during training.

Load-bearing premise

Everything rests on the assumption that the samples used to compute the confusion matrix are set aside from the test samples used to measure accuracy, because the paper describes taking a portion of the test set for the update set during training; if those samples are also scored, the reported accuracy is not a fair held-out measurement.

Editorial extensions

If this is right

  • If DICCAE's central claim is correct, dynamic confusion weighting offers a principled alternative to hand-tuned loss weights for confusable classes in multimodal recognition.
  • The cluster-guided pretraining step means unlabeled audio-video data from several datasets can be merged into a larger training pool, easing the labeled-data bottleneck for activity recognition.
  • The framework's components are modular: confusion loss, dynamic weighting, contrastive alignment, and cluster refinement can each be toggled, as the ablation study demonstrates.
  • The reported audio-only 60.1% and audio-visual 65.5% on VGGSound suggest that the approach transfers gains from fusion back to single-modality encoders.
  • The paper's conclusion states that the same inter-class confusion metric could be extended to other complex spatio-temporal tasks, guiding where a model should spend capacity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper's described use of a portion of the test set as the update set for the confusion matrix (Sections 2.1.3 and 2.2) means the headline 65.5% is not a standard held-out number unless those samples are excluded from the scored test split; this is our reading of the text, not something the paper's own evaluation verifies.
  • Editorial inference: the inter-class confusion distribution (mean down 0.75, variance down 4.89 with DICCAE) could serve as a general diagnostic for multimodal classifiers, letting practitioners identify which class pairs remain hard without needing the full loss machinery.
  • Editorial inference: because the confusion weights are recomputed every epoch, the method behaves like a learned curriculum over class pairs; one testable extension is whether freezing the matrix early changes final accuracy, and whether the same weighting transfers to new label sets.
  • Editorial inference: if the test-set update issue is fixed by using a validation-only update set, the expected drop in accuracy would quantify how much of the reported gain came from in-sample weighting; this is a concrete experiment the authors could run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DICCAE, an audio-visual encoder for human activity recognition. The method consists of a dynamic inter-class confusion loss that estimates confusion from feature-space centroid overlap and uses it to weight a pairwise contrast/induction loss, a joint audio-video training framework, and a cluster-guided self-supervised pre-training strategy. On VGGSound, the paper reports 60.1% audio-only and 65.5% audio-visual top-1 accuracy, with ablations attributing gains to the confusion loss, dynamic weighting, contrastive learning, cluster refinement, and cluster-guided pre-training.

Significance. If the reported numbers were clean, the paper would make a useful contribution: explicitly quantifying inter-class confusion during training and combining it with cluster-guided pre-training are reasonable ideas, and the ablation battery is thoughtfully organized. However, the central evaluation is contaminated by test-set information, and the paper does not ship code or checkpoints. As it stands, the empirical contribution cannot be credited, and the main claims are not supported by an unbiased evaluation protocol.

major comments (4)
  1. [Section 2.1.3 and Section 2.2, Eq. (4)] At the end of each training epoch, a portion of the test set is used as the update set to compute the inter-class confusion degree matrix. Computing the per-class centroids and enclosing circles in Section 2.1.2 requires the true class labels of those update-set samples, and the resulting matrix is then applied as the loss weight in Eq. (4). The final evaluation in Table 1 is therefore not independent of the training process: test labels and the test feature distribution have been used to shape the model. The reported 65.5% accuracy and the ablation deltas in Table 2 cannot be read as unbiased generalization estimates. Please re-run all experiments with an update set drawn from the training split or from a separate validation split, and state explicitly which split is used.
  2. [Section 3.1] The pre-training corpus is formed by combining AudioSet, VGGSound, EPIC-KITCHENS-100, and UCF101 into approximately 2.3 million samples, while fine-tuning and evaluation are performed on VGGSound. If any part of the VGGSound test split is included in that corpus, the self-supervised phase has already seen the test videos, which can only inflate the reported accuracy. Please specify the exact VGGSound splits used in pre-training and exclude all evaluation videos from every pre-training stage.
  3. [Table 2] The ablation deltas are small, for example 0.5% for audio accuracy when dynamic weighting is removed, and the manuscript reports no number of seeds, standard deviations, or significance tests. Without this information it is impossible to tell whether the Table 2 differences are real or noise, and this matters because dynamic weighting and the confusion loss are the paper's central mechanisms.
  4. [Section 2.1.1, Eq. (1)] The similarity function F_p(f_i, f_j) is never defined. It appears to be used in a binary cross-entropy style loss, but no parametric form, temperature, feature normalization, or pair-sampling procedure is given, so the loss is not reproducible. Please specify the similarity function and describe how the pairs (i, j) are sampled, including whether Eq. (4) sums over all class pairs or over sampled pairs.
minor comments (4)
  1. [Section 3.1] The sentence 'the supervised phase was limited to 30 steps' is unclear; it should likely read '30 epochs' or the intended unit should be stated, and the meaning of '500 iterations' should also be clarified.
  2. [Table 1] The header 'A V A-V' is ambiguous, and DICCAE's 65.5% audio-visual result exactly equals CAV-MAE's 65.5%, so the text should say that DICCAE ties rather than merely approaches that result.
  3. [Table 2] The row labeled 'w/o K-means' is described as using only coarse-grained alignment on AudioSet, so it simultaneously changes the dataset and removes clustering; the ablation design should separate the effect of the dataset choice from the effect of cluster-guided pre-training.
  4. [Figure 2] The comparison of inter-class confusion distributions reports only a mean decrease and variance decrease; it would be helpful to state how the distribution was computed, how many classes were included, and whether the result is stable across runs.

Circularity Check

3 steps flagged · score 7.0 of 10

Reported VGGSound accuracy is compromised: the test set is explicitly used to compute dynamic confusion weights during training, and VGGSound is included in the pre-training corpus.

  1. fitted input called prediction [Section 2.1.3 (Dynamic Weighting, item 1), Eq. (4), Section 2.2, Table 1]
    "At the end of each training epoch, we select a portion of the test set as the update set, which is used to calculate the inter-class confusion degree matrix. ... Additionally, during training, a portion of the test set is separated as the update set for calculating the inter-class confusion degree matrix, which is used to dynamically adjust the weight."

    The inter-class confusion degree matrix Mij is computed from the update set's feature-space centroids and enclosing circles (Section 2.1.2), which requires true class labels for test samples. The normalized matrix is then inserted directly into the training loss in Eq. (4) as the weights. The final top-1 accuracy is reported on the same test split (Table 1). Thus the test set is used to fit the loss weights during training before being used for evaluation; the reported 65.5% is not an independent generalization measurement but a fitted-input-called-prediction.

  2. other [Section 3.1 (pre-training datasets), Section 3.2 (experimental setup), Table 1]
    "we combine Audioset [9], VGGSound [4], EPIC-KITCHENS-100 [6], and UCF101 [24] into a larger dataset of approximately 2.3 million samples. ... fine-tuning experiments were conducted on the VGGSound dataset."

    The pre-training corpus includes VGGSound, the same dataset used for fine-tuning and evaluation. The paper does not state that the VGGSound test split is excluded from this corpus. If the test videos are part of the pre-training data, the model has already seen their audio-visual content during self-supervised pre-training, so the reported VGGSound accuracy is not an unbiased estimate of generalization to unseen data.

1 more flagged steps
  1. self definitional [Section 2.1.2 (Feature Quality Evaluation), Eq. (4), Section 3.3]
    "Feature Quality Evaluation: The distribution of inter-class confusion degrees is used to evaluate the quality of the feature representation. ... Thanks to the application of DICCAE, the mean of the distribution decreased by 0.75, and the variance reduced by 4.89, indicating that the feature representations produced by DICCAE have less overlap between similar classes."

    The inter-class confusion degree is precisely the quantity that Eq. (4) minimizes during training through dynamic weighting. Reporting a decrease in this same quantity as evidence that DICCAE improves feature quality is self-referential: the model is optimized to reduce Mij, so observing a reduction in Mij is expected and does not independently validate the method.

full rationale

The paper's central empirical claim is the 65.5% top-1 audio-visual accuracy on VGGSound (Table 1). That claim is not supported by an independent evaluation protocol. Section 2.1.3 explicitly selects 'a portion of the test set as the update set' each epoch; Section 2.1.2 computes per-class centroids and enclosing circles, which requires true labels; Eq. (4) uses the resulting confusion-degree matrix as weights in the training loss. The final accuracy is then measured on the same test split, so test labels and test feature statistics are used to fit the model before evaluation. This is a fitted-input-called-prediction pattern rather than a derivation-level circularity, but it directly invalidates the headline number as a generalization estimate. Additionally, Section 3.1 includes VGGSound in the pre-training corpus without stating an exclusion of the test split, so the self-supervised phase may have already seen the test videos. The confusion-degree distribution used in Section 3.3 as evidence of improvement is the same objective that Eq. (4) minimizes, making that ablation evidence self-referential. No self-citation chain is involved; the circularity is empirical/evaluative rather than a formal derivation equivalence, but it is load-bearing for the paper's main reported result.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on hand-chosen hyperparameters and on untested modeling assumptions: circle overlap as a confusion metric, K-means pseudo-labels as semantic categories, and an evaluation protocol that includes the target dataset and portions of the test set in training. No new physical entities are introduced.

free parameters (4)
  • K-means cluster count K = 300
    Set by default in Section 3.1. It defines the granularity of pseudo-labels in the self-supervised pre-training stage.
  • Enclosing circle coverage = 95%
    Section 2.1.2. The 95% threshold defines the inter-class confusion Mij and is chosen to avoid outlier influence.
  • Confusion matrix normalization range = [0, 2]
    Eq. (3). The arbitrary scaling affects the magnitude of the dynamic loss weights.
  • Number of K-means random initializations = 20
    Section 2.3. A computational choice made to stabilize clustering results.
assumptions (4)
  • domain assumption Audio and video modalities provide complementary information that resolves single-modality class confusion.
    Stated in the Introduction and used to justify the entire fusion approach.
  • ad hoc to paper Overlap of class centroids and enclosing circles accurately quantifies inter-class confusion.
    Eq. (2) is introduced without derivation or external validation; the circle-fitting procedure is only sketched.
  • ad hoc to paper K-means cluster assignments on merged audio-video data serve as meaningful category labels for fine-grained alignment.
    Section 2.3. Success depends on clusters capturing semantic categories, which is not tested.
  • ad hoc to paper The update set can be drawn from the test set without affecting generalization measurement.
    Sections 2.1.3 and 2.2. This is a methodological assumption that is likely invalid and is not defended.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition." pith.science (2026). https://pith.science/paper/FCXIMANM

@misc{pith2026250709323,
  author       = {Pith},
  title        = {Pith review of: Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCXIMANM}},
  note         = {Machine review of arXiv:2507.09323}
}
read the original abstract

Humans do not understand individual events in isolation; rather, they generalize concepts within classes and compare them to others. Existing audio-video pre-training paradigms only focus on the alignment of the overall audio-video modalities, without considering the reinforcement of distinguishing easily confused classes through cognitive induction and contrast during training. This paper proposes the Dynamic Inter-Class Confusion-Aware Encoder (DICCAE), an encoder that aligns audio-video representations at a fine-grained, category-level. DICCAE addresses category confusion by dynamically adjusting the confusion loss based on inter-class confusion degrees, thereby enhancing the model's ability to distinguish between similar activities. To further extend the application of DICCAE, we also introduce a novel training framework that incorporates both audio and video modalities, as well as their fusion. To mitigate the scarcity of audio-video data in the human activity recognition task, we propose a cluster-guided audio-video self-supervised pre-training strategy for DICCAE. DICCAE achieves near state-of-the-art performance on the VGGSound dataset, with a top-1 accuracy of 65.5%. We further evaluate its feature representation quality through extensive ablation studies, validating the necessity of each module.

Figures

Figures reproduced from arXiv: 2507.09323 by the authors.

Figure 1
Figure 1. The diagram illustrates Dynamic Inter-Class Confusion [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of inter-class confusion degree distribution [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Audio classification accuracy as a function of confusion [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages

  1. [1]

    Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text

    Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS, 34:24206–24221, 2021. 2

    work page 2021

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, and Oriol Vinyals. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022

    work page 2022

  3. [3]

    Multimodal machine learning: A survey and tax- onomy

    Tadas Baltru ˇsaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and tax- onomy. IEEE TPAMI, 41(2):423–443, 2019. 2

    work page 2019

  4. [4]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721–725. IEEE, 2020. 2, 4

    work page 2020

  5. [5]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597–1607. PmLR, 2020. 1

    work page 2020

  6. [6]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018. 2, 4

    work page 2018

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929, 2020. 1

    work page 2020

  8. [8]

    Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

    Haytham M Fayek and Anurag Kumar. Large scale audio- visual learning of sounds with weakly labeled data. arXiv preprint arXiv:2006.01595, 2020. 5

    work page Pith review arXiv 2006

Show all 31 references
  1. [9]

    Audio set: An ontology and human- labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In 2017 IEEE interna- tional conference on acoustics, speech and signal processin...

    2017

  2. [10]

    Audiovisual masked autoencoders

    Mihai-Iulian Georgescu, Eduardo Fonseca, Radu Tudor Ionescu, et al. Audiovisual masked autoencoders. ICCV,

  3. [11]

    Ast: Audio spectrogram transformer

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778,

    arXiv

  4. [12]

    Uavm: Towards unifying audio and visual models

    Yuan Gong, Alexander H Liu, Andrew Rouditchenko, and James Glass. Uavm: Towards unifying audio and visual models. IEEE Signal Processing Letters , 29:2437–2441,

  5. [13]

    Contrastive audio-visual masked autoencoder.arXiv preprint arXiv:2210.07839, 2022

    Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James Glass. Contrastive audio-visual masked autoencoder.arXiv preprint arXiv:2210.07839, 2022. 5

    2022 arXiv

  6. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1

    2016

  7. [15]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:...

    2021

  8. [16]

    Perceiver io: A general architecture for structured inputs & outputs

    Andreas Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, and Oriol Vinyals. Perceiver io: A general architecture for structured inputs & outputs. pages 4651–4664, 2021. 2, 5

    2021

  9. [17]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673,

  10. [18]

    Kipf and Max Welling

    Thomas N. Kipf and Max Welling. Semi-supervised clas- sification with graph convolutional networks. ICLR, 2017. 1

    2017

  11. [19]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 1

    2021

  12. [20]

    Attention bottlenecks for multimodal fusion

    Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in neural information process- ing systems, 34:14200–14213, 2021. 5

    2021

  13. [21]

    Omninet: A unified architecture for multi-modal multi-task learning

    Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain. Omninet: A unified architecture for multi-modal multi-task learning. arXiv preprint arXiv:1907.07804, 2019. 2

    1907 arXiv

  14. [22]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, et al. Robust speech recognition via large-scale weak supervision. 2023. 1

    2023

  15. [23]

    Multimodal fusion for audio-image and video action recognition

    Muhammad Bilal Shaikh, Douglas Chai, Syed Mo- hammed Shamsul Islam, and Naveed Akhtar. Multimodal fusion for audio-image and video action recognition. Neural Computing and Applications, 36(10):5499–5513, 2024. 2

    2024

  16. [24]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 2, 4

    2012 arXiv

  17. [25]

    Omnivec: Learn- ing robust representations with cross modal sharing

    Siddharth Srivastava and Gaurav Sharma. Omnivec: Learn- ing robust representations with cross modal sharing. In Pro- ceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1236–1248, 2024. 2

    2024

  18. [26]

    One-peace: Ex- ploring one general representation model toward unlimited modalities

    Peng Wang, Shuai Wang, Jing Lin, et al. One-peace: Ex- ploring one general representation model toward unlimited modalities. arXiv preprint arXiv:230511172, 2023. 2

    2023

  19. [27]

    What makes train- ing multi-modal classification networks hard? In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695–12705, 2020

    Weiyao Wang, Du Tran, and Matt Feiszli. What makes train- ing multi-modal classification networks hard? In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695–12705, 2020. 5

    2020

  20. [28]

    Multi-stream multi-class fusion of deep net- works for video classification

    Zuxuan Wu, Yu-Gang Jiang, Xi Wang, Hao Ye, and Xi- angyang Xue. Multi-stream multi-class fusion of deep net- works for video classification. In Proceedings of the 24th ACM international conference on Multimedia , pages 791– 800, 2016. 2

    2016

  21. [29]

    Multimodal learning with transformers: A survey

    Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(10):12113– 12132, 2023. 1

    2023

  22. [30]

    Peters, and Yejin Choi

    Rowan Zellers, Ximing Lu, Ari Holtzman, Swaroop Mishra, Matthew E. Peters, and Yejin Choi. Merlot: Multimodal neu- ral script knowledge models. NeurIPS, 2021. 2

    2021

  23. [31]

    Multimodal representation learning: Advances, trends and challenges

    Su-Fang Zhang, Jun-Hai Zhai, Bo-Jun Xie, Yan Zhan, and Xin Wang. Multimodal representation learning: Advances, trends and challenges. In 2019 International Conference on Machine Learning and Cybernetics (ICMLC) , pages 1–

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.