Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Open-set Cross Modal Generalization via Multimodal Unified Representation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MICU's masked contrastive learning and modality-agnostic jigsaw puzzles let one multimodal representation handle unseen classes.

desk verdict Useful new task, plausible method, but the missing open-set decision rule makes the headline HOS numbers unreproducible. read the letter →

arxiv 2507.14935 v1 pith:KJ7QF4KW submitted 2025-07-20 cs.CV

Hai Huang , Yan Xia , Shulei Wang , Hanting Wang , Minghui Fang , Shengpeng Ji , Sashuai Zhou , Tao Jin
show 1 more author
Zhou Zhao
This is my paper
classification cs.CV
keywords open-setrecognitioncross-modalgeneralizationmultimodalunifiedrepresentationmaskedcontrastivelearningself-supervisedjigsawpuzzlevectorquantizationaudio-visual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Open-set Cross Modal Generalization (OSCMG), a task in which a model pretrained on paired multimodal data must transfer from one modality to another while also recognizing classes never seen during downstream training. It argues that prior cross-modal generalization benchmarks are closed-set and therefore miss a common real-world failure mode. To address this, it proposes MICU, which adds masked contrastive learning at both whole-sample and per-timestep levels plus a modality-agnostic jigsaw-puzzle task on discretized features. The paper reports that MICU outperforms prior models on the new OSCMG benchmark and on the established CMG benchmark.

What carries the argument

The load-bearing machinery is a pair of training signals on a shared discrete codebook: FCMI, an InfoNCE loss applied at coarse and fine temporal scales with sample-specific masking aligned across modalities, and CUJP, a permutation-recognition task over randomly selected quantized segments drawn from any modality. The codebook maps features from all modalities into a common discrete space, and the jigsaw task forces the model to use block-level structure without relying on which modality a block came from. Ablations in the paper show that the coarse contrastive loss is the foundation, while the fine-grained contrastive loss and the jigsaw loss each improve open-set performance.

What would settle it

Re-run the OSCMG evaluations with one explicit, fixed unknown-detection rule, for example a threshold on the maximum class score or an energy score, applied identically to every model, and check whether MICU still leads on HOS.

Watch

Extended reading notes

Core claim

The central claim is that a multimodal unified representation can be made open-set-ready by combining two complementary self-supervised objectives. Fine-Coarse Masked Multimodal InfoNCE (FCMI) aligns modalities at two scales, whole-sequence semantics and individual timesteps, with masking applied at aligned positions across modalities. Cross-modal Unified Jigsaw Puzzles (CUJP) quantizes features through a shared codebook, then shuffles randomly chosen segments across modalities and asks the model to identify the permutation, treating all modalities as one. The paper argues this builds a representation that transfers across modalities and lets a linear probe trained on known classes separate known from unknown classes. It reports higher HOS than previous models in 11 of 12 OSCMG settings and consistent gains on the closed-set CMG tasks.

Load-bearing premise

The reported numbers assume that every model being compared decides which inputs are 'unknown' in the same way, but the paper never states what rule the models used to make that decision.

Editorial extensions

If this is right

  • If OSCMG becomes a standard benchmark, cross-modal models will be judged on whether they can reject unseen classes, not just on known-class accuracy.
  • Downstream systems for rare modalities such as optical flow could inherit audio and video pretraining with a linear probe and still detect inputs that do not belong to the training classes.
  • The modality-agnostic jigsaw design cuts the permutation search from factorial in all modalities combined to factorial in a single modality's segment count, making open-set pretraining with three or more modalities practical.
  • MICU's reported gains on both open-set and closed-set tasks suggest that the added objectives do not trade away ordinary cross-modal alignment quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The aligned masking in FCMI likely acts as a regularizer that removes modality-specific timesteps, forcing the contrastive loss to rely on shared semantic content, but the paper does not isolate that mechanism directly.
  • CUJP's benefits may extend beyond video, audio, and optical flow: any paired modalities that can be quantized into segments, such as text spans or skeleton sequences, could use the same permutation task.
  • Because the paper never specifies the unknown-detection rule behind the UNK accuracy, the reported HOS comparisons across different models should be read as conditional on an unstated evaluation protocol.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Open-set Cross Modal Generalization (OSCMG), a new evaluation protocol that extends the existing Cross Modal Generalization (CMG) task to settings where the target modality contains unknown classes. To address this task, the authors propose MICU, which combines Fine-Coarse Masked multimodal InfoNCE (FCMI) for fine- and coarse-grained cross-modal alignment with Cross-modal Unified Jigsaw Puzzles (CUJP), a modality-agnostic self-supervised task over a shared discrete codebook. The paper reports state-of-the-art results on the new OSCMG benchmark and on the established CMG benchmark, with HOS, precision, F1, and retrieval metrics, alongside ablations and supplementary analyses.

Significance. The paper addresses a timely and relevant problem: how to evaluate and improve multimodal unified representations when downstream test classes are not fully known during training. The proposed task is a natural extension of the authors' prior CMG work and the benchmark construction is reasonable. The methodological components, especially the combination of masked contrastive learning with unified jigsaw puzzles, are interesting and the reported margins over the controlled DCID baseline are nontrivial. The paper also provides code and detailed supplementary analyses. However, the central SOTA claim currently rests on an incompletely specified open-set evaluation protocol, and at least one ablation number is arithmetically inconsistent. These issues must be resolved before the results can be fully trusted.

major comments (4)
  1. [Section 4.1] The open-set decision rule is never specified. The paper defines OS*, UNK, and HOS but does not describe how the linear probe trained only on known classes decides whether a test sample is unknown. No threshold on maximum softmax probability, energy score, distance to prototype, or additional background class is mentioned. This is load-bearing because every UNK value in Tables 2 and 7 depends on this rule, and comparisons across methods are only meaningful if the rule is identical for all methods and is not tuned per method. Please specify the inference-time decision rule, the threshold-selection protocol (e.g., whether a held-out validation set is used), and confirm that the same rule is applied to all baselines.
  2. [Table 7] The row '✓ - ✓' for AVE Split1 V→A reports OS*=7.17, UNK=12.23, HOS=41.34. Using the paper's formula HOS = 2 * OS* * UNK / (OS* + UNK), the expected HOS is approximately 9.04, not 41.34. This is a concrete arithmetic inconsistency in a central ablation table. Please correct the value and audit all entries in Tables 7 and 8 for consistency with the HOS formula.
  3. [Section 3.3] The formal definition of CUJP is difficult to parse. The notation for the permuted segment sequence appears corrupted, and the paper does not specify how the permutation labels are generated, what P is (a hyperparameter or a fixed number), or how the random selection across modalities is implemented. Since CUJP is one of the two main contributions, a precise and unambiguous description is required for reproducibility.
  4. [Section 4.1 / Table 4] The pretraining uses 'text provided by [51]', i.e., the text from the DCID paper, and DCID is also the strongest baseline. Please clarify whether the same pretraining text is used for all baselines and whether the downstream class splits are exactly the same for all methods. This is necessary to rule out an indirect advantage to MICU from shared pretraining data.
minor comments (6)
  1. [Section 3.3] The superscript notation in the permutation example is garbled; please reformat the equation for readability.
  2. [Supplementary Figure 5] The captions refer to 'MICA' instead of 'MICU'; please correct the name.
  3. [Table 5] The rows for DCID and MICU in the UCF(vf)-VGG(a) block contain concatenated numbers (e.g., '69.8762.4974.23 65.01'); please ensure all entries are separated clearly.
  4. [Abstract] The provided code link (https://github.com/haihuangcode/CMG) points to a repository named 'CMG', which may correspond to the prior work rather than this paper; please update to the actual repository for MICU.
  5. [Section 4.2] The sentence 'The 5th row shows the same effect as the 2nd row' is ambiguous; given the arithmetic issue in row 5, the statement should be clarified and verified.
  6. [Section 3.2] The mask generation mechanism (e.g., whether masks are random per sample or fixed) is not described; please add a sentence specifying how the binary masks are sampled.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claimed SOTA rests on measured external evaluations; the open-set metric reproducibility issues are correctness concerns, not circular reductions.

full rationale

This paper's derivation chain is self-contained with respect to circularity. The core losses (Eqs. 4-8) are standard contrastive, VQ, reconstruction, and jigsaw objectives; no output metric or predicted quantity is defined in terms of the model's own fitted parameters. The OSCMG evaluation uses external datasets (AVE, UCF, VGGSound, AVVP) and compares against published baselines; the reported HOS numbers, whatever their reproducibility, are measured rather than derived. The pretraining text and backbone settings inherited from the authors' prior DCID paper [51] are data and protocol reuse, not load-bearing circularity, because the claimed improvements are empirical comparisons against DCID rather than consequences of assuming DCID is correct. The only self-citation-heavy elements are the baseline DCID and the CMG task definition, and neither is used to forbid alternatives or to define MICU's outputs. The skeptic's objection that no unknown-class decision rule is specified (Section 4.1) and the apparent inconsistency in ablation Table 7 (e.g., OS*=7.17, UNK=12.23 with HOS=41.34, while the stated formula HOS = 2*OS*UNK/(OS+UNK) gives about 9.04) are serious reproducibility and correctness concerns, but they are not circular reasoning: they do not show an input being renamed as a prediction. Accordingly, no circular step can be quoted with a specific reduction, and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical entities or theoretical constructs are introduced. The shared codebook is a standard VQ module. The main assumptions are domain assumptions about data alignment and evaluation metrics.

free parameters (5)
  • codebook size = 400
    Selected by experiments in Figure 5 of the supplementary; 400 gave the best HOS and precision on the downstream tasks, so it is tuned on the evaluation data.
  • mask ratio of FCMI = 30%
    Reported in implementation details; the paper says 'the mask ratio of FCMI is set to 30%' without a search, but the masking strategy is later evaluated in Figure 4, so the ratio is a chosen hyperparameter.
  • CUJP segment count = 8 (CUJP8)
    The main tables and Table 9 use CUJP8, the best-performing variant in Figure 3. The number of splits is chosen by downstream performance.
  • loss weights lambda1..lambda4 = 1, 2, 1, 1
    Set manually; no sensitivity analysis reported.
  • temperature tau = 1.0
    Reported as 1.0 for both Lfine and Lcoarse.
assumptions (4)
  • standard math InfoNCE contrastive loss is a valid alignment objective for paired multimodal data
    Used in FCMI (Eq. 4, 5).
  • domain assumption The paired samples (video, audio, text) in VGGSound-A VEL40K with text from [51] are semantically aligned
    The pretraining relies on this alignment to form positive pairs.
  • domain assumption OS*, UNK, and HOS are appropriate metrics for open-set evaluation
    Adopted from prior open-set works [2,14,28], but the decision rule for UNK is not specified.
  • ad hoc to paper Masking the same time positions across modalities preserves semantic alignment and helps contrastive learning
    Supported only by the authors' own Figure 4 experiments; no theoretical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Open-set Cross Modal Generalization via Multimodal Unified Representation." pith.science (2026). https://pith.science/paper/KJ7QF4KW

@misc{pith2026250714935,
  author       = {Pith},
  title        = {Pith review of: Open-set Cross Modal Generalization via Multimodal Unified Representation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJ7QF4KW}},
  note         = {Machine review of arXiv:2507.14935}
}
read the original abstract

This paper extends Cross Modal Generalization (CMG) to open-set environments by proposing the more challenging Open-set Cross Modal Generalization (OSCMG) task. This task evaluates multimodal unified representations in open-set conditions, addressing the limitations of prior closed-set cross-modal evaluations. OSCMG requires not only cross-modal knowledge transfer but also robust generalization to unseen classes within new modalities, a scenario frequently encountered in real-world applications. Existing multimodal unified representation work lacks consideration for open-set environments. To tackle this, we propose MICU, comprising two key components: Fine-Coarse Masked multimodal InfoNCE (FCMI) and Cross modal Unified Jigsaw Puzzles (CUJP). FCMI enhances multimodal alignment by applying contrastive learning at both holistic semantic and temporal levels, incorporating masking to enhance generalization. CUJP enhances feature diversity and model uncertainty by integrating modality-agnostic feature selection with self-supervised learning, thereby strengthening the model's ability to handle unknown categories in open-set tasks. Extensive experiments on CMG and the newly proposed OSCMG validate the effectiveness of our approach. The code is available at https://github.com/haihuangcode/CMG.

Figures

Figures reproduced from arXiv: 2507.14935 by the authors.

Figure 1
Figure 1. After unsupervised pretraining, the model is directly [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The architecture of MICU, illustrated with an example of fine and coarse InfoNCE with masked audio and video, as well as [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Experimental results of different Jigsaw Puzzles. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Experimental results of different Mask. 7. Codebook Size The size of the representation space also affects the model’s performance. As shown in [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Experimental results of different Codebook Size. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Purple (avt) indicates where all three modalities have quantized activations [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A two-stage prompt-tuning method with low-rank and contrastive prompt enhancement claims all-in-one adverse weather removal at 2.75M parameters.

Reference graph

Works this paper leans on

65 extracted references · 51 canonical work pages · cited by 1 Pith paper

  1. [51]

    Achiev- ing cross modal generalization with multimodal unified rep- resentation

    Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. Achiev- ing cross modal generalization with multimodal unified rep- resentation. Advances in Neural Information Processing Sys- tems, 36, 2024. 1, 2, 3, 5, 6, 7

    work page 2024

  2. [1]

    Robust cross-modal representation learning with progressive self- distillation

    Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self- distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430– 16441, 2022. 3

    work page 2022

  3. [2]

    On the effectiveness of image rotation for open set domain adaptation

    Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tom- masi. On the effectiveness of image rotation for open set domain adaptation. In European conference on computer vi- sion, pages 422–438. Springer, 2020. 6

    work page 2020

  4. [3]

    Domain generalization by solving jigsaw puzzles

    Fabio M Carlucci, Antonio D’Innocente, Silvia Bucci, Bar- bara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2229–2238, 2019. 2, 5

    work page 2019

  5. [4]

    Collecting highly paral- lel data for paraphrase evaluation

    David Chen and William B Dolan. Collecting highly paral- lel data for paraphrase evaluation. InProceedings of the 49th annual meeting of the association for computational linguis- tics: human language technologies , pages 190–200, 2011. 6

    2011

  6. [5]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721–725. IEEE, 2020. 5, 6

    work page 2020

  7. [6]

    Hts-at: A hierarchi- cal token-semantic audio transformer for sound classifica- tion and detection

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchi- cal token-semantic audio transformer for sound classifica- tion and detection. In ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 646–650. IEEE, 2022. 6

    work page 2022

  8. [7]

    Valor: Vision-audio- language omni-perception pretraining model and dataset

    Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Wein- ing Wang, Jinhui Tang, and Jing Liu. Valor: Vision-audio- language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345, 2023. 1

    arXiv 2023

Show all 65 references
  1. [8]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104–120. Springer,

  2. [9]

    Sinkd: Sinkhorn distance minimization for knowledge distillation

    Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, and Houqiang Li. Sinkd: Sinkhorn distance minimization for knowledge distillation. TNNLS, 2024

    2024

  3. [10]

    Sinkhorn distance minimization for knowledge distilla- tion

    Xiao Cui, Yulei Qin, Yuting Gao, Enwei Zhang, Zihan Xu, Tong Wu, Ke Li, Xing Sun, Wengang Zhou, and Houqiang Li. Sinkhorn distance minimization for knowledge distilla- tion. In LREC-COLING, pages 14846–14858, 2024

    2024

  4. [11]

    Optical: Leveraging optimal trans- port for contribution allocation in dataset distillation

    Xiao Cui, Yulei Qin, Liang Xie, Wengang Zhou, Hongsheng Li, and Houqiang Li. Optical: Leveraging optimal trans- port for contribution allocation in dataset distillation. CVPR, 2025

    2025

  5. [12]

    Layoutenc: Leveraging enhanced layout rep- resentations for transformer-based complex scene synthesis

    Xiao Cui, Qi Sun, Min Wang, Li Li, Wengang Zhou, and Houqiang Li. Layoutenc: Leveraging enhanced layout rep- resentations for transformer-based complex scene synthesis. ACM Transactions on Multimedia Computing, Communica- tions and Applications, 2025

    2025

  6. [13]

    Streetsurfgs: Scalable ur- ban street surface reconstruction with planar-based gaussian splatting

    Xiao Cui, Weicai Ye, Yifan Wang, Guofeng Zhang, Wengang Zhou, Tong He, and Houqiang Li. Streetsurfgs: Scalable ur- ban street surface reconstruction with planar-based gaussian splatting. IEEE Transactions on Circuits and Systems for Video Technology, 2025

    2025

  7. [14]

    Towards mul- timodal open-set domain generalization and adaptation through self-supervision

    Hao Dong, Eleni Chatzi, and Olga Fink. Towards mul- timodal open-set domain generalization and adaptation through self-supervision. arXiv preprint arXiv:2407.01518,

    arXiv

  8. [15]

    Simmmdg: A simple and effective framework for multi-modal domain generalization

    Hao Dong, Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. Simmmdg: A simple and effective framework for multi-modal domain generalization. Advances in Neural In- formation Processing Systems, 36, 2024. 2, 3

    2024

  9. [16]

    Clotho: An audio captioning dataset

    Konstantinos Drossos, Samuel Lipping, and Tuomas Virta- nen. Clotho: An audio captioning dataset. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 736–740. IEEE,

    2020

  10. [17]

    Multi-modal align- ment using representation codebook

    Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi-modal align- ment using representation codebook. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15651–15660, 2022. 1, 3, 6, 7

    2022

  11. [18]

    Ace: A generative cross-modal retrieval framework with coarse-to-fine semantic modeling

    Minghui Fang, Shengpeng Ji, Jialong Zuo, Hai Huang, Yan Xia, Jieming Zhu, Xize Cheng, Xiaoda Yang, Wenrui Liu, Gang Wang, et al. Ace: A generative cross-modal retrieval framework with coarse-to-fine semantic modeling. arXiv preprint arXiv:2406.17507, 2024

    2024

  12. [19]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 6

    2019

  13. [20]

    Domain-adversarial training of neural networks

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pas- cal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research , 17(59):1–35, 2016. 3

    2016

  14. [21]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023. 2

    2023

  15. [22]

    Enhancing multimodal unified rep- 9 resentations for cross modal generalization

    Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Minghui Fang, Jieming Zhu, Zhenhua Dong, Sashuai Zhou, and Zhou Zhao. Enhancing multimodal unified rep- 9 resentations for cross modal generalization. arXiv preprint arXiv:2403.05168, 2024. 3

    2024 arXiv

  16. [23]

    Semantic residual for multimodal unified discrete representation

    Hai Huang, Shulei Wang, and Yan Xia. Semantic residual for multimodal unified discrete representation. InICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  17. [24]

    Bridging domain generalization to multimodal domain generalization via unified represen- tations

    Hai Huang, Yan Xia, Sashuai Zhou, Hanting Wang, Shulei Wang, and Zhou Zhao. Bridging domain generalization to multimodal domain generalization via unified represen- tations. arXiv preprint arXiv:2507.03304, 2025. 3

    2025 arXiv

  18. [25]

    Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling.arXiv preprint arXiv:2408.16532, 2024

    Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling.arXiv preprint arXiv:2408.16532, 2024

    2024 arXiv

  19. [26]

    Learning to generalize: Meta-learning for do- main generalization

    Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for do- main generalization. In Proceedings of the AAAI conference on artificial intelligence, 2018. 3

    2018

  20. [27]

    Domain generalization for med- ical imaging classification with linear-dependency regular- ization

    Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang, Tie- Qiang Li, and Alex Kot. Domain generalization for med- ical imaging classification with linear-dependency regular- ization. Advances in neural information processing systems, 33:3118–3129, 2020. 3

    2020

  21. [28]

    Adjustment and alignment for unbiased open set domain adaptation

    Wuyang Li, Jie Liu, Bo Han, and Yixuan Yuan. Adjustment and alignment for unbiased open set domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 24110–24119, 2023. 6

    2023

  22. [29]

    Cross- modal discrete representation learning

    Alexander H Liu, SouYoung Jin, Cheng-I Jeff Lai, An- drew Rouditchenko, Aude Oliva, and James Glass. Cross- modal discrete representation learning. arXiv preprint arXiv:2106.05438, 2021. 1, 2, 3, 6, 7

    2021 arXiv

  23. [30]

    Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous fre- quency space

    Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous fre- quency space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , page...

    2021

  24. [31]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 6

    2021

  25. [32]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022. 1, 3

    2022 arXiv

  26. [33]

    Unsupervised learning of visual representations by solving jigsaw puzzles

    Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Euro- pean conference on computer vision, pages 69–84. Springer,

  27. [34]

    Two at once: Enhancing learning and generalization capacities via ibn-net

    Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In Proceedings of the european conference on computer vision (ECCV), pages 464–479, 2018. 3

    2018

  28. [35]

    Estimat- ing visual information from audio through manifold learn- ing

    Fabrizio Pedersoli, Dryden Wiebe, Amin Banitalebi, Yong Zhang, George Tzanetakis, and Kwang Moo Yi. Estimat- ing visual information from audio through manifold learn- ing. arXiv preprint arXiv:2208.02337, 2022. 3

    2022 arXiv

  29. [36]

    Audio-visual speech recognition with a hybrid ctc/attention architecture

    Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Geor- gios Tzimiropoulos, and Maja Pantic. Audio-visual speech recognition with a hybrid ctc/attention architecture. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 513–520. IEEE, 2018. 3

    2018

  30. [37]

    Domain generalization through audio- visual relative norm alignment in first person action recog- nition

    Mirco Planamente, Chiara Plizzari, Emanuele Alberti, and Barbara Caputo. Domain generalization through audio- visual relative norm alignment in first person action recog- nition. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1807–18...

    2022

  31. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [39]

    Mask2anomaly: Mask transformer for uni- versal open-set segmentation

    Shyam Nandan Rai, Fabio Cermelli, Barbara Caputo, and Carlo Masone. Mask2anomaly: Mask transformer for uni- versal open-set segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 4

    2024

  33. [40]

    Xkd: Cross-modal knowl- edge distillation with domain alignment for video represen- tation learning

    Pritam Sarkar and Ali Etemad. Xkd: Cross-modal knowl- edge distillation with domain alignment for video represen- tation learning. arXiv preprint arXiv:2211.13929, 2022. 1, 3

    2022 arXiv

  34. [41]

    Open domain generalization with domain- augmented meta-learning

    Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain- augmented meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9624–9633, 2021. 2, 3

    2021

  35. [42]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 5, 6

    2012 arXiv

  36. [43]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chen- liang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Com- puter Vision (ECCV), pages 247–263, 2018. 5, 6

    2018

  37. [44]

    Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing

    Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified mul- tisensory perception: Weakly-supervised audio-visual video parsing. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part III 16, pages 436–454. Springer, 2020. 6

    2020

  38. [45]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Woj- ciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 2...

    2017

  39. [46]

    Deep domain confusion: Maximizing for domain invariance

    Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014. 3

    2014 arXiv

  40. [47]

    Irbridge: Solving image restoration bridge with pre-trained generative diffusion mod- els

    Hanting Wang, Tao Jin, Wang Lin, Shulei Wang, Hai Huang, Shengpeng Ji, and Zhou Zhao. Irbridge: Solving image restoration bridge with pre-trained generative diffusion mod- els. arXiv preprint arXiv:2505.24406, 2025. 10

    2025 arXiv

  41. [48]

    Towards transformer-based aligned generation with self-coherence guidance

    Shulei Wang, Wang Lin, Hai Huang, Hanting Wang, Si- hang Cai, WenKang Han, Tao Jin, Jingyuan Chen, Jiacheng Sun, Jieming Zhu, et al. Towards transformer-based aligned generation with self-coherence guidance. arXiv preprint arXiv:2503.17675, 2025

    2025 arXiv

  42. [49]

    Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix

    Teng Wang, Wenhao Jiang, Zhichao Lu, Feng Zheng, Ran Cheng, Chengguo Yin, and Ping Luo. Vlmixer: Unpaired vision-language pre-training via cross-modal cutmix. In In- ternational Conference on Machine Learning, pages 22680– 22690. PMLR, 2022. 3

    2022

  43. [50]

    General- izable decision boundaries: Dualistic meta-learning for open set domain generalization

    Xiran Wang, Jian Zhang, Lei Qi, and Yinghuan Shi. General- izable decision boundaries: Dualistic meta-learning for open set domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11564– 11573, 2023. 2

    2023

  44. [52]

    Hyper- spectral image classification based on unsupervised hetero- geneous domain adaptation cyclegan

    W ANG Xuesong, LI Yiran, and CHENG Yuhu. Hyper- spectral image classification based on unsupervised hetero- geneous domain adaptation cyclegan. Chinese Journal of Electronics, 29(4):608–614, 2020. 3

    2020

  45. [53]

    Class semantics modulation for open-set in- stance segmentation

    Yifei Yang, ZhongXiang Zhou, Jun Wu, Yue Wang, and Rong Xiong. Class semantics modulation for open-set in- stance segmentation. IEEE Robotics and Automation Letters,

  46. [54]

    Learn- ing domain-invariant and discriminative features for homo- geneous unsupervised domain adaptation

    ZHANG Yun, W ANG Nianbin, and CAI Shaobin. Learn- ing domain-invariant and discriminative features for homo- geneous unsupervised domain adaptation. Chinese Journal of Electronics, 29(6):1119–1125, 2020. 3

    2020

  47. [55]

    A du- ality based approach for realtime tv-l 1 optical flow

    Christopher Zach, Thomas Pock, and Horst Bischof. A du- ality based approach for realtime tv-l 1 optical flow. In Pat- tern Recognition: 29th DAGM Symposium, Heidelberg, Ger- many, September 12-14, 2007. Proceedings 29 , pages 214–

    2007

  48. [56]

    mixup: Beyond empirical risk management

    H Zhang, M Cisse, Y Dauphin, and D Lopez-Paz. mixup: Beyond empirical risk management. In 6th Int. Conf. Learn- ing Representations (ICLR), pages 1–13, 2018. 3

    2018

  49. [57]

    Extending multi-modal contrastive rep- resentations

    Ziang Zhang, Zehan Wang, Luping Liu, Rongjie Huang, Xize Cheng, Zhenhui Ye, Huadai Liu, Haifeng Huang, Yang Zhao, Tao Jin, et al. Extending multi-modal contrastive rep- resentations. Advances in Neural Information Processing Systems, 37:91880–91903, 2024. 3

    2024

  50. [58]

    Towards effective multi-modal interchanges in zero-resource sounding object localization

    Yang Zhao, Chen Zhang, Haifeng Huang, Haoyuan Li, and Zhou Zhao. Towards effective multi-modal interchanges in zero-resource sounding object localization. Advances in Neural Information Processing Systems , 35:38089–38102,

  51. [59]

    Positive sample propagation along the audio- visual event line

    Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, and Meng Wang. Positive sample propagation along the audio- visual event line. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 8436–8444, 2021. 6

    2021

  52. [60]

    Contrastive pos- itive sample propagation along the audio-visual event line

    Jinxing Zhou, Dan Guo, and Meng Wang. Contrastive pos- itive sample propagation along the audio-visual event line. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2022. 5, 6 11 Open-set Cross Modal Generalization via Multimodal Unified Representation Suppleme...

    2022

  53. [61]

    As shown in Figure 4, applying the same mask to paired multimodal samples helps improve model perfor- mance

    Mask of FCMI We also conducted an analysis on different masking strate- gies. As shown in Figure 4, applying the same mask to paired multimodal samples helps improve model perfor- mance. This approach facilitates more precise and detailed alignment between modalities, ensuring...

  54. [62]

    As shown in Figure 5, we experimented with five different settings: 256, 400, 512, 800, and 1024

    Codebook Size The size of the representation space also affects the model’s performance. As shown in Figure 5, we experimented with five different settings: 256, 400, 512, 800, and 1024. Among these, 400 led by a significant margin over the other settings. Therefore, we chose ...

  55. [63]

    Lcoarse serves as the foundation of the model, while Lf ine and Lcujp further refine the unified representation space and enhance the model’s open-domain detection capabilities

    Ablation on CMG The experimental results of Table 8 and Table 7 are similar. Lcoarse serves as the foundation of the model, while Lf ine and Lcujp further refine the unified representation space and enhance the model’s open-domain detection capabilities. Lfine Lcoarse Lcujp A ...

  56. [64]

    CUJP8, despite having more split block reorder- ing, optimizes memory usage and reduces training time compared to MMJP6 [14]

    Computational Efficiency As shown in Table 9, compared to CMCM [29] and DCID [51], our method requires more GPU memory and longer per-epoch training time, but achieves better perfor- mance, reflecting a trade-off between performance and re- sources. CUJP8, despite having more ...

  57. [65]

    The visualization maps audio-video-text triplets from the Valor32K dataset [7] into the unified rep- resentation space (codebook)

    Unified Representation Space Visualization As shown in Figure 6, the two subfigures illustrate the rep- resentation spaces of DCID [51] after pre-training and our proposed model. The visualization maps audio-video-text triplets from the Valor32K dataset [7] into the unified re...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.