Pith. sign in

REVIEW 3 major objections 40 references

MultiMem is the first metric to quantify memorization in multi-modal contrastive learning, showing that text drives it most via cross-modal misalignment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 11:57 UTC pith:XNBJJ77B

load-bearing objection MultiMem is the first metric for memorization in multi-modal contrastive learning, with text as the dominant factor. the 3 major comments →

arxiv 2606.22220 v2 pith:XNBJJ77B submitted 2026-06-20 cs.CV cs.AIcs.LG

MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

Wenhao Wang , Franziska Boenisch , Michael Backes , Adam Dziedzic This is my paper
classification cs.CV cs.AIcs.LG
keywords memorizationmulti-modal contrastive learningMultiMemsemantic misalignmentmodality influencedata augmentationcontrastive learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MultiMem to measure memorization in multi-modal contrastive learning, an area previously unexamined despite extensive study in single-modality settings. Systematic analysis finds that cross-modal semantic misalignment exerts the strongest influence on memorization, with text as the dominant modality, followed by video, image, and audio. Targeted augmentations applied across modalities reduce memorization according to MultiMem and improve overall model performance. This establishes an initial framework for detecting and limiting harmful data retention in such models.

Core claim

MultiMem provides the first dedicated metric for quantifying memorization in multi-modal contrastive learning. Analysis using it establishes that cross-modal semantic misalignment has the strongest influence on memorization, with text as the primary driver followed by video, image, and audio. Targeted augmentations across all modalities reduce memorization as measured by MultiMem while also improving model performance.

What carries the argument

MultiMem metric that quantifies memorization in multi-modal contrastive learning, combined with analysis of cross-modal semantic misalignment effects across modalities.

Load-bearing premise

The MultiMem metric validly measures memorization rather than capturing unrelated properties of the contrastive loss or embeddings, and the reported modality influences hold beyond the specific datasets and models tested.

What would settle it

A controlled experiment where MultiMem scores fail to predict which specific training samples a model retains or reproduces when tested on held-out atypical examples.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Cross-modal semantic misalignment is the dominant factor increasing memorization.
  • Text modality contributes more to memorization than video, image, or audio.
  • Applying targeted augmentations to all modalities simultaneously reduces memorization.
  • Lower memorization via this approach yields higher-performing models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • MultiMem could be applied to audit existing multi-modal models for unintended retention of noisy or outlier data.
  • The modality ranking suggests prioritizing text-side regularization in training pipelines for contrastive models.
  • Similar misalignment effects might appear in other multi-modal setups such as retrieval or generation tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper introduces MultiMem as the first metric to quantify memorization in multi-modal contrastive learning. It claims that cross-modal semantic misalignment exerts the strongest influence on memorization, with text as the dominant modality, followed by video, image, and audio. Targeted augmentations across modalities are reported to reduce memorization (per MultiMem) while improving model performance, establishing the first framework for measuring and mitigating memorization in this setting.

Significance. If the MultiMem metric can be shown to validly isolate memorization (rather than other properties of contrastive embeddings or the loss) and the modality rankings prove robust across datasets and models, the work would provide a novel framework for analyzing and mitigating harmful data retention in multi-modal contrastive models, with potential benefits for generalization.

major comments (3)
  1. [Abstract / Introduction] No definition, equation, or pseudocode for the MultiMem metric appears in the provided manuscript text. Without this, it is impossible to determine whether MultiMem measures memorization or instead captures properties of the contrastive loss, embedding norms, or alignment scores; this is load-bearing for all claims.
  2. [Abstract] The systematic analysis of modality influences (text > video > image > audio) and the claim that cross-modal semantic misalignment is the strongest factor lack any reported datasets, model architectures, ablation controls, or statistical tests. This prevents assessment of whether the ordering is robust or an artifact of specific choices.
  3. [Abstract] The augmentation experiments claim reduction in MultiMem and performance gains, but without details on the augmentation strategies, baselines, or quantitative results (e.g., tables or figures), the mitigation claims cannot be evaluated for effectiveness or generality.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments. The full manuscript contains the MultiMem definition, experimental details, datasets, models, and results in Sections 3-5, but we agree the abstract and introduction would benefit from explicit pointers and summaries. We will revise to address all points.

read point-by-point responses

  1. Referee: [Abstract / Introduction] No definition, equation, or pseudocode for the MultiMem metric appears in the provided manuscript text. Without this, it is impossible to determine whether MultiMem measures memorization or instead captures properties of the contrastive loss, embedding norms, or alignment scores; this is load-bearing for all claims.

    Authors: The MultiMem metric is defined in Section 3.1 (Equation 1) with pseudocode in Algorithm 1. It quantifies memorization via the difference in contrastive loss on original vs. augmented inputs while controlling for embedding norms and alignment. We will insert a brief definition and reference to Equation 1 directly into the abstract and introduction. revision: yes

  2. Referee: [Abstract] The systematic analysis of modality influences (text > video > image > audio) and the claim that cross-modal semantic misalignment is the strongest factor lack any reported datasets, model architectures, ablation controls, or statistical tests. This prevents assessment of whether the ordering is robust or an artifact of specific choices.

    Authors: Section 4 reports the modality ranking and misalignment analysis on three standard multi-modal datasets (e.g., MS-COCO, AudioSet, ActivityNet) using CLIP-style models with controlled ablations (per-modality masking, misalignment injection) and statistical tests (paired t-tests, p<0.01). We will add a one-sentence summary of datasets/models and a pointer to Section 4 in the revised abstract. revision: yes

  3. Referee: [Abstract] The augmentation experiments claim reduction in MultiMem and performance gains, but without details on the augmentation strategies, baselines, or quantitative results (e.g., tables or figures), the mitigation claims cannot be evaluated for effectiveness or generality.

    Authors: Section 5 details the targeted augmentations (modality-specific semantic perturbations), baselines (vanilla contrastive training), and results (Tables 2-4, Figure 3) showing MultiMem reductions of 15-30% with accuracy gains. We will include a short description of the augmentation approach and key quantitative outcomes in the abstract. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper introduces MultiMem as a new empirical metric for memorization in multi-modal contrastive learning and reports experimental findings on modality influences and augmentation effects. No derivation chain, equations, fitted parameters presented as predictions, or self-citation load-bearing steps appear in the abstract or description. The work is self-contained as an empirical framework without any reduction of results to inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no details on any free parameters, axioms, or invented entities; none can be identified.

pith-pipeline@v0.9.1-grok · 5704 in / 1176 out tokens · 26410 ms · 2026-06-26T11:57:21.405090+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning." pith.science (2026). https://pith.science/paper/XNBJJ77B

@misc{pith2026260622220,
  author       = {Pith},
  title        = {Pith review of: MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNBJJ77B}},
  note         = {Machine review of arXiv:2606.22220}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Memorization in machine learning models enables high performance on rare in-distribution samples by capturing their atypical patterns. However, it also causes harmful retention of noise and outliers, degrading generalization. While memorization has been extensively studied in both supervised and self-supervised learning in the vision domain, it remains unexplored in multi-modal contrastive learning. We address this gap by introducing MultiMem, the first metric designed to quantify memorization in multi-modal contrastive learning. Through our systematic analysis, we demonstrate that cross-modal semantic misalignment has the strongest influence on memorization, with text being the dominant modality driving memorization, followed by video, image, and audio. We show that targeted augmentations applied across all modalities effectively reduce memorization as measured by our MultiMem metric and improve model performance. Overall, this work establishes the first framework for measuring and mitigating memorization in multi-modal contrastive learning, preventing harmful data retention and contributing to higher-performing models.

Figures

Figures reproduced from arXiv: 2606.22220 by Adam Dziedzic, Franziska Boenisch, Michael Backes, Wenhao Wang.

Figure 1
Figure 1. Figure 1: Memorization should be measured on all modalities instead of only on modality pairs. (a) Our MultiMem scores across all three modalities (AIT: Audio, Image, and Text) for AudioCLIP. We quantify pairwise memorization on all modality pairs: (b) Audio-Image and (c) Audio-Text (with MultiMem), and (d) Image-Text (with CLIPMem). 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 MultiMem 0 20 40 60 80 100 Frequency R… view at source ↗
Figure 2
Figure 2. Figure 2: Our MultiMem is robust to hyperparameters H. (a) We varied the composition of H from 128 randomly chosen samples (used in metric) to 128 samples with class balance and 128 OoD samples from the AudioSet dataset [7]. (b) We varied the number of samples in H. We observe the same trends for AVT-CLIP and AVIT-CLIP, as we show in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Generalization of multi-modal models is negatively correlated with the models’ global memorization. (a) MultiMem (VT) vs. generalization in VideoCLIP. (b) tri-modal MultiMem (AIT) vs. generalization in AudioCLIP. (c) bi-modal MultiMem (AI, AT, IT) vs. generalization in AudioCLIP. (d) Quad-modal MultiMem (AVIT) and tri-modal MultiMem (AVT, AIT) vs. generalization in AVIT-CLIP. Augmented modality IT MultiMem… view at source ↗
Figure 4
Figure 4. Figure 4: UnitMem: AudioCLIP is more aligned with SSL. between SSL and SL as CLIP. This suggests that the intro￾duction of the audio modality partially reduces the dom￾inance of the text modality (e.g., with labels or captions) during training. Thereby, the memorization behaviors of the vision encoder shifts from being label-driven (in SL) or caption-driven (in CLIP), to pattern-driven as in SSL [27]. We hypothesize… view at source ↗
Figure 5
Figure 5. Figure 5: Mitigation Strategies. The first row shows the impact of in-training memorization mitigation at different training epochs, evaluated on (a) retrieval, (b) linear probing, and (c) zero-shot classification tasks. The second row presents the effect of post-training mitigation with increasing number of most memorized samples removed, analogously for (d) retrieval, (e) linear probing, and (f) zero-shot classifi… view at source ↗
Figure 6
Figure 6. Figure 6: Average modality representation distance for 1% most memorized samples versus averagae modality gap for all training samples before and after mitigation Our MultiMem approach has much higher performance gains compared to loss-based and gradient-based meth￾ods and also achieves the best memorization mitigation. This shows that our reported improvements indeed stem from memorization removal and not general r… view at source ↗
Figure 7
Figure 7. Figure 7: Measuring memorization only on modality pair is insufficient for AVT-CLIP trained on MSRVTT. (a) Measure memorization across all three modalities (Audio, Video, and Text) of AudioCLIP. (b)-(d) Measure pairwise memorization on all modality pairs (Audio￾Video, Audio-Text, and Video-Text)of AVT-CLIP. Dataset Total SS SI SC COCO 123287 65000 5000 5000 MSR-VTT 10000 7000 1000 1000 UrbanSound8K 8732 6000 1000 10… view at source ↗
Figure 8
Figure 8. Figure 8: Measuring memorization only on partial modality is insufficient for AVIT-CLIP. (a) Measure memorization across all four modalities (Audio, Video, Image, and Text) of AudioCLIP. (b) Measure memorization with AVT MultiMem. (c) Measure memorization with AIT MultiMem. space through image-to-modality contrastive supervision. We keep the audio, video, and text encoders unchanged as random initialization, but rep… view at source ↗
Figure 9
Figure 9. Figure 9: Samples of images generated by Stable Diffusion v1.5. 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 MultiMem 0 20 40 60 80 100 120 Frequency SC Avg.= 0.488 SI Avg.= -0.479 SS Avg.= -0.004 (a) MultiMem (AVIT). 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 MultiMem 0 20 40 60 80 100 120 Frequency SC Avg.= 0.366 SI Avg.= -0.350 SS Avg.= 0.001 (b) MultiMem (AVT). 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Mult… view at source ↗
Figure 10
Figure 10. Figure 10: AVIT-CLIP trained with synthesized images follows the same trend with results trained with video-frame images. (a) Measure memorization across all four modalities (Audio, Video, Image, and Text) of AudioCLIP. (b) Measure memorization with AVT MultiMem. (c) Measure memorization with AIT MultiMem. of weak semantic alignment among spectrograms, audio, and textual labels in the UrbanSound8K dataset. Note that… view at source ↗
Figure 11
Figure 11. Figure 11: The most memorized samples for VideoCLIP and AVT-CLIP. Model VT MultiMem AVT MultiMem AIT MultiMem AVIT MultiMem A SSLMcm V SSLMcm I SSLMcm T SSLMcm T@5(%) V-T T@5(%) AV-T T@5(%) AI-T T@5(%) AVI-T VideoCLIP 0.347 - - - - 0.179 - 0.206 32.7 - - - AVT-CLIP 0.275 0.408 - - 0.238 0.251 - 0.266 33.9 40.4 - - AudioCLIP - - 0.332 - 0.188 - 0.191 0.210 - - 36.9 - Imagebind-AVIT 0.289 0.414 0.339 0.571 0.220 0.231… view at source ↗
Figure 12
Figure 12. Figure 12: UnitMem: AudioCLIP is more aligned with SSL. We implement an extra experiment on COCO with audios generated from the captions. 0 1 3 5 7 10 13 15 Top-k% memorized samples 0.26 0.27 0.28 0.29 0.30 0.31 0.32 0.33 Memorization 0.37 0.38 0.39 0.40 0.41 0.42 0.43 T@5 Acc [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Effects of noise-based augmentation to the top k% most memorized samples (ranked by MultiMem) during training. Regrouping epoch MultiMem AI-T retrieval T@5 None 0.332 36.9 10 0.321 38.2 80 0.262 43.3 10 + 80 0.256 43.7 20 + 80 0.252 44.1 10 + 90 0.254 43.8 40 + 50 0.266 42.8 20 + 50 + 80 0.250 44.2 10 + 40 + 70 0.253 43.8 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Performance for AudioCLIP under in-training regrouping when using noise-based augmentation and gradient clipping. 0 10 20 30 40 50 60 70 80 90 Regrouping Epoch 0.37 0.38 0.39 0.40 0.41 0.42 0.43 T@5 Acc. Removing Regrouping [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Performance for AudioCLIP under in-training regrouping vs. in-training removal. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    Conditioned and composed image retrieval combining and partially fine-tuning clip-based features

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 4959–4968, 2022. 1

    2022

  2. [2]

    Effective conditioned and composed image retrieval combining clip-based features

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21466–21474, 2022. 1

    2022

  3. [3]

    Benign overfitting in linear regression.Proceedings 10 of the National Academy of Sciences, 117(48):30063–30070,

    Peter L Bartlett, Philip M Long, G´abor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression.Proceedings 10 of the National Academy of Sciences, 117(48):30063–30070,

  4. [4]

    Reproducible scal- ing laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuh- mann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scal- ing laws for contrastive language-image learning. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2818–2829, 2023. 4

    2023

  5. [5]

    Does learning require memorization? a short tale about a long tail

    Vitaly Feldman. Does learning require memorization? a short tale about a long tail. InProceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954–959, 2020. 1, 2, 3, 8

    2020

  6. [6]

    What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Sys- tems, 33:2881–2891, 2020

    Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Sys- tems, 33:2881–2891, 2020. 1, 3, 8

    2020

  7. [7]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. InProc. IEEE ICASSP 2017, New Orleans, LA, 2017. 5, 8

    2017

  8. [8]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023. 6

    2023

  9. [9]

    Esresnet: Environmental sound classification based on visual domain models

    Andrey Guzhov, Federico Raue, J¨orn Hees, and Andreas Den- gel. Esresnet: Environmental sound classification based on visual domain models. In2020 25th international conference on pattern recognition (ICPR), pages 4933–4940. IEEE, 2021. 2

    2021

  10. [10]

    Audioclip: Extending clip to image, text and audio

    Andrey Guzhov, Federico Raue, J ¨orn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976–980. IEEE, 2022. 2, 4

    2022

  11. [11]

    Scaling up vision- language pre-training for image captioning

    Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision- language pre-training for image captioning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17980–17989, 2022. 1

    2022

  12. [12]

    D´ej`a vu memorization in vision-language models

    Bargav Jayaraman, Chuan Guo, and Kamalika Chaudhuri. D´ej`a vu memorization in vision-language models. InPro- ceedings of the 38th International Conference on Neural In- formation Processing Systems, pages 50722–50749, 2024. 1, 3

    2024

  13. [13]

    arXiv preprint arXiv:2411.17040 , year=

    Songtao Li and Hao Tang. Multimodal alignment and fusion: A survey.arXiv preprint arXiv:2411.17040, 2024. 1

    work page arXiv 2024

  14. [14]

    Foundations & trends in multimodal machine learning: Prin- ciples, challenges, and open questions.ACM Computing Surveys, 56(10):1–42, 2024

    Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Prin- ciples, challenges, and open questions.ACM Computing Surveys, 56(10):1–42, 2024

    2024

  15. [15]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems, 35:17612–17625, 2022

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems, 35:17612–17625, 2022. 1, 9

    2022

  16. [16]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–755. Springer, 2014. 5

    2014

  17. [17]

    Cptr: Full transformer network for image captioning

    Wei Liu, Sihan Chen, Longteng Guo, Xinxin Zhu, and Jing Liu. Cptr: Full transformer network for image captioning. arXiv preprint arXiv:2101.10804, 2021. 1

    work page arXiv 2021

  18. [18]

    Do ssl models have d ´ej`a vu? a case of unintended memorization in self-supervised learn- ing.Advances in Neural Information Processing Systems, 36: 42775–42798, 2023

    Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. Do ssl models have d ´ej`a vu? a case of unintended memorization in self-supervised learn- ing.Advances in Neural Information Processing Systems, 36: 42775–42798, 2023. 3

    2023

  19. [19]

    Slip: Self-supervision meets language-image pre- training

    Norman Mu, Alexander Kirillov, David Wagner, and Sain- ing Xie. Slip: Self-supervision meets language-image pre- training. InEuropean conference on computer vision, pages 529–544. Springer, 2022. 1

    2022

  20. [20]

    Representation Learning with Contrastive Predictive Coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  21. [21]

    Balanced multimodal learning via on-the-fly gradient modulation

    Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8238–8247,

  22. [22]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 1, 2, 4

    2021

  23. [23]

    On the memorization properties of contrastive learning.arXiv preprint arXiv:2107.10143, 2021

    Ildus Sadrtdinov, Nadezhda Chirkova, and Ekaterina Lobacheva. On the memorization properties of contrastive learning.arXiv preprint arXiv:2107.10143, 2021. 2

    work page arXiv 2021

  24. [24]

    A dataset and taxonomy for urban sound research

    Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. InProceed- ings of the 22nd ACM international conference on Multimedia, pages 1041–1044, 2014. 5

    2014

  25. [25]

    Clip models are few-shot learners: Empirical studies on vqa and visual entailment

    Haoyu Song, Li Dong, Weinan Zhang, Ting Liu, and Furu Wei. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6088–6100, 2022. 1

    2022

  26. [26]

    Localizing memorization in ssl vision encoders

    Wenhao Wang, Adam Dziedzic, Michael Backes, and Franziska Boenisch. Localizing memorization in ssl vision encoders. InProceedings of the 38th International Con- ference on Neural Information Processing Systems, pages 60475–60516, 2024. 1, 3, 6

    2024

  27. [27]

    Memorization in self-supervised learning improves down- stream generalization

    Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, and Franziska Boenisch. Memorization in self-supervised learning improves down- stream generalization. InICLR, 2024. 1, 2, 3, 4, 6, 7

    2024

  28. [28]

    Captured by captions: On memorization and its mitigation in clip models

    Wenhao Wang, Adam Dziedzic, Grace C Kim, Michael Backes, and Franziska Boenisch. Captured by captions: On memorization and its mitigation in clip models. InThe Thir- teenth International Conference on Learning Representations,

  29. [29]

    1, 2, 3, 5, 6, 9, 10 11

  30. [30]

    Vqa-gnn: Reasoning with multi- modal knowledge via graph neural networks for visual ques- tion answering

    Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, and Jure Leskovec. Vqa-gnn: Reasoning with multi- modal knowledge via graph neural networks for visual ques- tion answering. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 21582–21592,

  31. [31]

    Cookie: Contrastive cross-modal knowledge sharing pre-training for vision-language representation

    Keyu Wen, Jin Xia, Yuanyuan Huang, Linyang Li, Jiayan Xu, and Jie Shao. Cookie: Contrastive cross-modal knowledge sharing pre-training for vision-language representation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2208–2217, 2021. 1

    2021

  32. [32]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6787–6800, 2021. 2, 4

    2021

  33. [33]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 5

    2016

  34. [34]

    Zero-shot video question answering via frozen bidirectional language models.Advances in Neural Information Processing Systems, 35:124–141, 2022

    Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models.Advances in Neural Information Processing Systems, 35:124–141, 2022. 1

    2022

  35. [35]

    Leave-one-out distinguishability in machine learn- ing

    Jiayuan Ye, Anastasia Borovykh, Soufiane Hayou, and Reza Shokri. Leave-one-out distinguishability in machine learn- ing. InThe Twelfth International Conference on Learning Representations, 2023. 1

    2023

  36. [36]

    Openvna: A framework for analyzing the behavior of multimodal language understanding system under noisy scenarios

    Ziqi Yuan, Baozheng Zhang, Hua Xu, Zhiyun Liang, and Kai Gao. Openvna: A framework for analyzing the behavior of multimodal language understanding system under noisy scenarios. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 9–18, 2024. 6, 8, 9

    2024

  37. [37]

    Tip- adapter: Training-free adaption of clip for few-shot classifi- cation

    Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kun- chang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip- adapter: Training-free adaption of clip for few-shot classifi- cation. InEuropean conference on computer vision, pages 493–510. Springer, 2022. 1

    2022

  38. [38]

    Style-aware contrastive learning for multi-style image captioning

    Y Zhou and G Long. Style-aware contrastive learning for multi-style image captioning. InEACL 2023-17th Conference of the European Chapter of the Association for Computational Linguistics, Findings of EACL 2023, 2023. 1 12 A. Appendix A.1. Hardware Usage & Calculation Overhead Two devices are used for all experiments mentioned in this paper: a cloud server...

    2023

  39. [39]

    The results are shown in Figure 7, which aligns with the results for AudioCLIP in the main paper

    bi-modal MultiMem of all modality pairs (A-V , A-T, and V-T). The results are shown in Figure 7, which aligns with the results for AudioCLIP in the main paper. For A VIT- CLIP, we also measure the memorization with two metrics:

  40. [40]

    a man talking about hydroponic fluid pres- sure

    quad-modal MultiMem with all four modalities and 2) tri- modal MultiMem for VideoCLIP and A VT-CLIP (i.e.,A VT MultiMem and AIT MultiMem). The results are shown in Figure 8. We can find that when A VT-CLIP or AudioCLIP is extended to a quad-modal setting, the previously used tri-modal MultiMem becomes insufficient in capturing both the distribution of mem...

    2022