Pith. sign in

REVIEW 3 major objections 6 minor 42 references

I0T: Embedding Standardization Method Towards Zero Modality Gap

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By subtracting each modality's mean embedding and re-normalizing, this paper nearly eliminates CLIP's modality gap while preserving, and often improving, downstream performance.

desk verdict Useful analysis of CLIP's modality gap, but the headline post-hoc result is an artifact of evaluating on the same set used to estimate the standardization statistics. read the letter →

arxiv 2412.14384 v1 pith:M4FA5K7O submitted 2024-12-18 cs.LG cs.AIcs.CV

Na Min An , Eunki Kim , James Thorne , Hyunjung Shim This is my paper · ORCID
classification cs.LGcs.AIcs.CV
keywords modalitygapCLIPembeddingstandardizationimage-textretrievalbatchnormalizationreference-freeevaluationcentroiddistancecosinesimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contrastive language-image models such as CLIP place images and texts in separate regions of embedding space, a 'modality gap' that breaks the intended semantic comparison. The paper claims this gap is caused by modality-specific activation patterns, especially peak activations on a few fixed dimensions, that each encoder learns, and that removing those patterns brings the gap to near zero. Its post-hoc method, I0Tpost, subtracts the per-modality mean vector from normalized embeddings and re-normalizes, cutting centroid distance to 0.0102 and linear separability to 0.5374 on Flickr30k. The paper also proposes a trainable version, I0Tasync, that adds per-modality batch normalization layers and reaches a moderate gap without test-set access. A reader should care because a nearly closed gap makes CLIP usable as an explainable, reference-free image-text evaluation metric and clarifies what the embeddings actually encode.

What carries the argument

The central object is the 'modality-specific characteristic' in normalized embeddings: mean activation vectors that are nearly constant across samples of one modality, with peaks in fixed dimensions (for Long-CLIP, a negative peak at dimension 93 for images and positive peaks at 134 and 313 for texts). The argument-carrying mechanism is standardization of each modality's normalized embeddings by subtracting its own mean vector and renormalizing with the Frobenius norm, x'_i = Normalize(x_i - \bar{x}), y'_i = Normalize(y_i - \bar{y}). For the trainable variant the same statistics are learned by independent batch normalization layers placed after each encoder and trained asynchronously with the multimodal MCSIE objective. A supporting analytic claim shows that peak activations p and q cap the attainable cosine similarity at $\sqrt$((1-$p^{2}$)(1-$2q^{2}$)), so removing peaks and mean structure is what permits similarity scores to approach their semantic range.

What would settle it

Compute I0Tpost means on the Flickr30k training split, apply them to the Flickr30k test split, and report centroid distance on test: if CD jumps well above 0.19, the near-zero gap is an artifact of test-set statistics; alternatively, evaluate on COCO captions with means computed on Flickr30k.

Watch

Extended reading notes

Core claim

On the paper's account, CLIP's image and text encoders do not merely differ by where their data live; each encoder stamps every embedding with a characteristic activation profile, negative peaks in the same dimensions for all images and positive peaks in other dimensions for all texts, that limits how close cross-modal cosine similarity can get. The discovery is that discarding this modality-specific signature across all dimensions, not just clipping the peaks, collapses the gap. The post-hoc procedure subtracts the mean of all normalized image embeddings from each image embedding and the mean of all normalized text embeddings from each text embedding, then re-normalizes by Frobenius norm; on frozen encoders this yields centroid distance 0.0102 from 0.7642 and linear separability 0.5374 from 0.9985 on Flickr30k, while text-to-image retrieval improves to 76.30. The same idea is made trainable as I0Tasync with separate batch normalization layers per modality, reducing the gap to a moderate level without needing the evaluation distribution. The paper presents I0Tpost, under the name I0T-S, as an automatic reference-free captioning metric whose raw cosine scores are spread across a wider, interpretable range than CLIPScore.

Load-bearing premise

The near-zero centroid distance is measured with mean vectors computed from the same test set whose embeddings are standardized, so the method assumes those per-modality means are stable properties of the deployment distribution rather than a quirk of that particular test split.

Editorial extensions

If this is right

  • I0Tpost brings Flickr30k centroid distance from 0.7642 to 0.0102 and linear separability from 0.9985 to 0.5374, with text-to-image retrieval rising from 67.10 to 76.30.
  • I0Tasync reaches a moderate gap (CD 0.4795) without access to the test distribution, adding about 10M parameters, and improves T2I retrieval over the starting CLIP.
  • Using I0Tpost as I0T-S, cosine similarity between an image and its correct caption is higher and, for a wrong caption, lower than CLIP-S, without the 2.5 scaling CLIPScore needs.
  • The paper explicitly does not claim the gap and downstream performance are causally linked; the result is that the gap can be cut without sacrificing retrieval or classification.
  • The same standardization idea works on a ResNet-based CLIP (CLIPpost), suggesting the mechanism is not tied to ViT architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the per-modality mean vectors are stable, I0Tpost implies CLIP embeddings are approximately mean-shift separable; one testable extension is calibrating the means on a small unlabeled in-domain pool at deployment time and checking whether near-zero CD persists across datasets.
  • The peak-activation diagnosis is a cheap diagnostic: detecting the same peak dimensions across samples of one modality could flag modality-specific artifacts in other contrastive models, not only CLIP.
  • I0T-S's interpretable absolute scores suggest reference-free evaluation metrics could be standardized in place of scaling cosine similarities, a change that would make captioning scores comparable across domains; this is an extension beyond the paper's claims.
  • The trainable variant leaves a moderate gap, so a natural next step the paper does not take is combining async BN statistics with I0Tpost-style standardization at inference to get near-zero gap without test-set means.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes I0T, a two-stage framework to reduce the modality gap in CLIP embeddings. Stage one is an optional plug-in fine-tuning strategy based on Long-CLIP with cyclic losses and MCSIE; stage two consists of two alternatives: I0Tpost, which standardizes normalized image and text embeddings by subtracting per-modality mean vectors and re-normalizing, and I0Tasync, which adds separate batch-normalization layers trained asynchronously on frozen encoders. The authors report that I0Tpost reduces centroid distance to 0.0102 and linear separability to 0.5374 on Flickr30k while preserving or improving retrieval, and that I0Tasync reaches a moderate gap of 0.4795. They further propose I0T-S as an explainable reference-free image-captioning metric based on I0Tpost.

Significance. If the claims were established, the paper would make a useful practical contribution: a simple post-hoc transformation that removes a large part of the modality gap without retraining, plus a trainable alternative, with code released and a broad set of downstream evaluations. The connection to CLIPScore is also interesting and could lead to more interpretable evaluation scores. The paper deserves credit for releasing code, for checking both centroid distance and linear separability, and for including ablations on batch size and normalization choices. However, the central evidence for I0Tpost is currently not independent: the same test set supplies the standardization statistics and the evaluation numbers, so the near-zero centroid distance is largely a consequence of the evaluation protocol. The paper is transparent about this in its Limitations section, but transparency does not make the result supportive.

major comments (3)
  1. [Section 4.2, Section 5.2, Table 2, Section 8] The headline result for I0Tpost is circular. The per-modality mean vectors xbar and ybar are computed from the same Flickr30k test split on which centroid distance, linear separability, retrieval, and correlation are then measured. Since the centroid distance is defined as the distance between the modality means, subtracting the empirical mean of the same set and re-normalizing removes the first-order gap by construction; the reported CD of 0.0102 is therefore not independent evidence that a modality gap has been removed. The same reasoning explains the drop in linear separability. The paper acknowledges in Section 8 that I0Tpost relies on the entire test dataset, but it does not provide the necessary control experiment: estimate xbar and ybar on a disjoint training or validation split, apply them to the test split, and report CD, LS, retrieval, and metric correlation. Without such an experiment, the claim that I0Tpost reduces the modality gap to approximately zero, and the related claim that I0T-S is a usable evaluation metric on unseen captions, are unsupported. I0Tasync avoids this circularity but reaches only CD 0.4795, a moderate gap, so the paper's central 'zero gap' claim rests entirely on the circular I0Tpost result.
  2. [Appendix D.3, Table 2] The comparison with PAC-S and BLIP is confounded by different checkpoints. According to Appendix D.3, PAC-S and BLIP are evaluated using the provided checkpoints, while the I0T models are Long-CLIP-based models fine-tuned on COCO with the authors' first-stage recipe. Differences in retrieval and correlation could therefore come from the base model and training data rather than from the proposed method. A fair comparison should apply I0Tpost to the same base CLIP checkpoint and train or fine-tune all comparison methods under an identical protocol, or at least include a matching-control experiment. This is load-bearing because the paper claims a 9.2% improvement in T2I retrieval and competitiveness with PAC-S based on these tables.
  3. [Tables 1, 2, 4, 6, 7 and Figure 8] The main tables and figures report point estimates without error bars or significance tests. Many of the claimed differences are small relative to the likely run-to-run variability (for example, the relative correlation values on Expert and CF in Table 2 differ by only a few points across models). The paper states in Appendix D.1 that three random seeds were used for dataset selection, but the main results do not report mean and standard deviation over seeds. Without this, the reader cannot assess whether the reported improvements and the ranking in Table 2 are statistically reliable. Please report mean +/- std over at least three seeds for all main metrics, or clearly state if only a single run is reported.
minor comments (6)
  1. [Section 3.3 and Appendix C] The upper-bound proof assumes that all non-peak activations are uniformly distributed, an assumption that is stated but not justified empirically or theoretically. The derived limit is therefore not a general upper bound for the cosine similarity of arbitrary normalized embeddings with peak activations; the authors should clarify the scope of the claim or support the uniform assumption with activation statistics.
  2. [Table 7] The linear separability values for MG and CLIPpost in Table 7 are negative (-1.4094 and -1.8497), which is inconsistent with the definition LS = 1 - MSE if the model is trained and evaluated as described. The authors should explain how negative values arise or fix the computation.
  3. [Section 5.3 and Figure 5] The claim that I0T-S is more explainable than CLIP-S is supported mainly by anecdotal examples in Figures 1 and 5. It would be stronger to report the full distribution statistics (mean, variance, range) of the cosine similarity scores over the test set, as well as a quantitative comparison of the ordering of correct versus incorrect pairs.
  4. [Section 8] The term 'reference-free' for I0T-S is misleading in the single-sample setting: to compute the text mean vector for standardization, the method needs a set of captions from the test distribution, so it is not a purely single-sample metric. The authors should qualify this terminology and state the minimum corpus size required for stable estimates.
  5. [Table 4 and Appendix D.2] The retrieval protocol in Table 4 differs from the common practice of using all five captions for Flickr30k. The paper should clearly define the one-caption versus five-caption evaluation and report both consistently, since the choice affects the I2T/T2I balance and the comparability with prior work.
  6. [Ethical Statement] There is a typo in the Ethical Statement: 'ClIP-S' should be 'CLIP-S'.

Circularity Check

1 steps flagged · score 8.0 of 10

I0Tpost's near-zero modality gap is an artifact of computing per-modality means on the same test set used for CD/LS evaluation; the zero-gap claim reduces to the standardization operation itself.

  1. fitted input called prediction [Section 4.2 (The Second Stage of I0T) and Section 5.2 (Table 2)]
    "We standardize the normalized embedding activations (xi, yi ∈ R^d) by subtracting the mean vectors (¯x, ¯y ∈ R^d) for each modality and re-normalize them by dividing by the Frobenius norms ... x′i = Normalize(xi − ¯x), y′i = Normalize(yi − ¯y). ... I0Tpost reduces the modality gap to almost zero with significantly low CD and LS scores of 0.0102 and 0.5374."

    The subtracted vectors ¯x and ¯y are the empirical means of the same Flickr30k image/text embedding sets on which CD and LS are then reported. Centering each modality by its own mean makes the pre-normalization centroids exactly zero, so the renormalized centroid difference (CD=0.0102) is a mathematical consequence of the standardization rather than evidence that the modality gap is removed as a transferable model property. The same centering removes the dominant between-modality direction and mechanically lowers linear separability (LS=0.5374). Section 8 admits 'I0Tpost relies on the entire test dataset,' and no disjoint-split experiment shows that near-zero CD persists when the means are estimated on one set and applied to another.

full rationale

The trainable I0Tasync variant and the retrieval/classification benchmark numbers are not circular: I0Tasync estimates batch-normalization statistics from training data and is evaluated on held-out Flickr30k, and the first-stage fine-tuning is a standard empirical pipeline. However, the paper's central 'zero modality gap' claim rests on I0Tpost, whose standardization means are fitted on the exact test split whose centroid distance, linear separability, retrieval, and correlations are then measured. The paper's own limitation (Section 8: 'I0Tpost relies on the entire test dataset') confirms that the near-zero CD is an evaluation-protocol artifact. No held-out transfer experiment is provided, and I0Tasync reaches only moderate CD (0.4795), so the near-zero result cannot be independently supported. Score 8 reflects that the headline zero-gap claim reduces by construction, not that the entire paper lacks empirical content.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two data-derived statistics: the per-modality mean vectors for I0Tpost and the trained batch-norm parameters for I0Tasync. The causal story about peak activations is an additional assumption that is not strictly necessary for the standardization to work.

free parameters (2)
  • Per-modality mean vectors (xbar, ybar) for I0Tpost = d-dimensional mean of normalized Flickr30k image and text embeddings
    Subtracting these means is the core of I0Tpost; they are estimated from the same data used for evaluation, which drives the reported centroid distance to near zero.
  • Batch-normalization weights and biases for I0Tasync = 10M additional parameters trained on COCO captions
    Learned by the MCSIE contrastive objective, not derived from first principles.
assumptions (3)
  • domain assumption Normalized embeddings are the right representation for measuring semantic similarity via cosine.
    The entire method and evaluation operate on Frobenius-normalized embeddings, following standard CLIP practice.
  • ad hoc to paper Modality-specific activation patterns are stable across samples and can be summarized by a single mean vector per modality.
    The paper asserts consistent peak activations at fixed dimensions across images and texts, and assumes a single mean vector captures the modality-specific characteristic.
  • ad hoc to paper Non-peak activations are uniformly distributed in the upper-bound proof.
    Appendix C assumes uniform non-peak activations to derive the cosine upper bound; this is a simplification, not a proven property of CLIP embeddings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of I0T: Embedding Standardization Method Towards Zero Modality Gap." pith.science (2026). https://pith.science/paper/M4FA5K7O

@misc{pith2026241214384,
  author       = {Pith},
  title        = {Pith review of: I0T: Embedding Standardization Method Towards Zero Modality Gap},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M4FA5K7O}},
  note         = {Machine review of arXiv:2412.14384}
}
abstract

Contrastive Language-Image Pretraining (CLIP) enables zero-shot inference in downstream tasks such as image-text retrieval and classification. However, recent works extending CLIP suffer from the issue of modality gap, which arises when the image and text embeddings are projected to disparate manifolds, deviating from the intended objective of image-text contrastive learning. We discover that this phenomenon is linked to the modality-specific characteristic that each image/text encoder independently possesses and propose two methods to address the modality gap: (1) a post-hoc embedding standardization method, $\text{I0T}_{\text{post}}$ that reduces the modality gap approximately to zero and (2) a trainable method, $\text{I0T}_{\text{async}}$, to alleviate the modality gap problem by adding two normalization layers for each encoder. Our I0T framework can significantly reduce the modality gap while preserving the original embedding representations of trained models with their locked parameters. In practice, $\text{I0T}_{\text{post}}$ can serve as an alternative explainable automatic evaluation metric of widely used CLIPScore (CLIP-S).

Figures

Figures reproduced from arXiv: 2412.14384 by the authors.

Figure 1
Figure 1. Refined scoring system using our proposal [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Linear separability and minimum cosine distance (dashed line) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of normalized embedding activations (avg: salmon, std: gray) and modality gap across three [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of non-CLIP-based model BLIP and ours on the efficiency and performances. modality gap. We find that asynchronous train￾ing strategy (+BN*) performs better than non￾asynchronous training strategy (+BN) in terms of both modality gap and retrieval performances…
Figure 6
Figure 6. Figure 6: The existence of positive and negative peak activations in normalized embedding activations (pointed by [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Sample image-caption pairs and corresponding human scores from Flickr8k-Expert dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Effect of varying batch sizes on centroid distance, retrieval, and correlation performances of I0T [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 17 canonical work pages

  1. [1]

    Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. 2014. Birdsnap: Large-scale fine-grained visual categorization of birds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2011--2018

    work page 2014

  2. [2]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023. Sharegpt4v: Improving large multi-modal models with better captions. CoRR

    work page 2023

  3. [3]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations

    2020

  4. [4]

    Sedigheh Eslami and Gerard de Melo. 2024. Mitigate the gap: Investigating approaches for improving cross-modal alignment in clip. arXiv preprint arXiv:2406.17639

    arXiv 2024

  5. [5]

    Abrar Fahim, Alex Murphy, and Alona Fyshe. 2024. Its not a modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405.18570

    arXiv 2024

  6. [6]

    u rst, Elisabeth Rumetshofer, Johannes Lehner, Viet T Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp, G \

    Andreas F \"u rst, Elisabeth Rumetshofer, Johannes Lehner, Viet T Tran, Fei Tang, Hubert Ramsauer, David Kreil, Michael Kopp, G \"u nter Klambauer, Angela Bitto, et al. 2022. Cloob: Modern hopfield networks with infoloob outperform clip. Advances in neural information processing systems, 35:20450--20468

    work page 2022

  7. [7]

    Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. 2024. Softclip: Softer cross-modal alignment makes clip stronger. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1860--1868

    work page 2024

  8. [8]

    Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. 2022. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704--6719

    work page 2022

Show all 42 references
  1. [9]

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514--7528

    2021

  2. [10]

    Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853--899

    2013

  3. [11]

    Anwen Hu, Shizhe Chen, Liang Zhang, and Qin Jin. 2023. Infometic: An informative metric for reference-free image caption evaluation. In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  4. [12]

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. https://openreview.net/forum?id=BH8TYy0r6u Position: The platonic representation hypothesis . In Forty-first International Conference on Machine Learning

    2024

  5. [13]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904...

    2021

  6. [14]

    Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. 2023. Understanding and constructing latent modality structures in multi-modal representation learning. In Proceedings of the IEEE/CVF Conference on Computer V...

    2023

  7. [15]

    Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128--3137

    2015

  8. [16]

    Alex Krizhevsky et al. 2009. Learning multiple layers of features from tiny images. Technical report

    2009

  9. [17]

    Yebin Lee, Imseong Park, and Myungjoo Kang. 2024. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.06004

    2024 arXiv

  10. [18]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

    2022

  11. [19]

    Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. 2023. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390--23400

    2023

  12. [20]

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612--17625

    2022

  13. [21]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  14. [22]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  15. [23]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  16. [24]

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Gro berger. 2018. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29)

    2018

  17. [25]

    Changdae Oh, Junhyuk So, Hoyoon Byun, YongTaek Lim, Minchul Shin, Jong-June Jeon, and Kyungwoo Song. 2024. Geodesic multi-modal mixup for robust fine-tuning. Advances in Neural Information Processing Systems, 36

    2024

  18. [26]

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer ...

    2015

  19. [27]

    Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. 2019. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171--5180. PMLR

    2019

  20. [28]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  21. [29]

    Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Ajanthan Thalaiyasingam. 2024. Accept the modality gap: An exploration in the hyperbolic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27263--27272

    2024

  22. [30]

    a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, G \

    Hubert Ramsauer, Bernhard Sch \"a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, G \"u nter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. 2021. https://openreview.net/forum?id=tL89RnzIi...

    2021

  23. [31]

    Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2023. Positive-augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6914--6924

    2023

  24. [32]

    Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. 2024. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. arXiv preprint arXiv:2404.07983

    2024 arXiv

  25. [33]

    Peiyang Shi, Michael C Welle, M rten Bj \"o rkman, and Danica Kragic. 2023. Towards understanding the modality gap in clip. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls

    2023

  26. [34]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  27. [35]

    Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pages 9929--9939. PMLR

    2020

  28. [36]

    Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. 2024. Achieving cross modal generalization with multimodal unified representation. Advances in Neural Information Processing Systems, 36

    2024

  29. [37]

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Nat...

    2021

  30. [38]

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403.15378

    2024 arXiv

  31. [39]

    Miaoran Zhang, Marius Mosbach, David Adelani, Michael Hedderich, and Dietrich Klakow. 2022 a . Mcse: Multimodal contrastive learning of sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2022

  32. [40]

    Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. 2022 b . Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2--25. PMLR

    2022

  33. [41]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.