The Double-Ellipsoid Geometry of CLIP

Guy Gilboa; Meir Yossef Levi

REVIEW 3 major objections 5 minor 3 cited by

The Double-Ellipsoid Geometry of CLIP

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CLIP's raw embeddings of images and text form two linearly separable, tilted ellipsoid shells shifted from the origin, and that offset is what lets the model balance alignment against uniformity and blur uncertain concepts.

desk verdict The empirical double-ellipsoid geometry of CLIP is real and worth knowing, but the conformity proof in Supp. C.1 has a slope inversion and the key validation is in-sample, so treat the theory as a plausible hypothesis rather than a proven theorem. read the letter →

arxiv 2411.14517 v3 pith:JELNO6UH submitted 2024-11-21 cs.CV

Meir Yossef Levi , Guy Gilboa This is my paper · ORCID

classification cs.CV

keywords CLIPcontrastivelearningembeddinggeometryellipsoidshellthinmodalitygapconformityfalsenegatives

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines what CLIP's embeddings look like before the usual L2 normalization and argues that the seemingly arbitrary geometry is doing real work. It claims that image and text vectors each lie on a tilted ellipsoid shell, with the two shells linearly separable and shifted away from the origin. That shift is what lets the training loss balance alignment of correct pairs against uniformity of negatives, and it gives CLIP a way to soften its response for frequent, easily confused concepts. To make this quantitative, the paper defines conformity, the average cosine similarity of an instance to every other instance, and shows that it is proportional to the cosine similarity to the modality mean. If true, the geometry explains the modality gap and narrow cone effects as consequences of optimizing contrastive loss, not as artifacts.

What carries the argument

The load-bearing object is the double-ellipsoid geometry of the raw embedding, with each modality having its own mean vector $m_i$, $m_t$, a shell radius $R$, and a tilt from feature correlations. The argument runs through the thin-shell property (mass concentrated near a shell), linear separability of the two modality clouds, the conformity identity $C(v)\approx A\cos(v,m)$ stated as Proposition 1 in the supplement, and an experiment that varies the offset $\alpha$ and shows the full CLIP loss and the KL divergence between image and text conformity distributions are both optimized near the learned offset. The mechanism is that an offset shell creates a gradient of sharpness, so uncertain, frequently co-occurring concepts can be blurred toward the mean, reducing false-negative damage without hurting confident instances.

What would settle it

Take a different CLIP-style contrastive model or a held-out split of the original training distribution and compute the Pearson correlation between true conformity $C(v)$ and estimated conformity $a\cos(m,v)+b$; if the correlation drops well below the reported 0.9998, or if the KL-divergence minimum between image and text conformity distributions moves away from the learned offset, the central geometric claim fails. A second check is whether concept frequency in the training set actually correlates with cosine similarity to the modality mean as the paper predicts.

Watch

Extended reading notes

Core claim

The central discovery is a geometric description of the pre-normalized CLIP latent space: images occupy one ellipsoid shell, text another, and the two are linearly separable, tilted because features are correlated, and shifted from the origin. The paper argues this structure is not accidental. Shifting the shells away from the origin lets the model trade alignment against uniformity in the contrastive loss, and it makes cosine similarity to the mean behave like a measure of how common an instance is: frequent concepts, which generate more false negatives and higher uncertainty, are embedded closer to the modality mean. The paper introduces conformity $C(v)=E[\cos(v,v_k)]$ and shows $C(v)\approx a\cos(m,v)+b$ with near-perfect correlation on MS-COCO, and shows that the alignment of conformity distributions between modalities is optimized at the offset CLIP actually learned. In short, the paper claims the modality gap and the narrow cone are not bugs but features of the learned geometry.

Load-bearing premise

The proof that conformity equals a linear function of cosine similarity to the mean assumes embeddings lie on an extremely thin shell with deviations orthogonal to the mean, yet the paper's own whitening numbers show the shell is 8 to 9 times wider than a chi distribution, so that proportionality is empirically established on MS-COCO but not as a general proven fact.

Editorial extensions

If this is right

If the geometry claim holds, the modality gap and narrow cone observed on the unit sphere are direct consequences of the pre-normalization ellipsoids, so analyses on the hypersphere alone miss the mechanism.
Conformity can be estimated in constant time per instance by cosine similarity to the modality mean, enabling cheap diversity and expressiveness scoring for image generators and captioning models.
The loss experiment predicts that moving the ellipsoid offset away from its learned value should systematically worsen either alignment for correct pairs or uniformity for misclassified pairs.
The proposed vSLERP interpolation should let downstream editing preserve the same object identity because interpolation is performed around the shifted ellipsoid center.
The learned offset should align the conformity distributions of image and text, so deviating from that offset should increase the KL divergence between those distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: if concept frequency is what sets the offset, then a controlled experiment with a dataset where image-caption pairs are artificially corrupted to increase false negatives should measurably increase the learned modality offset; this test is not run in the paper.
Editorial extension: the thin-shell proof's assumptions are not strictly met, since the paper's own whitening analysis reports a shell 8 to 9 times wider than a chi distribution, so the conformity proportionality is an empirical fact on MS-COCO rather than a fully general proven result.
Editorial extension: the same conformity-to-mean logic could be applied to other contrastive vision-language models to see whether their raw spaces are also double-ellipsoid; the paper only examines two CLIP encoder variants.
Editorial extension: the semantic-blur mechanism suggests a testable prediction that deliberately removing false negatives from training data should make the ellipsoid offset shrink, a prediction that could be checked by retraining or fine-tuning.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real news is the empirical geometry: on MS-COCO, CLIP's pre-normalized image and text embeddings form two linearly separable, tilted ellipsoid shells offset from the origin, and the paper documents that carefully. That is the part worth citing. The conformity surrogate—average cosine to all other vectors approximated by cosine to the modality mean—is genuinely useful and the correlation of 0.9998 is striking on the tested set. But the proof in Supp. C.1 is not a proof. Proposition 1 defines A as sqrt(mu^2+R^2)/mu (about 1.41) yet the derivation produces A = mu/sqrt(mu^2+R^2) (about 0.71). The authors notice the mismatch and say the derived quantity is A^-1, which matches the empirical slope, but that is an acknowledgment that the proposition as stated is internally inconsistent. The thin-shell assumption is also shaky: their own whitening analysis shows the shell is 8-9 times wider than a chi distribution. So the conformity relation is an empirical fact on MS-COCO, not a general result under the stated assumptions.

The other soft spot is the in-sample nature of the key validation. The 0.9998 correlation is computed on the same set that defines both conformity and the mean, so it is internal consistency, not a held-out test. The claim that 'common themes' sit near the mean is supported by qualitative examples rather than by a quantitative frequency analysis. The loss experiment and modality-gap matching are post-hoc examinations of a trained model, not tests of the proposed mechanism. None of these are fatal; they are limitations that could be addressed by a held-out evaluation and a corrected proposition.

What the paper does well is the careful data-driven characterization: the histograms, the covariance analysis, the two-feature linear separability, and the thin-shell plots are convincing and reproducible. The vSLERP interpolation trick is minor but a nice practical byproduct.

This paper deserves a serious referee. The empirical discovery is solid and the flaws are fixable in revision. I would ask for a corrected proof or an explicit reframing of the conformity relation as empirical, plus held-out verification of the surrogate on another dataset or model. I would cite it for the geometry.

Referee Report

3 major / 5 minor

Summary. The paper analyzes the raw (pre-L2-normalization) CLIP embedding space and reports that image and text embeddings lie on linearly separable, tilted ellipsoid shells that are shifted away from the origin. It argues that this geometry is beneficial under the CLIP contrastive loss because it allows the network to place uncertain, frequent concepts closer to the modality mean, thereby mitigating false negatives. The paper introduces a notion of 'conformity' (average cosine similarity to all other instances) and proposes a fast surrogate based on cosine similarity to the mean vector; a proof of this proportionality is given in the supplementary material under a thin-shell assumption. It further connects the modality gap to matching the conformity distributions of images and text, and demonstrates applications in ranking generative models and a new interpolation scheme (vSLERP). The empirical geometry is established mainly on MS-COCO with ViT-B/32 and supplemented by ViT-L/14 results.

Significance. If the findings hold, the paper provides a useful structural description of CLIP's latent space that goes beyond the standard unit-sphere view, and it offers a practical quantity (conformity) with potential utility for evaluating generative diversity and for interpolation. The empirical evidence for the double-ellipsoid geometry is strong: the paper reports thin-shell norm distributions, anisotropic and correlated features, separable modalities, and loss and KL-divergence experiments that consistently favor non-origin-centered ellipsoids. The proposed conformity surrogate has an extremely high empirical correlation (0.9998) on MS-COCO. However, the theoretical derivation of this surrogate is internally inconsistent (the derived constant is the reciprocal of the fitted slope), and the paper's own whitening analysis shows that the thin-shell assumption used in that derivation is not quantitatively satisfied. The generality of the conformity relation is therefore not established beyond the single dataset and model where it was fitted.

major comments (3)

[Supp. C.1, Eq. (13) and Proposition 1] The proof of Proposition 1 is internally inconsistent: the text states that A ≈ √(μ²+R²)/μ, but the derivation obtains A = μ/√(R²+μ²) ≈ 0.707 for the reported MS-COCO values, which is the reciprocal of the fitted slope a ≈ 1.41 in Fig. 9. The paper acknowledges that the derived quantity is A^{-1}, but as written the proposition asserts C(v_i) ≈ A · cos(v_i, v̄) with the wrong value of A. This is a load-bearing error because the abstract and Section 6.1 claim that the surrogate is 'proved' under the thin-shell assumption, whereas the proof as stated does not establish the claimed proportionality with the correct constant.
[Supp. Sec. 2.1 (Claim 1 and whitened CLIP)] The proof of Proposition 1 relies on the thin-shell approximation ∥v_j∥ ≈ √(R²+μ²) and approximate orthogonality between z_j and v̄. Yet the paper's own whitening analysis reports std(∥y∥) ≈ 4.04–4.64 for n = 512, which is 8–9 times the chi-distribution value of about 0.5, and the authors explicitly state that 'the shell is not that thin, and certainly not as thin as chi distribution.' For the original (non-whitened) embeddings, the ratio var(y)/μ²_norm is small (0.1914/57.57), so the norm is concentrated on a relative scale, but the proof also needs ∥v_j∥ to be nearly constant in the denominator of the cosine; the intermediate-scale fluctuations are exactly what determine the constant A. Thus the proportionality between conformity and cosine-to-mean is an empirical observation on MS-COCO, not a consequence of the stated assumptions. The authors should either revise the proof to account for the actual norm variability or explicitly present Eq. (13) as an empirical relation.
[Section 6.1 (Fig. 9) and Prediction 1] The Pearson correlation of 0.9998 is measured between C and Ĉ on the same set S that defines both the empirical conformity and the mean m used in Ĉ. This makes the high correlation an internal consistency property of the embedding on MS-COCO, not a test of whether conformity tracks concept frequency or whether the surrogate generalizes to other datasets, models, or distributions. Since Prediction 1 is validated only through this same-set correlation, and since Sections 6.2 and 7 use the surrogate to draw conclusions about the modality gap and to rank generators, the absence of a held-out or cross-model check is a substantive gap. A simple split-half or cross-dataset evaluation would substantially strengthen the claim.

minor comments (5)

[Section 6.1] The sentence 'validating with close to perfect alignment with the rigorous mathematic derivation' is misleading given that the derived constant is the reciprocal of the fitted slope; please correct the wording to reflect the actual relationship between the derived and fitted quantities.
[Supplementary material (general)] The supplementary material contains several editorial artifacts: the 'Anonymous Authors' header, a dangling 'AUTHORERR: Missing \icmlcorrespondingauthor', repeated figure-caption text such as 'There is no image here to provide a caption for', and typos such as 'Enlraged'. These should be cleaned up before publication.
[Section 5, Eq. (9) and Fig. 6] The loss experiment shifts the image embeddings by -α·m_i but does not explicitly state whether the text embeddings or the normalization are recomputed after the shift; please clarify the exact procedure, since the normalization step affects the reported loss values.
[Section 6, Fig. 7] The Gaussian-sphere toy experiment is illustrative but does not model the actual CLIP covariance structure; consider adding a sentence noting that the effect is shown qualitatively and that the real-embedding loss experiments in Fig. 6 provide the quantitative support.
[Section 7.2, Eq. (12)] The vSLERP equation uses α both in the interpolation offset and to denote the interpolation magnitude; the notation is clear from the figure but could be disambiguated to avoid confusion with the α used in the loss experiments.

Circularity Check

1 steps flagged · score 6.0 of 10

Conformity surrogate's theoretical constant is inverted to match the fitted slope; core ellipsoid measurements remain independent.

fitted input called prediction [Supplementary C.1, Proposition 1 / Eq. (13); main text Sec. 6.1, Definition 2 and Fig. 9]
"Thus, the scalar factor A in Equation (13) is given by: A = µnorm/∥vj∥ ≈ µnorm/sqrt(R²+µ²norm). Empirically we know for Vit-B/32 that µnorm = 7.587 and R ≈ 7.59, thus the mathematical derivation state that A^{-1}=sqrt(7.59²+7.587²)/7.587 = 1.414 For images and A^{-1}=sqrt(5.59²+5.75²)/5.75 = 1.4, very close to the empirical observations (note that the correlation is reversed in the main paper)."

Proposition 1's proof derives the proportionality constant as μ/√(R²+μ²) ≈ 0.707, while Eq. (13) states A ≈ √(R²+μ²)/μ ≈ 1.414. The paper then matches the empirically fitted slope a=1.411/1.461 by saying the derivation gives A^{-1}=1.414. So the validated constant is the reciprocal of the derived constant; the theory is inverted to agree with the regression fit. Since Definition 2 sets a,b as 'scalars determined by the embedding' and the 0.9998 correlation is computed on the same MS-COCO set used for that fit, the agreement is an in-sample goodness-of-fit, not an independent prediction of the slope. The linear form is plausible, but the derivation does not predict the fitted slope.

full rationale

The double-ellipsoid geometry claims (Properties 1-6) are direct, self-contained measurements on the MS-COCO embeddings, and the thin-shell background is cited from external mathematics (Klartag, Paouris), so those parts are not circular. No load-bearing self-citation is present. The circularity is concentrated in the conformity-surrogate chain: Proposition 1 in Supp. C.1 proves a slope of about 0.707 but states Eq. (13) with the reciprocal constant, and the main text's fitted slope a≈1.41 is declared to match the reciprocal A^{-1}. Thus the 'first-principles' confirmation of the surrogate reduces to inverting the derived constant to fit the regression slope, and the near-perfect correlation is measured on the same set used to determine a and b. This is partial circularity in the conformity-estimation claim, while the geometric observations themselves retain independent empirical content. The A/A^{-1} inversion also marks a correctness weakness in the proof, but the circularity-specific issue is the post-hoc matching of a fitted parameter.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The paper's central calculations introduce two fitted scalars (a, b) for estimated conformity; the proof of proportionality relies on a thin-shell assumption that is only roughly satisfied; and the causal stories (false negatives drive uncertainty; the gap exists to match conformity distributions) are assumed rather than derived from training data.

free parameters (2)

a (estimated conformity slope) = 1.461 (image), 1.411 (text)
Slope in Eq. 11, fitted by linear regression to match estimated conformity to true conformity on MS-COCO; used for all conformity-based claims.
b (estimated conformity intercept) = -0.002 (image), -0.008 (text)
Intercept in Eq. 11, fitted together with a; without it the affine relation between conformity and cosine-to-mean is incomplete.

assumptions (4)

standard math Thin shell concentration: most mass of an isotropic log-concave vector lies near radius sqrt(n).
Used in Sec. 3.2 and Supp. C.1 to justify norm concentration and to derive the conformity estimator.
domain assumption MS-COCO validation set is representative of CLIP's training distribution.
All statistics (means, variances, loss curves, conformity) are computed on 5k MS-COCO validation pairs; stated in Supp. section 2.1 as 'assuming MS-COCO represents well image and text statistics'.
ad hoc to paper False negatives are more frequent for common concepts and induce greater embedding uncertainty.
Introduced in Sec. 6 to motivate the semantic blur mechanism and Prediction 1; no dataset statistics of false-negative rates are provided.
ad hoc to paper The learned embedding offset is the minimizer of the conformity-distribution KL divergence.
Sec. 6.2 and Fig. 11 show KL is minimized near alpha=0 only for the already-trained CLIP; no training-from-scratch or causal evidence is given.

invented entities (1)

Conformity C(v)
purpose: Score how typical/common an instance is by its average cosine similarity to all other instances in a set.
Defined in Eq. 10; its meaning as a measure of concept frequency is supported by qualitative examples (Figs. 8, 19, 20) and by strong internal correlation with cosine-to-mean, but there is no independent external benchmark establishing that it measures commonness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Double-Ellipsoid Geometry of CLIP." pith.science (2026). https://pith.science/paper/JELNO6UH

@misc{pith2026241114517,
  author       = {Pith},
  title        = {Pith review of: The Double-Ellipsoid Geometry of CLIP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JELNO6UH}},
  note         = {Machine review of arXiv:2411.14517}
}

read the original abstract

Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning applications within a large variety of domains. We investigate the geometry of this embedding, which is still not well understood. We examine the raw unnormalized embedding and show that text and image reside on linearly separable ellipsoid shells, not centered at the origin. We explain the benefits of having this structure, allowing to better embed instances according to their uncertainty during contrastive training. Frequent concepts in the dataset yield more false negatives, inducing greater uncertainty. A new notion of conformity is introduced, which measures the average cosine similarity of an instance to any other instance within a representative data set. We show this measure can be accurately estimated by simply computing the cosine similarity to the modality mean vector. Furthermore, we find that CLIP's modality gap optimizes the matching of the conformity distributions of image and text.

Figures

Figures reproduced from arXiv: 2411.14517 by the authors.

**Figure 1.** Sketch of CLIP general geometry: image and text are embedded on linearly separable ellipsoid shells, not centered at the origin. This allows to control uncertainty in contrastive learning, where as themes become more rare (lower uncertainty) they reside farther from the mean modality vector. 2022). Despite these advances, the structure of CLIP’s latent space remains poorly understood. Existing studies focus on prope… view at source ↗

**Figure 2.** Normalized histograms of certain CLIP features. Image and text are clearly drawn from different statistics. On the right it is shown that even two features are sufficient to obtain full linear separability. The results of a linear SVM classifier are shown (blue dashed line, with 100% accuracy on MS-COCO). 4. Geometric Analysis We begin by examining the statistics of image and text in the CLIP embedding space X . Thi… view at source ↗

**Figure 3.** Separability of features (left) and 10 most significant features ℓ for image and text, with high absolute mean, compared to the feature’s standard deviation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (31 more)

**Figure 4.** Figure 4: Statistics of image and text features after mean subtraction. Top: The first 10 features for image (top) and text (bottom). Bottom: Histograms of ∥v˜∥ for images and text, showing a thinshell phenomenon with no volume below a threshold, typical for high dimensions. t…

**Figure 6.** Figure 6: Loss vs. embedding center position. The parameter α controls the embedding center (Eq. 9, with α = 0 as the current non-origin-centered CLIP position). (Top). The unified loss balances uniformity and alignment optimally for non-origincentered positions. (Bottom). The …

**Figure 8.** Figure 8: High and low conformity of MS-COCO. Lowconformity images often depict unique, distinguishable individuals or objects, whereas high-conformity images capture common scenes that could be found anywhere. the modality gap. Blur through a non-origin centered sphere [PITH…

**Figure 7.** Figure 7: Top: Example of segmentation score blur (right), common in semantic segmentation, as object-membership uncertainty increases. Bottom: Similarity histograms of normally distributed samples for the mean vector (blue) and the furthest vector from the mean (”extreme”, ora…

**Figure 11.** Figure 11: Modality Gap matches conformity distributions. The parameter α controls the embedding offset from the origin (as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]

**Figure 10.** Figure 10: Conformity Differences. The conformity distributions of text and image modalities differ, as a common image may be described by a unique caption, and vice versa. To provide more intuition, we present examples of high and low conformity from MS-COCO in [PITH_FULL_IMAG…

**Figure 12.** Figure 12: Vertical SLERP (vSLERP) enables optimization-free, semantic editing. Interpolated images preserve the object with pose variations and roughly maintain backgrounds, with interpolation magnitude controlled by α. Caption Reward a large green and grey passenger train driv…

**Figure 13.** Figure 13: Conformity analysis of captioning and image synthesis. Image Synthesis (top): Glide generates more common images with less fine detail, while unCLIP creates more detailed images closer to natural distributions. Captioning (bottom): ClipCap produces more common capti…

**Figure 14.** Figure 14: Norm distribution. While norm magnitudes are disregarded during training due to the normalization inherent in cosine similarity, they still capture meaningful semantic information. We give below additional analysis related to applying a linear transformation that turn…

**Figure 15.** Figure 15: Enlarged plots from Section 4. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]

**Figure 16.** Figure 16: Enlarged plots from Section 4. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]

**Figure 17.** Figure 17: Enlarged plots for CLIP embedding of n = 768. There are dominant features with clearly different distribution between image and text. Both modalities can be separated (with perfect accuracy) by a linear SVM classifier based on only 2 features. With respect to separabi…

**Figure 18.** Figure 18: CLIP n = 768, thin shell phenomenon. We can observe similar geometry (as in the case of n = 512) of two tilted ellipsoids, one for each modality, not centered at the origin. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]

**Figure 19.** Figure 19: High and low conformity of sketches from ImageNet-R. Images with high conformity tend to be simpler and cleaner, while low-conformity images often feature complex details covered by large portions of text descriptions. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]

**Figure 20.** Figure 20: Conformity on ImageNet-a. It is possible that high conformity images are with more unique colors, perhaps contains people or text, whereas low conformity images tends to contain low amount of information. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_20.png]

**Figure 21.** Figure 21: Shifting text ellipsoid only. Conformity distribution matching and loss experiments when shifting text ellipsoid only as in Equation (14) 𝜶 𝑲𝑳 𝑫𝒊𝒗𝒆𝒓𝒈𝒆𝒏𝒄𝒆 𝜶 𝑳𝒐𝒔𝒔𝑴𝒊𝒔𝒄𝒍𝒂𝒔𝒔𝒊𝒇𝒊𝒆𝒅 𝑩𝒐𝒕𝒉 𝒊𝒎𝒂𝒈𝒆 𝒕𝒆𝒙𝒕 𝜶 = 𝟎 𝜶 = 𝟏 𝜶 = −𝟏 𝒐𝒓𝒊𝒈𝒊𝒏 𝜶 = 𝟎 𝜶 = −𝟏 [PITH_FULL_IMAGE:figures/full_fig_p020…

**Figure 22.** Figure 22: Shifting both ellipsoids. Conformity distribution matching and loss experiments when shifting both text and image ellipsoids as in Equation (15). Source Target Increasing 𝚯 𝜶Positive Negative [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]

**Figure 23.** Figure 23: vSLERP lamp to vase. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]

**Figure 24.** Figure 24: vSLERP Kevin Durant to Lebron James. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_24.png]

**Figure 1.** Figure 1: Top - covariance matrices of image and of text (first 100 features of each modality are shown). Both distributions are not isotropic. Bottom - histogram of norm of whitened CLIP, ∥yi∥, ∥yt∥. Here the spread of the norms is wider than the original embedding and not admi…

**Figure 2.** Figure 2: Norm distribution. While norm magnitudes are disregarded during training due to the normalization inherent in cosine similarity, they still capture meaningful semantic information. be well modeled by m(y) := E[∥y∥] ≈ √ n = ˆm(y), (1) s(y) := std(∥y∥) = p n − m2(y) = ˆs…

**Figure 3.** Figure 3: Enlarged plots from Section 4. 3 [PITH_FULL_IMAGE:figures/full_fig_p024_3.png]

**Figure 4.** Figure 4: Enlarged plots from Section 4. 4 [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]

**Figure 5.** Figure 5: Enlarged plots for CLIP embedding of n = 768. There are dominant features with clearly different distribution between image and text. Both modalities can be separated (with perfect accuracy) by a linear SVM classifier based on only 2 features. With respect to separabil…

**Figure 6.** Figure 6: CLIP n = 768, thin shell phenomenon. We can observe similar geometry (as in the case of n = 512) of two tilted ellipsoids, one for each modality, not centered at the origin. 6 [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]

**Figure 7.** Figure 7: High and low conformity of sketches from ImageNetR. Images with high conformity tend to be simpler and cleaner, while low-conformity images often feature complex details covered by large portions of text descriptions. 7 [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]

**Figure 8.** Figure 8: Conformity on ImageNet-a. It is possible that high conformity images are with more unique colors, perhaps contains people or text, whereas low conformity images tends to contain low amount of information. 8 [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]

**Figure 9.** Figure 9: Shifting text ellipsoid only. Conformity distribution matching and loss experiments when shifting text ellipsoid only as in Equation (3) 𝜶 𝑲𝑳 𝑫𝒊𝒗𝒆𝒓𝒈𝒆𝒏𝒄𝒆 𝜶 𝑳𝒐𝒔𝒔𝑴𝒊𝒔𝒄𝒍𝒂𝒔𝒔𝒊𝒇𝒊𝒆𝒅 𝑩𝒐𝒕𝒉 𝒊𝒎𝒂𝒈𝒆 𝒕𝒆𝒙𝒕 𝜶 = 𝟎 𝜶 = 𝟏 𝜶 = −𝟏 𝒐𝒓𝒊𝒈𝒊𝒏 𝜶 = 𝟎 𝜶 = −𝟏 [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]

**Figure 10.** Figure 10: Shifting both ellipsoids. Conformity distribution matching and loss experiments when shifting both text and image ellipsoids as in Equation (4). Source Target Increasing 𝚯 𝜶Positive Negative [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]

**Figure 11.** Figure 11: vSLERP lamp to vase. 9 [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]

**Figure 12.** Figure 12: vSLERP Kevin Durant to Lebron James. 10 [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

On the modality gap and the contrastive loss in multi-modal representation learning
cs.LG 2026-07 conditional novelty 6.5 of 10

InfoNCE with independent encoders actively creates a modality gap at low temperature; mixing intra- and inter-modality negatives (xNCE) removes the gap while improving zero-shot transfer.
The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model
cs.LG 2026-07 conditional novelty 5.0 of 10

CLIP embeddings are modeled as a mixture of von Mises-Fisher distributions on the unit sphere, improving out-of-distribution detection and semantic decomposition over single-Gaussian baselines.
On the rankability of visual embeddings
cs.CV 2025-07 conditional novelty 5.0 of 10

Visual embeddings from CLIP and other vision encoders encode ordinal attributes along linear directions, recoverable from as few as two extreme reference images, without full supervision.

Reference graph

Works this paper leans on

59 extracted references · 34 canonical work pages · cited by 3 Pith papers

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding

Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9902--9912, 2022

work page 2022
[3]

A theoretical analysis of contrastive unsupervised representation learning

Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 , 2019

arXiv 1902
[4]

Grit-vlp: Grouped mini-batch sampling for efficient vision and language pre-training

Jaeseok Byun, Taebaek Hwang, Jianlong Fu, and Taesup Moon. Grit-vlp: Grouped mini-batch sampling for efficient vision and language pre-training. In European Conference on Computer Vision , pages 395--412. Springer, 2022

work page 2022
[5]

Mafa: Managing false negatives for vision-language pre-training

Jaeseok Byun, Dohoon Kim, and Taesup Moon. Mafa: Managing false negatives for vision-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 27314--27324, 2024

work page 2024
[6]

Clip2scene: Towards label-efficient 3d scene understanding by clip

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7020--7030, 2023

work page 2023
[7]

A simple framework for contrastive learning of visual representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597--1607. PMLR, 2020

2020
[8]

Fine-grained image captioning with clip reward

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward. arXiv preprint arXiv:2205.13115 , 2022

arXiv 2022

Show all 59 references

[9]

Debiased contrastive learning

Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in neural information processing systems , 33:8765--8775, 2020

2020
[10]

Eccv caption: Correcting false negatives by collecting machine-and-human-verified image-caption associations for ms-coco

Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang, and Seong Joon Oh. Eccv caption: Correcting false negatives by collecting machine-and-human-verified image-caption associations for ms-coco. In European Conference on Computer Vision , pages 1--19. Springer, 2022

2022
[11]

Eldar and Alan V

Yonina C. Eldar and Alan V. Oppenheim. Mmse whitening and subspace whitening. IEEE Transactions on Information Theory , 49(7):1846--1851, 2003

2003
[12]

Its not a modality gap: Characterizing and addressing the contrastive gap

Abrar Fahim, Alex Murphy, and Alona Fyshe. Its not a modality gap: Characterizing and addressing the contrastive gap. arXiv preprint arXiv:2405.18570 , 2024

2024 arXiv
[13]

Datacomp: In search of the next generation of multimodal datasets

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Syste...

2024
[14]

An image is worth one word: Personalizing text-to-image generation using textual inversion

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022

2022 arXiv
[15]

Simcse: Simple contrastive learning of sentence embeddings

Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 , 2021

2021 arXiv
[16]

Audioclip: Extending clip to image, text and audio

Andrey Guzhov, Federico Raue, J \"o rn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 976--980. IEEE, 2022

2022
[17]

Proxedit: Improving tuning-free real image editing with proximal guidance

Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuning-free real image editing with proximal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applic...

2024
[18]

Momentum contrast for unsupervised visual representation learning

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729--9738, 2020

2020
[19]

Open-vocabulary multi-label classification via multi-modal knowledge transfer

Sunan He, Taian Guo, Tao Dai, Ruizhi Qiao, Xiujun Shu, Bo Ren, and Shu-Tao Xia. Open-vocabulary multi-label classification via multi-modal knowledge transfer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 808--816, 2023

2023
[20]

Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition

Deepti Hegde, Jeya Maria Jose Valanarasu, and Vishal Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2028--2038, 2023

2023
[21]

The many faces of robustness: A critical analysis of out-of-distribution generalization

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international c...

2021
[22]

Natural adversarial examples

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15262--15271, 2021

2021
[23]

Boosting contrastive self-supervised learning with false negative cancellation

Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boosting contrastive self-supervised learning with false negative cancellation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2785--2795, 2022

2022
[24]

A slightly improved bound for the kls constant

Arun Jambulapati, Yin Tat Lee, and Santosh S Vempala. A slightly improved bound for the kls constant. arXiv preprint arXiv:2208.11644 , 2022

2022 arXiv
[25]

The power of contrast for feature learning: A theoretical analysis

Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. Journal of Machine Learning Research , 24(330):1--78, 2023

2023
[26]

Hard negative mixing for contrastive learning

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in neural information processing systems , 33:21798--21809, 2020

2020
[27]

Isoperimetric problems for convex bodies and a localization lemma

Ravi Kannan, L \'a szl \'o Lov \'a sz, and Mikl \'o s Simonovits. Isoperimetric problems for convex bodies and a localization lemma. Discrete & Computational Geometry , 13:541--559, 1995

1995
[28]

Imagic: Text-based real image editing with diffusion models

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6007--6017, 2023

2023
[29]

Optimal whitening and decorrelation

Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician , 72(4):309--314, 2018

2018
[30]

Diffusionclip: Text-guided diffusion models for robust image manipulation

Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2426--2435, 2022

2022
[31]

Self-guided contrastive learning for bert sentence representations

Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. Self-guided contrastive learning for bert sentence representations. arXiv preprint arXiv:2106.07345 , 2021

2021 arXiv
[32]

Logarithmic bounds for isoperimetry and slices of convex sets

Bo'az Klartag. Logarithmic bounds for isoperimetry and slices of convex sets. Ars Inveniendi Analytica , 4, 2023

2023
[33]

Bourgain’s slicing problem and kls isoperimetry up to polylog

Bo’az Klartag and Joseph Lehec. Bourgain’s slicing problem and kls isoperimetry up to polylog. Geometric and functional analysis , 32(5):1134--1159, 2022

2022
[34]

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning , pages 12888--12900. PMLR, 2022

2022
[35]

Open-vocabulary semantic segmentation with mask-adapted clip

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , page...

2023
[36]

Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems , 35:17612--17625, 2022

2022
[37]

Microsoft coco: Common objects in context

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Procee...

2014
[38]

Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing , 508:293--304, 2022

2022
[39]

T-mars: Improving visual representations by circumventing text feature learning

Pratyush Maini, Sachin Goyal, Zachary C Lipton, J Zico Kolter, and Aditi Raghunathan. T-mars: Improving visual representations by circumventing text feature learning. arXiv preprint arXiv:2307.03132 , 2023

2023 arXiv
[40]

Null-text inversion for editing real images using guided diffusion models

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6038--6047, 2023

2023
[41]

Clipcap: Clip prefix for image captioning

Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 , 2021

2021 arXiv
[42]

Glide: Towards photorealistic image generation and editing with text-guided diffusion models

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021

2021 arXiv
[43]

Representation learning with contrastive predictive coding

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018

2018 arXiv
[44]

Concentration of mass on convex bodies

Grigoris Paouris. Concentration of mass on convex bodies. Geometric & Functional Analysis GAFA , 16(5):1021--1049, 2006

2006
[45]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

2021
[46]

Hierarchical text-conditional image generation with clip latents

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022

2022 arXiv
[47]

Contrastive learning with hard negative samples

Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 , 2020

2010 arXiv
[48]

Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning

Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language representation learning. arXiv preprint arXiv:2404.07983 , 2024

2024 arXiv
[49]

Towards understanding the modality gap in clip

Peiyang Shi, Michael C Welle, M rten Bj \"o rkman, and Danica Kragic. Towards understanding the modality gap in clip. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls , 2023

2023
[50]

Clip4caption: Clip for video caption

Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. Clip4caption: Clip for video caption. In Proceedings of the 29th ACM International Conference on Multimedia , pages 4858--4862, 2021

2021
[51]

Too large; data reduction for vision-language pre-training

Alex Jinpeng Wang, Kevin Qinghong Lin, David Junhao Zhang, Stan Weixian Lei, and Mike Zheng Shou. Too large; data reduction for vision-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3147--3157, 2023

2023
[52]

Understanding contrastive representation learning through alignment and uniformity on the hypersphere

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning , pages 9929--9939. PMLR, 2020

2020
[53]

Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.13457 , 2022

2022 arXiv
[54]

Wav2clip: Learning robust audio representations from clip

Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4563--4567. IEEE, 2022

2022
[55]

Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching

Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7031--7040, 2023

2023
[56]

Pointcontrast: Unsupervised pre-training for 3d point cloud understanding

Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part III 16 , page...

2020
[57]

Vision-language pre-training with triple contrastive learning

Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-language pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1...

2022
[58]

Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems , 36, 2024

2024
[59]

Pointclip: Point cloud understanding by clip

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8552--8562, 2022

2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding

Mohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri, Kanchana Thilakarathna, and Ranga Rodrigo. Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9902--9912, 2022

work page 2022

[3] [3]

A theoretical analysis of contrastive unsupervised representation learning

Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 , 2019

arXiv 1902

[4] [4]

Grit-vlp: Grouped mini-batch sampling for efficient vision and language pre-training

Jaeseok Byun, Taebaek Hwang, Jianlong Fu, and Taesup Moon. Grit-vlp: Grouped mini-batch sampling for efficient vision and language pre-training. In European Conference on Computer Vision , pages 395--412. Springer, 2022

work page 2022

[5] [5]

Mafa: Managing false negatives for vision-language pre-training

Jaeseok Byun, Dohoon Kim, and Taesup Moon. Mafa: Managing false negatives for vision-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 27314--27324, 2024

work page 2024

[6] [6]

Clip2scene: Towards label-efficient 3d scene understanding by clip

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7020--7030, 2023

work page 2023

[7] [7]

A simple framework for contrastive learning of visual representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597--1607. PMLR, 2020

2020

[8] [8]

Fine-grained image captioning with clip reward

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward. arXiv preprint arXiv:2205.13115 , 2022

arXiv 2022