Pith. sign in

REVIEW 4 major objections 6 minor 37 references

CLIP's embedding space is a hyperspherical semantic mixture, and modeling it that way improves long-tailed and out-of-distribution detection.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:28 UTC pith:RXS565GQ

load-bearing objection A plausible combination of standard MovMF with whitened CLIP, but the headline OOD/long-tail gains rest on a same-split evaluation protocol and may not survive held-out testing. the 4 major comments →

arxiv 2607.13660 v1 pith:RXS565GQ submitted 2026-07-15 cs.LG

The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model

Zijie Yu , Gaowen Liu , Ramana Rao Kompella , Philip S. Yu , Yue Song This is my paper
classification cs.LG
keywords CLIPvon Mises-Fisher mixturehyperspherical geometryout-of-distribution detectionlong-tailed recognitionlikelihood estimationsemantic decompositionwhitening
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CLIP embeddings are compared by cosine similarity, so their natural home is the unit sphere; yet existing probabilistic treatments fit them with a single Gaussian. The paper argues that this Gaussian assumption is the wrong geometry and the wrong shape: the space is multimodal, with rare but valid concepts occupying their own directional clusters. It proposes MovMF-CLIP, which whitens raw embeddings to remove global covariance anisotropy, normalizes to the unit sphere, and fits a mixture of von Mises-Fisher distributions by expectation-maximization. The resulting closed-form likelihood separates in-distribution from out-of-distribution samples far better than the Gaussian baseline, especially for tail concepts, and the mixture responsibilities double as a fast, interpretable semantic decomposition. If the claim holds, likelihood-based reasoning about CLIP should be done on the sphere with mixtures, not with Euclidean Gaussians.

Core claim

The central claim is that CLIP latent space is faithfully characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian. Concretely, after a linear whitening transform z̃ = Σ^{-1/2}(z - μ) and unit normalization u = z̃/‖z̃‖, the paper models p(u) = Σ_k π_k C_d(κ_k) exp(κ_k μ_k^T u) and uses the log of this density as the likelihood score. Each von Mises-Fisher component corresponds to a coherent semantic prototype, so rare concepts are represented by their own mixture components instead of being penalized for distance from a global mean. The paper reports that this score cuts FPR95 on MS-COCO from 67.76% to 48.00% overall and from 75.05% to 33.48% on tail-only concepts

What carries the argument

The engine is a Mixture of von Mises-Fisher (MovMF) distributions on the unit hypersphere S^{d-1}, fit by the EM algorithm after whitening. The vMF density f(u; μ, κ) = C_d(κ) exp(κ μ^T u) has a mean direction (the semantic prototype) and a concentration κ controlling cluster tightness; the mixture log-likelihood log Σ_k π_k C_d(κ_k) exp(κ_k μ_k^T u) is the score. The same components produce closed-form posterior responsibilities γ_k(u), which give sparse semantic attribution and a projection operator used to anchor drifting embeddings.

Load-bearing premise

The load-bearing premise is that a single linear whitening transform removes only nuisance global covariance from raw CLIP embeddings, so after unit normalization every useful semantic signal is captured by direction on the sphere; if semantic information also lives in radial magnitude or in nonlinear structure whitening leaves behind, the fitted mixture and all its likelihood scores are distorted.

What would settle it

Take two semantic clusters with different average embedding norms (one consistently larger in radius). If, after whitening and unit normalization, likelihood scores can no longer separate the clusters while a model that uses radial magnitude still can, the core claim fails. Alternatively, rerun the OOD benchmark replacing whitened directions with raw unit-normalized directions: if the OOD gains vanish, the whitening assumption, not the mixture, is carrying the result.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Rare categories can receive high likelihood when they are near their own mixture component, so semantic rarity need no longer be conflated with distributional abnormality.
  • OOD detection improves because low likelihood now means 'outside all learned semantic modes' rather than 'far from the global mean.'
  • Semantic decomposition of an embedding reduces to one matrix multiplication against component centers, removing the need for sparse linear solvers or auxiliary decoders.
  • The learned component directions can be manipulated directly, so the model doubles as a set of controllable semantic axes for generation.
  • The same recipe carries over to other contrastive vision-language models, where the reported gains persist.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the central claim is right, density estimation for contrastive embeddings generally should move from Euclidean Gaussians to directional mixtures; the same whitened-sphere plus vMF recipe is a natural default for any cosine-trained representation.
  • The whitening step is doing a lot of work: a direct test would fit the mixture on raw unit-normalized embeddings without whitening and compare likelihood calibration, since the paper's own appendix argues that components then align with covariance directions rather than semantics.
  • The flat mixture structure leaves the number of components fixed; a hierarchical or nonparametric extension (e.g., a Dirichlet-process vMF) is a plausible route to adaptive semantic granularity and may help very rare concepts share strength with parent clusters.
  • Because the likelihood is a sum of exponentials in μ_k^T u, the model yields a closed-form score on the sphere, suggesting it could act as a geometry-aware prior or guidance term in latent diffusion.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes MovMF-CLIP, a probabilistic model for CLIP embeddings. The method whitens raw CLIP embeddings (Eq. 1), unit-normalizes them (Eq. 2), and fits a mixture of von Mises-Fisher distributions by EM (Eqs. 3-8). The resulting closed-form likelihood (Eq. 9) is used as a density score, and posterior responsibilities (Eq. 10) provide a semantic decomposition. The authors report improvements over W-CLIP and other baselines on long-tail likelihood fairness (Table 1), OOD detection on MS-COCO vs. OpenImages (Table 2), semantic relevance and inference speed (Table 3), and stability under iterative CLIP-UnCLIP drift (Table 4). The mathematical development is standard and clearly presented.

Significance. Assuming the empirical results survive a rigorous, held-out evaluation, this would be a worthwhile contribution: it offers a simple, closed-form, geometry-consistent density for CLIP-like embeddings, with interpretable components and no auxiliary decoders. The use of vMF mixtures on the sphere is well matched to cosine-similarity training, and the paper is refreshingly clear about the whitening step. However, the current evaluation is compromised by fitting the whitening transform, the mixture density, and the hyperparameters (K,N) on the same MS-COCO validation split that is later used as the in-distribution set. The reported gains may therefore reflect memorization rather than a genuinely better geometric model. The central claim that CLIP latent space is 'more faithfully characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian' needs direct, held-out likelihood evidence.

major comments (4)
  1. [Sec. 4.1, Tables 1-2] The density is fit on the evaluation split. The MS-COCO 2017 validation split is used for density estimation (whitening in Eq. 1 and EM in Eq. 5) and also as the ID set for long-tail and OOD evaluation. With K=500 and about 5k reference samples, each component has on average 10 points, so the mixture can memorize the ID set; high-concentration components can assign artificially high likelihood to training samples. W-CLIP's single Gaussian has less capacity, so the comparison is biased. The FPR95 reductions (67.76 to 48.00, 75.05 to 33.48) are not reliable evidence for the geometry claim. Please re-run with the density estimated on a disjoint split (e.g., COCO train or one half of validation) and evaluated on held-out ID samples, for all baselines.
  2. [Sec. 4.3, Table 7] Hyperparameters K and N are selected on the same Semantic Relevance metric and the same MS-COCO validation split that is reported in Table 3. The grid search in Table 7 shows K=500, N=10 as the best on this split; reporting the same number as the method's result is selection on the test set. This inflates the 0.673 Semantic Relevance and the comparison with SPLICE. Select hyperparameters on a separate validation split (or use nested CV) and report test-split results.
  3. [Tables 1-4] No uncertainty quantification is provided. Every table reports a single point estimate without standard errors, confidence intervals, or significance tests. Given the high-capacity mixture and the split leakage, this is not sufficient to support the 'significantly improves' claims in the abstract. Report results over multiple EM restarts or bootstrap resamples of the ID set.
  4. [Sec. 5, Appendix B] The headline geometric conclusion is inferred from task performance rather than directly tested. To support the claim that a hyperspherical mixture is more faithful than an isotropic Gaussian, report held-out log-likelihood comparisons against W-CLIP and against a GMM in the whitened space, plus an ablation with vMF fitted on raw normalized embeddings (no whitening). Appendix B argues that whitening preserves semantics, but no experiment tests whether radial information is semantically informative; this is a correctness risk for the central claim.
minor comments (6)
  1. [Sec. 3.3 vs. Appendix A] The notation is inconsistent: 'MovMF' and 'movMF' are used interchangeably. Please unify.
  2. [Appendix C, Eq. (11)] The Semantic Relevance metric relies on keyword extraction and content-word filtering; the exact procedure is not specified, making the metric hard to reproduce.
  3. [Sec. 4.1] The OpenImages subset is described as 'cleaned following VOS', but the number of OOD images and the filtering procedure are not reported.
  4. [Sec. 4.4] The projection step that reconstructs embeddings as weighted combinations of retained centers and rescales to the original norm is described only in prose; pseudo-code or an equation would help.
  5. [Sec. 4.1] The paper says hyperparameters are 'fixed across tasks' yet Sec. 4.3 uses K=500, N=10 selected in Appendix D; please clarify this statement.
  6. [Figure 1] Figure 1 is described as 'visualized via dimensionality reduction on real data'; the reduction method is not stated.

Circularity Check

2 steps flagged

Reported long-tail and OOD gains are largely in-sample: the MovMF density is fit on the same MS-COCO validation split that is then scored as ID, and K/N are tuned on the reported Semantic Relevance metric.

specific steps
  1. fitted input called prediction [Sec. 4.1-4.2, Tables 1-2]
    "We adopt MS-COCO 2017 as the primary in-distribution (ID) dataset. Its validation split is used for both density estimation and long-tailed evaluation across semantic categories. ... Long-Tailed Likelihood Fairness (texts). We analyze captions of MS-COCO, partitioning into head and tail groups based on concept frequency statistics."

    The MovMF density (Eq. 5) is fit by maximum likelihood on exactly the MS-COCO validation captions/embeddings that Table 1 then scores as head vs tail and Table 2 scores as ID. With K=500 components over roughly 5k reference samples (about 10 points per component on average), EM can place high-concentration vMF components on individual training captions, inflating their in-sample likelihood. The reported AUROC 0.5819 and FPR95 reductions are therefore in-sample measures of how well the fitted density remembers the ID split, not held-out predictions separating semantic rarity from distributional abnormality. The W-CLIP comparison is biased because a single Gaussian has far less capacity to memorize the evaluation split.

  2. fitted input called prediction [Sec. 4.3 and Appendix D, Tables 3 and 7]
    "For our MovMF-CLIP, we use K=500 mixture components and retain the top N=10 keywords per component. Ablation studies over K and N are provided in Appendix D. ... We report the Semantic Relevance (1−d_H) on the MS-COCO validation set ... The best performance is achieved with K=500 and N=10, reaching a Semantic Relevance of 0.6726."

    The hyperparameters K and N are selected by grid search on the same MS-COCO validation Semantic Relevance metric (Table 7), and the same metric and dataset are then reported as the headline result in Table 3 (0.673). Thus the final score is the maximum of the tuning curve, not an independent evaluation; part of the reported advantage over SPLICE comes from selection on the test metric. This is a milder form of fitted-input-called-prediction, but the reported number is not an out-of-sample estimate.

full rationale

No equation-level self-definition or load-bearing self-citation was found. The only author-overlap citation, Kang et al. 2025 (which includes author Yue Song), appears in related-work background and is not load-bearing for the MovMF derivation. The MovMF formulation itself, the EM updates, and the likelihood score are standard and not circular. However, the empirical support for the central geometric claim is compromised by an in-sample evaluation protocol: the density is estimated on the MS-COCO validation split that is later scored as ID in both the long-tail fairness and OOD tables, and the semantic-decomposition hyperparameters are tuned on the same Semantic Relevance metric used for the final comparison. These issues make the reported improvements partly artifacts of memorization and selection rather than demonstrating that hyperspherical mixtures are intrinsically better. The central claim still has independent modeling content, so the score is below 6, but the key empirical predictions are not fully out-of-sample.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new entities are introduced. The load-bearing choices are the whitening step, the number of components K, and the identification of directions with semantics; the mixture parameters themselves are ordinary fitted model parameters.

free parameters (4)
  • Whitening mean μ and covariance Σ = Estimated from MS-COCO validation set
    Eq. (1); this defines the calibrated geometry and is evaluated on the same split used for fitting.
  • Number of mixture components K = K=500
    Grid-searched on Semantic Relevance over K=70–1000 (Table 7); also used in OOD and stability experiments.
  • Top keywords per component N = N=10
    Selected together with K on the same validation metric; directly affects the reported Semantic Relevance score.
  • Mixture parameters (π_k, μ_k, κ_k) = EM-MLE on MS-COCO validation
    These are standard fitted model parameters rather than ad hoc constants, but they are fit to the same data used for evaluation.
axioms (5)
  • domain assumption All semantic information relevant to CLIP similarity is contained in the direction of the whitened embedding.
    Sec. 3.1, Eq. (2): embeddings are normalized to the unit sphere and radial information is discarded.
  • domain assumption A linear whitening transform W = Σ^{-1/2} removes 'nuisance' covariance without destroying semantic clusters.
    Sec. 3.1 and Appendix B: whitening is treated purely as metric normalization; this is not proven for CLIP embeddings.
  • domain assumption A finite mixture of vMF distributions with K components approximates the true distribution on S^{d-1} well enough for likelihood-based OOD detection.
    Sec. 3.2: the vMF mixture is postulated as the natural density model; K is chosen empirically.
  • ad hoc to paper Each EM-learned component corresponds to a coherent semantic concept interpretable by humans.
    Sec. 3.4 and Sec. 4.3: component centers are treated as semantic prototypes; no independent measure verifies component-level purity.
  • domain assumption OpenImages (cleaned by VOS) is a valid OOD distribution relative to MS-COCO as the ID distribution.
    Sec. 4.1: evaluation protocol depends on this dataset pairing.

pith-pipeline@v1.3.0-alltime-deepseek · 15945 in / 11762 out tokens · 625756 ms · 2026-08-02T04:28:48.654932+00:00 · methodology

0 comments
read the original abstract

Contrastive Language-Image Pretraining (CLIP) representations form a semantic embedding space governed by cosine similarity, reflecting an intrinsic hyperspherical geometry. However, existing probabilistic interpretations typically rely on Gaussian assumptions, which fail to capture this directional and multimodal structure. We propose a principled density model for the CLIP latent space based on Mixtures of von Mises-Fisher (MovMF) distributions defined on the unit hypersphere. Using the Expectation-Maximization (EM) algorithm, we efficiently learn a probabilistic model in which each mixture component corresponds to a coherent semantic concept. This formulation yields a closed-form likelihood naturally aligned with hyperspherical geometry, enabling accurate and interpretable density estimation. Empirically, our model significantly improves long-tailed and out-of-distribution detection and provides a natural semantic decomposition, representing each embedding as a sparse probabilistic combination of interpretable concepts. These results suggest that CLIP latent space is more faithfully characterized as a hyperspherical semantic mixture rather than an isotropic Gaussian, establishing a simple and geometrically consistent probabilistic framework for modeling and understanding multimodal representations. Project page is available at https://xiaoyuzhizi.github.io/movmf-clip/.

Figures

Figures reproduced from arXiv: 2607.13660 by Gaowen Liu, Philip S. Yu, Ramana Rao Kompella, Yue Song, Zijie Yu.

Figure 1
Figure 1. Figure 1: Density modeling of CLIP latent space (visualized via dimensionality reduction on real data). (a) Gaussian-based approaches such as W-CLIP model the latent space with a single global distribution, which can assign low likelihood to valid long-tail concepts due to their distance from the global mean. (b) MovMF-CLIP models the space as a hyperspherical semantic mixture, capturing the multimodal structures an… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MovMF-CLIP Framework. We first extract raw embeddings using the CLIP encoder, which exhibit severe anisotropy. To address this, we apply geometric calibration via whitening (z˜ = W(z − µ)) and normalize the features onto a unit hypersphere (u = ˜z/∥z˜∥2). Finally, we fit a MovMF distributions on the hypersphere using the EM algorithm, yielding a prin￾cipled multimodal density model. Cao et … view at source ↗
Figure 3
Figure 3. Figure 3: Concept localization. MovMF-CLIP represents each image as a sparse mixture of se￾mantic prototypes. Concept-specific heatmaps highlight image regions associated with each se￾mantic component. Qualitative Analysis. To provide an intuitive illustration of the learned semantic structure, we visualize and compare the decompositions produced by MovMF-CLIP and SPLICE on several complex real-world scenes (see fig… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of semantic decomposition. Concept extraction results for SPLICE and MovMF-CLIP on representative scenes. Projection-based methods may emphasize generic or context-dependent terms, whereas our MovMF-CLIP produces a sparse probabilistic combination of semantically coherent cluster prototypes aligned with object-centric content [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Directional semantic intervention in CLIP latent space. We identify a vMF component associated with the cat concept and directly manipulate its direction in the whitened CLIP space before UnCLIP decoding. Removing the component suppresses the cat from the generated image, while amplifying the same direction strengthens the corresponding visual semantics. This illustrates that MovMF-CLIP learns semantically… view at source ↗
Figure 6
Figure 6. Figure 6: Pipeline for iterative generative drift analysis. We insert MovMF￾CLIP into the encoding-decoding loop to stabilize the underlying semantics. Semantic Stabilization via Hyperspherical Mixture Projection. Given a whitened embedding z˜, we first normalize it onto the hypersphere and compute its posterior responsibilities {γk}. Compo￾nents with negligible posterior mass are discarded, and the embedding is rec… view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison under iterative generative drift. For each example, the original image is followed by its evolution over six encoding-decoding iterations. The three rows demonstrate the trajectories using vanilla CLIP-UnCLIP (top), W-CLIP (middle), and MovMF-CLIP (bottom). Our hyperspherical mixture aligns well with coherent semantic prototypes, resulting in more stable semantic trajectories. 11 [P… view at source ↗
Figure 8
Figure 8. Figure 8: Likelihood distributions for OOD detection. Histograms of log-likelihood scores for ID and OOD samples. Left: W-CLIP exhibits substantial overlap between ID and OOD distribu￾tions, particularly when the ID set contains rare concepts. Right: MovMF-CLIP produces better separation, reflecting improved robustness to long-tail semantics while maintaining strong OOD discrimination. 18 [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 9
Figure 9. Figure 9: Extended Results of Semantic Decomposition. Comparison of text descriptions decoded from the latent embeddings. MovMF-CLIP consistently extracts precise, fine-grained semantic con￾cepts (e.g., “wii”, “skateboard”), while the baseline SPLICE often struggles with hallucinated or coarse terms (e.g., “bored”, “hvac”). preserving the core semantic identities (skater, pizza, surfer, tennis player, elephant) with… view at source ↗
Figure 10
Figure 10. Figure 10: Extended Results of Concept Localization. (Continued on next page.) 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: Extended Results of Concept Localization (Continued). MovMF-CLIP successfully disentangles complex visual scenes into orthogonal semantic components. Each heatmap accurately grounds specific keywords derived from our mixture components, demonstrating precise spatial isolation of objects. 1 2 3 4 5 6 Round 0.68 0.70 0.72 0.74 0.76 LPIPS (AlexNet) LPIPS-Alex MovMF-CLIP Vanilla CLIP-UnCLIP W-CLIP 1 2 3 4 5 6… view at source ↗
Figure 11
Figure 11. Figure 11: Quantitative Trends across Iterations. We track the average LPIPS (left, ↓) and CLIP Cosine Similarity (right, ↑) over 1,000 COCO images across 6 CLIP-UnCLIP iterations. While W-CLIP and Vanilla CLIP-UnCLIP show a slight initial advantage, they suffer from accelerating degradation. In contrast, MovMF-CLIP stabilizes the trajectory, significantly outperforming the baseline from round 3 onward. 22 [PITH_FU… view at source ↗
Figure 12
Figure 12. Figure 12: Extended Results of Iterative Generative Drift. Trajectories over 6 CLIP-UnCLIP iterations. Vanilla CLIP-UnCLIP suffers from severe mode collapse (e.g., generating meaningless stripes). W-CLIP loses semantic details. Our MovMF-CLIP correctly anchors the embeddings to high-density semantic prototypes, better preserving the original concepts. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 28 linked inside Pith

  1. [1]

    Why not fit movMF directly onz/∥z∥?If we normalize first,u 0 =z/∥z∥, the subsequent movMF fit must simultaneously explain (i) global anisotropic scaling and (ii) genuine semantic multimodal- ity. In practice, this causes mixture components to align with dominant covariance directions rather than semantic modes, leading to redundant components, unstable co...

    arXiv

  2. [2]

    Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P

    URLhttps://arxiv.org/abs/2505.06934. Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. In- terpreting clip with sparse linear concept embeddings (splice).Neurips,

    Pith/arXiv arXiv

  3. [3]

    remain competitive, indicating that the learned hyperspherical mixture structure is robust across different sets of hyperparameters. E EXTENDEDQUALITATIVEANALYSES We provide extensive results to further demonstrate the robustness, fine-grained interpretability of concept localization, and semantic stability under generative drifts of our MovMF-CLIP framew...

    arXiv 2000

  4. [6]

    Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li

    URLhttps://arxiv.org/abs/ 2602.06218. Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. V os: Learning what you don’t know by virtual outlier synthesis.ICLR,

    arXiv

  5. [7]

    Yossi Gandelsman, Alexei A

    URLhttps://arxiv.org/abs/2202.01197. Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip’s image representa- tion via text-based decomposition.ICLR,

    Pith/arXiv arXiv

  6. [8]

    org/abs/2205.14459

    URLhttps://arxiv. org/abs/2205.14459. Tao Gong, Qi Chu, Bin Liu, Zhou Wei, and Nenghai Yu. Fe-clip: Frequency enhanced clip model for zero-shot anomaly detection and segmentation.ICCV,

    Pith/arXiv arXiv

  7. [9]

    com/virtual/2025/poster/2571

    URLhttps://iccv.thecvf. com/virtual/2025/poster/2571. 12 Jianfang He, Min Cao, Silong Peng, and Qiong Xie. Rareclip: Rarity-aware online zero-shot indus- trial anomaly detection. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 24478–24487, October

    2025

  8. [10]

    Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han

    URLhttps://arxiv.org/abs/2507.09118. Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Negative label guided ood detection with pretrained vision-language models.ICLR,

    Pith/arXiv arXiv

  9. [11]

    Raphi Kang, Yue Song, Georgia Gkioxari, and Pietro Perona

    URLhttps: //arxiv.org/abs/2403.20078. Raphi Kang, Yue Song, Georgia Gkioxari, and Pietro Perona. Is clip ideal? no. can we fix it? yes! ICCV,

    Pith/arXiv arXiv

  10. [12]

    Elias Kempf, Simon Schrodi, Max Argus, and Thomas Brox

    URLhttps://arxiv.org/abs/2503.08723. Elias Kempf, Simon Schrodi, Max Argus, and Thomas Brox. When and how does clip enable domain and compositional generalization?ICML,

    arXiv

  11. [13]

    URLhttps://arxiv.org/abs/ 2502.09507. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relation- ship detection at scale.Inter...

    Pith/arXiv arXiv 1956

  12. [15]

    Meir Yossef Levi and Guy Gilboa

    URLhttps://arxiv.org/abs/2407.12442. Meir Yossef Levi and Guy Gilboa. The double-ellipsoid geometry of clip.ICML,

    Pith/arXiv arXiv

  13. [16]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi

    URL https://arxiv.org/abs/2411.14517. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation.ICML,

    Pith/arXiv arXiv

  14. [17]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi

    URLhttps: //arxiv.org/abs/2201.12086. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.ICML,

    Pith/arXiv arXiv

  15. [19]

    Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li

    URLhttps://arxiv.org/abs/2203.02053. Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with vision-language representations.Advances in neural information pro- cessing systems, 35:35087–35102,

    Pith/arXiv arXiv

  16. [20]

    URLhttps: //arxiv.org/abs/2406.08074. 13 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision.ICML,

    Pith/arXiv arXiv

  17. [22]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer

    URLhttps://arxiv.org/abs/ 2204.06125. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High- resolution image synthesis with latent diffusion models.CVPR,

    Pith/arXiv arXiv

  18. [23]

    org/abs/2112.10752

    URLhttps://arxiv. org/abs/2112.10752. Jie-Jing Shao, Jiang-Xin Shi, Xiao-Wen Yang, Lan-Zhe Guo, and Yu-Feng Li. Examining the achilles’ heel of CLIP models: The worst-performing categories,

    Pith/arXiv arXiv

  19. [24]

    Harald Steck, Chaitanya Ekanadham, and Nathan Kallus

    URLhttps://arxiv.org/abs/2011.13456. Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. Is cosine-similarity of embeddings re- ally about similarity?WWW,

    Pith/arXiv arXiv 2011

  20. [25]

    URLhttp://dx.doi.org/10.1145/3589335. 3651526. Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language- image pre-training (clip).Neurips,

    doi:10.1145/3589335

  21. [26]

    Arash Vahdat, Karsten Kreis, and Jan Kautz

    URLhttps://arxiv.org/abs/2402.07410. Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. NeurIPS,

    Pith/arXiv arXiv

  22. [27]

    Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang

    URLhttps://arxiv.org/abs/2106.05931. Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of clips to spurious features.Neurips,

    Pith/arXiv arXiv

  23. [28]

    org/abs/2403.11497

    URLhttps://arxiv. org/abs/2403.11497. Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere.ICML,

    Pith/arXiv arXiv

  24. [29]

    Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, and Xiaojuan Qi

    URLhttps://arxiv.org/abs/ 2005.10242. Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, and Xiaojuan Qi. What makes clip more robust to long-tailed pre-training data? a controlled study for transferable insights.Neurips,

    Pith/arXiv arXiv 2005

  25. [30]

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu

    URLhttps://arxiv.org/abs/2405.21070. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models.TMLR,

    Pith/arXiv arXiv

  26. [31]

    Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek

    URLhttps: //arxiv.org/abs/2205.01917. Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting clip with hierarchical sparse autoencoders.ICML,

    Pith/arXiv arXiv

  27. [32]

    Richard Zhang, Phillip Isola, Alexei A

    URLhttps://arxiv.org/abs/2502.20578. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric.CVPR,

    Pith/arXiv arXiv

  28. [34]

    URLhttps://arxiv.org/ abs/2310.18961. 14 A FUTUREWORK A.1 CONNECTION WITHSCORE-BASEDGENERATIVEMODELS Beyond density estimation, the hyperspherical density learned by MovMF-CLIP may provide a use- ful geometric prior for generative modeling. Modern latent diffusion and score-based models (Song et al., 2021; Rombach et al., 2022; Vahdat et al.,

    arXiv 2021

  29. [35]

    In contrast, the movMF mixture defines a directional density on the hypersphere, which naturally yields a closed-form score function on the manifold

    typically assume simple Gaussian priors in the latent space. In contrast, the movMF mixture defines a directional density on the hypersphere, which naturally yields a closed-form score function on the manifold. This observation suggests a potential connection between hyperspherical density modeling and Riemannian score-based genera- tive models (Bortoli e...

    2022

  30. [2018]

    org/abs/1801.03924

    URLhttps://arxiv. org/abs/1801.03924. Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection.ICLR,

    Pith/arXiv arXiv

  31. [2020]

    doi: 10.1007/s11263-020-01316-z

    ISSN 1573-1405. doi: 10.1007/s11263-020-01316-z. URLhttp://dx.doi.org/ 10.1007/s11263-020-01316-z. Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision-language inference.ECCV,

    doi:10.1007/s11263-020-01316-z

  32. [2021]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

    URLhttps://arxiv.org/abs/2103.00020. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents,

    Pith/arXiv arXiv

  33. [2022]

    Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han

    URLhttps: //arxiv.org/abs/2202.02763. Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envisioning outlier exposure by large language models for out-of-distribution detection.ICML, 2024a. URL https://arxiv.org/abs/2406.00806. Yunkang Cao, Jiangning Zhang, Luca Frittoli, Yuqi Cheng, Weiming Shen, and Giacomo Boracchi. Adaclip: Ada...

    Pith/arXiv arXiv

  34. [2023]

    Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou

    URLhttps: //arxiv.org/abs/2301.12597. Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Un- derstanding the modality gap in multi-modal contrastive representation learning.Neurips,

    Pith/arXiv arXiv

  35. [2024]

    Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet

    URLhttps: //arxiv.org/abs/2402.10376. Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modelling.NeurIPS,

    Pith/arXiv arXiv

  36. [2025]

    Arindam Banerjee, Inderjit S

    URLhttps://arxiv.org/abs/2502.19842. Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hyper- sphere using von mises-fisher distributions.JMLR,

    Pith/arXiv arXiv

  37. [2026]

    Gr´egoire Dhimo ¨ıla, Thomas Fel, Victor Boutin, and Agustin Picard

    URLhttps://arxiv.org/abs/2602.03282. Gr´egoire Dhimo ¨ıla, Thomas Fel, Victor Boutin, and Agustin Picard. Cross-modal redundancy and the geometry of vision-language embeddings,

    Pith/arXiv arXiv