{"id":"8cc29b19-9674-43f8-bb30-02ce02eba0ab","arxiv_id":"2411.14517","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The pre-normalized CLIP space consists of two linearly separable, offset ellipsoid shells, and cosine similarity to the modality mean closely estimates how typical an image or caption is.","lead":"CLIP's raw image and text embeddings are not spread on a ball centered at zero; they form two separate, tilted ellipsoid shells shifted away from the origin. The paper explains this shape as a way to handle uncertain image-text matches, and uses it to build a cheap measure of how common a sample is.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Supp. C.1's Proposition 1 derives a slope μ/√(R²+μ²) ≈ 0.707 but states and needs A ≈ 1.414, and the whitened shell is 8–9 times wider than assumed; the conformity-surrogate claim is therefore empirical, not proven.","rationale":"The reader's weakest-assumption analysis correctly flags that Supp. C.1's Proposition 1 relies on thin-shell and orthogonality assumptions that are not strictly satisfied, and that the derived and fitted slopes disagree. My read agrees with that diagnosis but sharpens it into an internal inconsistency: as written, the proposition states A ≈ √(μ²+R²)/μ ≈ 1.414, while the proof's own algebra yields A = μ/√(R²+μ²) ≈ 0.707. The paper even labels the derived quantity A⁻¹ and notes that it matches the empirical slope, which means the stated proportionality constant is the reciprocal of the one needed to match Fig. 9. This is more severe than a loose approximation: the theorem, if taken literally, predicts a slope of about 0.7, whereas the data show a slope of about 1.4. The paper's own whitening analysis (Supp. Sec. 2.1) reports that the whitened shell is 8–9 times wider than a chi distribution, so the assumed concentration regime is not present. Thus the conformity surrogate is supported by an excellent in-sample correlation on MS-COCO/ViT-B/32, but not by the stated proof, and no held-out or cross-architecture test is provided. I do not think this warrants rejection: the empirical geometry results and the high correlation on MS-COCO are credible, and the applications in Sec. 7 are interesting conditional on the surrogate. The appropriate outcome remains the reader's CONDITIONAL verdict, since the load-bearing theoretical claim needs correction and validation before the paper's more ambitious conclusions can be accepted without reservation.","tokens_in":20189,"tokens_out":6739,"duration_ms":63583,"concrete_test":"Recompute the Fig. 9 regression slope a for images from the released features and compare it with A = μ/√(R²+μ²) and with its reciprocal √(R²+μ²)/μ. If the fitted a is ≈ 1.46 while the derived A ≈ 0.707, the stated Proposition 1 does not explain the observed surrogate. Then, on the same encoder, compute C(v) and cos(m,v) on a held-out dataset such as Flickr30K or CC3M; if the Pearson correlation drops materially below 0.9998 or the optimal slope moves substantially away from 1.4, the conformity surrogate is a dataset-specific empirical relation rather than a geometric law.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Supp. C.1, Eq. (13) asserts E_j cos(v_i,v_j) ≈ A cos(v_i, v̄) with A ≈ √(μ²+R²)/μ, which for the reported MS-COCO values gives A ≈ 1.414. The proof, however, replaces ||v_j|| by √(R²+μ²) and obtains A = μ/√(R²+μ²) ≈ 0.707. The paper acknowledges this by describing the derived quantity as A⁻¹ and saying it matches the empirical slope, but the proposition as stated is then internally inconsistent: its A is the reciprocal of the fitted slope (a ≈ 1.41 in Fig. 9). Separately, Supp. Sec. 2.1 reports std(||y||) ≈ 4.04–4.64 for whitened CLIP, 8–9 times the chi-distribution value of about 0.5, so the 'thin shell and approximate orthogonality' approximation regime is not satisfied. Consequently, the high Pearson correlation of 0.9998 supports the affine relation on MS-COCO/ViT-B/32, but the theoretical derivation does not establish it, and no held-out or cross-model test demonstrates generality. This matters because conformity is used in Sec. 6.2 to explain the modality gap and in Sec. 7 to rank generators; if the surrogate is only a fitted empirical fact on one dataset, those applications rest on a narrower foundation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes the raw (pre-L2-normalization) CLIP embedding space and reports that image and text embeddings lie on linearly separable, tilted ellipsoid shells that are shifted away from the origin. It argues that this geometry is beneficial under the CLIP contrastive loss because it allows the network to place uncertain, frequent concepts closer to the modality mean, thereby mitigating false negatives. The paper introduces a notion of 'conformity' (average cosine similarity to all other instances) and proposes a fast surrogate based on cosine similarity to the mean vector; a proof of this proportionality is given in the supplementary material under a thin-shell assumption. It further connects the modality gap to matching the conformity distributions of images and text, and demonstrates applications in ranking generative models and a new interpolation scheme (vSLERP). The empirical geometry is established mainly on MS-COCO with ViT-B/32 and supplemented by ViT-L/14 results.","tokens_in":20521,"tokens_out":2899,"duration_ms":27782,"significance":"If the findings hold, the paper provides a useful structural description of CLIP's latent space that goes beyond the standard unit-sphere view, and it offers a practical quantity (conformity) with potential utility for evaluating generative diversity and for interpolation. The empirical evidence for the double-ellipsoid geometry is strong: the paper reports thin-shell norm distributions, anisotropic and correlated features, separable modalities, and loss and KL-divergence experiments that consistently favor non-origin-centered ellipsoids. The proposed conformity surrogate has an extremely high empirical correlation (0.9998) on MS-COCO. However, the theoretical derivation of this surrogate is internally inconsistent (the derived constant is the reciprocal of the fitted slope), and the paper's own whitening analysis shows that the thin-shell assumption used in that derivation is not quantitatively satisfied. The generality of the conformity relation is therefore not established beyond the single dataset and model where it was fitted.","major_comments":[{"comment":"The proof of Proposition 1 is internally inconsistent: the text states that A ≈ √(μ²+R²)/μ, but the derivation obtains A = μ/√(R²+μ²) ≈ 0.707 for the reported MS-COCO values, which is the reciprocal of the fitted slope a ≈ 1.41 in Fig. 9. The paper acknowledges that the derived quantity is A^{-1}, but as written the proposition asserts C(v_i) ≈ A · cos(v_i, v̄) with the wrong value of A. This is a load-bearing error because the abstract and Section 6.1 claim that the surrogate is 'proved' under the thin-shell assumption, whereas the proof as stated does not establish the claimed proportionality with the correct constant.","section":"Supp. C.1, Eq. (13) and Proposition 1"},{"comment":"The proof of Proposition 1 relies on the thin-shell approximation ∥v_j∥ ≈ √(R²+μ²) and approximate orthogonality between z_j and v̄. Yet the paper's own whitening analysis reports std(∥y∥) ≈ 4.04–4.64 for n = 512, which is 8–9 times the chi-distribution value of about 0.5, and the authors explicitly state that 'the shell is not that thin, and certainly not as thin as chi distribution.' For the original (non-whitened) embeddings, the ratio var(y)/μ²_norm is small (0.1914/57.57), so the norm is concentrated on a relative scale, but the proof also needs ∥v_j∥ to be nearly constant in the denominator of the cosine; the intermediate-scale fluctuations are exactly what determine the constant A. Thus the proportionality between conformity and cosine-to-mean is an empirical observation on MS-COCO, not a consequence of the stated assumptions. The authors should either revise the proof to account for the actual norm variability or explicitly present Eq. (13) as an empirical relation.","section":"Supp. Sec. 2.1 (Claim 1 and whitened CLIP)"},{"comment":"The Pearson correlation of 0.9998 is measured between C and Ĉ on the same set S that defines both the empirical conformity and the mean m used in Ĉ. This makes the high correlation an internal consistency property of the embedding on MS-COCO, not a test of whether conformity tracks concept frequency or whether the surrogate generalizes to other datasets, models, or distributions. Since Prediction 1 is validated only through this same-set correlation, and since Sections 6.2 and 7 use the surrogate to draw conclusions about the modality gap and to rank generators, the absence of a held-out or cross-model check is a substantive gap. A simple split-half or cross-dataset evaluation would substantially strengthen the claim.","section":"Section 6.1 (Fig. 9) and Prediction 1"}],"minor_comments":[{"comment":"The sentence 'validating with close to perfect alignment with the rigorous mathematic derivation' is misleading given that the derived constant is the reciprocal of the fitted slope; please correct the wording to reflect the actual relationship between the derived and fitted quantities.","section":"Section 6.1"},{"comment":"The supplementary material contains several editorial artifacts: the 'Anonymous Authors' header, a dangling 'AUTHORERR: Missing \\icmlcorrespondingauthor', repeated figure-caption text such as 'There is no image here to provide a caption for', and typos such as 'Enlraged'. These should be cleaned up before publication.","section":"Supplementary material (general)"},{"comment":"The loss experiment shifts the image embeddings by -α·m_i but does not explicitly state whether the text embeddings or the normalization are recomputed after the shift; please clarify the exact procedure, since the normalization step affects the reported loss values.","section":"Section 5, Eq. (9) and Fig. 6"},{"comment":"The Gaussian-sphere toy experiment is illustrative but does not model the actual CLIP covariance structure; consider adding a sentence noting that the effect is shown qualitatively and that the real-embedding loss experiments in Fig. 6 provide the quantitative support.","section":"Section 6, Fig. 7"},{"comment":"The vSLERP equation uses α both in the interpolation offset and to denote the interpolation magnitude; the notation is clear from the figure but could be disambiguated to avoid confusion with the α used in the loss experiments.","section":"Section 7.2, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The empirical geometry part of the paper is solid and likely to be of interest to the community, and the conformity surrogate is practically useful even if the proof is flawed. The main risk is that the authors' overclaim of a proof could mislead readers; I would urge the editor to require that the proof in Supp. C.1 be corrected or explicitly downgraded to an empirical observation, and that a cross-model or held-out validation of the conformity relation be added. The paper is within the scope of the journal and the contribution is significant enough to merit revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news is the empirical geometry: on MS-COCO, CLIP's pre-normalized image and text embeddings form two linearly separable, tilted ellipsoid shells offset from the origin, and the paper documents that carefully. That is the part worth citing. The conformity surrogate—average cosine to all other vectors approximated by cosine to the modality mean—is genuinely useful and the correlation of 0.9998 is striking on the tested set. But the proof in Supp. C.1 is not a proof. Proposition 1 defines A as sqrt(mu^2+R^2)/mu (about 1.41) yet the derivation produces A = mu/sqrt(mu^2+R^2) (about 0.71). The authors notice the mismatch and say the derived quantity is A^-1, which matches the empirical slope, but that is an acknowledgment that the proposition as stated is internally inconsistent. The thin-shell assumption is also shaky: their own whitening analysis shows the shell is 8-9 times wider than a chi distribution. So the conformity relation is an empirical fact on MS-COCO, not a general result under the stated assumptions.\n\nThe other soft spot is the in-sample nature of the key validation. The 0.9998 correlation is computed on the same set that defines both conformity and the mean, so it is internal consistency, not a held-out test. The claim that 'common themes' sit near the mean is supported by qualitative examples rather than by a quantitative frequency analysis. The loss experiment and modality-gap matching are post-hoc examinations of a trained model, not tests of the proposed mechanism. None of these are fatal; they are limitations that could be addressed by a held-out evaluation and a corrected proposition.\n\nWhat the paper does well is the careful data-driven characterization: the histograms, the covariance analysis, the two-feature linear separability, and the thin-shell plots are convincing and reproducible. The vSLERP interpolation trick is minor but a nice practical byproduct.\n\nThis paper deserves a serious referee. The empirical discovery is solid and the flaws are fixable in revision. I would ask for a corrected proof or an explicit reframing of the conformity relation as empirical, plus held-out verification of the surrogate on another dataset or model. I would cite it for the geometry.","headline":"The empirical double-ellipsoid geometry of CLIP is real and worth knowing, but the conformity proof in Supp. C.1 has a slope inversion and the key validation is in-sample, so treat the theory as a plausible hypothesis rather than a proven theorem.","tokens_in":21047,"tokens_out":3499,"would_cite":true,"duration_ms":32291,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP's raw embeddings of images and text form two linearly separable, tilted ellipsoid shells shifted from the origin, and that offset is what lets the model balance alignment against uniformity and blur uncertain concepts.","keywords":["CLIP","contrastive learning","embedding geometry","ellipsoid shell","thin shell","modality gap","conformity","false negatives"],"falsifier":"Take a different CLIP-style contrastive model or a held-out split of the original training distribution and compute the Pearson correlation between true conformity $C(v)$ and estimated conformity $a\\cos(m,v)+b$; if the correlation drops well below the reported 0.9998, or if the KL-divergence minimum between image and text conformity distributions moves away from the learned offset, the central geometric claim fails. A second check is whether concept frequency in the training set actually correlates with cosine similarity to the modality mean as the paper predicts.","tokens_in":19959,"feed_emoji":"📐","tokens_out":4866,"duration_ms":44433,"temperature":0.7,"pith_summary":"The paper examines what CLIP's embeddings look like before the usual L2 normalization and argues that the seemingly arbitrary geometry is doing real work. It claims that image and text vectors each lie on a tilted ellipsoid shell, with the two shells linearly separable and shifted away from the origin. That shift is what lets the training loss balance alignment of correct pairs against uniformity of negatives, and it gives CLIP a way to soften its response for frequent, easily confused concepts. To make this quantitative, the paper defines conformity, the average cosine similarity of an instance to every other instance, and shows that it is proportional to the cosine similarity to the modality mean. If true, the geometry explains the modality gap and narrow cone effects as consequences of optimizing contrastive loss, not as artifacts.","feed_headline":"Images and text in CLIP live on two tilted ellipsoid shells","feed_subtitle":"The shift from the origin explains the modality gap and how CLIP handles uncertain concepts.","key_machinery":"The load-bearing object is the double-ellipsoid geometry of the raw embedding, with each modality having its own mean vector $m_i$, $m_t$, a shell radius $R$, and a tilt from feature correlations. The argument runs through the thin-shell property (mass concentrated near a shell), linear separability of the two modality clouds, the conformity identity $C(v)\\approx A\\cos(v,m)$ stated as Proposition 1 in the supplement, and an experiment that varies the offset $\\alpha$ and shows the full CLIP loss and the KL divergence between image and text conformity distributions are both optimized near the learned offset. The mechanism is that an offset shell creates a gradient of sharpness, so uncertain, frequently co-occurring concepts can be blurred toward the mean, reducing false-negative damage without hurting confident instances.","core_discovery":"The central discovery is a geometric description of the pre-normalized CLIP latent space: images occupy one ellipsoid shell, text another, and the two are linearly separable, tilted because features are correlated, and shifted from the origin. The paper argues this structure is not accidental. Shifting the shells away from the origin lets the model trade alignment against uniformity in the contrastive loss, and it makes cosine similarity to the mean behave like a measure of how common an instance is: frequent concepts, which generate more false negatives and higher uncertainty, are embedded closer to the modality mean. The paper introduces conformity $C(v)=E[\\cos(v,v_k)]$ and shows $C(v)\\approx a\\cos(m,v)+b$ with near-perfect correlation on MS-COCO, and shows that the alignment of conformity distributions between modalities is optimized at the offset CLIP actually learned. In short, the paper claims the modality gap and the narrow cone are not bugs but features of the learned geometry.","pith_inferences":["Editorial extension: if concept frequency is what sets the offset, then a controlled experiment with a dataset where image-caption pairs are artificially corrupted to increase false negatives should measurably increase the learned modality offset; this test is not run in the paper.","Editorial extension: the thin-shell proof's assumptions are not strictly met, since the paper's own whitening analysis reports a shell 8 to 9 times wider than a chi distribution, so the conformity proportionality is an empirical fact on MS-COCO rather than a fully general proven result.","Editorial extension: the same conformity-to-mean logic could be applied to other contrastive vision-language models to see whether their raw spaces are also double-ellipsoid; the paper only examines two CLIP encoder variants.","Editorial extension: the semantic-blur mechanism suggests a testable prediction that deliberately removing false negatives from training data should make the ellipsoid offset shrink, a prediction that could be checked by retraining or fine-tuning."],"forward_implications":["If the geometry claim holds, the modality gap and narrow cone observed on the unit sphere are direct consequences of the pre-normalization ellipsoids, so analyses on the hypersphere alone miss the mechanism.","Conformity can be estimated in constant time per instance by cosine similarity to the modality mean, enabling cheap diversity and expressiveness scoring for image generators and captioning models.","The loss experiment predicts that moving the ellipsoid offset away from its learned value should systematically worsen either alignment for correct pairs or uniformity for misclassified pairs.","The proposed vSLERP interpolation should let downstream editing preserve the same object identity because interpolation is performed around the shifted ellipsoid center.","The learned offset should align the conformity distributions of image and text, so deviating from that offset should increase the KL divergence between those distributions."],"supporting_citations":[{"why":"Defines CLIP, its encoders, and the contrastive loss that the paper analyzes.","marker":"Radford et al., 2021"},{"why":"Supplies the MS-COCO validation set used for all empirical statistics, separability tests, and conformity correlations.","marker":"Lin et al., 2014"},{"why":"Documents the modality gap and narrow cone on the normalized sphere, the phenomena this paper re-explains via the ellipsoids.","marker":"Liang et al., 2022"},{"why":"Provides the alignment and uniformity decomposition of the contrastive loss that grounds the offset-optimality argument.","marker":"Wang & Isola, 2020"},{"why":"Supplies the thin-shell concentration bound used to justify treating embedding norms as concentrated around a shell.","marker":"Klartag, 2023"},{"why":"Relates the modality gap to entropy and information imbalance, serving as a comparison for the paper's geometric explanation.","marker":"Schrodi et al., 2024"},{"why":"Provides the unCLIP framework used in the vSLERP interpolation application.","marker":"Ramesh et al., 2022"},{"why":"Provides ClipCap, one of the captioning methods whose conformity is compared in the diversity evaluation.","marker":"Mokady et al., 2021"}],"fun_headline_variants":["CLIP's hidden geometry: two tilted ellipsoids, not a sphere","Double-ellipsoid geometry explains CLIP's modality gap","CLIP embeddings form two tilted ellipsoid shells","Why CLIP's text and image spaces are ellipsoid shells"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that conformity equals a linear function of cosine similarity to the mean assumes embeddings lie on an extremely thin shell with deviations orthogonal to the mean, yet the paper's own whitening numbers show the shell is 8 to 9 times wider than a chi distribution, so that proportionality is empirically established on MS-COCO but not as a general proven fact.","fun_headline_variants_meta":{"raw":{"variants":["CLIP's hidden geometry: two tilted ellipsoids, not a sphere","Double-ellipsoid geometry explains CLIP's modality gap","CLIP embeddings form two tilted ellipsoid shells","Why CLIP's text and image spaces are ellipsoid shells"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3286,"prompt_tokens":882,"completion_tokens":2404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2331}},"tokens_in":498,"tokens_out":2404,"duration_ms":16294,"temperature":1.0,"reasoning_tokens":2331,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:21:44.064139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a different CLIP-style contrastive model or a held-out split of the original training distribution and compute the Pearson correlation between true conformity $C(v)$ and estimated conformity $a\\cos(m,v)+b$; if the correlation drops well below the reported 0.9998, or if the KL-divergence minimum between image and text conformity distributions moves away from the learned offset, the central geometric claim fails. A second check is whether concept frequency in the training set actually correlates with cosine similarity to the modality mean as the paper predicts.","supporting_citations":[{"cited_title":"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning","cited_arxiv_id":null,"evidence_quote":"Documents the modality gap and narrow cone on the normalized sphere, the phenomena this paper re-explains via the ellipsoids."},{"cited_title":"Understanding contrastive representation learning through alignment and uniformity on the hypersphere","cited_arxiv_id":null,"evidence_quote":"Provides the alignment and uniformity decomposition of the contrastive loss that grounds the offset-optimality argument."},{"cited_title":"Logarithmic bounds for isoperimetry and slices of convex sets","cited_arxiv_id":null,"evidence_quote":"Supplies the thin-shell concentration bound used to justify treating embedding norms as concentrated around a shell."}],"review_version":1}