Pith. sign in

REVIEW 2 major objections 2 minor 6 references

CLIP-style vision-language models embed a shared 164-dimensional noise subspace that pruning leaves downstream performance intact or better.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-06-30 21:06 UTC pith:LAEOYYRH

load-bearing objection They report 164 shared dimensions in CLIP embeddings that can be pruned with little or no downstream loss, but the evidence that these are non-semantic noise rather than shared signal is not yet convincing. the 2 major comments →

arxiv 2605.14893 v1 pith:LAEOYYRH submitted 2026-05-14 cs.CV cs.AIcs.LG

Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers

Jakub Grzywaczewski , Dawid P{\l}udowski , Przemys{\l}aw Biecek This is my paper
classification cs.CV cs.AIcs.LG
keywords vision-language modelsCLIPembeddings covariancespectral decompositionnoise subspacepruningrepresentational geometry
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper applies spectral decomposition to the covariance matrices of embeddings from contrastively pretrained vision-language models. This separates the latent space into a semantic signal component and a shared noise subspace that stays consistent across different data subsets. Removing the noise dimensions turns out to be largely harmless and sometimes improves results on downstream tasks. The work concludes that a substantial portion of the models' latent geometry arises from architecture-level noise rather than task-relevant semantics.

Core claim

Spectral decomposition of the covariance matrix of VLM embeddings isolates a multi-modal semantic signal from a shared noise subspace that exhibits strong subgroup invariance; pruning the noise dimensions is mainly harmless and can preserve or actively improve performance on downstream tasks.

What carries the argument

Spectral decomposition of the embeddings covariance matrix, used to isolate and remove the shared noise subspace.

Load-bearing premise

The dimensions flagged by the decomposition truly contain only non-semantic shared noise and can be removed without discarding information needed for any downstream task.

What would settle it

A concrete downstream task or dataset where removing the identified noise dimensions produces a clear drop in performance would show the subspace is not harmless to prune.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Pruning the shared noise dimensions preserves or improves accuracy on standard downstream vision-language tasks.
  • The noise subspace remains invariant when the same model is evaluated on distinct data subsets.
  • A large fraction of the latent geometry in these models is governed by shared architectural artifacts rather than semantic content.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same covariance decomposition might reveal comparable noise subspaces in other contrastively trained transformer families.
  • Systematic pruning of the noise dimensions could serve as a lightweight way to compress embeddings for deployment.
  • The separation between signal and noise subspaces offers a route to diagnose and reduce certain forms of representational bias.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims that spectral decomposition of the covariance matrix of embeddings from contrastively pretrained VLMs (e.g., CLIP) separates a multi-modal semantic signal from a shared noise subspace; this subspace exhibits strong invariance across data subgroups, and pruning the noise dimensions is largely harmless or beneficial for downstream task performance, implying that a substantial fraction of VLM latent geometry is architecture-level noise rather than task-relevant semantics.

Significance. If the decomposition reliably isolates non-semantic noise, the work would supply mechanistic insight into VLM representational structure and a practical route to cleaner, lower-dimensional features. The reported subgroup invariance and pruning outcomes constitute concrete empirical observations that could inform future analyses of contrastive models.

major comments (2)
  1. [§4] §4 (Pruning Experiments): the claim that pruning the identified subspace is 'mainly harmless' and isolates non-semantic noise would require a control that prunes an equal number of random dimensions (or dimensions from an orthogonal subspace) to rule out that performance preservation simply reflects general dimensionality reduction; without this, the mechanistic interpretation that the subspace contains only artifactual noise rather than shared semantics remains under-supported.
  2. [§3.2] §3.2 (Invariance Analysis): subgroup invariance demonstrates stability of the subspace but does not establish its semantic content; an attribute-probing experiment (e.g., linear classification of semantic properties on the removed components) would be needed to distinguish noise from task-relevant directions.
minor comments (2)
  1. Notation for the covariance matrix and its eigenspectrum should be defined explicitly at first use to avoid ambiguity between sample and population quantities.
  2. Figure captions for the eigenspectrum plots should state the exact number of embeddings and data subsets used in each panel.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The two major comments identify valid gaps in the current empirical support. We address each below and will revise the manuscript to incorporate the suggested controls and experiments.

read point-by-point responses

  1. Referee: [§4] §4 (Pruning Experiments): the claim that pruning the identified subspace is 'mainly harmless' and isolates non-semantic noise would require a control that prunes an equal number of random dimensions (or dimensions from an orthogonal subspace) to rule out that performance preservation simply reflects general dimensionality reduction; without this, the mechanistic interpretation that the subspace contains only artifactual noise rather than shared semantics remains under-supported.

    Authors: We agree that the current pruning results would be strengthened by an explicit control. In the revised manuscript we will add experiments that prune an equal number of randomly chosen dimensions (and, separately, dimensions from an orthogonal complement of the identified subspace) and directly compare downstream task performance against the noise-subspace pruning condition. This will allow us to quantify whether performance preservation is specific to the identified directions or is a generic effect of dimensionality reduction. revision: yes

  2. Referee: [§3.2] §3.2 (Invariance Analysis): subgroup invariance demonstrates stability of the subspace but does not establish its semantic content; an attribute-probing experiment (e.g., linear classification of semantic properties on the removed components) would be needed to distinguish noise from task-relevant directions.

    Authors: The referee is correct that invariance alone does not directly demonstrate absence of semantic content. While the downstream-task results already provide indirect evidence (performance is preserved or improved after removal), we will add the requested attribute-probing analysis in the revision: linear probes will be trained on the removed components to predict a range of semantic attributes (object categories, attributes, relations) and we will report the resulting accuracies relative to probes trained on the retained semantic subspace. This will supply a direct test of whether the pruned directions encode task-relevant semantics. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical decomposition and pruning results stand on external observations

full rationale

The abstract and description present spectral decomposition of covariance matrices as a method to separate signal from noise, followed by empirical checks of subgroup invariance and downstream performance after pruning. No equations, self-definitions, fitted parameters renamed as predictions, or self-citation chains are shown that would make any claimed result equivalent to its inputs by construction. The central claims rest on observable stability and task metrics rather than any reduction to prior fitted quantities or author-specific uniqueness theorems.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the unverified premise that spectral decomposition isolates non-semantic noise; no free parameters or invented entities are stated in the abstract.

axioms (1)
  • domain assumption Covariance eigenspectrum decomposition separates semantic signal from shared non-semantic noise in VLM embeddings
    This assumption underpins the identification of the 164-dimensional noise subspace and the pruning results.

reviewed 2026-06-30 · how reviews work

0 comments
read the original abstract

Contrastively pre-trained Vision-Language Models (VLMs) serve as powerful feature extractors. Yet, their shared latent spaces are prone to structural anomalies and act as repositories for non-semantic, multi-modal noise. To address this phenomenon, we employ spectral decomposition of covariance matrices to decompose the VLM latent space into a multi-modal semantic signal component and a shared noise subspace. We observe that this noise geometry exhibits strong subgroup invariance across distinct data subsets. Crucially, pruning these shared noise dimensions is mainly harmless, preserving or actively improving downstream task performance. By isolating true semantic signals from artifactual noise, this work provides new mechanistic insights into the representational structure of modern VLMs, suggesting that a substantial fraction of their latent geometry is governed by shared, architecture-level noise rather than task-relevant semantics alone.

Figures

Figures reproduced from arXiv: 2605.14893 by Dawid P{\l}udowski, Jakub Grzywaczewski, Przemys{\l}aw Biecek.

Figure 1
Figure 1. Figure 1: The eigenspectra of sample covariance matrices, com￾puted from CLIP ViT-L/14 embeddings for two unrelated Ima￾geNet classes, are highly similar (shown here as the log10 of the eigenvalues). Despite no apparent correlation nor common sam￾ples, their identified class-specific noise directions exhibit strong overlap, achieving mSCSA of 91% (Equation 3). 2013; Arditi et al., 2024; Lee et al., 2025), uncovering… view at source ↗
Figure 2
Figure 2. Figure 2: Progression of the log10 of eigenvalues of the average covariance matrix (Equation (2)) for all considered VLM back￾bones. All evaluated models exhibit a sharp dip in eigenvalues immediately after the shared noise threshold. 3. Methodology Following Jing et al. (2022), we begin by examining the empirical covariance matrices of the model embeddings. Let zi ∈ R d represent the model’s embedding of an input x… view at source ↗
Figure 3
Figure 3. Figure 3: Overlap of the noise dimensions, computed using the average covariance matrix, and the lower eigenvectors of each ImageNet class covariance. For CLIP ViT-L/14 and SigLIP2 ViT￾L/16, the biggest models in their families, this overlap is very high at above 90%. 3 [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The distribution of the difference between the cosine sim￾ilarities of image-caption pairs from LAION-2B after and before the noise-removing projection. We can observe that in all cases the significantly largest portion of the probability mass is located on the right of 0, indicating an increase in similarity. 5. Discussion While a uniform noise threshold is conceptually elegant, we must acknowledge that i… view at source ↗
Figure 5
Figure 5. Figure 5: Progression of log eigenvalues for image, text, and average sample covariance matrix with the indicated noise threshold computed as the minimum of knee points for each model. 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Eigenvalue Index 10 8 6 4 2 Log10 Eigenvalue Image Covariance Eigenvalues 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Eigenvalue Index Text Covariance Eigenvalues 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Eigenvalue… view at source ↗
Figure 6
Figure 6. Figure 6: We observe that this does not change the noise threshold much nor the underlying eigenspectra. 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Eigenvalue Index 10 8 6 4 2 Log10 Eigenvalue Image Covariance Eigenvalues 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Eigenvalue Index Text Covariance Eigenvalues 0.0 0.2 0.4 0.6 0.8 1.0 Normalized Eigenvalue Index Average Covariance Eigenvalues CLIP ViT-B/16 CLIP ViT-B/32 CLIP ViT-L/14 … view at source ↗
Figure 7
Figure 7. Figure 7: Distributions of Top-5 accuracy computed on VLMs zero-shot ImageNet classification. The histograms present the performance of the VLMs after removing random directions from their embeddings. The amount of removed directions is equal to the discovered noisy directions (presented in the second column of [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Examples of images most activated in noise direction. One of the most common images is a placeholder for unavailable images. The noise directions are estimated using only the LAION dataset [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Examples of images most activated in noise direction. Images that contain content other than a placeholder for unavailable images are in the minority in this set. The noise directions are estimated using only the LAION dataset [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Examples of images most activated in noise direction. The noise directions are estimated using both ImageNet (images) and the LAION dataset (text). We argue that the topic of Marvel movies in noise space might be explained by the fact that ImageNet was collected before the latest Marvel movies (visible in the figure), which places them as out-of-distribution. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · 4 internal anchors

  1. [1]

    Is the Modality Gap a Bug or a Feature? A Robustness Perspective

    Chowers, R., Naparstek, O., Barzelay, U., and Weiss, Y . Is the modality gap a bug or a feature? a robustness perspective.arXiv preprint arXiv:2603.29080,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings

    Ethayarajh, K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. InProceedings of the 2019 Con- ference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 55– 65,

    work page 2019

  3. [3]

    Representation Learning with Contrastive Predictive Coding

    URL https://openreview. net/forum?id=VVVfuIcmKR. Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Oquab, M., Darcet, T., Moutakanni, T., V o, H

    Accessed: 2026-05-07. Oquab, M., Darcet, T., Moutakanni, T., V o, H. V ., Szafraniec, M., Khalidov, V ., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision.Transactions on Machine Learning Research,

    work page 2026

  5. [5]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y ., Mustafa, B., et al. Siglip 2: Multilingual vision-language encoders with improved semantic under- standing, localization, and dense features.arXiv preprint arXiv:2502.14786,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

    Yu, X., Xin, Y ., Zhang, W., Liu, C., Zhao, H., Hu, X., Yu, X., Qiao, Z., Tang, H., Yang, X., et al. Modality gap-driven subspace alignment training paradigm for multimodal large language models.arXiv preprint arXiv:2602.07026,

    work page internal anchor Pith review Pith/arXiv arXiv

This paper was first reviewed by grok-4.3 on June 30, 2026.