REVIEW 6 cited by
Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
CLIP embeddings have demonstrated remarkable performance across a wide range of multimodal applications. However, these high-dimensional, dense vector representations are not easily interpretable, limiting our understanding of the rich structure of CLIP and its use in downstream applications that require transparency. In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, allowing for the decomposition of representations into semantic concepts. We formulate this problem as one of sparse recovery and propose a novel method, Sparse Linear Concept Embeddings, for transforming CLIP representations into sparse linear combinations of human-interpretable concepts. Distinct from previous work, SpLiCE is task-agnostic and can be used, without training, to explain and even replace traditional dense CLIP representations, maintaining high downstream performance while significantly improving their interpretability. We also demonstrate significant use cases of SpLiCE representations including detecting spurious correlations and model editing.
Forward citations
Cited by 6 Pith papers
-
Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach
A new framework combines AI-derived concept embeddings with high-dimensional selective inference to enable statistically principled, interpretable discovery from unstructured data in empirical economics.
-
Beyond Accuracy: Metrics that Uncover What Makes a 'Good' Visual Descriptor
Two label-free metrics, DINO alignment and CLIP similarity, rank text descriptor sets in line with downstream accuracy and track iterative refinement.
-
Quantifying Structure in CLIP Embeddings: A Statistical Framework for Concept Interpretation
A rotation-sensitivity hypothesis test plus Varimax rotation produces sparse concept dictionaries from CLIP embeddings and improves worst-group accuracy after spurious concept removal.
-
Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models
Class-name-free accuracy of LLM descriptors is 15.5% versus 59.5% with class names; selecting attributes from target images raises attribute-only accuracy to 23.8% (45.5% uncapped).
-
The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Model
CLIP embeddings are modeled as a mixture of von Mises-Fisher distributions on the unit sphere, improving out-of-distribution detection and semantic decomposition over single-Gaussian baselines.
-
Model Science: getting serious about verification, explanation and control of AI systems
Proposes 'Model Science' as a model-centric paradigm for AI with four pillars: verification, explanation, control, and interface.
Discussion (0). Continue with ORCID to comment.