REVIEW 3 cited by
Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Vision-language models (VLMs) allow to embed texts and images in a shared representation space. However, it has been shown that these models are subject to a modality gap phenomenon meaning there exists a clear separation between the embeddings from one modality and another in the embedding space. While this misalignment is detrimental for downstream tasks such as multimodal retrieval, multimodal clustering or zero-shot classification, etc. no generic and practical methods have so far been proposed to assess it precisely and even reduce it. We therefore propose novel measures and effective techniques (spectral- and optimal transport-based methods) to achieve this goal. Extensive experiments conducted on several image-text datasets and models demonstrate their effectiveness and beneficial effects on downstream tasks. Our code is available at the URL provided in the paper's abstract.
Forward citations
Cited by 3 Pith papers
-
On the modality gap and the contrastive loss in multi-modal representation learning
InfoNCE with independent encoders actively creates a modality gap at low temperature; mixing intra- and inter-modality negatives (xNCE) removes the gap while improving zero-shot transfer.
-
PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning
Keeping only the first 12 layers of Qwen2-VL plus self-distillation and a modality-aware contrastive loss yields a 3B unified multimodal retriever within 1.8 points of the 7B model on M-BEIR.
-
Asynchronous Federated Learning with non-convex client objective functions and heterogeneous dataset
A claimed asynchronous federated learning method with staleness-aware aggregation and dynamic learning rates targets convergence on nonconvex, heterogeneous data, but the submitted text does not support verification.
Discussion (0). Continue with ORCID to comment.