RECORD
CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
Machine Learning · Submitted 21 Oct 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks. Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features. This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data. We suggest to use modern Hopfield networks to tackle the problem of explaining away. Their retrieved embeddings have an enriched covariance structure derived from co-occurrences of features in the stored embeddings. However, modern Hopfield networks increase the saturation effect of the InfoNCE objective which hampers learning. We propose to use the InfoLOOB objective to mitigate this saturation effect. We introduce the novel "Contrastive Leave One Out Boost" (CLOOB), which uses modern Hopfield networks for covariance enrichment together with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks. | background 0.99 | 0.06 | 0.02 | asserted 1.00 |
| s2 | Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features. | background 0.82 | 0.13 | 0.03 | asserted 1.00 |
| s3 | This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data. | background 0.64 | 0.17 | 0.02 | asserted 1.00 |
| s4 | We suggest to use modern Hopfield networks to tackle the problem of explaining away. | contribution 0.59 | 0.40 | 0.05 | hedged possibility 0.76 |
| s5 | Their retrieved embeddings have an enriched covariance structure derived from co-occurrences of features in the stored embeddings. | method 0.70 | 0.29 | 0.03 | asserted 1.00 |
| s6 | However, modern Hopfield networks increase the saturation effect of the InfoNCE objective which hampers learning. | limitation 0.45 | 0.44 | 0.02 | asserted 1.00 |
| s7 | We propose to use the InfoLOOB objective to mitigate this saturation effect. | method 0.68 | 0.68 | 0.05 | asserted 0.61 |
| s8 | We introduce the novel "Contrastive Leave One Out Boost" (CLOOB), which uses modern Hopfield networks for covariance enrichment together with the InfoLOOB objective. | contribution 0.93 | 0.87 | 0.09 | asserted 0.99 |
| s9 | In experiments we compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. | method 0.96 | 0.95 | 0.03 | asserted 0.81 |
| s10 | CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets. | result 1.00 | 0.95 | 0.93 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v4
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks. Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features. This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data. We suggest to use modern Hopfield networks to tackle the problem of explaining away. Their retrieved embeddings have an enriched covariance structure derived from co-occurrences of features in the stored embeddings. However, modern Hopfield networks increase the saturation effect of the InfoNCE objective which hampers learning. We propose to use the InfoLOOB objective to mitigate this saturation effect. We introduce the novel "Contrastive Leave One Out Boost" (CLOOB), which uses modern Hopfield networks for covariance enrichment together with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.
Human review
How the literature uses it
30 arXiv papers cite this in 47 sentences
Papers citing it · 25 sentences without a confident label
Uses its method or tool 1 paper
-
Following prior work [ref] , using a foundation model for all approaches we first compute sample input embeddings (which we denote as sample embeddings) for all samples we wish to classify, and $C$ class label embeddings (which we denote as class embeddings) for all $C$ classes.
Contrastive Adapters for Foundation Model Group Robustness · Empirical findings of poor foundation model group robustness · 0.93
Cites it as background 16 papers
-
Vision-language pre-training has witnessed significant progress in the last few years, which generally learns an image encoder and a text encoder jointly [ref] .
CLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose · Vision-language models · 0.97
-
Notably, modern Hopfield models [ref] [For an in-depth tutorial, see [ref] .] demonstrate not only a strong connection to transformer attention mechanisms in deep learning but also superior performance and a theoretically guaranteed exponential memory capacity Their applicability spans diverse areas such as large language models [ref] , immunology [ref] , time series forecasting [ref] , reinforcement learning [ref] , and vision models [ref] .
BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model · Related Works · 0.92
-
Recently, a resurgence of interest in Hopfield models within the machine learning field is attributed to developments in understanding memory storage capacities [ref] , innovative architecture [ref] , and their biological plausibility [ref] .
Nonparametric Modern Hopfield Models · Related Works · 0.90
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.