Common Data Properties Limit Object-Attribute Binding in CLIP

Bijay Gurung; David T. Hoffmann; Thomas Brox

REVIEW 3 major objections 5 minor 1 cited by

Common Data Properties Limit Object-Attribute Binding in CLIP

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that CLIP's object-attribute binding failure is caused by properties of the training data—sparse captions, few multi-object images, and captioners' saliency bias—rather than by the model architecture or loss.

desk verdict Careful synthetic study that convincingly shows data properties control CLIP binding, but the saliency claim is confounded and the real-data transfer is asserted rather than shown. read the letter →

arxiv 2507.07985 v2 pith:UUVPTLZW submitted 2025-07-10 cs.CV

Bijay Gurung , David T. Hoffmann , Thomas Brox This is my paper · ORCID

classification cs.CV

keywords CLIPobject-attributebindingbag-of-wordsrepresentationsyntheticdatasetsaliencybiascontrastivelearningcompositionalitydataproperties

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that CLIP's well-known failure to bind attributes to objects—confusing “a yellow submarine and a blue bus” with “a blue submarine and a yellow bus”—is caused mainly by properties of the training data, not by the contrastive loss or the architecture. Using a fully synthetic dataset called MADMAN, the authors vary four data properties one at a time and measure a two-alternative forced-choice binding accuracy. With data properties set to values estimated from real web captions, small CLIP models stay near chance even when batch size and model size are scaled up; with an "ideal" data configuration, the same models reach near-perfect binding. The paper concludes that CLIP can learn object-attribute binding, but common properties of natural image-caption data—especially saliency bias, sparse captions, and few multi-object images—prevent it. This matters because it redirects the search for a fix from architecture and loss design toward data curation.

What carries the argument

The central object is MADMAN (Multi-Attribute and Digit for Multi-Attribute biNding), a synthetic dataset built from transformed MNIST digits placed in a 3x3 grid, two per image, with template captions that chain attributes before each digit class. It makes four data properties independently controllable: the probability that an image contains two objects, the probability that a two-object image has both objects captioned, the distribution of the number of attributes per object in the caption, and the presence of a saliency bias (a center-placed object that is always captioned and mentioned first). The load-bearing evaluation is a forced-choice binding-accuracy score: for each attribute that is recognized in both objects, the authors swap that attribute between the two objects' captions and check that the model assigns higher similarity to the ground-truth caption. Realistic parameter values come from manual annotation of 100 CC12M image-caption pairs, which yields the estimates of about 95% multi-object images, 60% two-object captions, expected attributes per object of 0.57, and saliency in about 90% of images.

What would settle it

Take a large web-scale corpus, automatically re-caption it so that every visible object is mentioned, the expected number of attributes per object is around 3.5, and no single object is systematically favored, then retrain a CLIP-scale model and measure binding accuracy on SugarCrepe-style swapped-caption tests. If binding accuracy does not rise well above the current near-chance level, the paper's transfer from MADMAN to real data fails.

Watch

Extended reading notes

Core claim

The central claim, stated in Section 4.1, is that "CLIP models are able to learn object-attribute binding but properties of the data inhibit this." Trained on a realistic MADMAN configuration—two objects in about 95% of images, both objects captioned in about 60% of two-object cases, an expected 0.57 attributes per object in captions, and saliency bias in roughly 90% of images—the model's binding accuracy sits at or near chance for all six attributes, and raising batch size to 256 or embedding size to 256 changes little. Varying the properties individually shows a steady rise in binding as multi-object images and multi-object captions become more frequent; an inverted-U curve in attributes per object, with a peak near 3.5; and a steep drop as saliency bias exceeds 0.75. Combining all optimal settings yields above 90% binding for most attributes and near 100% with a larger batch and model. Hard-negative training (NegCLIP) improves binding on realistic data but remains far below simply switching to ideal data. Out-of-distribution tests on unseen object-attribute combinations show the same trends, indicating the model learned a compositional representation rather than memorized combinations.

Load-bearing premise

The load-bearing premise is that a small CLIP trained on synthetic images behaves like a large CLIP trained on real web data, and that four data-property estimates taken from 100 manually annotated web images represent natural data; if either fails, the claim that data properties are the main cause of CLIP's binding problem may not hold for real models.

Editorial extensions

If this is right

If the central claim is right, then improving CLIP's binding on real data is mainly a data-curation problem: re-captioning or filtering web-scraped corpora to describe all objects, include a moderate number of attributes per object, and reduce saliency bias should transfer the gains seen on MADMAN.
Batch-size scaling and hard-negative mining are not the cure: the paper shows these produce at best small gains on realistic data, so future fixes should focus on data rather than only on loss design.
Saliency bias is a candidate primary cause on natural data: at realistic levels above 0.75, adding saliency to otherwise ideal data drops binding accuracy from about 90–95% to roughly 51–62%, and it diminishes the influence of the other three properties.
Because the same trends appear on held-out object-attribute combinations, the learned representations are compositional rather than memorized class labels, so the data-property effects are expected to generalize beyond the synthetic vocabulary.
The inverted-U in attribute density predicts an optimal caption length on real data; captions that are too sparse (like most web captions) or too dense (a bag-of-words can already disambiguate) both hurt binding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors do not test this on real data, but their account implies that re-captioning a large web corpus to mention every object, keep attribute density near 3.5 per object, and break saliency bias should produce a measurable jump in CLIP's binding accuracy on compositional benchmarks.
A corollary they leave implicit is that original human captions are not neutral descriptions: because humans preferentially caption salient objects, even perfectly matched image-caption pairs carry a bias that teaches the model to ignore binding.
Another extension: the inverse-U predicts an optimal caption length, so caption-enrichment methods should be tuned to a target density rather than made as complete as possible.
If data properties dominate, comparisons between CLIP-like models trained on different corpora are confounded by differences in these four properties, and controlling them would isolate architectural contributions.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line up front: this is a careful, controlled study that makes a real contribution to the CLIP binding literature. The authors build a synthetic dataset (MADMAN) and show that four data properties—fraction of multi-object images, fraction of multi-object captions, attributes per object, and saliency bias—each move binding accuracy. The inverse U-shaped effect of attribute density is new and believable. So is the result that hard negatives alone don't get you there. The experiments are well executed: three seeds, confidence intervals, batch/embedding size sweeps, OOD checks, and they report unfiltered versions of the binding curves as well. The citation pattern is fair; they engage with the hard-negative and architecture-centric prior work without overclaiming novelty. Code is released. I'd trust the within-synthetic-regime claims.

The soft spots are at the boundary between the synthetic regime and real data. The authors want to conclude that data properties, not architecture or loss, limit CLIP's binding on natural data. That conclusion leans on two things: a stated but unverified transfer assumption (TinyCLIP on MADMAN behaves like CLIP at scale), and realistic parameter estimates from 100 manually annotated CC12M images. They acknowledge the transfer gap in their limitations paragraph, which is honest, but the paper's abstract and conclusion lean on it more than the evidence supports.

The bigger issue, which I think a careful referee should push on, is the saliency result. Salient objects in MADMAN are always centered, always mentioned in the caption, and always mentioned first. The footnote in Section 4.2 even says the text encoder can infer which object is salient because it appears first. So the drop in binding accuracy as p(saliency) increases could come from a spurious center/order shortcut rather than from saliency as a captioning tendency. Natural captions don't have that deterministic positional rule. Since the paper singles out saliency as a key limiting factor—Table 6 shows it causes the largest single-factor drop—this confound cuts the headline claim. The other three data properties don't have this problem, and the multi-object and attribute-density results stand on their own.

Minor point: the 'ideal data' setup is chosen post hoc from the same curves used to demonstrate its success. That's not circular in a damaging way—they're identifying good data conditions, not testing a prediction—but it means the near-perfect numbers in Table 1 are a selected best case, not an independent confirmation.

Who's this for? People who work on CLIP and compositional reasoning, and anyone designing data filtering or re-captioning pipelines. It deserves a serious refereeing. My own recommendation: send it out, but make the authors untangle the saliency confound—e.g., vary saliency without changing caption order or spatial position—and either temper the real-data conclusion or add a check on a larger model.

Referee Report

3 major / 5 minor

Summary. The paper introduces MADMAN, a synthetic dataset built from transformed MNIST digits in a grid, to study how four data properties affect object-attribute binding in contrastive vision-language models. The properties are p(two-obj-img), p(two-obj-cap), expected number of attributes per object in a caption, and a saliency bias. The authors train small CLIP-like transformers on controlled variants and measure zero-shot binding accuracy with recognition filtering. They report that settings calibrated to 100 CC12M image-caption pairs produce near-chance binding, while a manually chosen 'ideal' setting produces almost perfect binding; they also report that batch size, embedding size, and hard negatives (NegCLIP) do not close the gap, and that the trends persist on out-of-distribution object-attribute combinations. They conclude that CLIP's binding failures are largely caused by common properties of the training data, with saliency bias singled out as a key factor.

Significance. The study is well-executed within its synthetic regime: three seeds with confidence intervals, hyperparameter sweeps across batch and embedding sizes, OOD checks, and unfiltered ablations are included, and code is provided. A data-centric testbed for binding is useful to the community and could redirect attention from loss and architecture fixes toward data curation. If the transfer assumption holds, the work would motivate filtering and re-captioning as concrete remedies. The main unvalidated link is the extrapolation from TinyCLIP on MADMAN to CLIP on natural data, and the saliency operationalization conflates several cues. These issues affect the headline causal attribution but not the internal consistency of the controlled trends.

major comments (3)

[Section 3.1, Fig. 4b, Table 6] The saliency manipulation is confounded. In MADMAN, a salient object is always placed in the center cell, is always mentioned in the caption, and is always mentioned first. The footnote in Section 4.2 concedes that the text encoder can infer the salient object from first-mention order. Thus increasing p(saliency) simultaneously increases center-position predictability, caption-inclusion rate, and word-order determinism, and the decline in binding accuracy in Fig. 4b and the large drop in the 'Ideal except Saliency bias' row of Table 6 cannot be uniquely attributed to saliency as a real-world captioning tendency. Natural image-caption data does not contain these deterministic positional and order cues. Please add ablations that disentangle the three factors, for example, randomize grid placement while keeping first mention and guaranteed inclusion, or randomize order and omission while keeping center placement, and report the binding curves for each sub-manipulation separately.
[Section 4.1 and Introduction] The headline conclusion that 'CLIP models are able to learn object-attribute binding but properties of the data inhibit this' is stated for CLIP, while the experiments train a small transformer (TinyCLIP, Appendix Table 4) on a synthetic dataset. The bridge is the assertion in the Introduction that 'the learning algorithm does not change its behavior fundamentally in larger scale training on more complex data,' plus the realistic parameter estimates from 100 manually annotated CC12M images (Appendix 5.2). A 100-image sample gives wide confidence intervals, and the annotation procedure for saliency in CC12M differs from the synthetic definition: in CC12M an image is labeled as having salient objects if there are any non-salient objects, whereas in MADMAN saliency means center placement plus guaranteed first mention. The paper should provide direct evidence for transfer, such as a real-data re-captioning or filtering experiment, or at least a larger-scale synthetic scaling study, otherwise the title and abstract generalize beyond the evidence.
[Section 4.2 and Table 1] The 'ideal data' setting is selected by reading off the best-performing value of each property from the same binding-accuracy curves that are then used to report the ideal result in Table 1. This makes the high binding in Table 1 a post-hoc optimum rather than an independent validation of the prescription. If the paper claims that 'only when the data expresses our identified data properties does CLIP learn almost perfect binding,' an independent check is needed, for example, choosing the best parameters on a development split and reporting performance on a held-out split, or performing a small random search over the four-dimensional property grid and reporting the out-of-sample best. Without such a check, the 'only when' phrasing overstates what the one-at-a-time sweeps establish.

minor comments (5)

[Throughout main text] The term 'CLIP' is used for a small transformer trained from scratch on MADMAN; this is only clarified as TinyCLIP in Appendix Table 4. State this at first use in the main text to avoid ambiguity.
[Appendix 5.3, Table 5] The note that the search was performed 'by chance partially in the optimal setup' is confusing; it appears that the base setup already equals the best setting for three of the four properties, so Table 5 does not isolate the individual contribution of each property. Please rewrite the note to explain what was and was not varied.
[Appendix Figure 7] The figure shows two distributions labeled E[na] = 3.50 with different shapes; one of the labels is likely a typo and should be corrected.
[Appendix 5.2] The realistic saliency annotation defines an image as having salient objects if there are any non-salient objects, which is not the same as the MADMAN parameter p(saliency). The mapping between the binary annotation and the synthetic probability should be explained explicitly.
[Section 3.2] The recognition-filtering threshold of 1.1 times chance level is arbitrary and is not sensitivity-analyzed; please report how many samples are retained per attribute and condition, and whether the main trends change with a stricter or looser threshold.

Circularity Check

1 steps flagged · score 3.0 of 10

Ideal-data claim is a selected optimum rather than an independent prediction; the core controlled sweeps remain non-circular.

fitted input called prediction [Section 4.2 ('The ideal data setting'); Table 1 caption; Abstract]
"Finally, we use the previous experiments to select the best data setup for each of the properties. We end up with: no saliency bias, p(multi-obj-img) = 1, p(multi-obj-cap) = 1 and expected number of Attributes-per-object-in-caption = 3.5. As shown in Tab. 1, combining the optimal setups for all our data properties indeed leads to high binding... Only when the data expresses our identified data properties does CLIP learn almost perfect binding... Ideal Data refers to the best performing dataset properties as found in Section 4.2."

The 'Ideal Data' setting is constructed by reading off the maxima of the same binding-accuracy curves that are then used to support the conclusion. Since the ideal values are definitionally the best-performing values from Section 4.2, the near-perfect binding reported for them is a selected optimum, not an independent prediction. The abstract 'Only when the data expresses our identified data properties does CLIP learn almost perfect binding' therefore reduces, for the ideal row, to 'when the data is set to the values that maximize binding, binding is maximized.' The non-tautological evidence is carried by the full parameter sweeps in Figs. 3-4 and the single-factor ablations in Table 6, not by the Ideal Data row itself.

full rationale

The central empirical content of the paper comes from controlled interventions: each of the four data properties is varied independently while others are held fixed, and binding accuracy is measured on held-out MADMAN evaluation sets. Those sweeps are not circular, because the data property values are inputs and the binding accuracy is an output measured under a fixed evaluation protocol. The one genuinely circular element is the 'Ideal Data' claim: the ideal settings are selected post hoc from the binding-accuracy curves, then presented as showing that ideal data yields almost perfect binding. This is a selection effect rather than an out-of-sample confirmation. The saliency-bias confound raised by the skeptic (salient objects are always centered, always captioned, and always mentioned first) is a serious validity concern for transferring the result to natural data, but it is not a circularity under the paper's own definitions: the paper measures the effect of its operationalized saliency bias rather than deriving the effect from the definition. Self-citation to the MAD dataset [25] is a data-building block and is not load-bearing for the central claim. The transfer assumption from synthetic to real data is explicitly acknowledged as an assumption, not disguised as a derivation. Overall, the main controlled trends are self-contained; only the 'ideal data' summary inflates a selected optimum into what reads like a predicted requirement.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

MADMAN is a synthetic instrument, not a postulated physical entity, so it is not listed as an invented entity. The free parameters are the realistic data-property values estimated from a small annotation and the ideal data values selected post hoc from the evaluation curves. The axioms capture the transfer and representativeness assumptions that the general CLIP conclusion depends on.

free parameters (5)

p(two-obj-img) realistic = 0.95
Estimated from manual annotation of 100 CC12M images (Appendix 5.2); used in the Realistic setup that shows poor binding.
p(two-obj-cap) realistic = 0.6
Estimated from the same 100-image annotation as the conditional probability that both objects appear in the caption.
E[na] realistic = 0.57
Estimated attribute density from the same annotation; most objects have zero attributes in captions.
saliency probability realistic = 0.9
Estimated as the fraction of CC12M images with at least one salient object; drives the conclusion that realistic saliency is detrimental.
ideal data setup = p(two-obj-img)=1, p(two-obj-cap)=1, E[na]=3.5, saliency=0
Selected post hoc from the binding-accuracy curves in Section 4.2, then shown to yield near-perfect binding; a selection effect rather than a pre-registered prediction.

assumptions (3)

domain assumption We do not expect that the learning algorithm changes its behavior fundamentally in larger scale training on more complex data.
Stated in the Introduction; this transfer premise lets the authors generalize from TinyCLIP on MADMAN to CLIP on natural data.
domain assumption The data properties measured on 100 manually annotated CC12M images are representative of natural image-caption data.
Section 3.1 and Appendix 5.2; the realistic parameters rest on a small sample with fuzzy object-counting rules.
domain assumption Saliency is adequately modeled as a centered object that is always mentioned first in the caption.
Section 3.1; this couples visual center bias with caption ordering, and the text encoder can exploit the ordering as a shortcut, so the saliency effect conflates two factors.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Common Data Properties Limit Object-Attribute Binding in CLIP." pith.science (2026). https://pith.science/paper/UUVPTLZW

@misc{pith2026250707985,
  author       = {Pith},
  title        = {Pith review of: Common Data Properties Limit Object-Attribute Binding in CLIP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UUVPTLZW}},
  note         = {Machine review of arXiv:2507.07985}
}

read the original abstract

Contrastive vision-language models like CLIP are used for a large variety of applications, such as zero-shot classification or as vision encoder for multi-modal models. Despite their popularity, their representations show major limitations. For instance, CLIP models learn bag-of-words representations and, as a consequence, fail to distinguish whether an image is of ``a yellow submarine and a blue bus'' or ``a blue submarine and a yellow bus''. Previous attempts to fix this issue added hard negatives during training or modified the architecture, but failed to resolve the problem in its entirety. We suspect that the missing insights to solve the binding problem for CLIP are hidden in arguably the most important part of learning algorithms: the data. In this work, we fill this gap by rigorously identifying the influence of data properties on CLIP's ability to learn binding using a synthetic dataset. We find that common properties of natural data such as low attribute density, incomplete captions, and the saliency bias, a tendency of human captioners to describe the object that is ``most salient'' to them, have a detrimental effect on binding performance. In contrast to common belief, we find that neither scaling the batch size, i.e., implicitly adding more hard negatives, nor explicitly creating hard negatives enables CLIP to learn reliable binding. Only when the data expresses our identified data properties does CLIP learn almost perfect binding.

Figures

Figures reproduced from arXiv: 2507.07985 by the authors.

**Figure 1.** (a) The object-attribute binding problem: CLIP can’t distinguish between two captions with swapped attributes. (b) We identify data properties that contribute to poor object-attribute binding. If 1) too few or too many attributes per object are in the caption, or 2) too few images with multiple objects or 3) caption mentions too few objects, 4) captioners focus more on salient objects, a bag-of-words (BOW) represent… view at source ↗

**Figure 2.** MADMAN example and evaluation protocols. To evaluate recognition, we create zero-shot classification prompts by changing the target attribute. To evaluate binding, we swap the target attribute to create a negative and perform zero-shot classification with the true and negative caption. In practice, we evaluate on captions with 3 and 4 attributes per object. 3.1 MADMAN: Fully Controllable Synthetic Dataset We show… view at source ↗

**Figure 3.** Influence of (a) Two-object-in-image-probability and (b) Twoobject-in-caption-probability on binding accuracy on MADMAN. Having more images and captions containing multiple objects improves binding. a large enough batch size sufficient to encourage CLIP to learn binding via hard negatives? We can explore these questions using MADMAN with relatively small batch sizes since the number of factors of variation is low.… view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: Influence of (a) Attributes-per-object-in-caption and (b) Saliency bias on binding accuracy on MADMAN. (a) Too few and too many Attributes-per-object-in-caption hamper the binding accuracy. For large Attributes-per-object-in-caption binding is not necessary anymore, as…

**Figure 5.** Figure 5: Experimental results are independent of most important hyperparameters of the experimental setup on MADMAN. We test various combinations of the embedding size and batch size. We observe the same trends for all data properties as found in our main setting, showing that…

**Figure 6.** Figure 6: Impact of the data properties on the binding for out-ofdistribution combinations on MADMAN-OOD. It follows similar trends as in-distribution (denoted by the dashed line) [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Distributions for number of attributes per object ( [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

**Figure 8.** Figure 8: Illustration of saliency bias in MADMAN. Salient objects always occupy [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

**Figure 9.** Figure 9: Samples from the MADMAN dataset. The images are generated from [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

**Figure 10.** Figure 10: Impact of the data properties on the attribute recognition for in [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]

**Figure 11.** Figure 11: Impact of the data properties on the attribute recognition for out-of [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]

**Figure 12.** Figure 12: Influence of (a) Two-object-in-image-probability and (b) Twoobject-in-caption-probability on binding accuracy on MADMAN, evaluated without recognition filtering. The trend here is the same i.e. having more images and captions containing multiple objects improves bin…

**Figure 13.** Figure 13: Influence of (a) Attributes-per-object-in-caption and (b) Saliency bias on binding accuracy on MADMAN evaluated without recognition filtering. It follows the same trend as in [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS)
cs.CV 2025-08 conditional novelty 6.0 of 10

SLiCS learns group-structured non-negative dictionaries that disentangle dense image embeddings into concept components, improving concept-filtered retrieval and enabling image-to-prompt generation.

Reference graph

Works this paper leans on

36 extracted references · 29 canonical work pages · cited by 1 Pith paper

[1]

arXiv preprint arXiv:2502.14113 (2025)

Assouel, R., Astolfi, P., Bordes, F., Drozdzal, M., Romero-Soriano, A.: Object-centric binding in contrastive language-image pretraining. arXiv preprint arXiv:2502.14113 (2025)

arXiv 2025
[2]

arXiv preprint arXiv:2407.07726 (2024)

Beyer, L., Steiner, A., Pinto, A.S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., et al.: Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 (2024)

arXiv 2024
[3]

Advances in Neural Information Processing Systems 36, 45020–45054 (2023)

Bordes, F., Shekhar, S., Ibrahim, M., Bouchacourt, D., Vincent, P., Morcos, A.: Pug: Photorealistic and semantically controllable synthetic data for rep- resentation learning. Advances in Neural Information Processing Systems 36, 45020–45054 (2023)

work page 2023
[4]

IEEE Transactions on Image Processing22(1), 55–69 (2012)

Borji, A., Sihite, D.N., Itti, L.: Quantitative analysis of human-model agree- ment in visual saliency modeling: A comparative study. IEEE Transactions on Image Processing22(1), 55–69 (2012)

work page 2012
[5]

Vision research91, 62–77 (2013)

Borji, A., Sihite, D.N., Itti, L.: What stands out in a scene? a study of human explicit saliency judgment. Vision research91, 62–77 (2013)

work page 2013
[6]

Jour- nal of Machine Learning Research20(178), 1–29 (2019)

Castro, D.C., Tan, J., Kainz, B., Konukoglu, E., Glocker, B.: Morpho-mnist: Quantitative assessment and diagnostics for representation learning. Jour- nal of Machine Learning Research20(178), 1–29 (2019)

work page 2019
[7]

In: CVPR (2021)

Changpinyo,S.,Sharma,P.,Ding,N.,Soricut,R.:Conceptual12M:Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In: CVPR (2021)

work page 2021
[8]

In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=mWVoBz4W0u

Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thap- liyal, A.V., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B.K., Ruiz, C.R., Steiner, A.P., Angelova, A., Zhai, X., Houlsby, N...

work page 2023

Show all 36 references

[9]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gor- don, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp...

2023
[10]

Advances in Neural Information Processing Systems36, 76137–76150 (2023)

Doveh, S., Arbelle, A., Harary, S., Herzig, R., Kim, D., Cascante-Bonilla, P., Alfassy, A., Panda, R., Giryes, R., Feris, R., et al.: Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances in Neural Information Processing Systems36, 76137–76150 (2023)

2023
[11]

In: Proceedings of the Common Data Properties Limit Object-Attribute Binding in CLIP 15 IEEE/CVF Conference on Computer Vision and Pattern Recognition

Doveh, S., Arbelle, A., Harary, S., Schwartz, E., Herzig, R., Giryes, R., Feris, R., Panda, R., Ullman, S., Karlinsky, L.: Teaching structured vision & language concepts to vision & language models. In: Proceedings of the Common Data Properties Limit Object-Attribute Binding i...

2023
[12]

Advances in Neural Information Processing Systems 37, 17972–18018 (2024)

Dumpala, S.H., Jaiswal, A., Shama Sastry, C., Milios, E., Oore, S., Sajjad, H.: Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations. Advances in Neural Information Processing Systems 37, 17972–18018 (2024)

2024
[13]

Advances in neural information processing systems36, 31096–31116 (2023)

Hsieh, C.Y., Zhang, J., Ma, Z., Kembhavi, A., Krishna, R.: Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems36, 31096–31116 (2023)

2023
[14]

In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Zitnick, C.L., Girshick, R.: CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1988–1997. IEEE, Ho...

2017
[15]

arXiv preprint arXiv:2502.03566 (2025)

Koishigarina,D.,Uselis,A.,Oh,S.J.:Clipbehaveslikeabag-of-wordsmodel cross-modally but not uni-modally. arXiv preprint arXiv:2502.03566 (2025)

2025
[16]

ATT Labs [Online]

LeCun, Y., Cortes, C., Burges, C.: Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist2 (2010)

2010
[17]

In: Graham, Y., Purver, M

Lewis, M., Nayak, N., Yu, P., Merullo, J., Yu, Q., Bach, S., Pavlick, E.: Does CLIP bind concepts? probing compositionality in large image models. In: Graham, Y., Purver, M. (eds.) Findings of the Association for Compu- tational Linguistics: EACL 2024. pp. 1487–1500. Associati...

2024
[18]

In: International Conference on Learning Represen- tations (2022), https://openreview.net/forum?id=zq1iJkNk3uN

Li, Y., Liang, F., Zhao, L., Cui, Y., Ouyang, W., Shao, J., Yu, F., Yan, J.: Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In: International Conference on Learning Represen- tations (2022), https://openreview.net/forum?id=zq1iJkNk3uN

2022
[19]

Ma, Z., Hong, J., Gul, M.O., Gandhi, M., Gao, I., Krishna, R.: Crepe: Can vision-language foundation models reason compositionally? In: Proceedings oftheIEEE/CVFConferenceonComputerVisionandPatternRecognition. pp. 10910–10921 (2023)

2023
[20]

arXiv preprint arXiv:2112.10741 (2021)

Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)

2021 arXiv
[21]

Advances in neural information processing sys- tems 37, 32731–32760 (2024)

Patel, M., Kusumba, N.S.A., Cheng, S., Kim, C., Gokhale, T., Baral, C., et al.: Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives. Advances in neural information processing sys- tems 37, 32731–32760 (2024)

2024
[22]

In: International conference on machine learning

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sas- try, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) ...

2021
[23]

arXiv preprint arXiv:2204.06125 1(2), 3 (2022)

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchi- cal text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)

2022 arXiv
[24]

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High- resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

2022
[25]

In: International Conference on Learning Representations (ICLR) (2025)

Schrodi, S., Hoffmann, D.T., Argus, M., Fischer, V., Brox, T.: Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language model. In: International Conference on Learning Representations (ICLR) (2025)

2025
[26]

In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Tang, Y., Yamada, Y., Zhang, Y., Yildirim, I.: When are lemons purple? the concept association bias of vision-language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 14333–14348 (2023)

2023
[27]

In: Proceedings of the IEEE/CVF International Conference on Computer Vision

Trager, M., Perera, P., Zancato, L., Achille, A., Bhatia, P., Soatto, S.: Lin- ear spaces of meanings: compositional structures in vision-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15395–15404 (2023)

2023
[28]

In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=5BCFlnfE1g

Xu, H., Xie, S., Tan, X., Huang, P.Y., Howes, R., Sharma, V., Li, S.W., Ghosh, G., Zettlemoyer, L., Feichtenhofer, C.: Demystifying CLIP data. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=5BCFlnfE1g

2024
[29]

Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? In: International Conference on Learning Representations (2023), https://openreview.net/forum?id=KRLUvxh8uaX

2023
[30]

In: Proceedings of the IEEE/CVF international confer- ence on computer vision

Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: Proceedings of the IEEE/CVF international confer- ence on computer vision. pp. 11975–11986 (2023)

2023
[31]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L.H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al.: Regionclip: Region-based language-image pretraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16793–16803 (2022) Common...

2022
[32]

Select oi=(1or2)objectstoaddtotheimage.Wevarythisusingaparameter p(two-obj-img) that determines whether the second object is included or not
[33]

they are not inserted in the same cells

Randomly select the n cell locations within the 3x3 grid and place the objects making sure they do not overlap, i.e. they are not inserted in the same cells. Creating the captions. When creating the corresponding captions for the image, we want to be able to vary how much of t...
[34]

We vary this by a parame- ter p(two-obj-cap|two-obj-img = True) that determines whether the second object is included or not

Select ot=(1 or 2) objects to add to the caption. We vary this by a parame- ter p(two-obj-cap|two-obj-img = True) that determines whether the second object is included or not. Note: if there is only one object in the image, we can take only that one object in the caption
[35]

– We vary the number of attributes (na) used to describe an object by sampling it from a categorical distribution (see Fig

Create a caption for each object using the attributes and the object class as words. – We vary the number of attributes (na) used to describe an object by sampling it from a categorical distribution (see Fig. 7). – When stringing together the words, we separate them with a spa...
[36]

Dataset Properties of interest.With the above data generation process, we can vary the data properties of interest

Combine the captions for theot objects separated by the wordand. Dataset Properties of interest.With the above data generation process, we can vary the data properties of interest. Notably, we can also vary image and captions independently to create an imbalance in the informa...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:2502.14113 (2025)

Assouel, R., Astolfi, P., Bordes, F., Drozdzal, M., Romero-Soriano, A.: Object-centric binding in contrastive language-image pretraining. arXiv preprint arXiv:2502.14113 (2025)

arXiv 2025

[2] [2]

arXiv preprint arXiv:2407.07726 (2024)

Beyer, L., Steiner, A., Pinto, A.S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., et al.: Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 (2024)

arXiv 2024

[3] [3]

Advances in Neural Information Processing Systems 36, 45020–45054 (2023)

Bordes, F., Shekhar, S., Ibrahim, M., Bouchacourt, D., Vincent, P., Morcos, A.: Pug: Photorealistic and semantically controllable synthetic data for rep- resentation learning. Advances in Neural Information Processing Systems 36, 45020–45054 (2023)

work page 2023

[4] [4]

IEEE Transactions on Image Processing22(1), 55–69 (2012)

Borji, A., Sihite, D.N., Itti, L.: Quantitative analysis of human-model agree- ment in visual saliency modeling: A comparative study. IEEE Transactions on Image Processing22(1), 55–69 (2012)

work page 2012

[5] [5]

Vision research91, 62–77 (2013)

Borji, A., Sihite, D.N., Itti, L.: What stands out in a scene? a study of human explicit saliency judgment. Vision research91, 62–77 (2013)

work page 2013

[6] [6]

Jour- nal of Machine Learning Research20(178), 1–29 (2019)

Castro, D.C., Tan, J., Kainz, B., Konukoglu, E., Glocker, B.: Morpho-mnist: Quantitative assessment and diagnostics for representation learning. Jour- nal of Machine Learning Research20(178), 1–29 (2019)

work page 2019

[7] [7]

In: CVPR (2021)

Changpinyo,S.,Sharma,P.,Ding,N.,Soricut,R.:Conceptual12M:Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In: CVPR (2021)

work page 2021

[8] [8]

In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=mWVoBz4W0u

Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., Kolesnikov, A., Puigcerver, J., Ding, N., Rong, K., Akbari, H., Mishra, G., Xue, L., Thap- liyal, A.V., Bradbury, J., Kuo, W., Seyedhosseini, M., Jia, C., Ayan, B.K., Ruiz, C.R., Steiner, A.P., Angelova, A., Zhai, X., Houlsby, N...

work page 2023