Pith. sign in

REVIEW 3 major objections 6 minor 33 references

Dual-encoder vision-language models do not execute Boolean operators in their similarity scores; the paper shows the failure is interface-level and fixes it with a training-free score edit called LCSE, lifting FACTOR-Bench accuracy from 58.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:46 UTC pith:3GMPPKU6

load-bearing objection A careful, well-scoped paper that identifies a real interface-level failure and offers a training-free fix; the main risk is the global calibration assumption, but the evidence so far supports the claim. the 3 major comments →

arxiv 2607.23052 v1 pith:3GMPPKU6 submitted 2026-07-25 cs.CV cs.CLcs.LG

Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

Sultan Alshehri , Zhantao Yang , Han Zhang , Marios Savvides This is my paper
classification cs.CV cs.CLcs.LG
keywords vision-language modelsdual encodersBoolean operatorsnegationcompositional constraintssimilarity scoringCLIPFACTOR-Bench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to prove that the failure of CLIP-style models on logical queries is not in what the encoders represent but in how their similarity scores combine evidence: holistic scores approximate the mean of per-concept evidence, so negation, conjunction, and NOR are not executed truth-functionally. The proposed method, LCSE, separates evidence extraction from constraint execution by parsing a query into concepts and polarities, scoring each concept with the frozen encoder, applying 1-p for negation and power-mean aggregation for AND/OR, then adding only the logit-space correction to the original similarity. If this is right, fine-tuning for negation is the wrong bottleneck, and a lightweight external score edit can enforce Boolean semantics without retraining or degrading standard retrieval. On the introduced FACTOR-Bench, LCSE raises accuracy from 58.3% (holistic CLIP) to 85.5%, and to 90.7% on SigLIP 2, while NegBench COCO MCQ improves from 27.2% to 65.2%.

Core claim

The central claim: dual-encoder similarity behaves like a bag-of-concepts model - the holistic score approximates the mean of atomic concept scores regardless of operator, so Boolean constraints are not executed. Span-residual decomposition shows why: the span (balanced linear combination of atomic embeddings) dominates and pools, while operator residual directions are weak or misaligned (inverted for NOR). Since the encoder already yields reliable atomic evidence, the bottleneck is execution, not representation; fine-tuned negation models still fail polarity tests. LCSE fixes this: parse concepts, calibrate similarities to presence probabilities via sigmoid, use 1-p for negation, power-mean

What carries the argument

The load-bearing mechanism is the span-residual decomposition: a compound text embedding splits into a span component (projection onto atomic concept embeddings, which produces mean-like pooling) and an orthogonal residual (operator-specific signal that is weak or systematically inverted under dot-product scoring). LCSE replaces implicit mean pooling with explicit factored inference: calibrated sigmoid presence probabilities p_i = sigm(beta(s_i - mu)), polarity complement 1-p_i for negation, power-mean aggregation M_gamma (gamma<0 for conjunction, gamma>0 for disjunction), and an additive score edit in logit space that is nonzero only when the constraint aggregation differs from simple avera

Load-bearing premise

Everything hinges on one global sigmoid calibration (mu, beta) per backbone: the sign of every negation correction is set by whether a concept's similarity falls above or below a single threshold mu, so if a concept's score distribution lies on the wrong side of that threshold the complement 1-p steers rankings the wrong way and LCSE falls below holistic scoring on that concept - the paper's robustness evidence covers aggregate benchmarks, not per-concept failure, and its own

What would settle it

Take a fixed backbone and a held-out set of concepts outside the 80 COCO categories (or rare/abstract ones). For each concept, estimate the per-concept optimal threshold mu_c that maximizes negation accuracy under LCSE; if a substantial fraction of concepts have |mu_c - mu_global| > 0.08, the single-calibration premise fails because for those concepts the LCSE correction sign is wrong and per-concept NOT accuracy should drop below holistic scoring. A direct check: measure FACTOR-Bench NOT accuracy restricted to concepts whose median present-similarity is below mu=0.22.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Negation-aware fine-tuning is not the way to fix compositional failures; separating evidence extraction from constraint execution lets frozen encoders satisfy Boolean constraints without retraining.
  • Better atomic evidence translates directly into better constraint execution: LCSE on SigLIP 2 reaches 90.7% on FACTOR-Bench, versus 85.5% on CLIP.
  • Standard retrieval is preserved because for the roughly 97.5% of natural captions without explicit operators or negation the correction vanishes; COCO R@5 stays at 55.2% versus 55.3% for holistic CLIP, with rank correlation 0.999.
  • Anti-monotone constraints the holistic interface inverts - negation, NOR, and exclusion - are where LCSE has the largest gains, rising from 54-66% to 82-88% on CLIP.
  • The same factored template extends to 3-4 concept and nested Boolean queries by recursive power-mean aggregation, improving nested accuracy from 51.0% to 80.5%.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The factored template may transfer to non-Boolean compositional structure - attribute binding, relations, counting - wherever atomic evidence can be scored independently, though the paper only claims Boolean two-to-four-concept operators.
  • Inference: The single global calibration threshold (mu) per backbone is the part most likely to fail on rare or abstract concepts whose score distributions sit on the wrong side of that threshold; a per-concept or rank-based calibration would be a natural robustness extension.
  • Inference: Reading holistic similarity as a calibrated arithmetic mean of concept probabilities suggests the score edit could also improve practical tasks that filter or exclude concepts, such as safe image search or negative-prompt retrieval, not just benchmark accuracy.
  • Inference: Because the mechanism is attributed to dot-product scoring of contrastive dual encoders generally, the same bag-of-concepts behavior should appear in other dual-encoder models; testing LCSE on non-CLIP architectures would show whether the fix is truly architecture-agnostic.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that compositional failures of dual-encoder vision-language models on Boolean queries are caused by the similarity interface rather than by missing representations. It claims that holistic dot-product scoring behaves like mean pooling of atomic concept evidence, that operator-specific signals exist in text embeddings but are too weak or misaligned to affect rankings, and that fine-tuning does not fix the bottleneck. The proposed method, LCSE, parses a query into concepts and an operator, computes calibrated concept-presence scores with the frozen encoder, applies complement for negation and power-mean aggregation for conjunction/disjunction, and adds a logit-space correction to the holistic score. The authors introduce FACTOR-Bench and report large gains on it and on NegBench while preserving retrieval performance.

Significance. If the claims hold, the paper provides a valuable mechanistic account of a known failure mode and a simple, practical, training-free fix. The evidence is unusually well triangulated: operator-swap and rephrasing rank analyses, span-residual decomposition with null baselines, a polarity test in which all six fine-tuned baselines are below chance, controlled FACTOR-Bench construction with OWL-ViT validation, oracle-vs-LLM parse comparisons, calibration ablations, and retention checks. The paper is also honest about its scope (Boolean operators over one/two concepts, reliance on atomic evidence). The main risk is the transferability of a single global calibration to arbitrary concepts, which the current validation addresses only in aggregate.

major comments (3)
  1. [§4.2–4.3, Appendix A (Eqs. 4–7, Tables 8–9)] The sign and magnitude of LCSE corrections are controlled by one backbone-wide calibration center μ. For a single negated concept, Eq. (7) reduces to a correction of −2(s_i−μ) after β cancels, so whether the correction moves a given caption in the right direction depends on the global μ. The paper’s own ablation (Table 8) shows NOT accuracy swinging from 58.3% to 88.0% over a 0.04 shift in μ, and Table 9 shows a similar effect for SigLIP 2. Yet validation is aggregate-only: Table 4 stratifies by per-concept detection AUC, not by per-concept calibration error. I request a per-concept analysis: for each concept, estimate the empirical present/absent decision boundary, compare it with the global μ, report the fraction of concepts for which LCSE’s correction is misdirected or under-/over-weighted, and give FACTOR-Bench accuracy restricted to those concepts. Until this is provided, the claim
  2. [§3, §5, Appendix D] The mechanistic claim that compound prompts behave like mean pooling is derived and validated entirely on COCO val2017 categories (80 object classes, ViT-B/32), and FACTOR-Bench is COCO-derived. The abstract and introduction phrase the conclusion as a property of ‘dual-encoder VLMs’ generally. I would like the authors to either add a small cross-domain or cross-category analysis (e.g., abstract concepts, attributes, or a non-COCO dataset) or explicitly restrict the claim to the evaluated setting. This is not a demand for a new benchmark, but the current phrasing is broader than the evidence.
  3. [§5.1, Appendix A] The method is called ‘training-free’, but μ, β, γ_and, and γ_or are selected on COCO train2017 by grid search over operator tests, and the ablation shows meaningful sensitivity to these choices, especially μ. This is acceptable methodology, but the term should be defined precisely (e.g., ‘no encoder updates’) so readers do not equate it with ‘no validation-set tuning’.
minor comments (6)
  1. [Table 2] Many of the headline comparisons are between numbers differing by a few points. Confidence intervals or repeated-run standard errors would help, even if only for the main FACTOR-Bench and NegBench rows.
  2. [§3, Figure 3] The text reports concept replacement ρ≈0.47 while the figure caption says ρ≈0.64. The numbers should be reconciled in the final version.
  3. [Appendix E] The empirical verification reports mean |ρ|≈0.31 between span and residual score components. This weakens the independence assumption used in Proposition 1; please state explicitly how the inversion-rate formula is affected by this correlation.
  4. [Appendix A, Table 6] The sentence ‘no parse error induces a wrong-sign correction’ is easy to misread as ‘no parse error has any effect’; Table 6 shows AND dropping from 84.7 to 82.7. Please clarify that parse errors are benign only in the sense that they leave the correction at zero.
  5. [Figure 2] The caption says green borders indicate expected categories, but the figure as rendered in the manuscript does not show borders. This needs to be fixed in the production figure.
  6. [Appendix A] The statement ‘wrong μ causes negation to fail’ is important enough to be in the main text Limitations, not only in the appendix.

Circularity Check

0 steps flagged

Derivation is self-contained; the mechanistic claims and LCSE gains are supported by direct measurements and external benchmarks, not reduced to inputs.

full rationale

The paper's derivation chain does not reduce to its own inputs. The bag-of-concepts diagnosis in §3 is measured directly: operator swaps preserve rankings (ρ≈0.94) while concept swaps reorder them, and Figure 4 shows holistic scores track MEAN(s_A,s_B). The span-residual decomposition (Eq. 1) defines gain and alignment from measured embedding projections and ground-truth quadrants, and Proposition 1 is a mathematical relation, not a hidden assumption. LCSE is a constructive design: Eqs. (4)–(7) explicitly implement complement and power-mean aggregation, and the paper openly states that the factored polarity result is 'correct by construction' (Table 1), which is honest rather than a disguised prediction. The headline FACTOR-Bench evaluations use ground truth from COCO annotations plus OWL-ViT validation, and the central results are corroborated by the external NegBench benchmark and by retrieval retention (R@5 55.2 vs 55.3, ρ=0.999), so the main claims do not depend solely on the authors' own benchmark. There are no load-bearing self-citations or imported uniqueness theorems; the cited prior work supplies context, not the argument's load-bearing premise. The paper's own limitations admit that weak atomic evidence is not recoverable, and the global (µ,β) calibration is validated only in aggregate (Tables 8–9). That is a robustness/fragility concern about per-concept transfer, not a circularity of the derivation.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central mechanism claim is a measurement, not a derivation; LCSE's edge rests on two fitted calibration parameters (µ, β), two hand-chosen power-mean exponents (γ), the transferability of the calibrated concept score, and the parity between the self-authored benchmark and the method's operator set.

free parameters (4)
  • Calibration center µ = 0.22 (CLIP ViT-B/32), 0.05 (SigLIP 2)
    Grid-searched on COCO train2017 operator tests; sets the sign of negation corrections (Eq. 4–5). NOT accuracy swings 58→88% over a 0.08-wide sweep (Table 8).
  • Calibration slope β = 30 (both backbones)
    Controls probability spread and correction magnitude; grid-searched, stable over [30,60] (Appendix A, Tables 8–9).
  • AND power-mean exponent γ_and = -1
    Power-mean exponent for conjunction (−1 ≈ harmonic mean); chosen from ablation balancing FACTOR-Bench vs NegBench (Appendix G).
  • OR power-mean exponent γ_or = 10
    Power-mean exponent for disjunction (10 ≈ max); chosen from ablation (Appendix G).
axioms (6)
  • standard math Span-residual decomposition t = t_span + t_perp is a valid orthogonal split of the compound text embedding (Eq. 1).
    Standard linear algebra; the decomposition itself is exact, though its interpretive weight is empirical.
  • domain assumption Gaussian pairwise-difference and approximate isotropy assumptions in Proposition 1 (Appendix E) hold well enough.
    Empirically verified with mean |ρ(z_span,z_perp)| ≈ 0.31; authors call the error 'modest' and it does not affect qualitative conclusions.
  • ad hoc to paper Calibrated concept score p_i = sigmoid(β(s_i − µ)) is a transferable presence probability across arbitrary concepts and captions (Eqs. 4–5, Appendix A).
    The global (µ, β) per backbone is specific to this method; its transfer to rare/abstract concepts is assumed and only aggregate robustness is shown.
  • domain assumption Truth-functional Boolean semantics (A∧B iff both present, ¬A iff absent, etc.) is the correct normative criterion for evaluating these queries.
    Adopted in §1, §3 and Appendix B to define both the benchmark ground truth and the method's execution rules.
  • domain assumption OWL-ViT object detections validate concept presence/absence for FACTOR-Bench ground truth.
    Ground truth uses COCO annotations plus OWL-ViT validation (Appendix B); detector errors would mislabel benchmark items.
  • domain assumption LLM parser (GPT-4.1-mini, 83 few-shot examples) extracts correct concepts/polarity/operators on natural captions.
    Main results use oracle parses; end-to-end results cost only −0.3pp on FACTOR-Bench with the LLM parser (97.5% operator accuracy), so parser error is small but not zero.

pith-pipeline@v1.3.0-alltime-deepseek · 23986 in / 24525 out tokens · 239679 ms · 2026-08-01T03:46:02.011827+00:00 · methodology

0 comments
read the original abstract

Dual-encoder vision-language models (VLMs) expose a similarity interface that enables zero-shot retrieval but fails compositional constraints: queries like "umbrella and no person" retrieve images containing both, even when concept detection is reliable. We trace this to an interface-level Bag-of-Concepts effect, where similarity scores approximate mean pooling of concept evidence regardless of operators. Although operator-dependent signals exist in text embeddings, they are too weak or misaligned to affect rankings. Fine-tuning does not reliably resolve this failure because the dominant bottleneck is how similarity aggregates evidence rather than what encoders represent. We propose factored inference, which separates evidence extraction from constraint execution, and introduce LCSE (Logic-Constrained Score Editing), a training-free method that executes constraints externally using concept scores from frozen encoders. We also introduce FACTOR-Bench, where LCSE achieves 85.5% accuracy versus 73.2% for the best fine-tuned baseline, 90.7% when applied to SigLIP 2, and improves NegBench COCO MCQ accuracy from 27.2% to 65.2% while preserving retrieval performance.

Figures

Figures reproduced from arXiv: 2607.23052 by Han Zhang, Marios Savvides, Sultan Alshehri, Zhantao Yang.

Figure 1
Figure 1. Figure 1: CLIP-style retrieval models violate compositional constraints. Negation and exclusion queries retrieve excluded concepts, and conjunctions miss required concepts. CLIP and its composition-aware and negation-aware (Yuksekgonul et al., 2023; Alhamoud et al., 2025) variants show unreliable top-1 retrieval results. LCSE (§4) reduces these failures while preserving standard retrieval performance. is not a const… view at source ↗
Figure 2
Figure 2. Figure 2: Holistic scoring inverts anti-monotone operators. Cells show % of concept pairs where each image category achieves the highest mean similarity score. Green borders indicate expected (correct) categories. ¬A ∧ ¬B should prefer NEITHER but never does (0%). ¬A prefers images containing A for 91% of pairs. A ∧ B: BOTH wins only 60%. A∧B A∨B A∧¬B ¬A∧¬B 0.00 0.01 0.02 0.03 Mean |Δ score| a) Score change under op… view at source ↗
Figure 5
Figure 5. Figure 5: Two failure regimes. (a) Gain vs. alignment (2σ ellipses over concept pairs, dashed line = chance). A ∧ B, A ∨ B, and ¬A cluster above chance (aligned), while ¬A ∧ ¬B clusters far below (misaligned). (b) Amplification sweep (λ): as residual influence increases, scores converge toward their alignment ceiling. ¬A improves dramatically (12%→66%), while ¬A ∧ ¬B remains inverted. complement) must come from the … view at source ↗
Figure 6
Figure 6. Figure 6: Span coefficient distribution. Normalized weights α/(α + ω) cluster around 0.5 (shaded region) for all operators, consistent with the span component approximating mean pooling. White dots indicate medians, bars show interquartile range. A∧¬B is slightly A-biased (0.59) because concept A is asserted while B is negated [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: visualizes score distributions over all COCO im￾ages for two AND queries. In cat and dog, the extreme right tail is dominated by BOTH images. This yields few top-k violations, yet substantial overlap remains. In stop sign and bus, overlap is severe and top-k collapses entirely into violating quadrants [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Logical equivalence violations. Violation rate (%) when logically equivalent queries produce different scores (> 0.001 threshold in cosine similarity). Factored scoring achieves 0% by construction. LCSE matches its holistic backbone exactly, since the oracle parser canonicalizes equivalent forms [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 4 linked inside Pith

  1. [1]

    H., Kim, Y., and Ghassemi, M

    Alhamoud, K., Alshammari, S., Tian, Y., Li, G., Torr, P. H., Kim, Y., and Ghassemi, M. Vision-language models do not understand negation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 29612--29622, 2025

    2025

  2. [2]

    Conceptual 12 M : Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12 M : Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3558--3568, 2021

    2021

  3. [3]

    K., Winn, J., and Zisserman, A

    Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The PASCAL visual object classes ( VOC ) challenge. International Journal of Computer Vision, 88 0 (2): 0 303--338, 2010

    2010

  4. [4]

    and Kembhavi, A

    Gupta, T. and Kembhavi, A. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 14953--14962, 2023

    2023

  5. [5]

    SugarCrepe : Fixing hackable benchmarks for vision-language compositionality

    Hsieh, C.-Y., Zhang, J., Ma, Z., Kembhavi, A., and Krishna, R. SugarCrepe : Fixing hackable benchmarks for vision-language compositionality. Advances in Neural Information Processing Systems, 36: 0 31096--31116, 2023

    2023

  6. [6]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp.\ 4904--4916. PMLR, 2021

    2021

  7. [7]

    ComCLIP : Training-free compositional image and text matching

    Jiang, K., He, X., Xu, R., and Wang, X. ComCLIP : Training-free compositional image and text matching. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 6639--6659, 2024

    2024

  8. [8]

    The hard positive truth about vision-language compositionality

    Kamath, A., Hsieh, C.-Y., Chang, K.-W., and Krishna, R. The hard positive truth about vision-language compositionality. In European Conference on Computer Vision, pp.\ 37--54. Springer, 2024

    2024

  9. [9]

    Is CLIP ideal? No

    Kang, R., Song, Y., Gkioxari, G., and Perona, P. Is CLIP ideal? No . can we fix it? Yes ! In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025

    2025

  10. [10]

    Koishigarina, D., Uselis, A., and Oh, S. J. CLIP behaves like a bag-of-words model cross-modally but not uni-modally. arXiv preprint arXiv:2502.03566, 2025

    arXiv 2025

  11. [11]

    Does CLIP bind concepts? probing compositionality in large image models

    Lewis, M., Nayak, N., Yu, P., Merullo, J., Yu, Q., Bach, S., and Pavlick, E. Does CLIP bind concepts? probing compositionality in large image models. In Findings of the Association for Computational Linguistics: EACL 2024, pp.\ 1487--1500, 2024

    2024

  12. [12]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft COCO : Common objects in context. In European Conference on Computer Vision, pp.\ 740--755. Springer, 2014

    2014

  13. [13]

    O., Gandhi, M., Gao, I., and Krishna, R

    Ma, Z., Hong, J., Gul, M. O., Gandhi, M., Gao, I., and Krishna, R. CREPE : Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10910--10921, 2023

    2023

  14. [14]

    Simple open-vocabulary object detection

    Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pp.\ 728--755. Springer, 2022

    2022

  15. [15]

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman...

    Pith/arXiv arXiv 2024

  16. [16]

    Know `` No '' better: A data-driven approach for enhancing negation awareness in CLIP

    Park, J., Lee, J., Song, J., Yu, S., Jung, D., and Yoon, S. Know `` No '' better: A data-driven approach for enhancing negation awareness in CLIP . In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025

    2025

  17. [17]

    D., and Hein, M

    Peleg, A., Singh, N. D., and Hein, M. Advancing compositional awareness in CLIP with efficient fine-tuning. In Advances in Neural Information Processing Systems, 2025

    2025

  18. [18]

    How and where does CLIP process negation? In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), 2024

    Quantmeyer, V., Mosteiro, P., and Gatt, A. How and where does CLIP process negation? In Proceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), 2024

    2024

  19. [19]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp.\ 8748--8763. PMLR, 2021

    2021

  20. [20]

    Collecting image annotations using A mazon ' s M echanical T urk

    Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J. Collecting image annotations using A mazon ' s M echanical T urk. In Callison-Burch, C. and Dredze, M. (eds.), Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with A mazon ' s Mechanical Turk , pp.\ 139--147, Los Angeles, June 2010. Association for Computational Lin...

    2010

  21. [21]

    T., Argus, M., Fischer, V., and Brox, T

    Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., and Brox, T. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision-language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=uAFHCZRmXk

    2025

  22. [22]

    Learning the power of `` No '': Foundation models with negations

    Singh, J., Shrivastava, I., Vatsa, M., Singh, R., and Bharati, A. Learning the power of `` No '': Foundation models with negations. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.\ 8002--8012. IEEE, 2025

    2025

  23. [23]

    ViperGPT : Visual inference via Python execution for reasoning

    Sur \' s, D., Menon, S., and Vondrick, C. ViperGPT : Visual inference via Python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 11888--11898, 2023

    2023

  24. [24]

    Winoground : Probing vision and language models for visio-linguistic compositionality

    Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., and Ross, C. Winoground : Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5238--5248, 2022

    2022

  25. [25]

    F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M. F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al. SigLIP 2 : Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    A good CREPE needs more than just Sugar : Investigating biases in compositional vision-language benchmarks

    Udandarao, V., Cherti, M., Karthik, S., Jitsev, J., Albanie, S., and Bethge, M. A good CREPE needs more than just Sugar : Investigating biases in compositional vision-language benchmarks. arXiv preprint arXiv:2506.08227, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Y., Lee, M.-L., et al

    Xu, J., Fei, H., Zhang, Y., Pan, L., Huang, Q., Liu, Q., Nakov, P., Kan, M.-Y., Wang, W. Y., Lee, M.-L., et al. MuSLR : Multimodal symbolic logical reasoning. In Advances in Neural Information Processing Systems, 2025

    2025

  28. [28]

    When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023

    Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023

    2023

  29. [29]

    LiT : Zero-shot transfer with locked-image text tuning

    Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L. LiT : Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 18123--18133, 2022

    2022

  30. [30]

    Sigmoid loss for language image pre-training

    Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 11975--11986, 2023

    2023

  31. [31]

    NegVQA : Can vision language models understand negation? In Findings of the Association for Computational Linguistics: ACL 2025, 2025

    Zhang, Y., Su, Y., Liu, Y., and Yeung-Levy, S. NegVQA : Can vision language models understand negation? In Findings of the Association for Computational Linguistics: ACL 2025, 2025

    2025

  32. [32]

    VL-CheckList : Evaluating pre-trained vision-language models with objects, attributes and relations

    Zhao, T., Zhang, T., Zhu, M., Shen, H., Lee, K., Lu, X., and Yin, J. VL-CheckList : Evaluating pre-trained vision-language models with objects, attributes and relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2022

    2022

  33. [33]

    Logic unseen: Revealing the logical blindspots of vision-language models

    Zhou, Y., Tang, J., Yang, S., Xiao, X., Dai, Y., Yang, W., Gou, C., Xia, X., and Chua, T.-S. Logic unseen: Revealing the logical blindspots of vision-language models. arXiv preprint arXiv:2508.11317, 2025

    Pith/arXiv arXiv 2025