Pith. sign in

REVIEW 4 major objections 5 minor 29 references

CIVET: Systematic Evaluation of Understanding in VLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art vision-language models recognize shapes well but fail on relational and position-robust understanding in controlled tests.

desk verdict A solid, honest controlled-evaluation paper whose core position-dependence finding survives its arbitrary position-label convention, but the human comparison needs re-analysis under a human-aligned mapping. read the letter →

arxiv 2506.05146 v2 pith:VEJ4LBZ4 submitted 2025-06-05 cs.CV cs.CL

Massimo Rizzoli , Simone Alghisi , Olha Khomyn , Gabriel Roccabruna , Seyed Mahed Mousavi , Giuseppe Riccardi This is my paper
classification cs.CVcs.CL
keywords vision-languagemodelscontrolledstimulisystematicevaluationobjectpropertyrecognitionspatialrelationspositionbiassceneunderstandinghuman
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that leading vision-language models do not robustly understand the structure of visual scenes, even when they score well on standard question-answering benchmarks. To test this, it builds CIVET, a generator of controlled images on a 9-by-9 grid in which every object property, position, and relation is known exactly and every question is closed-ended. Across five models, shape recognition is near-perfect but sheen tops out at 64%, color accuracy varies sharply across hues, and absolute position accuracy reaches only 62% versus 73% for human annotators. The paper concludes that high benchmark performance can coexist with shallow, position-dependent visual understanding, and that relational reasoning about relative position, size, and distance remains weak or near chance.

What carries the argument

The load-bearing object is CIVET, a deterministic stimulus generator. Each world is a structured representation: a set of objects described by property-value pairs (shape, color, sheen) and relations, rendered as a $9 \times 9$ grid. For a given probe, the generator fixes the aspect under test and enumerates all combinations of the other variables—each of the 81 positions for single objects, all cell-pair combinations for relative position and size, and section-level placements for relative distance—so performance is averaged over a balanced, complete stimulus set. Questions are closed-ended with uniformly shuffled option order, which removes label imbalance and order bias. This design lets the authors isolate what a model knows about one property or relation without confounding from dataset statistics or scene complexity.

What would settle it

A concrete test: if a VLM scored above 90% in all 81 cells on shape, color, sheen, and absolute position and above 80% on all eight relative-position classes, the paper's central conclusion would be contradicted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that state-of-the-art VLMs have only partial, brittle scene understanding. In single-object probes, shape recognition is near-perfect (over 95% accuracy for all models) and color is often strong, but sheen recognition peaks at 64%, and absolute position in a 9-by-9 grid reaches at most 62% accuracy under a closed-ended nine-way choice, where chance is 11%. Accuracy for a fixed property changes with the object's cell: the best models show almost perfect shape recognition in most bottom-row cells but drop in top corners, a pattern consistent with attending to later visual tokens. On relations, relative position accuracy tops out at 46% (chance 13%), relative size at 54% (chance 33%), and relative distance at 83% (chance 50%), with several models scoring at or near chance on specific relation classes, such as 0% F1 on 'same' for relative size. The paper also finds that humans reading the same stimuli assign absolute position with 73% accuracy and a different horizontal convention, and that all evaluated VLMs fall short of that.

Load-bearing premise

The mapping from grid cells to absolute-position labels—nine equal $3 \times 3$ blocks—is a convention rather than ground truth, and the paper itself shows human annotators use a narrower center band horizontally, so the human-versus-model gap on absolute position rests on that unvalidated choice.

Editorial extensions

If this is right

  • Standard VQA benchmarks should not be read as evidence of robust scene understanding, because a model can score near ceiling on object properties while being near chance on relations.
  • Object-position effects are a first-class failure mode: accuracy maps over the 81-cell grid expose biases (for example, near-perfect shape accuracy in most bottom-row cells but drops in top corners) that aggregate accuracy hides.
  • Relational reasoning is separable from property recognition, since some models recognize a shape almost perfectly yet cannot use that shape name to identify the closest object; evaluation and training should treat reference resolution as its own skill.
  • Human comparison offers a calibration target: humans reached 73% on absolute position with a narrower horizontal center band, so the labels used in position questions are not neutral.
  • Scaling the text decoder is not a reliable cure, since the 13B version of one model was sometimes worse than its 7B counterpart for color and for real-world object category.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The equal 3-by-3-block ground truth for absolute position is a choice, not a fact; re-scoring the human annotations with the annotators' observed narrower horizontal center would change the reported 73%-versus-62% gap.
  • Editorial inference: The same exhaustive-generation logic could be applied to rotations, mirroring, background color, occlusion, and object count, and the existing position-dependence pattern already suggests the models respond to token order rather than to spatial geometry.
  • Editorial inference: The appendix shows that without an 'answer as few words as possible' instruction, several models produce multi-token or 'other' answers; this implies open-ended VQA formats can distort measured competence, and format control belongs in standard evaluation.
  • Editorial inference: CIVET's balanced stimulus sets are a natural fine-tuning or probing signal; if position and relation accuracy improve after training on such balanced data, that would confirm the bias interpretation and offer a concrete remedy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CIVET, a framework for controlled evaluation of vision-language models on basic scene understanding tasks using procedurally generated 9x9 grid scenes with known ground-truth properties. The authors evaluate five VLMs (LLaVA-NeXT 7B/13B, Molmo-O 7B, Qwen2-VL 7B, and CLIP) on shape, color, sheen, absolute position, and three relational tasks (relative position, relative distance, relative size). They report that shape recognition is near-perfect, sheen and some colors are poorly recognized, accuracy varies systematically with object position, and relational judgments are weak; a human annotation study on absolute position finds humans (73%) outperform all models (42-62%).

Significance. If the results hold, CIVET is a valuable complementary evaluation tool: its controlled, balanced, deterministic stimuli avoid the annotation noise and scene complexity of natural-image benchmarks, and the per-cell accuracy analysis offers a transparent view of positional biases. The release of the framework and the exhaustive stimulus sets are clear strengths. However, two issues limit the strength of the human-comparison and real-world generalization claims: the arbitrary 3x3 ground-truth partition for absolute position and the CLIP-based selection of COCO instances.

major comments (4)
  1. [5.2, Fig. 3; 4.1] The equal 3x3 block partition used as ground truth for absolute position is an unvalidated convention, and the paper's own human data contradict it. In Section 5.2 the authors state that the assignment is arbitrary and report that humans shrink the horizontal center to roughly one column, whereas the ground truth assigns three columns to 'center'. Both the human accuracy (73%) and the model accuracies (42-62%) are therefore computed under a labeling scheme that does not match the labels humans actually produce. The abstract's claim that 'VLMs still fall short of achieving human-level accuracy' is anchored in this comparison and is not robust to the choice of partition; the gap could shrink or disappear under a human-aligned mapping, especially for Molmo and Qwen2-VL, whose position assignments resemble the human mapping. I ask the authors to re-analyze the position results under a human-aligned partition (or an agreement-based metric), and to temper the human-level claim accordingly.
  2. [4.1, Single Object w. COCO; Table 2] The COCO subset is selected using CLIP: the authors keep the three categories with the best CLIP performance and, within each category, the ten objects with the highest CLIP similarity to the category, then manually pick the best. This procedure selects easy instances for CLIP and for the two VLMs that use CLIP's vision encoder (LLaVA-NeXT and Molmo), so Table 2's comparison with CLIP is confounded by selection. The paper should either use a model-independent selection criterion (e.g., random sample or human-verified typicality) or clearly label this experiment as an easy-case analysis and avoid claiming that it validates real-world generalization.
  3. [Abstract, §3, §5] The paper repeatedly claims 'statistical rigor' and 'statistical guarantees', but all reported results are point accuracies/F1 with no confidence intervals, significance tests, or error bars. For example, Table 3 reports differences of a few percent between models without indicating whether they are meaningful, and the human-model comparison in Section 5.2 is based on 81 stimuli with 8 annotations per stimulus yet has no uncertainty quantification. Adding binomial confidence intervals (or bootstrapped intervals) and, where appropriate, paired tests would substantiate the claim of statistical rigor and support the qualitative conclusions.
  4. [5.2, Abstract] The human evaluation covers only one task—absolute position of a single yellow star on a black grid—but the abstract and conclusion generalize the 'human-level accuracy' shortfall to VLMs' understanding at large. Even after fixing the ground-truth partition, the comparison would not support a global statement about human-level competence on properties and relations. The authors should either restrict the claim to the tested task or add human evaluations for the other stimulus types.
minor comments (5)
  1. [Table 2 caption] Typo in the Table 2 caption: 'Since Object' should be 'Single Object'.
  2. [Table 9 caption] Typo in the Table 9 caption: 'sccuracy' should be 'accuracy'.
  3. [4.1, Relative Distance] The sentence 'we sample the cell uniformly from the section' does not specify how many scenes are sampled per configuration; the total 4,374 implies six samples per 9-section configuration, and this should be stated explicitly.
  4. [5.2, CLIP position discussion] The discussion of CLIP's position behavior ('assigned top center to the central section' and 'top left to most other cells') conflates a label bias with randomness; consider rephrasing to describe the bias.
  5. [A.3] The selection of the 'best setting' (regular objects, 672x672 for synthetic, 1344x1344 for COCO) is made without stating the exact criterion (e.g., mean accuracy across tasks) or whether the qualitative conclusions change under other settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CIVET is an empirical benchmark study whose findings are computed directly from model and human responses, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

This paper reports a controlled empirical evaluation of VLMs, not a derivation. The central quantities (accuracy, F1, position-response maps) are computed directly from model outputs and human annotations against deterministic ground truths. No parameter is fitted to a subset of the data and then used to predict a closely related quantity; the framework does not presuppose the conclusions of RQ1-RQ3. The absolute-position convention (dividing the 9x9 grid into nine equal 3x3 blocks) is a measurement/labeling choice, and the paper explicitly acknowledges that this assignment is arbitrary ('Since we arbitrarily assigned each cell to a particular absolute position'), then tests it with a human study and reports the humans' different horizontal mapping. That is a validity concern about how to interpret the human-vs-model comparison, not a circular step: the model accuracies are not defined in terms of human accuracies, and the core RQ2 finding of position-dependent accuracy is supported independently by cell-by-cell accuracy maps (Figures 2, 4, 5, 6) that do not depend on the 3x3 partition. The paper cites no prior work by the same authors to justify a load-bearing premise, and no uniqueness theorem or ansatz is imported from self-citations. The limitations section notes practical constraints (model size, object size, crowd-worker variation) but does not assert any missing proof or circular dependency. Therefore there is no self-definitional, fitted-input-as-prediction, or self-citation circularity in the claimed evaluation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No fitted parameters are used for prediction; the only hand-chosen numerical convention is the position grid sectioning. The conclusions rely on domain assumptions about what controlled VQA accuracy measures and on the validity of the arbitrary position labeling, plus a CLIP-selected COCO subset.

free parameters (1)
  • Absolute-position sectioning = 3x3 equal blocks over a 9x9 grid
    Section 4.1 assigns each cell to one of nine position labels using equal 3x3 blocks; this hand-chosen convention defines the ground truth for absolute position and materially affects the reported accuracies.
assumptions (4)
  • domain assumption Accuracy on closed-ended questions about shape, color, sheen, absolute position, and pairwise relations is a valid measure of a VLM's 'understanding' of a scene.
    Section 3 defines worlds and stimuli, and Section 4.1 operationalizes understanding through these question templates; the paper does not independently validate this operationalization.
  • ad hoc to paper The arbitrary 3x3 equal-block labeling of the 9x9 grid is a valid ground truth for absolute position.
    Section 4.1 assigns labels top-to-bottom, left-to-right; Section 5.2 shows humans use a different horizontal division, so the convention is not independently justified.
  • domain assumption Closed-ended answers can be scored by exact match, with 'other' responses counted as mistakes.
    Appendix A.2 describes how verbose and 'other' answers are handled; this assumes the option set is exhaustive and unambiguous.
  • domain assumption COCO objects selected via CLIP similarity (giraffe, elephant, zebra; best instance per category) are a meaningful real-world complement.
    Section 4.1, Single Object w. COCO; selection by CLIP performance biases toward easy instances, so COCO results are conditional on this selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CIVET: Systematic Evaluation of Understanding in VLMs." pith.science (2026). https://pith.science/paper/VEJ4LBZ4

@misc{pith2026250605146,
  author       = {Pith},
  title        = {Pith review of: CIVET: Systematic Evaluation of Understanding in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VEJ4LBZ4}},
  note         = {Machine review of arXiv:2506.05146}
}
read the original abstract

While Vision-Language Models (VLMs) have achieved competitive performance in various tasks, their comprehension of the underlying structure and semantics of a scene remains understudied. To investigate the understanding of VLMs, we study their capability regarding object properties and relations in a controlled and interpretable manner. To this scope, we introduce CIVET, a novel and extensible framework for systematiC evaluatIon Via controllEd sTimuli. CIVET addresses the lack of standardized systematic evaluation for assessing VLMs' understanding, enabling researchers to test hypotheses with statistical rigor. With CIVET, we evaluate five state-of-the-art VLMs on exhaustive sets of stimuli, free from annotation noise, dataset-specific biases, and uncontrolled scene complexity. Our findings reveal that 1) current VLMs can accurately recognize only a limited set of basic object properties; 2) their performance heavily depends on the position of the object in the scene; 3) they struggle to understand basic relations among objects. Furthermore, a comparative evaluation with human annotators reveals that VLMs still fall short of achieving human-level accuracy.

Figures

Figures reproduced from arXiv: 2506.05146 by the authors.

Figure 1
Figure 1. CIVET framework enables a systematic eval [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Accuracy (%) of LLaVA-NeXT 13B in each cell of our [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. "Where is the yellow star?" - Responses of CLIP, Qwen, Molmo-O 7B, and Humans when asking a closed-ended question about the position of a yellow star on a black background. The question was asked by placing the object in all cells of a 9 × 9 grid. Since we obtained multiple human annotations for the same stimulus, we report the majority vote. Dashed lines delimit the ground truth sections, and colors indicate the re… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Accuracy (%) of LLaVA-NeXT 7B & Molmo 7B in each cell of our [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Accuracy (%) of Molmo 7B & Qwen2-VL 7B in each cell of our [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Accuracy (%) of Molmo 7B in each cell of our [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Guidelines for the proposed human evaluation task. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: User interface for the proposed human evaluation task. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 11 canonical work pages

  1. [1]

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. 2019. https://doi.org/10.1609/aaai.v33i01.33018076 Tallyqa: Answering complex counting questions . Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):8076--8084

    doi:10.1609/aaai.v33i01.33018076 2019

  2. [2]

    Lipton, and J

    Sumukh K Aithal, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/f29369d192b13184b65c6d2515474d78-Paper-Conference.pdf Understanding hallucinations in diffusion models through mode interpolation . In Advances in Neural Information Processing Systems, volume 37, pages 134614--134644. C...

    work page 2024

  3. [3]

    Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2016

  4. [4]

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. 2024. https://doi.org/10.1109/CVPR52733.2024.01370 Spatialvlm: Endowing vision-language models with spatial reasoning capabilities . In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455--14465

    arXiv 2024

  5. [5]

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146

    arXiv 2024

  6. [6]

    Reza Esfandiarpoor, Cristina Menghini, and Stephen Bach. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.547 If CLIP could talk: Understanding vision-language model representations through their preferred concept descriptions . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9797--9819, Miami, Florida, US...

    work page doi:10.18653/v1/2024.emnlp-main.547 2024

  7. [7]

    Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378

    1971

  8. [8]

    Xingyu Fu, Sheng Zhang, Gukyeong Kwon, Pramuditha Perera, Henghui Zhu, Yuhao Zhang, Alexander Hanbo Li, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Patrick Ng, Dan Roth, and Bing Xiang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.147 Generate then select: Open-ended visual question answering guided by world knowledge . In Findings of the As...

    doi:10.18653/v1/2023.findings-acl.147 2023

Show all 29 references
  1. [9]

    Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Vondrick, and Dan Roth. 2022. https://doi.org/10.18653/v1/2022.acl-long.81 There`s a time and place for reasoning beyond the image . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volum...

    2022 doi

  2. [10]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  3. [11]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017. https://doi.org/10.1109/CVPR.2017.215 Clevr: A diagnostic dataset for compositional language and elementary visual reasoning . In 2017 IEEE Conference on Comput...

    2017 doi

  4. [12]

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. 2024. Prismatic vlms: Investigating the design space of visually-conditioned language models. In International Conference on Machine Learning (ICML)

    2024

  5. [13]

    Seunghoi Kim, Chen Jin, Tom Diethe, Matteo Figini, Henry F. J. Tregidgo, Asher Mullokandov, Philip Teare, and Daniel C. Alexander. 2025. Tackling structural hallucination in image translation with local diffusion. In Computer Vision -- ECCV 2024, pages 87--103, Cham. Springer ...

    2025

  6. [14]

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015--4026

    2023

  7. [15]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of com...

    2017

  8. [16]

    Hugo Lauren c on, Leo Tronchon, Matthieu Cord, and Victor Sanh. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/a03037317560b8c5f2fb4b6466d4c439-Paper-Conference.pdf What matters when building vision-language models? In Advances in Neural Information Processin...

    2024

  9. [17]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  10. [18]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  11. [20]

    Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. 2024. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13279--13288

    2024

  12. [21]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. https://proceedings.mlr.press/v139/radford21a.html Learning transferable visual model...

    2021

  13. [22]

    Bradley Rettler and Andrew M. Bailey. 2024. Object . In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy , S ummer 2024 edition. Metaphysics Research Lab, Stanford University

    2024

  14. [23]

    Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata

    Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. 2023. https://doi.org/10.1109/ICCV51070.2023.01443 Waffling around for performance: Visual classification with random words and broad concepts . In 2023 IEEE/CVF International Conf...

    2023

  15. [24]

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114

    2021 arXiv

  16. [25]

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. https://doi.org/10.1109/CVPR52688.2022.00517 Winoground: Probing vision and language models for visio-linguistic compositionality . In 2022 IEEE/CVF Conference on Com...

    2022

  17. [26]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  18. [27]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. Mmm...

    2024

  19. [28]

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936

    2022 arXiv

  20. [29]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  21. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.