Pith. sign in

REVIEW 3 major objections 8 minor 50 references

Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that basic visual knowledge is present in early VLM representations but mostly fails to reach the text output, and that the joint fine-tuning of the projection and decoder is the weakest link.

desk verdict A systematic three-space probe of VLMs that finds a plausible decoder bottleneck for fine-grained recognition, but the response-space readout is too crude to pin the blame. read the letter →

arxiv 2507.10442 v1 pith:DSTEQN7Y submitted 2025-07-10 cs.LG cs.AI

Shivam Chandhok , Wan-Cyuan Fan , Vered Shwartz , Vineeth N Balasubramanian , Leonid Sigal This is my paper
classification cs.LGcs.AI
keywords vision-languagemodelslinearprobingintermediaterepresentationsfine-grainedrecognitionobjectcountingspatialunderstandingVLMrobustnessinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to locate, inside a vision-language model, where basic visual abilities break down. The authors compare three stages of the model—the visual encoder's latent space, the vision-language projection space, and the language decoder's response—on object recognition, fine-grained recognition, counting, and spatial reasoning. They find that for recognition and counting, linear probes read the correct answer out of the early visual and projection spaces with accuracy above 90 percent, while the model's actual text response often falls to 10–35 percent. For spatial relations the pattern reverses: the early spaces are weak and the response is comparatively stronger. The paper concludes that the bottleneck for most basic visual tasks is not the visual encoder but the transfer of already-encoded knowledge into the decoder's output, and that this points to the joint fine-tuning stage as the target for improvement.

What carries the argument

The central object is the three-space diagnostic: the output of the visual encoder, the output of the vision-language projection, and the output of the language decoder. A linear logistic-regression probe is trained on average-pooled features from each space, with a shuffled-label control task verifying that probe accuracy reflects encoded information rather than probe memorization; the response space is evaluated both by such a probe and by the model's own text output. This decomposition is what lets the paper attribute failures to the translation between components rather than to any single component.

What would settle it

Train a multi-layer probe or a per-token probe on the language decoder's hidden states for the Stanford Dogs or CUB fine-grained task. If such a probe exceeds roughly 90 percent accuracy while the model's text answer stays near 30 percent, then the paper's 'knowledge loss' is an artifact of its linear average-pooled readout; if every readout stays low, the information is truly blocked before the decoder.

Watch

Extended reading notes

Core claim

The paper's central discovery is that current open vision-language models already encode the information needed for fine-grained recognition and object counting in their visual and vision-language projection spaces—linear probes exceed 90 percent accuracy there—yet this information does not reach the final text response, where accuracy on fine-grained recognition can fall to 10–35 percent. This contradicts the idea that the visual encoder is the main source of these failures. Spatial understanding is the exception: there the visual and projection representations themselves are weak, so improving the visual encoder is the right direction. The authors attribute the recognition and counting gap to ineffective joint fine-tuning of the projection layer and language decoder, supported by the observation that instruction-tuned models transfer better than the non-instruction-tuned BLIP-2, and that scaling the decoder only partially closes the gap.

Load-bearing premise

The diagnosis assumes that a linear probe on average-pooled features measures the information genuinely available to the next component, so a probe that cannot read the answer may be a limitation of the readout rather than an absence of information.

Editorial extensions

If this is right

  • Improving the joint fine-tuning of the projection layer and language decoder, with more fine-grained examples in the alignment data, should transfer the high-accuracy early-space knowledge into VLM text responses.
  • Scaling the language decoder alone is not a remedy: larger decoders improve response-space accuracy only partially, leaving a large gap on fine-grained recognition.
  • For spatial understanding, the visual encoder is the information bottleneck, so better spatial encoders or spatial training data are needed rather than decoder-side fixes.
  • Visual prompting effects such as clean backgrounds and reverse blur are absorbed mostly by early spaces; the response space benefits less, indicating that alignment loss is a separate target for improvement.
  • The response space's apparent robustness to image corruptions is partly a side effect of information not flowing into the response, so robustness numbers should be interpreted with that in mind.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this diagnosis is right, then adding a modest number of fine-grained examples to the alignment or instruction-tuning stage is a cheap testable fix: response-space accuracy should jump without changing the visual encoder.
  • The same three-space probing design could sort encoder-side from decoder-side causes for other reported VLM failures, such as negation, compositional reasoning, and hallucination.
  • Because the probes are supervised while the text evaluation is zero-shot, part of the measured gap may be a measurement asymmetry rather than lost information; token-level or nonlinear readouts would separate the two.
  • The spatial reversal suggests a two-part design rule: improve encoders for geometry and improve alignment training for semantics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. This paper proposes a three-space analysis of vision-language models: linear probes on the visual encoder output, the VL projection output, and the language decoder output, compared against the free-form textual response. Using PaintSkills, Pascal VOC, Stanford Dogs, and CUB, the authors find that for coarse and fine-grained recognition and counting, probe accuracy in the visual and VL-projection spaces is high (often >90%) while response-space probe accuracy and text accuracy drop sharply, especially for fine-grained recognition. For spatial understanding the trend reverses: visual and projection probes are weak (~50%) while the response space is relatively stronger for some models. The paper also studies robustness to corruptions and background/visual-prompting transformations, and hypothesizes that the response-space drop stems from ineffective joint fine-tuning of the projection and decoder. The three-space methodology and the broad model/task coverage are valuable, but the central claim relies on a response-space probe whose readout (average pooling over generated tokens) is likely to discard the token-localized information in autoregressive decoder outputs.

Significance. If the central claim holds, it would challenge the common attribution of VLM visual failures to the visual encoder (Tong et al., 2024) and redirect efforts to the projection/decoder alignment stage for most tasks. The paper contributes a reusable three-space evaluation protocol, open-source model coverage, a shuffled-label control, and detailed per-corruption/transformation results. However, the significance is contingent on the validity of the response-space probe; the current average-pooling readout is not a faithful information measure for autoregressive decoders, so the paper's headline conclusion is not yet established.

major comments (3)
  1. [§3.4, Response Space; Tables 2 and 4] The response-space probe averages the T×F language-decoder output across all generated tokens before fitting a linear classifier. For autoregressive LLMs, task-relevant information is typically concentrated in the answer token (or a small set of tokens), and mean pooling over the full generated sequence—including prompt continuations, filler tokens, and separators—dilutes this signal. The shuffled-label control in Appendix A.3 rules out label memorization by the probe, but it does not address readout mismatch: a probe that cannot access the informative token will report low accuracy even when the decoder output contains the information. The visual and VL-projection probes, in contrast, use average-pooled features with a readout that is standard and near-lossless for those encoders. The comparison is therefore not apples-to-apples, and the low response-space probe numbers in Tables 2 and 4 cannot support the claim that 'knowledge does not translate to the response space' until a token-level probe (e.g., probing the final token or the answer span) or a nonlinear probe is added and the drop persists.
  2. [§4.2, Table 5] The spatial-understanding conclusion that the visual encoder is the bottleneck rests on the same average-pooling readout. For a task defined by relative position (left/right/above/below), mean pooling over image patches discards the spatial layout that the task requires. The near-chance visual-probe accuracy in Table 5 (47-50%) may therefore understate the information available in the visual encoder. Before asserting that 'the visual encoder (i.e., CLIP) is responsible for the loss of information' (§4.2), the authors should probe the visual space with a readout that preserves position (e.g., probing patch tokens or using positional features), or explicitly caveat that the conclusion is conditional on the pooling-based protocol.
  3. [§4.1, Tables 1-5] The paper interchangeably compares supervised probe accuracy (intermediate spaces) with zero-shot text generation (response space). These are different evaluation protocols: the probe is fit with labels on the task, while the text response is generated with a VQA prompt and matched via string/fuzzy matching. Low text accuracy can result from instruction-following failures or decoding choices rather than absence of information; the authors themselves note in Appendix A.2 that adding 'answer in one word' reduced performance for BLIP2/InstructBLIP. To support the 'information is lost' framing, the paper should either use a supervised readout for the text channel (e.g., a probe on the generated answer tokens) or restrict the conclusion to a 'verbalization failure' rather than 'knowledge loss.'
minor comments (8)
  1. [Section 3] The phrase 'We conjuncture that these spaces' should read 'We conjecture that these spaces.'
  2. [Table 1] The heading 'Course-grained' is a typo for 'Coarse-grained.'
  3. [Section 5.3] The numbered summary list jumps from item 2 to item 4; item 3 is missing.
  4. [Appendix A.2] The sentence 'For BLIP2 and InstructBLIP model we use the same prompt i.e., What is the central object...' lacks punctuation, and the phrase 'continuous outputs' is ambiguous; please clarify.
  5. [Appendix A.6] The phrase 'sinces it is similar' should be 'since it is similar.'
  6. [Section 5.2] The statement 'We use COCO (Malik et al., 2024)' mis-credits the COCO dataset; the reference should distinguish the COCO dataset from the ObjectCompose-derived masks or cite the appropriate COCO source.
  7. [Figure 4] The legend uses 'Text space' while the paper elsewhere uses 'VL Proj.'; please align the terminology.
  8. [Appendix A.3] The shuffled-label control is reported for LLaVA-NEXT only; consider reporting it for at least one additional model family (e.g., InstructBLIP) since the probing conclusions are drawn across seven models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central probe comparison is an empirically measured outcome with independent text-response evaluation and a shuffled-label control.

full rationale

The paper's load-bearing claim is that linear probes on visual and VL-projection features achieve high held-out accuracy while probe and text accuracy in the language-decoder response space drop sharply. This is an empirical comparison, not a derivation from its own inputs: probes are trained on training splits and evaluated on validation/test splits, and the shuffled-label control task (Sec. 3.4 and App. A.3) addresses the concern that the probe itself learns the task from labels. The text-response evaluation is independent of the probing pipeline. The choice of average-pooled linear probing is an operational definition of what counts as 'accessible information'; if token-level or non-linear information were present but not linearly readable from mean-pooled features, the observed response-space drop could be an artifact of the readout rather than a true information loss. That is a validity limitation, not circularity, because the paper does not define 'knowledge is present' and 'knowledge is lost' in terms of the same fitted quantity, and it does not rename a fitted parameter as a prediction. Self-citations (Chandhok 2024; Chou et al. 2025; Fan et al. 2024) appear only in background lists and are not load-bearing; no uniqueness theorem, ansatz, or known result is smuggled in via self-citation. The paper is self-contained against external benchmarks and its central claim does not reduce to its inputs by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the probing methodology, specifically linear probes, average pooling, and cross-space comparison, rather than on any fitted constants or new theoretical entities. The key assumptions are about what probes measure and how spaces are compared.

assumptions (4)
  • domain assumption Linear probes on frozen feature representations accurately measure the task-relevant information contained in that representation.
    Invoked throughout Section 3.4 to compare visual, projection, and response spaces; only validated with a shuffled-label control task, which checks probe validity but not the choice of pooling or linear readout.
  • domain assumption Average pooling of visual encoder, VL projection, and decoder output tokens preserves the information needed for the probed tasks.
    Section 3.4: 'we train a probe on an average pooled features from output of vision encoder and VL proj.' and average pool T x F decoder output; this discards spatial and token-level structure that may be crucial for counting and spatial reasoning.
  • domain assumption The language decoder's textual response is directly comparable to a supervised linear probe accuracy.
    The paper compares probe accuracy, trained with labels, to zero-shot text response accuracy, treating both as measurements of the same 'space'. Different training protocols and readout mechanisms make this comparison uneven.
  • domain assumption PaintSkills, Pascal VOC, CUB, and Stanford Dogs subsets are valid proxies for fundamental visual capabilities.
    Section 3.2: the tasks are chosen because they are 'fundamental capabilities'; the 15-class subsets are not validated against the full benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities." pith.science (2026). https://pith.science/paper/DSTEQN7Y

@misc{pith2026250710442,
  author       = {Pith},
  title        = {Pith review of: Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DSTEQN7Y}},
  note         = {Machine review of arXiv:2507.10442}
}
read the original abstract

Vision-language Models (VLMs) have emerged as general-purpose tools for addressing a variety of complex computer vision problems. Such models have been shown to be highly capable, but, at the same time, lacking some basic visual understanding skills. In this paper, we set out to understand the limitations of SoTA VLMs on fundamental visual tasks by constructing a series of tests that probe which components of design, specifically, may be lacking. Importantly, we go significantly beyond the current benchmarks, which simply measure the final performance of VLM response, by also comparing and contrasting it to the performance of probes trained directly on features obtained from the visual encoder, intermediate vision-language projection and LLM-decoder output. In doing so, we uncover shortcomings in VLMs and make a number of important observations about their capabilities, robustness and how they process visual information. We hope our insights will guide progress in further improving VLMs.

Figures

Figures reproduced from arXiv: 2507.10442 by the authors.

Figure 1
Figure 1. Overview of our VLM analysis. Going beyond existing efforts that analyze VLMs as a whole, we study per￾formance of VLMs in terms of intermediate spaces that repre￾sent knowledge as it is processed through the VLM network. Specifically, we consider three spaces in VLMs: visual, VL projection and response space; to understand what aspects of visual information are captured (not captured) and where. relying more on lar… view at source ↗
Figure 2
Figure 2. Qualitative results supporting the findings of our analysis. We show prediction (correct vs incorrect) for three spaces i.e visual, VL projection and response. We notice correct predictions in intermediate spaces and incorrect predictions in response space for object recognition and counting task. Furthermore, we notice a reversal in trend for spatial understanding task, where the response space has more correct pre… view at source ↗
Figure 3
Figure 3. Effect of Scaling LLM Decoder on Response space. Refer to Appendix A.1 for elaborate results instruction-tuning for the latter models. The better performance of the linear prob (probe) compared to (text) space on InstructBLIP also supports this hypothesis; i.e., it is able to capture the answer, but not verbalize it in (text). (Fine-grained) object recognition [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Consolidation of performance for instruction￾tuned models from the BLIP family and the LLaVA family. Legend: Visual space, Text space, Response space. spatial arrangement of a man and a chair is that the man is on the chair. These models are often able to solve VQA tas…
Figure 5
Figure 5. Figure 5: Background Transformations. Samples illustrat￾ing image transformations that we consider in our analysis of performance in visual, text and response spaces of VLMs. proj spaces (∼11-12%) and relatively less in the response space (∼4-5%), pointing to loss of infor￾matio…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 28 canonical work pages

  1. [1]

    Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. 2025. Vision-language models do not understand negation. In arXiv

    work page 2025

  2. [2]

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv\'e J\'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV)

    work page 2021

  3. [3]

    Shivam Chandhok. 2024. https://arxiv.org/abs/2408.06926 Scenegpt: A language model for 3d scene understanding . Preprint, arXiv:2408.06926

    arXiv 2024

  4. [4]

    Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In International Conference on Computer Vision ( ICCV )

    work page 2023

  5. [5]

    MM-R$^3$: On (In-)Consistency of Vision-Language Models (VLMs)

    Shih-Han Chou, Shivam Chandhok, James J. Little, and Leonid Sigal. 2025. https://arxiv.org/abs/2410.04778 Mm-r ^3 : On (in-)consistency of vision-language models (vlms) . Preprint, arXiv:2410.04778

    work page Pith review arXiv 2025

  6. [6]

    Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, and K

    Tushar Choudhary, Vikrant Dewangan, Shivam Chandhok, Shubham Priyadarshan, Anushka Jain, Arun K. Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, and K. Madhava Krishna. 2024. https://doi.org/10.1109/ICRA57147.2024.10611485 Talk2bev: Language-enhanced bird’s-eye view maps for autonomous driving . In 2024 IEEE International Conference on Robotic...

    arXiv 2024

  7. [7]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. https://arxiv.org/abs/2305.06500 Instructblip: Towards general-purpose vision-language models with instruction tuning . Preprint, arXiv:2305.06500

    arXiv 2023

  8. [8]

    Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. 2024. Probing the 3D Awareness of Visual Foundation Models . In CVPR

    work page 2024

Show all 50 references
  1. [9]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2010. The pascal visual object classes ( VOC ) challenge. International Journal of Computer Vision (IJCV) , 88(2):303--338

    2010

  2. [10]

    Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Lu Yuan, and Leonid Sigal. 2024. https://arxiv.org/abs/2407.14506 On pre-training of multimodal language models customized for chart understanding . Preprint, arXiv:2407.14506

    2024 arXiv

  3. [11]

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. 2018. https://api.semanticscholar.org/CorpusID:54101493 Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness . ArXiv, ...

    2018 arXiv

  4. [12]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. https://arxiv.org/abs/1612.00837 Making the v in vqa matter: Elevating the role of image understanding in visual question answering . Preprint, CVPR:1612.00837

    2017 arXiv

  5. [13]

    Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, and Kun Zhang. 2024. https://arxiv.org/abs/2312.07424 How well does gpt-4v(ision) adapt to distribution shifts? a preliminary investigation . Preprint, arXiv:2312.07424

    2024 arXiv

  6. [14]

    Dietterich

    Dan Hendrycks and Thomas G. Dietterich. 2019. https://arxiv.org/abs/1807.01697 Benchmarking neural network robustness to common corruptions and surface variations . Preprint, arXiv:1807.01697

    2019 arXiv

  7. [15]

    John Hewitt and Percy Liang. 2019. https://doi.org/10.18653/v1/D19-1275 Designing and interpreting probes with control tasks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua...

    2019 doi

  8. [16]

    Zhe Hu, Tuo Liang, Jing Li, Yiren Lu, Yunlai Zhou, Yiran Qiao, Jing Ma, and Yu Yin. 2024. https://arxiv.org/abs/2405.19088 Cracking the code of juxtaposition: Can ai models understand the humorous contradictions . Preprint, arXiv:2405.19088

    2024 arXiv

  9. [17]

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.568 What`s up with vision-language models? investigating their struggle with spatial reasoning . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce...

    2023 doi

  10. [18]

    Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. 2011. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization (FGVC)

    2011

  11. [19]

    Jeonghwan Kim and Heng Ji. 2024. https://arxiv.org/abs/2402.16315 Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models . Preprint, arXiv:2402.16315

    2024 arXiv

  12. [20]

    Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. 2022. Illustrating reinforcement learning from human feedback (rlhf). Hugging Face Blog. Https://huggingface.co/blog/rlhf

    2022

  13. [21]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench-2: Benchmarking multimodal large language models. Computer Vision and Pattern Recognition (CVPR)

    2024

  14. [22]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 a . BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML)

    2023

  15. [23]

    Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi

    Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS

    2021

  16. [24]

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023 b . VideoChat : Chat-centric video understanding. arXiv preprint arXiv:2305.06355

    2023 arXiv

  17. [25]

    Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, and Deva Ramanan. 2024. Revisiting the role of language priors in vision-language models. In International Conference on Machine Learning. PMLR

    2024

  18. [26]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  19. [27]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Conference on Neural Information Processing Systems (NeurIPS)

    2023

  20. [28]

    Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Jia-Wang Bian, Le Zhang, Xiang Bai, and Jinhui Tang. 2019. Richer convolutional features for edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1939--1946

    2019

  21. [29]

    Hashmat Shadab Malik, Muhammad Huzaifa, Muzammal Naseer, Salman Khan, and Fahad Shahbaz Khan. 2024. Objectcompose: Evaluating resilience of vision-based models on object-to-background compositional changes. arXiv preprint arXiv:2403.04701

    2024 arXiv

  22. [30]

    Ecker, Matthias Bethge, and Wieland Brendel

    Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. 2019. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484

    2019 arXiv

  23. [31]

    Chaithanya Kumar Mummadi, Ranjitha Subramaniam, Robin Hutmacher, Julien Vitay, Volker Fischer, and Jan Hendrik Metzen. 2021. https://api.semanticscholar.org/CorpusID:233306976 Does enhanced shape bias improve neural network robustness to common corruptions? ArXiv, abs/2104.09789

    2021 arXiv

  24. [32]

    Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. 2023. https://api.semanticscholar.org/CorpusID:257102813 Teaching clip to count to ten . 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3147--3157

    2023

  25. [33]

    Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. 2024. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In CVPR

    2024

  26. [34]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Internation...

    2021

  27. [35]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. https://api.semanticscholar.org/CorpusID:258959321 Direct preference optimization: Your language model is secretly a reward model . ArXiv, abs/2305.18290

    2023 arXiv

  28. [36]

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. 2023. What does clip know about a red circle? visual prompt engineering for vlms. ICCV

    2023

  29. [37]

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. 2022. https://api.semanticscholar.org/CorpusID:248006414 Winoground: Probing vision and language models for visio-linguistic compositionality . 2022 IEEE/CVF Conference on ...

    2022

  30. [38]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Computer Vision and Pattern Recognition (CVPR)

    2024

  31. [39]

    Burg, Samuel Albanie, and Matthias Bethge

    Vishaal Udandarao, Max F. Burg, Samuel Albanie, and Matthias Bethge. 2023. https://arxiv.org/abs/2310.08577 Visual data-type understanding does not emerge from scaling vision-language models . Preprint, arXiv:2310.08577

    2023 arXiv

  32. [40]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology

    2011

  33. [41]

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. 2024. https://arxiv.org/abs/2403.10517 Videoagent: Long-form video understanding with large language model as agent . Preprint, arXiv:2403.10517

    2024 arXiv

  34. [42]

    Zongyu Wu, Yuwei Niu, Hongcheng Gao, Minhua Lin, Zhiwei Zhang, Zhifang Zhang, Qi Shi, Yilong Wang, Sike Fu, Junjie Xu, Junjie Ao, Enyan Dai, Lei Feng, Xiang Zhang, and Suhang Wang. 2025. https://arxiv.org/abs/2502.12359 Lanp: Rethinking the impact of language priors in large v...

    2025 arXiv

  35. [43]

    Danfei Xu, Yuke Zhu, Christopher Choy, and Li Fei-Fei. 2017. Scene graph generation by iterative message passing. In Computer Vision and Pattern Recognition (CVPR)

    2017

  36. [44]

    Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. 2023. https://arxiv.org/abs/2306.04356 Fine-grained visual prompting . Preprint, arXiv:2306.04356

    2023 arXiv

  37. [45]

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. https://arxiv.org/abs/2205.01917 Coca: Contrastive captioners are image-text foundation models . Preprint, arXiv:2205.01917

    2022 arXiv

  38. [46]

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Y. Zou. 2022. https://api.semanticscholar.org/CorpusID:252734947 When and why vision-language models behave like bags-of-words, and what to do about it? ArXiv, abs/2210.01936

    2022 arXiv

  39. [47]

    Kelly Zhang and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5448 Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : Analyzing and Interpreting ...

    2018 doi

  40. [48]

    Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. 2024. Why are visually-grounded language models bad at image classification? Conference on Neural Information Processing Systems (NeurIPS)

    2024

  41. [49]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  42. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.