On the Limitations of Vision-Language Models in Understanding Image Transforms

Ahmad Mustafa Anis; Hasnain Ali; Saquib Sarfraz

On the Limitations of Vision-Language Models in Understanding Image Transforms

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.09837 v2 pith:EGOL54Y7 submitted 2025-03-12 cs.CV cs.AIcs.CL

Ahmad Mustafa Anis , Hasnain Ali , Saquib Sarfraz This is my paper

classification cs.CVcs.AIcs.CL

keywords imagemodelsunderstandingdownstreamimage-leveltaskstransformationsvideo

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Vision Language Models (VLMs) have demonstrated significant potential in various downstream tasks, including Image/Video Generation, Visual Question Answering, Multimodal Chatbots, and Video Understanding. However, these models often struggle with basic image transformations. This paper investigates the image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by Google. Our findings reveal that these models lack comprehension of multiple image-level augmentations. To facilitate this study, we created an augmented version of the Flickr8k dataset, pairing each image with a detailed description of the applied transformation. We further explore how this deficiency impacts downstream tasks, particularly in image editing, and evaluate the performance of state-of-the-art Image2Image models on simple transformations.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks
cs.CV 2026-06 unverdicted novelty 7.0 of 10

The paper constructs the VIBE benchmark and evaluates six visual in-context learning models on 14 datasets, 12 tasks, and 106 combinations under a unified one-shot protocol, revealing limitations and failure modes.
Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspective
cs.CV 2026-04 unverdicted novelty 6.0 of 10

TaCo contrastively embeds semantic, generative, and transformation tasks from medical imaging into a joint space to reveal which tasks cluster, blend, or remain distinct.
Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
cs.CV 2025-09 conditional novelty 5.0 of 10

Tree-based reasoning prompts consistently underperform standard zero-shot prompting for VLM image classification on GTSRB and CIFAR-10 across three models.
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
cs.CV 2026-04 unverdicted novelty 4.0 of 10

Robust CLIP models amplify vulnerabilities to natural adversarial scenarios while standard CLIP shows large performance drops on natural language-induced adversarial examples in zero-shot classification, segmentation,...

Pith tools