REVIEW 6 cited by
Why are Visually-Grounded Language Models Bad at Image Classification?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset.
Forward citations
Cited by 6 Pith papers
-
Unpacking Hateful Memes: Presupposed Context and False Claims
A hateful-meme detector that combines presupposed-context fusion, LLM-based social perception, and cross-modal reference graphs outperforms prior models on three benchmarks and transfers to fake news.
-
Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models
MuGCP adapts CLIP by decoding instance-specific prompts from a frozen MLLM's KV cache and fusing them with visual prompts, achieving state-of-the-art few-shot classification on 14 datasets.
-
AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models
AIGI-Holmes combines visual expert pretraining, SFT on explanation data, and direct preference optimization to deliver human-verifiable explanations and top detection accuracy on unseen AI generators.
-
Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
Tree-based reasoning prompts consistently underperform standard zero-shot prompting for VLM image classification on GTSRB and CIFAR-10 across three models.
-
Filter-And-Refine: A MLLM Based Cascade System for Industrial-Scale Video Content Moderation
A cascade of an embedding router and a fine-tuned multimodal LLM ranker is claimed to improve content moderation F1 by 66.5% while using 1.5% of the compute of direct LLM deployment.
-
Foundation Models for Astrophysics
Astronomical 'foundation models' largely reuse transformers and self-supervised pretraining, but evidence of transfer to new instruments, populations, or tasks remains rare; the paper argues such evidence, not archite...
Discussion (0). Sign in to comment.