Pith

open record

sign in

arxiv: 2310.16534 · v1 · pith:KJCXGZXO · submitted 2023-10-25 · cs.CL · cs.CV

An Early Evaluation of GPT-4V(ision)

Yang Wu , Shilong Wang , Hao Yang , Tian Zheng , Hongbo Zhang , Yanyan Zhao , Bing Qin This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:KJCXGZXOrecord.jsonopen to challenge →

classification cs.CL cs.CV
keywords gpt-4vunderstandinglanguageperformancevisualbenchmarksresultsevaluate
0
0 comments X
read the original abstract

In this paper, we evaluate different abilities of GPT-4V including visual understanding, language understanding, visual puzzle solving, and understanding of other modalities such as depth, thermal, video, and audio. To estimate GPT-4V's performance, we manually construct 656 test instances and carefully evaluate the results of GPT-4V. The highlights of our findings are as follows: (1) GPT-4V exhibits impressive performance on English visual-centric benchmarks but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows inconsistent refusal behavior when answering questions related to sensitive traits such as gender, race, and age; (3) GPT-4V obtains worse results than GPT-4 (API) on language understanding tasks including general language understanding benchmarks and visual commonsense knowledge evaluation benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both visual understanding and language understanding; (5) GPT-4V struggles to find the nuances between two similar images and solve the easy math picture puzzles; (6) GPT-4V shows non-trivial performance on the tasks of similar modalities to image, such as video and thermal. Our experimental results reveal the ability and limitations of GPT-4V and we hope our paper can provide some insights into the application and research of GPT-4V.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation

    cs.CL 2023-11 unverdicted novelty 6.0

    AMBER is an LLM-free multi-dimensional benchmark for evaluating hallucinations in MLLMs across generative and discriminative tasks.

  2. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    cs.CV 2023-06 unverdicted novelty 6.0

    MME is a manually annotated benchmark evaluating MLLMs on perception and cognition across 14 subtasks to avoid data leakage and support fair model comparisons.