MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

Bo Li; Caifeng Shan; Chaoyou Fu; Haodong Duan; Liang Wang; Ran He; Shukang Yin; Sirui Zhao; Xing Sun; Xinyu Fang

REVIEW 2 major objections 7 minor 20 cited by

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

T0 review · 2 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This survey maps the entire landscape of how multimodal large language models are evaluated, from capability benchmarks to construction pipelines to scoring methods.

desk verdict A useful but curated map of MLLM evaluation; needs a stated selection protocol and citation fixes before it can be the definitive reference it claims to be. read the letter →

arxiv 2411.15296 v2 pith:K6XCMMNJ submitted 2024-11-22 cs.CV cs.AIcs.CL

Chaoyou Fu , Yi-Fan Zhang , Shukang Yin , Bo Li , Xinyu Fang , Sirui Zhao , Haodong Duan , Xing Sun

show 4 more authors

Ziwei Liu Liang Wang Caifeng Shan Ran He

This is my paper · ORCID

classification cs.CVcs.AIcs.CL

keywords multimodallargelanguagemodelsMLLMevaluationbenchmarktaxonomyconstructionmetricstoolkitshallucinationvision-language

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey aims to organize the sprawling field of multimodal large language model (MLLM) evaluation into a single map with four coordinated parts. It claims that every current benchmark can be filed under three top-level branches: foundational capabilities (perception, OCR, charts, math, multilingual, instruction following, multi-round and multi-image understanding, video), model self-analysis (hallucination, bias, safety, causation), and extended applications (medicine, emotion, remote sensing, agents, code, GUI, transfer, knowledge editing, embodied AI, driving). It then lays out the pipeline for building a benchmark, the ways to judge outputs (human, LLM/MLLM, or script), the metric families that summarize scores, and the toolkits that run evaluations. If the map is accurate, a researcher can locate the right benchmark for a given question, avoid known construction pitfalls such as data leakage and text-only solvable questions, and see where the field's blind spots are.

What carries the argument

The central object is the taxonomy-pipeline pair. The taxonomy (Fig. 4) sorts benchmarks into three top-level branches and many sub-branches, giving each benchmark a location in capability space so that a researcher can search the field by what they want to test. The pipeline (Fig. 6) organizes the choices a benchmark builder faces: where data comes from (existing datasets, modified data, or internet gathering), how QA pairs are annotated (automatic construction, LLM/MLLM prompting, or manual annotation), which judge evaluates responses (human, model, or script), which metric family summarizes results (deterministic or non-deterministic), and which toolkit executes the evaluation. The taxonomy does the work of making the field searchable, while the pipeline does the work of making new benchmarks constructible and comparable.

What would settle it

Enumerate every MLLM evaluation paper released in a fixed window (for example, 2023–2024) using a neutral literature index, attempt to file each benchmark into the survey's three-branch taxonomy, and record any benchmark family that fits nowhere; if a substantial family such as audio-only or 3D evaluation is missing or misclassified, the survey's comprehensiveness claim is weakened.

Watch

Extended reading notes

Core claim

The paper's central claim is that MLLM evaluation can be surveyed systematically along four dimensions: what capabilities are assessed, how benchmarks are built, how performance is measured, and where the next benchmarks should focus. Its main organizing device is a three-branch taxonomy of benchmarks, with sub-branches ranging from comprehensive evaluation and OCR to hallucination, safety, and autonomous driving. Its second device is a construction-and-measurement pipeline that runs from data collection through annotation to judge, metric, and toolkit. The paper introduces no new benchmark; its contribution is the synthesis itself, together with practical guidance for choosing among existing benchmarks and for building new ones that avoid known failure modes.

Load-bearing premise

The survey's field-level conclusions rest on its benchmark sample being representative, yet it states no inclusion or exclusion criteria, and several of the most prominently featured benchmarks were developed by the authors themselves.

Editorial extensions

If this is right

A researcher can use the taxonomy as a checklist, locating the capability branch a new model claims to improve and selecting the corresponding benchmarks instead of relying on one aggregate leaderboard.
A benchmark builder can use the pipeline discussion to anticipate failure modes such as multiple-choice leakage, data contamination, and questions answerable without looking at the image.
The survey's gap analysis identifies where new benchmarks are most needed: instruction following, multi-turn dialogue, creativity, task-specific commercial applications, and audio and 3D modalities.
Because judge choice affects open-ended scores, results produced by different LLM judges or human judges are not directly comparable across papers.
The toolkit section implies that standardized evaluation infrastructure is becoming available, which should reduce the cost of reproducing and comparing MLLM results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A reader should weigh the benchmark sample: several of the most prominently featured benchmarks were developed by this paper's own author group, and no inclusion or exclusion criteria for the survey are stated, so the field-level takeaways should be read as conditioned on that sample.
The taxonomy invites a testable extension: compute rank correlations of models across benchmark branches; if rankings diverge, 'MLLM capability' is not a single scalar and results should be reported per capability branch.
The reported gaps in audio and 3D evaluation suggest that an omni-modal benchmark suite reusing this taxonomy could reveal whether current models' cross-modal reasoning is general or mostly vision-language.
Since benchmarks are grouped by their declared capability rather than the skills actually required, a follow-up could re-annotate each benchmark by the minimal skill set needed to solve it, then redraw the taxonomy to test the map's validity.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a genuinely useful map of the MLLM evaluation landscape, but it is a curated map, not a systematic one. The lack of a stated selection protocol and a handful of citation errors keep it from being the definitive reference it claims to be.

What is new: the organizing taxonomy itself—foundational capabilities, model self-analysis, extended applications—and the four-part pipeline (benchmark construction, judge, metric, toolkit) are a sensible way to structure the field. Section 4 on construction is the most useful part; it lays out data collection methods, annotation strategies, and practical pitfalls like MCQ leakage and data contamination that are easy to overlook. Section 5's breakdown of human, LLM-based, and script-based judges is also clear and accurate. The coverage is broad: video, OCR, math, hallucination, medical, driving, agents—most of the benchmarks a researcher would need are here.

Where it is soft: the selection of benchmarks in Fig. 4 and Table 1 is never justified. The authors lead MME, Video-MME, MME-RealWorld, MMBench, VLMEvalKit, and LMMs-Eval, and those projects appear prominently. The footnote discloses this, which is good, but disclosure is not the same as a protocol. Without inclusion/exclusion criteria, "representative benchmarks" is doing a lot of work, and any field-level conclusion (e.g. open-source matching closed-source) inherits the selection bias. This is an internal methodological gap, not a fatal one—the taxonomy stands on its own—but it prevents the paper from being the systematic survey the abstract promises.

There are also mechanical errors that a reference work cannot afford. Section 2.1 cites [12] (Chameleon, compositional reasoning) for purely discrete multimodal modeling; the actual Chameleon paper is about plug-and-play reasoning, not discrete tokens. Table 1 lists RefCOCO+ and RefCOCOg with reference [81], but those are from [82]. Small, but they erode trust in a survey whose value is precisely its reliability.

Who this is for: graduate students and researchers new to MLLM evaluation who need a bird's-eye view of the benchmark zoo. It is also a decent checklist for someone building a new benchmark. It is not a source to cite for specific benchmark statistics without checking the original.

Recommendation: send it to review. A serious referee can push for a stated selection protocol and fix the citation errors; the core taxonomy and construction guidance are worth publishing.

Referee Report

2 major / 7 minor

Summary. This paper presents a survey of evaluation methods and benchmarks for multimodal large language models (MLLMs). It organizes the benchmark landscape into a hierarchy of three top-level branches: foundational capabilities (comprehensive evaluation, OCR, chart/document understanding, mathematics, multidisciplinary knowledge, multilingual ability, instruction following, multi-round QA, multi-image understanding, interleaved content, high-resolution input, visual grounding, fine-grained perception, and video understanding), model self-analysis (hallucination, bias, safety, causation), and extended applications (medical imaging, emotion analysis, remote sensing, agents, code generation, GUI understanding, transfer capability, knowledge editing, embodied AI, and autonomous driving). It then discusses benchmark construction (Section 4), evaluation judges and metrics (Sections 5–6), four toolkits (Section 7), and future directions (Section 8). The paper's stated goal is to help researchers select and build benchmarks and to systematize MLLM evaluation, and its four-part structure matches the promises of the abstract.

Significance. The manuscript's main strength is its breadth and organization: roughly 200 benchmarks are arranged in a readable three-branch taxonomy, and the pipeline discussion in Sections 4–6 (construction, judge, metric, toolkit) gives practitioners concrete guidance. The discussion of multiple-choice-question leakage, data leakage, and vision-centric design in Section 4.3 distills recurring methodological lessons that are useful beyond any single benchmark. The companion project page (https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Benchmarks) makes the survey a living resource, which is a genuine practical asset. The survey does not introduce new theory or new measurements, which is appropriate for its genre; its validity depends on whether the displayed benchmark selection is representative of the field, and that is where the manuscript is currently weakest (see Major Comment 1).

major comments (2)

[§3, Fig. 4, Table 1, author footnote] The central claim of the paper is that it provides a comprehensive and systematic survey of MLLM evaluation, but the paper never states inclusion or exclusion criteria for the benchmarks shown in Fig. 4 and Table 1. The author footnote discloses that the authors lead the MME, MMBench, and LLaVA teams, and benchmarks or toolkits from those teams are among the most prominently featured (MME [24], MMBench [22], MME-RealWorld [35], Video-MME [87], MMBench-Video [91], VLMEvalKit, LMMs-Eval, and OpenCompass). Because the lists are presented without a protocol, the reader cannot determine whether the taxonomy in Fig. 4 and the aggregate conclusions in Sections 3.1.1 and 3.1.9 reflect the field or a curated subset of the literature; the concern is not the (disclosed) involvement of the authors in several featured projects but the fact that the comprehensiveness claim is unverifiable. A concrete instance is the claim in §3.1.1 that 'open-source models have increasingly matched or even surpassed closed-source counterparts,' which cites only [22], [24], [35], all from the authors' own teams. I note that the survey does include many third-party benchmarks (e.g., MathVista, MMMU, POPE, HallusionBench), so the selection is not exclusive; nevertheless, the paper should state how the benchmark sets were assembled (search scope, time window, selection criteria) and add a limitations statement, or explicitly reframe the selection as curated rather than comprehensive.
[§2.1 (Fig. 2 caption), Table 1, reference list] Several citation errors undermine the survey's reliability as a literature map. (a) The Fig. 2 caption attributes 'purely discrete modeling to achieve both understanding and generation' to reference [12], which is Lu et al., 'Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models'; that paper describes compositional prompting with LLMs, not discrete multimodal modeling, and the intended citation appears to be missing from the reference list. (b) Table 1 lists RefCOCO+ and RefCOCOg with citation [81] (Kazemzadeh et al., ReferItGame), while the text in §3.1.12 attributes them to [82] (Mao et al.); the RefCOCO family originates from [82], so the table entries should be corrected and made consistent with the text. (c) Several papers are assigned two reference numbers: [48] and [180] (AI2D), [70] and [210] (MIA-Bench), [87] and [178] (Video-MME), [104] and [204] (Bingo), and [109] and [186] (MHaluBench). (d) Reference [118] is used for what appear to be several distinct items: the OOD benchmarks in §3.2.3, the VLLM-safety-benchmark in §3.2.3, and VLAA in §3.3.7; if these are all from the same source, a cross-reference note is needed, otherwise separate citations are required. Because readers of a survey rely on the bibliography to locate the benchmarks discussed, this batch of errors should be corrected systematically before publication.

minor comments (7)

[Abstract, Fig. 3, §3.1.1, §4.1, Fig. 6, §8] The manuscript contains numerous typos that should be fixed in a proofreading pass: 'extented applications' and 'counstruction' in the abstract, 'instrctions' in the Fig. 3 caption, 'benmarks' in §3.1.1, 'edites' in §4.1, 'lager' in §3.1.1, 'BLUE4' for BLEU4 in Fig. 6, and 'taxonmomy' and 'Recongi tion' in §8.1 and §8.4.
[§3.1.1, §5.1, Fig. 4] The benchmark from reference [29] (WildVision) is referred to inconsistently as 'WV-Bench' (Fig. 4, §3.1.1, §6.2.1) and 'WV-Arena' (§5.1); one name should be adopted throughout.
[Table 1, Fig. 6] In Table 1, the category labels 'EmbodiedAI' and 'EmbodiedAI(Video)' do not match the survey's own naming in the text ('Embodied AI', §3.3.9), and 'MMHAL-BENCH' should be 'MMHal-Bench' as in Fig. 4; the duplicated citation '[149], [149]' in Fig. 6's metric list should be de-duplicated.
[§3.3.5] The text refers to 'WCGB [145]' for webpage-to-code generation, but Fig. 4 and the reference list use 'Web2Code' for [145]; the names should be aligned, or a separate citation provided for WCGB if it is a different dataset.
[§7.2] The mention of LiveBench in Section 7.2 has no citation and no reference-list entry; given that the surrounding paragraph is about LMMs-Eval, the connection to multimodal evaluation should also be made explicit.
[§3.1.5, Fig. 4] MMMU-Pro [62] appears in Fig. 4 but is not discussed in Section 3.1.5; since it is a notable recent robustness-oriented extension of MMMU, a one-sentence discussion would better match the survey's coverage claims.
[Title] The title 'MME-Survey' foregrounds a single benchmark from the authors' own team rather than the survey's general scope; a scope-reflecting title would match the content and the disclosure footnote more accurately.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey is a literature review whose claims are citations, not derivations; self-citation is present but not load-bearing.

full rationale

The paper makes no 'predictions' or 'first-principles' derivations. Its contribution is an organizational taxonomy of existing benchmarks and an account of benchmark construction and evaluation practice. Each substantive claim is a summary of cited benchmark papers, including many third-party benchmarks (VQA v2, VizWiz, MMMU, MathVista, etc.) alongside the authors' own (MME, MMBench, Video-MME, VLMEvalKit, LMMs-Eval), and the author footnote discloses those affiliations. The absence of an explicit benchmark selection protocol for Fig. 4 and Table 1 creates a possible representativeness or self-selection limitation, but it does not make any claim equivalent to its inputs by construction: the survey does not fit parameters, define a capability in terms of a benchmark, or import a uniqueness theorem from the authors' prior work. Section 7's toolkit coverage is dominated by the authors' own VLMEvalKit and LMMs-Eval, and some field-level takeaways cite the authors' own benchmarks, but these are descriptive citations to externally published, falsifiable results with independent content. The central organizational claim remains a literature review rather than a self-referential derivation, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey introduces no fitted parameters and no new postulated entities. Its load-bearing content is taxonomic and interpretive: the three-way capability split, the reliance on cited benchmark statistics, and the generalization from selected benchmarks to MLLMs as a whole.

assumptions (3)

domain assumption MLLM evaluation can be organized into three top-level categories: foundational capability, model self-analysis, and extended applications.
This taxonomy is presented in Section 3 and Fig. 4; the entire survey structure depends on the claim that these categories reflect the actual space of MLLM evaluation.
domain assumption The benchmark statistics cited in Table 1 and throughout Section 3 accurately reflect the original papers.
The survey aggregates QA-pair counts, answer types, and metrics from dozens of external papers without re-verifying them, and several citation errors suggest this assumption is fragile.
domain assumption The qualitative conclusions, such as 'MLLMs struggle with fine-grained perception' and 'longer videos cause performance degradation', generalize from the selected benchmarks to the model class.
These conclusions are drawn by aggregating results from the benchmarks in Section 3.1; if the benchmark sample is biased, the generalizations may not hold.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs." pith.science (2026). https://pith.science/paper/K6XCMMNJ

@misc{pith2026241115296,
  author       = {Pith},
  title        = {Pith review of: MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K6XCMMNJ}},
  note         = {Machine review of arXiv:2411.15296}
}

read the original abstract

As a prominent direction of Artificial General Intelligence (AGI), Multimodal Large Language Models (MLLMs) have garnered increased attention from both industry and academia. Building upon pre-trained LLMs, this family of models further develops multimodal perception and reasoning capabilities that are impressive, such as writing code given a flow chart or creating stories based on an image. In the development process, evaluation is critical since it provides intuitive feedback and guidance on improving models. Distinct from the traditional train-eval-test paradigm that only favors a single task like image classification, the versatility of MLLMs has spurred the rise of various new benchmarks and evaluation methods. In this paper, we aim to present a comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the summarised benchmarks types divided by the evaluation capabilities, including foundation capabilities, model self-analysis, and extented applications; 2) the typical process of benchmark counstruction, consisting of data collection, annotation, and precautions; 3) the systematic evaluation manner composed of judge, metric, and toolkit; 4) the outlook for the next benchmark. This work aims to offer researchers an easy grasp of how to effectively evaluate MLLMs according to different needs and to inspire better evaluation methods, thereby driving the progress of MLLM research.

Figures

Figures reproduced from arXiv: 2411.15296 by the authors.

**Figure 2.** Typical MLLM architecture. Tokenizer and De [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Illustration of three training stages of MLLMs. In [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Categories of MLLM benchmarks. the largest image resolution. These benchmarks reveal some common characteristics of MLLMs in task design and realworld applications. Fine-grained perception tasks continue to challenge existing models [27], [29]. In contrast, models per…

**Figure 5.** Figure 5: Examples of different MLLM evaluation tasks. The answer can be Open-Ended, Yes-or-No, or Multi-Choice. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Evaluation pipeline of MLLMs. combined with subsequent human review can yield a reasonably high-quality benchmark. Benchmarks like MMStar [28], Seed-Bench [23], MMT-Bench [26], and SEEDBench [23] have adopted this approach. This annotation process is inherently limi…

**Figure 7.** Figure 7: Major components and evaluation pipeline of the toolkit. By integrating various types of datasets and models, the [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
cs.AI 2026-07 conditional novelty 7.0 of 10

Context-guided multi-turn interviews reveal more VLM hallucinations than static benchmarks, and those hallucinations increase with conversational history and false-premise questions.
Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis
cs.CV 2026-07 conditional novelty 6.5 of 10

Fabricating appearance-varied images and modality contexts from RGB lets LMMs improve perception and understanding on real and synthetic non-RGB modalities without in-modality training.
SABRE: Scalable and Automated Benchmarking of VLMs under Stress
cs.CV 2026-08 conditional novelty 6.0 of 10

SABRE is a modular pipeline for automatically building and refreshing VLM stress tests, and its SABRE-Prior instantiation finds that six frontier VLMs score 17.8 to 31.3 percent when visual evidence conflicts with wor...
Robix: A Unified Model for Robot Interaction, Reasoning and Planning
cs.AI 2025-09 conditional novelty 6.0 of 10

A three-stage-trained VLM unifies robot planning and dialogue, and beats commercial VLMs on the authors' interactive-task benchmarks.
LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
cs.CL 2025-08 unverdicted novelty 6.0 of 10

LLaSO releases a 3.8B speech-language model, 25.5M training instances, and an evaluation benchmark, claiming a normalized score of 0.72.
COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark
cs.CV 2025-07 conditional novelty 6.0 of 10

COREVQA introduces a 5,608-pair true/false visual entailment benchmark for crowd images on which the strongest tested vision-language models reach only 77.57% accuracy.
VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?
cs.CV 2025-06 conditional novelty 6.0 of 10

A new benchmark with edited-image question pairs shows that multimodal reasoning models lose accuracy when visual cues change, suggesting their reasoning is often not faithfully tied to the image.
Mitigating Object Hallucination via Robust Local Perception Search
cs.CV 2025-06 conditional novelty 6.0 of 10

A training-free decoding method that uses an MLLM's own local object descriptions as a reward prior, combined with CLIP similarity, to cut object hallucination, especially under adversarial image noise.
BYO-Eval: Build Your Own Dataset for Fine-Grained Visual Assessment of Multimodal Language Models
cs.CV 2025-06 conditional novelty 6.0 of 10

A Blender-based diagnostic toolkit that tests VLMs on fine-grained visual skills by varying one visual attribute at a time, exposing failure modes that coarse benchmarks miss.
Abstractive Visual Understanding of Multi-modal Structured Knowledge: A New Perspective for MLLM Evaluation
cs.CV 2025-06 conditional novelty 6.0 of 10

A new benchmark, M3STR, renders knowledge-graph subgraphs as images and shows current MLLMs score near random on anomaly detection and poorly on entity counting.
FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
cs.CL 2025-05 conditional novelty 6.0 of 10

FinMME is a new 11,099-sample financial chart benchmark where top AI models average around 50% and FinScore adds penalties for guessing.
STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMs
cs.CV 2025-05 conditional novelty 6.0 of 10

STAR-R1 uses single-stage reinforcement learning with fine-grained rewards to improve spatial transformation reasoning in multimodal LLMs, outperforming supervised fine-tuning on cross-view TVR tasks.
MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
cs.CL 2025-02 conditional novelty 6.0 of 10

A human-annotated multimodal preference dataset plus critique-based reward modeling and reward-margin-weighted DPO improves MLLM performance across many benchmarks.
DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Risks
cs.RO 2025-11 reject novelty 5.0 of 10

A new benchmark claims to be the first to test VLMs on both external and in-cabin driving risks, and reports a fine-tuned model far outperforming all baselines.
Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
cs.CV 2025-07 conditional novelty 5.0 of 10

Selecting the intermediate layer where image-conditioned and text-only predictions diverge most, and adding that layer's contrastive visual signal back to the final logits, reduces object hallucinations in four large ...
Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning
cs.CV 2025-06 reject novelty 5.0 of 10

A new micro-edit dataset and fine-tuning recipe appear to help multimodal LLMs notice small visual changes, but the central 'feature consistency loss' claim is not present in the method.
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
cs.CV 2025-05 conditional novelty 5.0 of 10

HumaniBench, a news-image benchmark with 32,000 expert-verified items across seven tasks, reveals that no evaluated multimodal model aligns well across all seven human-centric principles.
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
cs.CL 2024-12 conditional novelty 4.0 of 10

A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.
Vision Generalist Model: A Survey
cs.CV 2025-06 conditional novelty 3.0 of 10

A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.
Visual Large Language Models for Generalized and Specialized Applications
cs.CV 2025-01 conditional novelty 3.0 of 10

This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

Reference graph

Works this paper leans on

232 extracted references · 139 linked inside Pith · cited by 20 Pith papers

[12]

Chameleon: Plug-and-play compositional reasoning with large language models,

P . Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.- C. Zhu, and J. Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,” in NeurIPS, 2024. 2

2024
[81]

Refer- itgame: Referring to objects in photographs of natural scenes,

S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg, “Refer- itgame: Referring to objects in photographs of natural scenes,” in EMNLP, 2014. 4, 7, 8

2014
[82]

Generation and comprehension of unambiguous object descriptions,

J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions,” in CVPR, 2016. 4, 7

2016
[24]

Mme: A comprehensive eval- uation benchmark for multimodal large language models,

C. Fu, P . Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun et al. , “Mme: A comprehensive eval- uation benchmark for multimodal large language models,” arXiv:2306.13394, 2023. 3, 4, 8, 14, 15, 16, 19

arXiv 2023
[22]

Mmbench: Is your multi-modal model an all-around player?

Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu et al. , “Mmbench: Is your multi-modal model an all-around player?” arXiv:2307.06281, 2023. 3, 4, 8, 13, 14, 15, 16, 19

arXiv 2023
[35]

Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?

Y.-F. Zhang, H. Zhang, H. Tian, C. Fu, S. Zhang, J. Wu, F. Li, K. Wang, Q. Wen, Z. Zhang et al., “Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?” arXiv:2408.13257, 2024. 3, 4, 7, 8, 13, 14, 15, 16

arXiv 2024
[91]

Mmbench-video: A long-form multi-shot benchmark for holistic video understanding,

X. Fang, K. Mao, H. Duan, X. Zhao, Y. Li, D. Lin, and K. Chen, “Mmbench-video: A long-form multi-shot benchmark for holistic video understanding,” arXiv:2406.14515, 2024. 4, 9

arXiv 2024
[48]

A diagram is worth a dozen images,

A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “A diagram is worth a dozen images,” inECCV, 2016. 4, 6

2016
[180]

A diagram is worth a dozen images,

A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “A diagram is worth a dozen images,” inECCV, 2016. 8

2016
[210]

Mia-bench: Towards better instruction following evaluation of multimodal llms,

Y. Qian, H. Ye, J.-P . Fauconnier, P . Grasch, Y. Yang, and Z. Gan, “Mia-bench: Towards better instruction following evaluation of multimodal llms,” arXiv:2407.01509, 2024. 16

arXiv 2024
[178]

Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis,

C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang et al. , “Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis,” arXiv:2405.21075, 2024. 6, 14, 15, 16

arXiv 2024
[204]

Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges,

C. Cui, Y. Zhou, X. Yang, S. Wu, L. Zhang, J. Zou, and H. Yao, “Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges,” arXiv:2311.03287, 2023. 14, 16

arXiv 2023
[186]

Unified hallucination detection for multimodal large language models,

X. Chen, C. Wang, Y. Xue, N. Zhang, X. Yang, Q. Li, Y. Shen, J. Gu, and H. Chen, “Unified hallucination detection for multimodal large language models,” arXiv:2402.03190, 2024. 9

arXiv 2024
[118]

How many unicorns are in this image? a safety evaluation benchmark for vision llms,

H. Tu, C. Cui, Z. Wang, Y. Zhou, B. Zhao, J. Han, W. Zhou, H. Yao, and C. Xie, “How many unicorns are in this image? a safety evaluation benchmark for vision llms,” arXiv:2311.16101, 2023. 4, 10, 12, 16

arXiv 2023

Show all 232 references

[1]

A survey of large language models,

W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv:2303.18223, 2023. 1

2023 arXiv
[2]

Instruction tuning with gpt-4,

B. Peng, C. Li, P . He, M. Galley, and J. Gao, “Instruction tuning with gpt-4,” arXiv:2304.03277, 2023. 1

2023 arXiv
[3]

Lan- guage models are few-shot learners,

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhari- wal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell et al., “Lan- guage models are few-shot learners,” in NeurIPS, 2020. 1

2020
[4]

Multimodal ma- chine learning: A survey and taxonomy,

T. Baltrušaitis, C. Ahuja, and L.-P . Morency, “Multimodal ma- chine learning: A survey and taxonomy,” IEEE Trans. Pattern Anal. Mach. Intell., 2018. 1

2018
[5]

A survey on multimodal large language models,

S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” arXiv:2306.13549,

arXiv
[6]

Vita: Towards open-source interactive omni multimodal llm,

C. Fu, H. Lin, Z. Long, Y. Shen, M. Zhao, Y. Zhang, X. Wang, D. Yin, L. Ma, X. Zheng et al. , “Vita: Towards open-source interactive omni multimodal llm,” arXiv:2408.05211, 2024. 1, 20

2024 arXiv
[7]

Multimodal learning with transformers: A survey,

P . Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” IEEE Trans. Pattern Anal. Mach. Intell. ,
[8]

Vision-language models for vision tasks: A survey,

J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell.,
[9]

Vision+ x: A survey on multimodal learning in the light of data,

Y. Zhu, Y. Wu, N. Sebe, and Y. Yan, “Vision+ x: A survey on multimodal learning in the light of data,” IEEE Trans. Pattern Anal. Mach. Intell., 2024. 2

2024
[10]

Diffusion models in vision: A survey,

F.-A. Croitoru, V . Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 2023. 2

2023
[11]

Multimodal image synthesis and editing: The generative ai era,

F. Zhan, Y. Yu, R. Wu, J. Zhang, S. Lu, L. Liu, A. Kortylewski, C. Theobalt, and E. Xing, “Multimodal image synthesis and editing: The generative ai era,” IEEE Trans. Pattern Anal. Mach. Intell., 2023. 2

2023
[13]

Self-instruct: Aligning language model with self generated instructions,

Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language model with self generated instructions,” arXiv:2212.10560, 2022. 3

2022 arXiv
[14]

Visual instruction tuning,

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in NeurIPS, 2023. 3, 10, 15, 16

2023
[16]

Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback,

T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H.-T. Zheng, M. Sun et al., “Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback,” arXiv:2312.00849, 2023. 3

2023 arXiv
[17]

Silkie: Preference distillation for large visual language models,

L. Li, Z. Xie, M. Li, S. Chen, P . Wang, L. Chen, Y. Yang, B. Wang, and L. Kong, “Silkie: Preference distillation for large visual language models,” arXiv:2312.10665, 2023. 3

2023 arXiv
[18]

Making the v in vqa matter: Elevating the role of image un- derstanding in visual question answering,

Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image un- derstanding in visual question answering,” in CVPR, 2017. 3, 4, 8, 14

2017
[19]

Vizwiz grand challenge: Answering visual questions from blind people,

D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P . Bigham, “Vizwiz grand challenge: Answering visual questions from blind people,” in CVPR, 2018. 3, 4, 8, 14, 16

2018
[20]

Lvlm-ehub: A comprehen- sive evaluation benchmark for large vision-language models,

P . Xu, W. Shao, K. Zhang, P . Gao, S. Liu, M. Lei, F. Meng, S. Huang, Y. Qiao, and P . Luo, “Lvlm-ehub: A comprehen- sive evaluation benchmark for large vision-language models,” arXiv:2306.09265, 2023. 3, 4, 8, 13, 14

2023 arXiv
[21]

Lamm: Language-assisted multi- modal instruction-tuning dataset, framework, and benchmark,

Z. Yin, J. Wang, J. Cao, Z. Shi, D. Liu, M. Li, X. Huang, Z. Wang, L. Sheng, L. Baiet al., “Lamm: Language-assisted multi- modal instruction-tuning dataset, framework, and benchmark,” in NeurIPS, 2024. 3, 4, 14, 16

2024
[23]

Seed- bench: Benchmarking multimodal llms with generative compre- hension,

B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan, “Seed- bench: Benchmarking multimodal llms with generative compre- hension,” arXiv:2307.16125, 2023. 3, 4, 8, 14

2023 arXiv
[25]

Seed-bench-2: Benchmarking multimodal large language mod- els,

B. Li, Y. Ge, Y. Ge, G. Wang, R. Wang, R. Zhang, and Y. Shan, “Seed-bench-2: Benchmarking multimodal large language mod- els,” arXiv:2311.17092, 2023. 3, 4, 13, 14, 16

2023 arXiv
[26]

Mmt-bench: A comprehen- sive multimodal benchmark for evaluating large vision-language models towards multitask agi,

K. Ying, F. Meng, J. Wang, Z. Li, H. Lin, Y. Yang, H. Zhang, W. Zhang, Y. Lin, S. Liu et al. , “Mmt-bench: A comprehen- sive multimodal benchmark for evaluating large vision-language models towards multitask agi,” arXiv:2404.16006, 2024. 3, 4, 13, 14, 19

2024 arXiv
[27]

Blink: Multimodal large language JOURNAL OF LATEX CLASS FILES, NOVEMBER 2024 21 models can see but not perceive,

X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna, “Blink: Multimodal large language JOURNAL OF LATEX CLASS FILES, NOVEMBER 2024 21 models can see but not perceive,” arXiv:2404.12390, 2024. 3, 4, 14, 15

2024 arXiv
[28]

Are we on the right way for evaluating large vision-language models?

L. Chen, J. Li, X. Dong, P . Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin et al. , “Are we on the right way for evaluating large vision-language models?” arXiv:2403.20330,

arXiv
[29]

Wildvision: Evaluating vision-language models in the wild with human preferences,

Y. Lu, D. Jiang, W. Chen, W. Y. Wang, Y. Choi, and B. Y. Lin, “Wildvision: Evaluating vision-language models in the wild with human preferences,” arXiv:2406.11069, 2024. 3, 4, 14, 15, 19

2024 arXiv
[30]

Visit-bench: A benchmark for vision-language instruction following inspired by real-world use,

Y. Bitton, H. Bansal, J. Hessel, R. Shao, W. Zhu, A. Awadalla, J. Gardner, R. Taori, and L. Schimdt, “Visit-bench: A benchmark for vision-language instruction following inspired by real-world use,” arXiv:2308.06595, 2023. 3, 4, 14, 17, 19

2023 arXiv
[31]

Mm-vet: Evaluating large multimodal models for integrated capabilities,

W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, “Mm-vet: Evaluating large multimodal models for integrated capabilities,” in ICML, 2024. 4, 5, 8, 13, 14, 15

2024
[32]

Touchstone: Evaluating vision-language models by language models,

S. Bai, S. Yang, J. Bai, P . Wang, X. Zhang, J. Lin, X. Wang, C. Zhou, and J. Zhou, “Touchstone: Evaluating vision-language models by language models,” arXiv:2308.16890, 2023. 4, 5, 14, 15, 16

2023 arXiv
[33]

Infimm-eval: Complex open- ended reasoning evaluation for multi-modal large language mod- els,

X. Han, Q. You, Y. Liu, W. Chen, H. Zheng, K. Mrini, X. Lin, Y. Wang, B. Zhai, J. Yuan et al. , “Infimm-eval: Complex open- ended reasoning evaluation for multi-modal large language mod- els,” arXiv:2311.11567, 2023. 4, 5, 14, 16

2023 arXiv
[34]

Cambrian-1: A fully open, vision-centric exploration of multimodal llms,

S. Tong, E. Brown, P . Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan et al., “Cambrian-1: A fully open, vision-centric exploration of multimodal llms,” arXiv:2406.16860,

arXiv
[36]

Towards vqa models that can read,

A. Singh, V . Natarjan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” in CVPR, 2019. 4, 5, 8, 14

2019
[37]

Ocr-vqa: Visual question answering by reading text in images,

A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty, “Ocr-vqa: Visual question answering by reading text in images,” in ICDAR,
[38]

Websrc: A dataset for web-based structural reading comprehension,

X. Chen, Z. Zhao, L. Chen, D. Zhang, J. Ji, A. Luo, Y. Xiong, and K. Yu, “Websrc: A dataset for web-based structural reading comprehension,” arXiv:2101.09465, 2021. 4, 5, 14, 16

2021 arXiv
[39]

On the hidden mystery of ocr in large multimodal models,

Y. Liu, Z. Li, B. Yang, C. Li, X. Yin, C.-l. Liu, L. Jin, and X. Bai, “On the hidden mystery of ocr in large multimodal models,” arXiv:2305.07895, 2023. 4, 5, 8, 14

2023 arXiv
[40]

Seed-bench- 2-plus: Benchmarking multimodal large language models with text-rich visual comprehension,

B. Li, Y. Ge, Y. Chen, Y. Ge, R. Zhang, and Y. Shan, “Seed-bench- 2-plus: Benchmarking multimodal large language models with text-rich visual comprehension,” arXiv:2404.16790, 2024. 4, 5, 6, 14

2024 arXiv
[41]

Vcr: Visual caption restoration,

T. Zhang, S. Wang, L. Li, G. Zhang, P . Taslakian, S. Rajeswar, J. Fu, B. Liu, and Y. Bengio, “Vcr: Visual caption restoration,” arXiv:2406.06462, 2024. 4, 5, 13, 14, 16

2024 arXiv
[42]

Chartqa: A benchmark for question answering about charts with visual and logical reasoning,

A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” arXiv:2203.10244, 2022. 4, 6, 8, 14

2022 arXiv
[43]

Docvqa: A dataset for vqa on document images,

M. Mathew, D. Karatzas, and C. Jawahar, “Docvqa: A dataset for vqa on document images,” in WACV, 2021. 4, 6, 8

2021
[44]

Infographicvqa,

M. Mathew, V . Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar, “Infographicvqa,” in WACV, 2022. 4, 5, 6, 8, 14

2022
[45]

Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models,

R. Xia, S. Mao, X. Yan, H. Zhou, B. Zhang, H. Peng, J. Pi, D. Fu, W. Wu, H. Ye et al., “Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models,” arXiv:2406.11633, 2024. 4, 6

2024
[46]

Mmlongbench-doc: Benchmark- ing long-context document understanding with visualizations,

Y. Ma, Y. Zang, L. Chen, M. Chen, Y. Jiao, X. Li, X. Lu, Z. Liu, Y. Ma, X. Dong et al. , “Mmlongbench-doc: Benchmark- ing long-context document understanding with visualizations,” arXiv:2407.01523, 2024. 4, 6

2024 arXiv
[47]

Charxiv: Charting gaps in realistic chart understanding in multimodal llms,

Z. Wang, M. Xia, L. He, H. Chen, Y. Liu, R. Zhu, K. Liang, X. Wu, H. Liu, S. Malladi et al., “Charxiv: Charting gaps in realistic chart understanding in multimodal llms,” arXiv:2406.18521, 2024. 4, 6

2024 arXiv
[49]

Tabpedia: Towards comprehensive visual table understanding with concept synergy,

W. Zhao, H. Feng, Q. Liu, J. Tang, S. Wei, B. Wu, L. Liao, Y. Ye, H. Liu, H. Li et al. , “Tabpedia: Towards comprehensive visual table understanding with concept synergy,” arXiv:2406.01326,

arXiv
[50]

Visualmrc: Machine reading comprehension on document images,

R. Tanaka, K. Nishida, and S. Yoshida, “Visualmrc: Machine reading comprehension on document images,” in AAAI, 2021. 4, 6, 8

2021
[51]

Leaf-qa: Locate, encode attend for figure question answering,

R. Chaudhry, S. Shekhar, U. Gupta, P . Maneriker, P . Bansal, and A. Joshi, “Leaf-qa: Locate, encode attend for figure question answering,” in WACV, 2020. 4, 6, 8

2020
[52]

Figureqa: An annotated figure dataset for visual reasoning,

S. E. Kahou, V . Michalski, A. Atkinson, Á. Kádár, A. Trischler, and Y. Bengio, “Figureqa: An annotated figure dataset for visual reasoning,” arXiv:1710.07300, 2017. 4, 6, 8

2017 arXiv
[53]

Mathvista: Evaluating math- ematical reasoning of foundation models in visual contexts,

P . Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.- W. Chang, M. Galley, and J. Gao, “Mathvista: Evaluating math- ematical reasoning of foundation models in visual contexts,” arXiv:2310.02255, 2023. 4, 6, 8, 13, 19

2023 arXiv
[54]

Measuring multimodal mathematical reasoning with math-vision dataset,

K. Wang, J. Pan, W. Shi, Z. Lu, M. Zhan, and H. Li, “Measuring multimodal mathematical reasoning with math-vision dataset,” arXiv:2402.14804, 2024. 4, 6, 19

2024 arXiv
[55]

Olympiadbench: A challenging bench- mark for promoting agi with olympiad-level bilingual multi- modal scientific problems,

C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhanget al., “Olympiadbench: A challenging bench- mark for promoting agi with olympiad-level bilingual multi- modal scientific problems,” arXiv:2402.14008, 2024. 4, 6

2024 arXiv
[56]

Mathverse: Does your multi- modal llm truly see the diagrams in visual math problems?

R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P . Qiu, A. Zhou, P . Lu, K.-W. Chang, P . Gaoet al., “Mathverse: Does your multi- modal llm truly see the diagrams in visual math problems?” arXiv:2403.14624, 2024. 4, 6, 19

2024 arXiv
[57]

We-math: Does your large mul- timodal model achieve human-like mathematical reasoning?

R. Qiao, Q. Tan, G. Dong, M. Wu, C. Sun, X. Song, Z. GongQue, S. Lei, Z. Wei, M. Zhang et al., “We-math: Does your large mul- timodal model achieve human-like mathematical reasoning?” arXiv:2407.01284, 2024. 4, 6

2024 arXiv
[58]

Learn to explain: Multimodal reasoning via thought chains for science question answering,

P . Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P . Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” in NeurIPS,
[59]

Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi,

X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun et al. , “Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi,” in CVPR, 2024. 4, 6, 7, 8, 14, 15, 19

2024
[60]

Cmmu: A benchmark for chinese multi-modal multi- type question understanding and reasoning,

Z. He, X. Wu, P . Zhou, R. Xuan, G. Liu, X. Yang, Q. Zhu, and H. Huang, “Cmmu: A benchmark for chinese multi-modal multi- type question understanding and reasoning,” arXiv:2401.14011,

arXiv
[61]

Cmmmu: A chinese mas- sive multi-discipline multimodal understanding benchmark,

G. Zhang, X. Du, B. Chen, Y. Liang, T. Luo, T. Zheng, K. Zhu, Y. Cheng, C. Xu, S. Guo et al. , “Cmmmu: A chinese mas- sive multi-discipline multimodal understanding benchmark,” arXiv:2401.11944, 2024. 4, 6, 19

2024 arXiv
[62]

Mmmu- pro: A more robust multi-discipline multimodal understanding benchmark,

X. Yue, T. Zheng, Y. Ni, Y. Wang, K. Zhang, S. Tong, Y. Sun, M. Yin, B. Yu, G. Zhang, H. Sun, Y. Su, W. Chen, and G. Neubig, “Mmmu- pro: A more robust multi-discipline multimodal understanding benchmark,” arXiv:2409.02813, 2024. 4

2024 arXiv
[63]

Alignmmbench: Evaluating chinese multimodal alignment in large vision-language models,

Y. Wu, W. Yu, Y. Cheng, Y. Wang, X. Zhang, J. Xu, M. Ding, and Y. Dong, “Alignmmbench: Evaluating chinese multimodal alignment in large vision-language models,” arXiv:2406.09295,

arXiv
[65]

M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models,

W. Zhang, M. Aljunied, C. Gao, Y. K. Chia, and L. Bing, “M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models,” in NeurIPS, 2023. 4, 6

2023
[66]

Dataset and benchmark for urdu natural scenes text detection, recognition and visual question answering,

H. Maryam, L. Fu, J. Song, T. A. Shafayet, Q. Luo, X. Bai, and Y. Liu, “Dataset and benchmark for urdu natural scenes text detection, recognition and visual question answering,” arXiv:2405.12533, 2024. 4, 6

2024 arXiv
[67]

The first swahili language scene text detection and recognition dataset,

F. W. Douamba, J. Song, L. Fu, Y. Liu, and X. Bai, “The first swahili language scene text detection and recognition dataset,” arXiv:2405.11437, 2024. 4, 6

2024 arXiv
[68]

Viocrvqa: Novel benchmark dataset and vision reader for visual ques- tion answering by understanding vietnamese text in images,

H. Q. Pham, T. K.-B. Nguyen, Q. Van Nguyen, D. Q. Tran, N. H. Nguyen, K. Van Nguyen, and N. L.-T. Nguyen, “Viocrvqa: Novel benchmark dataset and vision reader for visual ques- tion answering by understanding vietnamese text in images,” arXiv:2404.18397, 2024. 4, 6

2024 arXiv
[69]

Cvlue: A new benchmark dataset for chinese vision-language understanding evaluation,

Y. Wang, Y. Liu, F. Yu, C. Huang, K. Li, Z. Wan, and W. Che, “Cvlue: A new benchmark dataset for chinese vision-language understanding evaluation,” arXiv:2407.01081, 2024. 4, 6

2024 arXiv
[71]

Convbench: A multi-turn con- versation evaluation benchmark with hierarchical capability for large vision-language models,

S. Liu, K. Ying, H. Zhang, Y. Yang, Y. Lin, T. Zhang, C. Li, Y. Qiao, P . Luo, W. Shao et al. , “Convbench: A multi-turn con- versation evaluation benchmark with hierarchical capability for large vision-language models,” arXiv:2403.20194, 2024. 4, 7, 19

2024 arXiv
[72]

Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms,

Z. Liu, T. Chu, Y. Zang, X. Wei, X. Dong, P . Zhang, Z. Liang, Y. Xiong, Y. Qiao, D. Linet al., “Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms,” arXiv:2406.11833, 2024. 4, 7, 14, 19

2024 arXiv
[73]

A corpus for reasoning about natural language grounded in photographs,

A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi, “A corpus for reasoning about natural language grounded in photographs,” arXiv:1811.00491, 2018. 4, 7, 8

2018 arXiv
[74]

Sparkles: Unlocking chats across multiple images for multi- modal instruction-following models,

Y. Huang, Z. Meng, F. Liu, Y. Su, N. Collier, and Y. Lu, “Sparkles: Unlocking chats across multiple images for multi- modal instruction-following models,” arXiv:2308.16463, 2023. 4, 7

2023 arXiv
[75]

Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences,

X. Wang, Y. Zhou, X. Liu, H. Lu, Y. Xu, F. He, J. Yoon, T. Lu, G. Bertasius, M. Bansal et al. , “Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences,” arXiv:2401.10529, 2024. 4, 7

2024 arXiv
[76]

Benchmark- ing multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning,

B. Zhao, Y. Zong, L. Zhang, and T. Hospedales, “Benchmark- ing multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning,” arXiv:2406.12742, 2024. 4, 7

2024 arXiv
[77]

Remi: A dataset for reasoning with multiple images,

M. Kazemi, N. Dikkala, A. Anand, P . Devic, I. Dasgupta, F. Liu, B. Fatemi, P . Awasthi, D. Guo, S. Gollapudiet al., “Remi: A dataset for reasoning with multiple images,” arXiv:2406.09175, 2024. 4, 7

2024 arXiv
[78]

Muirbench: A comprehensive bench- mark for robust multi-image understanding,

F. Wang, X. Fu, J. Y. Huang, Z. Li, Q. Liu, X. Liu, M. D. Ma, N. Xu, W. Zhou, K. Zhang et al., “Muirbench: A comprehensive bench- mark for robust multi-image understanding,” arXiv:2406.09411,

arXiv
[79]

Vega: Learning interleaved image-text comprehension in vision-language large models,

C. Zhou, M. Zhang, P . Chen, C. Fu, Y. Shen, X. Zheng, X. Sun, and R. Ji, “Vega: Learning interleaved image-text comprehension in vision-language large models,” arXiv:2406.10228, 2024. 4, 7

2024 arXiv
[80]

V?: Guided visual search as a core mechanism in multimodal llms,

P . Wu and S. Xie, “V?: Guided visual search as a core mechanism in multimodal llms,” in CVPR, 2024. 4, 7

2024
[83]

Revisiting referring expression compre- hension evaluation in the era of large multimodal models,

J. Chen, F. Wei, J. Zhao, S. Song, B. Wu, Z. Peng, S.-H. G. Chan, and H. Zhang, “Revisiting referring expression compre- hension evaluation in the era of large multimodal models,” arXiv:2406.16866, 2024. 4, 8

2024 arXiv
[84]

African or european swallow? benchmarking large vision-language models for fine- grained object classification,

G. Geigle, R. Timofte, and G. Glavaš, “African or european swallow? benchmarking large vision-language models for fine- grained object classification,” arXiv:2406.14496, 2024. 4, 8

2024 arXiv
[85]

Eyes wide shut? exploring the visual shortcomings of multimodal llms,

S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie, “Eyes wide shut? exploring the visual shortcomings of multimodal llms,” in CVPR, 2024. 4, 8

2024
[86]

Q-bench: A benchmark for general-purpose foundation models on low-level vision,

H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, A. Wang, C. Li, W. Sun, Q. Yan, G. Zhai et al. , “Q-bench: A benchmark for general-purpose foundation models on low-level vision,” arXiv:2309.14181, 2023. 4, 8

2023 arXiv
[88]

Mvbench: A comprehensive multi-modal video understanding benchmark,

K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P . Luoet al., “Mvbench: A comprehensive multi-modal video understanding benchmark,” in CVPR, 2024. 4, 8, 9

2024
[89]

Mlvu: A comprehen- sive benchmark for multi-task long video understanding,

J. Zhou, Y. Shu, B. Zhao, B. Wu, S. Xiao, X. Yang, Y. Xiong, B. Zhang, T. Huang, and Z. Liu, “Mlvu: A comprehen- sive benchmark for multi-task long video understanding,” arXiv:2406.04264, 2024. 4, 9

2024 arXiv
[90]

Lvbench: An extreme long video understanding benchmark,

W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, S. Huang, B. Xu, Y. Dong, M. Ding et al., “Lvbench: An extreme long video understanding benchmark,” arXiv:2406.08035, 2024. 4, 9

2024 arXiv
[92]

Towards event-oriented long video understanding,

Y. Du, K. Zhou, Y. Huo, Y. Li, W. X. Zhao, H. Lu, Z. Zhao, B. Wang, W. Chen, and J.-R. Wen, “Towards event-oriented long video understanding,” arXiv:2406.14129, 2024. 4, 9

2024 arXiv
[93]

Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms,

Z. Zhao, H. Lu, Y. Huo, Y. Du, T. Yue, L. Guo, B. Wang, W. Chen, and J. Liu, “Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms,” arXiv:2406.09367,

arXiv
[94]

Egoschema: A diagnostic benchmark for very long-form video language under- standing,

K. Mangalam, R. Akshulakov, and J. Malik, “Egoschema: A diagnostic benchmark for very long-form video language under- standing,” in NeurIPS, 2024. 4, 8, 9

2024
[95]

Tempcompass: Do video llms really understand videos?

Y. Liu, S. Li, Y. Liu, Y. Wang, S. Ren, L. Li, S. Chen, X. Sun, and L. Hou, “Tempcompass: Do video llms really understand videos?” arXiv:2403.00476, 2024. 4, 9

2024 arXiv
[96]

Video question answering via gradually refined attention over appearance and motion,

D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang, “Video question answering via gradually refined attention over appearance and motion,” in ACM MM, 2017. 4, 8

2017
[97]

Tgif-qa: Toward spatio-temporal reasoning in visual question answering,

Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim, “Tgif-qa: Toward spatio-temporal reasoning in visual question answering,” in CVPR, 2017. 4, 8

2017
[98]

Activitynet-qa: A dataset for understanding complex web videos via question answering,

Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao, “Activitynet-qa: A dataset for understanding complex web videos via question answering,” in AAAI, 2019. 4, 8, 9

2019
[99]

Eval- uating object hallucination in large vision-language models,

Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Eval- uating object hallucination in large vision-language models,” in EMNLP, 2023. 4, 8, 9

2023
[100]

Mitigating hallucination in large multi-modal models via robust instruction tuning,

F. Liu, K. Lin, L. Li, J. Wang, Y. Yacoob, and L. Wang, “Mitigating hallucination in large multi-modal models via robust instruction tuning,” in ICLR, 2023. 4, 8, 9, 14

2023
[101]

Detecting and preventing halluci- nations in large vision language models,

A. Gunjal, J. Yin, and E. Bas, “Detecting and preventing halluci- nations in large vision language models,” in AAAI, 2024. 4, 8, 9, 14, 15

2024
[102]

Evaluation and analysis of hallucination in large vision-language models,

J. Wang, Y. Zhou, G. Xu, P . Shi, C. Zhao, H. Xu, Q. Ye, M. Yan, J. Zhang, J. Zhu et al., “Evaluation and analysis of hallucination in large vision-language models,” arXiv:2308.15126, 2023. 4

2023 arXiv
[103]

Aligning large multimodal models with factually augmented rlhf,

Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L.-Y. Gui, Y.-X. Wang, Y. Yang et al. , “Aligning large multimodal models with factually augmented rlhf,” arXiv:2309.14525, 2023. 4, 8

2023 arXiv
[105]

Phd: A prompted visual hallucination evaluation dataset,

J. Liu, Y. Fu, R. Xie, R. Xie, X. Sun, F. Lian, Z. Kang, and X. Li, “Phd: A prompted visual hallucination evaluation dataset,” arXiv:2403.11116, 2024. 4, 9

2024 arXiv
[106]

Hallusionbench: an advanced di- agnostic suite for entangled language hallucination and visual illusion in large vision-language models,

T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob et al. , “Hallusionbench: an advanced di- agnostic suite for entangled language hallucination and visual illusion in large vision-language models,” in CVPR, 2024. 4, 8, 9, 13, 14

2024
[107]

An llm-free multi-dimensional benchmark for mllms hallucination evaluation,

J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, M. Yan, J. Zhang, and J. Sang, “An llm-free multi-dimensional benchmark for mllms hallucination evaluation,” arXiv:2311.07397, 2023. 4, 9

2023 arXiv
[108]

Mocha: Multi-objective reinforcement mitigating caption hallucinations,

A. Ben-Kish, M. Yanuka, M. Alper, R. Giryes, and H. Averbuch- Elor, “Mocha: Multi-objective reinforcement mitigating caption hallucinations,” arXiv:2312.03631, 2023. 4, 9

2023 arXiv
[110]

Visual hallucina- tions of multi-modal large language models,

W. Huang, H. Liu, M. Guo, and N. Z. Gong, “Visual hallucina- tions of multi-modal large language models,” arXiv:2402.14683,

arXiv
[111]

Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models,

H. Qiu, W. Hu, Z.-Y. Dou, and N. Peng, “Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models,” arXiv:2404.13874, 2024. 4

2024 arXiv
[112]

Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video- language models,

Y. Wang, Y. Wang, D. Zhao, C. Xie, and Z. Zheng, “Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video- language models,” arXiv:2406.16338, 2024. 4, 9

2024 arXiv
[113]

Evaluating the quality of hallucination benchmarks for large vision-language models,

B. Yan, J. Zhang, Z. Yuan, S. Shan, and X. Chen, “Evaluating the quality of hallucination benchmarks for large vision-language models,” arXiv:2406.17115, 2024. 4

2024
[114]

Evaluating and analyzing relationship hallucinations in lvlms,

M. Wu, J. Ji, O. Huang, J. Li, Y. Wu, X. Sun, and R. Ji, “Evaluating and analyzing relationship hallucinations in lvlms,” arXiv:2406.16449, 2024. 4

2024 arXiv
[115]

Vlind-bench: Measuring language priors in large vision-language models,

K.-i. Lee, M. Kim, S. Yoon, M. Kim, D. Lee, H. Koh, and K. Jung, “Vlind-bench: Measuring language priors in large vision-language models,” arXiv:2406.08702, 2024. 4, 9

2024 arXiv
[116]

Vlbiasbench: A comprehensive benchmark for evaluating bias in large vision-language model,

J. Zhang, S. Wang, X. Cao, Z. Yuan, S. Shan, X. Chen, and W. Gao, “Vlbiasbench: A comprehensive benchmark for evaluating bias in large vision-language model,” arXiv:2406.14194, 2024. 4, 9

2024 arXiv
[117]

Mm-spubench: Towards better understanding of spurious bi- ases in multimodal llms,

W. Ye, G. Zheng, Y. Ma, X. Cao, B. Lai, J. M. Rehg, and A. Zhang, “Mm-spubench: Towards better understanding of spurious bi- ases in multimodal llms,” arXiv:2406.17126, 2024. 4, 10 JOURNAL OF LATEX CLASS FILES, NOVEMBER 2024 23

2024
[119]

Benchmarking trustworthiness of multimodal large language models: A comprehensive study,

Y. Zhang, Y. Huang, Y. Sun, C. Liu, Z. Zhao, Z. Fang, Y. Wang, H. Chen, X. Yang, X. Wei et al., “Benchmarking trustworthiness of multimodal large language models: A comprehensive study,” arXiv:2406.07057, 2024. 4, 10

2024 arXiv
[120]

On evaluating adversarial robustness of large vision- language models,

Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, “On evaluating adversarial robustness of large vision- language models,” in NeurIPS, 2024. 4, 8, 10

2024
[121]

Efficiently adversarial exam- ples generation for visual-language models under targeted trans- fer scenarios using diffusion models,

Q. Guo, S. Pang, X. Jia, and Q. Guo, “Efficiently adversarial exam- ples generation for visual-language models under targeted trans- fer scenarios using diffusion models,” arXiv:2404.10335, 2024. 4, 10

2024 arXiv
[122]

Mossbench: Is your multimodal language model oversensitive to safe queries?

X. Li, H. Zhou, R. Wang, T. Zhou, M. Cheng, and C.-J. Hsieh, “Mossbench: Is your multimodal language model oversensitive to safe queries?” arXiv:2406.17806, 2024. 4, 10

2024 arXiv
[123]

Cello: Causal evaluation of large vision-language models,

M. Chen, B. Peng, Y. Zhang, and C. Lu, “Cello: Causal evaluation of large vision-language models,” arXiv:2406.19131, 2024. 4, 10

2024 arXiv
[124]

A dataset of clinically generated visual questions and answers about radiology images,

J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman, “A dataset of clinically generated visual questions and answers about radiology images,” Sci. Data, 2018. 4, 8, 10

2018
[125]

Pathvqa: 30000+ questions for medical visual question answering,

X. He, Y. Zhang, L. Mou, E. Xing, and P . Xie, “Pathvqa: 30000+ questions for medical visual question answering,” arXiv:2003.10286, 2020. 4, 8, 10

2003 arXiv
[126]

Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering,

B. Liu, L.-M. Zhan, L. Xu, L. Ma, Y. Yang, and X.-M. Wu, “Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering,” in ISBI, 2021. 4, 8, 10

2021
[127]

Pmc-vqa: Visual instruction tuning for medical visual question answering,

X. Zhang, C. Wu, Z. Zhao, W. Lin, Y. Zhang, Y. Wang, and W. Xie, “Pmc-vqa: Visual instruction tuning for medical visual question answering,” arXiv:2305.10415, 2023. 4, 8, 10, 11

2023 arXiv
[128]

Om- nimedvqa: A new large-scale comprehensive evaluation bench- mark for medical lvlm,

Y. Hu, T. Li, Q. Lu, W. Shao, J. He, Y. Qiao, and P . Luo, “Om- nimedvqa: A new large-scale comprehensive evaluation bench- mark for medical lvlm,” in CVPR, 2024. 4, 11

2024
[129]

Towards generalist foundation model for radiology,

C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “Towards generalist foundation model for radiology,” arXiv:2308.02463,

arXiv
[130]

Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai,

P . Chen, J. Ye, G. Wang, Y. Li, Z. Deng, W. Li, T. Li, H. Duan, Z. Huang, Y. Su et al. , “Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai,” arXiv:2408.03361, 2024. 4, 10

2024 arXiv
[131]

Emollm: Multimodal emotional understanding meets large language models,

Q. Yang, M. Ye, and B. Du, “Emollm: Multimodal emotional understanding meets large language models,” arXiv:2406.16442,

arXiv
[132]

Facial affective behavior analysis with instruction tuning,

Y. Li, A. Dao, W. Bao, Z. Tan, T. Chen, H. Liu, and Y. Kong, “Facial affective behavior analysis with instruction tuning,” arXiv:2404.05052, 2024. 4, 11

2024 arXiv
[133]

Rsvqa: Visual question answering for remote sensing data,

S. Lobry, D. Marcos, J. Murray, and D. Tuia, “Rsvqa: Visual question answering for remote sensing data,” IEEE Trans. Geosci. Remote Sens., 2020. 4, 8, 11

2020
[134]

Mutual attention inception network for remote sensing visual question answering,

X. Zheng, B. Wang, X. Du, and X. Lu, “Mutual attention inception network for remote sensing visual question answering,” IEEE Trans. Geosci. Remote Sens., 2021. 4, 8

2021
[135]

Open-ended remote sensing visual question answering with transformers,

M. M. Al Rahhal, Y. Bazi, S. O. Alsaleh, M. Al-Razgan, M. L. Mekhalfi, M. Al Zuair, and N. Alajlan, “Open-ended remote sensing visual question answering with transformers,” Int. J. Remote Sens., 2022. 4

2022
[136]

Visual grounding in remote sensing images,

Y. Sun, S. Feng, X. Li, Y. Ye, J. Kang, and X. Huang, “Visual grounding in remote sensing images,” in ACM MM , 2022. 4, 11

2022
[137]

Rsvg: Exploring data and models for visual grounding on remote sensing data,

Y. Zhan, Z. Xiong, and Y. Yuan, “Rsvg: Exploring data and models for visual grounding on remote sensing data,” IEEE Trans. Geosci. Remote Sens., 2023. 4, 11

2023
[138]

Rsgpt: A remote sensing vision language model and benchmark,

Y. Hu, J. Yuan, C. Wen, X. Lu, and X. Li, “Rsgpt: A remote sensing vision language model and benchmark,” arXiv:2307.15266, 2023. 4, 11

2023 arXiv
[139]

Rotated multi-scale interaction network for referring remote sensing image segmentation,

S. Liu, Y. Ma, X. Zhang, H. Wang, J. Ji, X. Sun, and R. Ji, “Rotated multi-scale interaction network for referring remote sensing image segmentation,” in CVPR, 2024. 4, 11

2024
[140]

Vrsbench: A versatile vision- language benchmark dataset for remote sensing image under- standing,

X. Li, J. Ding, and M. Elhoseiny, “Vrsbench: A versatile vision- language benchmark dataset for remote sensing image under- standing,” arXiv:2406.12384, 2024. 4, 11

2024 arXiv
[143]

Gpt4tools: Teaching large language model to use tools via self- instruction,

R. Yang, L. Song, Y. Li, S. Zhao, Y. Ge, X. Li, and Y. Shan, “Gpt4tools: Teaching large language model to use tools via self- instruction,” in NeurIPS, 2024. 4, 8, 11

2024
[144]

Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation,

C. Shi, C. Yang, Y. Liu, B. Shui, J. Wang, M. Jing, L. Xu, X. Zhu, S. Li, Y. Zhang et al. , “Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation,” arXiv:2406.09961, 2024. 4, 11, 12

2024 arXiv
[145]

Web2code: A large- scale webpage-to-code dataset and evaluation framework for multimodal llms,

S. Yun, H. Lin, R. Thushara, M. Q. Bhat, Y. Wang, Z. Jiang, M. Deng, J. Wang, T. Tao, J. Li et al. , “Web2code: A large- scale webpage-to-code dataset and evaluation framework for multimodal llms,” arXiv:2406.20098, 2024. 4, 11

2024 arXiv
[146]

Resolving referring expressions in images with labeled elements,

N. Wichers, D. Hakkani-Tür, and J. Chen, “Resolving referring expressions in images with labeled elements,” in SLT, 2018. 4, 12, 16

2018
[147]

Screen2words: Automatic mobile ui summarization with mul- timodal learning,

B. Wang, G. Li, X. Zhou, Z. Chen, T. Grossman, and Y. Li, “Screen2words: Automatic mobile ui summarization with mul- timodal learning,” in UIST, 2021. 4, 8, 12, 14, 15, 16

2021
[148]

Screenqa: Large-scale question- answer pairs over mobile app screenshots,

Y.-C. Hsiao, F. Zubach, G. Baechler, V . Carbune, J. Lin, M. Wang, S. Sunkara, Y. Zhu, and J. Chen, “Screenqa: Large-scale question- answer pairs over mobile app screenshots,” arXiv:2209.08199,

arXiv
[149]

Towards better semantic understanding of mobile interfaces,

S. Sunkara, M. Wang, L. Liu, G. Baechler, Y.-C. Hsiao, A. Sharma, J. Stout et al., “Towards better semantic understanding of mobile interfaces,” arXiv:2210.02663, 2022. 4, 12, 14, 16

2022 arXiv
[150]

Screenai: A vision-language model for ui and infographics understanding,

G. Baechler, S. Sunkara, M. Wang, F. Zubach, H. Mansoor, V . Et- ter, V . C˘ arbune, J. Lin, J. Chen, and A. Sharma, “Screenai: A vision-language model for ui and infographics understanding,” arXiv:2402.04615, 2024. 4, 16

2024 arXiv
[151]

Widget captioning: Generating natural language description for mobile user interface elements,

Y. Li, G. Li, L. He, J. Zheng, H. Li, and Z. Guan, “Widget captioning: Generating natural language description for mobile user interface elements,” arXiv:2010.04295, 2020. 4, 8, 12, 14, 16

2010 arXiv
[152]

Benchlmm: Benchmarking cross-style visual capability of large multimodal models,

R. Cai, Z. Song, D. Guan, Z. Chen, X. Luo, C. Yi, and A. Kot, “Benchlmm: Benchmarking cross-style visual capability of large multimodal models,” arXiv:2312.02896, 2023. 4, 12, 14

2023 arXiv
[153]

Bench- marking large multimodal models against common corruptions,

J. Zhang, T. Pang, C. Du, Y. Ren, B. Li, and M. Lin, “Bench- marking large multimodal models against common corruptions,” arXiv:2401.11943, 2024. 4, 12, 13, 14

2024 arXiv
[154]

Can we edit multimodal large language models?

S. Cheng, B. Tian, Q. Liu, X. Chen, Y. Wang, H. Chen, and N. Zhang, “Can we edit multimodal large language models?” in EMNLP, 2023. 4, 12, 14

2023
[155]

Vlkeb: A large vision-language model knowledge editing benchmark,

H. Huang, H. Zhong, T. Yu, Q. Liu, S. Wu, L. Wang, and T. Tan, “Vlkeb: A large vision-language model knowledge editing benchmark,” arXiv:2403.07350, 2024. 4, 12, 14

2024 arXiv
[156]

Embodied question answering,

A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra, “Embodied question answering,” in CVPR, 2018. 4, 8, 12, 13, 14

2018
[157]

The epic-kitchens dataset: Collection, challenges and baselines,

D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “The epic-kitchens dataset: Collection, challenges and baselines,” IEEE Trans. Pattern Anal. Mach. Intell. , 2021. 4, 8, 12, 14

2021
[158]

Ego4d: Around the world in 3,000 hours of egocentric video,

K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al., “Ego4d: Around the world in 3,000 hours of egocentric video,” in CVPR,
[159]

Episodic memory question answering,

S. Datta, S. Dharur, V . Cartillier, R. Desai, M. Khanna, D. Batra, and D. Parikh, “Episodic memory question answering,” inCVPR,
[160]

Sqa3d: Situated question answering in 3d scenes,

X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang, “Sqa3d: Situated question answering in 3d scenes,” inICLR, 2023. 4, 12, 14, 16

2023
[161]

A dataset for interactive vision-language navigation with unknown command feasibility,

A. Burns, D. Arsan, S. Agrawal, R. Kumar, K. Saenko, and B. A. Plummer, “A dataset for interactive vision-language navigation with unknown command feasibility,” in ECCV, 2022. 4, 12, 14, 16

2022
[162]

Rh20t-p: A primitive- level robotic dataset towards composable generalization agents,

Z. Chen, Z. Shi, X. Lu, L. He, S. Qian, H. S. Fang, Z. Yin, W. Ouyang, J. Shao, Y. Qiao et al. , “Rh20t-p: A primitive- level robotic dataset towards composable generalization agents,” arXiv:2403.19622, 2024. 4, 12, 13

2024 arXiv
[163]

Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai,

T. Wang, X. Mao, C. Zhu, R. Xu, R. Lyu, P . Li, X. Chen, W. Zhang, K. Chen, T. Xue et al., “Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai,” in CVPR, 2024. 4, 12, 14

2024
[164]

Textual explanations for self-driving vehicles,

J. Kim, A. Rohrbach, T. Darrell, J. Canny, and Z. Akata, “Textual explanations for self-driving vehicles,” ECCV, 2018. 4, 8, 13, 16 JOURNAL OF LATEX CLASS FILES, NOVEMBER 2024 24

2018
[165]

Grounding human-to-vehicle advice for self-driving vehicles,

J. Kim, T. Misu, Y.-T. Chen, A. Tawari, and J. Canny, “Grounding human-to-vehicle advice for self-driving vehicles,” in CVPR,
[166]

Talk2car: Taking control of your self-driving car,

T. Deruyttere, S. Vandenhende, D. Grujicic, L. Van Gool, and M.-F. Moens, “Talk2car: Taking control of your self-driving car,” arXiv:1909.10838, 2019. 4, 8, 13

1909 arXiv
[167]

Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning,

E. Sachdeva, N. Agarwal, S. Chundi, S. Roelofs, J. Li, M. Kochen- derfer, C. Choi, and B. Dariush, “Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning,” in WACV,
[168]

Drama: Joint risk localization and captioning in driving,

S. Malla, C. Choi, I. Dwivedi, J. H. Choi, and J. Li, “Drama: Joint risk localization and captioning in driving,” in WACV, 2023. 4, 13

2023
[169]

Nuscenes- qa: A multi-modal visual question answering benchmark for autonomous driving scenario,

T. Qian, J. Chen, L. Zhuo, Y. Jiao, and Y.-G. Jiang, “Nuscenes- qa: A multi-modal visual question answering benchmark for autonomous driving scenario,” in AAAI, 2024. 4, 13, 14

2024
[170]

Drivelm: Driving with graph visual ques- tion answering,

C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P . Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual ques- tion answering,” arXiv:2312.14150, 2023. 4, 13, 14

2023 arXiv
[171]

Lingoqa: Video question answering for autonomous driving,

A.-M. Marcu, L. Chen, J. Hünermann, A. Karnsund, B. Hanotte, P . Chidananda, S. Nair, V . Badrinarayanan, A. Kendall, J. Shot- ton, and O. Sinavski, “Lingoqa: Video question answering for autonomous driving,” arXiv:2312.14115, 2023. 4, 13, 14, 16

2023 arXiv
[172]

Language prompt for autonomous driving,

D. Wu, W. Han, T. Wang, Y. Liu, X. Zhang, and J. Shen, “Language prompt for autonomous driving,” arXiv:2309.04379, 2023. 4, 13, 16

2023 arXiv
[173]

Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving,

M. Nie, R. Peng, C. Wang, X. Cai, J. Han, H. Xu, and L. Zhang, “Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving,” arXiv:2312.03661, 2023. 4, 13, 16

2023 arXiv
[174]

Can lvlms obtain a driver’s license? a benchmark towards reliable agi for autonomous driving,

Y. Lu, Y. Yao, J. Tu, J. Shao, Y. Ma, and X. Zhu, “Can lvlms obtain a driver’s license? a benchmark towards reliable agi for autonomous driving,” arXiv:2409.02914, 2024. 4, 13

2024 arXiv
[175]

Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation,

Y. Shi, D. Peng, W. Liao, Z. Lin, X. Chen, C. Liu, Y. Zhang, and L. Jin, “Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation,” arXiv:2310.16809, 2023. 5

2023 arXiv
[176]

Mtvqa: Benchmarking multi- lingual text-centric visual question answering,

J. Tang, Q. Liu, Y. Ye, J. Lu, S. Wei, C. Lin, W. Li, M. F. F. B. Mahmood, H. Feng, Z. Zhao et al., “Mtvqa: Benchmarking multi- lingual text-centric visual question answering,” arXiv:2405.11985,

arXiv
[177]

Trins: Towards multimodal language models that can read,

R. Zhang, Y. Zhang, J. Chen, Y. Zhou, J. Gu, C. Chen, and T. Sun, “Trins: Towards multimodal language models that can read,” in CVPR, 2024. 6

2024
[179]

Llava- next: Improved reasoning, ocr, and world knowledge,

H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee, “Llava- next: Improved reasoning, ocr, and world knowledge,” https:// llava-vl.github.io/blog/2024-01-30-llava-next/, 2024. 7, 17

2024
[181]

Learning to compose and reason with language tree structures for visual grounding,

R. Hong, D. Liu, X. Mo, X. He, and H. Zhang, “Learning to compose and reason with language tree structures for visual grounding,” IEEE Trans. Pattern Anal. Mach. Intell., 2019. 7

2019
[182]

Context disen- tangling and prototype inheriting for robust visual grounding,

W. Tang, L. Li, X. Liu, L. Jin, J. Tang, and Z. Li, “Context disen- tangling and prototype inheriting for robust visual grounding,” IEEE Trans. Pattern Anal. Mach. Intell., 2023. 7

2023
[183]

Qwen-vl: A frontier large vision-language model with versatile abilities,

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P . Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv:2308.12966, 2023. 7

2023 arXiv
[184]

Cogvlm: Visual expert for pretrained language models,

W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song et al. , “Cogvlm: Visual expert for pretrained language models,” arXiv:2311.03079, 2023. 7

2023 arXiv
[185]

Woodpecker: Hallucination correction for multimodal large language models,

S. Yin, C. Fu, S. Zhao, T. Xu, H. Wang, D. Sui, Y. Shen, K. Li, X. Sun, and E. Chen, “Woodpecker: Hallucination correction for multimodal large language models,” arXiv:2310.16045, 2023. 9

2023 arXiv
[187]

Improved baselines with visual instruction tuning,

H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” in CVPR, 2024. 9

2024
[188]

Shikra: Unleashing multimodal llm’s referential dialogue magic,

K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, “Shikra: Unleashing multimodal llm’s referential dialogue magic,” arXiv:2306.15195, 2023. 9

2023 arXiv
[189]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., “Gemini: a family of highly capable multimodal models,” arXiv:2312.11805,

arXiv
[190]

Minigpt-4: Enhancing vision-language understanding with advanced large language models,

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv:2304.10592, 2023. 10

2023 arXiv
[191]

Jailbroken: How does llm safety training fail?

A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” in NeurIPS, 2024. 10

2024
[192]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in ICML, 2023. 10, 11

2023
[193]

Introducing the next generation of claude,

Anthropic, “Introducing the next generation of claude,” https: //www.anthropic.com/news/claude-3-family, 2024. 10

2024
[194]

Llava-plus: Learning to use tools for creating multimodal agents,

S. Liu, H. Cheng, H. Liu, H. Zhang, F. Li, T. Ren, X. Zou, J. Yang, H. Su, J. Zhu et al., “Llava-plus: Learning to use tools for creating multimodal agents,” arXiv:2311.05437, 2023. 11

2023 arXiv
[195]

Phi-3 technical report: A highly capable language model locally on your phone,

M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl et al. , “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv:2404.14219, 2024. 11

2024 arXiv
[196]

Ferret-ui: Grounded mobile ui understand- ing with multimodal llms,

K. You, H. Zhang, E. Schoop, F. Weers, A. Swearngin, J. Nichols, Y. Yang, and Z. Gan, “Ferret-ui: Grounded mobile ui understand- ing with multimodal llms,” arXiv:2404.05719, 2024. 12, 16

2024 arXiv
[197]

Egotaskqa: Understand- ing human tasks in egocentric videos,

B. Jia, T. Lei, S.-C. Zhu, and S. Huang, “Egotaskqa: Understand- ing human tasks in egocentric videos,” in NeurIPS, 2022. 12

2022
[198]

Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration,

N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, “Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration,” arXiv:2311.12015, 2023. 12, 13

2023 arXiv
[199]

Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning,

Y. Hu, F. Lin, T. Zhang, L. Yi, and Y. Gao, “Look before you leap: Unveiling the power of gpt-4v in robotic vision-language planning,” arXiv:2311.17842, 2023. 13

2023 arXiv
[200]

Mp5: A multi-modal open-ended embodied system in minecraft via active perception,

Y. Qin, E. Zhou, Q. Liu, Z. Yin, L. Sheng, R. Zhang, Y. Qiao, and J. Shao, “Mp5: A multi-modal open-ended embodied system in minecraft via active perception,” arXiv:2312.07472, 2023. 13

2023 arXiv
[201]

Drivevlm: The convergence of autonomous driving and large vision-language models,

X. Tian, J. Gu, B. Li, Y. Liu, C. Hu, Y. Wang, K. Zhan, P . Jia, X. Lang, and H. Zhao, “Drivevlm: The convergence of autonomous driving and large vision-language models,” arXiv:2402.12289, 2024. 13

2024 arXiv
[202]

A survey on multimodal large language models for autonomous driving,

C. Cui, Y. Ma, X. Cao, W. Ye, Y. Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liao et al., “A survey on multimodal large language models for autonomous driving,” in WACV, 2024. 13

2024
[203]

On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving,

L. Wen, X. Yang, D. Fu, X. Wang, P . Cai, X. Li, T. Ma, Y. Li, L. Xu, D. Shang et al. , “On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving,” arXiv:2311.05332, 2023. 13

2023 arXiv
[205]

Ferret: Refer and ground anything anywhere at any granularity,

H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y. Yang, “Ferret: Refer and ground anything anywhere at any granularity,” arXiv:2310.07704, 2023. 14

2023 arXiv
[206]

Mmevalpro: Calibrating mul- timodal benchmarks towards trustworthy and efficient evalua- tion,

J. Huang, L. Chen, T. Guo, F. Zeng, Y. Zhao, B. Wu, Y. Yuan, H. Zhao, Z. Guo, Y. Zhang et al., “Mmevalpro: Calibrating mul- timodal benchmarks towards trustworthy and efficient evalua- tion,” arXiv:2407.00468, 2024. 14, 15

2024 arXiv
[207]

Fakebench: Uncover the achilles’ heels of fake images with large multimodal models,

Y. Li, X. Liu, X. Wang, S. Wang, and W. Lin, “Fakebench: Uncover the achilles’ heels of fake images with large multimodal models,” arXiv:2404.13306, 2024. 14, 16

2024 arXiv
[208]

Multi-modal arena,

S. A. Lab, “Multi-modal arena,” https://opencompass.org.cn/ arena?type=multimodal, 2023. 14

2023
[209]

Large language models are not fair evalua- tors,

P . Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui, “Large language models are not fair evalua- tors,” arXiv:2305.17926, 2023. 15

2023 arXiv
[211]

Opencompass: A universal evaluation platform for foundation models,

O. Contributors, “Opencompass: A universal evaluation platform for foundation models,” https://github.com/open-compass/ opencompass, 2023. 17

2023
[212]

Vlmevalkit: An open- source toolkit for evaluating large multi-modality models,

H. Duan, J. Yang, Y. Qiao, X. Fang, L. Chen, Y. Liu, X. Dong, Y. Zang, P . Zhang, J. Wang et al. , “Vlmevalkit: An open- source toolkit for evaluating large multi-modality models,” arXiv:2407.11691, 2024. 17

2024 arXiv
[213]

Lmms-eval: Reality check on the evaluation of large multimodal models,

K. Zhang, B. Li, P . Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y. Zhang, J. Yang, C. Li et al., “Lmms-eval: Reality check on the evaluation of large multimodal models,” arXiv:2407.12772, 2024. 17 JOURNAL OF LATEX CLASS FILES, NOVEMBER 2024 25

2024 arXiv
[214]

Agentstudio: A toolkit for building general virtual agents,

L. Zheng, Z. Huang, Z. Xue, X. Wang, B. An, and S. Yan, “Agentstudio: A toolkit for building general virtual agents,” arXiv:2403.17918, 2024. 17, 18

2024 arXiv
[215]

Multimedeval: A benchmark and a toolkit for evaluating medical vision-language models,

C. Royer, B. Menze, and A. Sekuboyina, “Multimedeval: A benchmark and a toolkit for evaluating medical vision-language models,” arXiv:2402.09262, 2024. 17, 18

2024
[216]

Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models,

P . Padlewski, M. Bain, M. Henderson, Z. Zhu, N. Relan, H. Pham, D. Ong, K. Aleksiev, A. Ormazabal, S. Phua et al. , “Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models,” arXiv:2405.02287, 2024. 17

2024 arXiv
[217]

Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,” in NeurIPS, 2024. 18

2024
[218]

Biomedgpt: a unified and gen- eralist biomedical generative pre-trained transformer for vision, language, and multimodal tasks,

K. Zhang, J. Yu, Z. Yan, Y. Liu, E. Adhikarla, S. Fu, X. Chen, C. Chen, Y. Zhou, X. Li et al. , “Biomedgpt: a unified and gen- eralist biomedical generative pre-trained transformer for vision, language, and multimodal tasks,” arXiv:2305.17100, 2023. 18

2023 arXiv
[219]

Prism: A framework for decoupling and assessing the capabilities of vlms,

Y. Qiao, H. Duan, X. Fang, J. Yang, L. Chen, S. Zhang, J. Wang, D. Lin, and K. Chen, “Prism: A framework for decoupling and assessing the capabilities of vlms,” arXiv:2406.14544, 2024. 19

2024 arXiv
[220]

Pp-ocr: A practical ultra lightweight ocr system,

Y. Du, C. Li, R. Guo, X. Yin, W. Liu, J. Zhou, Y. Bai, Z. Yu, Y. Yang, Q. Dang et al., “Pp-ocr: A practical ultra lightweight ocr system,” arXiv:2009.09941, 2020. 20

2009 arXiv
[221]

Mmocr: a comprehensive toolbox for text detection, recognition and understanding,

Z. Kuang, H. Sun, Z. Li, X. Yue, T. H. Lin, J. Chen, H. Wei, Y. Zhu, T. Gao, W. Zhanget al., “Mmocr: a comprehensive toolbox for text detection, recognition and understanding,” inACM MM, 2021. 20

2021
[222]

You only look once: Unified, real-time object detection,

J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in CVPR,
[223]

Mask r-cnn,

K. He, G. Gkioxari, P . Dollár, and R. Girshick, “Mask r-cnn,” in ICCV, 2017. 20

2017
[224]

Faster r-cnn: Towards real-time object detection with region proposal networks,

S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” IEEE Trans. Pattern Anal. Mach. Intell., 2016. 20

2016
[225]

End-to-end object detection with transformers,

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in ECCV, 2020. 20

2020
[226]

Temporal segment networks: Towards good prac- tices for deep action recognition,

L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good prac- tices for deep action recognition,” in ECCV, 2016. 20

2016
[227]

Slowfast networks for video recognition,

C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in ICCV, 2019. 20

2019
[228]

Revisiting skeleton-based action recognition,

H. Duan, Y. Zhao, K. Chen, D. Lin, and B. Dai, “Revisiting skeleton-based action recognition,” in CVPR, 2022. 20

2022
[229]

Appagent: Multimodal agents as smartphone users,

Z. Yang, J. Liu, Y. Han, X. Chen, Z. Huang, B. Fu, and G. Yu, “Appagent: Multimodal agents as smartphone users,” arXiv:2312.13771, 2023. 20

2023 arXiv
[230]

Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,

J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,” arXiv:2401.16158, 2024. 20

2024 arXiv
[231]

Osworld: Benchmarking mul- timodal agents for open-ended tasks in real computer environ- ments,

T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei et al. , “Osworld: Benchmarking mul- timodal agents for open-ended tasks in real computer environ- ments,” arXiv:2404.07972, 2024. 20

2024 arXiv
[232]

Androidworld: A dynamic benchmarking environment for autonomous agents,

C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala et al., “Androidworld: A dynamic benchmarking environment for autonomous agents,” arXiv:2405.14573, 2024. 20

2024 arXiv
[233]

Visual- webarena: Evaluating multimodal agents on realistic visual web tasks,

J. Y. Koh, R. Lo, L. Jang, V . Duvvur, M. C. Lim, P .-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “Visual- webarena: Evaluating multimodal agents on realistic visual web tasks,” arXiv:2401.13649, 2024. 20

2024 arXiv
[234]

Qwen-audio: Advancing universal audio un- derstanding via unified large-scale audio-language models,

Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio un- derstanding via unified large-scale audio-language models,” arXiv:2311.07919, 2023. 20

2023 arXiv
[235]

Lib- rispeech: an asr corpus based on public domain audio books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015. 20

2015
[236]

Covost 2 and massively multilingual speech translation

C. Wang, A. Wu, J. Gu, and J. Pino, “Covost 2 and massively multilingual speech translation.” in Interspeech, 2021. 20

2021
[237]

Clotho: An audio captioning dataset,

K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio captioning dataset,” in ICASSP, 2020. 20

2020
[238]

Vocalsound: A dataset for improving human vocal sounds recognition,

Y. Gong, J. Yu, and J. Glass, “Vocalsound: A dataset for improving human vocal sounds recognition,” in ICASSP, 2022. 20

2022
[239]

Scanrefer: 3d object localization in rgb-d scans using natural language,

D. Z. Chen, A. X. Chang, and M. Nießner, “Scanrefer: 3d object localization in rgb-d scans using natural language,” in ECCV,
[240]

Em- powering 3d visual grounding with reasoning capabilities,

C. Zhu, T. Wang, W. Zhang, K. Chen, and X. Liu, “Em- powering 3d visual grounding with reasoning capabilities,” arXiv:2407.01525, 2024. 20

2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [12]

Chameleon: Plug-and-play compositional reasoning with large language models,

P . Lu, B. Peng, H. Cheng, M. Galley, K.-W. Chang, Y. N. Wu, S.- C. Zhu, and J. Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,” in NeurIPS, 2024. 2

2024

[2] [81]

Refer- itgame: Referring to objects in photographs of natural scenes,

S. Kazemzadeh, V . Ordonez, M. Matten, and T. Berg, “Refer- itgame: Referring to objects in photographs of natural scenes,” in EMNLP, 2014. 4, 7, 8

2014

[3] [82]

Generation and comprehension of unambiguous object descriptions,

J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy, “Generation and comprehension of unambiguous object descriptions,” in CVPR, 2016. 4, 7

2016

[4] [24]

Mme: A comprehensive eval- uation benchmark for multimodal large language models,

C. Fu, P . Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun et al. , “Mme: A comprehensive eval- uation benchmark for multimodal large language models,” arXiv:2306.13394, 2023. 3, 4, 8, 14, 15, 16, 19

arXiv 2023

[5] [22]

Mmbench: Is your multi-modal model an all-around player?

Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu et al. , “Mmbench: Is your multi-modal model an all-around player?” arXiv:2307.06281, 2023. 3, 4, 8, 13, 14, 15, 16, 19

arXiv 2023

[6] [35]

Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?

Y.-F. Zhang, H. Zhang, H. Tian, C. Fu, S. Zhang, J. Wu, F. Li, K. Wang, Q. Wen, Z. Zhang et al., “Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?” arXiv:2408.13257, 2024. 3, 4, 7, 8, 13, 14, 15, 16

arXiv 2024

[7] [91]

Mmbench-video: A long-form multi-shot benchmark for holistic video understanding,

X. Fang, K. Mao, H. Duan, X. Zhao, Y. Li, D. Lin, and K. Chen, “Mmbench-video: A long-form multi-shot benchmark for holistic video understanding,” arXiv:2406.14515, 2024. 4, 9

arXiv 2024

[8] [48]

A diagram is worth a dozen images,

A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “A diagram is worth a dozen images,” inECCV, 2016. 4, 6

2016

[9] [180]

A diagram is worth a dozen images,

A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “A diagram is worth a dozen images,” inECCV, 2016. 8

2016

[10] [210]

Mia-bench: Towards better instruction following evaluation of multimodal llms,

Y. Qian, H. Ye, J.-P . Fauconnier, P . Grasch, Y. Yang, and Z. Gan, “Mia-bench: Towards better instruction following evaluation of multimodal llms,” arXiv:2407.01509, 2024. 16

arXiv 2024

[11] [178]

Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis,

C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang et al. , “Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal llms in video analysis,” arXiv:2405.21075, 2024. 6, 14, 15, 16

arXiv 2024

[12] [204]

Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges,

C. Cui, Y. Zhou, X. Yang, S. Wu, L. Zhang, J. Zou, and H. Yao, “Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges,” arXiv:2311.03287, 2023. 14, 16

arXiv 2023

[13] [186]

Unified hallucination detection for multimodal large language models,

X. Chen, C. Wang, Y. Xue, N. Zhang, X. Yang, Q. Li, Y. Shen, J. Gu, and H. Chen, “Unified hallucination detection for multimodal large language models,” arXiv:2402.03190, 2024. 9

arXiv 2024

[14] [118]

How many unicorns are in this image? a safety evaluation benchmark for vision llms,

H. Tu, C. Cui, Z. Wang, Y. Zhou, B. Zhao, J. Han, W. Zhou, H. Yao, and C. Xie, “How many unicorns are in this image? a safety evaluation benchmark for vision llms,” arXiv:2311.16101, 2023. 4, 10, 12, 16

arXiv 2023