Pith. sign in

REVIEW 4 major objections 6 minor 82 references

Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that multi-image hallucination in multimodal LLMs can be reduced by two-level preference training that first aligns whole-sequence captions and then aligns region-level details, improving seven multi-image benchmarks…

desk verdict Genuine two-level DPO method with credible multi-image gains, but the headline hallucination reductions are confounded by format matching and need a controlled re-evaluation. read the letter →

arxiv 2505.22396 v1 pith:GUMQBZFM submitted 2025-05-28 cs.CV

Xudong Li , Mengdan Zhang , Peixian Chen , Xiawu Zheng , Yan Zhang , Jingyuan Zheng , Yunhang Shen , Ke Li
show 3 more authors
Chaoyou Fu Xing Sun Rongrong Ji
This is my paper
classification cs.CV
keywords multi-imageunderstandingdirectpreferenceoptimizationhallucinationreductionmultimodallargelanguagemodelscaptiondatavisualpromptingcontext-levelneedle-level
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern multimodal LLMs can chat about a single photo but often fail when several images are presented together: they drop some images from their answer, mix up content between images, or miss small details. This paper claims that the reason is weak per-image perception, and that the missing skill can be trained cheaply with preference optimization. It proposes CcDPO, which first teaches the model to prefer complete, correctly attributed per-image captions of an entire sequence, then to prefer descriptions that match a highlighted local region. Training on the automatically constructed MultiScope-42k dataset, the method lifts average multi-image benchmark accuracy by 4.6 points on LLaVA-OV-7B and 2.7 points on Qwen2-VL-7B over the previous MIA-DPO baseline, and sharply lowers measured hallucination rates on Context-AMBER-1K. If the transfer from captioning to reasoning holds, this offers a scalable recipe for hardening multi-image understanding without dense manual annotation.

What carries the argument

The engine is a two-level DPO loss applied to caption-style preferences. Context-level optimization rewrites the task as "For Image 1: ..., For Image 2: ..." and creates rejected responses by sequence truncation, caption shortening, and content swapping, so the model learns to prefer complete, correctly attributed sequences. Needle-level optimization adds region-targeted visual prompts to the input image, then contrasts correct region captions with captions from non-overlapping regions, and adds vision-contrastive losses $L_{\mathrm{Focus}}$ and $L_{\mathrm{Reject}}$ that reward preferring the aligned image over a contradictory one. All preference pairs come from MultiScope-42k, assembled automatically from existing caption pools (LLaVA-23K, MDVP, MVC) with perturbations applied only to the answer side.

What would settle it

Take a fixed set of multi-image QA questions, run CcDPO, and separately measure per-image caption accuracy on the same test images. If caption CHAIR/SCover improve while QA accuracy stays flat, or if end-to-end hallucination rates measured on the final answers do not drop, the central transfer claim is falsified. A simpler check is to swap the automatically generated captions for human-verified captions and see whether the reported gains survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that the persistent hallucinations of multi-image MLLMs—context omission, context conflation, and detail misinterpretation—are caused by insufficient per-image perception under cross-image interference, and that direct preference optimization can fix them at two complementary levels. At the context level, the model is trained with DPO to prefer a structured response that captions every image in order over rejected responses in which images are truncated, shortened, or swapped; at the needle level, visual prompts such as boxes and keypoints focus the model on a marked region, with rejected captions drawn from non-matching regions plus vision-contrastive pairs that reward attending to the aligned image and rejecting a contradictory one. Training both stages on MultiScope-42k, an automatically generated preference dataset, yields consistent gains on seven multi-image benchmarks and maintains or slightly improves single-image performance, with the largest gains on MuirBench and the longest needle-in-a-haystack contexts.

Load-bearing premise

The load-bearing premise is that improving the model's per-image captioning—trained on model-generated captions that the paper does not human-audit—transfers to multi-image question answering and reasoning, rather than only improving the captioning task itself.

Editorial extensions

If this is right

  • If the transfer claim holds, multi-image QA accuracy should improve on unseen benchmarks beyond the seven tested, since the training signal is caption-level and never exposes the model to QA-style instructions.
  • With the same 42k-scale preference data, both LLaVA-OV-7B and Qwen2-VL-7B improve, so the recipe should transfer to other open MLLMs without new data collection.
  • Hallucination rates on long sequences (8 images) fall by a large margin: CHAIR drops from 50.6 to 15.3 and sequence coverage rises from 10.3% to 83.3%, implying the model handles longer context better after training.
  • Needle-in-a-haystack retrieval gains grow with context length (up to +19.7 points at 24K tokens), suggesting the method strengthens long-range fine-grained attention rather than only memorizing training distributions.
  • Single-image performance is not sacrificed: average gains of +1.2 (LLaVA-OV) and +0.3 (Qwen2-VL) on single-image benchmarks indicate the per-image captioning training is broadly beneficial.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the authors' two-stage probe implies that captioning quality is a causal bottleneck; a direct test would be to intervene on caption accuracy at inference and measure QA gains as a function of caption CHAIR/SCover.
  • Editorial extension: because the rejected responses are cheap text perturbations, the same truncate/swap recipe could be applied to any existing image-text corpus; if the active ingredient is perturbation diversity rather than the caption pools, gains should reproduce on a completely new corpus.
  • Editorial extension: the paper's stated limitation is temporal reasoning; treating video frames as an image sequence and applying truncation and swapping across frames is a natural, untested extension suggested by the framework itself.
  • Editorial extension: the claimed hallucination reductions are measured on a caption-generation proxy; end-to-end hallucination in final answer text should be measured separately before assuming the same margin transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Context-to-Cue Direct Preference Optimization (CcDPO), a two-level DPO framework for multi-image MLLMs. Context-Level DPO (Sec. 4.1, Eq. 1) trains the model to prefer structured per-image captions over truncated, shortened, or content-swapped versions, targeting context omission and conflation. Needle-Level DPO (Sec. 4.2) combines language-based DPO on region-marked visual prompts with vision-contrastive objectives (L_Focus, L_Reject) to improve fine-grained detail perception. The authors construct MultiScope-42k, a fully automatic preference dataset from LLaVA-23K, MDVP, and MVC caption pools. On LLaVA-OV-7B and Qwen2-VL-7B, CcDPO reports average gains of +4.6 and +2.7 over MIA-DPO on seven multi-image benchmarks (Table 1), large improvements on MM-NIAH retrieval (Table 3), reduced hallucination on a self-constructed Context-AMBER-1K (Table 2), and roughly stable single-image performance (Table 4).

Significance. Assuming the evaluation concerns below are resolved, the paper's contribution is valuable: it offers a low-cost, scalable preference construction pipeline, a useful taxonomy of multi-image hallucinations, and consistent gains on external multi-image QA benchmarks with two different base models. The manuscript also contains several strengths that are sometimes missing in DPO papers: a data-size-matched ablation against MIA-DPO (Table 9), an ablation of multi-stage vs. one-stage training (Table 10), and an explicit disclosure of train/test image-source overlap (App. A.3). The main external result is not fit-by-construction because the seven QA benchmarks are largely independent of the caption-preference training objective. However, the strong hallucination-reduction claim currently rests on a benchmark whose scoring rules reward the exact output format the method is trained to emit, and the 'consistent performance gains' phrasing overstates the single-image results.

major comments (4)
  1. [Table 2; App. B.1] The large hallucination reductions in Table 2 are confounded with output-format compliance. CcDPO's chosen responses are explicitly formatted as 'For Image *: <caption>' (Sec. 4.1, Fig. 3b), and App. B.1 states that responses with fewer than N captions are padded with a repeated caption and 'heavily penalized' while out-of-order captions are penalized as conflation. Because no SFT baseline is reported in Table 2, the reader cannot separate the effect of learning the output format from the effect of DPO preference learning on factual grounding; part of the SCover jump from 10.3% to 83.3% may be pure format adherence. Please add an SFT baseline trained with the same structured format, and report format-agnostic content metrics, for example by matching each generated caption to its best-scoring image instead of enforcing strict order.
  2. [Abstract; Table 4] The abstract claims 'consistent performance gains across general single- and multi-image tasks,' but Table 4 contains several declines: Qwen2-VL CcDPO drops on SQA (84.1 to 82.8) and OCR (85.6 to 83.5), and LLaVA-OV drops on MMMU (47.7 to 45.7); the Qwen single-image average gain is only +0.3. No error bars or significance tests are reported anywhere in the paper. Please either soften the claim to 'gains on most benchmarks,' or report repeated-seed statistics and confidence intervals so the reader can judge whether the declines are noise.
  3. [Appendix A.3] The paper itself discloses partial overlap between training image sources (COCO-2014, COCO-2017, Flickr30k) and three of the seven multi-image benchmarks: MIRB, MIBench, and MUIRBench. The argument that MIA-DPO is also trained on LLaVA-23K addresses baseline parity but not the external validity of the absolute scores; images seen during caption training could still aid counting and referring tasks through memorized content. Please report results with overlapping subsets removed, or quantify how much of the Table 1 average gain (+4.6/+2.7) is contributed by the three overlapping benchmarks.
  4. [Sec. 4.1; App. A.1] The 'high-quality chosen responses' in MultiScope-42k are model-generated captions from LLaVA-23K, MDVP, and MVC; no human audit, automatic quality filter, or error-rate statistic is reported. Since DPO explicitly optimizes the model toward these captions (Eq. 1), any systematic errors in the chosen captions are directly reinforced and could propagate into the evaluation-time gains. Please include at least a sampled human audit of chosen captions (or an automatic filtering step with measured error rates) to substantiate the 'high-quality' claim.
minor comments (6)
  1. [Table 5] The column abbreviations GC, SD, TR, LR, FVR, TRI, VTK, TVK, Know., Reas., Perc., and M-Hop are not defined in the caption; some are introduced in App. B.2, but a reader of Table 5 alone cannot decode them.
  2. [Eq. (1); Sec. 4.1] The notation 'y_l ∈ {y_trunc_l, y_short_l, y_swap_l}' is used before the perturbations are formally defined; give a precise definition of each rejected variant, including what 'short' means in terms of token length or caption source.
  3. [Table 6] Table 6 contains typos ('Context-Levl', 'Needle-Levl') and the '⊕' symbol for method superposition is never defined; also state in the caption that the last row reports CcDPO (Context + Needle-TDPO + VDPO).
  4. [Sec. 3; Fig. 2] Sec. 3 and Fig. 2(c) report '∆ Accuracy = 7.37' without specifying which benchmark or subset this number refers to; please state the evaluation set and the exact prompting protocol.
  5. [Limitations] The limitations paragraph is a single sentence and does not mention the evaluation-format confound or the training-data overlap discussed in App. A.3; adding a sentence that acknowledges these measurement caveats would help readers calibrate the hallucination claims.
  6. [Table 7; Abstract] Table 9 reports CcDPO with 41.8K pairs while the paper's title says MultiScope-42k; reconcile the count (the abstract says 42k, Table 7 sums to 41.8k) and update the dataset name or the total.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline hallucination reduction on Context-AMBER-1K is partially the CcDPO training objective re-measured: the benchmark's omission and conflation penalties are exact codifications of the truncated and swapped rejected responses used in Eq. (1).

  1. self definitional [Sec. 4.1 (Eq. 1), Sec. 3 / App. B.1 (Context-AMBER-1K scoring), Table 2]
    "We reformulate the response generation task as a structured, per-image captioning problem... yw = [For Image 1:<caption 1>, For Image 2:<caption 2>,..., For Image N:<caption N>.] ... yl ∈ {ytrunc_l, yshort_l, yswap_l} ... For context omission, if fewer than N captions are generated ... Missing or incomplete captions are heavily penalized. ... If the model cannot clearly associate each description with its corresponding image ... the resulting hallucination scores will be higher for out-of-order descriptions."

    Context-AMBER operationalizes hallucination as (i) producing fewer than N per-image captions and (ii) associating descriptions with wrong image indices, penalizing both by construction. These are exactly the rejected perturbations (truncation and content swapping) that Eq. (1) trains the model to avoid, while the chosen format 'For Image i:<caption>' is identical to the benchmark prompt's required format. Therefore the large reported gains (SCover 10.3% to 83.3%, CHAIR 50.6 to 15.3 at 8 images) are in substantial part a re-measurement of the training objective rather than an independent test of factual grounding.

full rationale

The seven-benchmark multi-image evaluation (Table 1) and the single-image and MM-NIAH results are external, pre-existing benchmarks, so those gains are not fit-by-construction. The two-stage motivating experiment (Sec. 3, Fig. 2) is an empirical observation rather than a definitional equivalence. The disclosed partial overlap of COCO-derived training captions with images in MIRB, MIBench, and MUIRBench (App. A.3) is a data-contamination risk for those tables, but it is not a derivation-by-construction circularity because the training objective is caption preference, not QA, and the paper flags the overlap. The one load-bearing circular element is the hallucination evaluation: Context-AMBER-1K is a self-constructed benchmark whose omission and conflation scoring rules encode the same structured-format behaviors that the CcDPO context-level DPO objective explicitly rewards (full 'For Image i' sequences) and penalizes (truncation, swapping). The headline claim that CcDPO 'significantly reduces hallucinations' therefore rests partly on a metric that is an echo of the training loss. This is partial circularity, not full: CHAIR-style object grounding on AMBER images and external benchmarks still provide independent content, and no fitted parameter was renamed as a prediction. Score 6 reflects the one central claim that partially reduces by construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the quality of the automatically generated caption pools, the transfer from caption-style preferences to QA, and the absence of material train/test overlap. These are domain assumptions rather than formal axioms; they are not proved in the paper.

free parameters (5)
  • beta (DPO temperature) = 0.1
    Set for all three DPO losses following MIA-DPO [28]; controls preference strength and is not varied per benchmark.
  • gamma (NLL coefficient) = 0.1
    Mentioned in Section 5.1 as following [28], but gamma does not appear in Equations 1-3, so its role in the reported loss is underspecified.
  • LoRA rank = 128
    Chosen for efficiency in Stages 1 and 2 (Section 5.1); affects how much of the model is updated.
  • Learning rates = 5e-6 / 5e-5 / 1e-6
    Per-stage learning rates for context, needle, and vision-contrastive stages (Section 5.1); hand-picked, no sensitivity study.
  • Training data mix = 27.3k context + 10.8k TDPO + 3.7k VDPO
    The composition of MultiScope-42k is a design choice; Appendix C shows performance varies with data volume, so the reported gains depend on this mix.
assumptions (6)
  • domain assumption The three hallucination types (context omission, conflation, detail misinterpretation) are the principal failure modes in multi-image MLLMs.
    Introduced in Sec. 3 via Context-AMBER-1K and used to justify the two optimization levels. No external validation that this taxonomy is complete.
  • domain assumption Caption pools (LLaVA-23K, MDVP, MVC) contain accurate image- and region-level descriptions suitable as chosen responses.
    Sec. 4.1-4.2 and Appendix A.1; if these machine-generated captions contain hallucinations, DPO trains toward those errors.
  • domain assumption Improving intrinsic per-image captioning transfers to multi-image question answering and reasoning.
    Sec. 3 argues from a two-stage proxy (Fig. 2) that accurate context raises VQA accuracy; the method optimizes caption-style preferences rather than QA directly.
  • domain assumption DPO preferences built from synthetic perturbations are valid proxies for human preferences.
    The rejected responses are constructed by truncation, swapping, and region mismatch; no human validation of the preference pairs is reported.
  • domain assumption Benchmark performance is not materially inflated by training/test image overlap.
    Appendix A.3 argues overlap is acceptable because training is caption-style, but MIRB, MIBench, and MUIRBench are admitted to contain COCO images that also appear in training sources.
  • standard math DPO loss form (Rafailov et al. 2023) is valid for preference optimization.
    Equations 1-3 use the standard DPO objective; no new derivation is attempted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs." pith.science (2026). https://pith.science/paper/GUMQBZFM

@misc{pith2026250522396,
  author       = {Pith},
  title        = {Pith review of: Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GUMQBZFM}},
  note         = {Machine review of arXiv:2505.22396}
}
read the original abstract

Multi-modal Large Language Models (MLLMs) excel at single-image tasks but struggle with multi-image understanding due to cross-modal misalignment, leading to hallucinations (context omission, conflation, and misinterpretation). Existing methods using Direct Preference Optimization (DPO) constrain optimization to a solitary image reference within the input sequence, neglecting holistic context modeling. We propose Context-to-Cue Direct Preference Optimization (CcDPO), a multi-level preference optimization framework that enhances per-image perception in multi-image settings by zooming into visual clues -- from sequential context to local details. It features: (i) Context-Level Optimization : Re-evaluates cognitive biases underlying MLLMs' multi-image context comprehension and integrates a spectrum of low-cost global sequence preferences for bias mitigation. (ii) Needle-Level Optimization : Directs attention to fine-grained visual details through region-targeted visual prompts and multimodal preference supervision. To support scalable optimization, we also construct MultiScope-42k, an automatically generated dataset with high-quality multi-level preference pairs. Experiments show that CcDPO significantly reduces hallucinations and yields consistent performance gains across general single- and multi-image tasks.

Figures

Figures reproduced from arXiv: 2505.22396 by the authors.

Figure 1
Figure 1. (a) Prior multi-image DPO (e.g., MIA-DPO) is constrained by its reliance on predefined [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Baseline: Direct inference without context as a condition. (b) Two-stage approach: Gen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of CcDPO. (a) Caption pools are built from LLaVA-23K [62], MDVP [63], and MVC [49] for image- and region-level supervision. (b) Context-Level DPO aligns model outputs with complete, coherent image sequences and penalizes omissions, conflation, and misalignments. (c) Needle-Level DPO incorporates visual prompts to enhance local detail understanding. chosen responses describe marked regions accurately, while … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Token length distributions of chosen and rejected responses in our MultiScope-42k and [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: The dataset word cloud comparison between our MultiScope-42k and MIA-DPO [28]. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Example of Context-Level preference pair with truncation perturbation (Context Omission). [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Example of Context-Level preference pair with caption shortening perturbation (Context [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Example of Context-Level preference pair with swapping perturbation (Context Conflation). [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]
Figure 9
Figure 9. Figure 9: Example of Needle-Level preference pair with bbox region mismatches perturbation (Detail [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Example of Needle-Level preference pair with point region mismatches perturbation [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Example of Needle-Level preference pair with image contrastive perturbation (Detail [PITH_FULL_IMAGE:figures/full_fig_p028_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 14 canonical work pages

  1. [2]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [3]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025

    arXiv 2025

  3. [4]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  4. [5]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  5. [8]

    Large language models are visual reasoning coordinators.Advances in Neural Information Processing Systems, 36:70115–70140, 2023

    Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Large language models are visual reasoning coordinators.Advances in Neural Information Processing Systems, 36:70115–70140, 2023

    work page 2023

  6. [9]

    Chameleon: Plug-and-play compositional reasoning with large language models.Advances in Neural Information Processing Systems, 36:43447–43478, 2023

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models.Advances in Neural Information Processing Systems, 36:43447–43478, 2023. 10

    work page 2023

  7. [10]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023

    work page 2023

  8. [11]

    A picture is worth more words over time: Multimodality and narrative structure across eight decades of american superhero comics

    Neil Cohn, Ryan Taylor, and Kaitlin Pederson. A picture is worth more words over time: Multimodality and narrative structure across eight decades of american superhero comics. Multimodal Communication, 6(1):19–37, 2017

    2017

Show all 82 references
  1. [12]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

    2024

  2. [13]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  3. [14]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  4. [15]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  5. [16]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017

    2017

  6. [17]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019

    2019

  7. [20]

    What matters when building vision-language models?Advances in Neural Information Processing Systems, 37:87874–87907, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?Advances in Neural Information Processing Systems, 37:87874–87907, 2024

    2024

  8. [21]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2024

  9. [22]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  10. [23]

    V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization.arXiv preprint arXiv:2411.02712, 2024

    Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization.arXiv preprint arXiv:2411.02712, 2024

    2024 arXiv

  11. [24]

    Aligning modali- ties in vision large language models via preference fine-tuning.arXiv preprint arXiv:2402.11411, 2024

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modali- ties in vision large language models via preference fine-tuning.arXiv preprint arXiv:2402.11411, 2024. 11

    2024 arXiv

  12. [25]

    Vistadpo: Video hierarchical spatial-temporal direct preference optimiza- tion for large video models.arXiv preprint arXiv:2504.13122, 2025

    Haojian Huang, Haodong Chen, Shengqiong Wu, Meng Luo, Jinlan Fu, Xinya Du, Hanwang Zhang, and Hao Fei. Vistadpo: Video hierarchical spatial-temporal direct preference optimiza- tion for large video models.arXiv preprint arXiv:2504.13122, 2025

    2025 arXiv

  13. [26]

    Automated multi-level preference for mllms.Advances in Neural Information Processing Systems, 37:26171–26194, 2024

    Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng, Jingdong Wang, et al. Automated multi-level preference for mllms.Advances in Neural Information Processing Systems, 37:26171–26194, 2024

    2024

  14. [27]

    Clip-dpo: Vision- language models as a source of preference for fixing hallucinations in lvlms

    Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Clip-dpo: Vision- language models as a source of preference for fixing hallucinations in lvlms. InEuropean Conference on Computer Vision, pages 395–413. Springer, 2024

    2024

  15. [28]

    Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models

    Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models. 2025

    2025

  16. [29]

    Identifying and mitigating position bias of multi-image vision-language models.arXiv preprint arXiv:2503.13792, 2025

    Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Identifying and mitigating position bias of multi-image vision-language models.arXiv preprint arXiv:2503.13792, 2025

    2025 arXiv

  17. [30]

    Cmmcot: Enhancing complex multi-image comprehension via multi-modal chain-of-thought and memory augmentation.arXiv preprint arXiv:2503.05255, 2025

    Guanghao Zhang, Tao Zhong, Yan Xia, Zhelun Yu, Haoyuan Li, Wanggui He, Fangxun Shu, Mushui Liu, Dong She, Yi Wang, et al. Cmmcot: Enhancing complex multi-image comprehension via multi-modal chain-of-thought and memory augmentation.arXiv preprint arXiv:2503.05255, 2025

    2025

  18. [31]

    Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference.arXiv preprint arXiv:2406.18139, 2024

    Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference.arXiv preprint arXiv:2406.18139, 2024

    2024 arXiv

  19. [32]

    Chip: Cross-modal hierarchical direct preference optimization for multimodal llms.arXiv preprint arXiv:2501.16629, 2025

    Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, and See-Kiong Ng. Chip: Cross-modal hierarchical direct preference optimization for multimodal llms.arXiv preprint arXiv:2501.16629, 2025

    2025 arXiv

  20. [33]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  21. [34]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  22. [35]

    Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms.arXiv preprint arXiv:2406.11833, 2024

    Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms.arXiv preprint arXiv:2406.11833, 2024

    2024 arXiv

  23. [36]

    Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    2018 arXiv

  24. [37]

    An llm-free multi-dimensional benchmark for mllms hallucination evaluation.CoRR, 2023

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for mllms hallucination evaluation.CoRR, 2023

    2023

  25. [38]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProcee...

    2024

  26. [39]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  27. [40]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents.Advances in Neural Information Processing Systems, 36:71683–71702, 2023

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents.Advances in Neural Information Pro...

    2023

  28. [41]

    Multimodal c4: An open, billion-scale corpus of images interleaved with text.Advances in Neural Information Processing Systems, 36:8958–8974, 2023

    Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text.Advances in Neural Information Processing Systems,...

    2023

  29. [42]

    Muirbench: A comprehensive benchmark for robust multi-image understanding.arXiv preprint arXiv:2406.09411, 2024

    Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding.arXiv preprint arXiv:2406.09411, 2024

    2024 arXiv

  30. [43]

    Remi: A dataset for reasoning with multiple images.Advances in Neural Information Processing Systems, 37:60088– 60109, 2024

    Mehran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Sreenivas Gollapudi, Dee Guo, et al. Remi: A dataset for reasoning with multiple images.Advances in Neural Information Processing Systems, 37:60088– 60109, 2024

    2024

  31. [44]

    Mantis: Interleaved multi-image instruction tuning.arXiv preprint arXiv:2405.01483, 2024

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning.arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  32. [45]

    Understanding and improving in-context learning on vision-language models.arXiv preprint arXiv:2311.18021, 2023

    Shuo Chen, Zhen Han, Bailan He, Mark Buckley, Philip Torr, V olker Tresp, and Jindong Gu. Understanding and improving in-context learning on vision-language models.arXiv preprint arXiv:2311.18021, 2023

    2023 arXiv

  33. [46]

    Mmiu: Multimodal multi-image understanding for evaluating large vision-language models.arXiv preprint arXiv:2408.02718, 2024

    Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models.arXiv preprint arXiv:2408.02718, 2024

    2024 arXiv

  34. [47]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  35. [48]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  36. [49]

    Symmetrical visual contrastive optimization: Aligning vision-language models with minimal contrastive images.arXiv preprint arXiv:2502.13928, 2025

    Shengguang Wu, Fan-Yun Sun, Kaiyue Wen, and Nick Haber. Symmetrical visual contrastive optimization: Aligning vision-language models with minimal contrastive images.arXiv preprint arXiv:2502.13928, 2025

    2025 arXiv

  37. [50]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization.arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  38. [51]

    Tarsier2: Advanc- ing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025

    Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advanc- ing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025

    2025 arXiv

  39. [52]

    Modality-fair preference optimization for trustworthy mllm alignment.arXiv preprint arXiv:2410.15334, 2024

    Songtao Jiang, Yan Zhang, Ruizhe Chen, Yeying Jin, and Zuozhu Liu. Modality-fair preference optimization for trustworthy mllm alignment.arXiv preprint arXiv:2410.15334, 2024

    2024 arXiv

  40. [53]

    Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258, 2024

    2024 arXiv

  41. [54]

    Fine-grained verifiers: Preference modeling as next-token prediction in vision-language alignment.arXiv preprint arXiv:2410.14148, 2024

    Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, and Tat-Seng Chua. Fine-grained verifiers: Preference modeling as next-token prediction in vision-language alignment.arXiv preprint arXiv:2410.14148, 2024

    arXiv

  42. [55]

    Token preference optimization with self-calibrated visual-anchored rewards for hallucination mitigation.arXiv preprint arXiv:2412.14487, 2024

    Jihao Gu, Yingyao Wang, Meng Cao, Pi Bu, Jun Song, Yancheng He, Shilong Li, and Bo Zheng. Token preference optimization with self-calibrated visual-anchored rewards for hallucination mitigation.arXiv preprint arXiv:2412.14487, 2024. 13

    2024

  43. [56]

    Denseclip: Language-guided dense prediction with context-aware prompting

    Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18082–18...

    2022

  44. [57]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

    2023

  45. [58]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  46. [59]

    What does clip know about a red circle? visual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 11987–11997, 2023

    2023

  47. [60]

    Cpt: Colorful prompt tuning for pre-trained vision-language models.AI Open, 5:30–38, 2024

    Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models.AI Open, 5:30–38, 2024

    2024

  48. [61]

    Controlmllm: Training-free visual prompt learning for multimodal large language models.Advances in Neural Information Processing Systems, 37:45206–45234, 2024

    Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, Guannan Jiang, Xiaoshuai Sun, and Rongrong Ji. Controlmllm: Training-free visual prompt learning for multimodal large language models.Advances in Neural Information Processing Systems, 37:45206–45234, 2024

    2024

  49. [62]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  50. [63]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want.arXiv preprint arXiv:2403.20271, 2024

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want.arXiv preprint arXiv:2403.20271, 2024

    2024 arXiv

  51. [64]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings,...

    2014

  52. [66]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  53. [67]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024

    2024 arXiv

  54. [68]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  55. [69]

    Benchmarking multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning.arXiv preprint arXiv:2406.12742, 2024

    Bingchen Zhao, Yongshuo Zong, Letian Zhang, and Timothy Hospedales. Benchmarking multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning.arXiv preprint arXiv:2406.12742, 2024

    2024 arXiv

  56. [70]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, pages 148–166. Springer, 2024. 14

    2024

  57. [71]

    A corpus for reasoning about natural language grounded in photographs.arXiv preprint arXiv:1811.00491, 2018

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs.arXiv preprint arXiv:1811.00491, 2018

    2018 arXiv

  58. [72]

    Q-bench: A benchmark for multi-modal foundation models on low-level vision from single images to pairs.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Zicheng Zhang, Haoning Wu, Erli Zhang, Guangtao Zhai, and Weisi Lin. Q-bench: A benchmark for multi-modal foundation models on low-level vision from single images to pairs.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  59. [73]

    Mibench: Evaluating multimodal large language models over multiple images.arXiv preprint arXiv:2407.15272, 2024

    Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, et al. Mibench: Evaluating multimodal large language models over multiple images.arXiv preprint arXiv:2407.15272, 2024

    2024 arXiv

  60. [74]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  61. [75]

    Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  62. [76]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  63. [77]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. S...

    2016

  64. [78]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  65. [79]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  66. [80]

    Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

    2024

  67. [81]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  68. [82]

    Needle in a multimodal haystack.Advances in Neural Information Processing Systems, 37:20540–20565, 2024

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack.Advances in Neural Information Processing Systems, 37:20540–20565, 2024

    2024

  69. [83]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  70. [84]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    2023 arXiv

  71. [85]

    Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390,...

    2023 arXiv

  72. [86]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  73. [87]

    Countercurate: Enhancing physical and semantic visio-linguistic compositional reasoning via counterfactual examples.arXiv preprint arXiv:2402.13254, 2024

    Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic visio-linguistic compositional reasoning via counterfactual examples.arXiv preprint arXiv:2402.13254, 2024

    2024 arXiv

  74. [88]

    Describe the following images

    Junzhuo Liu, Xuzheng Yang, Weiwei Li, and Peng Wang. Finecops-ref: A new dataset and task for fine-grained compositional referring expression comprehension.arXiv preprint arXiv:2409.14750, 2024. 16 Technical Appendices In this appendix, we provide additional materials to suppo...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.