Pith. sign in

REVIEW 3 major objections 7 minor 97 references

OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OpenING claims a 5,400-instance benchmark for mixed image-text generation and a judge model, IntJudge, that matches human pairwise preferences 82.42% of the time, outperforming GPT-4o-based evaluation by 11.34 percentage points.

desk verdict A genuinely larger and more diverse benchmark for interleaved generation, but IntJudge's headline agreement rests on human preferences that are never validated for reliability. read the letter →

arxiv 2411.18499 v3 pith:3QF3W7GS submitted 2024-11-27 cs.CV

Pengfei Zhou , Xiaopeng Peng , Jiajun Song , Chuanhao Li , Zhaopan Xu , Yue Yang , Ziyao Guo , Hao Zhang
show 10 more authors
Yuqi Lin Yefei He Lirui Zhao Shuo Liu Tianhua Li Yuxuan Xie Xiaojun Chang Yu Qiao Wenqi Shao Kaipeng Zhang
This is my paper · ORCID
classification cs.CV
keywords interleavedimage-textgenerationmultimodalevaluationbenchmarkjudgemodelhumanpreferencealignmentreference-augmentedpairwisecomparisonopen-ended
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fill a gap in how researchers measure models that produce interleaved image-and-text answers, such as a travel itinerary with a picture at each step. It introduces OpenING, a benchmark of 5,400 human-annotated examples spanning 56 everyday tasks, and IntJudge, a compact open judge model trained to compare two such outputs and pick the better one. The central claim is that IntJudge agrees with human preference 82.42% of the time, beating GPT-4o-based judging by 11.34 percentage points. If true, open-ended multimodal generation can be evaluated offline, cheaply, and reproducibly instead of relying on proprietary APIs or slow human panels.

What carries the argument

The load-bearing object is IntJudge, a Qwen2-VL-7B model fine-tuned with a four-term loss (cross-entropy, contrastive, MSE, and pairwise ranking) on 31,996 training samples: 6,014 arena pairwise judgments plus 25,982 reference-augmented pairs. The RAG pipeline generates contrastive pairs by prompting seen generators with gold answers and always labeling the gold-augmented output the winner; the paper reports that adding these pairs raises force-dividing-tie agreement on unseen models by 7.8%. Evaluation runs in an Interleaved Arena where two anonymous outputs are compared on seven ordered criteria—correctness, image-text coherency, multi-step consistency, content quality, human preference alignment, completeness, and content richness—with ties resolved by forced leaning judgments.

What would settle it

Take a random subset of 200 OpenING battle pairs, have two independent annotation teams judge them, and compute the agreement between teams; if inter-annotator agreement is markedly below IntJudge's 82.42% agreement with the original labels, then IntJudge is matching one noisy label set rather than measuring true quality. Separately, run a human preference test on the RAG training pairs: if humans do not prefer the gold-augmented output at near 100%, the RAG 'winner' labels embed a bias that inflates judge accuracy.

Watch

Extended reading notes

Core claim

The paper claims that open-ended interleaved generation can be judged reliably by a compact model trained on pairwise human preferences plus reference-augmented data. On 4,320 battle pairs sampled from the OpenING Test Set, IntJudge matches human decisions at 82.42% agreement under the force-dividing-tie metric, outperforming GPT-4o (71.08%) and random guessing (49.83%), with stronger agreement on seen models (84.05%) than unseen models (80.75%). The paper also reports that all current generators trail human-annotated content, that integrated pipelines such as GPT-4o+DALL·E-3 and Gemini1.5+Flux lead the leaderboard, and that end-to-end models such as Anole, MiniGPT-5, GILL, and NExT-GPT lag substantially.

Load-bearing premise

The whole evaluation rests on the assumption that the pairwise preferences collected from 28 annotators are a reliable and unbiased ground truth for open-ended interleaved quality; the paper reports no inter-annotator agreement, and part of IntJudge's training signal is constructed by a rule that always favors gold-augmented outputs.

Editorial extensions

If this is right

  • IntJudge can replace GPT-based scoring on OpenING with higher human agreement, enabling reproducible offline leaderboards for interleaved generation.
  • The 56-task Test Set provides a zero-shot evaluation suite for future interleaved models, including models never seen during judge training.
  • Integrated pipelines currently outperform end-to-end generators in image-text coherence and visual quality, while unified two-stage and end-to-end architectures retain substantial headroom.
  • Image-generator quality is a dominant factor: swapping in Flux-dev for Flux-schnell or SEED-X images changes win rates dramatically, so progress in image generators will directly lift interleaved generation.
  • Fine-tuning a small generator on the OpenING Dev Set improves its interleaved output, as shown by MiniGPT-5OpenING outperforming the MiniGPT-5 baselines across all tie-handling metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The RAG training signal labels gold-augmented outputs as winners by construction, so part of IntJudge's agreement gain may come from learning that rule rather than from genuine human preference; a human study on RAG pairs would reveal how much of the 82.42% is preference alignment versus shortcut learning.
  • The human ground truth itself lacks reported inter-annotator agreement, so the 82.42% figure is an agreement with one label set; measuring pairwise agreement between independent annotation teams on a subset would calibrate the ceiling of judge performance.
  • If IntJudge is adopted as a reward model for reinforcement learning, as the paper hints, its bias toward complete reference-augmented answers and against generated images would shape model behavior; a testable extension is training a generator with IntJudge rewards and checking whether human preference actually improves.
  • The benchmark's 23-meta-topic structure invites per-topic diagnostic leaderboards: models that win overall may lose on GUI navigation or embodied-AI tasks, which could guide where next-generation training data should come from.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces OpenING, a benchmark of 5,400 human-annotated interleaved image-text instances spanning 56 tasks across 23 meta-topics, and IntJudge, a Qwen2-VL-7B-based judge model trained on pairwise preferences from a Dev Set plus Reference-Augmented Generation (RAG) pairs. The authors report that IntJudge agrees with human judgments at 82.42% (FDT metric), an 11.34% improvement over GPT-4o, and they use the benchmark to rank ten to fourteen interleaved generation methods. The benchmark, judge model, and leaderboard are positioned as the paper's main contributions.

Significance. If the human ground truth is reliable, OpenING is a valuable resource: it is substantially larger and broader than OpenLEAF and InterleavedBench, it provides a reproducible open-source offline judge, and the RAG-based data pipeline for scaling judge training is a useful practical contribution. The paper also offers a detailed leaderboard, per-criterion analyses, and error analyses that can inform future model development. The key caveat is that the human-preference signal is treated as a stable gold label without any reliability or uncertainty analysis, so the headline agreement numbers and rankings are currently not fully interpretable.

major comments (3)
  1. [Sec. 4.1, 4.2, A.5; Table 3] The benchmark's ground truth is a set of pairwise human judgments, but the paper never reports how many annotators judged each battle pair, whether any pairs were double-annotated, or any inter-annotator agreement statistic such as Cohen's kappa or Krippendorff's alpha. Without such a reliability analysis, the 82.42% agreement figure in Table 3 and the win-rate rankings in Table 2 have an unvalidated target. Please add a reliability study on a double-annotated subset, describe the per-pair annotation workflow, and state how disagreements among annotators were resolved.
  2. [Table 3, Sec. 5.2, Fig. 7] The central quantitative claims are reported as point estimates without uncertainty quantification. The 82.42% vs. 71.08% agreement difference and the mid-table win-rate differences (e.g., SEED-X at 51.98% vs. Anole at 51.90% in human FDT) could easily be within noise. Please provide bootstrap confidence intervals for the agreement rates and win rates, and a paired significance test for the IntJudge versus GPT-4o comparison and for the key ranking differences.
  3. [Sec. 4.3, Fig. 8] The RAG training pairs are labeled by construction: the gold-answer-augmented output is always assigned as the winner, and the gold answers were produced by the same annotator pool that created the human-preference labels used for evaluation. The 7.8% FDT improvement on unseen models from RAG could therefore partly reflect a preference for those annotators' style rather than a general quality signal. Please provide evidence that the improvement transfers to independent human judgments, for example by evaluating on a held-out annotator group or by showing that the RAG-trained judge's agreement is stable across annotator subpopulations.
minor comments (7)
  1. [Sec. 5.1 vs. Table 12] The main text says 10 representative interleaved methods were evaluated, but the supplementary Table 12 reports 14 models; please harmonize the count and clarify which models the '10' refers to.
  2. [Sec. 4.3] The symbol g in 'A bag of models, including g seen interleaved generation methods' is introduced but never defined; please define it or rephrase.
  3. [Sec. 4.1, Eq. (5)] Equation (5) is typeset in a way that makes the coverage-time formula hard to parse; please rewrite with explicit floor/ceiling notation and define every symbol.
  4. [Sec. 4.3] The phrase 'tested in a zero-shot setting on both unseen and seen models' is misleading for seen models, since IntJudge was trained on Dev Set outputs from those models; please clarify that the evaluation is on held-out instances rather than zero-shot in the usual sense.
  5. [Sec. 2, Table 1] The claim of 'reducing data contamination risks' is not fully supported, because several OpenING tasks reuse public sources such as VIST and WikiHow that also appear in prior interleaved benchmarks; please document explicit overlap checks with evaluated models' training corpora and with prior benchmarks.
  6. [Various] There are several typos and formatting issues, including 'favarable' in Section 5.2 and 'promopts' in the Table 11 header; a careful proofreading pass is recommended.
  7. [Sec. F] The limitation statement in Section F acknowledges potential biases from crowdsourced aesthetic judgments, but the main text around Table 3 does not reference this limitation; please cross-link the two so that readers see the caveat where the headline numbers are presented.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation found: IntJudge is trained on Dev Set human preferences and RAG-augmented pairs, then evaluated against independent Test Set human judgments, so the 82.42% agreement is an empirical held-out result rather than a fitted input renamed as a prediction.

full rationale

The paper's central claims separate cleanly. OpenING is constructed independently of IntJudge through a human annotation pipeline, filtering protocols, and a Dev/Test split. IntJudge is trained on Dev Set pairwise preferences plus RAG pairs, where gold-augmented outputs are labeled winners by construction; however, that is a training-data synthesis choice, not a derivation of the evaluation result. The headline agreement is measured on Test Set battle pairs against human judgments that were not used for training, so the result is not forced by construction. The RAG ablation compares agreement with held-out human judgments and reports gains, especially on unseen models, which is an empirical claim rather than a tautology. The paper also reports seen versus unseen agreement, with unseen FDT agreement at 80.75%, indicating the improvement is not merely memorization of training models. There is no uniqueness theorem, no ansatz smuggled in via self-citation, and no equation in which a predicted quantity reduces to a fitted parameter. Self-citations such as GUI Odyssey, MMT-Bench, and InternLM-XComposer are used as data sources or baselines, not as load-bearing justifications of the core result. The paper itself flags in Sec. F that potential biases may arise from crowdsourced aesthetic judgments, and it does not report inter-annotator agreement; these are legitimate reliability and validity limitations on the ground-truth signal, but they do not make the derivation circular. Overall, the derivation chain is self-contained, with only minor non-load-bearing self-citations, so the circularity score is 1.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; IntJudge is a fine-tuned instantiation of Qwen2-VL-7B and OpenING is a dataset, not an entity in the sense of a new force or mediator.

free parameters (4)
  • IntJudge loss weights = lambda_1=1.0, lambda_2=0.01, lambda_3=0.01, lambda_4=0.01
    Chosen manually for training IntJudge; the reported agreement depends on this configuration.
  • Learning rate = 1e-4
    Training hyperparameter for IntJudge, not derived from first principles.
  • Training epochs = 20
    Manual choice for IntJudge training.
  • Sampling rounds E = 2
    Number of battle pair sampling rounds in the arena; affects the 4,320 pairs evaluated.
assumptions (3)
  • domain assumption Human annotator preferences are a valid ground truth for open-ended interleaved generation quality.
    The entire IntJudge training and agreement evaluation treat pairwise human votes as the reference standard (Section 4.1, Section C.3).
  • domain assumption The seven evaluation criteria (correctness, coherence, consistency, and others) jointly capture output quality.
    Used to instruct both human judges and IntJudge; no evidence is provided that this set is complete or redundant.
  • domain assumption RAG outputs conditioned on gold answers are always better than plain outputs.
    RAG pairs are labeled with the RAG result as winner by construction (Section 4.3, Figure 3b), introducing a synthetic preference signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation." pith.science (2026). https://pith.science/paper/3QF3W7GS

@misc{pith2026241118499,
  author       = {Pith},
  title        = {Pith review of: OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QF3W7GS}},
  note         = {Machine review of arXiv:2411.18499}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding and generation tasks. However, generating interleaved image-text content remains a challenge, which requires integrated multimodal understanding and generation abilities. While the progress in unified models offers new solutions, existing benchmarks are insufficient for evaluating these methods due to limitations in data size and diversity. To bridge this gap, we introduce OpenING, a comprehensive benchmark comprising 5,400 high-quality human-annotated instances across 56 real-world tasks. OpenING covers diverse daily scenarios such as travel guide, design, and brainstorming, offering a robust platform for challenging interleaved generation methods. In addition, we present IntJudge, a judge model for evaluating open-ended multimodal generation methods. Trained with a novel data pipeline, our IntJudge achieves an agreement rate of 82.42% with human judgments, outperforming GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that current interleaved generation methods still have substantial room for improvement. Key findings on interleaved image-text generation are further presented to guide the development of next-generation models.

Figures

Figures reproduced from arXiv: 2411.18499 by the authors.

Figure 1
Figure 1. Motivation: (a) Rapid progress of interleaved image [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. OpenING benchmark consists of 23 meta-topics (inner ring) which are further categorized into 56 specific tasks (see the number of [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of data curation and the proposed judge pipeline. (a) We construct our OpenING benchmark in a top-down manner, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: Win rate matrix of human and ten MLLM models, evalu [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Effect of sampling size on evaluation reliability. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 6
Figure 6. Figure 6: Evaluation results of GPT-based scores. (a)-(c): Average [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: Error distribution of three models: GPT-4o+DALL [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Interface of IntLabel, which shows a case where data is entered to finish annotation for an instance. [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Annotation Interface of Interleaved Arena for human judges to compare the anonymous outputs of model A and model B. Human [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: The system prompt for using GPT-4o as a judge to compare outputs from two interleaved generation methods. [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: The system prompts for using MLLMs as a judge to compare outputs of two interleaved generation methods. [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: The system prompt for obtaining detailed scores from GPT-based evaluators. Brief explanations are also required to support [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: The win rates of 14 interleaved generation methods across 23 meta-topics. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Win rate matrices of 14 interleaved genration methods, evaluated by Human, GPT-4o, and our IntJudge, respectively. [PITH_FULL_IMAGE:figures/full_fig_p031_16.png]
Figure 17
Figure 17. Figure 17: Win rate curves with respect to different sampling sizes. [PITH_FULL_IMAGE:figures/full_fig_p032_17.png]
Figure 19
Figure 19. Figure 19: Illustration of 23 pairwise comparison cases. The meta-topic name ( [PITH_FULL_IMAGE:figures/full_fig_p044_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

97 extracted references · 43 canonical work pages

  1. [1]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems,

  3. [3]

    BoQ: A place is worth a bag of learnable queries

    Amar Ali-bey, Brahim Chaib-draa, and Philippe Gigu `ere. BoQ: A place is worth a bag of learnable queries. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17794–17803, 2024. 14

    2024

  4. [4]

    Openleaf: Open-domain interleaved image-text generation and evalua- tion

    Jie An, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Lijuan Wang, and Jiebo Luo. Openleaf: Open-domain interleaved image-text generation and evalua- tion. arXiv preprint arXiv:2310.07749, 2023. 1, 2, 3

    arXiv 2023

  5. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 1

    arXiv 2023

  6. [6]

    Benchmarking foundation models with language-model- as-an-examiner

    Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with language-model- as-an-examiner. In Proceedings of the Advances in Neural Information Processing Systems, 2024. 3

    2024

  7. [7]

    Barron, Ben Mildenhall, Dor Verbin, Pratul P

    Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022. 14

    2022

  8. [8]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science, 2(3):8, 2023. 5, 17, 27

    2023

Show all 97 references
  1. [9]

    Black Forest Labs. Flux. https://github.com/ black- forest- labs/flux, 2024. Accessed: 2024- 11-05. 5, 17, 27

    2024

  2. [10]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 13

    2023

  3. [11]

    ARES: Alternating reinforcement learning and super- vised fine-tuning for enhanced multi-modal chain-of-thought reasoning through diverse AI feedback

    Ju-Seung Byun, Jiyun Chun, Jihyung Kil, and Andrew Per- rault. ARES: Alternating reinforcement learning and super- vised fine-tuning for enhanced multi-modal chain-of-thought reasoning through diverse AI feedback. arXiv preprint arXiv:2407.00087, 2024. 1

    2024 arXiv

  4. [12]

    Activitynet: A large-scale video bench- mark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video bench- mark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 15

    2015

  5. [13]

    Multi-modal sar- casm detection in Twitter with hierarchical fusion model

    Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sar- casm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 15

    2019

  6. [14]

    MLLM-as-a-Judge: Assessing multimodal llm-as-a-judge with vision-language benchmark

    Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. MLLM-as-a-Judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Proceed- ings of the International Conference on Machine Learning,

  7. [15]

    Gui-world: A dataset for gui-oriented multimodal llm-based agents, 2024

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. Gui-world: A dataset for gui-oriented ...

    2024

  8. [16]

    Anole: An open, autoregressive, native large multimodal mod- els for interleaved image-text generation

    Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal mod- els for interleaved image-text generation. arXiv preprint arXiv:2407.06135, 2024. 3, 5, 17, 27

    2024 arXiv

  9. [17]

    Artificial intelligence in dental education: Opportunities and challenges of large language models and multimodal foundation models

    Daniel Claman, Emre Sezgin, et al. Artificial intelligence in dental education: Opportunities and challenges of large language models and multimodal foundation models. JMIR Medical Education, 10(1):e52346, 2024. 1

    2024

  10. [18]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Visio...

  11. [19]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017. 15

    2017

  12. [20]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim En- tezari, Jonas M¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Proceedings of the International Conference...

    2024

  13. [21]

    Towards expressive communication with internet memes: A new multimodal conversation dataset and bench- mark

    Zhengcong Fei, Zekang Li, Jinchao Zhang, Yang Feng, and Jie Zhou. Towards expressive communication with internet memes: A new multimodal conversation dataset and bench- mark. arXiv preprint arXiv:2109.01839, 2021. 15

    2021 arXiv

  14. [22]

    Making llama see and draw with seed tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023. 5, 17, 27

    2023 arXiv

  15. [23]

    Seed-data-edit technical report: A hybrid dataset for instruc- tional image editing

    Yuying Ge, Sijie Zhao, Chen Li, Yixiao Ge, and Ying Shan. Seed-data-edit technical report: A hybrid dataset for instruc- tional image editing. arXiv preprint arXiv:2405.04007, 2024. 3, 5, 13, 17, 27

    2024 arXiv

  16. [24]

    Making LLaMA SEE and Draw with SEED Tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making LLaMA SEE and Draw with SEED Tokenizer. In Proceedings of the International Conference on Learning Representations, 2024. 3

    2024

  17. [25]

    A multi-modal parcellation of human cerebral cortex

    Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):171–178, 2016. 1

    2016

  18. [26]

    Opens- ketch: A richly-annotated dataset of product design sketches

    Yulia Gryaditskaya, Mark Sypesteyn, Jan Willem Hoftijzer, Sylvia Pont, Fr ´edo Durand, and Adrien Bousseau. Opens- ketch: A richly-annotated dataset of product design sketches. ACM Transactions on Graphics (Proc. SIGGRAPH Asia), 38,

  19. [27]

    Ava: A video dataset of spatio-temporally localized atomic visual actions

    Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Car- oline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE conf...

  20. [28]

    OpenDataLab: Empowering general ar- tificial intelligence with open datasets

    Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, and Dahua Lin. OpenDataLab: Empowering general ar- tificial intelligence with open datasets. arXiv preprint arXiv:2407.13773, 2024. 4

    2024 arXiv

  21. [29]

    CLIPScore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  22. [30]

    GANs trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the Advances in Neural Information Pro- cessing Systems, 2017. 3

    2017

  23. [31]

    Multimodal language processing in human communication

    Judith Holler and Stephen C Levinson. Multimodal language processing in human communication. Trends in Cognitive Sciences, 23(8):639–652, 2019. 1

    2019

  24. [32]

    Visual storytelling

    Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Is- han Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the Conference of the North American Chapter of the Association for Comp...

    2016

  25. [33]

    Pnp inversion: Boosting diffusion-based editing with 3 lines of code

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pnp inversion: Boosting diffusion-based editing with 3 lines of code. International Conference on Learning Repre- sentations (ICLR), 2024. 13

    2024

  26. [34]

    Large-scale text-to-image gener- ation models for visual artists’ creative works

    Hyung-Kwon Ko, Gwanmo Park, Hyeon Jeon, Jaemin Jo, Juho Kim, and Jinwook Seo. Large-scale text-to-image gener- ation models for visual artists’ creative works. InProceedings of the International Conference on Intelligent User Interfaces,

  27. [35]

    Gen- erating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Gen- erating images with multimodal language models. In Pro- ceedings of the Advances in Neural Information Processing Systems, 2024. 3, 5, 17, 27

    2024

  28. [36]

    Artificial general intelligence (AGI) for edu- cation

    Ehsan Latif, Gengchen Mai, Matthew Nyaaba, Xuansheng Wu, Ninghao Liu, Guoyu Lu, Sheng Li, Tianming Liu, and Xiaoming Zhai. Artificial general intelligence (AGI) for edu- cation. arXiv preprint arXiv:2304.12479, 1, 2023. 1

    2023 arXiv

  29. [37]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurenc ¸on, Lucile Saulnier, L´eo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Sid- dharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. In Proceedings of the Adva...

    2024

  30. [38]

    K-sort arena: Efficient and reliable benchmarking for generative models via k-wise human preferences

    Zhikai Li, Xuewen Liu, Dongrong Fu, Jianquan Li, Qingyi Gu, Kurt Keutzer, and Zhen Dong. K-sort arena: Efficient and reliable benchmarking for generative models via k-wise human preferences. arXiv preprint arXiv:2408.14468, 2024. 3, 10

    2024 arXiv

  31. [39]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL Workshop: Text Summarization Branches Out, pages 74–81, 2004. 3

    2004

  32. [40]

    Intelligent grimm - open-ended visual storytelling via latent diffusion models

    Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yan- feng Wang, and Weidi Xie. Intelligent grimm - open-ended visual storytelling via latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6190–6200, 2024. 13

    2024

  33. [41]

    Lumina-mGPT: Illuminate flex- ible photorealistic text-to-image generation with multimodal generative pretraining

    Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mGPT: Illuminate flex- ible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.02657 ,

    arXiv

  34. [42]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the Advances in Neural Information Processing Systems, 2024. 1

    2024

  35. [43]

    Holistic evaluation for interleaved text-and-image generation

    Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, and Lifu Huang. Holistic evaluation for interleaved text-and-image generation. arXiv preprint arXiv:2406.14643, 2024. 1, 2, 3

    2024 arXiv

  36. [44]

    MMBench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 1

    2023 arXiv

  37. [45]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024. 13

    2024 arXiv

  38. [46]

    Embodiedgpt: Vision-language pre-training via embodied chain of thought

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. In Proceedings of the Advances in Neural Information Processing Systems, 2024. 1

    2024

  39. [47]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE, 2008. 14

    2008

  40. [48]

    Hello GPT-4o

    OpenAI. Hello GPT-4o. https : / / openai . com / index/hello-gpt-4o/ , 2024. Accessed: 2024-05-26. 4, 5, 17, 27

    2024

  41. [49]

    BLEU: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the annual meeting of the Association for Computational Linguistics, 2002. 3

    2002

  42. [50]

    Doclaynet: A large human- annotated dataset for document-layout segmentation

    Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter W J Staar. Doclaynet: A large human- annotated dataset for document-layout segmentation. page 3743–3751, 2022. 13

    2022

  43. [51]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In Proceedings of the International Confer- ence on Learning Representations, 2024. 4

    2024

  44. [52]

    Hierarchical text-conditional image genera- tion with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image genera- tion with clip latents. arXiv preprint arXiv:2204.06125, 2022. 3

    2022 arXiv

  45. [53]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Bision and Pattern Recognition, 2022. 3

    2022

  46. [54]

    Improved techniques for training GANs

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Proceedings of the Advances in Neural Information Processing Systems, 2016. 3

    2016

  47. [55]

    Sfgram: a dataset containing thousands of scienc-fiction books and novels

    Nils Schaetti. Sfgram: a dataset containing thousands of scienc-fiction books and novels. https://github.com/ nschaetti/EchoTorch, 2018. 13

    2018

  48. [56]

    Building bridge across the time: Disruption and restoration of murals in the wild

    Huiyang Shao, Qianqian Xu, Peisong Wen, Peifeng Gao, Zhiyong Yang, and Qingming Huang. Building bridge across the time: Disruption and restoration of murals in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20259–20269, 2023. 3

    2023

  49. [57]

    Reverse-engineering information presentations: Recovering hierarchical grouping from layouts of visual ele- ments

    Danqing Shi, Weiwei Cui, Danqing Huang, Haidong Zhang, and Nan Cao. Reverse-engineering information presentations: Recovering hierarchical grouping from layouts of visual ele- ments. Visual Intelligence, 1(1):9, 2023. 13

    2023

  50. [58]

    Tvsum: Summarizing web videos using titles

    Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5179–5187, 2015. 14

    2015

  51. [59]

    How can LLMs transform the robotic design process? Nature Machine Intelligence, pages 1–4, 2023

    Francesco Stella, Cosimo Della Santina, and Josie Hughes. How can LLMs transform the robotic design process? Nature Machine Intelligence, pages 1–4, 2023. 1

    2023

  52. [60]

    Generative multimodal models are in- context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in- context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page...

    2024

  53. [61]

    Evalalign: Supervised fine-tuning multimodal llms with human-aligned data for evaluating text- to-image models

    Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Supervised fine-tuning multimodal llms with human-aligned data for evaluating text- to-image models. CoRR, 2024. 1

    2024

  54. [62]

    CoDi-2: In-context interleaved and interactive any-to-any generation

    Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chen- guang Zhu, and Mohit Bansal. CoDi-2: In-context interleaved and interactive any-to-any generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27425–27434, 2024. 1

    2024

  55. [63]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 1, 3, 27

    2024 arXiv

  56. [64]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1, 5, 17, 27

    2023 arXiv

  57. [65]

    InternLM: A multilingual language model with progressively enhanced capabilities, 2023

    InternLM Team. InternLM: A multilingual language model with progressively enhanced capabilities, 2023. 1

    2023

  58. [66]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Li- wei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In Proceedings of the Advances in Neural Information Processing Systems, 2024. 3

    2024

  59. [67]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  60. [68]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-ucsd birds. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 14

    2011

  61. [69]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-VL: Enhancing vision-language model’s ...

    2024 arXiv

  62. [70]

    A hybrid bandit model with visual priors for creative ranking in display advertising

    Shiyao Wang, Qi Liu, Tiezheng Ge, Defu Lian, and Zhiqiang Zhang. A hybrid bandit model with visual priors for creative ranking in display advertising. In Proceedings of the Web Conference 2021, pages 2324–2334, 2021. 13

    2021

  63. [71]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 1, 3, 17, 27

    2024 arXiv

  64. [72]

    PandaLM: An automatic evalua- tion benchmark for llm instruction tuning optimization

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, et al. PandaLM: An automatic evalua- tion benchmark for llm instruction tuning optimization. In Proceedings of the International Conference on Learning...

    2024

  65. [73]

    Argoverse 2: Next generation datasets for self-driving perception and fore- casting

    Benjamin Wilson, William Qi, Tanmay Agarwal, John Lam- bert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Rat- nesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception an...

    2021

  66. [74]

    Towards open-ended visual quality comparison

    Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, et al. Towards open-ended visual quality comparison. arXiv preprint arXiv:2402.16641, 2024. 3

    2024 arXiv

  67. [75]

    NExT-GPT: Any-to-Any Multimodal LLM

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NExT-GPT: Any-to-Any Multimodal LLM. In Pro- ceedings of the International Conference on Machine Learn- ing, 2024. 5, 17, 27

    2024

  68. [76]

    Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation

    Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Liang Pan Jiawei Ren, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In IEEE/CVF Conference on Computer V...

    2023

  69. [77]

    VILA-U: a unified foundation model inte- grating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. VILA-U: a unified foundation model inte- grating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 17, 27

    2024 arXiv

  70. [78]

    MMIE: Massive multimodal interleaved comprehension benchmark for large vision-language models

    Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, et al. MMIE: Massive multimodal interleaved comprehension benchmark for large vision-language models. arXiv preprint arXiv:2410.10139, 2024. 1

    arXiv

  71. [79]

    Show-o: One single transformer to unify multimodal understanding and genera- tion

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and genera- tion. arXiv preprint arXiv:2408.12528, 2024. 1, 5, 17, 27

    2024 arXiv

  72. [80]

    Architectural style classification using multinomial latent logistic regression

    Zhe Xu, Dacheng Tao, Ya Zhang, Junjie Wu, and Ah Chung Tsoi. Architectural style classification using multinomial latent logistic regression. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 600–61...

    2014

  73. [81]

    Seed-story: Multimodal long story generation with large language model

    Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Seed-story: Multimodal long story generation with large language model. arXiv preprint arXiv:2407.08683, 2024. 3, 13

    2024 arXiv

  74. [82]

    Inves- tigating the effectiveness of data augmentation from similarity and diversity: An empirical study

    Suorong Yang, Suhan Guo, Jian Zhao, and Furao Shen. Inves- tigating the effectiveness of data augmentation from similarity and diversity: An empirical study. Pattern Recognition, 148: 110204, 2024. 3

    2024

  75. [83]

    Visual goal-step inference using wikihow

    Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison-Burch. Visual goal-step inference using wikihow. arXiv preprint arXiv:2104.05845, 2021. 1

    2021 arXiv

  76. [84]

    The dawn of LMMs: Preliminary explorations with GPT-4V (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of LMMs: Preliminary explorations with GPT-4V (ision). arXiv preprint arXiv:2309.17421, 9(1):1, 2023. 1

    2023 arXiv

  77. [85]

    MMT-Bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. MMT-Bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006, 2024. 1

    2024 arXiv

  78. [86]

    MMMU: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  79. [87]

    InternLM-XComposer: A vision-language large model for advanced text-image compre- hension and composition

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. InternLM-XComposer: A vision-language large model for advanced text-image compre- hension and composition. arXiv preprint arXiv:2309.15112,

    arXiv

  80. [88]

    InternLM-XComposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

  81. [89]

    GPT-4V (ision) as a general- ist evaluator for vision-language tasks

    Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. GPT-4V (ision) as a general- ist evaluator for vision-language tasks. arXiv preprint arXiv:2311.01361, 2023. 3

    2023 arXiv

  82. [90]

    Benchmarking multi-image understanding in vi- sion and language models: Perception, knowledge, reasoning, and multi-hop reasoning

    Bingchen Zhao, Yongshuo Zong, Letian Zhang, and Timothy Hospedales. Benchmarking multi-image understanding in vi- sion and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv preprint, 2024. 14

    2024

  83. [91]

    A stitch in time saves nine: Small vlm is a precise guidance for accel- erating large vlms

    Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, and Yang You. A stitch in time saves nine: Small vlm is a precise guidance for accel- erating large vlms. arXiv preprint arXiv:2412.03324, 2024. 1

    2024 arXiv

  84. [92]

    MiniGPT- 5: Interleaved vision-and-language generation via generative vokens

    Kaizhi Zheng, Xuehai He, and Xin Eric Wang. MiniGPT- 5: Interleaved vision-and-language generation via generative vokens. arXiv preprint arXiv:2310.02239, 2023. 3, 5, 17, 27

    2023

  85. [93]

    Judging LLM-as-a-judge with MT-Bench and Chatbot Arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Proceedings of the Advances in Neural Information Processing Systems, 2023. 3, 5, 10

    2023

  86. [94]

    Pub- laynet: largest dataset ever for document layout analysis

    Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Pub- laynet: largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1015–1022. IEEE, 2019. 13

    2019

  87. [95]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Bar- riuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017. 14

    2017

  88. [96]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michi- hiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 1

    2024 arXiv

  89. [97]

    Detection and tracking meet drones challenge

    Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. Detection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7380–7399, 2021. 15 Figure 18. Examples of 23 Meta-Topics (bold font) and ...

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.