Pith. sign in

REVIEW 3 major objections 5 minor 11 cited by

VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VARGPT is the first unified model to support next-token visual understanding and next-scale visual generation in one autoregressive framework.

desk verdict A clearly described unified MLLM whose comprehension-over-LLaVA-1.5 claim is undercut by its own ablation data; still worth a serious referee. read the letter →

arxiv 2501.12327 v1 pith:P5UCNRH4 submitted 2025-01-21 cs.CV

Xianwei Zhuang , Yuxin Xie , Yufan Deng , Liming Liang , Jinghan Ru , Yuguo Yin , Yuexian Zou This is my paper · ORCID
classification cs.CV
keywords VARGPTvisualautoregressivegenerationnext-scalepredictionmultimodallargelanguagemodelunderstandingtext-to-imageunifiedinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VARGPT claims that one autoregressive multimodal model can do both visual understanding and image generation by using two different prediction rules: predict the next text token when answering questions, and predict the next visual scale when drawing an image. The paper builds this by extending the LLaVA-1.5 architecture with a 2-billion-parameter visual decoder and two visual feature projectors, then training the combined model in three stages on roughly 3.86 million instruction samples. On eleven visual-understanding benchmarks the model beats LLaVA-1.5 and several comparable unified models, while also producing 256x256 images from text instructions in the same conversation. A sympathetic reader would take the central claim to be that a single weight set can host both capabilities without either degrading the other.

What carries the argument

The central mechanism is dual-paradigm autoregression inside one Transformer. For understanding, the model predicts the next text token under a standard causal mask. For generation, the special <image_gen_start> token switches on a second autoregressive loop where the LLM's hidden states pass through a generation projector into a 2B visual decoder that predicts next-scale tokens under a block-causal attention mask; a multi-scale VQ-VAE then decodes the tokens into an image. The special tokens used to mark generation positions and the staged training schedule are the load-bearing connectors that allow one model to answer a question and then draw a picture in a single mixed-modal response.

What would settle it

Evaluate instruction-to-image generation on prompts built from a different distribution, such as COCO captions or ImageNet categories held out from training, while keeping VARGPT frozen; if FID and CLIP scores drop far below the ImageNet-derived results, the reported instruction-following is mostly in-distribution recall.

Watch

Extended reading notes

Core claim

The paper's central claim is that VARGPT is the first unified model to support predicting the next token for understanding and predicting the next scale for generation, and that this single model can outperform LLaVA-1.5-class multimodal LLMs on comprehension while adding autoregressive text-to-image generation. Concretely, the LLM emits text tokens normally; once it predicts the special <image_gen_start> token, the image-related features are projected into a separate 2B visual decoder that predicts multi-scale image tokens in block-causal order, and a multi-scale VAE decoder turns them into a 256x256 image. The three-stage training recipe, consisting of pretraining the generation projectors, mixed instruction tuning for understanding plus a small generation subset, and then generation-focused instruction tuning, is what lets the two paradigms coexist. Reported results include higher scores than LLaVA-1.5 on benchmarks such as MMBench, SEED-Bench, MME, POPE, GQA, TextVQA, VQAv2, and ScienceQA, with an FID of 12.6 and a CLIP score of 27.4 on the constructed instruction-to-image evaluation.

Load-bearing premise

The load-bearing premise is that the generation benchmarks measure generalization rather than memorization, because the test instructions and the FID reference both come from ImageNet-1K, the same dataset the model was trained on, so high scores could instead reflect recall of training images and captions.

Editorial extensions

If this is right

  • A single model can hold a conversation, answer visual questions, and then generate an image in the same response, without routing generation to an external diffusion model.
  • At the 7B-plus-2B scale, unified understanding and generation does not have to come at a comprehension cost; VARGPT reports higher scores than LLaVA-1.5 on the understanding benchmarks it evaluates.
  • The three-stage recipe and the 5K generation samples mixed into stage-2 instruction tuning teach the model when to emit generation tokens, so task routing emerges from instruction following rather than from a separate controller.
  • Generation quality is tied to data: the paper's stated limitation is that ImageNet-only training and 256x256 resolution leave a quality gap to large diffusion models, so scaling data and resolution is the explicit next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested implication is that next-scale generation may scale with data the way text autoregression does, potentially letting a single unified model approach diffusion-level image quality while retaining understanding.
  • A sharper test than the paper runs would evaluate instruction-to-image generation on a held-out distribution, since the test prompts are built from ImageNet-1K captions and FID is measured against ImageNet-1K, which the model saw during training.
  • The decoupling of the visual decoder from the LLM suggests a broader design principle: understanding and generation can share a language model only when generation has its own parameter space, and future unified models may vary how much of the generation burden the LLM itself carries.
  • A practical consequence the authors do not spell out is that mixed-modal conversations in one model remove the need to coordinate separate question-answering and image-generation systems, which could simplify deployment and reduce latency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. VARGPT proposes a single autoregressive multimodal model that performs visual understanding through next-token prediction and visual generation through next-scale prediction, extending the LLaVA-1.5 architecture with a 2B visual decoder and two generation projectors. The model is trained in three stages on a curated mixture of LLaVA-1.5, LLaVA-OneVision, and ImageNet-based instruction data, and it can output mixed text-and-image responses in a single conversation. The paper reports comprehension scores on eleven benchmarks and generation metrics (FID, CLIP score) on a 50,000-instruction ImageNet evaluation set, together with ablations of the training stages and data composition.

Significance. If the central claims hold, the architecture is a useful step toward unified multimodal models: it demonstrates that a next-scale visual generator can be attached to a next-token understanding LLM in a single model, and the qualitative mixed-modal outputs in Figures 2, 9, 13, and 14 show a genuine capability. The paper also provides a clearly described three-stage training recipe and ablation studies on both generation stages and understanding data, which are valuable for follow-up work. However, the headline comprehension claim is confounded by the addition of LLaVA-OneVision data, and the generation evaluation is performed on the training distribution without comparison to existing unified models or text-to-image baselines. The significance of the numerical results is therefore not yet established; the architectural contribution is plausible but requires stronger controlled evidence.

major comments (3)
  1. [Section 5.1, Table 5, Section 4.2] The claimed comprehension superiority over LLaVA-1.5 is not supported by a controlled comparison. The stage-2 training mixes 665K LLaVA-1.5 samples with 508K LLaVA-OneVision samples, and the ablation in Table 5 (row '- LLaVA-OneVision-502K') shows that after removing the OneVision portion the model attains MMMU 33.56 and MME 1403.5, below the reported LLaVA-1.5-hf scores of 35.24 and 1480.1. Since the abstract states that 'Despite its LLAVA-based architecture... significantly outperforms LLaVA-1.5,' the improvement is likely attributable to the addition of a newer, larger data mixture rather than to the next-token/next-scale architecture. The paper needs a matched comparison: either train LLaVA-1.5 on the same combined data, or train VARGPT only on the LLaVA-1.5 portion, and report both understanding and generation metrics.
  2. [Section 5.1, Evaluation on Instruction-to-image Task] The generation evaluation does not measure generalization to novel instructions. The 50,000 test instructions are constructed from ImageNet-1K captions with 50 samples per category, the model is trained on ImageNet-1K, and the FID is computed against the ImageNet-1K dataset itself; CLIP score is computed between instructions and images drawn from the same caption distribution used to build the training data. These numbers cannot distinguish genuine instruction following from memorization of the training distribution. The authors should evaluate on held-out categories, on captions from a different source, or with human raters, and should compare against at least one text-to-image or unified-model baseline under identical conditions.
  3. [Section 5, Tables 1 and 4; Section 6, Limitation] The generation quality claim is never benchmarked against the unified models listed in Table 1 (Chameleon, SEEDLLaMA, Show-o, VILA-U, Liquid) or against the VAR backbone used to initialize the visual decoder. Table 4 only ablates the training stages; without a comparison model, the reader cannot judge whether the reported FID 12.6 / CLIP 27.4 is strong or weak. The limitation section asserts that VARGPT's generation lags behind diffusion models and other unified models, but no quantitative comparison is provided to support this. Adding a standard text-to-image benchmark with matched baselines would substantiate the 'instruction-to-image' claim.
minor comments (5)
  1. [Abstract and Section 3.2] There are several typos and grammatical errors: 'multimodel understanding' should be 'multimodal understanding' in the abstract, and 'The unified training strategy are designed' should be 'is designed'.
  2. [Table 5 vs Section 4.2 and Figure 8b] The ablation row '- LLaVA-OneVision-502K' is inconsistent with the data description, which states that 508K samples were sampled from LLaVA-OneVision; please reconcile the numbers or clarify what exactly was removed.
  3. [Table 3] The legend 'We gray out the model has trained on the dataset' is ungrammatical and unclear; please specify which models were trained on which evaluation datasets and what the gray highlighting indicates.
  4. [Appendix 7.6, Eq. (3)] The CFG equation uses the same symbol R_t on both sides of the final line; please write the combination with distinct notation, e.g., \hat{R}_t = (1+\lambda) R_t^{cond} - \lambda R_t^{uncond}, to avoid confusion between the conditional distribution, the unconditional distribution, and the combined result.
  5. [Section 5, Details] Sampling hyperparameters (top-k=900, top-p=0.95, CFG=1.5) are reported without sensitivity analysis or a citation to the source of these defaults; a short sensitivity table or a reference to the VAR settings would make the results more reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VARGPT is an empirical system paper; the in-distribution generation metric and the LLaVA-OneVision data confound are evaluation-validity concerns, not circular derivations.

full rationale

VARGPT is an empirical systems paper with no derivation chain whose output is defined as its input. Equations (1) and (2) are operational definitions of next-token understanding and next-scale generation, and Eq. (3) is the CFG combination rule adopted from prior work; none of these equations reduces a result to a fitted input. The model is trained on external datasets (LLaVA-1.5, LLaVA-OneVision, ImageNet-Instruct) and then evaluated on standard zero-shot benchmarks, so no fitted parameter is relabeled as a prediction. The cited components (LLaVA-1.5, VAR, the multi-scale tokenizer) are externally validated and are not self-citations by these authors. Two passages raise validity concerns but not circularity. First, Section 5.1 builds the 50,000-instruction generation test from ImageNet-1K captions and computes FID against ImageNet-1K, the same distribution used for Stage-1 and Stage-3 training; this can reward memorization, but a good FID does not follow by construction from the training objective, and the paper discloses the ImageNet-only limitation. Second, the claimed comprehension advantage over LLaVA-1.5 is confounded by adding 508K LLaVA-OneVision samples in Stage-2: Table 5 shows that removing those samples drops MMMU to 33.56 and MME to 1403.5, below the reported LLaVA-1.5-hf scores of 35.24 and 1480.1. That weakens the attribution of the gain to the architecture, but it is an experimental confound, not a circular step. No self-definitional, fitted-input-called-prediction, self-citation-chain, uniqueness-imported-from-authors, or ansatz-smuggled-via-citation pattern is present, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The paper's central claims rest on a handful of hand-set hyperparameters (CFG scale, sampling thresholds, stage durations, data mix), on the transferability of pretrained LLaVA and VAR components, and on the quality of LLM-synthesized instruction data. The special tokens are invented interface elements with no external evidence beyond the model's own behavior.

free parameters (5)
  • CFG scale lambda = 1.5
    Hand-set hyperparameter controlling the strength of classifier-free guidance during image generation; directly affects FID and CLIP scores in Table 4.
  • Top-k sampling = 900
    Hand-set decoding parameter for image token sampling; influences generation quality and diversity.
  • Top-p sampling = 0.95
    Hand-set nucleus sampling threshold for image generation; influences output distribution.
  • Stage-3 epoch count = 12
    Training duration for generation SFT, chosen by hand; longer training would change final generation performance.
  • Stage-2 generation sample count = 5K
    Number of ImageNet-Instruct samples mixed into stage-2 understanding tuning; chosen by hand to teach special-token emission without hurting understanding.
assumptions (4)
  • domain assumption Pretrained LLaVA-1.5-7B-hf components retain understanding capability when spliced with a generation pathway.
    The paper initializes the LLM, visual encoder, and understanding projector from LLaVA-1.5 and assumes they can be fine-tuned alongside new image-generation components without destructive conflict; Section 3.1 and 5 describe this initialization.
  • domain assumption The pretrained VAR-d30 visual decoder and multi-scale VQ-VAE tokenizer can decode image features produced from LLaMA hidden states after projector training.
    The visual decoder is initialized from VAR-d30 and the tokenizer comes from VAR; the paper assumes these transfer when driven by features from the LLM through randomly initialized projectors; Section 3.1 and 7.1.
  • domain assumption DeepSeek-V3-generated instruction data adequately represents diverse user requests for image generation.
    The 130K and 1270K instruction datasets are built by prompting DeepSeek-V3 with seed templates; there is no human quality filtering or distributional validation; Section 4.1 and Appendix 8.
  • domain assumption Benchmark numbers from other papers are comparable and not affected by training-data overlap with VARGPT's stage-2 datasets.
    Tables 1 and 3 compare VARGPT against published baselines, but VARGPT trains on LLaVA-OneVision and LLaVA-1.5 data while the baselines use different data; contamination is not systematically analyzed.
invented entities (2)
  • <image_gen> special token
    purpose: Placeholder marking the position where generated image tokens should be produced in mixed-modal output.
    Internal model marker with no external falsifiable handle; its functionality is defined only within VARGPT.
  • <image_gen_start> and <image_gen_end> special tokens
    purpose: Delimit the beginning and end of the generated image token sequence.
    Internal control tokens used to switch the model from text generation to scale-wise image generation and back.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model." pith.science (2026). https://pith.science/paper/P5UCNRH4

@misc{pith2026250112327,
  author       = {Pith},
  title        = {Pith review of: VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P5UCNRH4}},
  note         = {Machine review of arXiv:2501.12327}
}
read the original abstract

We present VARGPT, a novel multimodal large language model (MLLM) that unifies visual understanding and generation within a single autoregressive framework. VARGPT employs a next-token prediction paradigm for visual understanding and a next-scale prediction paradigm for visual autoregressive generation. VARGPT innovatively extends the LLaVA architecture, achieving efficient scale-wise autoregressive visual generation within MLLMs while seamlessly accommodating mixed-modal input and output within a single model framework. Our VARGPT undergoes a three-stage unified training process on specially curated datasets, comprising a pre-training phase and two mixed visual instruction-tuning phases. The unified training strategy are designed to achieve alignment between visual and textual features, enhance instruction following for both understanding and generation, and improve visual generation quality, respectively. Despite its LLAVA-based architecture for multimodel understanding, VARGPT significantly outperforms LLaVA-1.5 across various vision-centric benchmarks, such as visual question-answering and reasoning tasks. Notably, VARGPT naturally supports capabilities in autoregressive visual generation and instruction-to-image synthesis, showcasing its versatility in both visual understanding and generation tasks. Project page is at: \url{https://vargpt-1.github.io/}

Figures

Figures reproduced from arXiv: 2501.12327 by the authors.

Figure 1
Figure 1. A comparative analysis of various MLLMs across multi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Some generated 256×256 samples by VARGPT trained on ImageNet [11]. VARGPT supports text-and-image instructions from user and outputs both text-and-image mixed modal data simultaneously. ure 3). Recent works [18, 90, 91] have attempted to assem￾ble models from these two distinct domains (e.g., LLMs and DDPMs) to form a unified system capable of process￾ing multimodal understanding and generation (as shown in [PITH_F… view at source ↗
Figure 3
Figure 3. Comparison of different model architectures, where, ‘AR’ denotes autoregressive, while ‘VAR’ signifies visual autoregressive. We [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: The illustration of the proposed VARGPT framework, which consists of (1) a large language model, visual encoder and a [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: The three training stages of the VARGPT, including stage-1 pretraining, stage-2 and stage-3 instruction fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: A sample from the ImageNet-Instruct-130K image gen [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The illustration of the proposed image generation instruction-following sample generation, which consists of (a) seed generation: [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: We present the data distribution we constructed and collected, encompassing: (a) the proportional breakdown of data across the [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Some generated 256×256 samples by our VARGPT trained on ImageNet-1K. VARGPT supports user text command input and outputs both text and image modal data simultaneously. (3) Compared with other unified models (e.g., SEEDL￾LaMA [17] and VILA-U [91]) that support generatio…
Figure 10
Figure 10. Figure 10: Loss curves of training stages 2 and 3 for VARGPT. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Illustration of the proposed unified prompting format. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Illustration of the proposed unified visual generation instruction fine-tuning dataset construction. [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Some generated 256×256 samples by VARGPT trained on ImageNet [11]. VARGPT supports user text command input and outputs both text and image modal data simultaneously. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Some generated 256×256 samples by VARGPT trained on ImageNet [11]. VARGPT supports user text command input and outputs both text and image modal data simultaneously. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Example of the instruction fine-tuning dataset we collected and constructed. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Demystifying Video Reasoning

    cs.CV 2026-03 conditional novelty 7.0 of 10

    Video diffusion models reason along the denoising trajectory (Chain-of-Steps), not primarily across frames, and this mechanism can be nudged by ensembling latent trajectories.

  2. Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens

    cs.CV 2025-04 conditional novelty 7.0 of 10

    Diffusion timestep tokens give large language models a recursive visual language that improves unified multimodal comprehension and generation relative to spatial patch tokens.

  3. SynVAR: Synergizing Spatial and Semantic Alignment in Visual Autoregressive Model

    cs.CV 2026-08 conditional novelty 6.0 of 10

    SynVAR improves compositional generation of VAR models by injecting spatial priors, constraining early self-attention, and enhancing high-frequency details.

  4. UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An MLLM-conditioned next-scale VAR decoder handles 15+ unified visual generation tasks with competitive quality and substantially lower latency than diffusion baselines.

  5. Unified Audio Intelligence Without Regressing on Text Intelligence

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A unified 30B MoE audio-text LLM achieves state-of-the-art audio understanding, generation, and speech tasks while preserving text reasoning comparable to its text-only backbone.

  6. Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    1D binary image latents reduce a 1024x1024 image to 128 discrete tokens and support text-to-image generation with diffusion and autoregressive models.

  7. ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependencies

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Introduces a benchmark for chain-dependent image editing instructions plus a region-aware consistency metric, and shows a chain-of-thought prompt improves a Gemini-based editor.

  8. Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A dual-level attention intervention that boosts salient visual-token attention and suppresses text/system attention during decoding reduces hallucination rates in LLaVA, MiniGPT-4, and mPLUG-Owl2 on POPE and CHAIR.

  9. Do we really have to filter out random noise in pre-training data for language models?

    cs.CL 2025-02 conditional novelty 6.0 of 10

    Random noise in pre-training data has a surprisingly small effect on language model next-token loss, but can still hurt downstream tasks; a new local gradient matching loss partially counteracts this.

  10. Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A new benchmark and agent framework for complex text-to-image generation, with an unvalidated AI-judge evaluation and claims that the agent outperforms GPT-4o on the authors' own benchmark.

  11. Open Set Domain Adaptation with Vision-language models via Gradient-aware Separation

    cs.LG 2025-05 reject novelty 5.0 of 10

    A gradient-norm heuristic applied to CLIP prompt weights is proposed to detect unknown classes in open-set domain adaptation, with Office-Home experiments against CLIP and CoOp.

Reference graph

Works this paper leans on

104 extracted references · 49 canonical work pages · cited by 11 Pith papers

  1. [1]

    Albergo and Eric Vanden-Eijnden

    Michael S. Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants, 2023. 3

    2023

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966,

    arXiv

  3. [3]

    messages

    Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and 15 [User]: Please design a crafted drawing of a lion fish swimming in the ocean. [VARGPT]: Here's a picture of a lion fish swimming in the ocean. <image_gen> [User]: Please render a designed image ofa ferret is standing in the snow. [VARGPT]: I have c...

    2023

  4. [4]

    Analytic- dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models, 2022

    Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic- dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models, 2022. 3

    2022

  5. [5]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. ArXiv, abs/2310.09478,

    arXiv

  6. [6]

    Halc: Object halluci- nation reduction via adaptive focal-contrast decoding.ArXiv, abs/2403.00425, 2024

    Zhaorun Chen, Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object halluci- nation reduction via adaptive focal-contrast decoding.ArXiv, abs/2403.00425, 2024. 14

    arXiv 2024

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Visual input example: User If you were a photographer looking to capture this lo- c...

  8. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6,

    2023

Show all 104 references
  1. [9]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  2. [10]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024. 6, 14

    2024

  3. [11]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. 2, 6, 14, 16, 17

    2009

  4. [12]

    Cogview: Mastering text-to- image generation via transformers, 2021

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to- image generation via transformers, 2021. 3

    2021

  5. [13]

    Dreamllm: Synergistic multimodal compre- hension and creation, 2024

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Dreamllm: Synergistic multimodal compre- hension and creation, 2024. 4

    2024

  6. [14]

    Taming transformers for high-resolution image synthesis, 2021

    Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2021. 3, 4, 8, 13

    2021

  7. [15]

    Fluid: Scaling autoregressive text-to-image generative models with continuous tokens

    Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13863, 2024. 3

    arXiv

  8. [16]

    Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024. 9

    2024

  9. [17]

    Making llama see and draw with seed tokenizer, 2023

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer, 2023. 4, 9, 10

    2023

  10. [18]

    Seed-x: Multimodal models with unified multi-granularity compre- hension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity compre- hension and generation. arXiv preprint arXiv:2404.14396,

    arXiv

  11. [19]

    Making the v in vqa matter: Ele- vating the role of image understanding in visual question answering, 2017

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Ele- vating the role of image understanding in visual question answering, 2017. 9

    2017

  12. [20]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  13. [21]

    Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

    Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people, 2018. 9

    2018

  14. [22]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024

    Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024. 1, 3

    2024

  15. [23]

    Scaling laws for autoregressive generative modeling

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. 1, 3

    2010 arXiv

  16. [24]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 1

    2020

  17. [25]

    Denoising diffu- sion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models, 2020. 3

    2020

  18. [26]

    Fleet, Mohammad Norouzi, and Tim Salimans

    Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation, 2021. 3

    2021

  19. [27]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 7

    2019

  20. [28]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. 9

    2019

  21. [29]

    Scaling laws for 20 neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for 20 neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  22. [30]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language pro- cessing (EMNLP), pages 787–798, 2014. 7

    2014

  23. [31]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of compute...

    2017

  24. [32]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and vi...

    1956

  25. [33]

    Black Forest Labs. Flux. https://github.com/ black-forest-labs/flux, 2023. 12

    2023

  26. [34]

    Datasets: imagenet-1k-vl-enriched

    Visual Layer. Datasets: imagenet-1k-vl-enriched. https: //huggingface.co/datasets/visual-layer/ imagenet-1k-vl-enriched , 2024. 6

    2024

  27. [35]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015. 3

    2015

  28. [36]

    Autoregressive image generation using residual quantization, 2022

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization, 2022. 3

    2022

  29. [37]

    Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shi- jian Lu, Chunyan Miao, and Li Bing. Mitigating object hallucinations in large vision-language models through vi- sual contrastive decoding. ArXiv, abs/2311.16922, 2023. 14

    2023 arXiv

  30. [38]

    Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension, 2023. 9

    2023

  31. [39]

    Llava-next: What else influences visual instruction tuning beyond data?, 2024

    Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influences visual instruction tuning beyond data?, 2024. 3, 7

    2024

  32. [40]

    Llava-next: Stronger llms supercharge multimodal capabilities in the wild, 2024

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chun- yuan Li. Llava-next: Stronger llms supercharge multimodal capabilities in the wild, 2024. 3, 7

    2024

  33. [41]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. 7

    2024

  34. [42]

    Llava-next: Tackling multi-image, video, and 3d in large multimodal models,

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next: Tackling multi-image, video, and 3d in large multimodal models,

  35. [43]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Interna- tional conference on machine learning, pages 19730–19742. PMLR, 2023. 3, 14

    2023

  36. [44]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 6

    2023

  37. [45]

    Evaluating object hallucination in large vision- language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Singapore, 2023. Association for C...

    2023

  38. [46]

    Dual diffusion for unified image generation and understanding, 2024

    Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding, 2024. 2, 4

    2024

  39. [47]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ArXiv, abs/2310.03744, 2023. 14

    2023 arXiv

  40. [48]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023. 1, 9

    2023 arXiv

  41. [49]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 2, 3, 7

    2023

  42. [50]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 3, 4

    2024

  43. [51]

    Improved baselines with visual instruction tuning, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. 7, 8

    2024

  44. [52]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 3, 8

    2024

  45. [53]

    World model on million-length video and language with ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint, 2024. 2, 4

    2024

  46. [54]

    Mmbench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. 9

    2024

  47. [55]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps, 2022. 3

    2022

  48. [56]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. 9

    2022

  49. [57]

    Unified multi-modal latent diffusion for joint subject and text conditional image generation, 2023

    Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation, 2023. 3

    2023

  50. [58]

    Generation and comprehension of unambiguous object descriptions

    Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016. 7

    2016

  51. [59]

    Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019. 9 21

    2019

  52. [60]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 7

    2019

  53. [61]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–

    2019

  54. [62]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR,

  55. [63]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. 3

    2023

  56. [64]

    Du, Zehuan Yuan, and Xin- glong Wu

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K. Du, Zehuan Yuan, and Xin- glong Wu. Tokenflow: Unified image tokenizer for multi- modal understanding and generation, 2024. 2

    2024

  57. [65]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. 3

    2018

  58. [66]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 4

    2021

  59. [67]

    Zero-shot text-to-image generation, 2021

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. 3

    2021

  60. [68]

    Hierarchical text-conditional image genera- tion with clip latents, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image genera- tion with clip latents, 2022. 5, 13

    2022

  61. [69]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. 12

    2021

  62. [70]

    A-okvqa: A bench- mark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A bench- mark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146–

  63. [71]

    https://sharegpt.com/, 2023

    ShareGPT. https://sharegpt.com/, 2023. 7

    2023

  64. [72]

    Textcaps: a dataset for image captioning with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 742–758. Springer,

    2020

  65. [73]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019. 9

    2019

  66. [74]

    Denois- ing diffusion implicit models, 2022

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models, 2022. 3

    2022

  67. [75]

    Generative modeling by estimating gradients of the data distribution, 2020

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020. 3

    2020

  68. [76]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 1, 3

    2024 arXiv

  69. [77]

    Autoregressive model beats diffusion: Llama for scalable image generation, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation, 2024. 3

    2024

  70. [78]

    Emu: Generative pretraining in multimodality, 2024

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality, 2024. 4

    2024

  71. [79]

    Hart: Efficient visual generation with hybrid autoregressive transformer

    Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint, 2024. 1, 3

    2024

  72. [80]

    Any-to-any generation via composable diffusion, 2023

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion, 2023. 4

    2023

  73. [81]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 2

    2024 arXiv

  74. [82]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024. 4, 9

    2024

  75. [83]

    Gemini: A family of highly capable multi- modal models, 2024

    Gemini Team. Gemini: A family of highly capable multi- modal models, 2024. 4

    2024

  76. [84]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024. 1, 3, 4, 5, 8, 13

    2024

  77. [85]

    Lacroix, Baptiste R., Naman Goyal, Eric Hambro, Faisal Azhar, et al

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, T. Lacroix, Baptiste R., Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 3

    2023 arXiv

  78. [86]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1, 3

    2023 arXiv

  79. [87]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 1, 3

    2024 arXiv

  80. [88]

    Janus: Decoupling visual encoding for unified multimodal understanding and genera- tion, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and genera- tion, 2024. 2, 4

    2024

  81. [89]

    Liq- uid: Language models are scalable multi-modal generators

    Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Heng- shuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liq- uid: Language models are scalable multi-modal generators. arXiv preprint arXiv:2412.04332, 2024. 2, 4

    2024 arXiv

  82. [90]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. 2, 4

    2023 arXiv

  83. [91]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie 22 Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 2, 9, 10

    2024 arXiv

  84. [92]

    Show-o: One single transformer to unify multimodal understanding and generation, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation, 2024. 4

    2024

  85. [93]

    X-vila: Cross-modality align- ment for large language model, 2024

    Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, and Hongxu Yin. X-vila: Cross-modality align- ment for large language model, 2024. 4

    2024

  86. [94]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, An- wen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. ArXiv, abs/2311.04257, 2023. 1, 9

    2023 arXiv

  87. [95]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  88. [96]

    Woodpecker: Hallucination correction for multi- modal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xingguo Sun, and En- hong Chen. Woodpecker: Hallucination correction for multi- modal large language models. ArXiv, abs/2310.16045, 2023. 14

    arXiv

  89. [97]

    Scaling autoregressive models for content-rich text-to-image generation, 2022

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content...

    2022

  90. [98]

    Mmmu: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert agi, 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Ren- liang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A...

    2024

  91. [99]

    Lmms- eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms- eval: Reality check on the evaluation of large multimodal models, 2024. 9

    2024

  92. [100]

    Var-clip: Text-to-image generator with visual auto-regressive modeling, 2024

    Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text-to-image generator with visual auto-regressive modeling, 2024. 5, 13

    2024

  93. [101]

    Llava- next: A strong zero-shot video understanding model, 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava- next: A strong zero-shot video understanding model, 2024. 3, 8

    2024

  94. [102]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 4

    2023

  95. [103]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 8 23

    2023 arXiv

  96. [2023]

    1, 4, 9 19 Visual input example: User If someone were to recommend a new flavor or topping to the dish, describe the reason for this change and how it might alter the overall taste. LLaV A-v1.5-7BIf someone were to recommend a new flavor or topping to the dish, they might sugg...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.