Pith. sign in

REVIEW 5 major objections 6 minor 12 cited by

X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that reinforcement learning, applied to a discrete autoregressive image model built from a semantic tokenizer and a frozen diffusion decoder, removes the artifacts that have pushed the field toward diffusion hybrids, and…

desk verdict A genuinely new combination—GRPO on discrete AR image tokens with a fixed diffusion decoder—that looks like it works, but the headline text-rendering claims are weakened by reward-evaluation overlap and missing ablations. read the letter →

arxiv 2507.22058 v1 pith:BMGHFCH7 submitted 2025-07-29 cs.CV

Zigang Geng , Yibing Wang , Yeyao Ma , Chen Li , Yongming Rao , Shuyang Gu , Zhao Zhong , Qinglin Lu
show 5 more authors
Han Hu Xiaosong Zhang Linus Di Wang Jie Jiang
This is my paper · ORCID
classification cs.CV
keywords reinforcementlearningdiscreteautoregressiveimagegenerationunifiedmultimodalmodelsemantictokenizerGRPOtext-to-imagelongtextrenderingclassifier-freeguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that reinforcement learning removes the long-standing weaknesses of discrete autoregressive image generation—low fidelity, distorted details, and failure to follow complex instructions—so that a single next-token-prediction model can handle both text and images and beat diffusion-centered unified systems. To show this, the authors build X-Omni, in which a semantic tokenizer (SigLIP-VQ) turns images into discrete tokens, a 7B autoregressive language model predicts those tokens, and a frozen diffusion decoder renders them into pixels. The autoregressive model is then trained with GRPO against a composite reward that scores aesthetics, text-image alignment, and OCR-verified text rendering. If the claim holds, unified open-source image models no longer need diffusion training objectives, and a 7B model can render long in-image text in English and Chinese competitively with specialized commercial systems.

What carries the argument

The load-bearing mechanism is the closed loop: an autoregressive policy samples discrete semantic tokens, a frozen diffusion decoder converts them into pixels, external reward models score the pixels, and GRPO converts group-normalized reward differences into policy-gradient updates. The tokenizer is SigLIP-VQ, which discretizes a frozen SigLIP2-g semantic encoder's outputs through a 16,384-entry vector-quantization codebook into the same kind of token stream as language, and the decoder is a frozen FLUX.1-dev model fed through a linear adapter that injects the semantic token embeddings into its intermediate features. GRPO, or Group Relative Policy Optimization, is the optimizer that avoids a separate critic by normalizing rewards within a group of 16 rollouts per prompt, with a KL penalty toward the reference policy. The composite reward—HPSv2, Unified Reward, Qwen2.5-VL-32B alignment, and GOT-OCR2.0/PaddleOCR accuracy—is what the paper argues carries the aesthetic, alignment, and text-rendering quality gains.

What would settle it

Run a blind human-preference study comparing X-Omni's RL outputs against its own SFT checkpoint with best-of-N sampling on prompts drawn from benchmarks outside the RL training and reward distribution, such as T2I-CompBench or DreamBench; if humans do not prefer the RL outputs, the central claim fails. As a mechanistic check, inject spelling errors into ground-truth text and verify whether the OCR-based reward actually tracks human readability, or whether it saturates once the OCR model can guess the text.

Watch

Extended reading notes

Core claim

The central discovery is that the quality gap between discrete autoregressive image generation and diffusion-based generation is not inherent to discrete tokens; it is a trainable distribution mismatch. The autoregressive model samples semantic tokens in a region of token space that the pixel decoder was not trained to accept, and reinforcement learning pulls the sampler into the decoder's expected distribution while suppressing errors that accumulate over long token sequences. Concretely, the paper reports that after three pre-training stages, supervised fine-tuning, and 200 steps of GRPO, a 7B model reaches state-of-the-art overall on DPG-Bench (87.65), the best OneIG-Bench English text-rendering score among unified and open models (0.901), a Chinese score of 0.895, and the best LongText-Bench Chinese long-text score (0.814), while matching or exceeding unified baselines on image understanding including OCRBench (704). A second reported finding is that the RL-trained model generates high-quality images without classifier-free guidance, unlike earlier autoregressive image models, and that RL exceeds the best-of-N sampling of its own SFT checkpoint, a contrast with typical language-modeling experience. The paper also introduces LongText-Bench, 160 prompts across eight text-rich scenarios, as an evaluation for long-text rendering.

Load-bearing premise

The entire quality gain is driven by external scoring models that judge aesthetics, text-image match, and OCR-readable text, with no published aggregation weights and no analysis of reward hacking, so if those scorers are miscalibrated the claimed improvement may not reflect real human preference or transfer to benchmarks outside their coverage.

Editorial extensions

If this is right

  • Unified discrete next-token prediction becomes a viable base for frontier image generation, removing the need for diffusion objectives inside the main model.
  • Autoregressive image sampling can run without classifier-free guidance, lowering inference cost and indicating that visual and language token generation share one consistent distribution.
  • A 7B open model can render long in-image text in English and Chinese, beating specialized commercial systems on Chinese long-text rendering.
  • Because generation and understanding use the same token stream, multi-turn joint understanding and generation no longer requires re-encoding generated images with a separate encoder.
  • The recipe of GRPO plus a frozen diffusion decoder plus external reward models is the component that closes the token-distribution gap, and it outperforms best-of-N sampling from the SFT model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the operative mechanism is distribution alignment rather than the specific model choices, the same GRPO-plus-frozen-decoder recipe should transfer to other discrete autoregressive image systems; the clean test is applying it to an Emu3- or Janus-Pro-style model without architectural changes.
  • The reported result that RL beats SFT best-of-N—atypical for language—is plausibly because one image yields many reward-bearing tokens, giving group-normalized advantages dense spatial credit; a testable corollary is that the advantage shrinks as tokens per image are compressed.
  • The missing reward weights and absent reward-hacking analysis mark the true risk surface: on dimensions the reward models do not see, such as counting and spatial relations, the method could regress while scores rise, and the relatively weaker GenEval result (0.83) is consistent with such blind spots.
  • The CFG-free finding suggests the reward signal absorbed the role of sampling-time guidance; whether that robustness persists on out-of-distribution or adversarial prompts is an open, testable question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes X-Omni, a unified autoregressive model for image generation and understanding. Images are tokenized with a SigLIP-VQ semantic tokenizer, modeled with a Qwen2.5-7B autoregressive backbone, and decoded through a frozen FLUX diffusion decoder. After large-scale pretraining and supervised fine-tuning, the model is optimized with GRPO using a weighted combination of reward models: HPSv2, Unified Reward, Qwen2.5-VL-32B text-image alignment, and GOT-OCR2.0/PaddleOCR for text rendering. The authors report state-of-the-art results on text-rendering benchmarks including a new LongText-Bench, competitive DPG-Bench and GenEval numbers, comparable image-understanding performance, and a finding that the model does not require classifier-free guidance. The central claim is that reinforcement learning effectively mitigates artifacts of discrete autoregressive generation and makes such models competitive again.

Significance. If validated, the result is significant: it would revive discrete autoregressive image generation, which has largely been replaced by diffusion hybrids, and would provide a clean recipe—semantic tokens plus a fixed diffusion decoder plus RL—for unifying image generation and understanding in one transformer. The paper is also timely in applying GRPO, which is well developed for language models, to image generation. The work includes a large-scale training pipeline and introduces a new long-text benchmark, which is useful if properly validated. The main weakness is that the headline claims rest on reward-model-based evaluation that substantially overlaps with the training reward stack, and the paper does not provide the ablations, reward weights, or human evaluation needed to rule out reward overoptimization. The core idea is promising and the evidence is suggestive, but the central claim is not yet established at the standard the paper states.

major comments (5)
  1. [§3.2.2 and §4.3.1 / Appendix A] The training rewards in §3.2.2 include OCR accuracy rewards (GOT-OCR2.0, PaddleOCR) and a Qwen2.5-VL-32B text-image alignment reward, while the headline text-rendering results are evaluated with OCR-family metrics: the LongText-Bench metric is explicitly Qwen2.5-VL-7B OCR accuracy (Appendix A), and the OneIG-Bench text-rendering score in Table 1 is composed of OCR-derived Edit Distance, Completion Rate, and Word Accuracy. The paper reports no human evaluation, no held-out OCR evaluator, no reward aggregation weights, and no reward-hacking analysis. With G=16 rollouts over 200 steps on 180K prompts, GRPO is exactly the regime in which a fixed proxy reward can be overoptimized, so the policy may produce token sequences that the OCR/VLM reward models read as correct even when human readability does not improve correspondingly. Please provide a human study on LongText-Bench, an independent OCR engine not used in training, and an analysis of reward versus true benchmark performance; without these, the SOTA long-text claim is not yet supported.
  2. [§4.4 and Figure 2(b)] The central attribution of gains to RL is supported mainly by the training curve in Figure 2(b), which claims that the RL reward surpasses the SFT best-of-N baseline. However, no final-checkpoint benchmark numbers are reported for the SFT or SFT+BoN baselines on any of the evaluation tables (Tables 1–4). Figure 2(b) also lacks axis labels and quantitative endpoints, making it impossible to assess the magnitude of the improvement. Please add a direct comparison of the final RL model, the SFT model, and the SFT+BoN baseline under identical evaluation protocols on DPG-Bench, GenEval, and the text-rendering benchmarks; this is necessary to substantiate the statement that RL 'outperforms SFT with best-of-N sampling'.
  3. [Tables 1–4 and Appendix A] All benchmark tables report single runs without confidence intervals or significance tests. This is particularly concerning for the new LongText-Bench, which has only 160 prompts with 4 generated images each (Appendix A), and for several comparisons in Tables 2 and 3 where the differences are small (e.g., Table 2: X-Omni 87.65 vs Show-o2 86.14; Table 3: X-Omni 0.83 vs Mogao 0.89). Given the stochasticity of sampling and the small size of the new benchmark, the reported margins may not be meaningful. Please provide error bars, significance tests, or at least per-prompt variance for the key claims, especially for the new benchmark.
  4. [§3.2.2 and Eq. (1)] The reward system is a central component of the method, but the paper does not specify the aggregation weights for HPSv2, Unified Reward, text-image alignment, and OCR rewards, nor the GRPO clipping parameter epsilon in Eq. (1). Section 4.2 reports learning rate, batch size, number of rollouts, and KL weight, but the missing reward weights and clipping value prevent reproduction and make it impossible to analyze reward overoptimization. Please report these values and, ideally, an ablation showing the contribution of each reward component; the current description is too underspecified for a method whose entire claim is that carefully designed rewards drive the improvement.
  5. [§4.2] The paper states that 'The Chinese text rendering model is derived by incorporating training on Chinese data at an intermediate checkpoint during the reinforcement learning stage.' This means the Chinese and English X-Omni results in Table 1 may come from different checkpoints or training recipes. Please clarify whether Table 1 reports a single unified model or separate English and Chinese variants, what the intermediate checkpoint is, how much additional Chinese data is used, and how this affects the comparison with other models. Without this clarification, the claim of a single unified model and the SOTA Chinese text-rendering result are ambiguous.
minor comments (6)
  1. [Table 3] The abstract claims 'state-of-the-art performance in image generation tasks using a 7B language model,' but on GenEval X-Omni (0.83) is below Mogao (0.89) and BAGEL (0.88). Please qualify the SOTA claim to refer to the specific benchmarks or unified-model category where it holds.
  2. [§4.4 and Figure 6] The finding that X-Omni does not rely on classifier-free guidance is supported only by qualitative examples in Figure 6. Please add a quantitative comparison of CFG vs no-CFG generation on at least one benchmark, since this is presented as a key result.
  3. [Figure 2(b)] The training curve in Figure 2(b) would benefit from labeled axes, units, and explicit endpoints for the SFT BoN baseline and the final RL reward; currently the reader cannot assess the magnitude of the claimed improvement.
  4. [Appendix A] The LongText-Bench evaluation metric 'Text Accuracy' is not formally defined. Please specify whether it is character-level, word-level, normalized, and how partial matches are scored, so that the numbers in Table 1 are reproducible.
  5. [§3.1] The training status of the diffusion decoder is ambiguous: the text says a linear layer is added to FLUX.1-dev and the decoder is 'trained with the objective of image reconstruction,' while later 'Both components are maintained frozen' appears to refer only to the visual encoder and vector quantizer. Please clarify whether FLUX.1-dev is fine-tuned or kept frozen during pretraining, SFT, and RL, and whether the linear adapter is trained in all stages.
  6. [Eq. (1)] There are minor typesetting issues in Eq. (1), including the rendering of the summation and the subscript of the reference policy. Please fix these for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the RL objective and the reported benchmarks are distinct, and the OCR/VLM reward-evaluation overlap is a validity risk rather than a circular reduction.

full rationale

The paper's derivation chain is empirical rather than formal: an autoregressive policy is optimized with GRPO (Eq. 1) against a composite reward system (HPSv2, Unified Reward, Qwen2.5-VL-32B alignment, and GOT-OCR2.0/PaddleOCR; Section 3.2.2), and the resulting model is then measured on DPG-Bench, GenEval, OneIG-Bench, and standard image-understanding benchmarks. None of these target benchmark labels or metrics enters the training objective, so the headline gains are not fitted to the reported numbers by construction. The self-proposed LongText-Bench uses Qwen2.5-VL-7B as an OCR scorer (Appendix A), which overlaps in model family with the Qwen2.5-VL-32B alignment reward; this overlap weakens the independence of the long-text-rendering result but does not make it equal to the training signal, because the reward tasks and evaluation metrics differ. The Unified Reward model [67] has an overlapping author, but it is a training-signal component rather than the evidence for the central conclusion, which also rests on external benchmarks. No equation-level reduction, fitted-parameter-as-prediction, or self-citation chain forces the paper's claims; the identified concerns are reward-overoptimization and external-validity risks, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's contribution is an empirical system; it does not derive equations. The central claim depends on the quality of the frozen tokenizer/decoder, on the validity of the chosen reward models, and on GRPO's stability. No revolutionary new entities are introduced; the closest new artifact is the LongText-Bench benchmark and the unspecified reward weights.

free parameters (3)
  • Reward aggregation weights for HPSv2, Unified Reward, alignment, and OCR scores = unspecified
    Section 3.2.2 says the rewards are combined through a weighted aggregation mechanism but does not give weights or a tuning procedure; these weights determine the RL objective.
  • GRPO clipping parameter epsilon = unspecified
    Equation (1) uses epsilon but the paper never states its value; a standard hyperparameter that affects policy update size.
  • Text-rendering prompt bucket boundaries for RL data sampling = unspecified
    Section 4.1 says prompts were sorted into buckets by text length with 10K samples per bucket, but bucket ranges are not defined.
assumptions (4)
  • domain assumption The frozen SigLIP-VQ semantic tokenizer and fixed FLUX diffusion decoder can together reconstruct high-quality images from semantic tokens.
    Section 3.1 describes the tokenizer and decoder; the entire method relies on this reconstruction ability before RL even starts.
  • domain assumption HPSv2, Unified Reward, Qwen2.5-VL-32B, GOT-OCR2.0, and PaddleOCR scores are valid proxies for real image quality and prompt alignment.
    Section 3.2.2 builds the RL reward on these models; if any is miscalibrated or hackable, RL will optimize the wrong target.
  • standard math GRPO's clipped surrogate objective and KL penalty yield a stable policy improvement for this autoregressive visual token generator.
    Equation (1) is taken from DeepSeekMath [61] without re-derivation.
  • ad hoc to paper Qwen2.5-VL-7B OCR accuracy is a reliable metric for long-text rendering in the new LongText-Bench.
    Appendix A introduces LongText-Bench and adopts Qwen2.5-VL-7B to parse rendered text; this evaluator is not independently validated against human reading.

how reviews work

0 comments
Cite this review

Pith. "Pith review of X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again." pith.science (2026). https://pith.science/paper/BMGHFCH7

@misc{pith2026250722058,
  author       = {Pith},
  title        = {Pith review of: X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMGHFCH7}},
  note         = {Machine review of arXiv:2507.22058}
}
read the original abstract

Numerous efforts have been made to extend the ``next token prediction'' paradigm to visual contents, aiming to create a unified approach for both image generation and understanding. Nevertheless, attempts to generate images through autoregressive modeling with discrete tokens have been plagued by issues such as low visual fidelity, distorted outputs, and failure to adhere to complex instructions when rendering intricate details. These shortcomings are likely attributed to cumulative errors during autoregressive inference or information loss incurred during the discretization process. Probably due to this challenge, recent research has increasingly shifted toward jointly training image generation with diffusion objectives and language generation with autoregressive objectives, moving away from unified modeling approaches. In this work, we demonstrate that reinforcement learning can effectively mitigate artifacts and largely enhance the generation quality of a discrete autoregressive modeling method, thereby enabling seamless integration of image and language generation. Our framework comprises a semantic image tokenizer, a unified autoregressive model for both language and images, and an offline diffusion decoder for image generation, termed X-Omni. X-Omni achieves state-of-the-art performance in image generation tasks using a 7B language model, producing images with high aesthetic quality while exhibiting strong capabilities in following instructions and rendering long texts.

Figures

Figures reproduced from arXiv: 2507.22058 by the authors.

Figure 1
Figure 1. Equipped with reinforcement learning, X-Omni integrates image and language modeling in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. During the reinforcement learning process, X-Omni’s image generation reward quickly [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The architecture of X-Omni. We propose integrating image and text tokens within a unified autoregressive architecture. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Text rendering comparison with other unified multimodal models. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative cases of X-Omni. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Comparison of dependency on classifier-free guidance (CFG). X-Omni Does Not Rely on Classifier-Free Guidance. A key observation is that our model can generate high￾quality images without relying on classifier-free guidance (CFG) in the autoregressive component. Autoreg…
Figure 7
Figure 7. Figure 7: Comparison between our proposed LongText-Bench and OneIG-Bench with respect to the [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

    cs.CV 2026-08 conditional novelty 7.0 of 10

    By asking a unified multimodal model to caption an image, regenerate it, and answer questions about the regenerated image, SGU consistently exposes accuracy drops that separate benchmarks do not reveal.

  2. Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

    cs.AI 2026-07 conditional novelty 7.0 of 10

    BRAID jointly optimizes text and image generation in interleaved multi-modal reasoning by casting the full trajectory as a unified MDP with shared advantages and a VLM process reward.

  3. DynEval: Holistic Evaluations of T2I Generative Models in the Wild

    cs.CV 2026-07 conditional novelty 6.5 of 10

    DynEval distills a 235B teacher VLM into 2B/4B evaluators via 250K synthetic instruction triplets, yielding higher human correlation than existing T2I metrics while enabling open-set dynamic QA and scene-graph quality checks.

  4. Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Multimodal pretraining transfers asymmetrically: language boosts vision, understanding boosts generation, generation is mostly neutral, and early unified training prevents vision laziness.

  5. dRAE: Representation Autoencoder with Hyper-Spherical Codes

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Switching codebook assignment and update to cosine similarity while keeping a magnitude-preserving commitment loss avoids codebook collapse and scales visual tokenizers to 131,072 codes with high utilization.

  6. Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    Z-Image is an efficient 6B-parameter foundation model for image generation that rivals larger commercial systems in photorealism and bilingual text rendering through a new single-stream diffusion transformer and strea...

  7. RubricRL: Simple Generalizable Rewards for Text-to-Image Generation

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Using an LLM to generate prompt-specific visual rubrics and grade each criterion independently gives a more interpretable reward that improves text-to-image model alignment beyond composite and learned scalar rewards.

  8. Reconstruction Alignment Improves Unified Multimodal Models

    cs.CV 2025-09 conditional novelty 6.0 of 10

    RECA, a self-supervised post-training objective that conditions unified multimodal models on their own visual understanding embeddings to reconstruct input images, improves text-to-image and editing benchmarks across ...

  9. TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    TokLIP semanticizes VQ image tokens with a causal CLIP-style encoder, improving multimodal comprehension while preserving autoregressive image generation.

  10. Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing

    cs.CV 2026-07 conditional novelty 5.0 of 10

    A compact 4B image generation/editing system with a fast one-step VAE, native-resolution packing, RL alignment, and 4-step distillation reports competitive benchmarks against 6B–80B open models.

  11. FAIL: Flow Matching Adversarial Imitation Learning for Image Generation

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Post-training of flow matching can be framed as adversarial imitation learning, and the proposed FAIL methods improve FLUX's generation quality using 13K expert images without preference pairs.

  12. Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

    cs.CV 2026-07 conditional novelty 4.0 of 10

    An open-source image-generation family shows that agentic prompt rewriting and a stronger text encoder can lift quality to near closed-source levels with only 208.62M images and about $400K of training compute.

Reference graph

Works this paper leans on

101 extracted references · 13 canonical work pages · cited by 12 Pith papers

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    2020

  2. [2]

    Chatgpt.https://chat.openai.com/, 2023

    OpenAI. Chatgpt.https://chat.openai.com/, 2023

    2023

  3. [3]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    arXiv 2024

  5. [5]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  6. [6]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  7. [7]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  8. [8]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

Show all 101 references
  1. [9]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  2. [10]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  3. [11]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2024

  4. [12]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  5. [13]

    Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499, 2023

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation.arXiv preprint arXiv:2309.11499, 2023

    2023 arXiv

  6. [14]

    Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    2024 arXiv

  7. [15]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024. 13

    2024 arXiv

  8. [16]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  9. [17]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  10. [18]

    Vector quantized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10696–10706, 2022

    2022

  11. [19]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information ...

    2022

  12. [20]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 2...

    2025

  13. [21]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  14. [22]

    Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multi- modal understanding and generation.arXiv preprint arXiv:2503.06764, 2025

    Zisheng Chen, Chunwei Wang, Xiuwei Chen, Hang Xu, Jianhua Han, and Xiaodan Liang. Semhitok: A unified image tokenizer via semantic-guided hierarchical codebook for multi- modal understanding and generation.arXiv preprint arXiv:2503.06764, 2025

    2025

  15. [23]

    Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement.arXiv preprint arXiv:2504.01934, 2025

    Runhui Huang, Chunwei Wang, Junwei Yang, Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Illuminating unified mllm with dual visual tokenization and diffusion refinement.arXiv preprint arXiv:2504.01934, 2025

    2025 arXiv

  16. [24]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  17. [25]

    Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024

    2024 arXiv

  18. [26]

    Orthus: Autoregressive interleaved image-text generation with modality-specific heads.arXiv preprint arXiv:2412.00127, 2024

    Siqi Kou, Jiachun Jin, Zhihong Liu, Chang Liu, Ye Ma, Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng. Orthus: Autoregressive interleaved image-text generation with modality-specific heads.arXiv preprint arXiv:2412.00127, 2024

    2024 arXiv

  19. [27]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    2025 arXiv

  20. [28]

    Transfer between modalities with metaqueries.arXiv preprint arXiv:2504.06256, 2025

    Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between modalities with metaqueries.arXiv preprint arXiv:2504.06256, 2025

    2025 arXiv

  21. [29]

    Uniworld: High-resolution semantic encoders for unified visual understanding and generation.arXiv preprint arXiv:2506.03147, 2025

    Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld: High-resolution semantic encoders for unified visual understanding and generation.arXiv preprint arXiv:2506.03147, 2025. 14

    2025 arXiv

  22. [30]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    2025 arXiv

  23. [31]

    Omnigen2: Exploration to advanced multimodal generation.arXiv preprint arXiv:2506.18871, 2025

    Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Ex...

    2025 arXiv

  24. [32]

    Show-o2: Improved native unified multimodal models.arXiv preprint arXiv:2506.15564, 2025

    Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models.arXiv preprint arXiv:2506.15564, 2025

    2025 arXiv

  25. [33]

    Mogao: An omni foundation model for interleaved multi-modal generation.arXiv preprint arXiv:2505.05472, 2025

    Chao Liao, Liyang Liu, Xun Wang, Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation.arXiv preprint arXiv:2505.05472, 2025

    2025 arXiv

  26. [34]

    Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning.arXiv preprint arXiv:2412.14164, 2024

    2024 arXiv

  27. [35]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. InForty-first International Conference on Machine Learning, 2024

    2024

  28. [36]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  29. [37]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. InProceedings of the Computer Vision and Pattern Recognition C...

    2025

  30. [38]

    Unified language-vision pretraining in llm with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Yadong Mu, et al. Unified language-vision pretraining in llm with dynamic discrete visual tokenization. InInternational Conference on Learning Representations, 2024

    2024

  31. [39]

    Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale.arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  32. [40]

    Lmfusion: Adapting pretrained language models for multimodal generation

    Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188, 2024

    2024 arXiv

  33. [41]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. InProceedings of the Computer Vision ...

    2025

  34. [42]

    Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    2024 arXiv

  35. [43]

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. InInternational Conference on Learning Representations, 2024

    2024

  36. [44]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. InAdvances in Neural Information Processing Systems, 2023

    2023

  37. [45]

    Video diffusion alignment via reward gradients.CoRR, abs/2407.08737, 2024

    Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients.CoRR, abs/2407.08737, 2024. 15

    2024 arXiv

  38. [46]

    Aligning text-to-image models using human feedback.CoRR, abs/2302.12192, 2023

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback.CoRR, abs/2302.12192, 2023

    2023 arXiv

  39. [47]

    DPOK: reinforcement learning for fine-tuning text-to-image diffusion models.CoRR, abs/2305.16381, 2023

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: reinforcement learning for fine-tuning text-to-image diffusion models.CoRR, abs/2305.16381, 2023

    2023 arXiv

  40. [48]

    RAFT: reward ranked finetuning for generative foundation model alignment.Trans

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. RAFT: reward ranked finetuning for generative foundation model alignment.Trans. Mach. Learn. Res., 2023, 2023

    2023

  41. [49]

    Instructdiffusion: A generalist modeling interface for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, Dong Chen, and Baining Guo. Instructdiffusion: A generalist modeling interface for vision tasks. InIEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  42. [50]

    Diffusion model alignment using direct preference optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. InIEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  43. [51]

    Using human feedback to fine-tune diffusion models without any reward model

    Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941–8951, 2024

    2024

  44. [52]

    Step-aware preference optimization: Aligning preference with denoising performance at each step.CoRR, abs/2406.04314, 2024

    Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance at each step.CoRR, abs/2406.04314, 2024

    2024 arXiv

  45. [53]

    Training diffusion models with reinforcement learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. InInternational Conference on Learning Representations, 2024

    2024

  46. [54]

    Training diffusion models towards diverse image generation with reinforcement learning

    Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10844–10853, 2024

    2024

  47. [55]

    Self-play fine-tuning of diffusion models for text-to-image generation

    Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. InAdvances in Neural Information Processing Systems, 2024

    2024

  48. [56]

    Towards better alignment: Training diffusion models with reinforce- ment learning against sparse rewards

    Zijing Hu, Fengda Zhang, Long Chen, Kun Kuang, Jiahui Li, Kaifeng Gao, Jun Xiao, Xin Wang, and Wenwu Zhu. Towards better alignment: Training diffusion models with reinforce- ment learning against sparse rewards. InIEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2025

  49. [57]

    A simple and effective reinforcement learning method for text-to-image diffusion fine-tuning.CoRR, abs/2503.00897, 2025

    Shashank Gupta, Chaitanya Ahuja, Tsung-Yu Lin, Sreya Dutta Roy, Harrie Oosterhuis, Maarten de Rijke, and Satya Narayan Shukla. A simple and effective reinforcement learning method for text-to-image diffusion fine-tuning.CoRR, abs/2503.00897, 2025

    2025

  50. [58]

    Flow-grpo: Training flow matching models via online rl, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl, 2025

    2025

  51. [59]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, 2023

    2023

  52. [60]

    Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347, 2017. 16

    2017 arXiv

  53. [61]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  54. [62]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

    2025 arXiv

  55. [63]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  56. [64]

    Flux.https://github.com/black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

  57. [65]

    Approximating kl divergence

    John Schulman. Approximating kl divergence. http://joschu.net/blog/kl-approx. html, 2020

    2020

  58. [66]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.arXiv preprint arXiv:2306.09341, 2023

    2023 arXiv

  59. [67]

    Unified reward model for multimodal understanding and generation.arXiv preprint arXiv:2503.05236, 2025

    Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation.arXiv preprint arXiv:2503.05236, 2025

    2025 arXiv

  60. [68]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Han...

    2025 arXiv

  61. [69]

    General ocr theory: Towards ocr-2.0 via a unified end-to-end model.arXiv preprint arXiv:2409.01704, 2024

    Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, et al. General ocr theory: Towards ocr-2.0 via a unified end-to-end model.arXiv preprint arXiv:2409.01704, 2024

    2024 arXiv

  62. [70]

    Paddleocr, awesome multilingual ocr toolkits based on paddlepaddle

    PaddlePaddle Authors. Paddleocr, awesome multilingual ocr toolkits based on paddlepaddle. https://github.com/PaddlePaddle/PaddleOCR, 2020

    2020

  63. [71]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/ coyo-dataset, 2022

    2022

  64. [72]

    Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M. Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Muss...

    2023

  65. [73]

    Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural information processing systems, 35:25278–25294, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models.Advances in neural informa...

    2022

  66. [74]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  67. [75]

    Blip3-kale: Knowledge augmented large-scale dense captions.arXiv preprint arXiv:2411.07461, 2024

    Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, et al. Blip3-kale: Knowledge augmented large-scale dense captions.arXiv preprint arXiv:2411.07461, 2024. 17

    2024 arXiv

  68. [76]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data, 2024

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, Zhenchong Hu, Bo-Wen Zhang, Jijie Li, Dong Liang, Yingli Zhao, Yulong Ao, Yaoqi Liu, Fangxiang Feng, and Guang Liu. Infinity-mm: Scaling multimodal pe...

    2024

  69. [77]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  70. [78]

    What matters when building vision-language models?, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024

    2024

  71. [79]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal llms, 2024

    2024

  72. [80]

    Midjourney prompts dataset

    Vivym. Midjourney prompts dataset. https://huggingface.co/datasets/vivym/ midjourney-prompts, 2024

    2024

  73. [81]

    Oneig-bench: Omni-dimensional nuanced evaluation for image generation.arXiv preprint arxiv:2506.07977, 2025

    Jingjing Chang, Yixiao Fang, Peng Xing, Shuhan Wu, Wei Cheng, Rui Wang, Xianfang Zeng, Gang Yu, and Hai-Bao Chen. Oneig-bench: Omni-dimensional nuanced evaluation for image generation.arXiv preprint arxiv:2506.07977, 2025

    2025 arXiv

  74. [82]

    Hidream-i1: A high-efficient image generative foundation model with sparse diffusion transformer.arXiv preprint arXiv:2505.22705, 2025

    Qi Cai, Jingwen Chen, Yang Chen, Yehao Li, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Yiheng Zhang, Fengbin Gao, Peihan Xu, et al. Hidream-i1: A high-efficient image generative foundation model with sparse diffusion transformer.arXiv preprint arXiv:2505.22705, 2025

    2025 arXiv

  75. [83]

    Kolors2.0.https://app.klingai.com/cn/, 2025

    Kuaishou Kolors team. Kolors2.0.https://app.klingai.com/cn/, 2025

    2025

  76. [84]

    Seedream 3.0 technical report.arXiv preprint arXiv:2504.11346, 2025

    Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xi- aochen Lian, Chao Liao, Liyang Liu, et al. Seedream 3.0 technical report.arXiv preprint arXiv:2504.11346, 2025

    2025 arXiv

  77. [85]

    Addendum to gpt-4o system card: 4o image generation, 2025

    OpenAI. Addendum to gpt-4o system card: 4o image generation, 2025. Accessed: April 2, 2025

    2025

  78. [86]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  79. [87]

    Dall·e 3.https://openai.com/index/dall-e-3/, 2024

    OpenAI. Dall·e 3.https://openai.com/index/dall-e-3/, 2024

    2024

  80. [88]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. InForty-first international conference on machine...

    2024

  81. [89]

    Ovis-u1 technical report.arXiv preprint arXiv:2506.23044, 2025

    Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Jianshan Zhao, Yang Li, and Qing-Guo Chen. Ovis-u1 technical report.arXiv preprint arXiv:2506.23044, 2025

    2025 arXiv

  82. [90]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.CoRR, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.CoRR, 2024

    2024

  83. [91]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  84. [92]

    Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation.arXiv preprint arXiv:2504.02782, 2025

    Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation.arXiv preprint arXiv:2504.02782, 2025

    2025 arXiv

  85. [93]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 18

    2024

  86. [94]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023

    2023

  87. [95]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint 2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. M...

    2024 arXiv

  88. [96]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  89. [97]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  90. [98]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  91. [99]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

    2024

  92. [100]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  93. [101]

    short” category of LongText-Bench are concentrated within the range of 10-30 words, while those in the “long

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024. 19 Figure 7: Comparison ...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.