Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SUDER claims unified multimodal models can improve both text-to-image generation and visual understanding using their own reversed input-output likelihoods as self-rewards, without any external supervision or paired data.

desk verdict Clever dual self-reward idea with consistent reported gains, but the self-referential reward is unvalidated and could be reward hacking; worth refereeing, not yet believable. read the letter →

arxiv 2506.07963 v3 pith:YINVLGZL submitted 2025-06-09 cs.AI cs.CLcs.CV

Jixiang Hong , Yiran Zhang , Guanzhong Wang , Yi Liu , Ji-Rong Wen , Rui Yan This is my paper
classification cs.AIcs.CLcs.CV
keywords largemultimodalmodeltext-to-imagegenerationvision-languagealignmentself-supervisedlearningdualself-rewardpreferenceoptimizationvisualunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that unified multimodal models—models that can both read and generate images—can improve their own performance on both abilities with no external feedback, human labels, or paired image-text data. SUDER treats image captioning and text-to-image generation as inverse tasks: it samples several candidate outputs for an input, then scores each candidate by asking the model how likely it is to regenerate the original input from that candidate. These scores become dual self-rewards that drive a preference-optimization or reinforcement-learning step. If the claim holds, a free self-supervised loop can reduce hallucinations in visual question answering and prompt-following failures in text-to-image generation, without any supervised alignment data.

What carries the argument

The dual self-reward (DSR) mechanism is the load-bearing object. For each sampled output, DSR computes the length-normalized log-likelihood of the model regenerating the original input when the output is used as the condition: $R_U(Y_T|X_V) = \frac{1}{\|X_V\|}\log\pi_\theta(X_V|Y_T)$ for a candidate caption in understanding, and $R_G(Y_V|X_T) = \frac{1}{\|X_T\|}\log\pi_\theta(X_T|Y_V)$ for a candidate image in generation. These reversed likelihoods serve as the rewards that select preferred candidates for SimPO or define group-normalized advantages for GRPO. The mechanism runs in a single forward pass, uses no external reward model, and length-normalization shields it from the length bias that affects reward-model scores.

What would settle it

Take a held-out set of prompts and images, rank candidate outputs by the dual self-reward, and compare those rankings against independent alignment judgments (human ratings or a strong external VQA/hallucination detector). If the dual self-reward's ranking does no better than chance at picking the output humans judge as more aligned, the training signal is not tracking alignment and the reported gains would not transfer to new data.

Watch

Extended reading notes

Core claim

The central claim is that the duality between understanding and generation can be converted into a training signal, so the two tasks supervise each other. For a visual-understanding input, SUDER samples multiple captions for an image, reverses each pair, and computes the likelihood of regenerating the image given each caption; the caption with the higher reverse likelihood is treated as the more accurate one. For text-to-image generation, sampled images are scored by the likelihood of regenerating the original prompt given the image. Optimizing with these self-rewards improves Janus-Pro-7B by 11.68% on average on T2I-CompBench and 5% overall on GenEval, while improving LLaVA-Bench by 7.5%; the same mechanism transfers to a second backbone, Show-o, with a more than 5% average gain on T2I-CompBench. These gains are obtained without any external supervision or parallel image-text pairs in training.

Load-bearing premise

The model's own probability of regenerating the original input is assumed to reliably measure how well the caption and image actually match, and this link must stay strong through training instead of being exploitable to inflate the score.

Editorial extensions

If this is right

  • Text-to-image alignment improves measurably on compositional benchmarks, with the largest gains in attribute binding (color, shape, texture) across both tested backbones.
  • Jointly optimizing understanding and generation beats optimizing either alone, so the dual reward does not trade one capability against the other.
  • The framework works with two different optimizers (SimPO and GRPO) and two different architectures (Janus-Pro-7B/1B and Show-o), which points to the reward signal rather than a specific training recipe as the source of improvement.
  • A two-model variant that trains understanding and generation alternately, using each model's rewards to improve the other, outperforms the single-model version on both tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same reverse-likelihood loop could apply to any modality pair with usable inverse directions—audio-to-text, video-to-text, or code-and-execution pairs—since the signal is effectively the mutual information between the two representations; the paper names audio as an open direction.
  • The stated requirement that the backbone already have basic alignment implies a testable prediction: on weaker or less-aligned backbones the dual self-reward is a noisier proxy, so the gains should shrink as initial alignment drops.
  • The alternating two-model scheme resembles a self-play critic-generator game; if scaled, it may develop instabilities and mode collapse like other self-play systems, an issue the paper does not examine.
  • A risk the paper leaves implicit is reward hacking: the model could raise its reverse likelihood by producing outputs that are simply easy to reconstruct—generic images or templated captions—which the benchmark gains do not rule out on out-of-distribution prompts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents SUDER, a self-supervised optimization framework for unified large multimodal models (LMMs). The core idea is a dual self-reward (DSR): for a sampled response in one modality, the model computes the likelihood of reconstructing the original input in the other modality, using the very model being optimized (Eq. 3). DSR scores are then used with SimPO or GRPO to reinforce both visual understanding and text-to-image generation, either jointly in one model or alternately in two models. On Janus-Pro-7B and Janus-Pro-1B, the authors report average T2I-CompBench increases of 11.68% and 10.5%, an overall GenEval gain of 5% for Janus-Pro-7B, and improvements or stability on understanding benchmarks such as LLaVA-Bench and HallusionBench. An appendix extends the method to Show-o. The experiments use non-parallel image and text data and require no external reward model or human annotations.

Significance. If robust, SUDER would be an important result in self-supervised multimodal alignment: it turns the model's own inverse prediction into a label-free reward, is computable in a single forward pass, and shows consistent gains across two scales of Janus-Pro and, in the appendix, on Show-o. The authors are explicit about their central assumption in Section 3.2 and provide qualitative rankings in Figures 9 and 10. However, the paper currently lacks three pieces of evidence needed to support the central claim: a demonstration that DSR correlates with external alignment, a control against reward hacking, and any measure of statistical reliability (error bars, seeds, significance). The reported gains are consistent and plausible, but the self-referential nature of the reward is a real risk that is not yet addressed; the significance is therefore conditional.

major comments (4)
  1. [§3.2, Eq. (3); §3.3, Eq. (4)] The reward in Eq. (3) is the log-likelihood of the original input under the same conditional distribution π_θ that SimPO (Eq. 4) or GRPO (Eq. 7) subsequently optimizes. Because SimPO is reference-free, there is no external anchor preventing the model from increasing DSR by exploiting its own reversible-computation shortcuts: for example, the generation branch can drift toward images that the captioning head can reconstruct regardless of prompt fidelity, and the understanding branch can drift toward captions that make any image likely. The manuscript offers anecdotal examples (Figs. 9–10) and an indirect CLIP comparison (Table 5), but no correlation measure between DSR and human or CLIP alignment on held-out data, no random-reward control, and no diversity or FID metric. Please add at least: (i) a control that optimizes with rewards computed by a frozen or random model, and (ii) a rank correlation of DSR against CLIP or human judgments on a held-out set. Without this, the central claim that the self-supervised loop improves semantic alignment is not established.
  2. [Tables 1–3 and §4.2–4.3] All benchmark numbers are reported as single point estimates with no standard deviations, number of seeds, or significance tests. Several gains are small enough to be within typical evaluation noise, e.g., Non-Spatial +1% for Janus-Pro-7B, HalluBench +1.9 points, and POPE −0.2 points in Table 3; even the headline +11.68% T2I average needs a variance estimate given the instability visible in the per-category numbers across scales. Please report mean and standard deviation over at least three independent training runs (or provide a justified single-run protocol), and state whether any hyperparameter selection (e.g., β, γ, clipping thresholds) was performed on the evaluation benchmarks.
  3. [§4.1, T2I-CompBench split] The training data uses the 5,600-prompt training set of T2I-CompBench, while the evaluation uses the benchmark's 6,000 'original text prompts.' The paper does not state whether these sets are disjoint. If the evaluation includes training prompts, the T2I-CompBench gains are partially a memorization effect and do not measure generalization. Please state the exact split, and if no disjoint evaluation split exists, re-evaluate on a held-out compositional benchmark (e.g., GenEval or a filtered subset of T2I-CompBench) to certify generalization.
  4. [Table 4 and §4.4] The ablation shows that optimizing only the generation branch already raises LLaVA-Bench from 74.0 to 77.3 and HalluBench from 37.0 to 37.3, even though no understanding-specific training is performed. This is a substantial transfer effect and is not explained; it complicates the attribution of the understanding gains to the dual reward, since a single-branch control produces most of the LLaVA-Bench improvement seen in the unified model (81.5). Please investigate whether this transfer is due to shared parameters, to the shared training distribution, or to instability, and ideally include a control where generation is optimized with a random reward to distinguish the effect of DSR from generic fine-tuning.
minor comments (6)
  1. [§4.5, Figure 3 vs Figure 4] The sentence 'The results can be seen in Figure 3, which shows that SimPO is slightly better than GRPO' should refer to Figure 4; Figure 3 is the separate-vs-unified strategy comparison.
  2. [Eq. (4)] Equation (4) contains an editing leftover: the sentence 'where R(·|·) denotes the corresponding dual self-reward from Section 3.2' appears inside the SimPO objective, but R is never used in that objective; the sentence should be removed or integrated into the surrounding text.
  3. [Table 3] The header 'LLaV ABench' contains an unintended space, and 'HallBench' / 'HalluBench' are used inconsistently across tables and text; please unify these names.
  4. [Reference [31]] Reference [31] is listed as 'Chameleon: Plug-and-play compositional reasoning...' by Lu et al., but the Chameleon model discussed in Tables 1–3 is the unified multimodal model; please cite the correct paper.
  5. [Appendix A.4, Table 6] The Show-o results show the method mostly improves attribute binding (about +10 points) while spatial and non-spatial relationships are essentially flat (−0.2 and −0.1); the discussion should acknowledge this concentration rather than claim a uniform gain.
  6. [General] The manuscript does not include a code or model availability statement; please add one, since the reproducibility of the RL training details (e.g., sampling temperature, reward computation for normalized likelihoods) would otherwise be hard to assess.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dual self-rewards are self-referential in implementation, but every claimed gain is measured on external fixed-prompt benchmarks, so no result reduces to its training objective by construction.

full rationale

No circular step can be exhibited from the paper's own equations. The dual self-rewards in Eq. (3) are defined as the log-likelihood of the original input under the current model (R_U(Y_T|X_V) = (1/||X_V||) log π_θ(X_V|Y_T) and R_G(Y_V|X_T) = (1/||X_T||) log π_θ(X_T|Y_V)), and Eq. (4) then uses those rewards to select preference pairs for the same model. That is a self-referential training signal, and the Section 3.2 statement that a description accurately reflecting the image should have high reverse likelihood is an empirical proxy assumption, not a definitional equivalence between the reward and external alignment. The central claim, however, is empirical: the headline numbers (+11.68% on T2I-CompBench, +5% on GenEval, +7.5% on LLaVA-Bench) come from standard external benchmarks with fixed prompts and metrics independent of the DSR values. The reported improvements are therefore not forced by construction, and no fitted parameter is renamed as a prediction. The only author self-citation (POPE, reference [22], which includes Ji-Rong Wen) is used as an evaluation benchmark and is not load-bearing support for the method. The paper invokes no uniqueness theorem and imports no ansatz through prior-work citation. The Limitations section's remark that the backbone must already have basic semantic alignment is a stated precondition rather than a circular derivation. Under the required standard of exhibiting a specific reduction, no circular step is present, so the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities; it relies on the model's own likelihoods as rewards. The key assumptions are the task duality, the validity of the dual likelihood as a semantic alignment measure, and the pre-alignment of the backbone. Free parameters are the training hyperparameters, which are manually chosen and could influence the reported results.

free parameters (4)
  • SimPO beta = 2.0
    Choice of margin scaling for preference optimization; set by hand, not derived, and likely affects the magnitude of improvements.
  • SimPO gamma = 0.5
    Reward margin in SimPO; chosen by hand and influences the separation between chosen and rejected samples.
  • GRPO beta = 0.04
    KL penalty coefficient for GRPO; manually set, controls deviation from reference policy.
  • GRPO clipping thresholds = epsilon_low=0.2, epsilon_high=0.28
    Clipping bounds in GRPO; chosen by hand, standard values from prior work.
assumptions (4)
  • domain assumption Understanding and generation are inverse dual tasks, such that the output of one can serve as the input of the other.
    Section 3.1 builds the entire method on this duality. It is a conceptual assumption about the structure of the tasks, not a proven theorem.
  • ad hoc to paper If a textual description accurately reflects the visual input, then the likelihood of generating the original image conditioned on this description should be high.
    This is the core reward assumption in Section 3.2. It is stated as 'intuitive assumption' and is not proven; the paper explicitly labels it as an assumption.
  • domain assumption The backbone LMM has undergone basic semantic alignment.
    The Limitations section admits that 'the dual self-reward mechanism requires the backbone LMM to have undergone basic semantic alignment.' If the backbone is misaligned, the dual likelihoods may be meaningless.
  • domain assumption The model's probability estimates are reliable and stable enough to serve as training rewards.
    The reward is a log-likelihood from the model itself. No calibration analysis is provided, and the paper acknowledges 'DSR contains a certain level of noise' in Section 4.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards." pith.science (2026). https://pith.science/paper/YINVLGZL

@misc{pith2026250607963,
  author       = {Pith},
  title        = {Pith review of: SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YINVLGZL}},
  note         = {Machine review of arXiv:2506.07963}
}
read the original abstract

Building upon large language models (LLMs), recent large multimodal models (LMMs) unify cross-model understanding and generation into a single framework. However, LMMs still struggle to achieve accurate vision-language alignment, prone to generating text responses contradicting the visual input or failing to follow the text-to-image prompts. Current solutions require external supervision (e.g., human feedback or reward models) and only address unidirectional tasks-either understanding or generation. In this work, based on the observation that understanding and generation are naturally inverse dual tasks, we propose \textbf{SUDER} (\textbf{S}elf-improving \textbf{U}nified LMMs with \textbf{D}ual s\textbf{E}lf-\textbf{R}ewards), a framework reinforcing the understanding and generation capabilities of LMMs with a self-supervised dual reward mechanism. SUDER leverages the inherent duality between understanding and generation tasks to provide self-supervised optimization signals for each other. Specifically, we sample multiple outputs for a given input in one task domain, then reverse the input-output pairs to compute the dual likelihood within the model as self-rewards for optimization. Extensive experimental results on visual understanding and generation benchmarks demonstrate that our method can effectively enhance the performance of the model without any external supervision, especially achieving remarkable improvements in text-to-image tasks.

Figures

Figures reproduced from arXiv: 2506.07963 by the authors.

Figure 1
Figure 1. Failures in visual understanding and image gener [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of SUDER. The model samples multiple outputs for a given input for one task and reverses the input-output [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of op￾timization strategies. Color Attributes Two Objects Single Object Counting Colors Position SimPO GRPO [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Evolution on LLaVABench. 5 Conclusion In this work, we proposed a self-improving framework SUDER, with a dual self-reward mechanism to enhance the generation and understanding capabilities of LMMs in an efficient, self-supervised, and unified manner. By leveraging the …
Figure 7
Figure 7. Figure 7: Qualitative comparison of visual generation. We sample multiple images for the same text prompts using Janus-Pro-7B [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison of visual understanding. We sample multiple responses for same image with a question using [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Demonstration of our dual self-reward for visual generation. We sample multiple images for a given text prompt and [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Demonstration of our dual self-reward for visual understanding. We sample multiple text descriptions for a given [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Models

    cs.CV 2025-10 conditional novelty 5.0 of 10

    A unified multimodal model can improve its own text-to-image generation by using its understanding module as a rewarder in a global-plus-local reward-weighted training loop.

Reference graph

Works this paper leans on

66 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anthropic. 2024. Introducing the next generation of Claude. (2024). https: //www.anthropic.com/news/claude-3-family

    work page 2024

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966 1, 2 (2023), 3

    arXiv 2023

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    arXiv 2025

  4. [4]

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al . 2023. Improving im- age generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf 2, 3 (2023), 8

    2023

  5. [5]

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. 2023. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–10

    work page 2023

  6. [6]

    Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. 2024. PixArt-𝑎𝑙𝑝ℎ𝑎 : Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. In The Twelfth International Conference on Learning Representations

    work page 2024

  7. [7]

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)

    arXiv 2025

  8. [8]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 24185–24198

    work page 2024

Show all 66 references
  1. [9]

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. Advances in Neural Information Processing Systems 36 (2023), 49250–49267

    2023

  2. [10]

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. 2024. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In International Conference on Machine Learni...

    2024

  3. [11]

    Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021. Taming Transformers for High-Resolution Image Synthesis. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 12868–12878

    2021

  4. [12]

    Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2022. Training- free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032 (2022)

    2022 arXiv

  5. [13]

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. 2024. Seed-x: Multimodal models with unified multi- granularity comprehension and generation. arXiv preprint arXiv:2404.14396 (2024)

    2024 arXiv

  6. [14]

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. 2023. GENEVAL: an object-focused framework for evaluating text-to-image alignment. In Proceedings of the 37th International Conference on Neural Information Processing Systems . 52132–52152

    2023

  7. [15]

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2I- CompBench: a comprehensive benchmark for open-world compositional text-to- image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems. 78723–78747

    2023

  8. [16]

    Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, and Yaodong Yang

  9. [17]

    Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. 2025. T2I-R1: Reinforc- ing Image Generation with Collaborative Semantic-level and Token-level CoT. arXiv:2505.00703 [cs.CV] https://arxiv.org/abs/2505.00703

    2025 arXiv

  10. [18]

    Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuo- fan Zong, Yu Liu, and Hongsheng Li. 2024. Comat: Aligning text-to-image diffu- sion model with image-to-text concept matching. Advances in Neural Information Processing Systems 37 (2024), 76177–76209

    2024

  11. [19]

    Black Forest Labs. 2024. FLUX. https://github.com/black-forest-labs/flux

    2024

  12. [20]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13299–13308

    2024

  13. [21]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 (2024)

    2024 arXiv

  14. [22]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 292–305

    2023

  15. [23]

    Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. 2024. Dual diffusion for unified image generation and understanding. arXiv preprint arXiv:2501.00289 (2024)

    2024 arXiv

  16. [24]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proce...

    2014

  17. [25]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26296–26306

    2024

  18. [26]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916

    2023

  19. [27]

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024. World model on million-length video and language with ringattention. arXiv e-prints (2024), arXiv–2402

    2024

  20. [28]

    Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. 2022. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision . Springer, 423–439

    2022

  21. [29]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al . 2024. MMBench: Is Your Multi-modal Model an All-Around Player?. In European Conference on Computer Vision. 216–233. Conference acronym ’XX, June 03–...

    2024

  22. [30]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024. Deepseek-vl: towards real- world vision-language understanding. arXiv preprint arXiv:2403.05525 (2024)

    2024 arXiv

  23. [31]

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play composi- tional reasoning with large language models. Advances in Neural Information Processing Systems 36 (2023), 43447–43478

    2023

  24. [32]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems 37 (2024), 124198–124235

    2024

  25. [33]

    OpenAI. 2023. GPT-4V(ision) System Card. (2023). https://cdn.openai.com/ papers/GPTV_System_Card.pdf

    2023

  26. [34]

    Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. 2024. Clip-dpo: Vision-language models as a source of preference for fixing hallucina- tions in lvlms. In European Conference on Computer Vision . Springer, 395–413

    2024

  27. [35]

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)

    2023 arXiv

  28. [36]

    Leigang Qu, Haochuan Li, Wenjie Wang, Xiang Liu, Juncheng Li, Liqiang Nie, and Tat-Seng Chua. 2025. SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation. arXiv:2412.05818 [cs.CV] https: //arxiv.org/abs/2412.05818

    2025 arXiv

  29. [37]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings...

    2021

  30. [38]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2023), 53728–53741

    2023

  31. [39]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

  32. [40]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 10674–10685

    2022

  33. [41]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al . 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    2024 arXiv

  34. [42]

    Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2023. Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 2859–2873

    2023

  35. [43]

    Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. 2023. JourneyDB: a benchmark for generative image understanding. In Proceedings of the 37th International Conference on Neural Information Processing Sy...

    2023

  36. [44]

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. 2024. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525 (2024)

    2024 arXiv

  37. [45]

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024. Genera- tive multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  38. [46]

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. 2024. Emu: Generative Pretraining in Multimodality. In The Twelfth International Conference on Learning Representations

    2024

  39. [47]

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems . 6309–6318

    2017

  40. [48]

    Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. 2024. Illume: Illuminating your llms to see, draw, and self-enhance. arXiv preprint arXiv:2412.06673 (2024)

    2024 arXiv

  41. [49]

    Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. 2025. SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL. arXiv:2504.11455 [cs.CV] https://arxiv.org/abs/2504.11455

    2025 arXiv

  42. [50]

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jin- sheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. 2024. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869 (2024)

    2024 arXiv

  43. [51]

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. 2024. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848 (2024)

    2024 arXiv

  44. [52]

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2024. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning

    2024

  45. [53]

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al . 2024. Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429 (2024)

    2024 arXiv

  46. [54]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. 2024. Deepseek- vl2: Mixture-of-experts vision-language models for advanced multimodal under- standing. arXiv preprint arXiv:2412.10302 (2024)

    2024 arXiv

  47. [55]

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. 2024. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528 (2024)

    2024 arXiv

  48. [56]

    Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. 2024. X-vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405.19335 (2024)

    2024 arXiv

  49. [57]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

  50. [58]

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. 2024. AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin...

    2024

  51. [59]

    Ma, Simon Stepputtis, Deva Ra- manan, Russ Salakhutdinov, Louis-Philippe Morency, Katia P

    Ce Zhang, Zifu Wan, Zhehan Kan, Martin Q. Ma, Simon Stepputtis, Deva Ra- manan, Russ Salakhutdinov, Louis-Philippe Morency, Katia P. Sycara, and Yaqi Xie. 2025. Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models. In ...

    2025

  52. [60]

    Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. 2025. d1: Scal- ing Reasoning in Diffusion Large Language Models via Reinforcement Learning. arXiv:2504.12216 [cs.CL] https://arxiv.org/abs/2504.12216

    2025 arXiv

  53. [61]

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839 (2023)

    2023 arXiv

  54. [62]

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. 2024. Trans- fusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039 (2024)

    2024 arXiv

  55. [63]

    M" and an

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024. MINIGPT-4: ENHANCING VISION-LANGUAGE UNDERSTANDING WITH AD- VANCED LARGE LANGUAGE MODELS. In 12th International Conference on Learning Representations, ICLR 2024 . A Technical Appendices A.1 Implementat...

    2024

  56. [2022]

    arXiv preprint arXiv:2204.06125 (2022)

    Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125 (2022)

    2022 arXiv

  57. [2024]

    arXiv:2412.15838 [cs.AI] https://arxiv.org/abs/2412.15838

    Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback. arXiv:2412.15838 [cs.AI] https://arxiv.org/abs/2412.15838

    arXiv

  58. [2025]

    arXiv:2503.14476 [cs.LG] https://arxiv.org/abs/2503.14476

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476 [cs.LG] https://arxiv.org/abs/2503.14476

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.