Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

Hao Zhang; Jianqiang Huang; Jiaxin Qi; Zhijiang Tang

REVIEW 3 major objections 5 minor 36 references

Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes SGU, a closed-loop evaluation score for unified multimodal models: the model describes an image, redraws it from its own caption, and answers the original questions from the redrawn image, with accuracy on that loop…

desk verdict SGU is a genuinely new closed-loop evaluation axis for unified multimodal models; the OCR-VQA numbers may partly measure text-rendering fidelity, and the paper needs tighter controls before the headline claim is safe. read the letter →

arxiv 2608.11907 v1 pith:TWGXJNF6 submitted 2026-08-12 cs.CV cs.AI

Hao Zhang , Jiaxin Qi , Zhijiang Tang , Jianqiang Huang This is my paper · ORCID

classification cs.CVcs.AI

keywords unifiedmultimodalmodelsself-generativeunderstandingclosed-loopevaluationvisualquestionansweringimagegenerationsystem-levelsemanticpreservationreasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Unified multimodal models are meant to combine visual understanding and image generation in one system, but existing benchmarks grade those two abilities separately. This paper argues that a model can score well on both yet still fail when the two must work together, and proposes Self-Generative-Understanding (SGU) to test the integrated loop. SGU is annotation-free: it reuses standard visual question-answering datasets, asking the model to caption an image, regenerate the image from its own caption, and then answer the original questions about the regenerated image. In experiments across four benchmarks and six models, accuracy under the loop drops far below direct question-answering accuracy, and models with similar base understanding diverge sharply on the looped score. The case the paper makes is that SGU exposes integration failures—information lost between seeing, describing, redrawing, and re-seeing—that separate component scores do not reveal.

What carries the argument

The machinery is the semantic closed-loop composition $M_U \circ M_G \circ M_U$ evaluated on VQA triplets, executed statelessly so each stage sees only the explicit intermediate artifact (caption or regenerated image) and no hidden state or memory is carried across stages. The supporting identity is a semantic-preservation upper bound: under the assumption that generation cannot introduce new task-solving evidence beyond what the understanding pathway extracted, direct VQA accuracy $s_\mathrm{base}$ bounds the SGU score from above, $s_\mathrm{umm}\le s_\mathrm{base}$. This makes $s_\mathrm{base}$ a model-specific reference, and the ratio $s_\mathrm{umm,r}=s_\mathrm{umm}/s_\mathrm{base}$ measures how much integrated capability survives the loop. The framework also defines answer matching for multiple-choice and open-form questions so the score can be instantiated on any existing VQA benchmark without new annotations.

What would settle it

Take a set of SGU samples and, after the model generates its caption, corrupt the regenerated image—for example, blur or crop exactly the region that contains the answer—while keeping the caption unchanged. If the final question-answering accuracy stays close to direct VQA accuracy on the original image, the loop is being solved from the caption or from language priors rather than from visual understanding of the regenerated image; that outcome would falsify the claim that SGU measures integrated visual understanding-and-generation.

Watch

Extended reading notes

Core claim

The central claim is that the SGU score, defined as $s_\mathrm{umm} = \mathbb{E}_{(v,q,a)\sim D}[\mathbb{I}(\mathrm{Match}(M_U(M_G(M_U(v)), q), a))]$, measures whether a unified multimodal model can carry task-relevant visual information through its own understanding-and-generation loop. The pipeline is a three-stage composition: the understanding function $M_U$ produces a caption $t_g$ from the original image $v$; the generation function $M_G$ reconstructs an image $\hat{v}$ from that caption alone; and $M_U$ again answers the original question $q$ using only $\hat{v}$, with the score being the expected match accuracy against ground-truth answers. A relative score $s_\mathrm{umm,r}=s_\mathrm{umm}/s_\mathrm{base}$ is used to compare how much performance survives the loop across models. The paper's key finding is that this closed-loop accuracy is consistently and often dramatically lower than direct VQA accuracy $s_\mathrm{base}$ on the original image, with the largest drops on visually grounded math and text-in-image questions. Stage-wise replacement experiments indicate the image-generation stage is a major bottleneck, and caption-only analyses show the intermediate textual description also loses information. The contribution is therefore an outcome-based, system-level evaluation signal that complements—rather than replaces—component-wise metrics for understanding and generation.

Load-bearing premise

The entire score depends on the assumption that correct answers on the reconstructed image genuinely require the model to have preserved and re-rendered the question-relevant visual evidence, rather than being obtained from language priors, information leaked through the caption, or self-consistency cues.

Editorial extensions

If this is right

A model's ranking on SGU can differ substantially from its ranking on direct VQA; in the reported data, models with similar direct accuracy differ by roughly 28 points on OCR-VQA, so system-level comparisons need a looped score rather than an aggregate of component scores.
Because replacing the generation stage with a stronger generator yields larger gains than replacing the captioning stage, current unified multimodal models' visual reconstruction is a principal bottleneck, and improving generation fidelity should directly raise SGU scores.
Since SGU is annotation-free and reuses standard VQA datasets, system-level evaluation can be scaled to any existing benchmark without human labeling or external judge models.
The relative score $s_\mathrm{umm}/s_\mathrm{base}$ separates the cost of the loop from base understanding ability, allowing practitioners to identify whether a model's failure comes from weak perception or from information loss in the integrated process.
Caption-length and caption-only analyses show the text bottleneck is not merely a token limit: increasing the caption limit from 256 to 512 tokens barely changes caption-only QA accuracy, so how the model organizes visual information matters more than caption verbosity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors leave implicit that the same closed-loop protocol could measure information flow quantitatively: comparing how much question-relevant evidence survives in the caption and in the regenerated image would decompose the SGU gap into understanding loss, generation loss, and reasoning loss.
A testable extension of the paper's logic is question-conditioned captioning: if the caption stage is allowed to see the question and SGU still does not approach $s_\mathrm{base}$, the bottleneck lies downstream in generation or self-reasoning rather than in the textual representation.
If SGU were adopted as a training objective, models might learn to regenerate images tuned for their own VQA success rather than for human-perceived fidelity; that reward-hacking risk is not addressed in the paper and would need to be checked against external human-judged generation benchmarks.
The image-swap shortcut check is run on one high-scoring subset; extending it across all four datasets and more model pairs would give stronger evidence that SGU images do not carry model-specific self-consistency cues.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Zhang et al. propose SGU: describe an image, regenerate it from your own caption, then answer the original VQA question on the regenerated image. The closed-loop design is new, and the consistent drop from direct VQA to SGU across all six models and four datasets is a real empirical pattern. This is a useful complementary evaluation axis, and the authors deserve credit for the stateless protocol, the stage-wise replacement experiments, and the prompt-sensitivity check. They also acknowledge the text bottleneck and that SGU is not a standalone theoretical criterion.

The soft spots are mostly about interpretation. On OCR-VQA the captioning stage can transcribe text, the generator can render it back into the image, and the final VQA can answer by reading the embedded text. That is a text round-trip shortcut, and it undercuts the claim that SGU measures 'semantic understanding of self-generated contexts' on text-heavy benchmarks. The image-swap test does not close this gap: it is run on a high-scoring MMBench subset (general perception, 198 samples) where text rendering is rarely the answer channel, and it only compares two models on the same rendered images. A text-masking control or an OCR-VQA-specific swap would be needed. Also, all headline numbers come from single runs with no error bars, and the prompt-robustness ablation is one model on one subset. No code or data is released, so the numbers are not independently verifiable.

The Appendix C.2 upper-bound proposition is circular: it assumes generation cannot introduce new task-solving semantic evidence and then concludes s_umm ≤ s_base. The proof essentially asserts the premise at Eq. (17). This matters less because s_base is only called a reference, not an unconditional guarantee, but it should be reframed.

The abstract's 'not captured by separate evaluations' is also stronger than the paper's own evidence: Figure 5 shows the SGU ranking is broadly consistent with an isolated average. SGU is a complementary signal, not a new capability that component metrics miss entirely.

Who is this for? People building or benchmarking unified multimodal models. It deserves a serious referee: the idea is simple, the protocol is reusable, and the negative result (models drop sharply on their own regenerated context) is worth publishing. But the authors need to add uncertainty quantification, address the text-rendering shortcut on OCR-VQA, and either fix or soften Appendix C.2 before I'd treat the numbers as reliable.

Referee Report

3 major / 5 minor

Summary. The paper proposes Self-Generative-Understanding (SGU), an evaluation framework for unified multimodal models (UMMs) that runs a three-stage closed loop: the model generates a textual description of an input image, generates a reconstructed image from that description, and then answers the original VQA question from the reconstructed image. The SGU score is the final VQA accuracy, optionally normalized by direct VQA accuracy on the original image. The authors evaluate six UMMs on four benchmarks (MMStar, MMBench, MathVista, OCR-VQA) and report a consistent drop from direct VQA accuracy to SGU accuracy, plus stage-wise replacement experiments, a prompt-sensitivity study, an image-swap shortcut check, and auxiliary CLIP-based similarities. The central claim is that SGU provides a complementary, annotation-free, system-level evaluation that exposes integration failures not captured by separate understanding or generation metrics.

Significance. If the SGU score measures what it claims, it is a useful complementary evaluation axis: it is annotation-free, reuses existing VQA benchmarks, is model-internal with no external judge, and its stage-wise diagnostics can localize bottlenecks. The empirical pattern across six models and four datasets is consistent and the stateless protocol is carefully described. However, the validity of the metric is not yet established for text-centric benchmarks because of a possible text round-trip shortcut, and the statistical support is limited to single-run point estimates. The proposition in Appendix C.2 is circular and formally under-specified. With additional controls and variance reporting, the framework could become a valuable benchmarking tool, but the current evidence is insufficient to support the strong interpretation placed on the OCR-VQA results.

major comments (3)

[§4.2 (Q4) and Table 4] The text round-trip shortcut is not controlled. On OCR-VQA, Phase 2 can render text from the caption into the reconstructed image, and Phase 3 can then answer by reading embedded text rather than by semantic visual understanding of the regenerated scene. The image-swap test is run only on a high-scoring MMBench subset of 198 general-perception samples and compares models on identical rendered images, so it cannot detect this channel; it also does not include OCR-VQA. I request a text-masking or text-removal control (e.g., blanking detected text regions of the reconstructed image before final VQA), a caption-only control without any image, and an OCR-VQA-specific breakdown or image-swap test. Without these, the claim that SGU measures integrated semantic understanding-generation on text-centric benchmarks is not established.
[§4.1 (Evaluation setup) and Table 1] Every reported s_base and s_umm is a single-run point estimate with no confidence intervals or significance tests. The headline example (OCR-VQA: UniWorld 81.33→28.67 vs OmniGen2 79.39→56.59) is presented as a robust cross-model difference, but generation is stochastic (temperature 1.0 for autoregressive models, diffusion sampling for others), and no seed sweep or variance estimate is reported. Table 3's prompt-sensitivity study covers one model (OmniGen2) on one 198-sample MMBench subset, so it does not establish stability across the six models and four datasets in Table 1. Please report means and standard deviations over at least 3–5 seeds for the full protocol, or at least for the constructed subsets in Appendix E, and state the sample sizes of the full splits.
[Appendix C.2, Eq. (10)] The proposition's proof is circular and formally under-specified. The assumption states that generation cannot introduce new task-solving semantic evidence, and Eq. (17) is essentially this assumption restated; the proof then derives G_SGU ⊆ G_base from it. Additionally, B_d is defined as a set of subsets s ⊆ H_v such that Match(M_U(s,q),a)=1, but M_U is defined on images, not on semantic subsets, so Eqs. (11)–(20) are type-inconsistent. The text immediately after the proof concedes that 'rare cases may occur' where s_umm > s_base, contradicting the universal inequality in Eq. (10). I recommend removing the proposition label and presenting s_base only as a heuristic model-specific reference, or supplying a correct proof under explicit conditions under which the inequality actually holds.

minor comments (5)

[Abstract and §1] Calling SGU 'zero-cost' is misleading: the pipeline runs six models on four benchmarks and includes image generation, which is computationally expensive. 'Annotation-free' is the accurate and sufficient claim.
[Table 1] The row labeled UPPERBOUND is never defined in the text; specify that it is the maximum s_base across models per dataset, and note that it is not an upper bound on s_umm in general.
[Table 3 header] The header 'Und./Gen. Promptdenote' is missing a space; it should read 'Und./Gen. Prompts' or 'Und./Gen. Prompt denotes'.
[Table 4 caption] The caption says 'We reports Δ'; it should be 'We report Δ'.
[Eq. (4)] For open-form answers, specify the matching procedure in more detail, including whether numerical equivalence and paraphrases are handled; the current description ('parsing and normalization') may not cover semantically equivalent free-form answers.

Circularity Check

1 steps flagged · score 2.0 of 10

Main SGU benchmark evaluation is self-contained; the only reduction is the Appendix C.2 upper-bound proposition, which restates its semantic-preservation premise.

self definitional [Appendix C.2, Proposition (upper-bound reference under semantic preservation), Eqs. (10)-(22).]
"Assume that the SGU loop does not introduce new task-solving semantic evidence beyond what is extracted from the original image by the model’s understanding pathway. Under this assumption, direct VQA accuracy on the original image serves as an upper-bound reference for the SGU score: s_umm ≤ s_base. ... Under the semantic-preservation assumption, the generation process cannot introduce new task-solving semantic evidence beyond what is encoded in m_U(v∗). Therefore, (m_G◦m_U(v∗)) ∩ B_d∗ ⊆ m_U(v∗) ∩ B_d∗."

The proposition is an explicit conditional, but its proof's key step (Eq. 16) is the semantic-preservation assumption itself: 'the generation process cannot introduce new task-solving semantic evidence beyond what is encoded in m_U(v∗)', rewritten as a containment of proof-relevant semantic sets. Eq. (17) then merely applies the definition of G_SGU—a loop-correct sample is one whose loop output contains the required evidence—to convert that containment into G_SGU ⊆ G_base, i.e., s_umm ≤ s_base. No independent evidence for the assumption is supplied, so the claimed upper bound is the premise expressed in accuracy notation rather than a derived theorem.

full rationale

The central SGU protocol is an operational definition plus an empirical measurement, not a derivation that reduces to its inputs: Eq. (3)-(4) define s_umm as VQA accuracy on the model's own reconstructed image, and Table 1 reports measured accuracies on external benchmarks (MMStar, MMBench, MathVista, OCR-VQA). No parameter is fitted and then renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem appears. The only identifiable circular step is the Appendix C.2 'upper-bound reference' proposition, where the proof's containing-set relation (Eq. 16) is just the semantic-preservation assumption restated, so the conclusion s_umm ≤ s_base follows by definition of the SGU-correct set rather than from independent evidence. Because that proposition is ancillary—the paper's main findings stand on the measured SGU drops and controlled replacement experiments—the overall circularity score is low. Concerns about text-rendered answers in OCR-VQA are validity threats to the metric's interpretation, not circularity, and are therefore not counted here.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central measurement depends on several hand-chosen inference settings (caption length, CFG, resolution, steps, seed) and small ablation subsets, plus domain assumptions about what reconstructed-image QA measures. No new physical or ontological entities are introduced; SGU is a metric and protocol, not a new force, particle, or conserved quantity.

free parameters (6)

caption length max_new_tokens = 256
Chosen as a unified resource constraint for the image-to-text stage; SGU scores and caption-only QA depend on this budget (Section B.1, Table 12).
CFG scale = 5.0
Default classifier-free guidance for diffusion and flow generation; affects reconstruction fidelity and therefore downstream VQA (Section B.1).
generation resolution = 512x512
Fixed image size for reconstructed context; resolution bounds the visual detail available to the final VQA stage.
diffusion inference steps = 30
Steps for diffusion and flow models; the quality of the reconstructed image depends on this choice.
random seed = 42
Used for generation sampling and subset construction; no seed sweep is reported, so sensitivity to stochasticity is not measured.
ablation subset sizes = 198 to 200 samples
Fixed stratified subsets for stage-replacement, prompt-sensitivity, and image-swap analyses; main tables use full splits but ablations rely on these small subsets.

assumptions (4)

ad hoc to paper Answering a VQA question correctly on the reconstructed image is a valid outcome-based proxy for the integrated understanding-and-generation loop.
This is the core measurement assumption of SGU introduced in Section 3.2-3.3 and Eq. (4). If reconstructed-image QA can be solved by language priors or caption remnants, the score overstates or misstates integrated capability.
domain assumption A textual caption is a sufficient intermediate representation for the question-relevant visual content.
SGU instantiates the loop with text as the bottleneck (Section 3.2, Appendix G). The authors acknowledge information loss, but the framework's interpretation depends on the caption retaining all answer-critical details.
ad hoc to paper The generation process cannot introduce new task-solving semantic evidence beyond what is extracted by the model's understanding pathway.
Used in Appendix C.2 to prove s_umm <= s_base; this premise already contains the proposition's conclusion (Eq. (10)).
domain assumption Stateless execution prevents hidden-state leakage and shortcut use across stages.
Assumed in Section 3.3 and Appendix B.3; if models encode cross-turn memory or use prompt-internal bias, the closed-loop isolation fails. Image-swap tests partially probe this on one subset.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models." pith.science (2026). https://pith.science/paper/TWGXJNF6

@misc{pith2026260811907,
  author       = {Pith},
  title        = {Pith review of: Do You See What You Draw? A Semantic Closed-Loop Framework for Holistic Evaluation of Unified Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TWGXJNF6}},
  note         = {Machine review of arXiv:2608.11907}
}

read the original abstract

As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified multimodal models (UMMs). In this work, we propose Self-Generative-Understanding (SGU), a novel, annotation-free evaluation framework that probes the integrated capabilities of unified models through a semantic closed-loop challenge. Without requiring new annotations, SGU leverages the dual understanding-and-generation abilities of UMMs by asking them to first perceive an image and produce a textual description, subsequently reconstruct a visual context based on that description, and finally perform reasoning over the self-generated output. This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems. Extensive experiments show that even high-performing UMMs often struggle to reason over their own generated contexts, revealing limitations that are not captured by separate evaluations of understanding or generation alone. Our work provides a complementary holistic evaluation framework and offers a foundation for benchmarking the development of next-generation unified multimodal models.

Figures

Figures reproduced from arXiv: 2608.11907 by the authors.

**Figure 2.** Overview of the Self-Generative-Understanding (SGU) protocol. Given a VQA triplet [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Circular bar plot of summ,r. To further compare SGU with conventional evaluation signals, we compute an isolated average score by aggregating commonly used component-level indicators, including direct VQA accuracy sbase and CLIPScore [11] between the original image and the reconstructed image or generated description. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Qualitative case studies under the SGU protocol. The examples illustrate different loop [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Comparison of ranking performance across average SGU score summ, average normalized ratio summ,r, and an isolated average aggregated from CLIP-T, CLIP-I, and sbase. 0 10 20 30 40 50 60 70 80 UniWorld-V1 Janus-Pro-7B Show-o2-7B Ovis-U1 BAGEL-7B OmniGen2 A c c u r a c …

**Figure 7.** Figure 7: Abstraction-level illustration of SGU success and failure. (a) The understanding–generation [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: Additional SGU case examples showing the complete loop from the input VQA sample to [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 5 canonical work pages

[1]

Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

arXiv 2025
[2]

Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025. URL https://arxiv.org/abs/2505.09568

arXiv 2025
[3]

Are we on the right way for evaluating large vision-language models?, 2024

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models?, 2024. URLhttps://arxiv.org/abs/2403.20330

arXiv 2024
[4]

Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025. URLhttps://arxiv.org/abs/2501.17811

arXiv 2025
[5]

Emerging properties in unified multimodal pretraining, 2025

Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining, 2025. URLhttps://arxiv.org/abs/2505.14683

arXiv 2025
[6]

Agent ai: Surveying the horizons of multimodal interaction, 2024

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi V o, Li Fei-Fei, and Jianfeng Gao. Agent ai: Surveying the horizons of multimodal interaction, 2024. URL https://arxiv.org/abs/2401.03568

arXiv 2024
[7]

Mme: A comprehensive evaluation benchmark for multimodal large language models, 2025

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2025. URL https://arxiv.org/abs/2306.13394

arXiv 2025
[8]

X-omni: Reinforcement learning makes discrete autoregressive image generative models great again, 2025

Zigang Geng, Yibing Wang, Yeyao Ma, Chen Li, Yongming Rao, Shuyang Gu, Zhao Zhong, Qinglin Lu, Han Hu, Xiaosong Zhang, Linus, Di Wang, and Jie Jiang. X-omni: Reinforcement learning makes discrete autoregressive image generative models great again, 2025. URL https://arxiv.org/abs/2507.22058

arXiv 2025

Show all 36 references

[9]

Geneval: An object-focused framework for evaluating text-to-image alignment, 2023

Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment, 2023. URL https://arxiv.org/abs/2310.11513

2023 arXiv
[10]

URL https://developers.googleblog.com/en/introducing-gemini-2-5- flash-image/

Google DeepMind, 2025. URL https://developers.googleblog.com/en/introducing-gemini-2-5- flash-image/

2025
[11]

Clipscore: A reference-free evaluation metric for image captioning, 2022

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022. URL https://arxiv.org/abs/ 2104.08718

2022 arXiv
[12]

Gans trained by a two time-scale update rule converge to a local nash equilibrium

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InAdvances in Neural Information Processing Systems, volume 30, 2017. 10

2017
[13]

Gir-bench: Versatile benchmark for generating images with reasoning, 2025

Hongxiang Li, Yaowei Li, Bin Lin, Yuwei Niu, Yuhang Yang, Xiaoshuang Huang, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Long Chen. Gir-bench: Versatile benchmark for generating images with reasoning, 2025. URLhttps://arxiv.org/abs/2510.11026

2025
[14]

Rover: Benchmarking reciprocal cross-modal reasoning for omnimodal generation, 2025

Yongyuan Liang, Wei Chow, Feng Li, Ziqiao Ma, Xiyao Wang, Jiageng Mao, Jiuhai Chen, Jiatao Gu, Yue Wang, and Furong Huang. Rover: Benchmarking reciprocal cross-modal reasoning for omnimodal generation, 2025. URLhttps://arxiv.org/abs/2511.01163

2025
[15]

Uniworld-v1: High-resolution semantic encoders for unified visual understanding and generation, 2025

Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, Yatian Pang, and Li Yuan. Uniworld-v1: High-resolution semantic encoders for unified visual understanding and generation, 2025. URL https:// arxiv.org/abs/2506.03147

2025 arXiv
[16]

Evaluating text-to-visual generation with image-to-text generation

Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer, 2024

2024
[17]

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URLhttps://arxiv.org/abs/2210.02747

2023 arXiv
[18]

Umnibench: Unified understand and generation model oriented omni-dimensional benchmark, 2025

Kai Liu, Leyang Chen, Wenbo Li, Zhikai Chen, Zhixin Wang, Renjing Pei, Linghe Kong, and Yulun Zhang. Umnibench: Unified understand and generation model oriented omni-dimensional benchmark, 2025. URLhttps://arxiv.org/abs/2512.17196

2025
[19]

Mmbench: Is your multi-modal model an all-around player?, 2024

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. URLhttps://arxiv.org/abs/2307.06281

2024 arXiv
[20]

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/ 2310.02255

2024 arXiv
[21]

Ocr-vqa: Visual question answering by reading text in images

Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In2019 International Conference on Document Analysis and Recognition (ICDAR), pages 947–952, 2019. doi: 10.1109/ICDAR. 2019.00156

2019
[22]

URL https://openai.com/index/image-generation-api/

OpenAI, 2025b. URL https://openai.com/index/image-generation-api/
[23]

Transfer between modalities with metaqueries, 2025

Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Ji Hou, and Saining Xie. Transfer between modalities with metaqueries, 2025. URLhttps://arxiv.org/abs/2504.06256

2025 arXiv
[24]

Chameleon: Mixed-modal early-fusion foundation models, 2025

Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2025. URL https://arxiv.org/abs/2405.09818

2025 arXiv
[25]

Ovis-u1 technical report, 2025

Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, Yang Li, and Qing-Guo Chen. Ovis-u1 technical report, 2025. URLhttps://arxiv.org/abs/2506.23044

2025 arXiv
[26]

From understanding the world to intervening in it: A unified multi-scale framework for embodied cognition, 2025

Maijunxian Wang. From understanding the world to intervening in it: A unified multi-scale framework for embodied cognition, 2025. URLhttps://arxiv.org/abs/2503.00727

2025 arXiv
[27]

Emu3: Next-token prediction is all you need, 2024

Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tie...

2024 arXiv
[28]

Skywork unipic 2.0: Building kontext model with online rl for unified multimodal model, 2025

Hongyang Wei, Baixin Xu, Hongbo Liu, Cyrus Wu, Jie Liu, Yi Peng, Peiyu Wang, Zexiang Liu, Jingwen He, Yidan Xietian, Chuanxin Tang, Zidong Wang, Yichen Wei, Liang Hu, Boyi Jiang, William Li, Ying He, Yang Liu, Xuchen Song, Eric Li, and Yahui Zhou. Skywork unipic 2.0: Building ...

2025
[29]

Ggbench: A geometric generative reasoning benchmark for unified multimodal models, 2026

Jingxuan Wei, Caijun Jia, Xi Bai, Xinglong Xu, Siyuan Li, Linzhuang Sun, Bihui Yu, Conghui He, Lijun Wu, and Cheng Tan. Ggbench: A geometric generative reasoning benchmark for unified multimodal models, 2026. URLhttps://arxiv.org/abs/2511.11134

2026
[30]

Qwen-image technical report, 2025

Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai...

2025 arXiv
[31]

Omnigen2: Exploration to advanced multimodal generation, 2025

Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Ex...

2025 arXiv
[32]

Reconstruction alignment improves unified multimodal models, 2025

Ji Xie, Trevor Darrell, Luke Zettlemoyer, and XuDong Wang. Reconstruction alignment improves unified multimodal models, 2025. URLhttps://arxiv.org/abs/2509.07295

2025 arXiv
[33]

Show-o2: Improved native unified multimodal models, 2025

Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models, 2025. URLhttps://arxiv.org/abs/2506.15564

2025 arXiv
[34]

Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

2024 arXiv
[35]

Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computatio...

2025
[36]

Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024

Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024. URL https://arxiv.org/ abs/2408.11039. A Data...

2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

arXiv 2025

[2] [2]

Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025. URL https://arxiv.org/abs/2505.09568

arXiv 2025

[3] [3]

Are we on the right way for evaluating large vision-language models?, 2024

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models?, 2024. URLhttps://arxiv.org/abs/2403.20330

arXiv 2024

[4] [4]

Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025. URLhttps://arxiv.org/abs/2501.17811

arXiv 2025

[5] [5]

Emerging properties in unified multimodal pretraining, 2025

Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining, 2025. URLhttps://arxiv.org/abs/2505.14683

arXiv 2025

[6] [6]

Agent ai: Surveying the horizons of multimodal interaction, 2024

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi V o, Li Fei-Fei, and Jianfeng Gao. Agent ai: Surveying the horizons of multimodal interaction, 2024. URL https://arxiv.org/abs/2401.03568

arXiv 2024

[7] [7]

Mme: A comprehensive evaluation benchmark for multimodal large language models, 2025

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, Rongrong Ji, Caifeng Shan, and Ran He. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2025. URL https://arxiv.org/abs/2306.13394

arXiv 2025

[8] [8]

X-omni: Reinforcement learning makes discrete autoregressive image generative models great again, 2025

Zigang Geng, Yibing Wang, Yeyao Ma, Chen Li, Yongming Rao, Shuyang Gu, Zhao Zhong, Qinglin Lu, Han Hu, Xiaosong Zhang, Linus, Di Wang, and Jie Jiang. X-omni: Reinforcement learning makes discrete autoregressive image generative models great again, 2025. URL https://arxiv.org/abs/2507.22058

arXiv 2025