Pith. sign in

REVIEW 6 major objections 5 minor 1 cited by

OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks

T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniGenBench constructs 57 generation tasks spanning perception and cognition, and its dual-mode evaluation finds GPT-4o-Native ahead on every dimension.

desk verdict A genuinely broad generation benchmark with a useful reverse-engineering pipeline; the evaluation rig has real gaps, but the resource deserves a serious referee. read the letter →

arxiv 2505.18775 v1 pith:OXKZ2EL4 submitted 2025-05-24 cs.CV cs.AI

Jiayu Wang , Yang Jiao , Yue Yu , Tianwen Qian , Shaoxiang Chen , Jingjing Chen , Yu-Gang Jiang This is my paper
classification cs.CVcs.AI
keywords OmniGenBenchtext-to-imagegenerationinstructionfollowinglargemultimodalmodelsLLM-as-a-judgeScoreimagebenchmarkevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OmniGenBench aims to answer a question existing text-to-image benchmarks cannot: which models actually follow general-purpose, real-world instructions when asked to generate an image, rather than only simple object-attribute prompts. The paper builds 57 generation sub-tasks spanning perception (appearance compliance, dynamics consistency) and cognition (world knowledge, situational reasoning, spatial reasoning, STEM-driven reasoning), systematically categorized by the capability each demands. It then evaluates nine mainstream generative models with a dual-mode protocol: off-the-shelf visual parsers check perception-centric outputs, and a Gemini-2.5-Pro judge with task-specific criteria scores cognition-centric outputs, all combined into a single OmniScore that weights instruction consistency above realism and aesthetics. If the benchmark is reliable, the result is a capability map of the field: GPT-4o-Native leads on all six capability axes, Gemini-2.0 is second overall, Seedream3 is the strongest closed model on perception alone, and open-source models trail by a wide margin. The paper's contribution is therefore not a new generator but a reusable, human-aligned way to measure instruction-following generation across more than 50 real-world task types.

What carries the argument

The load-bearing device is the dual-mode evaluation protocol with its single aggregate score. Perception-centric tasks are checked by off-the-shelf scene parsers, segmentation and dense-prediction tools that extract object-level information from the generated image and compare it against the elements the instruction requires. Cognition-centric tasks are scored by an LLM-as-judge: a two-stage pipeline first distills each task's key visual requirement from a batch of prompts, then derives the specific scene elements a given instruction demands; Gemini-2.5-Pro receives those criteria alongside the generated image and returns scores for Consistency, Realism, and Aesthetics Quality. These are collapsed into OmniScore, with instruction consistency weighted eight times as heavily as each of the two quality terms, which is what makes the ranking an instruction-following ranking rather than an aesthetics ranking.

What would settle it

Re-score the benchmark with an independent judge: run the same instructions through a second judge model, or through full human annotation on a random sample that explicitly includes cases where the original annotators disagreed, and re-rank the models. If GPT-4o-Native no longer leads on most axes, or if a model such as Seedream3 overtakes it on perception-centric tasks under parser-free human scoring, the claim that OmniGenBench reliably measures instruction-following ability, and the resulting ranking, is falsified.

Watch

Extended reading notes

Core claim

The paper claims that instruction-following image generation can be measured broadly and reliably by reversing an understanding benchmark into a generation benchmark. Starting from an established multimodal understanding task taxonomy, the authors classify each task as reversible or irreversible, rewrite the reversible tasks as image-generation instructions through a GPT-4o-assisted prompt-generation pipeline, and keep only the examples that three human annotators unanimously approve. Evaluation is dual-mode: perception-centric tasks are scored by off-the-shelf visual parsing tools in the style of GenEval, while cognition-centric tasks are judged by Gemini-2.5-Pro using task-specific key reference points derived by a two-stage prompting pipeline that first identifies each task's key requirement and then extracts the required scene elements for a given instruction. The four assessment dimensions are combined as OmniScore, with consistency weighted eight times as heavily as realism and aesthetic quality, deliberately making instruction adherence the dominant factor. On this benchmark GPT-4o-Native achieves the highest OmniScore on every capability axis, which the paper reads as evidence that unified multimodal generation models, rather than dedicated diffusion models, are currently the strongest general-purpose instruction followers.

Load-bearing premise

The entire model ranking rests on one premise: that OmniScore, computed from Gemini-2.5-Pro's judgments and off-the-shelf parsers, measures instruction alignment accurately across all 57 tasks, when the reported human-agreement check was performed only on a subset of samples where all three annotators happened to agree.

Editorial extensions

If this is right

  • Unified multimodal models such as GPT-4o-Native and Gemini-2.0 beat dedicated text-to-image diffusion models on nearly every dimension, so general instruction-following ability, not raw rendering quality, is now the main axis of model comparison.
  • Perception tasks with dense multi-object prompts and attribute bindings remain the weak spot for most generative models, pointing to a concrete target for training-data and architecture improvements.
  • Cognition-centric tasks, especially STEM-driven and situational reasoning, are the hardest category for every evaluated model, making reasoning-to-visualization the current bottleneck in image generation.
  • Open-source models lag closed models by large margins on nearly all metrics, indicating a gap in scale and instruction data that open-weight releases have not yet closed.
  • Because the task taxonomy and per-task evaluation prompts are fixed, future models can be scored on OmniGenBench without re-annotating the benchmark, giving a stable yardstick for tracking progress.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reverse-engineering pipeline, converting understanding tasks into generation prompts via an LLM plus human filtering, is itself a transferable method: any VQA-style benchmark with reversible tasks could be converted into a generation benchmark, so the construction recipe may matter as much as the specific task list.
  • Because OmniScore weights consistency at 0.8, the ranking is deliberately biased toward instruction adherence; a reader should expect that re-weighting toward realism or aesthetics would reshuffle the mid-tier models more than it would threaten the top position.
  • The state-at-a-specified-time tasks, such as aging a dog from 2015 to 2020, probe temporal commonsense in image generation; if such tasks become standard, they double as a simple behavioral check of whether a generative model has anything like a world model.
  • The paper's finding that GPT-4o-Native also wins perception-centric tasks, where dedicated diffusion models were expected to retain an edge, challenges the assumption that task-specific architectures dominate their specialty; a testable extension is tracking whether this gap narrows as unified open models scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper introduces OmniGenBench, a benchmark of 57 sub-tasks (the taxonomy actually lists more) for evaluating instruction-following in image generation. The benchmark is organized into perception-centric and cognition-centric categories, and a dual-mode evaluation is proposed: off-the-shelf parsers for perception tasks and an LLM-as-a-judge (specifically Gemini-2.5-Pro) for cognition tasks. The main empirical claim is that GPT-4o-Native is the state-of-the-art model across all six capability dimensions, based on scores in Table 1. The paper also reports human-agreement rates for OmniScore in Fig. 5 and provides qualitative examples in the appendix.

Significance. If the evaluation is valid, OmniGenBench would be a useful, broad resource for the community, covering a wider range of generation tasks than existing benchmarks such as GenEval or T2I-CompBench. The paper includes a code/data link and a reproducible evaluation protocol. However, the central ranking claim (GPT-4o-Native as SOTA) is currently not fully supported because the evaluation metric's validity across all tasks is not established. The paper's strengths are its broad task coverage and the effort to build a task-specific evaluation pipeline. The main weakness is that the validity arguments rest on an incompletely specified perception parser setup and on a human-agreement study with a strong selection bias.

major comments (6)
  1. [Section 3.3 / Eq. (1)] OmniScore is defined as 0.8*Consistency + 0.1*Realism + 0.1*Aesthetics Quality. The weights are asserted without justification or sensitivity analysis. Since the final model ranking depends on this weighted sum, the paper should report results with alternative weightings (e.g., equal weights, or weights varying within a reasonable range) to demonstrate that the ranking is robust. Without such an analysis, the choice of weights is free and could be tuned to produce a desired ordering.
  2. [Section 4.2 / Fig. 5] The human-agreement validation only retains samples for which all three annotators agreed, and then reports agreement between OmniScore and human judgments on that subset. This is a selection bias: it excludes exactly the ambiguous cases where the judge is most likely to disagree with human raters. The reported agreement rates (0.84–0.98) are therefore an upper bound on agreement over the full distribution. The authors should report inter-annotator agreement (e.g., Fleiss' kappa) on all sampled items, and compute OmniScore agreement on the full sample, including non-unanimous cases.
  3. [Section 3.3 / Table 1] The perception-centric evaluation is claimed to follow the protocols of GenEval [13] and [40], but the paper does not specify which parser is used for each perception sub-task. Several tasks in the Appearance Compliance category (e.g., Scene Text Generation, Poster Generation, Academic Document Generation, Absolute Spatial Positioning) require verifying content that Mask2Former/DPT-style parsers are not designed to check. The paper must provide a per-task description of the parsing tool, its inputs/outputs, and its per-task accuracy. Without this, the perception scores are not reproducible and their validity is unknown.
  4. [Section 4.1 / Table 1] The claim that "GPT-4o-Native demonstrates state-of-the-art performance across all dimensions" is contradicted by the paper's own Table 1, where Imagen3 scores higher on World Knowledge (85.1 vs. 85.0). Additionally, multiple scores are missing (e.g., SD3, Seedream3, and Imagen3 in Spatial Reasoning; SD3 and Imagen3 in Situational Reasoning), with no explanation of how missing entries are handled when computing category averages. The overclaim should be corrected, and the treatment of missing data should be explicitly described.
  5. [Section 3.2 / Fig. 3] GPT-4o is used to auto-generate the benchmark prompts, after human filtering. This introduces a potential confound: the prompt distribution may be aligned with GPT-4o's language style and generation biases, which could advantage GPT-4o-Native in the evaluation. The human filtering reduces but does not eliminate this concern. The authors should provide evidence that the ranking is not driven by this bias, e.g., by evaluating on a small held-out set of human-written prompts or by analyzing prompt difficulty across sources.
  6. [Section 3.3 / Fig. 4] The evaluation criteria for cognition-centric tasks are both automatically generated by Gemini-2.5-Pro and then used by Gemini-2.5-Pro to score the images. This creates a potential self-confirmation bias: the judge may favor images that adhere to its own generated criteria, and the criteria may not match human expectations in ambiguous cases. The authors should either use a different model for scoring than for criterion generation, or provide independent validation of the criteria (e.g., human assessment of the criteria themselves).
minor comments (5)
  1. [Section 3.1 / Table 2] The paper claims 57 sub-tasks, but Table 2 lists 65 task identifiers (12 + 11 + 10 + 14 + 10 + 8). Please reconcile the count.
  2. [Fig. 17] The task numbering in Fig. 17 is inconsistent with Table 2 (e.g., "Task 5.10 Relational Scene Composition" vs. Table 2's "5.9 Relational Spatial Positioning Composition" and "5.10 Absolute Spatial Positioning Generation"). Please align the numbering.
  3. [Section 2.2 / References] The reference list includes [12] Seedream 3.0, but the experiments mention "SeedDream 3" and "Seedream3" interchangeably; please standardize the spelling.
  4. [Appendix B / Fig. 8] Typo in Fig. 8: "vancient Egyptian hieroglyphs" should be "ancient Egyptian hieroglyphs". Also, in Fig. 13, "Rules and Procedures Vvisualization" should be "Rules and Procedures Visualization".
  5. [Section 4.1 / Table 1] The table header uses "Spa. Rea." and "Sit. Rea." for Spatial and Situational Reasoning, which is clear, but the paper does not state how many samples per task are used or whether scores are averages over multiple runs. Please add the number of prompts per category and, if possible, error bars or statistical significance tests.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's evaluation protocol is operational, and the model ranking is computed from measured outputs rather than being identical to any fitted input.

full rationale

The paper's central claim—that OmniGenBench is a broad benchmark and that GPT-4o-Native performs best on it—does not reduce to its own inputs. The benchmark prompts are constructed from MegaBench tasks via GPT-4o generation plus human filtering; this is an external data-curation step, not a definitional equivalence. OmniScore in Eq. (1) is an operational weighted combination of Consistency, Realism, and Aesthetics, where Consistency is obtained from off-the-shelf parsers for perception tasks and from a Gemini-2.5-Pro judge for cognition tasks. These scores are measured outputs, not parameters fitted to reproduce the conclusion. The 0.8/0.1/0.1 weights are arbitrary choices, but they are not inferred from the data and do not force the reported ranking. The human-agreement study in Section 4.2 retains only samples where all three annotators agreed; that is a validity limitation and a possible source of optimism bias, but it is an external empirical check rather than a circular step, because OmniScore is not defined as those agreed human labels. The self-citations in the paper (e.g., refs. [23] and [24]) appear in related-work and motivation contexts and are not load-bearing for the benchmark's construction or for the evaluation chain. Likewise, the fact that GPT-4o was used to draft prompts and Gemini was used to derive evaluation criteria may create measurement bias, but it does not make the claimed ranking logically equivalent to the benchmark's inputs. No equation, fitted parameter, or self-citation chain exhibits the required reduction for a circularity finding.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on the assumption that reverse-engineering understanding tasks from MegaBench, filtering GPT-4o-generated prompts, and scoring with a Gemini-based judge yields a valid measure of instruction-following. The OmniScore weights are arbitrary. No new physical or mathematical entities are introduced.

free parameters (1)
  • OmniScore weights = 0.8 Consistency, 0.1 Realism, 0.1 Aesthetics
    Chosen by hand in Eq. (1); no sensitivity analysis or justification beyond prioritizing instruction adherence.
assumptions (4)
  • domain assumption MegaBench task taxonomy is a valid basis for generation tasks
    The benchmark is built by reversing MegaBench understanding tasks; if MegaBench's categorization is not representative of real generation needs, the benchmark inherits that bias (Sec 3.2).
  • domain assumption GPT-4o-generated prompts, after human filtering, are diverse and unbiased
    The prompt construction uses GPT-4o to auto-generate queries (Sec 3.2); human filtering reduces but does not eliminate model-specific bias in the task distribution.
  • domain assumption LLM-as-a-judge with Gemini-2.5-Pro can reliably rate generated images
    The cognition-centric evaluation relies on Gemini-2.5-Pro to derive criteria and score images (Sec 3.3); no per-task validation of judge accuracy is provided.
  • domain assumption Off-the-shelf visual parsers correctly evaluate perception-centric tasks
    Perception-centric evaluation follows GenEval-style parsing (Sec 3.3); parser errors on complex scenes would corrupt those scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks." pith.science (2026). https://pith.science/paper/OXKZ2EL4

@misc{pith2026250518775,
  author       = {Pith},
  title        = {Pith review of: OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OXKZ2EL4}},
  note         = {Machine review of arXiv:2505.18775}
}
read the original abstract

Recent breakthroughs in large multimodal models (LMMs), such as the impressive GPT-4o-Native, have demonstrated remarkable proficiency in following general-purpose instructions for image generation. However, current benchmarks often lack the necessary breadth and depth to fully evaluate the diverse capabilities of these models. To overcome this limitation, we introduce OmniGenBench, a novel and comprehensive benchmark meticulously designed to assess the instruction-following abilities of state-of-the-art LMMs across both perception-centric and cognition-centric dimensions. Our OmniGenBench includes 57 diverse sub-tasks grounded in real-world scenarios, systematically categorized according to the specific model capabilities they demand. For rigorous evaluation, we further employ a dual-mode protocol. This protocol utilizes off-the-shelf visual parsing tools for perception-centric tasks and a powerful LLM-based judger for cognition-centric tasks to assess the alignment between generated images and user instructions. Using OmniGenBench, we evaluate mainstream generative models, including prevalent models like GPT-4o, Gemini-2.0-Flash, and Seedream, and provide in-depth comparisons and analyses of their performance.Code and data are available at https://github.com/emilia113/OmniGenBench.

Figures

Figures reproduced from arXiv: 2505.18775 by the authors.

Figure 1
Figure 1. Evaluation of leading generative models in handling the general-purpose instructional task. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The proposed taxonomy of OmniGenBench, accompanied by representative tasks sampled [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The data curation pipeline of our OmniGenBench. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Illustration of the evaluation pipeline of our OmniGenBench. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results of agreement rate between our OmniScore and human evaluation. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison of four leading text-to-image generation methods on the OmniGen [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Examples of tasks from the World Knowledge Anchored Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Examples of tasks from the World Knowledge Anchored Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Examples of tasks from the Situational Reasoning Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Examples of tasks from the Situational Reasoning Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Examples of tasks from the Spatial Reasoning Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Examples of tasks from the Spatial Reasoning Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Examples of tasks from the STEM-Driven Reasoning Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Examples of tasks from the STEM-Driven Reasoning Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Examples of tasks from the STEM-Driven Reasoning Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Examples of tasks from the Appearance Compliance Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Examples of tasks from the Appearance Compliance Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Examples of tasks from the Dynamics Consistency Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]
Figure 19
Figure 19. Figure 19: Examples of tasks from the Dynamics Consistency Generation section, along with the corresponding outputs generated by different models for qualitative comparison. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DynEval: Holistic Evaluations of T2I Generative Models in the Wild

    cs.CV 2026-07 conditional novelty 6.5 of 10

    DynEval distills a 235B teacher VLM into 2B/4B evaluators via 250K synthetic instruction triplets, yielding higher human correlation than existing T2I metrics while enabling open-set dynamic QA and scene-graph quality checks.

Reference graph

Works this paper leans on

57 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [13]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  2. [40]

    Evolving storytelling: benchmarks and methods for new character customization with diffusion models

    Xiyu Wang, Yufei Wang, Satoshi Tsutsui, Weisi Lin, Bihan Wen, and Alex Kot. Evolving storytelling: benchmarks and methods for new character customization with diffusion models. InProceedings of the 32nd ACM International Conference on Multimedia, pages 3751–3760, 2024

    work page 2024

  3. [1]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    arXiv 2023

  4. [2]

    Imagen 3.arXiv preprint arXiv:2408.07009, 2024

    Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Lluis Castre- jon, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, et al. Imagen 3.arXiv preprint arXiv:2408.07009, 2024

    arXiv 2024

  5. [3]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  6. [4]

    Flux1.1 [pro]: Advanced text-to-image generation

    Black Forest Labs. Flux1.1 [pro]: Advanced text-to-image generation. https://bfl.ai/ models/flux-pro, 2024. Accessed: 2025-05-16

    work page 2024

  7. [5]

    Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks.arXiv preprint arXiv:2410.10563, 2024

    Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengqing Wang, Kai Wang, Yubo Wang, Yuan- sheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, et al. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks.arXiv preprint arXiv:2410.10563, 2024

    arXiv

  8. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pages 370–387. Springer, 2025

    2025

Show all 57 references
  1. [7]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. InInternational conference on machine learning, pages 1691–1703. PMLR, 2020

    2020

  2. [8]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  3. [9]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022

    2022

  4. [10]

    Scaling rectified flow trans- formers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. InForty-first international conference on machine...

    2024

  5. [11]

    Fluid: Scaling autoregressive text-to-image generative models with continuous tokens.arXiv preprint arXiv:2410.13863, 2024

    Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens.arXiv preprint arXiv:2410.13863, 2024

    arXiv

  6. [12]

    Seedream 3.0 technical report.arXiv preprint arXiv:2504.11346, 2025

    Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xi- aochen Lian, Chao Liao, Liyang Liu, et al. Seedream 3.0 technical report.arXiv preprint arXiv:2504.11346, 2025

    2025 arXiv

  7. [14]

    Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703, 2025

    Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model.arXiv preprint arXiv:2503.07703, 2025

    2025 arXiv

  8. [15]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594, 2024. 10

    2024 arXiv

  9. [16]

    Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis

    Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.04431, 2024

    2024 arXiv

  10. [17]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  11. [18]

    Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    2024 arXiv

  12. [19]

    T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation.Advances in Neural Information Processing Systems, 36:78723–78747, 2023

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation.Advances in Neural Information Processing Systems, 36:78723–78747, 2023

    2023

  13. [20]

    T2i- compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i- compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  14. [21]

    Smartedit: Exploring complex instruction- based image editing with multimodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction- based image editing with multimodal large language models. InProceedings of the IEEE/CVF Conference...

    2024

  15. [22]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  16. [23]

    Lumen: Unleashing versatile vision-centric capabilities of large multimodal models.arXiv preprint arXiv:2403.07304, 2024

    Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Lumen: Unleashing versatile vision-centric capabilities of large multimodal models.arXiv preprint arXiv:2403.07304, 2024

    2024 arXiv

  17. [24]

    Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv preprint arXiv:2504.04423, 2025

    Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding.arXiv preprint arXiv:2504.04423, 2025

    2025 arXiv

  18. [25]

    Dual diffusion for unified image generation and understanding.arXiv preprint arXiv:2501.00289, 2024

    Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang. Dual diffusion for unified image generation and understanding.arXiv preprint arXiv:2501.00289, 2024

    2024 arXiv

  19. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings,...

    2014

  20. [27]

    Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

    Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, Chunrui Han, et al. Step1x-edit: A practical framework for general image editing.arXiv preprint arXiv:2504.17761, 2025

    2025 arXiv

  21. [28]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    2021 arXiv

  22. [29]

    Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    Yuwei Niu, Munan Ning, Mengren Zheng, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Bin Zhu, and Li Yuan. Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    2025 arXiv

  23. [30]

    Dreambench++: A human-aligned benchmark for personalized image generation.arXiv preprint arXiv:2406.16855, 2024

    Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned benchmark for personalized image generation.arXiv preprint arXiv:2406.16855, 2024

    2024 arXiv

  24. [31]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023. 11

    2023 arXiv

  25. [32]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  26. [33]

    Vision transformers for dense prediction

    René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. InProceedings of the IEEE/CVF international conference on computer vision, pages 12179– 12188, 2021

    2021

  27. [34]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  28. [35]

    Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information ...

    2022

  29. [36]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  30. [37]

    Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  31. [38]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  32. [39]

    Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024

    Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction.Advances in neural information processing systems, 37:84839–84865, 2024

    2024

  33. [41]

    Conceptmix: A compositional image generation benchmark with controllable difficulty.arXiv preprint arXiv:2408.14339, 2024

    Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty.arXiv preprint arXiv:2408.14339, 2024

    2024 arXiv

  34. [42]

    Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  35. [43]

    Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing.arXiv preprint arXiv:2504.02826, 2025

    Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Hao Li, Zicheng Zhang, Guangtao Zhai, Junchi Yan, Hua Yang, Xue Yang, and Haodong Duan. Envisioning beyond the pixels: Benchmarking reasoning-informed visual editing.arXiv preprint arXiv:2504.02826, 2025

    2025 arXiv

  36. [44]

    Not applicable

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model.arXiv preprint arXiv:2408.11039, 2024. 12 A Detailed...

    2024 arXiv

  37. [45]

    Name and Title Liam Bennett Senior Product Designer

  38. [46]

    Contact Numbers +1-310-987-6543 +1-310-987-6544

  39. [47]

    Address 501 Sunset Blvd., Los Angeles, CA 90028

  40. [48]

    Identify and highlight the inflection points of the function shown in the image

    Website and Email www.creativeline.design liam@creativeline.design The first image The second image Task 2.6 Instruction Translation Generation Please translate the English manual shown in the image below into French. The given image Not applicable The third image Figure 9: Ex...

    2020

  41. [49]

    a red backpack with a sunflower sticking out of the side pocket

  42. [50]

    a blue umbrella with white clouds printed on the canopy

  43. [51]

    a chocolate cake with five lit candles

  44. [52]

    a green rubber duck wearing sunglasses

  45. [53]

    a silver keychain in the shape of a dolphin

  46. [54]

    a yellow alarm clock showing 7:00

  47. [55]

    a purple skateboard with flame decals

  48. [56]

    a pink envelope sealed with a heart sticker

  49. [57]

    To Do Today

    a brown teddy bear holding a tiny book SeedDream 3 FLUX 1.1 Gemini-2.0 GPT-4o-Native ൜ඣ Instruction & Input Image Task 5.10 Relational Scene Composition Generate an image of a workspace tabletop with the following items: A black laptop in the center, with a spreadsheet open on...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.