Pith. sign in

REVIEW 3 major objections 5 minor 93 references

A small open evaluator can score text-to-image models as well as humans without any human training labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 06:19 UTC pith:ZZBRWZGY

load-bearing objection Solid systems paper: open compact T2I judges that beat most prior evaluators on human correlation without human training labels, plus useful diagnostics on 36 models. the 3 major comments →

arxiv 2607.11199 v1 pith:ZZBRWZGY submitted 2026-07-13 cs.CV

DynEval: Holistic Evaluations of T2I Generative Models in the Wild

Shyam Marjit , Dheeraj Baiju , Anuj Shikarkhane , Akhil Sakthieswaran , Sayak Paul , Anirban Chakraborty This is my paper
classification cs.CV
keywords text-to-image evaluationvision-language model judgedynamic question generationscene graphknowledge distillationcurriculum learningimage quality assessmentprompt-image alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Text-to-image models now produce realistic pictures, but automatic scores still miss partial mismatches, compositional errors, and images that look fine yet fail the prompt. DynEval is a compact vision-language judge that jointly checks whether an image matches its prompt and whether the image itself is free of distortions and structural failures. The authors first build GenDB: half a million real-user prompts paired with images from 36 generators, using a tiered matching of prompt difficulty to model strength so that informative failures appear. A large teacher then writes structured questions and scores for alignment and quality; the resulting 250K instruction set trains DynEval-2B and DynEval-4B by curriculum distillation. Across eleven public benchmarks the 4B model correlates more strongly with human ratings than fourteen prior automatic judges, and it also diagnoses the same persistent weaknesses (counting, humans, size binding, text rendering) across model generations.

Core claim

A fully open 4B evaluator, trained only on teacher-generated structured questions and scores and never on human ratings, attains higher overall Spearman and Pearson correlation with human judgments than prior automatic T2I evaluators on eleven benchmarks while jointly scoring text-image alignment and image quality through dynamically generated questions and scene graphs.

What carries the argument

DynEvalInstruct plus three task tokens: a 250K set of prompt-image-response triples in which a large teacher produces atomic yes/no questions for alignment and for quality (via an image-conditioned scene graph), then scores each answer 1-5; the student is curriculum-trained first to emit the questions and then to answer them under the tokens <T2IA>, <IQA>, and <EVALUATION>.

Load-bearing premise

The single large teacher model's written questions and 1-5 scores are unbiased enough that a student distilled on them will track real human preference rather than merely the teacher's blind spots.

What would settle it

Collect fresh human ratings on a held-out set of open-domain prompt-image pairs that none of the eleven benchmarks or the teacher training distribution cover; if DynEval-4B's rank correlation with those humans falls below the best competing open evaluator of comparable size, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces DynEval, a dynamic evaluator for text-to-image (T2I) models that jointly scores text-to-image alignment (T2IA) and image quality (IQA). The authors construct GenDB (500K tier-matched prompt-image pairs from DiffusionDB and 36 T2I models) and DynEvalInstruct (250K structured triplets distilled from Qwen3-VL-235B via prompt-grounded questions, scene graphs, and 1–5 VQA scores). Compact student models (DynEval-2B/4B) are fully fine-tuned with curriculum learning and task tokens (<T2IA>, <IQA>, <EVALUATION>). Across 11 benchmarks and 14 baselines, DynEval-4B reports higher overall SRCC/PLCC with human judgments (SOTA on 9/11) and supplies fine-grained failure analysis of 36 models over 42 subcategories and 9 semantic dimensions, without using human ratings for training.

Significance. If the reported human correlations hold under independent scrutiny, the work is a substantial contribution to T2I evaluation. It removes dependence on static prompt sets and external LLM question generation at inference, jointly treats alignment and perceptual quality, and demonstrates that large-scale distillation without human labels can outperform prior judges that rely on limited human annotations. The public-scale datasets, curriculum design, and diagnostic taxonomy over 36 models are concrete engineering assets that the community can reuse. The empirical gains on public human-annotated sets (Tables 2–3) and the scaling ablations (Tabs. S10–S11) give the claim practical weight.

major comments (3)
  1. Sec. 4–5 and Tab. S10: the central claim that DynEval tracks human judgments rests on the assumption that Qwen3-VL-235B’s structured T2IA+IQA 1–5 scores are sufficiently unbiased supervision. Tab. S10 only ranks teachers by average strictness on DynEval-1K; it does not report teacher–human SRCC/PLCC on the same prompt–image pairs used in Tables 2–3. Without that direct agreement measurement (or a multi-teacher ensemble check), inherited teacher bias on artistic styles, negation, or counting remains a load-bearing risk for the “higher overall correlation with human judgments” claim.
  2. Supp. A: human labels for GenEval2, TIIF-Bench, UniGenBench++, and T2I-CoReBench were collected with a single annotator per sample. Tables 2–3 treat these scores as ground truth for SRCC/PLCC. Single-annotator labels introduce unquantified noise that can inflate or deflate reported gains; at minimum, inter-annotator agreement (or a multi-annotator subset) should be reported so that the SOTA margins can be interpreted.
  3. Sec. 3–4 and free parameters (τ1, τ2, μ1, μ2, δi, α=β=0.5, wj): the tier-matched construction and failure-case filter are presented as essential for informative supervision, yet no ablation compares against uniform prompt–model sampling or against alternative α/β weights. Without that control, it is unclear how much of the human-correlation gain is attributable to the proposed pipeline versus simply training a larger VLM judge on more data.
minor comments (5)
  1. Fig. 1 and Fig. S1: normalize and report raw score ranges for every baseline so that visual comparisons of “overly high/low” scores are unambiguous.
  2. Tab. 1 capability matrix uses informal symbols; a short legend or binary columns would improve readability.
  3. Eq. (1)–(3) and Supp. C–F: the heuristic complexity weights and exact threshold values are given, but a short sensitivity plot (varying τ or δ) would help readers assess robustness.
  4. Clarify whether DynEval-1K prompts overlap with any of the 11 evaluation benchmarks used for final correlation tables.
  5. Minor typos and formatting: “V enue”, “F ramework”, and occasional missing spaces around citations should be cleaned.

Circularity Check

0 steps flagged

No circularity: human-correlation claims are external empirical results, not forced by teacher labels or self-definition.

full rationale

DynEval’s pipeline is standard knowledge distillation, not a closed derivation. GenDB is built from external DiffusionDB prompts and 36 T2I generators; DynEvalInstruct is teacher (Qwen3-VL-235B) structured T2IA/IQA responses; DynEval-2B/4B are full fine-tunes of that data. The load-bearing claim—higher SRCC/PLCC with human judgments on 11 benchmarks (Tabs. 2–3)—is measured against independent human ratings (public where available; single-annotator labels the authors collected for four newer sets). Nothing in the paper equates those human correlations to a fitted constant, a self-defined score, or a uniqueness theorem from the same authors. Teacher selection (Tab. S10) and model tiering on DynEval-1K use the teacher’s own scores only to choose a distillation source and to stratify training data; they do not algebraically force the reported human correlations. Relative failure-mode rankings of 36 T2I models (Fig. 4) are diagnostic outputs of the trained evaluator, not claimed first-principles predictions. No self-definitional equation, fitted-input-as-prediction, load-bearing self-citation chain, or renamed known result appears in the derivation. Teacher bias can still affect generalization (a correctness risk), but that is not circularity under the stated criteria.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 3 invented entities

The central claim rests on the quality of the teacher’s structured outputs, several hand-chosen thresholds for tiering and filtering, equal weighting of T2IA and IQA, and the assumption that DiffusionDB prompts plus the 36-model pool adequately cover real-world failure modes. No new physical entities are invented; free parameters are the usual engineering knobs of a distillation pipeline.

free parameters (5)
  • T2IA/IQA combination weights α, β = 0.5 / 0.5
    Fixed at 0.5 each with no sensitivity study reported in the main text.
  • prompt-complexity thresholds τ1, τ2 = 200 / 100
    Empirically set to 200 and 100 after inspecting DiffusionDB score distribution (Supp. D).
  • model-tier thresholds μ1, μ2 = 0.81 / 0.67
    Set to 0.81 and 0.67 on DynEval-1K teacher scores (Supp. E).
  • failure-case filter thresholds δi = 5
    Uniformly set to 5 (i.e., discard only perfect scores) for all tiers (Supp. F).
  • heuristic complexity weights wj = 0.2 / 0.1
    Length and object/attribute counts weighted 0.2, remaining seven factors 0.1 (Supp. C).
axioms (3)
  • domain assumption Qwen3-VL-235B structured T2IA+IQA responses are a sufficiently faithful proxy for human evaluation to serve as sole training supervision.
    Invoked throughout Sec. 4–5 and justified only by the teacher-selection ablation (Tab. S10) showing lower average scores (stricter) than alternatives.
  • ad hoc to paper Tier-matched pairing of prompt complexity with model capability yields more informative failure modes than uniform sampling.
    Stated as motivation in Sec. 3; no controlled ablation against random pairing is provided.
  • domain assumption Atomic yes/no questions plus scene-graph-derived quality questions cover the dimensions humans care about.
    Inherited from prior VQA/DSG literature and used as the distillation target (Fig. 3).
invented entities (3)
  • GenDB (500K tier-matched prompt-image pairs) independent evidence
    purpose: Provide diverse real-user failure cases for subsequent distillation.
    New dataset constructed for this work; independent evidence is the public DiffusionDB source plus the listed 36 generators.
  • DynEvalInstruct (250K prompt-image-response triplets) no independent evidence
    purpose: Supply structured supervision for curriculum fine-tuning of the student evaluators.
    Distilled from the teacher; no external human labels, so independent evidence is limited to the teacher’s own consistency.
  • task-specific tokens <T2IA>, <IQA>, <EVALUATION> no independent evidence
    purpose: Disentangle question generation from scoring during curriculum training and inference.
    Engineering device introduced in Sec. 5; no claim of broader theoretical status.

pith-pipeline@v1.1.0-grok45 · 39931 in / 3173 out tokens · 40355 ms · 2026-07-14T06:19:34.164825+00:00 · methodology

0 comments
read the original abstract

Recent advances in text-to-image (T2I) generation have led to models capable of producing highly realistic images. Yet, reliably evaluating their outputs remains challenging, especially at scale. Existing automatic evaluators, often relying on a static prompt set, struggle to capture subtle failure modes such as partial prompt misalignment, compositional errors, or visually plausible but semantically incorrect generations. In this work, we introduce DynEval, a Dynamic Evaluation framework designed to jointly assess text-to-image alignment and image quality of T2I models. To support scalable training beyond limited human-annotated data, we construct two large datasets. First, we build GenDB, a collection of 500K prompt-image pairs generated from human-written prompts drawn from DiffusionDB using a tiered prompt-model generation strategy. Second, building upon GenDB, we construct DynEvalInstruct, a 250K instruction dataset comprising prompt-image-response triplets distilled from a structured evaluation pipeline that decomposes evaluation into text-image alignment and visual quality reasoning. Using this dataset, we perform full fine-tuning of a compact evaluator through a curriculum learning strategy to effectively distill the superior evaluation capabilities of a larger teacher vision-language model, resulting in DynEval-2B and DynEval-4B. In extensive comparisons against existing evaluators across 11 benchmarks, our evaluator achieves a higher overall correlation with human judgments. Furthermore, it provides fine-grained analysis of the capabilities and failure modes of 36 T2I models across 42 subcategories and 9 semantic dimensions.

Figures

Figures reproduced from arXiv: 2607.11199 by Akhil Sakthieswaran, Anirban Chakraborty, Anuj Shikarkhane, Dheeraj Baiju, Sayak Paul, Shyam Marjit.

Figure 1
Figure 1. Figure 1: Qualitative comparison of DynEval-4B with four representative text-to-image evaluation methods [24, 28, 34, 35] across multiple T2I models [4, 7, 13, 15, 19, 20, 25, 32, 37, 41, 52, 58, 60, 64, 66, 80, 83, 85]. All scores are normalized to the [0, 1] range. The first GenEval example additionally includes a real reference image, illustrating the limitations of detector-based evaluation. Compared with existi… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DynEval Evaluation Framework. GenDB [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the proposed DynEvalInstruct generation pipeline and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Fine-grained capability analysis of 36 T2I models on the DynEval [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average scores assigned by existing methods [24, 28] and DynEval-4B to T2I mod￾els grouped by publication year. While existing evaluators show steadily increasing scores for newer models, DynEval-4B produces more calibrated as￾sessments by jointly evaluating text-image alignment and im￾age quality (visual realism). Despite rapid advances in text-to-image (T2I) generation, reliable evaluation remains an ope… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 31 linked inside Pith

  1. [1]

    In: European conference on computer vision

    Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: European conference on computer vision. pp. 382–

  2. [2]

    arXiv preprint arXiv:2312.03511 (2023)

    Arkhipkin, V., Filatov, A., Vasilev, V., Maltseva, A., Azizov, S., Pavlov, I., Aga- fonova, J., Kuznetsov, A., Dimitrov, D.: Kandinsky 3.0 technical report. arXiv preprint arXiv:2312.03511 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee,J.,Guo,Y.,etal.:Improvingimagegenerationwithbettercaptions.Computer Science.https://cdn.openai.com/papers/dall-e-3.pdf2(3), 8 (2023)

    2023

  5. [5]

    arXiv preprint arXiv:1801.01401 (2018)

    Bińkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying mmd gans. arXiv preprint arXiv:1801.01401 (2018)

    Pith/arXiv arXiv 2018

  6. [6]

    Advances in neural information processing systems33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020)

    1901

  7. [7]

    arXiv preprint arXiv:2511.22699 (2025)

    Cai, H., Cao, S., Du, R., Gao, P., Hoi, S., Hou, Z., Huang, S., Jiang, D., Jin, X., Li, L., et al.: Z-image: An efficient image generation foundation model with single-stream diffusion transformer. arXiv preprint arXiv:2511.22699 (2025)

    Pith/arXiv arXiv 2025

  8. [8]

    arXiv preprint arXiv:2505.22705 (2025)

    Cai, Q., Chen, J., Chen, Y., Li, Y., Long, F., Pan, Y., Qiu, Z., Zhang, Y., Gao, F., Xu, P., et al.: Hidream-i1: A high-efficient image generative foundation model with sparse diffusion transformer. arXiv preprint arXiv:2505.22705 (2025)

    Pith/arXiv arXiv 2025

  9. [9]

    arXiv preprint arXiv:2509.23951 (2025)

    Cao, S., Chen, H., Chen, P., Cheng, Y., Cui, Y., Deng, X., Dong, Y., Gong, K., Gu, T., Gu, X., et al.: Hunyuanimage 3.0 technical report. arXiv preprint arXiv:2509.23951 (2025)

    Pith/arXiv arXiv 2025

  10. [10]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021) DynEval: Holistic Evaluations of T2I Generative Models in the Wild 17

    2021

  11. [11]

    In: European Conference on Computer Vision

    Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., LiPixart-Σ, Z.: Weak-to-strong training of diffusion transformer for 4k text-to- image generation. In: European Conference on Computer Vision. pp. 74–91 (2024)

    2024

  12. [12]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Chen, J., Xue, S., Zhao, Y., Yu, J., Paul, S., Chen, J., Cai, H., Han, S., Xie, E.: Sana-sprint: One-step diffusion with continuous-time consistency distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 16185–16195 (2025)

    2025

  13. [13]

    In: International conference on learning representations

    Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wang, Z., Kwok, J., Luo, P., Lu, H., Li, Z.: PixArt-α: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In: International conference on learning representations. vol. 2024, pp. 57611–57640 (2024)

    2024

  14. [14]

    arXiv preprint arXiv:2509.24695 (2025)

    Chen, J., Zhao, Y., Yu, J., Chu, R., Chen, J., Yang, S., Wang, X., Pan, Y., Zhou, D.,Ling,H.,etal.:Sana-video:Efficientvideogenerationwithblocklineardiffusion transformer. arXiv preprint arXiv:2509.24695 (2025)

    arXiv 2025

  15. [15]

    arXiv preprint arXiv:2501.17811 (2025)

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C.: Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 (2025)

    Pith/arXiv arXiv 2025

  16. [16]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)

    2022

  17. [17]

    In: International conference on learning representations

    Cho, J., Hu, Y., Baldridge, J., Garg, R., Anderson, P., Krishna, R., Bansal, M., Pont-Tuset, J., Wang, S.: Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-to-image generation. In: International conference on learning representations. vol. 2024, pp. 15625–15645 (2024)

    2024

  18. [18]

    In: Proceedings of the IEEE/CVF in- ternational conference on computer vision

    Cho, J., Zala, A., Bansal, M.: Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In: Proceedings of the IEEE/CVF in- ternational conference on computer vision. pp. 3043–3054 (2023)

    2023

  19. [19]

    arXiv preprint arXiv:2505.14683 (2025)

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., Shi, G., Fan, H.: Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683 (2025)

    Pith/arXiv arXiv 2025

  20. [20]

    In: Proceedings of the 41st International Conference on Machine Learning

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high- resolution image synthesis. In: Proceedings of the 41st International Conference on Machine Learning. pp. 12606–12633 (2024)

    2024

  21. [21]

    Neural Networks144, 187–209 (2021)

    Frolov, S., Hinz, T., Raue, F., Hees, J., Dengel, A.: Adversarial text-to-image syn- thesis: A review. Neural Networks144, 187–209 (2021)

    2021

  22. [22]

    arXiv preprint arXiv:2312.11805 (2023)

    Gemini Team: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    Pith/arXiv arXiv 2023

  23. [23]

    arXiv preprint arXiv:2507.22058 (2025)

    Geng, Z., Wang, Y., Ma, Y., Li, C., Rao, Y., Gu, S., Zhong, Z., Lu, Q., Hu, H., Zhang, X., et al.: X-omni: Reinforcement learning makes discrete autoregressive image generative models great again. arXiv preprint arXiv:2507.22058 (2025)

    Pith/arXiv arXiv 2025

  24. [24]

    Advances in Neural Information Processing Systems36, 52132–52152 (2023)

    Ghosh, D., Hajishirzi, H., Schmidt, L.: Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems36, 52132–52152 (2023)

    2023

  25. [25]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., Guo, B.: Vector quantized diffusion model for text-to-image synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10696– 10706 (2022) 18 S. Marjit et al

    2022

  26. [26]

    arXiv preprint arXiv:2401.02677 (2024)

    Gupta, Y., Jaddipal, V.V., Prabhala, H., Paul, S., Von Platen, P.: Progressive knowledge distillation of stable diffusion xl using layer level loss. arXiv preprint arXiv:2401.02677 (2024)

    Pith/arXiv arXiv 2024

  27. [27]

    arXiv preprint arXiv:2511.06876 (2025)

    Gutflaish, E., Kachlon, E., Zisman, H., Hacham, T., Sarid, N., Visheratin, A., Huberman, S., Davidi, G., Bukchin, G., Goldberg, K., et al.: Generating an im- age from 1,000 words: Enhancing text-to-image with structured captions. arXiv preprint arXiv:2511.06876 (2025)

    arXiv 2025

  28. [28]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Han, S., Fan, H., Fu, J., Li, L., Li, T., Cui, J., Wang, Y., Tai, Y., Sun, J., Guo, C.L., et al.: Evalmuse-40k: A fine-grained benchmark with comprehensive human anno- tations for text-to-image generation model alignment evaluation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 4583–4591 (2026)

    2026

  29. [29]

    In: Proceedings of the 2021 conference on empirical methods in natural language processing

    Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., Choi, Y.: Clipscore: A reference- free evaluation metric for image captioning. In: Proceedings of the 2021 conference on empirical methods in natural language processing. pp. 7514–7528 (2021)

    2021

  30. [30]

    Advances in neural information processing systems30(2017)

    Heusel,M.,Ramsauer,H.,Unterthiner,T.,Nessler,B.,Hochreiter,S.:Ganstrained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems30(2017)

    2017

  31. [31]

    IEEE transactions on pattern analysis and machine intelligence 44(3), 1552–1565 (2020)

    Hinz, T., Heinrich, S., Wermter, S.: Semantic object accuracy for generative text- to-image synthesis. IEEE transactions on pattern analysis and machine intelligence 44(3), 1552–1565 (2020)

    2020

  32. [32]

    Holz, D.: Midjourney.https://www.midjourney.com(2023)

    2023

  33. [33]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Hong, S., Yang, D., Choi, J., Lee, H.: Inferring semantic layout for hierarchical text- to-image synthesis. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7986–7994 (2018)

    2018

  34. [34]

    arXiv preprint arXiv:2403.05135 (2024)

    Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., Yu, G.: Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135 (2024)

    Pith/arXiv arXiv 2024

  35. [35]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., Smith, N.A.: Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20406–20417 (2023)

    2023

  36. [36]

    IEEE Transactions on Pattern Analysis and Machine Intelligence47(5), 3563–3579 (2025)

    Huang, K., Duan, C., Sun, K., Xie, E., Li, Z., Liu, X.: T2i-compbench++: An en- hanced and comprehensive benchmark for compositional text-to-image generation. IEEE Transactions on Pattern Analysis and Machine Intelligence47(5), 3563–3579 (2025)

    2025

  37. [37]

    arXiv preprint arXiv:2410.23775 (2024)

    Huang,L.,Wang,W.,Wu,Z.F.,Shi,Y.,Dou,H.,Liang,C.,Feng,Y.,Liu,Y.,Zhou, J.: In-context lora for diffusion transformers. arXiv preprint arXiv:2410.23775 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    arXiv preprint arXiv:2512.16853 (2025)

    Kamath, A., Chang, K.W., Krishna, R., Zettlemoyer, L., Hu, Y., Ghazvinine- jad, M.: Geneval 2: Addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853 (2025)

    arXiv 2025

  39. [39]

    Advances in neural information processing systems36, 36652–36663 (2023)

    Kirstain,Y.,Polyak,A.,Singer,U.,Matiana,S.,Penna,J.,Levy,O.:Pick-a-pic:An open dataset of user preferences for text-to-image generation. Advances in neural information processing systems36, 36652–36663 (2023)

    2023

  40. [40]

    Labs, B.F.: Flux.https://github.com/black-forest-labs/flux(2024)

    2024

  41. [41]

    Labs, B.F.: FLUX.2: Frontier Visual Intelligence.https://bfl.ai/blog/flux-2 (2025)

    2025

  42. [42]

    1 kontext: Flow match- ing for in-context image generation and editing in latent space

    Labs, B.F., Batifol, S., Blattmann, A., Boesel, F., Consul, S., Diagne, C., Dock- horn, T., English, J., English, Z., Esser, P., et al.: Flux. 1 kontext: Flow match- ing for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742 (2025) DynEval: Holistic Evaluations of T2I Generative Models in the Wild 19

    Pith/arXiv arXiv 2025

  43. [43]

    Advances in Neural Information Processing Systems36, 69981– 70011 (2023)

    Lee, T., Yasunaga, M., Meng, C., Mai, Y., Park, J.S., Gupta, A., Zhang, Y., Narayanan, D., Teufel, H., Bellagente, M., et al.: Holistic evaluation of text-to- image models. Advances in Neural Information Processing Systems36, 69981– 70011 (2023)

    2023

  44. [44]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, B., Lin, Z., Pathak, D., Li, J., Fei, Y., Wu, K., Xia, X., Zhang, P., Neubig, G., Ramanan, D.: Evaluating and improving compositional text-to-visual generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5290–5301 (2024)

    2024

  45. [45]

    In: Proceedings of the 2022 conference on empirical methods in natural language processing

    Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al.: mplug: Effective and efficient vision-language learning by cross-modal skip-connections. In: Proceedings of the 2022 conference on empirical methods in natural language processing. pp. 7241–7259 (2022)

    2022

  46. [46]

    5: Three insights towards enhancing aesthetic quality in text-to-image generation

    Li, D., Kamko, A., Akhgari, E., Sabet, A., Xu, L., Doshi, S.: Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245 (2024)

    Pith/arXiv arXiv 2024

  47. [47]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

    2023

  48. [48]

    In: International confer- ence on machine learning

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International confer- ence on machine learning. pp. 12888–12900. PMLR (2022)

    2022

  49. [49]

    Li, O., Wang, Y., Hu, X., Huang, H., Chen, R., Ou, J., Tao, X., Wan, P., Qi, X., Feng, F.: Easier painting than thinking: Can text-to-image models set the stage, but not direct the play? arXiv preprint arXiv:2509.03516 (2025)

    arXiv 2025

  50. [50]

    arXiv preprint arXiv:2405.08748 (2024)

    Li, Z., Zhang, J., Lin, Q., Xiong, J., Long, Y., Deng, X., Zhang, Y., Liu, X., Huang, M., Xiao, Z., et al.: Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint arXiv:2405.08748 (2024)

    Pith/arXiv arXiv 2024

  51. [51]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liang, Y., He, J., Li, G., Li, P., Klimovskiy, A., Carolan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F., et al.: Rich human feedback for text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19401–19411 (2024)

    2024

  52. [52]

    arXiv preprint arXiv:2506.03147 (2025)

    Lin, B., Li, Z., Cheng, X., Niu, Y., Ye, Y., He, X., Yuan, S., Yu, W., Wang, S., Ge, Y., et al.: Uniworld-v1: High-resolution semantic encoders for unified visual understanding and generation. arXiv preprint arXiv:2506.03147 (2025)

    Pith/arXiv arXiv 2025

  53. [53]

    In: European Conference on Computer Vision

    Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., Ramanan, D.: Evaluating text-to-visual generation with image-to-text generation. In: European Conference on Computer Vision. pp. 366–384. Springer (2024)

    2024

  54. [54]

    OpenAI: GPT-5.1: Large Language Model.https://chat.openai.com/(Nov 2025), accessed on November 29, 2025

    2025

  55. [55]

    com / index / new - chatgpt - images-is-here/

    OpenAI: Gpt-image-1.5 (2025),https : / / openai . com / index / new - chatgpt - images-is-here/

    2025

  56. [56]

    In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021)

    Park, D.H., Azadi, S., Liu, X., Darrell, T., Rohrbach, A.: Benchmark for composi- tional text-to-image synthesis. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021)

    2021

  57. [57]

    Peebles,W.,Xie,S.:Scalablediffusionmodelswithtransformers.In:Proceedingsof the IEEE/CVF international conference on computer vision. pp. 4195–4205 (2023)

    2023

  58. [58]

    In: International Conference on Learning Representations

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. In: International Conference on Learning Representations. vol. 2024, pp. 1862–1874 (2024) 20 S. Marjit et al

    2024

  59. [59]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)

    2021

  60. [60]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  61. [61]

    In: International Conference on Medical image computing and computer-assisted intervention

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)

    2015

  62. [62]

    Advances in neural information processing systems29(2016)

    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. Advances in neural information processing systems29(2016)

    2016

  63. [63]

    arXiv preprint arXiv:2509.26641 (2025)

    Song, Y., Dong, W., Wang, S., Zhang, Q., Xue, S., Yuan, T., Yang, H., Feng, H., Zhou, H., Xiao, X., et al.: Query-kontext: An unified multimodal model for image generation and editing. arXiv preprint arXiv:2509.26641 (2025)

    arXiv 2025

  64. [64]

    arXiv preprint arXiv:2406.06525 (2024)

    Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., Yuan, Z.: Autoregres- sive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525 (2024)

    Pith/arXiv arXiv 2024

  65. [65]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the incep- tion architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)

    2016

  66. [66]

    Team, D.: Deepfloyd if: A modular cascaded diffusion model for text-to-image synthesis.https://github.com/deep-floyd/IF(2023), stability AI Research

    2023

  67. [67]

    arXiv preprint (2024)

    Team, K.: Kolors: Effective training of diffusion model for photorealistic text-to- image synthesis. arXiv preprint (2024)

    2024

  68. [68]

    arXiv preprint arXiv:2512.07584 (2025)

    Team, M.L., Ma, H., Tan, H., Huang, J., Wu, J., He, J.Y., Gao, L., Xiao, S., Wei, X., Ma, X., et al.: Longcat-image technical report. arXiv preprint arXiv:2512.07584 (2025)

    Pith/arXiv arXiv 2025

  69. [69]

    Tu, R.C., Ma, Z.A., Lan, T., Zhao, Y., Huang, H.Y., Mao, X.L.: Automatic evalu- ation for text-to-image generation: Task-decomposed framework, distilled training, andmeta-evaluationbenchmark.In:Proceedingsofthe63rdAnnualMeetingofthe Association for Computational Linguistics (Volume 1: Long Papers). pp. 22340– 22361 (2025)

    2025

  70. [70]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4566–4575 (2015)

    2015

  71. [71]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wang, J., Duan, H., Zhao, Y., Wang, J., Zhai, G., Min, X.: Lmm4lmm: Benchmark- ing and evaluating large-multimodal image generation with lmms. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 17312–17323 (2025)

    2025

  72. [72]

    arXiv preprint arXiv:2505.18775 (2025)

    Wang, J., Jiao, Y., Yu, Y., Qian, T., Chen, S., Chen, J., Jiang, Y.G.: Omni- genbench: A benchmark for omnipotent multimodal generation across 50+ tasks. arXiv preprint arXiv:2505.18775 (2025)

    Pith/arXiv arXiv 2025

  73. [73]

    arXiv preprint arXiv:2508.03320 (2025)

    Wang, P., Peng, Y., Gan, Y., Hu, L., Xie, T., Wang, X., Wei, Y., Tang, C., Zhu, B., Li, C., et al.: Skywork unipic: Unified autoregressive modeling for visual un- derstanding and generation. arXiv preprint arXiv:2508.03320 (2025)

    Pith/arXiv arXiv 2025

  74. [74]

    5: Advancing open-source multimodal models in versatility, reasoning, and efficiency

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025) DynEval: Holistic Evaluations of T2I Generative Models in the Wild 21

    Pith/arXiv arXiv 2025

  75. [75]

    arXiv preprint arXiv:2409.18869 (2024)

    Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., et al.: Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869 (2024)

    Pith/arXiv arXiv 2024

  76. [76]

    arXiv preprint arXiv:2510.18701 (2025)

    Wang, Y., Li, Z., Zang, Y., Bu, J., Zhou, Y., Xin, Y., He, J., Wang, C., Lu, Q., Jin, C., et al.: Unigenbench++: A unified semantic evaluation benchmark for text- to-image generation. arXiv preprint arXiv:2510.18701 (2025)

    arXiv 2025

  77. [77]

    In: Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers)

    Wang, Z.J., Montoya, E., Munechika, D., Yang, H., Hoover, B., Chau, D.H.: Diffu- siondb: A large-scale prompt gallery dataset for text-to-image generative models. In: Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers). pp. 893–911 (2023)

    2023

  78. [78]

    Wei, X., Zhang, J., Wang, Z., Wei, H., Guo, Z., Zhang, L.: Tiif-bench: How does your t2i model follow your instructions? arXiv preprint arXiv:2506.02161 (2025)

    Pith/arXiv arXiv 2025

  79. [79]

    arXiv preprint arXiv:2508.02324 (2025)

    Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., Yin, S.m., Bai, S., Xu, X., Chen, Y., et al.: Qwen-image technical report. arXiv preprint arXiv:2508.02324 (2025)

    Pith/arXiv arXiv 2025

  80. [80]

    arXiv preprint arXiv:2506.18871 (2025)

    Wu, C., Zheng, P., Yan, R., Xiao, S., Luo, X., Wang, Y., Li, W., Jiang, X., Liu, Y., Zhou, J., et al.: Omnigen2: Exploration to advanced multimodal generation. arXiv preprint arXiv:2506.18871 (2025)

    Pith/arXiv arXiv 2025

Showing first 80 references.