Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces EvalMuse-40K, a 40,000-pair benchmark with over one million fine-grained human annotations, along with two metrics—FGA-BLIP2 and PN-VQA—that correlate with human alignment judgments more closely than prior baselines.

desk verdict The benchmark is the real contribution, and it deserves peer review; the metric comparisons need trained baselines and honest threshold reporting before the SOTA claims can be trusted. read the letter →

arxiv 2412.18150 v2 pith:EMKALQZM submitted 2024-12-24 cs.CV cs.AI

Shuhao Han , Haotian Fan , Jiachen Fu , Liang Li , Tao Li , Junhui Cui , Yunqiu Wang , Yang Tai
show 3 more authors
Jingwei Sun Chunle Guo Chongyi Li
This is my paper · ORCID
classification cs.CVcs.AI
keywords text-to-imagegenerationimage-textalignmentfine-grainedevaluationbenchmarkhumanannotationsBLIP2fine-tuningzero-shotVQAmetricdiffusionmodelsrankingcompositional
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that text-to-image alignment can be evaluated reliably and at fine grain when the benchmark is large, balanced, and richly annotated. It contributes EvalMuse-40K: 4,000 prompts (half real user prompts, half synthetic skill probes), 40,000 generated image-text pairs across 20 different T2I models, and over one million human annotations covering overall alignment scores, per-element match labels, structural defects, and data-quality flags. On this benchmark the paper's two metrics, FGA-BLIP2 and PN-VQA, achieve the highest correlation with human annotations among the methods compared, and FGA-BLIP2 also generalizes to GenAI-Bench, TIFA, and RichHF. The authors use these metrics to rank current generation models and argue that the benchmark can serve as a shared reference for future text-to-image evaluation.

What carries the argument

FGA-BLIP2 extends the BLIP2 image-text matching head so that query embeddings produce an overall alignment score while text-token embeddings produce element-level scores; a mask predictor identifies which tokens are valid elements, and the joint loss is weighted by $e^{\sigma(p)}$, where $\sigma(p)$ is the variance of human scores across images from the same prompt, so training focuses on prompts that actually discriminate between models. PN-VQA asks each element question twice, once with the correct answer and once with the incorrect answer, and computes the element score as $(P_T + 1 - P_F)/2$ from the model's yes/no logits; the original generation prompt is inserted into the question to anchor context. These two mechanisms convert the benchmark's overall and element-level annotations into either end-to-end training signal or zero-shot probing.

What would settle it

Recompute inter-annotator agreement on the raw element-level labels (for example, Cohen's kappa between independent annotators) and compare GPT-4's element splits against a human-created gold split on a random subset; if kappa falls below roughly 0.4 on either, the fine-grained labels and any metric trained on them cannot support the claimed correlations.

Watch

Extended reading notes

Core claim

The central claim is that a large, diverse benchmark with fine-grained human annotations can make automated text-to-image alignment metrics both more accurate and more diagnosable. EvalMuse-40K supplies 4K prompts balanced across subject, logic, style, and semantic clusters, with images generated by 20 diffusion-based models; each pair receives ratings from 3 to 6 annotators on a 5-point Likert scale plus element-level yes/no labels, structural problem labels, a splitting-confidence flag, and a meaningless-prompt flag. The paper's own metrics exploit this design: FGA-BLIP2 fine-tunes BLIP2 end-to-end with a variance-weighted loss over overall and element scores, reaching SRCC 0.7742 on EvalMuse-40K and 0.5637 on GenAI-Bench, while the zero-shot PN-VQA reaches 0.5748 with Qwen2-VL; both exceed prior baselines such as CLIPScore, ImageReward, PickScore, HPSv2, VQAScore, TIFA, and VQ2. The paper further claims that the resulting ranking of 22 T2I models places proprietary models such as Dreamina v2.0Pro, DALL-E 3, FLUX 1.1, and Midjourney v6.1 at the top.

Load-bearing premise

The benchmark's reliability rests on treating averaged scores from 3 to 6 annotators and GPT-4-generated element splits as valid ground truth for image-text alignment.

Editorial extensions

If this is right

  • EvalMuse-40K gives the field a common yardstick large enough to rank alignment metrics statistically, rather than on the small prompt sets used by earlier benchmarks.
  • FGA-BLIP2 can serve as a reference scorer for ranking T2I models, with the paper's table placing Dreamina v2.0Pro, DALL-E 3, FLUX 1.1, and Midjourney v6.1 ahead of open and efficient models.
  • Element-level categories let practitioners pinpoint which skills a model fails, such as counting, shape, and activity, and also reveal which skills evaluators themselves mis-score.
  • PN-VQA provides a training-free fine-grained evaluation that improves as the underlying vision-language model improves.
  • The structural-problem annotations support measuring how well MLLMs detect anatomical and object defects, where the paper reports that current recall is low.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The variance-weighted loss is a general training trick: any reward model or preference pipeline where some prompts produce near-constant scores could reweight samples by label variance to focus learning on discriminative cases.
  • Positive-negative prompting is a generic verification pattern that could improve hallucination detection or compositional reasoning in vision-language models beyond text-to-image evaluation.
  • The paper reports only a maximum-difference histogram, not Cohen's kappa or ICC; a re-analysis of the released raw annotations could settle whether element-level labels are stable enough to support fine-grained training.
  • The benchmark's release enables a testable extension beyond evaluation: training a reward model on its element labels could steer T2I generation toward better alignment, not just measure it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces EvalMuse-40K, a text-to-image alignment benchmark with 4K prompts, 40K image-text pairs generated by 20 T2I models, and about 1M human annotations. Annotations include a 5-point overall alignment score, element-level binary alignment labels categorized by skill, structural problem labels, and extra labels such as splitting confidence and prompt meaningfulness. The paper also proposes two automated metrics: FGA-BLIP2, a BLIP2-based model fine-tuned to jointly predict overall and element-level alignment with a variance-weighted loss, and PN-VQA, a zero-shot positive-negative VQA protocol. Experiments report Spearman/Pearson correlations with human annotations on EvalMuse-40K, GenAI-Bench, TIFA, and RichHF, plus fine-grained accuracy results, an ablation study, and a T2I model ranking.

Significance. The benchmark is a potentially valuable community resource: it is substantially larger than existing fine-grained alignment benchmarks, includes balanced prompt sampling via MILP, multi-annotator labeling with re-annotation, and covers a wider range of T2I models. The method contributions are also relevant, particularly the idea of jointly supervising overall and element scores. The claims of state-of-the-art correlation are plausible, but the current evaluation protocol has two load-bearing weaknesses that must be addressed before the paper can be accepted: (1) the main in-domain comparison is confounded by FGA-BLIP2 being fine-tuned on a large portion of the same benchmark while baselines are zero-shot, and (2) the fine-grained accuracy numbers use test-set threshold selection. The dataset construction and reliability analysis are otherwise sound.

major comments (4)
  1. [Section 5.2, Table 2] FGA-BLIP2 is fine-tuned on 3K of the 4K EvalMuse-40K prompts and evaluated on the remaining 1K prompts, whereas all baselines are zero-shot and have never seen this distribution. The large in-domain SRCC advantage over VQAScore (0.7742 vs 0.4877) is therefore a distribution-overfitting indicator, not evidence of general superiority. The cross-benchmark gains are much smaller (GenAI-Bench: 0.5637 vs 0.5534; RichHF: 0.5123 vs 0.4826; TIFA: 0.7604 vs 0.6951) and no significance testing or confidence intervals are reported. Please add bootstrap confidence intervals for the SRCC differences, and consider evaluating FGA-BLIP2 under a protocol that is fair to zero-shot baselines, such as training on an external dataset and testing on EvalMuse-40K, or reporting results with varying fractions of EvalMuse-40K used for training.
  2. [Table 3 and Table 6] The fine-grained accuracy numbers are obtained by a fixed-step (0.01) threshold search on the test set to maximize overall accuracy. This uses test labels to select a free parameter, which inflates the reported accuracy and can make the comparison with methods that use a fixed threshold unfair. Please use a held-out validation set for threshold selection, or report accuracy both at a fixed threshold (e.g., 0.5) and at the optimally selected threshold separately, and apply the same protocol to every compared method, including Element-GT.
  3. [Section 3.4] The reliability analysis reports only the histogram of maximum pairwise score differences and the re-annotation rule (range >= 2). To support the claim that EvalMuse-40K is a 'reliable' benchmark, please report standard inter-rater agreement statistics, such as Cohen's kappa or intraclass correlation coefficient (ICC), for the Likert scores and for element-level binary labels. Also report the fraction of image-text pairs that triggered re-annotation and how re-annotation changed the final labels.
  4. [Section 3.3] The 'splitting confidence' label is collected to flag instances where GPT-4 element splitting is incorrect, but the paper never analyzes this label or reports its distribution. Please report how often annotators flagged incorrect splits, and assess whether the element-level results change when low-confidence splits are excluded. Without this, the validity of the element-level annotations and element-supervised training is not fully established.
minor comments (6)
  1. [Section 5.3] There is a typo in the heading: 'FGnA-BLIP2' should be 'FGA-BLIP2'.
  2. [Section 8.3] There is a typo in the supplementary section title: 'Sythetic Data Generation' should be 'Synthetic Data Generation'.
  3. [Table 2] The experiments text refers to 'TIFA800' but the benchmark is commonly referenced as TIFA160 elsewhere in the paper and in prior work; please clarify which TIFA split is used and keep the terminology consistent.
  4. [Equation (1)] Please define whether σ(p) is the variance among all images for prompt p in the training set, and clarify whether it is normalized. Adding this detail would make the variance-weighting strategy reproducible.
  5. [Table 4] The ablation row 'PN-VQA (w/o prompt)' should state explicitly whether the positive-negative question template still includes the original prompt except for the removal; otherwise the reader cannot tell which component is being ablated.
  6. [References] Several references have inconsistent formatting (e.g., [9], [17], [41]); please unify the reference style according to the journal template.

Circularity Check

1 steps flagged · score 3.0 of 10

No load-bearing circularity in the benchmark derivation or the main SRCC claims; the one circular-adjacent step is the test-set threshold search used to report fine-grained accuracy, which is a disclosed secondary fitted metric.

  1. fitted input called prediction [Table 6 caption and Table 3 footnote (Sections 9.1 and 5.2)]
    "All the above methods use a fixed-step (0.01) search to select the optimal threshold for binary classification, aiming to maximize overall accuracy."

    The fine-grained accuracy numbers in Tables 3 and 6 are reported after choosing a per-method threshold that maximizes accuracy on the same EvalMuse-40K test set. Thus the reported accuracy is a maximized fit, not an independent prediction: it is max_t Accuracy(t | test_labels) by construction. The table footnote discloses this, but the values are still presented as method performance. The SRCC/PLCC correlations and the external-benchmark results are not threshold-fitted, so this is a secondary, partial circularity rather than a defect in the central benchmark claim.

full rationale

The paper's central derivation is self-contained. EvalMuse-40K is constructed from human annotations, and the methods are evaluated against those annotations on a prompt-disjoint held-out test split plus external benchmarks (GenAI-Bench, TIFA, RichHF); FGA-BLIP2 is supervised on the training split and PN-VQA is zero-shot, so their reported correlations are genuine predictions rather than reproductions of training labels. No load-bearing self-citation or imported uniqueness theorem appears. The only circular-adjacent element is the fixed-step threshold search on the test set used to report fine-grained binary accuracy, which makes those accuracy numbers fitted values. Because this affects only a secondary metric and is explicitly acknowledged in the table captions, the overall circularity is mild.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The 'elements', 'skills', and 'structural problems' are annotation taxonomies adapted from TIFA and standard image-quality categories.

free parameters (3)
  • loss weight lambda = 0.1
    Chosen by hand in Eq. (1); balances element score loss Les in FGA-BLIP2 training and affects the trained model's correlation.
  • loss weight eta = 0.1
    Chosen by hand in Eq. (1); balances mask loss Lmask in FGA-BLIP2 training.
  • yes/no threshold = not reported; searched over [0,1] in 0.01 steps
    Fixed-step search on test data to maximize accuracy in Table 3 and Section 9.3. This is a fitted parameter that can inflate the reported accuracy numbers.
assumptions (4)
  • domain assumption GPT-4 element splitting and question generation produce faithful element decompositions of prompts.
    Section 3.1 uses LLM templates (Listings 1 and 2) to split prompts into elements and generate yes/no questions. The fine-grained labels inherit any errors in this decomposition; the 'splitting confidence' label partially mitigates this but is not a systematic validation.
  • domain assumption Averaged human Likert ratings and element labels are ground truth for image-text alignment.
    Section 3.3 averages 3 to 6 annotators and re-annotates disagreements of 2 or more points. No inter-annotator agreement coefficient (kappa or ICC) is reported, so the reliability claim rests on this assumption.
  • domain assumption DiffusionDB prompts are representative of real T2I user prompts.
    Section 3.1 samples 2K real prompts from DiffusionDB's 1.8M prompts; if this gallery is not representative of current user practice, the real-prompt half of the benchmark is skewed.
  • standard math MILP optimization yields a balanced representative sample.
    Equations (3) to (5) formulate the sampling as a standard mixed-integer linear program; correctness follows from linear programming, but the category labels input to the MILP come from GPT-4 and are noisy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation." pith.science (2026). https://pith.science/paper/EMKALQZM

@misc{pith2026241218150,
  author       = {Pith},
  title        = {Pith review of: EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMKALQZM}},
  note         = {Machine review of arXiv:2412.18150}
}
read the original abstract

Recently, Text-to-Image (T2I) generation models have achieved significant advancements. Correspondingly, many automated metrics have emerged to evaluate the image-text alignment capabilities of generative models. However, the performance comparison among these automated metrics is limited by existing small datasets. Additionally, these datasets lack the capacity to assess the performance of automated metrics at a fine-grained level. In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks. In the construction process, we employ various strategies such as balanced prompt sampling and data re-annotation to ensure the diversity and reliability of our benchmark. This allows us to comprehensively evaluate the effectiveness of image-text alignment metrics for T2I models. Meanwhile, we introduce two new methods to evaluate the image-text alignment capabilities of T2I models: FGA-BLIP2 which involves end-to-end fine-tuning of a vision-language model to produce fine-grained image-text alignment scores and PN-VQA which adopts a novel positive-negative VQA manner in VQA models for zero-shot fine-grained evaluation. Both methods achieve impressive performance in image-text alignment evaluations. We also use our methods to rank current AIGC models, in which the results can serve as a reference source for future study and promote the development of T2I generation. The data and code will be made publicly available.

Figures

Figures reproduced from arXiv: 2412.18150 by the authors.

Figure 1
Figure 1. Overview construction of EvalMuse-40K, consisting of (a) data collection, (b) data annotation, and (c) evaluation methods. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Statistical Charts. (a) Distribution of annotated alignment scores in real prompt samples and synthetic prompt samples. (b) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Structure of FGA-BLIP2. m and n denote query and text token lengths, respectively; so and se represent overall and element scores, respectively; Los represents the overall alignment score loss, Les is the element alignment score loss, and Lmask is the loss for predicting valid elements. FGA-BLIP2 jointly optimizes these losses to achieve fine-grained evaluation. to extract text features, which then are passed to an … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visualization of overall alignment scores from different methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of fine-grained alignment scores from different methods. In each figure, the question and answer are generated [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Distribution of data before (top) and after (bottom) sampling in four dimensions using MILP strategy. It can be observed that [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Visualization of overall alignment scores from different methods. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Visualization of fine-grained alignment scores from different methods. In the figure, the question and answer are generated [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark

    cs.CV 2025-09 conditional novelty 7.0 of 10

    The authors build a 6M-image, 20M-caption reasoning dataset with generation chain-of-thought and a 7-track VLM-judged benchmark, then rank 19 text-to-image models.

  2. LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A new benchmark with 58,500 AI videos and 2.6 million human annotations, plus an LMM-based metric that predicts perceptual quality, prompt-video correspondence, and task-specific correctness.

  3. ViDA-UGC: Detailed Image Quality Analysis via Visual Distortion Assessment for UGC Images

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A UGC distortion instruction dataset with 11.5K images and a human-revised benchmark improves MLLM distortion grounding, perception, and quality description, with several tuned models passing GPT-4o on the benchmark.

  4. MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MMIG-Bench is a unified benchmark of 4,850 prompts and 1,750 reference images with a three-level evaluation suite, including the VQA-based Aspect Matching Score that correlates with human ratings.

  5. LLM Code Customization with Visual Results: A Benchmark on TikZ

    cs.SE 2025-05 conditional novelty 6.0 of 10

    vTikZ, a 100-task benchmark for visual code customization, shows LLMs solve at most 28 percent of TikZ editing scenarios even with best-of-five sampling.

  6. Instruction-augmented Multimodal Alignment for Image-Text and Element Matching

    cs.CV 2025-04 conditional novelty 5.0 of 10

    A fine-tuned multimodal score model with soft Q-Align scoring, element-conditioned prompts, and self-training on validation pseudo-labels takes first place in NTIRE 2025 Track 1 image-text alignment.

Reference graph

Works this paper leans on

52 extracted references · 32 canonical work pages · cited by 6 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Kandinsky 3.0 technical report

    Vladimir Arkhipkin, Andrei Filatov, Viacheslav Vasilev, Anastasia Maltseva, Said Azizov, Igor Pavlov, Julia Aga- fonova, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky 3.0 technical report. arXiv preprint arXiv:2312.03511, 2023. 4, 7

    arXiv 2023

  3. [3]

    blackforestlabs. Flux1.1. https : / / blackforestlabs.ai/, 2024. 7

    work page 2024

  4. [4]

    Pixart- σ: Weak-to-strong training of dif- fusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- σ: Weak-to-strong training of dif- fusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692, 2024. 4, 7

    arXiv 2024

  5. [5]

    Pixart- δ: Fast and controllable image generation with latent consistency mod- els

    Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart- δ: Fast and controllable image generation with latent consistency mod- els. arXiv preprint arXiv:2401.05252, 2024. 4, 7

    arXiv 2024

  6. [6]

    Pixart-α: Fast training of diffusion trans- former for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-α: Fast training of diffusion trans- former for photorealistic text-to-image synthesis. In ICLR,

    work page

  7. [7]

    Dall-eval: Probing the reasoning skills and social biases of text-to- image generation models

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to- image generation models. In ICCV, 2023. 2

    work page 2023

  8. [8]

    If-i-xl-v1.0

    DeepFloyd. If-i-xl-v1.0. https://www.deepfloyd. ai, 2023. 4, 7

    work page 2023

Show all 52 references
  1. [9]

    Dreamina

    DreaminaAI. Dreamina. https : / / dreamina . capcut.com/, 2023. 4, 7

    2023

  2. [10]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In ICML, 2024. 1, 4, 7

    2024

  3. [11]

    Progressive knowledge dis- tillation of stable diffusion xl using layer level loss

    Yatharth Gupta, Vishnu V Jaddipal, Harish Prabhala, Sayak Paul, and Patrick V on Platen. Progressive knowledge dis- tillation of stable diffusion xl using layer level loss. arXiv preprint arXiv:2401.02677, 2024. 4

    2024 arXiv

  4. [12]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  5. [13]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. In NeurIPS, 2017. 1, 3

    2017

  6. [14]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In NeurIPS, 2020. 4

    2020

  7. [15]

    Midjourney

    David Holz. Midjourney. https://www.midjourney. com, 2023. 4, 7

    2023

  8. [16]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Os- tendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV, 2023. 1, 2, 3, 4, 6, 7

    2023

  9. [17]

    T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xi- hui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. In NeurIPS, 2023. 2

    2023

  10. [18]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In NAACL, 2019. 4

    2019

  11. [19]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. In NeurIPS, 2023. 1, 2, 3, 6, 7

    2023

  12. [20]

    Genai-bench: Evaluating and improv- ing compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Gra- ham Neubig, et al. Genai-bench: Evaluating and improv- ing compositional text-to-visual generation. arXiv preprint arXiv:2406.13743, 2024. 2, 6, 7

    2024 arXiv

  13. [21]

    Evaluating and improving composi- tional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan. Evaluating and improving composi- tional text-to-visual generation. In CVPR, 2024. 1, 6, 7, 8, 5

    2024

  14. [22]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2. 5: Three in- sights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024. 4, 7

    2024 arXiv

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  16. [24]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding. arXiv preprint arXiv:2405.087...

    2024 arXiv

  17. [25]

    Rich human feedback for text-to-image generation

    Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. Rich human feedback for text-to-image generation. In CVPR, 2024. 2, 4, 6, 7

    2024

  18. [26]

    Sdxl- lightning: Progressive adversarial diffusion distillation

    Shanchuan Lin, Anran Wang, and Xiao Yang. Sdxl- lightning: Progressive adversarial diffusion distillation. arXiv preprint arXiv:2402.13929, 2024. 4, 7

    2024 arXiv

  19. [27]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 2, 6

    2014

  20. [28]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 7, 5, 6

    2024

  21. [29]

    Latent consistency models: Synthesizing high- resolution images with few-step inference

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high- resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023. 4, 7

    2023 arXiv

  22. [30]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023. 1

    2023

  23. [31]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024. 1, 4

    2024

  24. [32]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  25. [33]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 1, 4, 7

    2022

  26. [34]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022. 1, 2

    2022

  27. [35]

    Improved techniques for training gans

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016. 3

    2016

  28. [36]

    Adversarial diffusion distillation

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In ECCV, 2024. 4, 7

    2024

  29. [37]

    Evalalign: Evaluating text- to-image models through precision alignment of multimodal large models with supervised fine-tuning to human annota- tions

    Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Evaluating text- to-image models through precision alignment of multimodal large models with supervised fine-tuning to human annota- tions. arXiv preprint arXiv:2406.16562, 2024. 2

    2024 arXiv

  30. [38]

    Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis

    Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint,

  31. [39]

    Shaping datasets: Optimal data selection for spe- cific target distributions across dimensions

    Vassilios V onikakis, Ramanathan Subramanian, and Stefan Winkler. Shaping datasets: Optimal data selection for spe- cific target distributions across dimensions. In ICIP, 2016. 1, 3, 4

    2016

  32. [40]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 7, 8, 5, 6

    2024 arXiv

  33. [41]

    Diffu- siondb: A large-scale prompt gallery dataset for text-to- image generative models

    Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffu- siondb: A large-scale prompt gallery dataset for text-to- image generative models. In ACL, 2023. 1, 3, 4

    2023

  34. [42]

    Revisiting text-to-image evaluation with gecko: On met- rics, prompts, and human ratings

    Olivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kaji´c, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, et al. Revisiting text-to-image evaluation with gecko: On met- rics, prompts, and human ratings. arXiv preprint arXiv...

    2024 arXiv

  35. [43]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 ,

    arXiv

  36. [44]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. In NeurIPS, 2023. 1, 3, 6, 7

    2023

  37. [45]

    What you see is what you read? improving text- image alignment evaluation

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text- image alignment evaluation. In NeurIPS, 2024. 1, 2, 3, 6, 7, 8, 5

    2024

  38. [46]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 7, 5, 6

    2024 arXiv

  39. [47]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. In TMLR,

  40. [48]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 3 EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Gen...

    2018

  41. [49]

    In data collection details, we de- tail the classification and sampling of real user prompts in Sections 8.1 and 8.2, ensuring diversity and balance among the real prompts

    Overview In this supplementary material, we provide data collection details, additional experiment results , and T2I model alignment evaluation. In data collection details, we de- tail the classification and sampling of real user prompts in Sections 8.1 and 8.2, ensuring diver...

  42. [50]

    1 cat and some dogs

    Data Collection Details 8.1. Real Prompt Categorization DiffusionDB [41] contains a large and diverse set of real user-used prompts. Our goal is to select a representative subset of prompts that accurately reflects the overall distri- bution. To achieve this, we classify the p...

    2009

  43. [51]

    Yes.” and “No

    More Experiments 9.1. Results of Metrics on Specific Skills We have categorized elements by different skills when an- notating elements at a fine-grained level. Here, we analyze the accuracy of our fine-grained evaluation metrics on spe- cific skills. The specific results are ...

  44. [52]

    Evaluation of image-text alignment across different T2I models

    T2I Model Alignment Evaluation Benefiting from the strong correlation between FGA-BLIP2 and human preferences in image-text alignment evaluation, we employ it to assess and rank the performance of widely Table 10. Evaluation of image-text alignment across different T2I models....

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.