Pith. sign in

REVIEW 5 major objections 4 minor 67 references

Multi-Modal Language Models as Text-to-Image Model Evaluators

T0 review · 5 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single open-source multimodal language model can rank eight text-to-image models using just 20 self-generated prompts, closely matching the ranking of a 1,600-prompt benchmark.

desk verdict Clever idea and one solid human-correlation result, but the 20-prompt efficiency headline is weakened by seed averaging, selection on the target benchmark, and a numeric mismatch between the main table and the appendix. read the letter →

arxiv 2505.00759 v2 pith:WSSNDNYW submitted 2025-05-01 cs.CV cs.AIcs.CL

Jiahui Chen , Candace Ross , Reyhane Askari-Hemmat , Koustuv Sinha , Melissa Hall , Michal Drozdzal , Adriana Romero-Soriano This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords text-to-imageevaluationmultimodallanguagemodelsdynamicbenchmarkspromptgenerationVQAScoreimage-textconsistencyadaptivelargeasevaluators
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to replace static, hand-written prompt benchmarks in text-to-image evaluation with a single multimodal language model that both writes prompts and scores the resulting images. Its central claim is that MT2IE, an iterative loop that makes each new prompt harder and then scores image-text consistency, reproduces the model ranking of a 1,600-prompt benchmark while using only 20 prompts. The authors also report that scores from this loop correlate better with human judgment than prior automatic metrics on a static benchmark. If the claim holds, evaluation of image-generation models would no longer depend on large curated prompt sets and could instead adapt automatically to each model's strengths and failure points.

What carries the argument

The load-bearing mechanism is the iterative prompt-difficulty loop: a multimodal language model rewrites a previous prompt into a harder one by adding an object, spatial relation, or attribute; the text-to-image model generates an image; the same MLLM scores the image-prompt consistency; and the score feeds the next rewrite. The scoring component is VQAScore, a named metric that measures how likely the model is to answer "yes" to the prompt-description question, validated against human ratings on a static benchmark. The loop is what makes few-prompt efficiency possible: instead of sampling from a static pool, the MLLM concentrates difficulty where it separates models.

What would settle it

Have human annotators score the images generated from MT2IE's 20 prompts for the eight text-to-image models, then compute the ranking correlation between MT2IE's scores and the human scores; if that correlation is significantly lower than the correlation reported on TIFA160, the 20-prompt ranking would match GenAIBench for the wrong reason.

Watch

Extended reading notes

Core claim

The paper's central discovery is that an open-source multimodal language model can serve as a complete evaluator for text-to-image models: it generates the evaluation material and grades the result. In MT2IE, the evaluator takes a seed prompt, adds one object, attribute, or spatial relation per iteration to make the prompt harder, has the text-to-image model draw the image, then scores the image with VQAScore, the likelihood that the model answers "yes" to "Does this figure show {prompt}?". On a standard static benchmark of 160 prompts, this single-model pipeline achieves higher Spearman and Kendall correlations with human ratings than several existing question-answering and CLIP-based metrics. Rank-ordering eight text-to-image models with 20 generated prompts gives a Kendall's tau of 0.8571 against the ranking from the full 1,600-prompt benchmark, while random 20-prompt subsamples of that benchmark give uninformative or inverted rankings. A variant that adapts prompt complexity to each model's running score preserves the same ranking, and an appendix extends the loop to aesthetic quality scoring with a reported Kendall's tau of 0.691 against a pretrained aesthetics predictor.

Load-bearing premise

The paper assumes that the MLLM's VQAScore remains a valid proxy for human judgment when applied to the longer, more complex prompts it generates, even though the human-correlation validation was done only on static TIFA160 prompts and the benchmark ranking it matches is itself VQAScore-based.

Editorial extensions

If this is right

  • Text-to-image model evaluation could be done with about 20 generated prompts instead of 1,600 hand-written ones, cutting generation and annotation cost by a factor of 80.
  • Static benchmarks that saturate or leak into training data could be replaced by dynamic, per-run prompts, making overfitting to fixed prompt sets less attractive.
  • Because the same open-source multimodal model generates and scores, evaluation would no longer require proprietary language models or multi-model pipelines.
  • An adaptive version gives each model a bespoke difficulty trajectory, so weaker models receive informative easier prompts and stronger models are pushed to harder ones, while the resulting ranking still matches the full benchmark.
  • The framework generalizes beyond consistency to other judgment axes, such as aesthetic quality, by swapping the scoring instruction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported 0.8571 ranking match holds beyond the eight models tested, MT2IE could enable nearly free regression testing of every new text-to-image checkpoint, making benchmark saturation a routine concern rather than an episodic one.
  • The method's validity depends on the scorer tracking human judgment on the harder, generated prompts; a direct human-rating study on those prompts is the natural next test, and one the paper does not report.
  • A risk left implicit is that text-to-image developers could optimize against the MLLM's scoring signal; whether that leads to genuine improvements or to reward hacking is an open question.
  • The same iterative difficulty loop could be pointed at other generative domains, such as video or 3D generation, where static benchmarks are even less stable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces Multimodal Text-to-Image Eval (MT2IE), a framework in which an open-source multimodal LLM (Llava-v1.6-34B) acts as both prompt generator and scorer for text-to-image (T2I) evaluation. In Experiment 1, the authors validate three open MLLMs as zero-shot scorers on the static TIFA160 benchmark, comparing GQA-style question generation and VQAScore-style likelihood scoring against human ratings, and report higher human correlations than existing GQA baselines. In Experiment 2, MT2IE iteratively expands four seed prompts over five iterations to produce 20 prompts, scores images from 8 T2I models with Llava-VQAScore, and reports a Kendall's tau of 0.8571 between the resulting model ranking and the 1,600-prompt GenAIBench VQAScore ranking. Experiment 3 adds adaptive prompting, in which prompt difficulty is adjusted based on the previous score, and difficulty-weighted scores are used to reproduce GenAIBench rankings. An appendix extends the framework to aesthetic scoring.

Significance. The paper addresses a real problem—static T2I benchmarks saturate and leak—and the proposal of a single open-source MLLM that both writes and scores prompts is a useful step toward adaptive evaluation. The TIFA160 human-correlation results are a genuine contribution: they show that open MLLMs can outperform multi-model GQA pipelines, which contradicts earlier findings with open models. The iterative prompt-generation procedure also produces verifiable difficulty progression. If the efficiency claim is re-established under a correct accounting of the evaluation budget and with proper hyperparameter selection, the framework would be practically valuable for cheap model comparison. At present the paper does not ship code, and the central 20-prompt/80x-efficiency claim is not supported by the reported aggregate analysis; the claims about matching human judgment are also limited to static prompts. These issues are addressable, so the manuscript is a candidate for major revision rather than rejection.

major comments (5)
  1. [Section 5, Table 3, Figure 7] The headline efficiency claim is not supported by the reported protocol. Section 5 says "We run MT2IE-benchmark generation over 5 random seeds" and Figure 7 plots average ranks over all seeds, so the Kendall's tau of 0.8571 in Table 3 appears to be computed on a seed-averaged ranking that consumes 5 × 20 = 100 prompt-image pairs per T2I model, not 20. Please report the per-seed rank correlations and their variance, and compare the seed-averaged MT2IE ranking against a seed-averaged random 100-prompt GenAIBench subset under the same aggregation; otherwise the abstract's "1/80th" claim is not established.
  2. [Appendix F, Table 5] The 5-iteration/20-prompt configuration is selected as the best among {2, 5, 10, 12} by comparing Kendall's tau against the full GenAIBench VQAScore ranking, which is the same target used to evaluate the final method in Section 5. With only 8 models, optimizing over four configurations can inflate the reported rank correlation, and no held-out benchmark or partition is used. Please specify the configuration before evaluation or use a separate validation set for hyperparameter selection, and report the tau for every iteration count as a sensitivity range rather than only the best configuration.
  3. [Appendix D, Section 6] Section 6's adaptive ranking depends on the Yngve difficulty weight, but Appendix D's post-hoc analysis (Table 4) shows that Yngve score has Kendall's tau of only -0.0970 with VQAScore and is not the most correlated metric (average word length, tau 0.1025, is higher). The paper says Yngve was selected based on a preliminary correlation analysis, which means the adaptive result in Figure 7 is conditional on a weakly supported and possibly overfit choice. Please report the Figure 7 adaptive ranking under all candidate difficulty metrics, and discuss why the selected weight is appropriate despite Table 4.
  4. [Section 4.2, Section 5] The human-correlation validation in Section 4.2 is performed only on static TIFA160 prompts, and no human ratings are collected for the longer, more compositional MT2IE-generated prompts used in Sections 5 and 6. Since the GenAIBench target ranking is itself obtained with CLIP-FlanT5 VQAScore, the reported agreement with GenAIBench could largely reflect agreement between two automatic VQAScore variants rather than human-aligned evaluation. Additionally, Section 3 states that MLLM-generated benchmarks "preserve relative model rankings computed by human annotators in prior work," but Section 5 compares against an automatic VQAScore ranking, not human-rated images. Please collect human judgments on a sample of MT2IE-generated prompts or explicitly re-frame the claim as agreement with an automatic benchmark.
  5. [Table 3 vs Appendix F, Table 5] The paper reports two different Kendall tau values for the same downsampled GenAIBench (20 prompts) baseline: Table 3 gives -0.6428 for VQAScore, while Appendix F, Table 5 gives -0.8693. Unless these correspond to different scoring metrics, seeds, or aggregation schemes, one of these numbers is incorrect; please reconcile the baseline and report the protocol used in each table.
minor comments (4)
  1. [Section 5] The text states MT2IE is "utilizing 20 times fewer prompts" when comparing 20 prompts to 1,600 prompts; the correct factor is 80 times fewer, as stated in the abstract and Section 6.
  2. [Section 5, Appendix A, Appendix D] The experimental counts are inconsistent across the paper: Section 5 mentions 756 full runs and 3,780 scores, Appendix A refers to 800 generated prompts and images from Section 5, and Appendix D reports 20,640 MLLM-generated prompts. Please reconcile these numbers.
  3. [Tables 1, 2, and 3] Tables 1 and 2 report Spearman and Kendall correlations as percentages (e.g., 46.0) whereas Table 3 reports Kendall's tau as a fraction (0.8571); please use a single convention throughout.
  4. [Appendix C] The appendix text references "12 shows ..." rather than "Figure 12 shows ...", and the appendix title differs from the main paper title; please fix these formatting issues.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the 20-prompt ranking is an empirical comparison, and the disclosed ablation/selection choices are not fitted reductions.

full rationale

No load-bearing step reduces to its own input. MT2IE's 20-prompt ranking is produced by Llava-VQAScore on generated prompts and compared against CLIP-FlanT5 VQAScore on the disjoint GenAIBench prompt set; no equation constructs one ranking from the other. Experiment 1's human-correlation validation on TIFA160 is an external check, not a fit. The choice of 5 iterations (Appendix F, Table 5) and the Yngve difficulty weight (Appendix D) are disclosed ablations/selection decisions; selecting them against the same GenAIBench benchmark is a statistical overfitting concern, not a circular reduction, and the reported tau is an observed outcome rather than an identity. Appendix D even concedes post hoc that Yngve is not the best difficulty proxy, which is a limitation, not a circular step. Self-citations (Ross et al. 2024, Gupta et al. 2024, Astolfi et al. 2024) are contextual and not load-bearing for the central ranking claim. No fitted parameter is renamed as a prediction and no uniqueness result or ansatz is imported from the authors' prior work.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central ranking claim rests on: (1) a VQAScore that is validated only on static prompts transferring to generated prompts; (2) the GenAIBench VQAScore ranking being a valid ground truth; (3) 20 generated prompts from 4 hand-picked seeds being representative. The method introduces no new physical entities. The main free parameters are the iteration count N=5 (selected on the target benchmark), the hand-written seed prompts, the hand-coded adaptive score bins, and the Yngve difficulty weight in adaptive scoring.

free parameters (4)
  • Number of prompt iterations N in MT2IE runs = 5
    Appendix F ablates N in {2, 5, 10, 12} and reports Kendall's tau with the GenAIBench ranking of 0.6672, 0.7275, 0.7004, 0.7025; N=5 is used because it gives the highest correlation. This is a post-hoc selection on the target benchmark.
  • Hand-written seed prompts = 4 seeds across categories: household, people, animals, locations
    Section 5: MT2IEbench20 starts from 4 moderately complex seed prompts chosen by the authors. The ranking outcome depends on these specific prompts; no selection procedure or robustness across alternative seed sets is given.
  • Adaptive score-to-complexity thresholds = 5 bins: [0,0.2], (0.2,0.4], (0.4,0.6], (0.6,0.8], (0.8,1.0]
    Section 6 defines hand-coded rules mapping the previous consistency score to the amount of prompt complexity change. These thresholds are not derived from data or ablated.
  • Prompt difficulty weight in adaptive scoring = Yngve score
    Appendix D: Yngve was selected based on a small preliminary set as the metric most correlated with VQAScore, but the paper's post-hoc analysis on 20,640 prompts (Table 4) shows Yngve has Kendall's tau of -0.097, worse than word count (-0.151) and average word length (0.103). The adaptive ranking is computed with this weighting.
assumptions (3)
  • domain assumption Llava-based VQAScore remains a valid proxy for human judgment on MT2IE-generated prompts.
    Validation in Section 4.2 is on static TIFA160 prompts only; Section 5 applies the same scorer to longer, more compositional generated prompts without collecting human ratings on those prompts.
  • domain assumption The VQAScore ranking on the full 1,600-prompt GenAIBench is a valid ground truth for relative T2I model quality.
    The 20-prompt ranking claim is measured against this ranking. The paper does not provide human rankings for the 8 models on GenAIBench; it relies on VQAScore as the reference.
  • domain assumption 20 MLLM-generated prompts starting from 4 hand-chosen seeds are a representative probe of T2I model capabilities.
    Section 5 assumes 4 seeds and 5 iterations yield a stable, representative evaluation; no coverage analysis or comparison with alternative seed sets is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Modal Language Models as Text-to-Image Model Evaluators." pith.science (2026). https://pith.science/paper/WSSNDNYW

@misc{pith2026250500759,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Language Models as Text-to-Image Model Evaluators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSSNDNYW}},
  note         = {Machine review of arXiv:2505.00759}
}
read the original abstract

The steady improvements of text-to-image (T2I) generative models lead to slow deprecation of automatic evaluation benchmarks that rely on static datasets, motivating researchers to seek alternative ways to evaluate the T2I progress. In this paper, we explore the potential of multi-modal large language models (MLLMs) as evaluator agents that interact with a T2I model, with the objective of assessing prompt-generation consistency and image aesthetics. We present Multimodal Text-to-Image Eval (MT2IE), an evaluation framework that iteratively generates prompts for evaluation, scores generated images and matches T2I evaluation of existing benchmarks with a fraction of the prompts used in existing static benchmarks. Moreover, we show that MT2IE's prompt-generation consistency scores have higher correlation with human judgment than scores previously introduced in the literature. MT2IE generates prompts that are efficient at probing T2I model performance, producing the same relative T2I model rankings as existing benchmarks while using only 1/80th the number of prompts for evaluation.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 9 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024

    Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    work page 2024

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022

    2022

  4. [4]

    Consistency-diversity-realism pareto fronts of conditional image generative models, 2024

    Pietro Astolfi, Marlene Careil, Melissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adriana Romero Soriano, and Michal Drozdzal. Consistency-diversity-realism pareto fronts of conditional image generative models, 2024. https://arxiv.org/abs/2406.10429

    arXiv 2024

  5. [5]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions, 2023. https://arxiv.org/abs/2211.09800

    arXiv 2023

  6. [6]

    Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers, 2023. https://arxiv.org/abs/2301.00704

    arXiv 2023

  7. [7]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558--3568, 2021

    2021

  8. [8]

    All you may need for vqa are image captions, 2022

    Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for vqa are image captions, 2022. https://arxiv.org/abs/2205.01883

    arXiv 2022

Show all 67 references
  1. [9]

    Pixart- : Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- : Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023. https://arxiv.org/abs/2310.00426

    2023 arXiv

  2. [10]

    Visual programming for text-to-image generation and evaluation, 2023

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image generation and evaluation, 2023. https://arxiv.org/abs/2305.15328

    2023 arXiv

  3. [11]

    Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation, 2024

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation, 2024. https://arxiv.org/abs/2310.18235

    2024 arXiv

  4. [12]

    The problem of zombie datasets: A framework for deprecating datasets

    Frances Corry, Hamsini Sridharan, Alexandra Sasha Luccioni, Mike Ananny, Jason Schultz, and Kate Crawford. The problem of zombie datasets: A framework for deprecating datasets. CoRR, abs/2111.04424, 2021. https://arxiv.org/abs/2111.04424

    2021 arXiv

  5. [13]

    Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch,...

    2024 arXiv

  6. [14]

    Scaling rectified flow transformers for high-resolution image synthesis, 2024

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transf...

    2024 arXiv

  7. [15]

    A new readability yardstick

    Rudolf Franz Flesch. A new readability yardstick. The Journal of applied psychology, 32 3: 0 221--33, 1948. https://api.semanticscholar.org/CorpusID:39344661

    1948

  8. [16]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913, 2017

    2017

  9. [17]

    Vector quantized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10696--10706, 2022

    2022

  10. [18]

    Visual programming: Compositional visual reasoning without training, 2022

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training, 2022. https://arxiv.org/abs/2211.11559

    2022 arXiv

  11. [19]

    Improving model evaluation using smart filtering of benchmark datasets

    Vipul Gupta, Candace Ross, David Pantoja, Rebecca J Passonneau, Megan Ung, and Adina Williams. Improving model evaluation using smart filtering of benchmark datasets. arXiv preprint arXiv:2410.20245, 2024

    2024 arXiv

  12. [20]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.08718, 2021. https://arxiv.org/abs/2104.08718

    2021 arXiv

  13. [21]

    Kingma, Ben Poole, Mohammad Norouzi, David J

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022. https://arxiv.org/abs/2210.02303

    2022 arXiv

  14. [22]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering, 2023

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering, 2023. https://arxiv.org/abs/2303.11897

    2023 arXiv

  15. [23]

    Perplexity—a measure of the difficulty of speech recognition tasks

    Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62 0 (S1): 0 S63--S63, 1977

    1977

  16. [24]

    Genai arena: An open evaluation platform for generative models, 2024

    Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models, 2024. https://arxiv.org/abs/2406.04485

    2024 arXiv

  17. [25]

    Text encoders bottleneck compositionality in contrastive vision-language models, 2023

    Amita Kamath, Jack Hessel, and Kai-Wei Chang. Text encoders bottleneck compositionality in contrastive vision-language models, 2023. https://arxiv.org/abs/2305.14897

    2023 arXiv

  18. [26]

    Scaling up gans for text-to-image synthesis, 2023

    Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis, 2023. https://arxiv.org/abs/2303.05511

    2023 arXiv

  19. [27]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation, 2023. https://arxiv.org/abs/2305.01569

    2023 arXiv

  20. [28]

    Imagenhub: Standardizing the evaluation of conditional image generation models

    Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models. arXiv preprint arXiv:2310.01596, 2023

    2023 arXiv

  21. [29]

    Viescore: Towards explainable metrics for conditional image synthesis evaluation, 2024

    Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation, 2024. https://arxiv.org/abs/2312.14867

    2024 arXiv

  22. [30]

    Flux.1-dev, 2024

    Black Forest Labs. Flux.1-dev, 2024. https://huggingface.co/black-forest-labs/FLUX.1-dev

    2024

  23. [31]

    Aesthetic-predictor, 2022

    LAION. Aesthetic-predictor, 2022. https://github.com/LAION-AI/aesthetic-predictor

    2022

  24. [32]

    Genai-bench: Evaluating and improving compositional text-to-visual generation

    Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, et al. Genai-bench: Evaluating and improving compositional text-to-visual generation. arXiv preprint arXiv:2406.13743, 2024 a

    2024 arXiv

  25. [33]

    Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024 b

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024 b . https://arxiv.org/abs/2402.17245

    2024 arXiv

  26. [34]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022. https://arxiv.org/abs/2201.12086

    2022 arXiv

  27. [35]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Procee...

    2014

  28. [36]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291, 2024

    2024 arXiv

  29. [37]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. https://arxiv.org/abs/2304.08485

    2023 arXiv

  30. [38]

    Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation, 2023

    Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation, 2023. https://arxiv.org/abs/2305.11116

    2023 arXiv

  31. [39]

    Crepe: Can vision-language foundation models reason compositionally?, 2023

    Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally?, 2023. https://arxiv.org/abs/2212.07796

    2023 arXiv

  32. [40]

    Sdedit: Guided image synthesis and editing with stochastic differential equations, 2022

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations, 2022. https://arxiv.org/abs/2108.01073

    2022 arXiv

  33. [41]

    Image regeneration: Evaluating text-to-image model via generating identical image with multimodal large language models

    Chutian Meng, Fan Ma, Jiaxu Miao, Chi Zhang, Yi Yang, and Yueting Zhuang. Image regeneration: Evaluating text-to-image model via generating identical image with multimodal large language models. arXiv preprint arXiv:2411.09449, 2024

    2024 arXiv

  34. [42]

    Mapping global dynamics of benchmark creation and saturation in artificial intelligence

    Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13 0 (1): 0 6793, 2022

    2022

  35. [43]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311–318, USA, 2002. Association for Computational ...

    2002

  36. [44]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M \"u ller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  37. [45]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020...

    2021 arXiv

  38. [46]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  39. [47]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj\"orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684--10695, June 2022

    2022

  40. [48]

    What makes a good metric? evaluating automatic metrics for text-to-image consistency

    Candace Ross, Melissa Hall, Adriana Romero-Soriano, and Adina Williams. What makes a good metric? evaluating automatic metrics for text-to-image consistency. In First Conference on Language Modeling, 2024

    2024

  41. [49]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023. https://arxiv.org/abs/2208.12242

    2023 arXiv

  42. [50]

    mindall-e on conceptual captions

    Kim Saehoon, Cho Sanghun, Kim Chiheon, Lee Doyup, and Baek Woonhyuk. mindall-e on conceptual captions. https://github.com/kakaobrain/minDALL-E, 2021

    2021

  43. [51]

    Adversarial diffusion distillation

    Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pages 87--103. Springer, 2025

    2025

  44. [52]

    Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2)

    Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, and William Yang Wang. Who evaluates the evaluations? objectively scoring text-to-image prompt coherence metrics with t2iscorescore (ts2). arXiv preprint arXiv:2404.04251, 2024

    2024 arXiv

  45. [53]

    Text-to-sticker: Style tailoring latent diffusion models for human expression, 2024

    Animesh Sinha, Bo Sun, Anmol Kalia, Arantxa Casanova, Elliot Blanchard, David Yan, Winnie Zhang, Tony Nelli, Jiahui Chen, Hardik Shah, Licheng Yu, Mitesh Kumar Singh, Ankit Ramchandani, Maziar Sanjabi, Sonal Gupta, Amy Bearman, and Dhruv Mahajan. Text-to-sticker: Style tailori...

    2024 arXiv

  46. [54]

    Evalalign: Supervised fine-tuning multimodal llms with human-aligned data for evaluating text-to-image models

    Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, Mengping Yang, Cheng Zhang, and Hao Li. Evalalign: Supervised fine-tuning multimodal llms with human-aligned data for evaluating text-to-image models. arXiv preprint arXiv:2406.16562, 2024

    2024 arXiv

  47. [55]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2022

  48. [56]

    Winoground: Probing vision and language models for visio-linguistic compositionality, 2022 b

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality, 2022 b . https://arxiv.org/abs/2204.03162

    2022 arXiv

  49. [57]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. https://arxiv.org/abs/2401.06209

    2024 arXiv

  50. [58]

    Equivariant similarity for vision-language foundation models, 2023

    Tan Wang, Kevin Lin, Linjie Li, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Equivariant similarity for vision-language foundation models, 2023. https://arxiv.org/abs/2303.14465

    2023 arXiv

  51. [59]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation, 2023 a

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation, 2023 a . https://arxiv.org/abs/2212.11565

    2023 arXiv

  52. [60]

    Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023 b

    Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023 b . https://arxiv.org/abs/2306.09341

    2023 arXiv

  53. [61]

    Conceptmix: A compositional image generation benchmark with controllable difficulty

    Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty. arXiv preprint arXiv:2408.14339, 2024

    2024 arXiv

  54. [62]

    Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023. https://arxiv.org/abs/2304.05977

    2023 arXiv

  55. [63]

    What you see is what you read? improving text-image alignment evaluation, 2023

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation, 2023. https://arxiv.org/abs/2305.10400

    2023 arXiv

  56. [64]

    Proceedings of the American Philosophical Society

    Victor H. Yngve. A model and an hypothesis for language structure. In "Proceedings of the American Philosophical Society", 1960. https://api.semanticscholar.org/CorpusID:18889404

    1960

  57. [65]

    Scaling autoregressive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2 0 (3): 0 5, 2022

    2022 arXiv

  58. [66]

    When and why vision-language models behave like bags-of-words, and what to do about it?, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it?, 2023. https://arxiv.org/abs/2210.01936

    2023 arXiv

  59. [67]

    Gpt-4v(ision) as a generalist evaluator for vision-language tasks, 2023

    Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v(ision) as a generalist evaluator for vision-language tasks, 2023. https://arxiv.org/abs/2311.01361

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.