{"id":"768665b3-3cc4-47ec-b108-d55c1e4c6058","arxiv_id":"2505.00759","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MT2IE uses a single open-source multimodal LLM to generate 20 progressively harder prompts and score image-text consistency, reproducing the 1,600-prompt GenAIBench ranking of 8 text-to-image models.","lead":"This paper presents MT2IE, a framework where one open-source multimodal language model writes new image prompts, generates images with text-to-image models, and scores how well each image matches its prompt. It reports that 20 such generated prompts reproduce the model ranking of a 1,600-prompt benchmark, and that the scoring component correlates with human judgment better than previous automatic metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 20-prompt claim rests on seed-averaged ranks and an iteration count tuned on the same GenAIBench ranking used as ground truth; per-run correlation is not reported.","rationale":"The reader's weakest assumption concerns whether Llava-based VQAScore remains a valid human proxy on longer, MT2IE-generated prompts. That is a real concern, but it attacks the human-alignment interpretation more than the paper's stated benchmark-matching claim: even an imperfect automatic scorer could, in principle, reproduce the GenAIBench VQAScore ranking that the paper uses as ground truth. The more load-bearing problem is that the evidence for 'only 20 prompts' may not be evidence about a single 20-prompt run at all. The text explicitly reports seed-averaged ranks over 5 seeds and chooses the number of iterations by comparing against the same GenAIBench ranking. Both practices can inflate the reported Kendall's tau, and neither was identified by the reader. A concrete per-seed and held-out comparison would settle whether the 20-prompt claim is real or an artifact of ensembling and target-benchmark tuning. I keep the verdict CONDITIONAL rather than REJECT because the concern is addressable with additional analysis and does not contradict the positive TIFA160 human-correlation result, but the conditions should include per-seed reporting and a held-out benchmark evaluation.","tokens_in":21160,"tokens_out":10419,"duration_ms":105498,"concrete_test":"Recompute the main comparison per seed and per fixed configuration. For each of the 5 random seeds, compute the Kendall's tau between the 20-prompt Llava-VQAScore ranking and the full GenAIBench VQAScore ranking, and report the distribution (min, median, max) rather than a seed-averaged rank. Then compare against a random 100-prompt GenAIBench VQAScore subset, also averaged over 10 seeds, using the same target ranking. If the median per-seed tau is not clearly above the random-100 baseline, or if individual per-seed tau values are low and highly variable, the claim that a single 20-prompt run reproduces the benchmark ranking is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim is that MT2IE, 'generating and scoring only 20 prompts per ranking,' reproduces the full 1,600-prompt GenAIBench VQAScore ranking with Kendall's tau of 0.8571 (Section 5, Table 3). The supporting analysis does not establish a single 20-prompt ranking. Section 5 says runs are over 5 random seeds, and Figure 7 reports 'Average rank ... over all seeds'; thus the reported tau is compatible with a seed-averaged ranking that consumes 5 × 20 = 100 prompt-image pairs, not 20. Additionally, Appendix F, Table 5 selects the 5-iteration setting as best among {2, 5, 10, 12} against the very same GenAIBench ranking used as the comparison target. With only 8 models, choosing the best of several configurations can inflate rank correlation, and no held-out benchmark is used. If individual 20-prompt runs are unstable, the true efficiency gain is 100 prompts (16x), not 20 prompts (80x), and the advantage over a random 100-prompt GenAIBench subset was never tested. This concern is about the evaluation protocol, not the scorer's human alignment: it attacks whether the reported '20 prompts' protocol is what produces the agreement.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Multimodal Text-to-Image Eval (MT2IE), a framework in which an open-source multimodal LLM (Llava-v1.6-34B) acts as both prompt generator and scorer for text-to-image (T2I) evaluation. In Experiment 1, the authors validate three open MLLMs as zero-shot scorers on the static TIFA160 benchmark, comparing GQA-style question generation and VQAScore-style likelihood scoring against human ratings, and report higher human correlations than existing GQA baselines. In Experiment 2, MT2IE iteratively expands four seed prompts over five iterations to produce 20 prompts, scores images from 8 T2I models with Llava-VQAScore, and reports a Kendall's tau of 0.8571 between the resulting model ranking and the 1,600-prompt GenAIBench VQAScore ranking. Experiment 3 adds adaptive prompting, in which prompt difficulty is adjusted based on the previous score, and difficulty-weighted scores are used to reproduce GenAIBench rankings. An appendix extends the framework to aesthetic scoring.","tokens_in":21405,"tokens_out":10396,"duration_ms":94849,"significance":"The paper addresses a real problem—static T2I benchmarks saturate and leak—and the proposal of a single open-source MLLM that both writes and scores prompts is a useful step toward adaptive evaluation. The TIFA160 human-correlation results are a genuine contribution: they show that open MLLMs can outperform multi-model GQA pipelines, which contradicts earlier findings with open models. The iterative prompt-generation procedure also produces verifiable difficulty progression. If the efficiency claim is re-established under a correct accounting of the evaluation budget and with proper hyperparameter selection, the framework would be practically valuable for cheap model comparison. At present the paper does not ship code, and the central 20-prompt/80x-efficiency claim is not supported by the reported aggregate analysis; the claims about matching human judgment are also limited to static prompts. These issues are addressable, so the manuscript is a candidate for major revision rather than rejection.","major_comments":[{"comment":"The headline efficiency claim is not supported by the reported protocol. Section 5 says \"We run MT2IE-benchmark generation over 5 random seeds\" and Figure 7 plots average ranks over all seeds, so the Kendall's tau of 0.8571 in Table 3 appears to be computed on a seed-averaged ranking that consumes 5 × 20 = 100 prompt-image pairs per T2I model, not 20. Please report the per-seed rank correlations and their variance, and compare the seed-averaged MT2IE ranking against a seed-averaged random 100-prompt GenAIBench subset under the same aggregation; otherwise the abstract's \"1/80th\" claim is not established.","section":"Section 5, Table 3, Figure 7"},{"comment":"The 5-iteration/20-prompt configuration is selected as the best among {2, 5, 10, 12} by comparing Kendall's tau against the full GenAIBench VQAScore ranking, which is the same target used to evaluate the final method in Section 5. With only 8 models, optimizing over four configurations can inflate the reported rank correlation, and no held-out benchmark or partition is used. Please specify the configuration before evaluation or use a separate validation set for hyperparameter selection, and report the tau for every iteration count as a sensitivity range rather than only the best configuration.","section":"Appendix F, Table 5"},{"comment":"Section 6's adaptive ranking depends on the Yngve difficulty weight, but Appendix D's post-hoc analysis (Table 4) shows that Yngve score has Kendall's tau of only -0.0970 with VQAScore and is not the most correlated metric (average word length, tau 0.1025, is higher). The paper says Yngve was selected based on a preliminary correlation analysis, which means the adaptive result in Figure 7 is conditional on a weakly supported and possibly overfit choice. Please report the Figure 7 adaptive ranking under all candidate difficulty metrics, and discuss why the selected weight is appropriate despite Table 4.","section":"Appendix D, Section 6"},{"comment":"The human-correlation validation in Section 4.2 is performed only on static TIFA160 prompts, and no human ratings are collected for the longer, more compositional MT2IE-generated prompts used in Sections 5 and 6. Since the GenAIBench target ranking is itself obtained with CLIP-FlanT5 VQAScore, the reported agreement with GenAIBench could largely reflect agreement between two automatic VQAScore variants rather than human-aligned evaluation. Additionally, Section 3 states that MLLM-generated benchmarks \"preserve relative model rankings computed by human annotators in prior work,\" but Section 5 compares against an automatic VQAScore ranking, not human-rated images. Please collect human judgments on a sample of MT2IE-generated prompts or explicitly re-frame the claim as agreement with an automatic benchmark.","section":"Section 4.2, Section 5"},{"comment":"The paper reports two different Kendall tau values for the same downsampled GenAIBench (20 prompts) baseline: Table 3 gives -0.6428 for VQAScore, while Appendix F, Table 5 gives -0.8693. Unless these correspond to different scoring metrics, seeds, or aggregation schemes, one of these numbers is incorrect; please reconcile the baseline and report the protocol used in each table.","section":"Table 3 vs Appendix F, Table 5"}],"minor_comments":[{"comment":"The text states MT2IE is \"utilizing 20 times fewer prompts\" when comparing 20 prompts to 1,600 prompts; the correct factor is 80 times fewer, as stated in the abstract and Section 6.","section":"Section 5"},{"comment":"The experimental counts are inconsistent across the paper: Section 5 mentions 756 full runs and 3,780 scores, Appendix A refers to 800 generated prompts and images from Section 5, and Appendix D reports 20,640 MLLM-generated prompts. Please reconcile these numbers.","section":"Section 5, Appendix A, Appendix D"},{"comment":"Tables 1 and 2 report Spearman and Kendall correlations as percentages (e.g., 46.0) whereas Table 3 reports Kendall's tau as a fraction (0.8571); please use a single convention throughout.","section":"Tables 1, 2, and 3"},{"comment":"The appendix text references \"12 shows ...\" rather than \"Figure 12 shows ...\", and the appendix title differs from the main paper title; please fix these formatting issues.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the TIFA160 validation is solid. I would be comfortable with acceptance after the efficiency accounting, per-seed analysis, and hyperparameter-selection issues are addressed. The inconsistency between Tables 3 and 5 and the numerical mismatches in experimental counts should be fixed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea—using a single open-source MLLM to iteratively generate and score prompts—is genuinely new, and the TIFA160 human-correlation result in Experiment 1 is a real, reproducible positive. But the headline efficiency claim, that 20 MLLM-generated prompts reproduce the 1600-prompt GenAIBench ranking, is shakier than the abstract suggests.\n\nWhat is actually new and good: prior work either used MLLMs as static judges or used LLMs to generate questions for fixed prompts. Here one open-source model (Llava-1.6-34B) both writes prompts and scores images, and the adaptive feedback loop is a natural extension. Experiment 1 stands on its own: on TIFA160, Llava's VQAScore correlates better with human ratings (ρ=67.5) than the previous SOTA CLIP-FlanT5 (ρ=66.9) and clearly better than TIFA (ρ=46.0) or DSG (ρ=57.1). That result is worth citing even if the rest of the paper needs work.\n\nSoft spots, in rough order of importance. First, the main table (Table 3, τ=0.8571) and the appendix ablation (Table 5, τ=0.7275) report different numbers for the same 20-prompt iterative setting, and the paper never explains the discrepancy. Second, the τ=0.8571 comes from ranking seed-averaged scores over 5 runs, so the actual evaluation consumed 5×20=100 prompt-image pairs; no per-seed Kendall's tau is reported, so we cannot tell whether an individual 20-prompt run is stable. Third, the iteration count N=5 is selected in Appendix F against the very same GenAIBench ranking used as the comparison target, which can inflate correlation when only 8 models are ranked. Fourth, the ground truth is VQAScore on GenAIBench, not human judgment, so the efficiency claim is really about reproducing an automatic ranking. Fifth, the adaptive experiments pick the Yngve difficulty weight based on preliminary correlation with VQAScore and then validate against a VQAScore-based ranking—mildly circular, and Table 4 in their own appendix shows Yngve has near-zero correlation on the 20k-prompt set. Finally, no human ratings are collected on MT2IE-generated prompts; scorer validity on long, compositional prompts is assumed. Minor: the paper says \"20 times fewer prompts\" in Section 5 when 1600/20=80, and there are a couple of citation slips.\n\nThese are all addressable. I would send this to a serious referee, not desk-reject, because the core idea and the Experiment 1 result deserve engagement. A good referee will ask for per-seed correlations, a held-out target (e.g., a human preference benchmark or a different static benchmark), a reconciliation of Tables 3 and 5, and ideally released code.","headline":"Clever idea and one solid human-correlation result, but the 20-prompt efficiency headline is weakened by seed averaging, selection on the target benchmark, and a numeric mismatch between the main table and the appendix.","tokens_in":22000,"tokens_out":4583,"would_cite":true,"duration_ms":42840,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single open-source multimodal language model can rank eight text-to-image models using just 20 self-generated prompts, closely matching the ranking of a 1,600-prompt benchmark.","keywords":["text-to-image evaluation","multimodal language models","dynamic benchmarks","prompt generation","VQAScore","image-text consistency","adaptive evaluation","large language models as evaluators"],"falsifier":"Have human annotators score the images generated from MT2IE's 20 prompts for the eight text-to-image models, then compute the ranking correlation between MT2IE's scores and the human scores; if that correlation is significantly lower than the correlation reported on TIFA160, the 20-prompt ranking would match GenAIBench for the wrong reason.","tokens_in":20938,"feed_emoji":"🖼️","tokens_out":6598,"duration_ms":63398,"temperature":0.7,"pith_summary":"This paper sets out to replace static, hand-written prompt benchmarks in text-to-image evaluation with a single multimodal language model that both writes prompts and scores the resulting images. Its central claim is that MT2IE, an iterative loop that makes each new prompt harder and then scores image-text consistency, reproduces the model ranking of a 1,600-prompt benchmark while using only 20 prompts. The authors also report that scores from this loop correlate better with human judgment than prior automatic metrics on a static benchmark. If the claim holds, evaluation of image-generation models would no longer depend on large curated prompt sets and could instead adapt automatically to each model's strengths and failure points.","feed_headline":"20 generated prompts match a 1,600-prompt T2I ranking","feed_subtitle":"One open-source multimodal model writes and scores its own prompts, reproducing benchmark rankings at 1/80th the cost.","key_machinery":"The load-bearing mechanism is the iterative prompt-difficulty loop: a multimodal language model rewrites a previous prompt into a harder one by adding an object, spatial relation, or attribute; the text-to-image model generates an image; the same MLLM scores the image-prompt consistency; and the score feeds the next rewrite. The scoring component is VQAScore, a named metric that measures how likely the model is to answer \"yes\" to the prompt-description question, validated against human ratings on a static benchmark. The loop is what makes few-prompt efficiency possible: instead of sampling from a static pool, the MLLM concentrates difficulty where it separates models.","core_discovery":"The paper's central discovery is that an open-source multimodal language model can serve as a complete evaluator for text-to-image models: it generates the evaluation material and grades the result. In MT2IE, the evaluator takes a seed prompt, adds one object, attribute, or spatial relation per iteration to make the prompt harder, has the text-to-image model draw the image, then scores the image with VQAScore, the likelihood that the model answers \"yes\" to \"Does this figure show {prompt}?\". On a standard static benchmark of 160 prompts, this single-model pipeline achieves higher Spearman and Kendall correlations with human ratings than several existing question-answering and CLIP-based metrics. Rank-ordering eight text-to-image models with 20 generated prompts gives a Kendall's tau of 0.8571 against the ranking from the full 1,600-prompt benchmark, while random 20-prompt subsamples of that benchmark give uninformative or inverted rankings. A variant that adapts prompt complexity to each model's running score preserves the same ranking, and an appendix extends the loop to aesthetic quality scoring with a reported Kendall's tau of 0.691 against a pretrained aesthetics predictor.","pith_inferences":["If the reported 0.8571 ranking match holds beyond the eight models tested, MT2IE could enable nearly free regression testing of every new text-to-image checkpoint, making benchmark saturation a routine concern rather than an episodic one.","The method's validity depends on the scorer tracking human judgment on the harder, generated prompts; a direct human-rating study on those prompts is the natural next test, and one the paper does not report.","A risk left implicit is that text-to-image developers could optimize against the MLLM's scoring signal; whether that leads to genuine improvements or to reward hacking is an open question.","The same iterative difficulty loop could be pointed at other generative domains, such as video or 3D generation, where static benchmarks are even less stable."],"forward_implications":["Text-to-image model evaluation could be done with about 20 generated prompts instead of 1,600 hand-written ones, cutting generation and annotation cost by a factor of 80.","Static benchmarks that saturate or leak into training data could be replaced by dynamic, per-run prompts, making overfitting to fixed prompt sets less attractive.","Because the same open-source multimodal model generates and scores, evaluation would no longer require proprietary language models or multi-model pipelines.","An adaptive version gives each model a bespoke difficulty trajectory, so weaker models receive informative easier prompts and stronger models are pushed to harder ones, while the resulting ranking still matches the full benchmark.","The framework generalizes beyond consistency to other judgment axes, such as aesthetic quality, by swapping the scoring instruction."],"supporting_citations":[{"why":"Supplies the TIFA160 static benchmark, its question sets, and human-correlation baselines that the single-MLLM scorer must match or beat.","marker":"Hu et al. (2023)"},{"why":"Introduces VQAScore, the scoring rule MT2IE uses for image-text consistency, and provides the previous state-of-the-art correlation baseline.","marker":"Lin et al. (2024)"},{"why":"Provides the full 1,600-prompt GenAIBench benchmark and the model ranking that MT2IE aims to reproduce with only 20 prompts.","marker":"Li et al. (2024a)"},{"why":"Supplies the human ratings and generated images used in Experiment 1 to validate MLLM-generated question scores on TIFA160.","marker":"Cho et al. (2024)"},{"why":"Reports that open-source multimodal models were ineffective evaluators, the claim MT2IE directly challenges.","marker":"Ku et al. (2024)"}],"fun_headline_variants":["20 self-generated prompts match a 1,600-prompt T2I ranking","One multimodal model evaluates T2I with self-written prompts","1/80th prompts, same ranking: MLLM as T2I evaluator","Self-graded prompts beat static benchmarks at 1/80th cost","MLLM writes prompts, scores images, matches human rankings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the MLLM's VQAScore remains a valid proxy for human judgment when applied to the longer, more complex prompts it generates, even though the human-correlation validation was done only on static TIFA160 prompts and the benchmark ranking it matches is itself VQAScore-based.","fun_headline_variants_meta":{"raw":{"variants":["20 self-generated prompts match a 1,600-prompt T2I ranking","One multimodal model evaluates T2I with self-written prompts","1/80th prompts, same ranking: MLLM as T2I evaluator","Self-graded prompts beat static benchmarks at 1/80th cost","MLLM writes prompts, scores images, matches human rankings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000876,"raw_usage":{"total_tokens":3804,"prompt_tokens":971,"completion_tokens":2833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2736}},"tokens_in":587,"tokens_out":2833,"duration_ms":21552,"temperature":1.0,"reasoning_tokens":2736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:36:25.481103+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators score the images generated from MT2IE's 20 prompts for the eight text-to-image models, then compute the ranking correlation between MT2IE's scores and the human scores; if that correlation is significantly lower than the correlation reported on TIFA160, the 20-prompt ranking would match GenAIBench for the wrong reason.","supporting_citations":[],"review_version":1}