Pith. sign in

REVIEW 2 major objections 3 minor 36 references

ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization

T0 review · 2 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Optimized prompts substantially improve compositional text-to-image generation across model architectures, showing that rigid benchmarks systematically underestimate model capabilities.

desk verdict Optimizing prompts before benchmarking reveals real gains and reorders models, but the 'true capability' claim needs an independent final judge, not just a swapped optimizer. read the letter →

arxiv 2507.03275 v1 pith:3HMQXG4W submitted 2025-07-04 cs.CV cs.LG

Haosheng Gan , Berk Tinaz , Mohammad Shahab Sepehri , Zalan Fabian , Mahdi Soltanolkotabi This is my paper
classification cs.CVcs.LG
keywords text-to-imagegenerationpromptoptimizationcompositionalbenchmarkingvision-languagemodelevaluationdiffusionmodelscross-modeltransferability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-to-image models are extremely sensitive to how a prompt is worded, and this paper argues that standard benchmarks, which freeze one wording per test item, therefore understate what the models can actually draw. It introduces ConceptMix++, an iterative loop that rewrites each benchmark prompt until a vision-language judge confirms the generated image satisfies every compositional criterion, then reruns the benchmark under those optimized prompts. Across three diffusion models and seven complexity levels ($k=1$ to $k=7$), optimization raises scores by up to roughly 20 percentage points in the mid-complexity range, with the largest gains in spatial relations, shapes, sizes, and textures. The paper's conclusion is that fixed-prompt leaderboards reward prompt sensitivity more than visual skill, and that per-model optimized prompting gives a fairer picture of generation ability.

What carries the argument

The load-bearing component is the iterative prompt-optimization loop. Starting from the ConceptMix prompt for a set of $k+1$ criteria, the diffusion model $D$ generates an image, a vision-language verifier $V$ scores it with the product of per-criterion yes-likelihoods and returns per-criterion textual feedback, and an LLM update function $U_{\mathrm{LLM}}$ rewrites the prompt using the best-scoring prompt so far plus the history of prompt-score-feedback tuples. This is a discrete analogue of gradient descent in which the LLM acts as both gradient estimator and update rule, and the history buffer plays the role of a trust region that keeps updates anchored to the best-performing prompt.

What would settle it

Have human annotators score the same original and optimized image sets against the same criteria (for example, 'exactly four cows', 'glass texture', 'tiny rose'). If humans find no consistent improvement, or prefer the original images, where the vision-language model reports large gains, then the optimization is chasing evaluator preference rather than visual capability and the underestimation claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the performance gap between original and optimized prompts is a measurement artifact of benchmark design rather than a fixed property of the models: the same list of visual criteria can be satisfied far more often once a phrasing is found that the model can follow. The authors demonstrate this by treating prompt selection as discrete optimization over natural language, with the product of a vision-language verifier's yes-likelihoods as the objective and an LLM as the update rule, and by showing consistent gains on DALL·E 3, Stable Diffusion 3.5, and Playground v2.5 at every complexity level from $k=1$ to $k=7$. They also argue the gains are genuine rather than evaluator leakage, because prompts optimized with a different judge (InternVL3-8B) still score nearly as well under GPT-4o, and that optimized prompts transfer across model families, suggesting shared preferences for effective phrasing.

Load-bearing premise

The argument collapses if the vision-language judge is not an honest measure of whether the image meets each stated criterion, because optimizing against a biased judge could raise scores by catering to the judge's preferences without improving the images.

Editorial extensions

If this is right

  • Rankings from fixed-prompt benchmarks may be unreliable, because a model that appears weak on spatial or shape criteria can look strong once its prompts are optimized.
  • Category-level results separate prompt-sensitivity from architectural limits: spatial, shape, size, and texture respond to rewording, while number and object do not, so counting failures are real model limitations.
  • Because optimized prompts transfer across the three tested model families, practitioners can optimize prompts on a cheaper model and apply them to a more expensive one.
  • Even optimized prompts decline sharply at complexity levels $k = 6$ and $k = 7$, indicating an architectural ceiling that no amount of prompt engineering removes.
  • Benchmark scores should be read as lower bounds on model capability rather than as ceilings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate stress test the paper only partially runs: hold out the final evaluator entirely (use a different VLM or human panel as the judge, not just as the optimizer) and require the improvement to survive; the InternVL3 control varies only the optimization judge.
  • If cross-model transferability is real, a shared library of optimized prompts could be distilled and reused across models, but that would make benchmark numbers depend on which prompt set is used, so future comparisons should report both original and optimized scores.
  • The equal-budget experiment hints at a practical byproduct the paper underplays: the same loop can serve as a prompt autotuner for end users with a fixed number of image-generation calls.
  • The paper's definition of 'true capability' is a choice rather than a measurement: a model that needs heavily engineered prompts to succeed could be said to lack robust instruction following, so the framing of optimized prompts as revealing hidden skill is contestable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. The paper proposes ConceptMix++, a prompt-optimization framework for text-to-image (T2I) benchmarking. Starting from ConceptMix's compositional prompts, it iteratively refines prompts for each T2I model using feedback from a vision-language model (VLM), with the goal of maximizing the same VLM-based criterion score used for evaluation. Experiments on DALL·E 3, Stable Diffusion 3.5, and Playground v2.5 report consistent gains of up to roughly 20% absolute in average and best-of-5 scores after optimization, plus category-specific gains and cross-model prompt transferability. The paper argues that fixed-prompt benchmarks systematically underestimate true model capabilities.

Significance. If the central claim holds, the paper would have significant implications for how T2I models are ranked and evaluated, since it suggests that observed differences across models may partly reflect prompt-sensitivity artifacts rather than visual-generation ability. The paper includes several appropriate safeguards: an alternative-judge control (Section 4.6), cross-model transfer experiments (Section 4.4), an iteration-count ablation (Section 4.5), and an equal-compute-budget comparison in Appendix E. These go beyond what is typical for prompt-optimization papers. However, the main claim rests on the assumption that the GPT-4o verifier is a valid and unbiased measure of whether a generated image satisfies the given criteria. The reported gains are optimized against exactly this score, and the only control changes the optimization-time judge while leaving the final evaluator unchanged, so the evidence is not yet sufficient to establish the 'hidden capabilities' / 'systematic underestimation' conclusion.

major comments (2)
  1. [Section 3.3, Section 4.6, Section 6.1] The optimization objective in Section 3.3 is exactly the evaluation score V(D(p)), so the headline improvements are fitted results rather than independent predictions. The InternVL3-8B control in Section 4.6 changes only the judge used during optimization; both optimized prompt sets are then scored by the same GPT-4o verifier ('The optimized prompts are then evaluated using GPT-4o to maintain consistency'). If the two VLMs share a systematic bias, such as rewarding lexical explicitness regardless of whether the object is actually depicted, the control would still show large improvements. The paper's own Section 6.1 concedes that 'the use of a single VLM for evaluation may not capture all aspects of visual quality that human evaluators would consider.' To support the central claim of Section 5.1 that existing benchmarks 'may be systematically underestimating model capabilities,' the manuscript needs a human evaluation on a representative subset of prompts or an evaluation with a held-out, non-optimizer final judge that is never used during optimization.
  2. [Section 4.1, Table 1] No significance testing is reported for any of the headline differences, and the best-of-5 columns in Table 1 are single point estimates without error bars. The ± values for average scores are not defined in the text: it is not stated whether they are standard errors over the 300 datapoints, over the 5 images, or something else, and the procedure for selecting '300 datapoints' per complexity level is unspecified, including whether the same datapoints are used across models. Without confidence intervals or paired significance tests, the claim of 'up to ≈20% absolute gains' could be consistent with substantial sampling noise, particularly for the larger-k rows where absolute scores are low and the relative variance is high.
minor comments (3)
  1. [Section 4.3, Figure 3] The category-wise heatmaps show negative deltas for several individual cells (e.g., DALL·E 3 style at k=1 is -2.8), which is not acknowledged in the text. The conclusion in Section 7 states that optimization 'consistently enhances model performance,' which is stronger than the category-level evidence supports. The aggregation of the per-category score S_{M,k,c} across datapoints and images is also not specified.
  2. [Section 4.6] The text says InternVL3-8B-optimized prompts 'closely match or exceed' GPT-4o-optimized prompts, but for DALL·E 3 the InternVL-optimized score (0.386) is below the GPT-4o-optimized score (0.419); only for SD 3.5 does it exceed (0.358 vs 0.354). The phrasing should be adjusted to match the reported numbers.
  3. [Appendix A.4.1] The text states 'we fix the number of iterations to 5 for all experiments,' but Section 4.5 reports an ablation over T ∈ {0,1,2,3,4,5,10,15}. This should be qualified as fixing T=5 for the main experiments.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline gains are the optimized objective itself; all final evaluations use GPT-4o, so the 'hidden capabilities' claim reduces to an unvalidated verifier.

  1. fitted input called prediction [Section 3.3 (optimization objective p* = arg max V(D(p))) and Section 5.1 (capability claim)]
    "Given a T2I diffusion model D and a vision-language evaluator V, the goal is to find an optimal prompt p∗ that maximizes the evaluation score: p∗ = arg max p V(D(p)) ... The fact that optimized prompts can improve performance by up to 20% suggests that existing benchmarks may be systematically underestimating model capabilities."

    The optimized prompt is selected by maximizing GPT-4o's criterion score V(D(p)), and the headline evidence is the increase in that same GPT-4o score (Table 1). The 'hidden capability' gain is therefore the value of the training objective itself, not an independent measurement: V(D(p*)) versus V(D(p0)) is forced by the argmax (modulo stochasticity/overfitting) rather than evidence about true visual ability. The conclusion requires V to be externally validated as ground truth; no human-label calibration or independent final evaluator is provided. The InternVL3-8B control changes only the optimization judge while GPT-4o remains final evaluator, so the maximized metric and the capability-defining metric are still the same V-family.

  2. other [Section 4.4 (Cross-Model Prompt Transferability) and Section 5.2]
    "Stable Diffusion 3.5 performs significantly better with DALL·E 3-optimized prompts compared to original prompts, and its performance closely approaches that achieved with prompts optimized for itself. This indicates a high degree of cross-model transferability, suggesting that the optimized prompts capture phrasing patterns that are effective among all models."

    Transfer is measured only by GPT-4o, the same V used to select the prompts. Since p* was chosen to maximize GPT-4o's score on source-model images, observing that the same prompts raise GPT-4o's score on target-model images is consistent with prompts that exploit GPT-4o's phrasal preferences, not necessarily with shared visual representations. The claim 'shared underlying representations across different architectures' (Section 5.2) is not implied by the data unless V is independent of the optimization objective; the paper provides no such independent judge.

full rationale

ConceptMix++ is not structurally self-citational: it builds on ConceptMix [29] by different authors and on TextGrad [30] as an external method. The circularity lies in the measurement chain. The optimization objective (Section 3.3) is exactly the GPT-4o verification score used as the benchmark metric (Section 3.2), so the 10–20% gains in Table 1 are the fitted value of the objective rather than an out-of-sample prediction. The central generalization that rigid benchmarks 'systematically underestimate model capabilities' (Section 5.1) therefore rests on assuming GPT-4o's yes/no answers are a valid, bias-free measure of criterion satisfaction. The alternative-judge experiment (Section 4.6) replaces the optimizer but keeps GPT-4o as the final evaluator, so it does not test shared-evaluator bias; the cross-model transfer experiments (Section 4.4) likewise score all models with GPT-4o. These controls show that the optimization signal transfers across generators and optimizer VLMs, which is useful engineering evidence, but they do not break the definitional link between the optimized target and the reported capability measure. Section 6.1 itself concedes that 'the use of a single VLM for evaluation may not capture all aspects of visual quality that human evaluators would consider.' Since the paper's headline claim goes beyond 'our optimization raises our metric' to 'true capabilities are higher than benchmarks show,' and that step is not independently grounded, the analysis is partially circular: the measured gain is real but is, by construction, a gain on the objective being optimized.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework rests on the validity of the VLM evaluator as a proxy for compositional correctness, on the representativeness of ConceptMix criteria, and on the LLM's ability to convert score-feedback history into improved prompts. The only tuned hyperparameter reported is the iteration count T=5; data sampling and image count are experimental choices that affect the estimates.

free parameters (3)
  • T (max optimization iterations) = 5
    Chosen by hand after ablation in Section 4.5; affects performance and overfitting.
  • Number of datapoints per complexity level = 300
    Selected in Section 4.1 without describing sampling method; affects all reported results.
  • Number of images per prompt = 5
    Used for average and best-of-5 scores (Section 4.1); arbitrary choice that influences variance and best-of-5 gains.
assumptions (4)
  • domain assumption GPT-4o as verifier V gives correct yes/no answers for whether an image satisfies a criterion.
    Section 3.2 treats V as ground truth; no human validation is provided.
  • domain assumption ConceptMix criteria and style categories define the relevant space of compositional capability.
    Section 3.2 adopts ConceptMix as the measurement instrument; the paper notes in Section 6.5 this may not capture all aspects.
  • domain assumption An LLM can reliably improve prompts from history of scores and feedback.
    Section 3.3 assumes the LLM update function improves the objective; supported only by empirical results.
  • domain assumption The diffusion models are stable enough that optimization-time scores generalize to test-time generations.
    Section 4.5 discusses stochasticity and overfitting, suggesting this assumption is only partially met; mitigated by 5 test images.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization." pith.science (2026). https://pith.science/paper/3HMQXG4W

@misc{pith2026250703275,
  author       = {Pith},
  title        = {Pith review of: ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3HMQXG4W}},
  note         = {Machine review of arXiv:2507.03275}
}
read the original abstract

Current text-to-image (T2I) benchmarks evaluate models on rigid prompts, potentially underestimating true generative capabilities due to prompt sensitivity and creating biases that favor certain models while disadvantaging others. We introduce ConceptMix++, a framework that disentangles prompt phrasing from visual generation capabilities by applying iterative prompt optimization. Building on ConceptMix, our approach incorporates a multimodal optimization pipeline that leverages vision-language model feedback to refine prompts systematically. Through extensive experiments across multiple diffusion models, we show that optimized prompts significantly improve compositional generation performance, revealing previously hidden model capabilities and enabling fairer comparisons across T2I models. Our analysis reveals that certain visual concepts -- such as spatial relationships and shapes -- benefit more from optimization than others, suggesting that existing benchmarks systematically underestimate model performance in these categories. Additionally, we find strong cross-model transferability of optimized prompts, indicating shared preferences for effective prompt phrasing across models. These findings demonstrate that rigid benchmarking approaches may significantly underrepresent true model capabilities, while our framework provides more accurate assessment and insights for future development.

Figures

Figures reproduced from arXiv: 2507.03275 by the authors.

Figure 1
Figure 1. Text-to-image models are sensitive to the specific phras [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our prompt optimization framework. Starting with an initial prompt, our iterative process first generates images [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Heatmaps showing improvement magnitude across vi [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Average score of Stable Diffusion 3.5 using original [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of different iteration numbers [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison between prompts optimized [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Example of the our prompt optimization process. Left: Before optimization, the image shows 3 black cows with reflections and [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison across models and complexity levels, showing original vs. optimized prompt performance. Left column: [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Radar graphs showing category improvements for complexity level [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Radar graphs showing category improvements for complexity level [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Radar graphs showing category improvements for complexity level [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Radar graphs showing category improvements for complexity level [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Radar graphs showing category improvements for complexity level [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Radar graphs showing category improvements for complexity level [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Radar graphs showing category improvements for complexity level [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: DALL·E 3 performance with SD 3.5 optimized prompts compared to original prompts and self-optimized prompts. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: DALL·E 3 performance with Playground v2.5 optimized prompts compared to original prompts and self-optimized prompts. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Stable Diffusion 3.5 performance with DALL·E 3 optimized prompts compared to original prompts and self-optimized prompts. [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: Stable Diffusion 3.5 performance with Playground v2.5 optimized prompts compared to original prompts and self-optimized [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]
Figure 20
Figure 20. Figure 20: Playground v2.5 performance with DALL·E 3 optimized prompts compared to original prompts and self-optimized prompts. [PITH_FULL_IMAGE:figures/full_fig_p020_20.png]
Figure 21
Figure 21. Figure 21: Playground v2.5 performance with Stable Diffusion 3.5 optimized prompts compared to original prompts and self-optimized [PITH_FULL_IMAGE:figures/full_fig_p020_21.png]
Figure 22
Figure 22. Figure 22: Comparison of DALL·E 3 performance under equal computational budget (5 images). The graph shows Best-of-5 Score for [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: Comparison of Stable Diffusion 3.5 performance under equal computational budget (5 images). The graph shows Best-of-5 [PITH_FULL_IMAGE:figures/full_fig_p021_23.png]
Figure 24
Figure 24. Figure 24: Comparison of Playground v2.5 performance under equal computational budget (5 images). The graph shows Best-of-5 Score [PITH_FULL_IMAGE:figures/full_fig_p022_24.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 13 canonical work pages

  1. [1]

    Stable diffusion 3.5, 2024

    Stability AI. Stable diffusion 3.5, 2024. 3

    work page 2024

  2. [2]

    Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models

    Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 20041– 20053, 2023. 1

    work page 2023

  3. [3]

    Diffusion Mod- els Beat GANs on Image Synthesis

    Prafulla Dhariwal and Alex Nichol. Diffusion Mod- els Beat GANs on Image Synthesis. arXiv preprint arXiv:2105.05233, 2021. 1

    arXiv 2021

  4. [4]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  5. [5]

    Denoising Diffu- sion Probabilistic Models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffu- sion Probabilistic Models. arXiv preprint arXiv:2006.11239,

    arXiv 2006

  6. [6]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1–33, 2022. 1

    work page 2022

  7. [7]

    Video dif- fusion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video dif- fusion models. arXiv preprint arXiv:2204.03458, 2022. 1

    arXiv 2022

  8. [8]

    T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- tion

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- tion. Advances in Neural Information Processing Systems , 36:78723–78747, 2023. 1, 2, 5

    work page 2023

Show all 36 references
  1. [9]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 3

    2024 arXiv

  2. [10]

    Diffwave: A versatile diffusion model for audio synthesis

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761, 2020. 1

    2009 arXiv

  3. [11]

    Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Lin- miao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image genera- tion. arXiv preprint arXiv:2402.17245, 2024. 3

    2024 arXiv

  4. [12]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

    2022

  5. [13]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  6. [14]

    Improved De- noising Diffusion Probabilistic Models

    Alex Nichol and Prafulla Dhariwal. Improved De- noising Diffusion Probabilistic Models. arXiv preprint arXiv:2102.09672, 2021. 1

    2021 arXiv

  7. [15]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models.arXiv preprint arXiv:2112.10741, 2021. 2

    2021 arXiv

  8. [16]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. Technical report, Ope- nAI, 2023. 2

    2023

  9. [17]

    Dall·e 3, 2024

    OpenAI. Dall·e 3, 2024. 2, 3

    2024

  10. [18]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 4195–4205,

  11. [19]

    Human evaluation of text-to-image models on a multi-task bench- mark

    Vitali Petsiuk, Alexander E Siemenn, Saisamrit Surbehera, Zad Chin, Keith Tyser, Gregory Hunter, Arvind Raghavan, Yann Hicke, Bryan A Plummer, Ori Kerret, et al. Human evaluation of text-to-image models on a multi-task bench- mark. arXiv preprint arXiv:2211.12112, 2022. 5

    2022 arXiv

  12. [20]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  13. [21]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International confer- ence on machine learning, pages 8821–8831. Pmlr, 2021. 2

    2021

  14. [22]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2

    2022

  15. [23]

    Fleet, and Mohammad Norouzi

    Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image Super-Resolution via Iterative Refinement. arXiv:2104.07636 [cs, eess], 2021. 1

    2021 arXiv

  16. [24]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  17. [25]

    Autoprompt: Eliciting knowl- edge from language models with automatically generated prompts

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV , Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowl- edge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020. 2

    2010 arXiv

  18. [26]

    Generative Model- ing by Estimating Gradients of the Data Distribution

    Yang Song and Stefano Ermon. Generative Model- ing by Estimating Gradients of the Data Distribution. arXiv:1907.05600 [cs, stat], 2020. 1, 2

    1907 arXiv

  19. [27]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 2

    2022

  20. [28]

    Investigating prompt engineering in diffusion models, 2022

    Sam Witteveen and Martin Andrews. Investigating prompt engineering in diffusion models, 2022. 1, 2

    2022

  21. [29]

    Conceptmix: A compositional image generation benchmark with controllable difficulty

    Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty. arXiv preprint arXiv:2408.14339, 2024. 1, 2

    2024 arXiv

  22. [30]

    Textgrad: Automatic” differentiation” via text

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic” differentiation” via text. arXiv preprint arXiv:2406.07496, 2024. 2, 3

    2024 arXiv

  23. [31]

    Respond ‘Yes’ or ‘No’

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 7 ConceptMix++: ...

    2025 arXiv

  24. [32]

    Requirements: The image must satisfy these cri- teria (all should receive ’Yes’ answers): {criteria}

  25. [33]

    History: Previous attempts sorted by perfor- mance (best to worst): {formatted history table} Based on the requirements and previous attempts, please provide a new, improved description for the image generation model. The description should: • Be specific to guide the image ge...

  26. [34]

    We generate an initial image with the original prompt p0

  27. [35]

    We run 4 iterations of our optimization process, generat- ing one image with each improved prompt p1, p2, p3, p4

  28. [36]

    We select the best image from these 5 generations (p0, p1, p2, p3, p4) based on our evaluation metric This approach maintains strict budget parity between the methods, with both generating exactly 5 images. The key difference is that our approach generates images from a se- qu...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.