Pith. sign in

REVIEW 3 major objections 7 minor 50 references

Interactive Visual Assessment for Text-to-Image Generation Models

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DyEval, an adaptive LLM-driven testing loop, finds up to 2.56 times more text-to-image failures than static evaluation at equal cost.

desk verdict Useful interactive T2I testing framework, but the 2.56x claim rests on a weak one-shot LLM baseline and an inconsistent bug definition. read the letter →

arxiv 2411.15509 v1 pith:42S73SNG submitted 2024-11-23 cs.CV cs.AI

Xiaoyue Mi , Fan Tang , Juan Cao , Qiang Sheng , Ziyao Huang , Peng Li , Yang Liu , Tong-Yee Lee This is my paper · ORCID
classification cs.CVcs.AI
keywords text-to-imagegenerationadaptivetestinglargelanguagemodelhuman-in-the-loopevaluationscenegraphfailurelocalizationvisualassessmentdebugging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Static text-to-image benchmarks fix their prompts ahead of time, so they can miss failures and leak into model training. DyEval replaces the one-shot evaluation with a loop: a language model writes prompts, the image model renders them, a human marks pass or fail, and the results steer the next round of prompts. The paper's central claim is that this feedback-driven process finds more real failures per evaluated text-image pair, up to 2.56 times as many in SD1-5, and that it surfaces rare, interpretable failure modes such as pronoun-induced degradation and cultural-context errors. The reader should care because, if true, adaptive interactive protocols could complement or outperform static benchmarks for debugging the next generation of generative models.

What carries the argument

The central mechanism is the feedback loop between a test tree and a divide-and-conquer failure locator. A test node stores a topic, its generated prompts, and pass/fail rates; high-pass nodes cause the LLM to propose finer subtopics, while low-pass nodes trigger the contextual reflection module. That module first converts a failing prompt into a scene graph—a structured representation of objects, attributes, and relations—then recursively splits it into minimal subsets, converts subsets back to text prompts, retests them on the model, and merges surviving parts to identify the smallest failure trigger. The scene-graph split and merge procedure is the load-bearing identity: it turns an opaque failing sentence into a concise, testable hypothesis about where the model's capability boundary lies.

What would settle it

Run a controlled comparison in which DyEval and a static benchmark (e.g., a fixed prompt set from COCO or T2I-CompBench) are each given the same number of text-image pairs and the same human annotation budget; if the static benchmark finds as many or more bugs, the paper's central claim fails. A simpler marker: if the non-adaptive LLM baseline already achieves the 2.56x advantage the moment it is allowed any feedback, then the gain comes from adaptation itself rather than from the interactive loop's specific structure.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that evaluating a text-to-image model with the DyEval protocol discovers substantially more bugs than evaluating the same number of text-image pairs statically, with a maximum factor of 2.56 on SD1-5, while also giving a more differentiated and more stable picture of model quality. The framework organizes the run as a test tree: each node collects a topic, generated prompts, images, and human pass/fail judgments; high-pass nodes spawn deeper child topics, and low-pass nodes trigger a contextual reflection module. That module splits a failing prompt into a scene graph, recursively tests smaller sub-prompts to isolate the minimal failing component, and then has the LLM summarize the failure pattern. Beyond the headline multiplier, the paper reports consistent cross-model weaknesses—culture and implicit relations are hardest, cardinal numbers and possessive pronouns are persistently error-prone—and argues that the dynamic, on-the-fly generation of prompts sidesteps data leakage.

Load-bearing premise

The quantitative superiority claim rests on the assumption that a one-shot LLM prompt generator fairly represents conventional static evaluation methods; if a comparable-cost curated static benchmark were used instead, the measured advantage could shrink or disappear.

Editorial extensions

If this is right

  • If the claim holds, static benchmarks undercount real text-to-image failures, and a fixed human budget spread over an adaptive loop yields more debugging signal than the same budget spent on a one-shot prompt set.
  • Because the protocol is decoupled from specific metrics, the same interactive loop can be pointed at alignment, bias, fairness, or any user-defined criterion for either white-box or black-box models.
  • The failure triggers produced by dynamic failure location give developers concrete targets for retraining or post-processing, including the two-stage redraw-flawed-objects strategy the paper suggests.
  • On-the-fly prompt generation makes targeted optimization against a fixed public test set harder, which should slow the benchmark overfitting that static evaluation suffers from.
  • Cross-model comparisons become more informative: DyEval reports larger pass-rate differences between models than static testing, so adaptive runs may better separate strong from weak generators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 2.56x figure compares against a one-shot LLM baseline rather than established static benchmarks; a matched comparison against curated prompt sets at equal human cost could shrink the gap, so the multiplier is best read as a proof of concept, not a settled superiority.
  • The failure locator assumes failures are compositional—that splitting a scene graph preserves whatever made the original prompt fail; holistic failures could be found but not localized by this machinery.
  • The observation that child topics sometimes pass at higher rates than their parents suggests the tree maps capability islands as well as boundaries, and exploiting that would require an evaluator to actively choose such topics, making human steering a key variable in the outcome.
  • A natural extension is to substitute a vision-language model for the human annotator in the loop, which would let the same adaptive protocol scale to much larger prompt budgets and to video generation, at the cost of trusting the VLM's judgments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes DyEval, an LLM-powered interactive visual assessment framework for text-to-image (T2I) models. It builds a test tree: an LLM generates test topics and prompts, humans label generated images as pass/fail, low-pass nodes trigger a scene-graph-based dynamic failure location and LLM self-reflection, and high-pass nodes branch into finer subtopics. Experiments on SD1-5, SD2-1, SDXL, and SD3 with 23 trained raters report that DyEval finds up to 2.56 times more failures than a non-adaptive baseline at equal numbers of text-image pairs, and the framework surfaces failure patterns involving pronouns, quantifiers, cultural concepts, and implicit relations.

Significance. The framework addresses a real need: static evaluation suites have fixed coverage and contamination risk, and adaptive human-in-the-loop testing is underexplored for T2I models. The study has genuine strengths: 23 trained raters, an inter-rater Kendall's tau of 0.7746, consistency checks via reintroduced pairs, and human pass/fail labels as the outcome, which avoids circularity even though GPT-3.5-turbo both generates prompts and writes reflections. The dynamic failure-location procedure is a useful interpretability contribution. However, the headline quantitative claim is not yet supported because the comparison baseline is not a static benchmark and the bug-counting definition shifts across sections; the contribution is still valuable if these issues are addressed.

major comments (3)
  1. [§5.2, Fig. 4] The 'conventional static evaluation' baseline in the comparison is not one of the static methods defined in §2.1; it is a one-shot non-adaptive LLM prompt generator. The paper calls this 'non-adaptive testing' but then interprets it as 'conventional static evaluation approaches' in the text and abstract. A fair comparison needs at least one fixed benchmark of comparable cost (for example, a sample from T2I-CompBench or HRS-Bench), or the claim must be restricted to 'one-shot LLM prompt generation'.
  2. [§5.1/§5.2 versus §4.1] The definition of a bug is inconsistent. Equation (1) defines a bug by an expected pass rate below ρ; §5.1 sets ρ=0; §5.2 redefines bugs as test inputs with average pass rate below 0.75; and Equation (7) counts failing test cases rather than buggy test inputs. The 2.56 ratio in Fig. 4 is computed from APR failure rates, not from #Bugs. Without a single consistent definition, '#Bugs' and the failure-ratio headline cannot be interpreted.
  3. [§5.2] No significance tests or confidence intervals are reported for the comparison. The 'max up to 2.56 times' is a maximum across models and topics, and Fig. 4 shows only means and variance. Given the moderate sample size (13 nodes per tree, 9 initial topics), the authors should report paired or bootstrap tests over initial topics and per-model confidence intervals, or soften the claim accordingly.
minor comments (7)
  1. [Appendix] The appendix says 'we collect a total of 24,054 text-image pairs for the three models,' but four models are tested; this should be corrected.
  2. [§5.2] The phrase '65 be exact' should read '65 to be exact'.
  3. [Fig. 5] The figure caption uses 'Spacial relationships'; the correct spelling is 'Spatial relationships'.
  4. [§5.4] The failure-location cases contain unresolved figure references ('Fig. ??') for the kimono and crescent-moon examples; these should be replaced with the actual figure numbers.
  5. [Appendix, Table 2] The table caption refers to 'Inf-Eval,' which appears to be an earlier name for the framework; it should be changed to DyEval for consistency.
  6. [§5.1] The sentence 'We set the topic stop extension pass rate ρ to 0' should clarify that this is the node-continuation threshold, not the bug threshold from Eq. (1), to avoid confusion with the 0.75 threshold introduced in §5.2.
  7. [Abstract] The abstract's 'max up to 2.56 times generation failures than conventional methods' should be revised to match the actual baseline and reported uncertainty, since the number is a maximum ratio against a non-adaptive LLM baseline.

Circularity Check

0 steps flagged · score 1.0 of 10

No derivation-level circularity: DyEval's quantitative claims are empirical, and the non-adaptive baseline and metric inconsistencies are benchmark-validity concerns rather than definitional reductions.

full rationale

DyEval's derivation chain is not circular. The central 2.56x claim is an empirical comparison (Sec. 5.2) between DyEval's adaptive testing tree and a non-adaptive LLM baseline generating the same number of text inputs; the primary labels are human pass/fail judgments (reported inter-rater Kendall's tau = 0.7746), not the LLM's own output, so the claimed advantage is not defined into existence. The bug-count metric is inconsistently defined (Eq. (1) uses rho, Sec. 5.1 sets rho = 0, and Sec. 5.2 defines bugs as average pass rate below 0.75), and the non-adaptive baseline is a one-shot LLM generator rather than a curated static benchmark such as HRS-Bench or T2I-CompBench, but these are benchmark-validity and metric-consistency concerns, not reductions to the paper's own inputs. The only self-citation, [41] in the static-evaluation list in Sec. 2.1, is not load-bearing. The paper honestly states in Limitations that effectiveness depends on LLM quality, and it contains a missing-reference artifact ('Compared with static testing methods [?], [20]' in Sec. 1) and unresolved figure placeholders in Sec. 5.4, none of which create circularity. No framework step, prompt generation, scene-graph splitting, or LLM reflection presupposes the failure patterns it reports. The claims are self-contained against external human labels and do not reduce by construction to their inputs.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claims rest on tunable hyperparameters (nt, ni, nx, dmax, rho), an unspecified CLIPScore threshold, and an inconsistent bug threshold. The framework assumes LLM text/scene-graph transformations are semantically faithful, assumes failures are compositional enough for divide-and-conquer localization, and assumes human labels are reliable. No new physical or conceptual entities are introduced beyond the software components of the system.

free parameters (7)
  • nt (test topics per iteration) = 3
    Chosen by authors in Sec. 5.1; controls tree breadth and the number of bugs that can be accumulated.
  • ni (test inputs per iteration) = 5
    Chosen in Sec. 5.1; the number of LLM-generated prompts per node determines the granularity of testing.
  • nx (images per input) = 4
    Chosen in Sec. 5.1; the number of generated images per prompt sets the resolution of the per-input pass rate estimate.
  • dmax (max test depth) = 3
    Chosen in Sec. 5.1; limits the size of the testing tree and therefore the total number of bugs found.
  • rho (stop-extension pass rate) = 0
    Set to 0 in Sec. 5.1 to always expand nodes; this diverges from the formal bug definition in Eq. 1 and affects the meaning of APR and #Bugs.
  • CLIPScore threshold = not specified
    Mentioned in Sec. 4.2.1 as an automatic fail filter for test pairs; the threshold value is never reported, making the labeling pipeline under-specified.
  • Bug threshold for #Bugs = 0.75
    Sec. 5.2 defines bugs as test inputs with average pass rate below 0.75, which is different from the rho-based definition in Sec. 4.1; this inconsistency affects the headline bug count.
assumptions (4)
  • domain assumption GPT-3.5-turbo can generate semantically relevant and diverse test prompts and topics from test records.
    Relied on throughout Sec. 4; the paper provides no quantitative validation of prompt quality apart from a 10-expert survey on reflections, and the same LLM is used for both generation and analysis.
  • ad hoc to paper LLM-based text-to-scene-graph and scene-graph-to-text transformations preserve the test semantics needed for failure localization.
    Assumed in Sec. 4.2.2 and the appendix; the conversion is heuristic and may drop or alter context, which would change which failure trigger is identified.
  • ad hoc to paper Failures are compositional: splitting a prompt into scene-graph subsets and testing them identifies minimal failure triggers.
    Core assumption of the dynamic failure location algorithm (Alg. 1); no ablation or proof shows that divide-and-conquer finds the true trigger rather than an artifact of the splitting procedure.
  • domain assumption Human pass/fail labels are ground truth for text-image alignment and visual quality.
    The evaluation relies on 23 raters; inter-rater reliability is checked with Kendall's tau of 0.7746, so the assumption is partially supported, but subjective topics were excluded from the headline comparison due to lower agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interactive Visual Assessment for Text-to-Image Generation Models." pith.science (2026). https://pith.science/paper/42S73SNG

@misc{pith2026241115509,
  author       = {Pith},
  title        = {Pith review of: Interactive Visual Assessment for Text-to-Image Generation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/42S73SNG}},
  note         = {Machine review of arXiv:2411.15509}
}
read the original abstract

Visual generation models have achieved remarkable progress in computer graphics applications but still face significant challenges in real-world deployment. Current assessment approaches for visual generation tasks typically follow an isolated three-phase framework: test input collection, model output generation, and user assessment. These fashions suffer from fixed coverage, evolving difficulty, and data leakage risks, limiting their effectiveness in comprehensively evaluating increasingly complex generation models. To address these limitations, we propose DyEval, an LLM-powered dynamic interactive visual assessment framework that facilitates collaborative evaluation between humans and generative models for text-to-image systems. DyEval features an intuitive visual interface that enables users to interactively explore and analyze model behaviors, while adaptively generating hierarchical, fine-grained, and diverse textual inputs to continuously probe the capability boundaries of the models based on their feedback. Additionally, to provide interpretable analysis for users to further improve tested models, we develop a contextual reflection module that mines failure triggers of test inputs and reflects model potential failure patterns supporting in-depth analysis using the logical reasoning ability of LLM. Qualitative and quantitative experiments demonstrate that DyEval can effectively help users identify max up to 2.56 times generation failures than conventional methods, and uncover complex and rare failure patterns, such as issues with pronoun generation and specific cultural context generation. Our framework provides valuable insights for improving generative models and has broad implications for advancing the reliability and capabilities of visual generation systems across various domains.

Figures

Figures reproduced from arXiv: 2411.15509 by the authors.

Figure 1
Figure 1. Diagram of the DyEval, a dynamic interactive testing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of DyEval. The DyEval process begins with setting initial topics and model parameters (Step A). In the test input [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Comparison experiments between DyEval and non-adaptive [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (11 more)
Figure 6
Figure 6. Figure 6: Average test pass rate AP R of SD1-5, SD2-1, SDXL, and SD3 models in nine dimensions. Dimensions of the same base color indicate belonging to the same category. The overall trend of image generation capabilities is SD3>= SDXL> SD2-1> SD1-5; culture and implicit relatio…
Figure 5
Figure 5. Figure 5: A test process case of DyEval. The initial test topic is [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Top 15 error-prone parts-of-speech from different models, ordered by fail rate. The red dashed line represents the average fail rate [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Frequency of object, attribute, and relation errors in different models. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: An error analysis example mentioned in the paper. Given the [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Violin plot on the length of fail test inputs and pass test [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Example of split scene graph and merge scene graphs. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Demo system pages. We provide words with high failure rates in [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Interesting test inputs generated by DyEval. We find that all four models were poorly generated, with red indicating the missing [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Interesting test inputs generated by DyEval. The red color indicates the missing parts of the model under test, we find that in the [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Number distribution of different part-of-speech tags for different models. The three models are basically consistent. [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 37 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Bakr, E.M., Sun, P., Shen, X., Khan, F.F., Li, L.E., Elhoseiny, M.: Hrs- bench: Holistic, reliable and scalable benchmark for text-to-image models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20041–20053 (2023)

    work page 2023

  2. [2]

    Computer Science

    Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y ., et al.: Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf (2023)

    work page 2023

  3. [3]

    Qualitative Failures of Image Generation Models and Their Application in Detecting Deepfakes

    Borji, A.: Qualitative failures of image generation models and their application in detecting deepfakes. arXiv preprint arXiv:2304.06470 (2023)

    work page Pith review arXiv 2023

  4. [4]

    arXiv preprint arXiv:2305.10843 (2023)

    Chen, Y .: X-iqe: explainable image quality evaluation for text-to- image generation with visual large language models. arXiv preprint arXiv:2305.10843 (2023)

    arXiv 2023

  5. [5]

    In: The Twelfth International Conference on Learning Representations (2024)

    Cho, J., Hu, Y ., Baldridge, J.M., Garg, R., Anderson, P., Krishna, R., Bansal, M., Pont-Tuset, J., Wang, S.: Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. In: The Twelfth International Conference on Learning Representations (2024)

    work page 2024

  6. [6]

    In: Thirty- seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=tesBViWnbx

    Du, C., Li, Y ., Qiu, Z., Xu, C.: Stable diffusion is unstable. In: Thirty- seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=tesBViWnbx

    work page 2023

  7. [7]

    In: Forty-first International Conference on Machine Learning (2024)

    Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y ., Lorenz, D., Sauer, A., Boesel, F., et al.: Scaling rectified flow transformers for high-resolution image synthesis. In: Forty-first International Conference on Machine Learning (2024)

    work page 2024

  8. [8]

    In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=FPCMqjI0jXN

    Eyuboglu, S., Varma, M., Saab, K.K., Delbrouck, J.B., Lee-Messer, C., Dunnmon, J., Zou, J., Re, C.: Domino: Discovering systematic errors with cross-modal embeddings. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=FPCMqjI0jXN

    work page 2022

Show all 50 references
  1. [9]

    IEEE Transactions on Visualization and Computer Graph- ics 30(1), 295–305 (2024)

    Feng, Y ., Wang, X., Wong, K.K., Wang, S., Lu, Y ., Zhu, M., Wang, B., Chen, W.: Promptmagician: Interactive prompt engineering for text-to- image creation. IEEE Transactions on Visualization and Computer Graph- ics 30(1), 295–305 (2024). https://doi.org/10.1109/TVCG.2023.3327168

    2024

  2. [10]

    arXiv preprint arXiv:2209.07858 (2022)

    Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y ., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al.: Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 (2022)

    2022 arXiv

  3. [11]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)

    Gao, I., Ilharco, G., Lundberg, S., Ribeiro, M.T.: Adaptive testing of computer vision models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4003–4014 (October 2023)

    2023

  4. [12]

    IEEE Transactions on Visualization and Computer Graphics pp

    Gao, X., Yang, Y ., Xie, Z., Du, S., Sun, Z., Wu, Y .: Guess: Gradually enriching synthesis for text-driven human motion generation. IEEE Transactions on Visualization and Computer Graphics pp. 1–13 (2024). https://doi.org/10.1109/TVCG.2024.3352002

    2024

  5. [13]

    In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

    Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., Choi, Y .: Clipscore: A reference-free evaluation metric for image captioning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 7514–7528 (2021)

    2021

  6. [14]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Hu, Y ., Liu, B., Kasai, J., Wang, Y ., Ostendorf, M., Krishna, R., Smith, N.A.: Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20406–20417 (2023)

    2023

  7. [15]

    Advances in Neural Information Processing Systems 36 (2024)

    Huang, K., Sun, K., Xie, E., Li, Z., Liu, X.: T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems 36 (2024)

    2024

  8. [16]

    arXiv preprint arXiv:2406.04485 (2024)

    Jiang, D., Ku, M., Li, T., Ni, Y ., Sun, S., Fan, R., Chen, W.: Genai arena: An open evaluation platform for generative models. arXiv preprint arXiv:2406.04485 (2024)

    2024 arXiv

  9. [17]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D., Bernstein, M., Fei-Fei, L.: Image retrieval using scene graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3668–3678 (2015)

    2015

  10. [18]

    In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=G5RwHpBUv0

    Kirstain, Y ., Polyak, A., Singer, U., Matiana, S., Penna, J., Levy, O.: Pick- a-pic: An open dataset of user preferences for text-to-image generation. In: Thirty-seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=G5RwHpBUv0

    2023

  11. [19]

    In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2023), https://openreview.net/forum?id=qY9LR74O3Z

    Lee, T., Yasunaga, M., Meng, C., Mai, Y ., Park, J.S., Gupta, A., Zhang, Y ., Narayanan, D., Teufel, H.B., Bellagente, M., Kang, M., Park, T., Leskovec, J., Zhu, J.Y ., Fei-Fei, L., Wu, J., Ermon, S., Liang, P.: Holistic evaluation of text-to-image models. In: Thirty-seventh C...

    2023

  12. [20]

    IEEE Transactions on Circuits and Systems for Video Technology pp

    Li, C., Zhang, Z., Wu, H., Sun, W., Min, X., Liu, X., Zhai, G., Lin, W.: Agiqa-3k: An open database for ai-generated image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology pp. 1–1 (2023). https://doi.org/10.1109/TCSVT.2023.3319020

    2023

  13. [21]

    In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL

    Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. XX, NO....

    2014

  14. [22]

    In: The Twelfth International Conference on Learning Representations (2023)

    Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y ., Wang, L.: Mitigating hallucination in large multi-modal models via robust instruction tuning. In: The Twelfth International Conference on Learning Representations (2023)

    2023

  15. [23]

    In: European Conference on Computer Vision

    Liu, N., Li, S., Du, Y ., Torralba, A., Tenenbaum, J.B.: Compositional visual generation with composable diffusion models. In: European Conference on Computer Vision. pp. 423–439. Springer (2022)

    2022

  16. [24]

    Advances in Neural Information Processing Systems 36 (2023)

    Lu, Y ., Yang, X., Li, X., Wang, X.E., Wang, W.Y .: Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems 36 (2023)

    2023

  17. [25]

    arXiv preprint arXiv:2203.08242 (2022)

    Magar, I., Schwartz, R.: Data contamination: From memorization to exploitation. arXiv preprint arXiv:2203.08242 (2022)

    2022 arXiv

  18. [26]

    In: The Twelfth International Conference on Learning Representations (2024), https:// openreview.net/forum?id=KS8mIvetg2

    Oren, Y ., Meister, N., Chatterji, N.S., Ladhak, F., Hashimoto, T.: Proving test set contamination for black-box language models. In: The Twelfth International Conference on Learning Representations (2024), https:// openreview.net/forum?id=KS8mIvetg2

    2024

  19. [27]

    arXiv preprint arXiv:2307.01952 (2023)

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023)

    2023 arXiv

  20. [28]

    In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers)

    Ribeiro, M.T., Lundberg, S.: Adaptive testing and debugging of nlp models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers). pp. 3253–3267 (2022)

    2022

  21. [29]

    In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

    Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: Behavioral testing of NLP models with CheckList. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4902–

  22. [30]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High- resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)

    2022

  23. [31]

    Advances in Neural Information Processing Systems 35, 36479–36494 (2022)

    Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35,...

    2022

  24. [32]

    Advances in Neural Information Processing Systems 36 (2023)

    Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36 (2023)

    2023

  25. [33]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) (2023)

    Wang, Z.J., Montoya, E., Munechika, D., Yang, H., Hoover, B., Chau, D.H.: Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) (2023)

    2023

  26. [34]

    In: NeurIPS ML Safety Workshop (2022)

    Wiles, O., Albuquerque, I., Gowal, S.: Discovering bugs in vision models using off-the-shelf image generation and captioning. In: NeurIPS ML Safety Workshop (2022)

    2022

  27. [35]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Wu, Q., Liu, Y ., Zhao, H., Bui, T., Lin, Z., Zhang, Y ., Chang, S.: Har- nessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7766–7776 (2023)

    2023

  28. [36]

    arXiv preprint arXiv:2306.09341 (2023)

    Wu, X., Hao, Y ., Sun, K., Chen, Y ., Zhu, F., Zhao, R., Li, H.: Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 (2023)

    2023 arXiv

  29. [37]

    IEEE Transactions on Visualization and Computer Graphics pp

    Xu, C., Xu, Y ., Zhang, H., Xu, X., He, S.: Dreamanime: Learning style-identity textual disentanglement for anime and beyond. IEEE Transactions on Visualization and Computer Graphics pp. 1–12 (2024). https://doi.org/10.1109/TVCG.2024.3397712

    2024

  30. [38]

    Advances in Neural Information Processing Systems 36 (2023)

    Xu, J., Liu, X., Wu, Y ., Tong, Y ., Li, Q., Ding, M., Tang, J., Dong, Y .: Imagereward: Learning and evaluating human preferences for text-to- image generation. Advances in Neural Information Processing Systems 36 (2023)

    2023

  31. [39]

    In: The Eleventh International Conference on Learning Representations (2022)

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y .: React: Synergizing reasoning and acting in language models. In: The Eleventh International Conference on Learning Representations (2022)

    2022

  32. [40]

    IEEE Transactions on Visualization and Computer Graphics 30(9), 6481–6492 (2024)

    Zhang, Z., Chen, J., Fu, H., Zhao, J., Chen, S.Y ., Gao, L.: Text2face: Text-based face generation with geometry and appearance control. IEEE Transactions on Visualization and Computer Graphics 30(9), 6481–6492 (2024). https://doi.org/10.1109/TVCG.2023.3349050

    2024

  33. [41]

    objects": {

    Zhou, Z., Tang, F., Zhang, Y ., Deussen, O., Cao, J., Dong, W., Li, X., Lee, T.Y .: A comprehensive evaluation of arbitrary image style transfer methods. IEEE Transactions on Visualization and Computer Graphics pp. 1–19 (2024). https://doi.org/10.1109/TVCG.2024.3466964 IEEE TR...

    2024

  34. [43]

    Misinterpretation of Actions: The model may misinterpret the actions described in the text prompt, resulting in inaccurate image generation. For example, in the first test record, the model may not understand the concept of a dog wagging its tail while its owner scratches its ...

  35. [44]

    Lack of Emotional Understanding: The model may struggle to understand and represent emotions accurately. In the third test record, where a dog is described as licking its owner’s face to show affection and love, the model may not be able to capture the emotional aspect, result...

  36. [45]

    A dog wagging its tail while its owner scratches its belly

    Limited Understanding of Relationships: The model may have a limited understanding of the dynamics and nuances of dog-human relationships. In the fourth test record, where a dog is described as cuddling up to its owner on a cold winter night, the model may not be able to accur...

  37. [46]

    A dog’s tail

    **Object Focus**: The model tends to perform better when the test input focuses on a specific object related to the dog, such as "A dog’s tail" or "A dog lying on the couch." In these cases, the model successfully generates the corresponding images. This suggests that the mode...

  38. [47]

    A dog lying next to its owner on the couch, both fast asleep

    **Contextual Relationships**: The model struggles when the test input involves contextual relationships between the dog and its owner, such as "A dog lying next to its owner on the couch, both fast asleep" or "A dog cuddling up to its owner on a cold winter night, keeping them...

  39. [48]

    A dog" or

    **Incomplete Descriptions**: In some cases, the test inputs provide incomplete descriptions, such as "A dog" or "A dog on a cold winter night." Surprisingly, the model successfully generates the corresponding images in these cases. It suggests that the model may rely more on t...

  40. [49]

    A dog’s owner

    **Owner Focus**: The model consistently struggles when the test input focuses solely on the dog’s owner, such as "A dog’s owner" or "A dog’s owner on the couch." These inputs consistently result in failures, indicating that the model may have difficulty generating images based...

    2000

  41. [50]

    context": { }

    and summarize failure patterns where the model may underperform. List by points. {test recordsr} Prompting LLMs to transform scene graphs into texts Task: Given a scene graph c, and describe c accurately in text. Do not output any Entity/Relation/Context that is not in c, espe...

  42. [4912]

    https://doi.org/10.18653/v1/2020.acl-main.442, https://aclanthology.org/ 2020.acl-main.442

    Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.442, https://aclanthology.org/ 2020.acl-main.442

    2020 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.