{"id":"7e355110-2595-4cef-b342-c17399946312","arxiv_id":"2411.15509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DyEval is an interactive, LLM-powered testing framework that dynamically generates text prompts to find up to 2.56x more failures in text-to-image models than static prompt sets.","lead":"The paper introduces DyEval, a system where humans and a language model work together to test text-to-image models by generating prompts on the fly based on what the models get wrong. It is relevant because it finds more model failures than static test sets and reveals new failure types, such as problems with pronouns and cultural details.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.56x claim is not established: the Sec. 5.2 'non-adaptive' baseline is a one-shot LLM prompt generator, not a conventional static benchmark, and #Bugs is measured with an inconsistent definition (ρ=0 vs the 0.75 threshold used in Sec. 5.2).","rationale":"The reader's weakest_assumption correctly focused on the non-adaptive baseline: the paper's headline 2.56x figure is computed against a one-shot LLM prompt generator, not against any of the conventional static benchmarks cited in the Related Work. I agree that this is a load-bearing weakness because the central contribution is framed as outperforming static evaluation. My read is partially aligned with the reader: the same passage also contains an inconsistent bug definition, which independently undermines the quantitative claim. The formal bug definition in Eq. (1) depends on a threshold ρ, but Sec. 5.1 sets ρ=0 and Sec. 5.2 uses 0.75; Eq. (7) counts failed text-image pairs rather than buggy test inputs. Thus the reported 'failures found' is not a well-defined quantity across the paper. The framework itself has genuine value as an interactive testing protocol, and the qualitative findings about culture, implicit relations, and pronouns are useful. The right outcome is therefore not rejection but a conditional requirement: rerun the comparison with a true static benchmark, preregister one bug definition, and report uncertainty. Since the reader already arrived at CONDITIONAL, I recommend no change to the verdict.","tokens_in":27,"tokens_out":4648,"duration_ms":156929,"concrete_test":"Re-run the Sec. 5.2 comparison against an actual static benchmark. Randomly sample 65 prompts from T2I-CompBench or HRS-Bench covering the same four initial topics (material objects, explicit relations, static attributes, style), present them to the same 23 annotators with the same pass/fail instructions, and compute #Bugs using one preregistered definition consistent with Eq. (1) (e.g., a test input is a bug iff its average pass rate over nx=4 images is below 0.75). Report per-model ratios and bootstrap 95% confidence intervals. If DyEval's advantage over the curated static benchmark is not significant for most models, or the max ratio drops below the reported 2.56x, the quantitative superiority claim should be weakened or removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim—that DyEval finds 'max up to 2.56 times' more failures than conventional static evaluation when assessing the same number of text-image pairs—rests on the comparison in Sec. 5.2. The baseline is described as 'non-adaptive testing, where the LLM generates the same quantity of text inputs directly (65...).' That is not a conventional static evaluation method as the paper itself defines them in Sec. 2.1 (pre-collected datasets such as T2I-CompBench, HRS-Bench, COCO-based sets, or template/LLM-generated prompt suites). It is a single-shot variant of DyEval's own generation mechanism without the feedback loop. A one-shot LLM may produce less challenging prompts than a curated benchmark specifically designed to stress compositional failures, so the measured advantage could be an artifact of choosing an easy baseline rather than evidence of superiority over static evaluation. The comparison also lacks confidence intervals or tests; only a maximum ratio across models is reported. Additionally, the bug definition is not stable: Eq. (1) defines a bug via expected pass rate ρ, Sec. 5.1 sets ρ=0, and Sec. 5.2 redefines bugs as test inputs with average pass rate below 0.75. Eq. (7) counts individual failing test pairs, not buggy test inputs. With three different notions in play, #Bugs cannot be interpreted consistently, and the ratio of 'failures found' is not a reliable headline. A fair test would use a fixed static benchmark of comparable cost and one preregistered bug definition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DyEval, an LLM-powered interactive visual assessment framework for text-to-image (T2I) models. It builds a test tree: an LLM generates test topics and prompts, humans label generated images as pass/fail, low-pass nodes trigger a scene-graph-based dynamic failure location and LLM self-reflection, and high-pass nodes branch into finer subtopics. Experiments on SD1-5, SD2-1, SDXL, and SD3 with 23 trained raters report that DyEval finds up to 2.56 times more failures than a non-adaptive baseline at equal numbers of text-image pairs, and the framework surfaces failure patterns involving pronouns, quantifiers, cultural concepts, and implicit relations.","tokens_in":24869,"tokens_out":5483,"duration_ms":47686,"significance":"The framework addresses a real need: static evaluation suites have fixed coverage and contamination risk, and adaptive human-in-the-loop testing is underexplored for T2I models. The study has genuine strengths: 23 trained raters, an inter-rater Kendall's tau of 0.7746, consistency checks via reintroduced pairs, and human pass/fail labels as the outcome, which avoids circularity even though GPT-3.5-turbo both generates prompts and writes reflections. The dynamic failure-location procedure is a useful interpretability contribution. However, the headline quantitative claim is not yet supported because the comparison baseline is not a static benchmark and the bug-counting definition shifts across sections; the contribution is still valuable if these issues are addressed.","major_comments":[{"comment":"The 'conventional static evaluation' baseline in the comparison is not one of the static methods defined in §2.1; it is a one-shot non-adaptive LLM prompt generator. The paper calls this 'non-adaptive testing' but then interprets it as 'conventional static evaluation approaches' in the text and abstract. A fair comparison needs at least one fixed benchmark of comparable cost (for example, a sample from T2I-CompBench or HRS-Bench), or the claim must be restricted to 'one-shot LLM prompt generation'.","section":"§5.2, Fig. 4"},{"comment":"The definition of a bug is inconsistent. Equation (1) defines a bug by an expected pass rate below ρ; §5.1 sets ρ=0; §5.2 redefines bugs as test inputs with average pass rate below 0.75; and Equation (7) counts failing test cases rather than buggy test inputs. The 2.56 ratio in Fig. 4 is computed from APR failure rates, not from #Bugs. Without a single consistent definition, '#Bugs' and the failure-ratio headline cannot be interpreted.","section":"§5.1/§5.2 versus §4.1"},{"comment":"No significance tests or confidence intervals are reported for the comparison. The 'max up to 2.56 times' is a maximum across models and topics, and Fig. 4 shows only means and variance. Given the moderate sample size (13 nodes per tree, 9 initial topics), the authors should report paired or bootstrap tests over initial topics and per-model confidence intervals, or soften the claim accordingly.","section":"§5.2"}],"minor_comments":[{"comment":"The appendix says 'we collect a total of 24,054 text-image pairs for the three models,' but four models are tested; this should be corrected.","section":"Appendix"},{"comment":"The phrase '65 be exact' should read '65 to be exact'.","section":"§5.2"},{"comment":"The figure caption uses 'Spacial relationships'; the correct spelling is 'Spatial relationships'.","section":"Fig. 5"},{"comment":"The failure-location cases contain unresolved figure references ('Fig. ??') for the kimono and crescent-moon examples; these should be replaced with the actual figure numbers.","section":"§5.4"},{"comment":"The table caption refers to 'Inf-Eval,' which appears to be an earlier name for the framework; it should be changed to DyEval for consistency.","section":"Appendix, Table 2"},{"comment":"The sentence 'We set the topic stop extension pass rate ρ to 0' should clarify that this is the node-continuation threshold, not the bug threshold from Eq. (1), to avoid confusion with the 0.75 threshold introduced in §5.2.","section":"§5.1"},{"comment":"The abstract's 'max up to 2.56 times generation failures than conventional methods' should be revised to match the actual baseline and reported uncertainty, since the number is a maximum ratio against a non-adaptive LLM baseline.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The central idea is worth pursuing, and the human study is solid in its design, but the quantitative claim needs either a much stronger baseline or careful rephrasing. In addition, the paper would benefit from releasing the interface and test data for reproducibility; the current submission does not mention code or data release. The positioning relative to prior adaptive testing work (Gao et al., ICCV 2023) could also be sharpened: the current differentiation is mostly domain-specific, whereas a methodological comparison would be more informative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is solid: use an LLM to generate test prompts adaptively, let humans assess images, and record failures in a tree with scene-graph-based minimization and self-reflection. Extending adaptive human-in-the-loop testing from NLP to text-to-image is a natural and worthwhile gap, and the paper fills it reasonably.\n\nWhat the paper does well: the human study is decent. 23 trained raters, Kendall's tau of 0.7746, consistency checks, and 24k annotated text-image pairs. The qualitative failure analysis is the strongest part. The PoS breakdown, the error-type taxonomy (object vs relation vs attribute), and the case studies showing specific triggers like \"kimono\" or combinatorial failures give real insight. The dynamic failure location module is clever and plausibly useful.\n\nThe soft spot is the headline quantitative claim. In Sec 5.2 the non-adaptive baseline is an LLM generating the same number of prompts directly in one shot. That is not a conventional static benchmark as the paper itself defines them in Sec 2.1 (pre-collected datasets like T2I-CompBench, HRS-Bench, COCO). It is a single-shot variant of the same generation mechanism, with no feedback. A curated benchmark designed to stress composition could shrink the measured advantage considerably. Also, the bug definition is not stable: Eq (1) defines a bug via expected pass rate rho; Sec 5.1 sets rho=0; Sec 5.2 redefines bugs as test inputs with average pass rate below 0.75; and Eq (7) counts individual failing test pairs. Those are three different notions, and the 2.56x number is reported as a maximum across models with no confidence intervals or significance tests. That is a load-bearing weakness for the quantitative superiority claim.\n\nMinor issues: missing figure references, a citation placeholder, and the appendix refers to the system as Inf-Eval. No code or data released, which limits reproducibility.\n\nThese problems do not sink the framework; they sink the specific quantitative comparison. The qualitative findings stand as exploratory and genuinely informative for T2I evaluation. I would send this to peer review, but ask for a comparison against an actual static benchmark matched for cost, a single preregistered bug definition, and confidence intervals. The paper deserves serious referee time; the central claim just needs to be rebuilt on a fair baseline.\n\nFor a reader working on T2I evaluation or interactive benchmarking, this is worth reading and discussing. I would probably cite it for the framework and the failure analyses, not for the 2.56x number.","headline":"Useful interactive T2I testing framework, but the 2.56x claim rests on a weak one-shot LLM baseline and an inconsistent bug definition.","tokens_in":25410,"tokens_out":2197,"would_cite":true,"duration_ms":21694,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DyEval, an adaptive LLM-driven testing loop, finds up to 2.56 times more text-to-image failures than static evaluation at equal cost.","keywords":["text-to-image generation","adaptive testing","large language model","human-in-the-loop evaluation","scene graph","failure localization","visual assessment","model debugging"],"falsifier":"Run a controlled comparison in which DyEval and a static benchmark (e.g., a fixed prompt set from COCO or T2I-CompBench) are each given the same number of text-image pairs and the same human annotation budget; if the static benchmark finds as many or more bugs, the paper's central claim fails. A simpler marker: if the non-adaptive LLM baseline already achieves the 2.56x advantage the moment it is allowed any feedback, then the gain comes from adaptation itself rather than from the interactive loop's specific structure.","tokens_in":24244,"feed_emoji":"🧪","tokens_out":6948,"duration_ms":59362,"temperature":0.7,"pith_summary":"Static text-to-image benchmarks fix their prompts ahead of time, so they can miss failures and leak into model training. DyEval replaces the one-shot evaluation with a loop: a language model writes prompts, the image model renders them, a human marks pass or fail, and the results steer the next round of prompts. The paper's central claim is that this feedback-driven process finds more real failures per evaluated text-image pair, up to 2.56 times as many in SD1-5, and that it surfaces rare, interpretable failure modes such as pronoun-induced degradation and cultural-context errors. The reader should care because, if true, adaptive interactive protocols could complement or outperform static benchmarks for debugging the next generation of generative models.","feed_headline":"Adaptive LLM testing finds 2.56x more image-model failures","feed_subtitle":"A feedback loop of prompts, human verdicts, and scene-graph splitting finds new model bugs at equal cost.","key_machinery":"The central mechanism is the feedback loop between a test tree and a divide-and-conquer failure locator. A test node stores a topic, its generated prompts, and pass/fail rates; high-pass nodes cause the LLM to propose finer subtopics, while low-pass nodes trigger the contextual reflection module. That module first converts a failing prompt into a scene graph—a structured representation of objects, attributes, and relations—then recursively splits it into minimal subsets, converts subsets back to text prompts, retests them on the model, and merges surviving parts to identify the smallest failure trigger. The scene-graph split and merge procedure is the load-bearing identity: it turns an opaque failing sentence into a concise, testable hypothesis about where the model's capability boundary lies.","core_discovery":"On its own terms, the paper claims that evaluating a text-to-image model with the DyEval protocol discovers substantially more bugs than evaluating the same number of text-image pairs statically, with a maximum factor of 2.56 on SD1-5, while also giving a more differentiated and more stable picture of model quality. The framework organizes the run as a test tree: each node collects a topic, generated prompts, images, and human pass/fail judgments; high-pass nodes spawn deeper child topics, and low-pass nodes trigger a contextual reflection module. That module splits a failing prompt into a scene graph, recursively tests smaller sub-prompts to isolate the minimal failing component, and then has the LLM summarize the failure pattern. Beyond the headline multiplier, the paper reports consistent cross-model weaknesses—culture and implicit relations are hardest, cardinal numbers and possessive pronouns are persistently error-prone—and argues that the dynamic, on-the-fly generation of prompts sidesteps data leakage.","pith_inferences":["The 2.56x figure compares against a one-shot LLM baseline rather than established static benchmarks; a matched comparison against curated prompt sets at equal human cost could shrink the gap, so the multiplier is best read as a proof of concept, not a settled superiority.","The failure locator assumes failures are compositional—that splitting a scene graph preserves whatever made the original prompt fail; holistic failures could be found but not localized by this machinery.","The observation that child topics sometimes pass at higher rates than their parents suggests the tree maps capability islands as well as boundaries, and exploiting that would require an evaluator to actively choose such topics, making human steering a key variable in the outcome.","A natural extension is to substitute a vision-language model for the human annotator in the loop, which would let the same adaptive protocol scale to much larger prompt budgets and to video generation, at the cost of trusting the VLM's judgments."],"forward_implications":["If the claim holds, static benchmarks undercount real text-to-image failures, and a fixed human budget spread over an adaptive loop yields more debugging signal than the same budget spent on a one-shot prompt set.","Because the protocol is decoupled from specific metrics, the same interactive loop can be pointed at alignment, bias, fairness, or any user-defined criterion for either white-box or black-box models.","The failure triggers produced by dynamic failure location give developers concrete targets for retraining or post-processing, including the two-stage redraw-flawed-objects strategy the paper suggests.","On-the-fly prompt generation makes targeted optimization against a fixed public test set harder, which should slow the benchmark overfitting that static evaluation suffers from.","Cross-model comparisons become more informative: DyEval reports larger pass-rate differences between models than static testing, so adaptive runs may better separate strong from weak generators."],"supporting_citations":[{"why":"Supplies the adaptive-testing-and-debugging paradigm with human feedback that DyEval extends from NLP to text-to-image.","marker":"[28]"},{"why":"Provides CheckList behavioral testing, the baseline human-in-the-loop testing approach this work generalizes.","marker":"[29]"},{"why":"Shows adaptive testing works for computer vision models, the direct precedent for test-then-refine loops over visual outputs.","marker":"[11]"},{"why":"Introduces scene graphs, the structured representation used by dynamic failure location to split and merge test inputs.","marker":"[17]"},{"why":"Supplies CLIPScore, the automatic pre-filter that marks likely failures before human review in the annotation pipeline.","marker":"[13]"},{"why":"Demonstrates adaptive and adversarial testing of text-to-image models, the closest prior dynamic approach this work contrasts with.","marker":"[6]"},{"why":"Defines the latent-diffusion architecture behind SD1-5 and SD2-1, the models under test in the headline quantitative comparison.","marker":"[30]"}],"fun_headline_variants":["LLM-driven visual tests find 2.56x more model bugs","Dynamic test tree uncovers 2.56x more text-to-image failures","Interactive AI probing exposes rare image generation flaws","Adaptive LLM evaluation finds 2.56x more failure cases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative superiority claim rests on the assumption that a one-shot LLM prompt generator fairly represents conventional static evaluation methods; if a comparable-cost curated static benchmark were used instead, the measured advantage could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["LLM-driven visual tests find 2.56x more model bugs","Dynamic test tree uncovers 2.56x more text-to-image failures","Interactive AI probing exposes rare image generation flaws","Adaptive LLM evaluation finds 2.56x more failure cases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00048,"raw_usage":{"total_tokens":2395,"prompt_tokens":987,"completion_tokens":1408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1334}},"tokens_in":603,"tokens_out":1408,"duration_ms":9380,"temperature":1.0,"reasoning_tokens":1334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:12:29.178786+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison in which DyEval and a static benchmark (e.g., a fixed prompt set from COCO or T2I-CompBench) are each given the same number of text-image pairs and the same human annotation budget; if the static benchmark finds as many or more bugs, the paper's central claim fails. A simpler marker: if the non-adaptive LLM baseline already achieves the 2.56x advantage the moment it is allowed any feedback, then the gain comes from adaptation itself rather than from the interactive loop's specific structure.","supporting_citations":[{"cited_title":"In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers)","cited_arxiv_id":null,"evidence_quote":"Supplies the adaptive-testing-and-debugging paradigm with human feedback that DyEval extends from NLP to text-to-image."},{"cited_title":"In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J","cited_arxiv_id":null,"evidence_quote":"Provides CheckList behavioral testing, the baseline human-in-the-loop testing approach this work generalizes."},{"cited_title":"In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","cited_arxiv_id":null,"evidence_quote":"Shows adaptive testing works for computer vision models, the direct precedent for test-then-refine loops over visual outputs."},{"cited_title":"In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Supplies CLIPScore, the automatic pre-filter that marks likely failures before human review in the annotation pipeline."},{"cited_title":"In: Thirty- seventh Conference on Neural Information Processing Systems (2023), https://openreview.net/forum?id=tesBViWnbx","cited_arxiv_id":null,"evidence_quote":"Demonstrates adaptive and adversarial testing of text-to-image models, the closest prior dynamic approach this work contrasts with."}],"review_version":1}