{"id":"012bb60f-b22f-435b-b450-8b47759fcff1","arxiv_id":"2412.05722","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A scene-graph plus LLM question-answering pipeline scores text-to-image faithfulness and categorizes hallucination types, but the measured correlation gain over prior metrics is small and not statistically tested.","lead":"To evaluate whether AI-generated images really match the text that requested them, the authors built a system that maps each image into a graph of objects and properties, then has a large language model answer questions drawn from the prompt. They tested it on 12,000 human-scored images from three Stable Diffusion models and report it agrees with human ratings slightly better than several existing metrics.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported correlation advantage over T2ICompBench is small, lacks significance tests or confidence intervals, and the best variant was run on only ~10% of the data, so the central claim 'aligns more closely with human scoring' is not yet established.","rationale":"The reader identified scene-graph completeness as the weakest assumption, and I agree that is a real limitation; the authors themselves concede in Section 5 that GroundedSAM misses key objects in landscape images, which would corrupt downstream QA. However, for the central claim as stated (closer alignment with human scoring than other metrics), the scene-graph fragility is mediated by the correlation result: if the graph extraction fails, the score may still be correlated with human judgments, and the paper's evidence for the correlation advantage is what is not securely established. The most load-bearing problem is therefore statistical: Table 1 reports correlations without significance testing, the improvement over T2ICompbench is small or tied, the GPT-4 variant was run on a subset, and the text/table configuration is inconsistent. This does not mean the method is wrong; the human-scored dataset and the graph-QA formulation are valuable contributions, and the ablation against caption-based knowledge (Table 3) is a useful control. The reader's CONDITIONAL verdict already requires significance testing and missing baselines, so my read does not change the verdict; it sharpens the condition that the headline claim should not be accepted until the correlation difference is shown to be statistically reliable on the same data.","tokens_in":10097,"tokens_out":4384,"duration_ms":40468,"concrete_test":"Recompute the headline comparison with clearly specified full-data or subset correlations and report inferential statistics. Concretely: (1) obtain GPT-4 scores on the full 12,000 images, or if cost prohibits this, restrict the comparison of Ours(w/GPT-4) versus T2ICompbench to the identical 10% sample and compute Fisher z / bootstrap 95% confidence intervals for the difference between dependent correlations; (2) add TIFA and DALL-Eval baselines to Table 1; (3) verify which LLM variant the 'Ours(w/GPT-3.5)' sentence in Section 4.2.1 refers to. If the correlation advantage over T2ICompbench is not significant at p < 0.05 or confidence intervals overlap, the claim that the method 'aligns more closely with human scoring' should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim rests on Table 1. Our method's advantage over T2ICompBench is small: Pearson 0.26 vs 0.24, Kendall 0.23 vs 0.20, Spearman 0.27 vs 0.27. No significance tests, confidence intervals, or effect-size error bars are reported anywhere in Section 4.2.1. With roughly 12,000 images, a Pearson difference of 0.02 is not clearly significant (Fisher z ≈ 1.6, p ≈ 0.11), and the Spearman values are tied. Moreover, Section 4.2.2 states the GPT-4 variant was evaluated on only about 10% of the data, yet Table 1 does not state whether its correlations are restricted to that subset or provide intervals; if they are subset correlations, the comparison is not directly against the full-data baselines. Section 4.2.1 refers to 'Ours(w/GPT-3.5)' while Table 1 lists only Gemini and GPT-4, leaving the exact configuration of the headline result ambiguous. The scene-graph completeness issue acknowledged in Section 5 is a genuine validity limitation, but it affects the metric's general faithfulness interpretation; the missing statistical support directly undercuts the specific comparative claim that the method aligns more closely with human scoring.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic evaluation metric for text-to-image (T2I) faithfulness, named scene-graph-based question-answering (SGQA). The pipeline first generates images from composite prompts, then builds an image-specific scene graph using GroundedSAM for object detection and BLIP-2 for attribute and relation extraction. Template questions are derived from the dependency tree of the prompt, and an LLM answers these questions by retrieving triples from the scene graph. A rule-based module converts the QA results into a hallucination severity score and categorizes errors into attribute, relation, object omission, and extraneous-object types. The authors also construct a dataset of 12,000 human-scored images from 1,000 prompts using three Stable Diffusion models. The central claim is that the proposed metric correlates more closely with human scoring than existing metrics such as CLIP-Score, BLIP-Score, LLM-Score, and T2ICompBench.","tokens_in":10344,"tokens_out":4276,"duration_ms":35853,"significance":"If the central claim holds, the method would provide a more interpretable and granular automatic evaluation for T2I models, with the additional contribution of a publicly released human-scored dataset. The paper explicitly ships the threat of a load-bearing assumption: the scene-graph completeness issue is acknowledged (Section 5), and the human gold standard is central to all correlation analyses. The paper also offers a useful categorization of hallucination types and a machine-checkable recipe for evaluation. However, the quantitative evidence for the headline claim is currently weak: the correlation advantages over T2ICompBench are small (Pearson 0.26 vs. 0.24; Spearman tied), no significance tests or confidence intervals are reported, and the best variant (GPT-4) was evaluated on only a 10% subset. These issues undermine the comparative claim as stated.","major_comments":[{"comment":"The central claim that the proposed metric 'aligns more closely with human scoring patterns' is not established by the reported numbers. The best full-data variant shows Pearson 0.26 vs. T2ICompBench 0.24, Kendall 0.23 vs. 0.20, and Spearman tied at 0.27. No confidence intervals or significance tests are provided, and with N≈12,000 the Pearson difference is not significant (approximate Fisher z-test p≈0.11). Moreover, Section 4.2.2 states that GPT-4 was run on only ~10% of the data, but Table 1 does not indicate whether its correlations are subset-restricted; if they are, the comparison against full-data baselines is not apples-to-apples. The text in Section 4.2.1 also refers to 'our method(w/GPT-3.5)', a configuration absent from Table 1, leaving the headline result ambiguous. Please report the exact configuration, run the best variant on the full data or state subset-restricted results explicitly, and provide confidence intervals or significance tests for all correlation differences.","section":"§4.2.1, Table 1"},{"comment":"The metric's validity rests on the completeness and correctness of the extracted scene graph. The authors acknowledge that GroundedSAM 'struggles to effectively detect key objects in synthesized landscape images, possibly due to these being treated as background'; in such cases every downstream QA answer and the final score are wrong regardless of actual image faithfulness. This is a direct violation of the core operating hypothesis stated in Section 3.5, and it limits the generality of the proposed metric. The manuscript should quantify the frequency of such detection failures (e.g., per prompt category) and either restrict the method's claimed scope or provide a recovery mechanism (e.g., additional VQA-based object verification).","section":"§3.2 and §5"},{"comment":"The human gold-standard scores are used as the target for all correlations, but the manuscript does not report inter-rater agreement (e.g., Cohen's kappa or Krippendorff's alpha), the number of annotators per image, or the annotator instructions beyond the 7-point rubric. Without reliability evidence, it is unclear how much of the correlation differences are attributable to the metric versus noise in the human labels. This is particularly important because the rubric requires subjective judgments such as 'more than two missing objects' and whether extraneous objects are 'unreasonable' (Section 1). Please report agreement statistics and, if feasible, the per-image majority-vote construction.","section":"§4.1"},{"comment":"The claim that 'our method achieves higher F1 scores across different hallucination types' is not supported by the table. For relation errors, T2ICompBench achieves an F1 of 0.62, equal to Ours(w/GPT-3.5-turbo) and higher than Ours(w/Gemini) at 0.61. Also, T2ICompBench is simply not evaluated on omission or extraneous-object types (shown as '–'), so the comparison is incomplete. The text should either qualify the claim to the types where the method is actually better or provide a fairer head-to-head evaluation with the missing cells filled in.","section":"§4.2.2, Table 2"}],"minor_comments":[{"comment":"The section heading 'Rasing Templated Questions' contains a typo; it should be 'Raising Templated Questions'.","section":"§3.3"},{"comment":"In the sentence 'the graph question-answering agent should be able to answer all the questions by retiring the relevant triples', 'retiring' should be 'retrieving'.","section":"§3.5"},{"comment":"The heading 'Future works and Limiations' contains a typo; it should be 'Future Works and Limitations'.","section":"§5"},{"comment":"The text says 'Tabel 1' and later 'From Tabel 1'; these should be 'Table 1'.","section":"§4.2.1"},{"comment":"In the sentence 'From out experiments, except for explicitly unmatched relationships...', 'out' should be 'our'.","section":"§1"},{"comment":"The dataset is described as 'will be made publicly available in the future' but no repository or release plan is given; please provide a concrete availability statement (e.g., URL or expected hosting platform) to enable reproducibility.","section":"Abstract and §4.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's real contributions are the pipeline and the dataset. Scene-graph extraction plus LLM-based graph QA is a legitimate new integration for T2I hallucination evaluation, and the 12,000 human-scored images across three Stable Diffusion models is a reusable resource. The interpretability angle, categorizing hallucinations into attribute/relation/object/extraneous, is useful. The ablation comparing graph triples against caption-based retrieval is a sensible control and shows the graph representation helps.\n\nThe soft spot is exactly where the stress test lands: the central claim that the method 'aligns more closely with human scoring' is not established by Table 1. Pearson over T2ICompBench is 0.26 vs 0.24, Spearman ties at 0.27, and there are no confidence intervals or significance tests anywhere. With ~12,000 images, a 0.02 difference in Pearson is not clearly meaningful (Fisher z gives p~0.11). The GPT-4 variant, which shows the best correlations, was run on only about 10% of the data, and the table doesn't clarify whether those correlations are subset-only or projected to full data. The text also refers to 'Ours(w/GPT-3.5)' while the table lists Gemini and GPT-4, so the exact configuration is ambiguous. Without error bars, the comparative claim is simply not supported.\n\nAdditional smaller gaps: inter-rater agreement for the human gold standard is not reported, which matters because human scores are the reference. The rule-based scoring module is described only in passing; thresholds and aggregation are free parameters. TIFA and DALL-Eval, directly relevant baselines, are missing. And the authors concede in Section 5 that GroundedSAM misses key objects in landscape images, undermining the scene-graph completeness assumption in a common setting.\n\nNone of this is fatal to the underlying idea. The method is plausible, the dataset is real, and the categorization output is genuinely useful. The paper needs to do the statistical homework before the central claim can be accepted.\n\nWho should read it: anyone working on automatic T2I evaluation or building on T2ICompBench. It deserves a serious referee, but the referee should demand confidence intervals, significance tests, missing baselines, and a full-data GPT-4 run or a clear statement of subset limitations.\n\nRecommendation: send to review, but expect major revision.","headline":"A plausible interpretable metric with a valuable human-scored dataset, but the headline claim of closer human alignment rests on statistically untested correlation differences of ~0.02.","tokens_in":10934,"tokens_out":2048,"would_cite":true,"duration_ms":19084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prompt-image consistency in text-to-image models can be scored automatically by asking an LLM to answer prompt-derived questions against a scene graph extracted from the generated image, and that this score tracks…","keywords":["text-to-image evaluation","hallucination detection","scene graph","question answering","diffusion models","human-aligned scoring","attribute binding","T2I consistency"],"falsifier":"Take images for which the open-vocabulary detector is known to miss prompt-relevant objects (the authors report this for landscape background elements), have humans score them, and check whether the metric's scores diverge from human scores on exactly those images while agreeing elsewhere. A sharper test is to feed a hand-verified complete scene graph into the question-answering stage instead of the extracted one; if the score still fails to track human ratings, the bottleneck is not the graph but the question-answering or scoring design.","tokens_in":9879,"feed_emoji":"🎯","tokens_out":9191,"duration_ms":71702,"temperature":0.7,"pith_summary":"The paper claims that the consistency between a text prompt and a generated image can be measured automatically as a question-answering task: build a scene graph of the image, turn the prompt into templated questions, and let an LLM answer them by retrieving triples from the graph. Wrong answers indicate what the authors call 'hallucinations' in the image, and the pattern of errors reveals whether the failure is an attribute, a relation, a missing object, or an extraneous object. On 12,000 images generated from 1,000 composite prompts by three Stable Diffusion models, the proposed score correlates with human ratings more closely than CLIP-score, BLIP-score, LLM-score, and T2ICompBench. The authors will release the images and human scores, giving the field a reusable benchmark for evaluating evaluators.","feed_headline":"Scene-graph quiz matches human ratings of text-to-image AI","feed_subtitle":"A graph-based QA quiz catches what text-to-image models get wrong, scoring closer to human raters than existing metrics.","key_machinery":"The load-bearing mechanism is the image-specific scene graph combined with a retrieval-based question-answering loop. Each detected object becomes a node, attributes are bound to object nodes, and edges encode spatial relations (from bounding boxes) and non-spatial relations (from BLIP-2 question answering). On the prompt side, a dependency parser splits the text into triples of subject, relation, and object, and an LLM generates template questions and ground-truth answers from those triples. A graph-retrieval module extracts entities from each question, pulls matching triples from the scene graph as memory, and the LLM decides whether that memory answers the question; a rule-based scorer then converts wrong answers into a 1-to-7 human-style rating.","core_discovery":"The central claim is that a graph question-answering agent can serve as a faithful proxy for human judgment of text-to-image consistency. The paper states the operating hypothesis explicitly: if a generated image faithfully reflects the prompt, the agent should be able to answer all prompt-derived questions by retrieving the relevant triples from the image's scene graph, so an erroneous answer marks a hallucinated part of the image. The scene graph is built with an open-vocabulary detector (GroundedSAM) for objects and a VQA model (BLIP-2) for attributes and non-spatial relations, with spatial relations computed from bounding-box coordinates; questions are generated from the prompt's dependency tree. The main quantitative evidence is that the method's Pearson, Kendall, and Spearman correlations with human scores on the 12,000-image dataset exceed those of four baseline metrics, and that its F1 scores for identifying attribute, omission, relation, and extraneous-object hallucinations exceed T2ICompBench's.","pith_inferences":["As open-vocabulary detection improves to handle background and overlapping objects, the same graph-QA design should extend to the landscape images where the current version demonstrably fails.","The framework could be inverted into a prompt-repair tool: the specific triples the agent fails to verify identify which phrase of the prompt the image did not realize, and could be fed back to the generator as a corrective signal.","A controlled perturbation test (swapping attributes or relations in the prompt and checking that the score drops accordingly) would strengthen the causal reading of the method, since the current evidence is correlational.","The hallucination-type frequencies, if published per model, could serve as a diagnostic of text-encoder capability independent of image quality, since the authors observe that attribute-binding errors are common across models."],"forward_implications":["The metric can rank and compare text-to-image models automatically, producing both an overall score and a breakdown of hallucination types for each model.","Model developers can use the per-type error counts (attribute, relation, omission, extraneous) to localize weaknesses, such as attribute-binding failures in the text encoder.","The released 12,000 human-scored image-prompt pairs become a benchmark for testing any future automatic evaluation metric against human judgment.","Because the pipeline uses detection, VQA, and LLM components with no training, it can be rerun on new prompts and models as they appear."],"supporting_citations":[{"why":"Supplies the 1,000 composite prompts used to generate the dataset and the T2ICompBench baseline that the paper compares against.","marker":"[4]"},{"why":"TIFA, the VQA-based faithfulness evaluation approach that this work extends by adding a scene graph.","marker":"[5]"},{"why":"CLIPScore, a baseline embedding-similarity metric in the human-correlation comparison.","marker":"[24]"},{"why":"LLM-score, a baseline caption-comparison metric in the human-correlation comparison.","marker":"[25]"},{"why":"Grounded-SAM, the open-vocabulary detector whose output objects become scene-graph nodes.","marker":"[27]"},{"why":"BLIP-2, the VQA model that produces attribute and non-spatial relation information for the scene graph.","marker":"[28]"},{"why":"The graph-retrieval question-answering module that retrieves triples from the scene graph and supplies them to the LLM as memory for answering.","marker":"[30]"}],"fun_headline_variants":["Scene-graph quiz matches human ratings of text-to-image","QA agent catches T2I hallucinations better than existing metrics","Graph-based interrogation detects image-text mismatches","New scene-graph metric measures T2I hallucination like humans","Scene-graph Q&A flags what text-to-image models get wrong"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire metric rests on the assumption that the automatically extracted scene graph is a complete and correct representation of everything in the image that the prompt cares about; if the detector misses a prompt-relevant object or the VQA model misreads an attribute, the score is wrong regardless of how faithful the image actually is.","fun_headline_variants_meta":{"raw":{"variants":["Scene-graph quiz matches human ratings of text-to-image","QA agent catches T2I hallucinations better than existing metrics","Graph-based interrogation detects image-text mismatches","New scene-graph metric measures T2I hallucination like humans","Scene-graph Q&A flags what text-to-image models get wrong"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1651,"prompt_tokens":1024,"completion_tokens":627,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":640,"tokens_out":627,"duration_ms":6157,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:25:14.008395+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take images for which the open-vocabulary detector is known to miss prompt-relevant objects (the authors report this for landscape background elements), have humans score them, and check whether the metric's scores diverge from human scores on exactly those images while agreeing elsewhere. A sharper test is to feed a hand-verified complete scene graph into the question-answering stage instead of the extracted one; if the score still fails to track human ratings, the bottleneck is not the graph but the question-answering or scoring design.","supporting_citations":[{"cited_title":"LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation","cited_arxiv_id":"2305.11116","evidence_quote":"LLM-score, a baseline caption-comparison metric in the human-correlation comparison."},{"cited_title":"Grounded-Segment-Anything, April 2023","cited_arxiv_id":null,"evidence_quote":"Grounded-SAM, the open-vocabulary detector whose output objects become scene-graph nodes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLIP-2, the VQA model that produces attribute and non-spatial relation information for the scene graph."},{"cited_title":"LangChain, October 2022","cited_arxiv_id":null,"evidence_quote":"The graph-retrieval question-answering module that retrieves triples from the scene graph and supplies them to the LLM as memory for answering."}],"review_version":1}