{"id":"c338b2ae-e262-4bc6-88a7-be2126fff5e0","arxiv_id":"2412.05685","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HMGIE detects image-caption inconsistencies by hierarchically questioning an MLLM along a caption-derived semantic graph, and reports accuracy and completeness scores.","lead":"The paper introduces HMGIE, a pipeline that checks whether captions match images by parsing the caption into a semantic graph, asking progressive visual questions, and scoring accuracy plus completeness. It also builds a new LLM-generated benchmark with captions at four detail levels and reports higher detection scores than prior methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline F1 advantage is measured on MVTID, a benchmark whose labels are produced by GPT-4o while every HMGIE component also uses GPT-4o; the reported gap may reflect self-consistency, not general inconsistency detection, and external results omit FPR.","rationale":"The reader's weakest_assumption focuses on the completeness score Hcomp, which is indeed a real flaw: Hcomp is computed from the caption-derived semantic graph, so it cannot detect omissions of visual elements that never appear in the caption, and N_l is undefined. However, Hcomp is not used in the binary consistency decision d that produces the headline TPR/F1 numbers on MVTID. The more load-bearing threat to the central claim is the evaluation circularity: MVTID is constructed with GPT-4o (Section 4.1) and HMGIE also uses GPT-4o for every stage (Section 5.1). This makes the reported 22.82-point F1 advantage over VDC potentially an artifact of self-agreement rather than a demonstration of general visual-textual inconsistency detection. The external benchmarks partially mitigate this because their labels are not GPT-4o-generated, but Table 5 reports only TPR, and the appendix's failure cases show HMGIE relabels many 'consistent' samples as inconsistent, which can inflate TPR while hiding a high FPR. The reader's overall CONDITIONAL verdict already captures this need for independent validation, so my analysis does not change the verdict; it shifts the emphasis from Hcomp to the benchmark protocol as the primary concern.","tokens_in":19468,"tokens_out":6843,"duration_ms":65043,"concrete_test":"Sample 300 image-caption pairs from MVTID (75 per granularity), have three independent human annotators label each as consistent/inconsistent, then compute HMGIE and VDC TPR/FPR/F1 on this human-labeled subset. If HMGIE's F1 advantage over VDC is not sustained (e.g., drops below 10 points) or its FPR exceeds 20%, the self-generated benchmark explains the reported superiority and the central claim must be re-scoped. As a complement, rerun Table 5 on NewsCLIPpings reporting FPR and F1 in addition to TPR.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 describes MVTID construction: ground-truth captions are generated by an ensemble that includes GPT-4o, fused by GPT-4o, adversarially perturbed by GPT-4o, with perturbation detectability judged by an MLLM; Section 5.1 states GPT-4o is the base model for all HMGIE components (semantic graph generation, question generation, VQA, answer evaluation). Thus on MVTID, HMGIE is checking whether GPT-4o's own caption-generation and perturbation process introduced a detectable inconsistency. The consistent class is what GPT-4o and two other models agree on; the inconsistent class is what GPT-4o's direct prompt missed but HMGIE's hierarchical decomposition catches. This can produce large F1 gains without implying general visual-textual inconsistency detection. On the external datasets (Table 5), only TPR is reported; the appendix shows HMGIE relabels many 'consistent' samples as inconsistent, so TPR alone cannot establish accuracy. The completeness metric Hcomp is also caption-derived and has an undefined N_l, but the binary detection superiority is independent of Hcomp; the benchmark circularity is the more load-bearing threat.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HMGIE, a hierarchical and multi-grained framework for evaluating visual-textual inconsistency. HMGIE converts a caption into a semantic graph, progressively constructs a hierarchical inconsistency evaluation graph (HIEG) through question-answer generation and VQA, and computes accuracy and completeness scores (Hacc and Hcomp) together with natural-language explanations. The authors introduce MVTID, a dataset of image-caption pairs with four granularities of inconsistency generated by an LLM/MLLM ensemble, and report TPR/F1/FPR on MVTID, TPR on external datasets (NewsCLIPpings, TIIL, SeeTRUE), human correlation results for the H-Scores, ablation studies, and qualitative analyses.","tokens_in":19738,"tokens_out":3415,"duration_ms":32774,"significance":"If the central empirical claim held, HMGIE would be a practically useful data-cleansing tool that also produces interpretable explanations. The paper has several genuine strengths: the modular design is clear, the ablation in Section 5.3 shows that the semantic graph contributes to detection, Figure 4 demonstrates the framework works with open-source LLMs/MLLMs, and Tables 3-4 report correlation with human judgments. However, the main claim of superior performance is substantially weakened by two issues: (i) the MVTID benchmark is constructed and evaluated with the same GPT-4o model family, so the reported F1 advantage may partly reflect self-consistency rather than general inconsistency detection; and (ii) the external benchmark results report only TPR, with the appendix dismissing high false positives as dataset flaws. These issues are central to the paper's headline result.","major_comments":[{"comment":"The MVTID labels and the HMGIE evaluation pipeline both rely on GPT-4o: ground-truth captions are generated by an ensemble that includes GPT-4o, fused by GPT-4o, adversarially perturbed by GPT-4o, with perturbation detectability judged by an MLLM (Section 4.1); and HMGIE uses GPT-4o for semantic graph generation, question generation, VQA, and answer evaluation (Section 5.1). The consistent class is what three models agree on, and the inconsistent class is what GPT-4o's perturbation process produced. On this benchmark, HMGIE is partly measuring how well its own reasoning decomposition catches inconsistencies that the same model family introduced. The reported 22.82% F1 advantage over VDC (Table 2) may therefore reflect self-consistency rather than general visual-textual inconsistency detection. Please provide a benchmark-independent validation: for example, human-verify a random subset of MVTID labels, evaluate HMGIE against human judgements as the gold standard, or report results on a benchmark with independently sourced inconsistency labels.","section":"§4.1 and §5.1 (MVTID construction and experimental setup)"},{"comment":"The external evaluations on NewsCLIPpings, TIIL, and SeeTRUE report only TPR, with no FPR, precision, or F1. Since HMGIE's overall consistency decision dL is the product of all correctness indicators, a single unverifiable claim (e.g., 'Barry's home' or 'Dulles airport' in Table 15) forces an 'inconsistent' decision; this mechanism can inflate TPR on fake-news datasets by construction. Appendix C.2 acknowledges 'relatively high FPR' and explains it as dataset flaws, but this is a post hoc relabeling of ground truth without independent validation, and it does not establish that HMGIE's detections are correct. Please report FPR and F1 on these datasets, and either use the original dataset labels without relabeling or provide external human validation for any relabeled samples.","section":"§5.2, Table 5, and Appendix C.2"},{"comment":"The completeness score Hcomp is computed as a weighted sum of n_j / N_j, where n_j is the number of evaluation nodes at level j derived from the caption's semantic graph Gs and N_j is 'the maximum allowable nodes at that level,' which is never defined. More importantly, because the nodes come only from the caption, Hcomp measures the fraction of the caption's semantic graph that was checked, not whether the caption covers the actual image content. If a caption omits a major visual element, no node exists in Gs, no question is generated, and Hcomp can remain high. The paper's claim of 'semantic completeness' (Abstract and Section 3.4) is therefore not supported by the described method. Please either redefine Hcomp using image-derived elements, or explicitly restrict the claim to caption-relative completeness. Also, Eq. (6) uses K in the summation bound and constraint while the text defines L evaluation levels; please reconcile this notation.","section":"§3.4, Eq. (6)"},{"comment":"The human correlation experiments use only 400 pairs sampled by the authors and rated by 10 annotators, and the annotators' consistency ratings are compared against Hacc and Hcomp. While this is a useful sanity check, the annotators were not asked to make the binary consistent/inconsistent decision that drives Table 2, and no inter-annotator agreement (e.g., Krippendorff's alpha) is reported. Given that the paper positions HMGIE for data cleansing, where binary filtering decisions matter, please report agreement statistics and, if possible, the annotators' binary decisions against which TPR/FPR can be computed.","section":"§5.2, Tables 2-4"}],"minor_comments":[{"comment":"Line 11 contains a typo: 'Expend HIEG' should be 'Expand HIEG'.","section":"Algorithm 1"},{"comment":"The constraint is written as 's.t. K_j=1 α_j = 1' in the submitted text, but the summation should presumably run over L levels to match the definition of Hacc; a consistent notation across Eqs. (5) and (6) would avoid confusion.","section":"§3.4, Eq. (6)"},{"comment":"References [21] and [22] list the same paper (Huang et al., 'Exposing text-image inconsistency using diffusion models') in different venues; please merge them.","section":"References"},{"comment":"The caption reads 'Filter8k test set' but the text and footer refer to Flickr8k; please correct the typo.","section":"Figure 7"},{"comment":"The paper states that the maximum evaluation level is set to five, but it does not explain how the level-specific weights ω and α (geometric sequence with ratio 1.2) were chosen or how sensitive the results are to this choice; a brief sensitivity study would strengthen the reproducibility of the H-Scores.","section":"§5.1, Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The core methodological idea is appealing and the paper is likely within scope for the journal, but the benchmark circularity is a serious concern that goes to the validity of the headline result. I would recommend asking the authors to either provide a benchmark-independent validation of MVTID (e.g., human-verified labels or an external gold standard) or substantially temper the claims about general superiority. The external-dataset evaluation without FPR is also not sufficient to support the data-cleansing claim. These issues are fixable within the scope of a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a solid, clearly written engineering contribution with a real design idea, but its central superiority claim is built on a self-confirming benchmark. The framework is coherent: parse a caption into a semantic graph, then hierarchically generate and answer questions with coverage masks, producing Hacc and Hcomp scores plus an explanation. That combination genuinely goes beyond TIFA, VDC, and DSG, and the prompt-level detail is unusually complete. The MVTID benchmark, with four caption granularities, is also a useful resource.\n\nThe soft spot is the one the stress-test note flags, and it is load-bearing. MVTID labels are made by a GPT-4o-centric ensemble, and HMGIE uses GPT-4o for every component: graph parsing, question generation, VQA, and answer evaluation. On MVTID, HMGIE is essentially checking whether GPT-4o's own caption generation and perturbation process introduced an inconsistency. That can inflate F1 substantially without implying general visual-textual inconsistency detection. The external benchmarks (NewsCLIPpings, TIIL, SeeTRUE) report only TPR, and the appendix shows HMGIE frequently relabels \"consistent\" samples as inconsistent because the image could not verify out-of-context claims. Without FPR, those TPR numbers do little to establish accuracy.\n\nThe Hcomp completeness score has a separate but related problem: it measures how much of the caption was checked, not whether the caption covers the image. If a caption omits a major visual element, no graph node exists, no question is generated, and Hcomp can stay high. Also, N_l in Eq. 6 is never defined. These are real issues, though they do not sink the binary detection claim.\n\nThere is still value here. The hierarchical QA design is plausible, the human correlation results for Hacc/Hcomp are modest but real, and the ablation showing semantic graph removal hurts performance is informative. The concerns are addressable: release code and data, report FPR on all external benchmarks, use a human-annotated validation set or at least ensure the benchmark creators and evaluation model are not the same family, and fix the definition of N_l.\n\nWho is this for? Researchers working on VLM data cleansing and QA-based faithfulness evaluation. It deserves a serious referee, not a desk reject, but I would not cite it as a demonstrated SOTA until the circularity is resolved. My recommendation: send to peer review with a request for major revision focusing on the benchmark and metrics.","headline":"A well-engineered hierarchical QA framework for VTI evaluation whose headline results mostly measure self-consistency with its own GPT-4o benchmark; the design is worth peer review, but the central claim needs independent validation.","tokens_in":20292,"tokens_out":1818,"would_cite":false,"duration_ms":19038,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that parsing a caption into a semantic graph and verifying each element with progressively finer visual questions detects image-caption mismatches at 94.94% recall and 92.68% F1, with scores that track human judgment.","keywords":["visual-textual inconsistency evaluation","vision-language data cleansing","semantic graph","hierarchical question answering","image-caption consistency","H-Scores","multimodal large language models","multi-granularity dataset"],"falsifier":"Take an image whose scene contains a clearly salient object (say, a fire truck) and write two fluent captions for it: one complete, one that silently omits that object. Because the omitted object never enters the caption-derived semantic graph, no question targets it, so $H_{\\text{comp}}$ should come out nearly identical for the two captions; if the completeness score is supposed to reflect coverage of the image, this test settles the claim whenever it fails to separate the pair. A second, cheaper check is reproducibility: rerun identical pairs with a different question-generation LLM and a different VQA model — if $H_{\\text{acc}}$ and $H_{\\text{comp}}$ shift measurably on unchanged inputs, the scores are properties of the model configuration rather than of the image-caption pair.","tokens_in":19242,"feed_emoji":"🔍","tokens_out":14366,"duration_ms":109994,"temperature":0.7,"pith_summary":"Visual-textual inconsistency evaluation decides whether a caption matches its image, a core step for cleaning vision-language datasets. This paper proposes HMGIE, which parses the caption into a semantic graph of entities, attributes, and relations, then walks the graph with a hierarchy of visual questions — coarse object-level checks first, subtle detail checks later — while tracking which elements have been covered. From the resulting hierarchical evaluation graph it derives an accuracy score, a completeness score, a binary consistency verdict, and a natural-language explanation of any problems found. On the new multi-granularity benchmark MVTID introduced by the paper, the method reaches a 94.94% true-positive rate and a 92.68% F1 score, beating the previous best method by 22.82 F1 points, and its scores correlate with human ratings better than the standard embedding baseline or raw caption length.","feed_headline":"Hierarchical questioning catches 94.9% of image-caption mismatches","feed_subtitle":"Accuracy and completeness scores track human judgment, and detection holds up on long captions where other methods fail.","key_machinery":"The load-bearing object is the semantic graph $G_s$, the structured parse of the caption, together with the Hierarchical Inconsistency Evaluation Graph (HIEG) built from it level by level. Each HIEG node is a quartet — question, reference answer taken from the graph, VQA answer taken from the image, and a correctness flag — and edges record dependencies between questions at different levels. The coverage mask $M_l$, a binary map marking which graph nodes and edges have not yet been verified, is the mechanism that turns a flat question list into a goal-directed interrogation: the question generator receives the mask and produces the next, finer level of questions for exactly the unexamined elements. The H-Scores compress the HIEG into two numbers: $H_{\\text{acc}}$ is a level-weighted mean of confidence-weighted correctness (Eq. 5), and $H_{\\text{comp}}$ is a level-weighted count of evaluation nodes normalized by the maximum allowable nodes per level (Eq. 6).","core_discovery":"The central claim is that image-caption inconsistency is best judged as a hierarchical, coverage-tracked interrogation of the caption's semantic content, rather than as one global similarity score or a single flat list of questions. The caption is first converted into a semantic graph $G_s$ whose nodes are entities, locations, concepts, events, and attributes and whose edges are actions, spatial relations, part-whole links, and quantitative links. A question generator then produces visual questions level by level — basic scene elements, then attributes and relationships, then fine-grained details — each with a reference answer read off the graph, and a VQA model answers each question from the image. A coverage mask records which graph elements have been examined and drives the next, finer round of questions, until everything is covered or the depth limit is reached. The accumulated question-answer nodes form the Hierarchical Inconsistency Evaluation Graph (HIEG), from which HMGIE computes an accuracy score $H_{\\text{acc}}$, a completeness score $H_{\\text{comp}}$, a strict all-correct consistency decision, and a natural-language explanation. The paper argues that this progressive, graph-guided design is why HMGIE outperforms embedding scores and flat QA methods, and why its performance degrades only mildly as captions grow longer and inconsistencies subtler.","pith_inferences":["Because the semantic graph is caption-derived, HMGIE's completeness is one-sided: it can certify that everything asserted in the caption was verified against the image, but not that all salient image content was described — a symmetric version that parses the image into its own graph and checks both directions would be the direct route to a true coverage score.","The reported H-Score values are functions of the LLM and VQA backbones; the paper's own ablation shows roughly 9% true-positive variation when the model changes, so absolute scores on new data should be read as rankings within a fixed model configuration rather than as model-independent measurements.","The all-correct decision rule marks a pair inconsistent whenever a single question fails, which likely drives both the high true-positive rate and the roughly 10% false-positive rate; a graded or per-level majority decision rule is a testable variant that could trade a few recall points for fewer false alarms.","The same HIEG machinery that flags inconsistencies also returns the specific failed questions, which makes it a ready-made generator of grounded hard negatives and repair targets for fine-tuning vision-language models, not just a filter."],"forward_implications":["On the paper's MVTID benchmark, HMGIE reaches a 94.94% true-positive rate and a 92.68% F1 score at a 9.94% false-positive rate, outperforming the second-best method by 22.82 F1 points.","Detection remains comparatively stable as captions lengthen, dropping only 6.36% in true-positive rate from the shortest to the longest granularity, while direct-prompt baselines drop by more than 44%.","The H-Scores align with human judgment more strongly than the standard baselines: $H_{\\text{acc}}$ beats the embedding-based score on every granularity, and $H_{\\text{comp}}$ beats caption length as a completeness proxy.","The framework transfers to other data-cleaning settings, reporting 95.56% true-positive rate on NewsCLIPpings, 96.46% on TIIL, and 76.99% on SeeTRUE.","A captioning model fine-tuned on HMGIE-cleansed data produced the best captions for 45.8% of test images, despite training on only half the data volume of the mixed-data baseline."],"supporting_citations":[{"why":"The embedding-based image-text similarity baseline that HMGIE compares against and beats in human-judgment correlation on every granularity level.","marker":"[18]"},{"why":"The QA-based faithfulness method whose flat question decomposition HMGIE extends into hierarchical, coverage-tracked levels.","marker":"[20]"},{"why":"The strongest QA-based baseline (second-best on MVTID) that HMGIE outperforms by 22.82% in F1.","marker":"[48]"},{"why":"The composition-aware embedding baseline used to show embedding methods miss fine-grained inconsistencies in longer captions.","marker":"[44]"},{"why":"The COCO image source used to construct the MVTID dataset and its adversarially perturbed captions.","marker":"[5]"},{"why":"GPT-4o, the proprietary model serving as the LLM/MLLM backbone for all HMGIE modules in the main experiments.","marker":"[32]"},{"why":"The SeeTRUE benchmark used for the text-to-image generation consistency extension.","marker":"[40]"},{"why":"The NewsCLIPpings fake-news dataset used for the out-of-context detection extension.","marker":"[30]"},{"why":"The TIIL fine-grained fake-news dataset used for the text-image mismatch extension.","marker":"[21]"}],"fun_headline_variants":["Graph-guided QA pinpoints caption mismatches at any length","Hierarchical graph probes find image-caption errors others miss","Multi-grained VQA probing scores caption accuracy and completeness","Semantic graph drills down to expose image-caption mismatches","Hierarchical multi-grained QA cleans caption data with precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The completeness score $H_{\\text{comp}}$ is computed only from the semantic graph parsed out of the caption itself, so the framework measures how much of the caption was checked, not whether the caption covers the actual image content: a caption that omits a salient object never creates a node for that object, never generates a question about it, and never lowers $H_{\\text{comp}}$, whose normalizing term, the 'maximum allowable nodes' $N_l$, is left undefined in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Graph-guided QA pinpoints caption mismatches at any length","Hierarchical graph probes find image-caption errors others miss","Multi-grained VQA probing scores caption accuracy and completeness","Semantic graph drills down to expose image-caption mismatches","Hierarchical multi-grained QA cleans caption data with precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001248,"raw_usage":{"total_tokens":5191,"prompt_tokens":1089,"completion_tokens":4102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":705,"completion_tokens_details":{"reasoning_tokens":4022}},"tokens_in":705,"tokens_out":4102,"duration_ms":27436,"temperature":1.0,"reasoning_tokens":4022,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:27:29.900746+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an image whose scene contains a clearly salient object (say, a fire truck) and write two fluent captions for it: one complete, one that silently omits that object. Because the omitted object never enters the caption-derived semantic graph, no question targets it, so $H_{\\text{comp}}$ should come out nearly identical for the two captions; if the completeness score is supposed to reflect coverage of the image, this test settles the claim whenever it fails to separate the pair. A second, cheaper check is reproducibility: rerun identical pairs with a different question-generation LLM and a different VQA model — if $H_{\\text{acc}}$ and $H_{\\text{comp}}$ shift measurably on unchanged inputs, the scores are properties of the model configuration rather than of the image-caption pair.","supporting_citations":[{"cited_title":"Clipscore: A reference-free evaluation met- ric for image captioning","cited_arxiv_id":null,"evidence_quote":"The embedding-based image-text similarity baseline that HMGIE compares against and beats in human-judgment correlation on every granularity level."},{"cited_title":"Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering","cited_arxiv_id":null,"evidence_quote":"The QA-based faithfulness method whose flat question decomposition HMGIE extends into hierarchical, coverage-tracked levels."},{"cited_title":"mirror” → “window","cited_arxiv_id":null,"evidence_quote":"The strongest QA-based baseline (second-best on MVTID) that HMGIE outperforms by 22.82% in F1."},{"cited_title":"When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023","cited_arxiv_id":null,"evidence_quote":"The composition-aware embedding baseline used to show embedding methods miss fine-grained inconsistencies in longer captions."},{"cited_title":"Hello gpt-4o, 2024","cited_arxiv_id":null,"evidence_quote":"GPT-4o, the proprietary model serving as the LLM/MLLM backbone for all HMGIE modules in the main experiments."},{"cited_title":"What you see is what you read? improving text- image alignment evaluation","cited_arxiv_id":null,"evidence_quote":"The SeeTRUE benchmark used for the text-to-image generation consistency extension."},{"cited_title":"Newsclip- pings: Automatic generation of out-of-context multimodal media","cited_arxiv_id":null,"evidence_quote":"The NewsCLIPpings fake-news dataset used for the out-of-context detection extension."},{"cited_title":"Exposing text-image inconsistency using dif- fusion models","cited_arxiv_id":null,"evidence_quote":"The TIIL fine-grained fake-news dataset used for the text-image mismatch extension."}],"review_version":1}