Pith. sign in

REVIEW 4 major objections 5 minor 68 references

HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that parsing a caption into a semantic graph and verifying each element with progressively finer visual questions detects image-caption mismatches at 94.94% recall and 92.68% F1, with scores that track human judgment.

desk verdict A well-engineered hierarchical QA framework for VTI evaluation whose headline results mostly measure self-consistency with its own GPT-4o benchmark; the design is worth peer review, but the central claim needs independent validation. read the letter →

arxiv 2412.05685 v1 pith:CIGCAN5G submitted 2024-12-07 cs.CV cs.AI

Zihao Zhu , Hongbao Zhang , Guanzong Wu , Siwei Lyu , Baoyuan Wu This is my paper · ORCID
classification cs.CVcs.AI
keywords visual-textualinconsistencyevaluationvision-languagedatacleansingsemanticgraphhierarchicalquestionansweringimage-captionconsistencyH-Scoresmultimodallargelanguagemodelsmulti-granularitydataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual-textual inconsistency evaluation decides whether a caption matches its image, a core step for cleaning vision-language datasets. This paper proposes HMGIE, which parses the caption into a semantic graph of entities, attributes, and relations, then walks the graph with a hierarchy of visual questions — coarse object-level checks first, subtle detail checks later — while tracking which elements have been covered. From the resulting hierarchical evaluation graph it derives an accuracy score, a completeness score, a binary consistency verdict, and a natural-language explanation of any problems found. On the new multi-granularity benchmark MVTID introduced by the paper, the method reaches a 94.94% true-positive rate and a 92.68% F1 score, beating the previous best method by 22.82 F1 points, and its scores correlate with human ratings better than the standard embedding baseline or raw caption length.

What carries the argument

The load-bearing object is the semantic graph $G_s$, the structured parse of the caption, together with the Hierarchical Inconsistency Evaluation Graph (HIEG) built from it level by level. Each HIEG node is a quartet — question, reference answer taken from the graph, VQA answer taken from the image, and a correctness flag — and edges record dependencies between questions at different levels. The coverage mask $M_l$, a binary map marking which graph nodes and edges have not yet been verified, is the mechanism that turns a flat question list into a goal-directed interrogation: the question generator receives the mask and produces the next, finer level of questions for exactly the unexamined elements. The H-Scores compress the HIEG into two numbers: $H_{\text{acc}}$ is a level-weighted mean of confidence-weighted correctness (Eq. 5), and $H_{\text{comp}}$ is a level-weighted count of evaluation nodes normalized by the maximum allowable nodes per level (Eq. 6).

What would settle it

Take an image whose scene contains a clearly salient object (say, a fire truck) and write two fluent captions for it: one complete, one that silently omits that object. Because the omitted object never enters the caption-derived semantic graph, no question targets it, so $H_{\text{comp}}$ should come out nearly identical for the two captions; if the completeness score is supposed to reflect coverage of the image, this test settles the claim whenever it fails to separate the pair. A second, cheaper check is reproducibility: rerun identical pairs with a different question-generation LLM and a different VQA model — if $H_{\text{acc}}$ and $H_{\text{comp}}$ shift measurably on unchanged inputs, the scores are properties of the model configuration rather than of the image-caption pair.

Watch

Extended reading notes

Core claim

The central claim is that image-caption inconsistency is best judged as a hierarchical, coverage-tracked interrogation of the caption's semantic content, rather than as one global similarity score or a single flat list of questions. The caption is first converted into a semantic graph $G_s$ whose nodes are entities, locations, concepts, events, and attributes and whose edges are actions, spatial relations, part-whole links, and quantitative links. A question generator then produces visual questions level by level — basic scene elements, then attributes and relationships, then fine-grained details — each with a reference answer read off the graph, and a VQA model answers each question from the image. A coverage mask records which graph elements have been examined and drives the next, finer round of questions, until everything is covered or the depth limit is reached. The accumulated question-answer nodes form the Hierarchical Inconsistency Evaluation Graph (HIEG), from which HMGIE computes an accuracy score $H_{\text{acc}}$, a completeness score $H_{\text{comp}}$, a strict all-correct consistency decision, and a natural-language explanation. The paper argues that this progressive, graph-guided design is why HMGIE outperforms embedding scores and flat QA methods, and why its performance degrades only mildly as captions grow longer and inconsistencies subtler.

Load-bearing premise

The completeness score $H_{\text{comp}}$ is computed only from the semantic graph parsed out of the caption itself, so the framework measures how much of the caption was checked, not whether the caption covers the actual image content: a caption that omits a salient object never creates a node for that object, never generates a question about it, and never lowers $H_{\text{comp}}$, whose normalizing term, the 'maximum allowable nodes' $N_l$, is left undefined in the paper.

Editorial extensions

If this is right

  • On the paper's MVTID benchmark, HMGIE reaches a 94.94% true-positive rate and a 92.68% F1 score at a 9.94% false-positive rate, outperforming the second-best method by 22.82 F1 points.
  • Detection remains comparatively stable as captions lengthen, dropping only 6.36% in true-positive rate from the shortest to the longest granularity, while direct-prompt baselines drop by more than 44%.
  • The H-Scores align with human judgment more strongly than the standard baselines: $H_{\text{acc}}$ beats the embedding-based score on every granularity, and $H_{\text{comp}}$ beats caption length as a completeness proxy.
  • The framework transfers to other data-cleaning settings, reporting 95.56% true-positive rate on NewsCLIPpings, 96.46% on TIIL, and 76.99% on SeeTRUE.
  • A captioning model fine-tuned on HMGIE-cleansed data produced the best captions for 45.8% of test images, despite training on only half the data volume of the mixed-data baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the semantic graph is caption-derived, HMGIE's completeness is one-sided: it can certify that everything asserted in the caption was verified against the image, but not that all salient image content was described — a symmetric version that parses the image into its own graph and checks both directions would be the direct route to a true coverage score.
  • The reported H-Score values are functions of the LLM and VQA backbones; the paper's own ablation shows roughly 9% true-positive variation when the model changes, so absolute scores on new data should be read as rankings within a fixed model configuration rather than as model-independent measurements.
  • The all-correct decision rule marks a pair inconsistent whenever a single question fails, which likely drives both the high true-positive rate and the roughly 10% false-positive rate; a graded or per-level majority decision rule is a testable variant that could trade a few recall points for fewer false alarms.
  • The same HIEG machinery that flags inconsistencies also returns the specific failed questions, which makes it a ready-made generator of grounded hard negatives and repair targets for fine-tuning vision-language models, not just a filter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HMGIE, a hierarchical and multi-grained framework for evaluating visual-textual inconsistency. HMGIE converts a caption into a semantic graph, progressively constructs a hierarchical inconsistency evaluation graph (HIEG) through question-answer generation and VQA, and computes accuracy and completeness scores (Hacc and Hcomp) together with natural-language explanations. The authors introduce MVTID, a dataset of image-caption pairs with four granularities of inconsistency generated by an LLM/MLLM ensemble, and report TPR/F1/FPR on MVTID, TPR on external datasets (NewsCLIPpings, TIIL, SeeTRUE), human correlation results for the H-Scores, ablation studies, and qualitative analyses.

Significance. If the central empirical claim held, HMGIE would be a practically useful data-cleansing tool that also produces interpretable explanations. The paper has several genuine strengths: the modular design is clear, the ablation in Section 5.3 shows that the semantic graph contributes to detection, Figure 4 demonstrates the framework works with open-source LLMs/MLLMs, and Tables 3-4 report correlation with human judgments. However, the main claim of superior performance is substantially weakened by two issues: (i) the MVTID benchmark is constructed and evaluated with the same GPT-4o model family, so the reported F1 advantage may partly reflect self-consistency rather than general inconsistency detection; and (ii) the external benchmark results report only TPR, with the appendix dismissing high false positives as dataset flaws. These issues are central to the paper's headline result.

major comments (4)
  1. [§4.1 and §5.1 (MVTID construction and experimental setup)] The MVTID labels and the HMGIE evaluation pipeline both rely on GPT-4o: ground-truth captions are generated by an ensemble that includes GPT-4o, fused by GPT-4o, adversarially perturbed by GPT-4o, with perturbation detectability judged by an MLLM (Section 4.1); and HMGIE uses GPT-4o for semantic graph generation, question generation, VQA, and answer evaluation (Section 5.1). The consistent class is what three models agree on, and the inconsistent class is what GPT-4o's perturbation process produced. On this benchmark, HMGIE is partly measuring how well its own reasoning decomposition catches inconsistencies that the same model family introduced. The reported 22.82% F1 advantage over VDC (Table 2) may therefore reflect self-consistency rather than general visual-textual inconsistency detection. Please provide a benchmark-independent validation: for example, human-verify a random subset of MVTID labels, evaluate HMGIE against human judgements as the gold standard, or report results on a benchmark with independently sourced inconsistency labels.
  2. [§5.2, Table 5, and Appendix C.2] The external evaluations on NewsCLIPpings, TIIL, and SeeTRUE report only TPR, with no FPR, precision, or F1. Since HMGIE's overall consistency decision dL is the product of all correctness indicators, a single unverifiable claim (e.g., 'Barry's home' or 'Dulles airport' in Table 15) forces an 'inconsistent' decision; this mechanism can inflate TPR on fake-news datasets by construction. Appendix C.2 acknowledges 'relatively high FPR' and explains it as dataset flaws, but this is a post hoc relabeling of ground truth without independent validation, and it does not establish that HMGIE's detections are correct. Please report FPR and F1 on these datasets, and either use the original dataset labels without relabeling or provide external human validation for any relabeled samples.
  3. [§3.4, Eq. (6)] The completeness score Hcomp is computed as a weighted sum of n_j / N_j, where n_j is the number of evaluation nodes at level j derived from the caption's semantic graph Gs and N_j is 'the maximum allowable nodes at that level,' which is never defined. More importantly, because the nodes come only from the caption, Hcomp measures the fraction of the caption's semantic graph that was checked, not whether the caption covers the actual image content. If a caption omits a major visual element, no node exists in Gs, no question is generated, and Hcomp can remain high. The paper's claim of 'semantic completeness' (Abstract and Section 3.4) is therefore not supported by the described method. Please either redefine Hcomp using image-derived elements, or explicitly restrict the claim to caption-relative completeness. Also, Eq. (6) uses K in the summation bound and constraint while the text defines L evaluation levels; please reconcile this notation.
  4. [§5.2, Tables 2-4] The human correlation experiments use only 400 pairs sampled by the authors and rated by 10 annotators, and the annotators' consistency ratings are compared against Hacc and Hcomp. While this is a useful sanity check, the annotators were not asked to make the binary consistent/inconsistent decision that drives Table 2, and no inter-annotator agreement (e.g., Krippendorff's alpha) is reported. Given that the paper positions HMGIE for data cleansing, where binary filtering decisions matter, please report agreement statistics and, if possible, the annotators' binary decisions against which TPR/FPR can be computed.
minor comments (5)
  1. [Algorithm 1] Line 11 contains a typo: 'Expend HIEG' should be 'Expand HIEG'.
  2. [§3.4, Eq. (6)] The constraint is written as 's.t. K_j=1 α_j = 1' in the submitted text, but the summation should presumably run over L levels to match the definition of Hacc; a consistent notation across Eqs. (5) and (6) would avoid confusion.
  3. [References] References [21] and [22] list the same paper (Huang et al., 'Exposing text-image inconsistency using diffusion models') in different venues; please merge them.
  4. [Figure 7] The caption reads 'Filter8k test set' but the text and footer refer to Flickr8k; please correct the typo.
  5. [§5.1, Implementation Details] The paper states that the maximum evaluation level is set to five, but it does not explain how the level-specific weights ω and α (geometric sequence with ratio 1.2) were chosen or how sensitive the results are to this choice; a brief sensitivity study would strengthen the reproducibility of the H-Scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No formal circularity in the claimed derivation chain; the GPT-4o overlap between MVTID construction and HMGIE components is a benchmark-validity caveat, not an equation-level reduction.

full rationale

HMGIE is not trained or fitted to MVTID labels; its consistency decision is a deterministic function of caption-derived reference answers and GPT-4o VQA answers computed through the HIEG, as defined in Eqs. (2)-(6) and Algorithm 1. The MVTID construction in Sec. 4.1 indeed uses GPT-4o for caption fusion, adversarial perturbation, and detectability checking, while Sec. 5.1 states that GPT-4o is the base model for all HMGIE components. This overlap is a legitimate external-validity concern and can inflate measured agreement, but it does not make the predicted binary label equal to the ground-truth label by construction: direct GPT-4o prompting, CoT prompting, and HMGIE all use the same backbone yet produce very different results in Table 2, so the framework's output is not identical to its input. The external benchmarks (NewsCLIPpings, TIIL, SeeTRUE) provide independent evidence, and the paper reports those results explicitly, even if only as TPR. The Hcomp metric (Eq. 6) is caption-derived and has an undefined N_l, which is a construct-validity weakness for measuring true image completeness, but Hcomp is defined from the caption's semantic graph and compared empirically with human ratings; it is not derived from those ratings or from the binary consistency labels. The paper also cites the authors' prior VDC work, but only as a baseline and motivation, not as a load-bearing uniqueness theorem or unverified premise. No self-definitional identity, fitted-input-as-prediction, or self-citation chain was found that meets the quoted-reduction standard required for a circularity finding.

Assumptions & free parameters 4 free parameters · 3 assumptions · 3 invented entities

The central claim rests on the reliability of four chained LLM components, on an LLM-generated benchmark, and on an undefined completeness denominator. The level weights and maximum depth are hand-chosen, and no exact values for N_l are given. These are the inputs the paper does not justify independently.

free parameters (4)
  • Level weights omega_j and alpha_j = geometric sequence ratio 1.2 (exact values not specified)
    Eqs. (5) and (6) require weights summing to 1; the paper fixes them as a geometric sequence with ratio 1.2, an arbitrary choice that shifts Hacc and Hcomp values.
  • Maximum allowable nodes N_l per level = unspecified
    Hcomp in Eq. (6) divides n_j by N_j but the paper never defines N_j, so the completeness score is not computable from the text.
  • Maximum evaluation depth K = 5
    Section 5.1 caps question generation at five levels; changing K changes the amount of content evaluated and thus Hcomp and the decision d.
  • LLM temperature = 0.3
    Section 5.1 sets temperature to 0.3 for all LLM and MLLM calls; sampling variability is not reported across runs.
assumptions (3)
  • domain assumption MVTID's LLM-generated and adversarially perturbed captions are valid ground truth for visual-textual inconsistency.
    Section 4.1 constructs all labels via MLLM generation and MLLM-based undetectability checks; no human verification or inter-annotator agreement is reported.
  • domain assumption A single incorrect answer to any generated question implies the image-caption pair is inconsistent.
    The final decision d is the product of all correctness flags in Section 3.4, so the framework assumes every generated question is meaningful and every feval judgment is correct.
  • ad hoc to paper Caption-derived semantic graph coverage equals image-content completeness.
    Eq. (6) computes Hcomp from the caption graph Gs only; the paper provides no evidence that counting caption-derived nodes measures whether a caption describes all important image content.
invented entities (3)
  • HIEG
    purpose: Structured graph of progressive question-answer evaluation nodes with dependencies
    Defined in Section 3.3; no external implementation, release, or validation outside this paper.
  • H-Scores (Hacc and Hcomp)
    purpose: Quantify semantic accuracy and completeness from the HIEG
    Defined in Section 3.4; the only validation is internal human-correlation data collected by the authors, with no external benchmark or released implementation.
  • MVTID dataset
    purpose: Multi-granularity benchmark for VTI evaluation with four caption detail levels
    Constructed in Section 4 with LLM-generated captions and perturbations; no release link or downstream use by independent groups is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing." pith.science (2026). https://pith.science/paper/CIGCAN5G

@misc{pith2026241205685,
  author       = {Pith},
  title        = {Pith review of: HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CIGCAN5G}},
  note         = {Machine review of arXiv:2412.05685}
}
read the original abstract

Visual-textual inconsistency (VTI) evaluation plays a crucial role in cleansing vision-language data. Its main challenges stem from the high variety of image captioning datasets, where differences in content can create a range of inconsistencies (\eg, inconsistencies in scene, entities, entity attributes, entity numbers, entity interactions). Moreover, variations in caption length can introduce inconsistencies at different levels of granularity as well. To tackle these challenges, we design an adaptive evaluation framework, called Hierarchical and Multi-Grained Inconsistency Evaluation (HMGIE), which can provide multi-grained evaluations covering both accuracy and completeness for various image-caption pairs. Specifically, the HMGIE framework is implemented by three consecutive modules. Firstly, the semantic graph generation module converts the image caption to a semantic graph for building a structural representation of all involved semantic items. Then, the hierarchical inconsistency evaluation module provides a progressive evaluation procedure with a dynamic question-answer generation and evaluation strategy guided by the semantic graph, producing a hierarchical inconsistency evaluation graph (HIEG). Finally, the quantitative evaluation module calculates the accuracy and completeness scores based on the HIEG, followed by a natural language explanation about the detection results. Moreover, to verify the efficacy and flexibility of the proposed framework on handling different image captioning datasets, we construct MVTID, an image-caption dataset with diverse types and granularities of inconsistencies. Extensive experiments on MVTID and other benchmark datasets demonstrate the superior performance of the proposed HMGIE to current state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2412.05685 by the authors.

Figure 1
Figure 1. Comparison of inconsistency evaluation methods on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall illustration of our proposed hierarchical and multi-grained inconsistency evaluation (HMGIE) framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 6
Figure 6. Comparison of explanation quality between HMGIE and [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: ROUGE-4 for cap￾tions before and after repair. across varying levels of granularity. Impact of Various LLMs and MLLMs. To eval￾uate the flexibility and robustness of our framework, we conducted experiments by replacing the GPT-4o in HMGIE with various open-source alter…
Figure 7
Figure 7. Figure 7: Distribution of caption quality among three models on Flickr8k test set, as evaluated by GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Examples of MVTID dataset. 5 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Examples of generated captions and evaluations by GPT-4o. Each example consists of an image, three captions generated by [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Example of semantic graph for raw caption. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Example of semantic graph for noisy caption. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Example of HIEG for the raw caption. 15 [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Example of HIEG for the noisy caption. 16 [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 56 canonical work pages

  1. [1]

    The kendall rank correlation coefficient

    Herv ´e Abdi. The kendall rank correlation coefficient. Ency- clopedia of measurement and statistics, 2007. 6

    work page 2007

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 2022. 1

    work page 2022

  3. [3]

    Claude 3.5 haiku, 2024

    Anthropic. Claude 3.5 haiku, 2024. 6

    work page 2024

  4. [4]

    A survey of multimodal large language model from a data- centric perspective

    Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, et al. A survey of multimodal large language model from a data- centric perspective. arXiv preprint arXiv:2405.16640, 2024. 1, 2

    arXiv 2024

  5. [5]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 5

    arXiv 2015

  6. [6]

    Gpt4sgg: Synthesizing scene graphs from holistic and region-specific narratives

    Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Changwen Chen. Gpt4sgg: Synthesizing scene graphs from holistic and region-specific narratives. arXiv preprint arXiv:2312.04314, 2023. 4

    arXiv 2023

  7. [7]

    Davidsonian scene graph: Improving reliabil- ity in fine-grained evaluation for text-to-image generation

    Jaemin Cho, Yushi Hu, Jason Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliabil- ity in fine-grained evaluation for text-to-image generation. In ICLR, 2024. 6

    work page 2024

  8. [8]

    Davidsonian Scene Graph: Improv- ing Reliability in Fine-Grained Evaluation for Text-to-Image Generation

    Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ran- jay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont- Tuset, and Su Wang. Davidsonian Scene Graph: Improv- ing Reliability in Fine-Grained Evaluation for Text-to-Image Generation. In ICLR, 2024. 3

    work page 2024

Show all 68 references
  1. [9]

    Diffusion models in vision: A survey

    Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023. 1

    2023

  2. [10]

    InstructBLIP: Towards general-purpose vision- language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In Conference on Neural Information Processing Systems, 2023. 2

    2023

  3. [11]

    Flashattention: Fast and memory-efficient exact at- tention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christo- pher R´e. Flashattention: Fast and memory-efficient exact at- tention with io-awareness. Advances in Neural Information Processing Systems, 2022. 8

    2022

  4. [12]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  5. [13]

    Diffusion self-guidance for control- lable image generation

    Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for control- lable image generation. Advances in Neural Information Processing Systems, 2023. 1

    2023

  6. [14]

    Exploring the potential of dense information in multimodal alignment

    Zhiyuan Fan, Zhihong Chen, and Benyou Wang. Exploring the potential of dense information in multimodal alignment. In Findings of the Association for Computational Linguistics ACL, 2024. 2

    2024

  7. [15]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Sys...

    2024

  8. [16]

    Llama-adapter v2: Parameter-efficient vi- sual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xi- angyu Yue, et al. Llama-adapter v2: Parameter-efficient vi- sual instruction model. arXiv preprint arXiv:2304.15010 ,

    arXiv

  9. [17]

    Geneval: An object-focused framework for evaluating text- to-image alignment

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment. Advances in Neural Information Pro- cessing Systems, 2024. 2

    2024

  10. [18]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing,

    2021

  11. [19]

    Cas: A probability-based approach for universal condition align- ment score

    Chunsan Hong, ByungHee Cha, and Tae-Hyun Oh. Cas: A probability-based approach for universal condition align- ment score. In The Twelfth International Conference on Learning Representations, 2024. 2

    2024

  12. [20]

    Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Os- tendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, 20...

    2023

  13. [21]

    Exposing text-image inconsistency using dif- fusion models

    Mingzhen Huang, Shan Jia, Zhou Zhou, Yan Ju, Jialing Cai, and Siwei Lyu. Exposing text-image inconsistency using dif- fusion models. In The Twelfth International Conference on Learning Representations, 2024. 6, 7, 17

    2024

  14. [22]

    Exposing text-image inconsistency using dif- fusion models

    Mingzhen Huang, Shan Jia, Zhou Zhou, Yan Ju, Jialing Cai, and Siwei Lyu. Exposing text-image inconsistency using dif- fusion models. arXiv preprint arXiv:2404.18033, 2024. 2

    2024 arXiv

  15. [23]

    Llm4sgg: Large language models for weakly supervised scene graph generation

    Kibum Kim, Kanghoon Yoon, Jaehyeong Jeon, Yeonjun In, Jinyoung Moon, Donghyun Kim, and Chanyoung Park. Llm4sgg: Large language models for weakly supervised scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

  16. [24]

    Quality estimation for image captions based on large-scale human evaluations

    Tomer Levinboim, Ashish V Thapliyal, Piyush Sharma, and Radu Soricut. Quality estimation for image captions based on large-scale human evaluations. In Proceedings of the 2021 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Lan...

    2021

  17. [25]

    Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with 9 frozen image encoders and large language models. In In- ternational conference on machine learning, 2023. 1

    2023

  18. [26]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004. 8

    2004

  19. [27]

    Evaluating text-to-visual generation with image-to-text gen- eration

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gen- eration. In European Conference on Computer Vision . Springer, 2025. 2

    2025

  20. [28]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 2024. 8

    2024

  21. [29]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arXiv preprint arXiv:2403.05525,

    arXiv

  22. [30]

    Newsclip- pings: Automatic generation of out-of-context multimodal media

    Grace Luo, Trevor Darrell, and Anna Rohrbach. Newsclip- pings: Automatic generation of out-of-context multimodal media. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. 6, 7, 2, 17

    2021

  23. [31]

    Llama 3.2, 2024

    Meta. Llama 3.2, 2024. 6, 7, 8

    2024

  24. [32]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. 6, 7

    2024

  25. [33]

    The synergy between data and multi-modal large language mod- els: A survey from co-development perspective

    Zhen Qin, Daoyuan Chen, Wenhao Zhang, Liuyi Yao, Yilun Huang, Bolin Ding, Yaliang Li, and Shuiguang Deng. The synergy between data and multi-modal large language mod- els: A survey from co-development perspective. arXiv preprint arXiv:2407.08583, 2024. 1

    2024 arXiv

  26. [34]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, 2021. 2

    2021

  27. [35]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023. 1

    2023

  28. [36]

    Efficient attention: Attention with lin- ear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with lin- ear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531– 3539, 2021. 8

    2021

  29. [37]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 6

    2024 arXiv

  30. [38]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 2022. 2, 6

    2022

  31. [39]

    Smoothquant: Accurate and ef- ficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and ef- ficient post-training quantization for large language models. In International Conference on Machine Learning, 2023. 8

    2023

  32. [40]

    What you see is what you read? improving text- image alignment evaluation

    Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text- image alignment evaluation. Advances in Neural Informa- tion Processing Systems, 2024. 2, 6, 7, 18

    2024

  33. [41]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  34. [42]

    mplug- owi2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug- owi2: Revolutionizing multi-modal large language model with modality collaboration. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 1

    2024

  35. [43]

    Delving into noisy label detection with clean data

    Chenglin Yu, Xinsong Ma, and Weiwei Liu. Delving into noisy label detection with clean data. In International Con- ference on Machine Learning, 2023. 5

    2023

  36. [44]

    When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- language models behave like bags-of-words, and what to do about it? In International Conference on Learning Repre- sentations, 2023. 2, 6, 7

    2023

  37. [45]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 1

    2023

  38. [46]

    Data cleaning using large language models

    Shuo Zhang, Zezhou Huang, and Eugene Wu. Data cleaning using large language models. arXiv preprint arXiv:2410.15547, 2024. 2

    2024 arXiv

  39. [47]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representa- tions, 2023. 1

    2023

  40. [48]

    mirror” → “window

    Zihao Zhu, Mingda Zhang, Shaokui Wei, Bingzhe Wu, and Baoyuan Wu. Vdc: Versatile data cleanser based on visual- linguistic inconsistency by multimodal large language mod- els. In The Twelfth International Conference on Learning Representations, 2024. 2, 5, 6, 7 10 HMGIE: Hiera...

    2024

  41. [49]

    - Location: Scenes or places - Concept: Abstract Concept - Event: Actions or events that occur - Attribute: Characteristics describing entities, locations, or events - Others

    Node Types: - Entity: People, animals, objects, etc. - Location: Scenes or places - Concept: Abstract Concept - Event: Actions or events that occur - Attribute: Characteristics describing entities, locations, or events - Others

  42. [50]

    - Has Attribute: Features of entities, locations, or events - Part Of: Representing compositional relationships - Quantity: Representing amounts or counts - Others

    Edge Types: - Action: Actions performed by a subject on an object - Spatial: Representing the spatial relationship between entities and locations. - Has Attribute: Features of entities, locations, or events - Part Of: Representing compositional relationships - Quantity: Repres...

  43. [51]

    nodes”: [ {“id

    Output JSON Format: { “nodes”: [ {“id”: “N1”, “type”: “Entity”, “label”: “entity name” }, {......} ], “edges”: [ { “from”: (“N1”, “entity-name-1”), “to”: (“N3”, “entity-name-3”), “type”: “Action”, “label”: “action description”, “description”: “A sentence describing this triple...

  44. [52]

    Semantic Graph (in JSON format): - Nodes: entities and concepts mentioned in the caption - Edges: relationships between nodes

  45. [53]

    True is correct, False is incorrect

    Previous HIEG, where each node contains: - Question-ID: unique identifier - Question: the actual question text - Verify-Fact: the fact that this question is trying to verify - Expected-Answer: answer derived from semantic graph - Actual-Answer: answer provided by VQA module - ...

  46. [54]

    Current Level: the depth for new questions to be generated

  47. [55]

    Each question should:

    Suggestion: guide the direction or focus for this level of questions You need to generate different new questions for the current level. Each question should:

  48. [56]

    Build upon the history HIEG without repetition

  49. [57]

    Use the semantic graph for reference

    Generate the questions based on the semantic graph. Use the semantic graph for reference

  50. [58]

    Explore unverified nodes and edges in the semantic graph that are not covered by the history HIEG

  51. [59]

    Increase in difficulty and granularity compared to the previous level, but not excessively

  52. [60]

    Consider the direction or points raised in the suggestion, but not totally depend on it

  53. [61]

    Is the pottery large?

    Avoid generating questions that involve vague or relative attributes (e.g., “Is the pottery large?” or “Is the object small?”). Questions should not require answers based on subjective sizes or undefined comparisons. Please Ensure questions align with the depth appropriate for...

  54. [62]

    Semantic graph parsed from the text description of an image

  55. [63]

    - Question: The actual question text

    Current HIEG, each node containing: - Question-ID: A unique identifier. - Question: The actual question text. - Verify-Fact: The fact that this question is trying to verify. - Expected-Answer: The answer expected if the image and text were match. - Actual-Answer: Real received...

  56. [64]

    - Question: The actual question text

    HIEG structure where each node contains: - Question-ID: A unique identifier. - Question: The actual question text. - Verify-Fact: The fact that this question is trying to verify. - Expected-Answer: The answer expected if the image and text were match. - Actual-Answer: Real rec...

  57. [65]

    Original Caption: The text description being evaluated

  58. [66]

    Consistent

    Final Consistency Decision: “Consistent” or “Inconsistent” Your task is to generate a comprehensive explanation that:

  59. [67]

    For Inconsistent Cases: - Start with a clear statement of inconsistency - Present inconsistencies in a hierarchical order (from basic to detailed) - For each inconsistency:Identify the specific semantic element involved; Explain the discrepancy between the caption and image; R...

  60. [68]

    Input: HIEG: [{hieg}] Original Caption: [{caption}] Final Consistency Decision:[{consistency-decision}] 11 Figure 9

    For Consistent Cases: - Begin with a confirmation of consistency - Summarize the key semantic elements verified - Highlight important relationships and attributes confirmed - Organize verification results by evaluation levels - Emphasize any notable detailed verifications - Co...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.