Pith. sign in

REVIEW 3 major objections 5 minor 82 references

This paper claims that static, single-turn benchmarks systematically understate vision-language model hallucinations, and that contextualized multi-turn interviews with an adaptive examiner reveal more failures—especially ones that resemble

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:53 UTC pith:M4OIGFMI

load-bearing objection A genuinely new interactive VLM hallucination evaluation design with a defensible central claim; the SVG graph-completeness issue is real but not fatal, and it deserves a serious referee. the 3 major comments →

arxiv 2607.14499 v1 pith:M4OIGFMI submitted 2026-07-16 cs.AI

Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

Yijiang Li , Huiqi Zou , Bingyang Wang , Ziang Xiao This is my paper
classification cs.AI
keywords vision-language modelsvisual hallucinationmulti-turn evaluationscene graphsgraph edit distancedynamic benchmarkingdialogue historypremise rejection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that static, single-turn benchmarks miss a large share of vision-language model hallucinations, and that the missed failures are exactly the ones that matter in practical use. It proposes CEDI, which recasts evaluation as a conversation among an examiner, the model, and a grader; the examiner plans questions by walking over a scene graph and adapts its strategy to the model's previous answers. Across models and datasets, the interactive protocol consistently surfaces more hallucinations than captioning, yes/no, or scripted prompts, and the errors it surfaces are judged more relevant to realistic user goals. A second claim is that hallucinations accumulate: once a model asserts something unsupported, later turns amplify it, and models are especially prone to fail when a question requires rejecting a false premise. If true, benchmark rankings based on static tests do not reflect deployment behavior, and evaluation should be rebuilt around interaction.

Core claim

The paper's central claim is that visual hallucination—output that is not faithful to the image—is a context-dependent behavior that static evaluation systematically under-samples. CEDI makes evaluation a three-party conversation: an examiner plans questions by walking over a scene graph, the evaluatee model answers across multiple turns, and a grader parses the full transcript into a predicted graph and compares it to ground truth. The examiner varies four question types—regular, follow-up, adversarial (probing plausible-but-absent objects), and unanswerable (false-premise) questions—and each turn conditions on the dialogue history. The paper reports that this protocol raises hallucination

What carries the argument

The load-bearing mechanism is the scene graph—a graph of objects, attributes, and relations representing image content. CEDI treats this graph as a state space: the examiner embeds the image in a goal-driven user scenario, selects a context-relevant subgraph, and traverses it turn by turn to generate questions and follow-ups; the grader parses the model's transcript into a predicted graph and scores the mismatch with graph edit distance (GED), isolating hallucinated content (GEDhal) from uncovered ground-truth content (GEDcov). The same representation carries both elicitation and scoring, which is what lets the evaluation remain grounded while adapting to the model's own answers.

Load-bearing premise

The pipeline assumes the ground-truth scene graphs are complete and accurate: the examiner can only ask about what the graphs contain and the grader counts anything outside them as a hallucination, so omitted objects inflate measured hallucinations and spurious elements hide real ones—an assumption the paper itself flags in its limitations.

What would settle it

Take a random sample of test images and have independent annotators build exhaustive scene graphs from scratch; re-run CEDI using both the original ground-truth graphs and the corrected graphs. If original graphs contain omissions, faithful statements get scored as hallucinations and the CEDI-vs-static gap shrinks or vanishes; if the gap survives with corrected graphs, the central claim is confirmed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, static benchmark scores understate real-world hallucination rates, so a model that looks strong on a leaderboard may fail more often in interactive use.
  • Hallucination is not a fixed property of a model: the same model speaks differently when its earlier claims are in the conversation, so evaluations should report interaction-dependent behavior rather than a single static accuracy number.
  • Long conversations are a stress test: because errors build on earlier errors, evaluation length and history handling directly affect measured performance.
  • Coverage can be controlled by the examiner through context sampling, which means evaluation breadth can be expanded without forcing models to produce unnaturally exhaustive captions.
  • The scene-graph grader catches attribute and relation hallucinations that object-only metrics miss, giving a fuller error profile beyond object presence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the gap is real, deployment choices should be validated interactively; two models with identical static scores may differ sharply under follow-up questioning.
  • Editorial inference: The same examiner/grader loop could be repurposed for other context-sensitive failures—unsupported tool use, unsafe answers, or instruction drift—where dialogue history is the trigger.
  • Editorial inference: The method could be turned into a calibration tool: measure a model's hallucination rate as a function of turn number to decide when to re-ground the model with a fresh image caption.
  • Editorial inference: The reported sample-level hallucination rates might be partly a function of question difficulty; one could test this by matching question length or lexical difficulty between CEDI and static prompts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CEDI, a three-party framework for evaluating visual hallucinations in VLMs. A scene-graph-guided examiner conducts multi-turn, goal-driven interviews using four question types (regular, follow-up, adversarial, unanswerable); a scene-graph grader parses the resulting transcript and computes GED and SG-ΔCon against ground-truth scene graphs. Experiments on VG and SVG across nine models compare CEDI to caption, prompt-only, and POPE baselines, using automatic metrics and human annotation. The authors report that CEDI exposes more hallucinations, achieves broader coverage, yields more human-realistic interactions, and that hallucinations accumulate over dialogue history, with GEDhal correlating with human annotations more consistently than CHAIR/MMHal.

Significance. If the headline result is robust, the paper is a useful methodological contribution: static benchmarks understate interactive failure modes, and the examiner/grader separation with a graph state space is a principled move beyond fixed rubrics. Strengths include a released implementation, a four-way question-type ablation that isolates refusal-demanding probes, multi-turn history ablations, parser validation (P=0.882, R=0.879, F1=0.881), human annotation of hallucinations and interaction realism, and a stability check showing the SG grader's std is below 1% of the metric mean. The central claim is currently conditional on the completeness of G* and on statistical support that the manuscript does not provide; both issues are addressable within the framework's scope.

major comments (3)
  1. [§3.1, §3.2, §7, Appendix F] The completeness/accuracy of the ground-truth scene graph G* is load-bearing for the central claim. §3.1 restricts examiner questions to a traversal over V_s ⊆ V*; §3.2 scores every predicted node/edge absent from G* as a deletion (hallucination); and the adversarial/unanswerable question types specifically target elements not in G*. If G* omits content that is actually present, a truthful model statement is scored as a hallucination, and a probe about an 'absent' element can in fact be about real content. The large-scale SVG graphs are automatically constructed (Sec. 7) with no completeness audit, and the human annotation protocol (Appendix F) provides annotators with the same GT captions and object locations, so it cannot independently adjudicate these cases. The paper needs (i) a recall/completeness audit of G* on a subsample, or (ii) a sensitivity analysis that reports headline compa
  2. [§5.1, Tables 1, 2, 5, Figure 4] Central quantitative comparisons lack uncertainty quantification and significance tests. The abstract claims 'significantly more hallucinations', and Table 2 says CEDI's six statistics 'reflect significantly more hallucinations', but no confidence intervals, bootstrap estimates, or p-values are reported for any of the main comparisons in Tables 1, 2, 5, or Figure 4. Given that the examiner is LLM-based and stochastic, and that samples/contexts are drawn per image, the absence of variability estimates weakens the quantitative claim. Some differences are modest (e.g., GPT-4o: 67.89% POPE vs 67.38% CEDI in Table 1). The paper should report per-metric CIs, paired statistical tests, or effect sizes for the headline comparisons, and adjust the wording of 'significantly' accordingly.
  3. [§5.1, Table 3, Figure 4] A substantial part of the headline effect is built into the question design: adversarial and unanswerable questions are defined as probes about elements absent from G*, while the baselines do not include such probes. The ablation removing these two types is the right control and shows the effect persists, but it is presented only as aggregate means over three evaluatees, without per-model spread, CIs, or significance tests; moreover, on MMHal the ablated variant ties with the prompt-only baseline (34.0 vs 34.9). The paper should report the ablation with the same statistical support as the main results and make the non-adversarial, contextualized-interaction effect the primary basis for the 'more hallucinations' claim. Otherwise the abstract overstates what the design can distinguish from task construction.
minor comments (5)
  1. [Figure 4, Appendix B] Radar axes are scaled differently per metric (GED by /150, SGΔCon by /5, 100−VALOR faithi by /50), but the axis labels and caption do not state this scaling. Please make the scaling explicit in the caption or use normalized, comparable units.
  2. [Table 3] The table body appears to run values together (e.g., '48.652.739.4 160.177.8 63.14.76'). Please separate columns clearly.
  3. [§3.2] The GED definition uses Π(Ĝ,G⋆) as the set of valid edit paths, but the minimization over this set is not precisely specified (e.g., whether edit costs are uniform, node-specific, or require isomorphism up to synset alignment). This matters for reproducibility.
  4. [§5.2, Figure 12] The GEDhal-human correlations (0.22–0.33) are modest, and for LLaVA-1.5 CHAIR nhall reaches 0.32, close to GEDhal's 0.33. The claim that GEDhal is the 'most consistent' signal is plausible but would be strengthened by confidence intervals and a direct comparison of correlation differences.
  5. [§5.1, 'Human Annotation'] The phrase 'significantly more hallucinations' in the text is not backed by a statistical test on the human-annotated counts; please add a paired test (e.g., bootstrap/resampling) or rephrase.

Circularity Check

1 steps flagged

Headline hallucination-exposure result is partly design-inherent, but independent ablations and human validation give it substantial non-circular content.

specific steps
  1. self definitional [Section 3.1, Question Types (Adversarial/Unanswerable) and Table 3 (Ablation Study on Question-type)]
    "Adversarial questions ask an existential question about a highly plausible but absent object v /∈ Vs ... Unanswerable questions introduce a false premise ... G |= p ⇒ G |= q(p). ... Removing the two types does reduce CEDI's hallucination exposure across most graders (GED 160.1→156.7 ...)."

    The head-to-head claim 'CEDI reveals significantly more hallucinations than conventional static evaluation' is partly built into the examiner design, because adversarial and unanswerable questions are defined as probes about content absent from the graph Gs; any model that answers rather than refuses is scored as hallucinating. The gradient from Table 3 shows that including these types raises CHAIRI (45.2→48.6) and other hallucination metrics. The paper itself stops short of claiming these types are the sole driver, and its 'without adv./unans.' ablation still exceeds baselines, so the headline effect is not fully definitional, but the increment from two of the four question types is by construction.

full rationale

The central quantitative claim — that CEDI elicits more hallucinations than static caption/POPE baselines — does not reduce to a fitted parameter or to a self-citation. No parameter is fit to the target claim, no self-citation is load-bearing: the only self-referential aspect is that the examiner and grader both use the same G* scene graphs for question generation and scoring, which creates a genuine graph-completeness assumption (acknowledged in the Limitations: scene graphs for SVG are 'automatically constructed,' with no completeness audit). If G* omits real content, truthful model statements can be scored as hallucinations. That concern, however, is a validity/correctness risk, not circularity in the derivation. The paper also contains independent, non-circular support for the headline effect: (1) the 'w/o adversarial and unanswerable' ablation still exceeds caption and prompt baselines on most graders; (2) the multi-round ablation (Figure 7) and the history/no-history comparison (Figure 9) show elicitation driven by interaction structure, holding question type constant; (3) human annotation (Table 2) independently confirms the relative ordering of examiner conditions, although annotators are given the same GT captions and object locations. The only step that is definitional is the contribution of adversarial/unanswerable probes to hallucination counts, and the paper openly treats those as the primary locus of the increase while also showing they are not the sole cause. That warrants a mildly elevated score (2), not a 4-6. The graph-completeness limitation is real but is a measurement-validity threat, not a case where the prediction is equivalent to its input by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

CEDI introduces no new physical entities, forces, or conserved quantities. The examiner and grader are software components built from existing LLMs and scene-graph concepts. The main numerical choices are metric weights and evaluation hyperparameters, while the deeper burden is the accuracy of the ground-truth scene graphs and the LLM parsing/human annotation chain.

free parameters (3)
  • lambda_attribute_weight = not reported
    In the SGDeltaCon grader, d = dDelta + lambda * dA (Section 3.2). The weight lambda is never specified; it changes the balance between structural distance and attribute mismatch and can affect model rankings.
  • GED_edit_costs = not reported
    Graph-edit-distance in Section 3.2 assigns cost c(e) to node/edge insertions, deletions, and substitutions, but no cost schedule is given. Unit costs are implicitly assumed; different costs would change GED magnitudes and possibly rankings.
  • num_contexts_per_image = 1, 2, or 5 in sweep
    Coverage scaling (Figure 5) depends on the number of sampled contexts per image. This is an evaluation hyperparameter rather than a fitted constant, but it materially changes all reported metrics.
axioms (6)
  • domain assumption Ground-truth scene graphs are accurate, complete representations of image content (Sections 3.2, 4.1).
    Both the examiner's question generation and the grader's distance computation assume G* captures all relevant objects, attributes, and relations.
  • domain assumption LLM-based semantic parsing faithfully reconstructs the claims in a transcript, validated at F1=0.881 on 50 responses (Section 5.2).
    The SG-based grader's predicted graph is only as good as the parser; the validation set is small and based on one protocol.
  • domain assumption Human-annotated hallucination labels are a reliable external criterion (Section 4.1, Appendix F).
    Human annotation is used to validate both examiners and graders, but annotators are given the same ground-truth captions and object locations, so their judgments inherit any GT errors.
  • domain assumption The GPT-4o examiner generates useful, goal-driven contexts and questions and follows graph constraints, with no independent quality control (Section 3.1, Limitations).
    The examiner is a single proprietary backbone; its failures or biases propagate into all conversations and metrics.
  • domain assumption WordNet physical-entity filtering removes non-visual nouns without excluding valid visual objects (Section 3.2).
    The parser restricts entities to WordNet physical classes; if a visible object is not in WordNet, it is silently dropped, affecting coverage and hallucination counts.
  • domain assumption Automatically constructed SVG scene graphs are adequate substitutes for human annotations (Sections 4.1, 7).
    The paper relies on SVG for cross-domain evaluation; Section 7 acknowledges this dependency but does not quantify SVG graph noise.

pith-pipeline@v1.3.0-alltime-deepseek · 21098 in / 11582 out tokens · 120408 ms · 2026-08-02T01:53:55.558039+00:00 · methodology

0 comments
read the original abstract

Multi-modal Large Language Models (MLLMs) have made substantial advances on benchmarks, yet their real-world effectiveness remains uncertain. This gap stems from the fundamental misalignment between benchmarks in controlled, static settings and the dynamic, interactive, and contextualized nature of real-world applications. To bridge this gap, we propose CEDI (Contextualized Evaluations of MLLMs through Dynamic, multi-round Interactions), a framework that recasts evaluation as a three-party interaction between an evaluatee model, an automated examiner, and a grader. The examiner conducts multi-turn, semi-structured conversation guided by a graph-based representation of the task. By navigating state-space transitions, CEDI deploys diverse strategies, from clarification requests to adversarial probes, to elicit performance evidence. We apply CEDI to visual hallucinations. Empirical results across multiple models, diverse settings, datasets, and domains show that contextualized, interactive evaluations reveal not only significantly more hallucinations than conventional static evaluation but also ones that more closely resemble those arising in practical use cases. We further show that hallucinations often accumulate over long contexts, through self-reinforcing dialogue history, and models are particularly vulnerable to questions requiring premise rejection or refusal. Together, these findings highlight CEDI as a step toward realistic, systematic, and ecologically valid assessments of MLLMs' capabilities. Code is available at github.com/williamium3000/cedi.

Figures

Figures reproduced from arXiv: 2607.14499 by Bingyang Wang, Huiqi Zou, Yijiang Li, Ziang Xiao.

Figure 1
Figure 1. Figure 1: CEDI’s three-party interaction (left) contrasted with prior static examiners (right). Building on this framing, we propose CEDI (Contextualized Evaluations of MLLMs through Dynamic multi-round Interactions). CEDI draws inspiration from evaluative interviews in hu￾man assessment: rather than relying on static, paper-based tests, a semi-structured interview allows the examiner to adapt question type and diff… view at source ↗
Figure 2
Figure 2. Figure 2: Workflow of CEDI, including our contextualized and interactive examiner and scene graph (SG)-based grader. multimodal reasoning in scientific domains. Other benchmarks, including GQA [37], OK-VQA [31], OwlEval [38], MMBench [39], and MMMU [40], evaluate visual perception and reasoning through large collections of question–answer pairs. To improve coverage, SEED-Bench [41] and SEED-Bench-2-Plus [42] extend … view at source ↗
Figure 3
Figure 3. Figure 3: Multi-turn conversation as a traversal over Gs, with one of four question types selected per turn. Examiner Backbone. To enable structured yet adaptive evaluation, we combine the flexibility of MLLMs with the graph-based workflow described above. The examiner is empowered by an MLLM such as GPT-4o [10]. Concretely, we prompt an MLLM to generate the scenario and objectives, actively select the question type… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of CEDI, caption-based examiners, and prompt-only examiners across six models on the SVG dataset. All axes are oriented such that a larger radius indicates more severe hallucination. See Appendix B for grader definitions and radar-plot scaling factors. LLaVA-1.5 Qwen2.5-VL Qwen3-VL 0.0 0.2 0.4 0.6 0.8 C H AIRI ( ↑ ) CHAIRI ( ↑ ) LLaVA-1.5 Qwen2.5-VL Qwen3-VL 0.0 0.1 0.2 0.3 0.4 C o v. ( ↑ ) Cov.… view at source ↗
Figure 5
Figure 5. Figure 5: Hallucination and coverage metrics with different numbers of contexts per image. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ablation study on node selection and contextualization. Ablation Study on Question-type. CEDI mixes four question types: regular, follow-up, adversarial (adv.), and unanswerable (unans.). To test the contribution of the two refusal-demanding question types (adv. and unans.), we ablate them, keeping only regular and follow-up, and compare against the two examiner baselines. Removing the two types does reduc… view at source ↗
Figure 7
Figure 7. Figure 7: ∆ (%) change of metrics without multi-round interaction compared to CEDI. Negative ∆ (%) indicates fewer hallucina￾tions. CEDI Caption POPE 0 20 40 60 80 100 Percentage of Annotations (%) 46% 30% 24% 32% 36% 32% 22% 34% 44% *p = . 011 Rank 1 (Best) Rank 2 Rank 3 (Worst) [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effect of conversational history on CEDI. Per-metric means over 5 evaluatees, with each w/-hist mean shown relative to its w/o-hist mean (= 100%); absolute values annotated. regular follow￾up adversarial unanswerable 0 20 40 60 80 100 VA L O R 1−faithi (%) w/o hist w/ hist [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Relevance of questions (left) and iden￾tified hallucinations (right) across examiners to the context. Question relevance is not evaluated in the Caption baseline since all queries are al￾most the same. CHAIRI CHAIR nhall MMHal GEDhal InternVL3-8B LLaVA-1.5-7B Opera-LLaVA-1.5 Qwen3-VL-8B -0.05 +0.06 +0.27 +0.32 +0.04 +0.32 -0.02 +0.33 -0.29 -0.03 +0.20 +0.32 -0.32 -0.24 +0.33 +0.22 −0.4 0.0 0.4 [PITH_FULL… view at source ↗
Figure 13
Figure 13. Figure 13: Per-turn hallucination behavior in CEDI, measured by VALOR 1−faithi . Four Types of Questions. Figure 13a shows that regular, follow-up, and adversarial questions account for most hallucinated turns. Although unanswerable questions occur less frequently, they yield the highest per-turn hallucination rate (see [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

82 extracted references · 1 canonical work pages

  1. [1]

    Euclid: Supercharging mul- timodal llms with synthetic high-fidelity visual descriptions.arXiv preprint arXiv:2412.08737, 2024

    Jiarui Zhang, Ollie Liu, Tianyu Yu, Jinyi Hu, and Willie Neiswanger. Euclid: Supercharging mul- timodal llms with synthetic high-fidelity visual descriptions.arXiv preprint arXiv:2412.08737, 2024. 13

    Pith/arXiv arXiv 2024

  2. [2]

    MLLMs know where to look: Training-free perception of small visual details with multimodal LLMs

    Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. MLLMs know where to look: Training-free perception of small visual details with multimodal LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=DgaY5mDdmT

    2025

  3. [3]

    Introducing visual perception token into multi- modal large language model.arXiv preprint arXiv:2502.17425, 2025

    Runpeng Yu, Xinyin Ma, and Xinchao Wang. Introducing visual perception token into multi- modal large language model.arXiv preprint arXiv:2502.17425, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Multi-modal instruction tuned llms with fine-grained visual perception

    Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned llms with fine-grained visual perception. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13980–13990. IEEE, 2024. doi: 10.1109/CVPR52733.2024. 01326. URLh...

    arXiv 2024

  5. [5]

    Lion: Empowering multimodal large language model with dual-level visual knowledge

    Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26540–26550, 2024

    2024

  6. [6]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

    2024

  7. [7]

    Image-of- thought prompting for visual reasoning refinement in multimodal large language models.arXiv preprint arXiv:2405.13872, 2024

    Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang. Image-of- thought prompting for visual reasoning refinement in multimodal large language models.arXiv preprint arXiv:2405.13872, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large language models

    Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind’s eye of llms: visualization-of-thought elicits spatial reasoning in large language models. Advances in Neural Information Processing Systems, 37:90277–90317, 2024

    2024

  9. [9]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9062–9072, 2025

    2025

  10. [10]

    Hello GPT-4o, 2024

    OpenAI. Hello GPT-4o, 2024. URLhttps://openai.com/index/hello-gpt-4o/

    2024

  11. [11]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Natural language can help bridge the sim2real gap

    Albert Yu, Adeline Foote, Ray Mooney, and Roberto Martín-Martín. Natural language can help bridge the sim2real gap. InRSS 2024 Workshop: Data Generation for Robotics, 2024. URL https://openreview.net/forum?id=wXP7nlzmAh

    2024

  14. [14]

    YiFan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang, and Rong Jin. MME-realworld: Could your multimodal LLM challenge high-resolution real-world scenarios that are difficult for humans? InThe Thirteenth International Conference on Learning Representations, 2025. URLhttp...

    2025

  15. [15]

    Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    Pith/arXiv arXiv 2018

  16. [16]

    Hallucination is inevitable: An innate limitation of large language models.arXiv preprint arXiv:2401.11817, 2024

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models.arXiv preprint arXiv:2401.11817, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Hallulens: Llm hallucination benchmark.arXiv preprint arXiv:2504.17550, 2025

    Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. Hallulens: Llm hallucination benchmark.arXiv preprint arXiv:2504.17550, 2025. 14

    Pith/arXiv arXiv 2025

  18. [18]

    A survey on hallucination in large vision-language models.arXiv preprint arXiv:2402.00253, 2024

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models.arXiv preprint arXiv:2402.00253, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13872–13882, June 2024

    2024

  20. [20]

    Gregor Geigle, Radu Timofte, and Goran Glavaš. Does object grounding really reduce hallucination of large vision-language models? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2728–2742, Miami, Florida, USA, November 2024. Association for Computat...

    doi:10.18653/v1/2024.emnlp-main.159 2024

  21. [21]

    Evaluating and analyzing relationship hallucinations in large vision-language models

    Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. Evaluating and analyzing relationship hallucinations in large vision-language models. In Proceedings of the 41st International Conference on Machine Learning, pages 53553–53570, 2024

    2024

  22. [22]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, 2023

    2023

  23. [23]

    Ciem: Contrastive instruction evaluation method for better instruction tuning.arXiv preprint arXiv:2309.02301, 2023

    Hongyu Hu, Jiyuan Zhang, Minyi Zhao, and Zhenbang Sun. Ciem: Contrastive instruction evaluation method for better instruction tuning.arXiv preprint arXiv:2309.02301, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Negative object presence evaluation (nope) to measure object hallucination in vision-language models

    Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. Negative object presence evaluation (nope) to measure object hallucination in vision-language models. InProceedings of the 3rd Workshop on Advances in Language and Vision Research (ALVR), pages 37–58, 2024

    2024

  25. [25]

    Hallu-pi: Evaluating hallucination in multi-modal large language models within perturbed inputs

    Peng Ding, Jingyu Wu, Jun Kuang, Dan Ma, Xuezhi Cao, Xunliang Cai, Shi Chen, Jiajun Chen, and Shujian Huang. Hallu-pi: Evaluating hallucination in multi-modal large language models within perturbed inputs. InProceedings of the 32nd ACM International Conference on Multimedia, pages 10707–10715, 2024

    2024

  26. [26]

    Detecting and preventing hallucinations in large vision language models

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135–18143, 2024

    2024

  27. [27]

    Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models.arXiv preprint arXiv:2404.13874, 2024

    Haoyi Qiu, Wenbo Hu, Zi-Yi Dou, and Nanyun Peng. Valor-eval: Holistic coverage and faithfulness evaluation of large vision-language models.arXiv preprint arXiv:2404.13874, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Synthetic visual genome

    Jae Sung Park, Zixian Ma, Linjie Li, Chenhao Zheng, Cheng-Yu Hsieh, Ximing Lu, Khyathi Chandu, Quan Kong, Norimasa Kobori, Ali Farhadi, et al. Synthetic visual genome. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9073–9086, 2025

    2025

  29. [29]

    Aligning large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. InFindings of the Association for Computational Linguistics ACL 2024, pages 13088–13110, 2024

    2024

  30. [30]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. ICCV ’15, page 2425–2433, USA,

  31. [31]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 15

    2019

  32. [32]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. InProceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015

    2015

  33. [33]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors,Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1

    2014

  34. [34]

    Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

    2024

  35. [35]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186. Springer, 2024

    2024

  36. [36]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35: 2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35: 2507–2521, 2022

    2022

  37. [37]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  38. [38]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  40. [40]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024

    2024

  41. [41]

    Seed- bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension.arXiv preprint arXiv:2404.16790, 2024

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension.arXiv preprint arXiv:2404.16790, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Mme-survey: A comprehensive survey on evaluation of multimodal llms.arXiv preprint arXiv:2411.15296, 2024

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal llms.arXiv preprint arXiv:2411.15296, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms.Advances in Neural Information Processing Systems, 37:8698–8733, 2024

    Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms.Advances in Neural Information Processing Systems, 37:8698–8733, 2024

    2024

  45. [45]

    Convbench: A multi-turn conversation evaluation benchmark with hierarchical capability for large vision-language models.arXiv preprint arXiv:2403.20194, 2024

    Shuo Liu, Kaining Ying, Hao Zhang, Yue Yang, Yuqi Lin, Tianle Zhang, Chuanhao Li, Yu Qiao, Ping Luo, Wenqi Shao, et al. Convbench: A multi-turn conversation evaluation benchmark with hierarchical capability for large vision-language models.arXiv preprint arXiv:2403.20194, 2024. 16

    Pith/arXiv arXiv 2024

  46. [46]

    Mmrc: A large-scale benchmark for understanding multimodal large language model in real-world conversation.arXiv preprint arXiv:2502.11903, 2025

    Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qidong Huang, Yulong Li, Chengzhi Liu, Zhongxing Xu, Chong Zhang, Chun-Mei Feng, et al. Mmrc: A large-scale benchmark for understanding multimodal large language model in real-world conversation.arXiv preprint arXiv:2502.11903, 2025

    Pith/arXiv arXiv 2025

  47. [47]

    Toward unsupervised realistic visual question answering

    Yuwei Zhang, Chih-Hui Ho, and Nuno Vasconcelos. Toward unsupervised realistic visual question answering. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 15613–15624, 2023

    2023

  48. [48]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. InThe Twelfth Inter- national Conference on Learning Representations, 2024. URL https://openreview.net/ forum?id=J44HfH4JCg

    2024

  49. [49]

    Evaluation and analysis of hallucination in large vision-language models.arXiv preprint arXiv:2308.15126, 2023

    Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models.arXiv preprint arXiv:2308.15126, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Mit- igating open-vocabulary caption hallucinations

    Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar Averbuch-Elor. Mit- igating open-vocabulary caption hallucinations. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22680–22698, 2024

    2024

  51. [51]

    Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.arXiv preprint arXiv:2311.03287, 2023

    Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.arXiv preprint arXiv:2311.03287, 2023

    Pith/arXiv arXiv 2023

  52. [52]

    Faithscore: Fine-grained evaluations of hallucinations in large vision-language models

    Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du. Faithscore: Fine-grained evaluations of hallucinations in large vision-language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 5042–5063, 2024

    2024

  53. [53]

    Lanp: Rethinking the impact of language priors in large vision-language models.arXiv preprint arXiv:2502.12359, 2025

    Zongyu Wu, Yuwei Niu, Hongcheng Gao, Minhua Lin, Zhiwei Zhang, Zhifang Zhang, Qi Shi, Yilong Wang, Sike Fu, Junjie Xu, et al. Lanp: Rethinking the impact of language priors in large vision-language models.arXiv preprint arXiv:2502.12359, 2025

    Pith/arXiv arXiv 2025

  54. [54]

    Beaf: Observing before- after changes to evaluate hallucination in vision-language models

    Moon Ye-Bin, Nam Hyeon-Woo, Wonseok Choi, and Tae-Hyun Oh. Beaf: Observing before- after changes to evaluate hallucination in vision-language models. InEuropean Conference on Computer Vision, pages 232–248. Springer, 2024

    2024

  55. [55]

    Image retrieval using scene graphs

    Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3668–3678, 2015

    2015

  56. [56]

    About WordNet, 2010

    Princeton University. About WordNet, 2010. URLhttps://wordnet.princeton.edu/

    2010

  57. [57]

    V ogelstein, Brian Gallagher, and Christos Faloutsos

    Danai Koutra, Neil Shah, Joshua T. V ogelstein, Brian Gallagher, and Christos Faloutsos. Deltacon: Principled massive-graph similarity function with attribution.ACM Trans. Knowl. Discov. Data, 10(3), February 2016. ISSN 1556-4681. doi: 10.1145/2824443. URL https://doi.org/10.1145/2824443

    work page doi:10.1145/2824443 2016

  58. [58]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    2017

  59. [59]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122–5130, 2017. doi: 10.1109/CVPR.2017.544

    doi:10.1109/cvpr.2017.544 2017

  60. [60]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024. 17

    2024

  61. [61]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 6dcf277ea32ce3288...

    2023

  62. [62]

    InternVL2, 2024

    OpenGVLab. InternVL2, 2024. URL https://internvl.github.io/blog/ 2024-07-02-InternVL-2.0/

    2024

  63. [63]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  65. [65]

    Introducing gpt -5.4 mini and nano, 2026

    OpenAI. Introducing gpt -5.4 mini and nano, 2026. URL https://openai.com/index/ introducing-gpt-5-4-mini-and-nano/

    2026

  66. [66]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Pith/arXiv arXiv 2025

  67. [67]

    outer = more hall

    Qwen Team. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388. 18 A Ethics Statement This paper presents work whose goal is to advance the evaluation of VLMs by introducing a contextualized and interactive framework for diagnosing hallucinations. Improved hallucination evaluations have the potential to positively impact the deployment of mu...

    Pith/arXiv arXiv 2025

  68. [69]

    Candidate pool(one LLM call). Given {background, goal}, prompt an LLM (Qwen3-30B-A3B- Instruct-2507) to list N= 50 concrete visible objects that could appear in the scenario, mixing plausible-present and plausible-absent

  69. [70]

    vehicle” counts asyes if the scene graph has “car

    Existence + co-occurrence check(one LLM call). Given the image’s ground-truth object list (from the scene graph) and the 50 candidates, the LLM (i) labels each candidate yes/no using synonym and parent-class matching (e.g. “vehicle” counts asyes if the scene graph has “car”), and (ii) flags which of the no-candidates are frequent real-world co-occurrences...

  70. [71]

    is there a ⟨object⟩in the image?

    Composition.From the labelled pool, draw exactly nrounds questions per image in the ratio 1 2 yes: 1 4 randomno: 1 4 co-occurrenceno, then shuffle. Each question is formatted as“is there a ⟨object⟩in the image?”with the correspondingyes/noground-truth answer. This pipeline preserves POPE’s binary-question format and its balanced yes/no ratio, but replaces...

  71. [72]

    You need to go through all the sentences in the model’s responses

  72. [73]

    You should pay attention to:

    Identify the segment / sub-sentence that’s wrong. You should pay attention to:

  73. [74]

    Object: Verify whether the objects in the image exist and if the quantity of objects conflicts with the object information

  74. [75]

    the color, position, action of objects in the image conflict with the attribute information

    Attributes: Check whether the attributes, e.g. the color, position, action of objects in the image conflict with the attribute information

  75. [76]

    Relations: Check whether the relation between the objects and their interaction is correct

  76. [77]

    For instance, if the texts in the image are also mentioned in the conversation, check whether they are the same

    Scene Text: Confirm whether the textual information in the scene of the image conflicts with the textual information given in the conversation. For instance, if the texts in the image are also mentioned in the conversation, check whether they are the same. Conversation Realism Ranking Guidelines.Annotators were shown three candidate conversations generate...

  77. [78]

    Read theBackgroundandGoal

  78. [79]

    Some questions show an answer snippet below, highlighted in red and tagged H#

    Each question line starts with Q# . Some questions show an answer snippet below, highlighted in red and tagged H# . Questions with no snippet show no answer

  79. [80]

    Tick Q# if that question is related to the goal; leave it unchecked if not

  80. [81]

    Note: some highlighted answers may appear correct to you

    Tick H# if that highlighted mistake is related to the goal; leave it unchecked if not. Note: some highlighted answers may appear correct to you. If so, you do not need to tick it

Showing first 80 references.