{"id":"8e86a2c7-d77c-4bf7-9586-0bd7011bfa71","arxiv_id":"2511.13420","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A recheck-based evaluation shows LVLMs often fail to correctly judge the presence of objects they themselves generated, and current mitigation methods do not fix this.","lead":"VOPE is a new benchmark that asks vision-language models to recheck whether objects they mentioned in tasks like story-writing actually appear in the image, separating true voluntary imagination from hallucination. On this benchmark, most models show high hallucination rates on imagined objects, and existing hallucination-mitigation methods barely help.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Presence recheck may misclassify 'No' answers about story-specific instances as hallucination, potentially inflating Hal-I and the claim that models hallucinate heavily.","rationale":"The reader's weakest_assumption identified both COCO ground-truth incompleteness and the lack of independent verification of the model's self-reported interpretation. I agree that these are concerns, but the COCO issue is less load-bearing because incomplete labels make Hal-I conservative: objects present but missing from COCO would be counted as IT (true imagination) rather than IH, so fixing the labels would likely increase measured Hal-I, not decrease it. The self-report issue, however, could cut the other way: the recheck question's referential ambiguity may systematically overstate the model's failure to recognize present objects. This directly underpins the paper's headline finding and the claim that mitigation methods fail. The paper offers no evidence that 'No' answers are grounded in the same object instance as the story mention, and no probe for response bias in the presence question. My proposed test would directly resolve whether this ambiguity materially affects Hal-I. If it does, the quantitative conclusions would weaken; if it does not, the original findings stand. Given this uncertainty, the reader's CONDITIONAL verdict remains appropriate, so I do not change it.","tokens_in":16197,"tokens_out":6697,"duration_ms":73695,"concrete_test":"Sample a few hundred IH cases from Table 1 (object mention + model's 'No' + COCO present). Have two independent annotators view the original story, the image, and the exact recheck question, and judge whether the model's 'No' is plausibly about the specific object instance described in the story rather than the image category. Then, on the same models, run a controlled comparison: for each object, ask (a) 'Is there a [object] in the image?' and (b) 'Is the [object] you mentioned in your story present in the image?' (with the story provided as context). If a substantial fraction (e.g., >20%) of 'No' answers to (a) become 'Yes' to (b), the Hal-I metric conflates referential denial with misrecognition, and the reported rates need downward correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that LVLMs hallucinate heavily during voluntary imagination—rests on Hal-I, which counts as hallucination (IH) any object the model mentions and then denies in the recheck question \"Is there a [object] in the image?\" when the object is present per COCO. This assumes the model's \"No\" is a claim about the image category's presence, but in a story-writing context, the model may be referring to the specific fictional instance it described, not the generic category. For example, a model may write \"a few books\" as part of a narrative scene, and when asked about \"book,\" it may answer \"No\" because the particular book it imagined is not visibly present, even though another book appears in the image. The recheck question is underspecified relative to the generative task, and no control experiments are provided to validate that \"No\" responses correspond to genuine failures to recognize present objects. Unlike COCO label incompleteness—which would likely bias Hal-I downward, making the finding conservative—this ambiguity could inflate Hal-I systematically, thus directly threatening the magnitude (and possibly the existence) of the headline hallucination rates and the conclusion that mitigation methods fail.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VOPE, a recheck-based protocol for evaluating hallucination in LVLMs during tasks that invite voluntary imagination, such as story writing. Objects are extracted from a model-generated response, the model is asked a yes/no presence question about each object, and the answers are compared with image ground truth. Objects the model says are present are scored by a factual-description metric Hal-D; objects the model says are absent are scored by a voluntary-imagination metric Hal-I, with an additional Exp metric measuring the fraction of imagined objects. Experiments on MSCOCO with eight LVLMs across captioning, reasoning, and writing report that while Hal-D is low, Hal-I is high across models, and that several existing hallucination-mitigation methods do not reduce Hal-I. The paper also adds an MLLM-based relevance assessment of imagined objects and a small attention analysis on LLaVA1.5.","tokens_in":16476,"tokens_out":6847,"duration_ms":70721,"significance":"The core idea is timely and useful: existing hallucination metrics penalize any object not in the image and therefore conflate genuine hallucination with legitimate creative imagination. Distinguishing factual description from voluntary imagination and rechecking the model's own interpretation of its output is a sensible operationalization. The paper also provides a human evaluation of its relevance judge (weighted kappa 0.45), which is a useful validation step. If the headline numbers are correct, the finding that current mitigation methods do not improve Hal-I is practically important. However, the evidence currently rests on an unstated object-extraction pipeline, an underspecified ground-truth mapping, and an assumption about what a \"No\" answer means in the presence recheck; these issues directly affect the magnitude and even the existence of the main empirical claims.","major_comments":[{"comment":"The object-extraction step is never described, yet every metric is defined over the extracted object set. The paper does not state whether extraction is done by a parser, an LLM, a detector, or human annotators, how plurals/synonyms are resolved to canonical categories, or whether counts are per mention or per unique object. Without this, the numbers in Tables 1–3 are not reproducible.","section":"Sec. 3.2 / Fig. 2 / Sec. 4.1"},{"comment":"Ground-truth presence is said to come from MSCOCO, but MSCOCO annotations cover only 80 categories. Several objects used as examples in the paper, e.g., \"sun\", \"forest\", \"man\", \"rack\" in Fig. 2, and \"books\", \"television\" in Figs. 4 and 6, do not map cleanly to those 80 categories. The paper does not explain how such objects are labeled. If an object is present in the image but not in the COCO category list, a model's correct \"No\" is counted as true imagination and an incorrect \"Yes\" is counted as a factual-description hallucination; this can bias Hal-D upward and Hal-I downward, and it makes the exact magnitudes and model rankings unreliable.","section":"Sec. 4.1 / Fig. 2"},{"comment":"The Hal-I metric counts as hallucination any object the model mentions and then denies in the recheck question when the object is present per ground truth. In a story-writing context, the model's \"No\" may refer to the specific fictional instance it generated, not to the category as a whole. For example, a story that places \"a few books\" on a windowsill can be followed by \"No, there is no book visible in the image\" because the imagined books are not the image books, even if another book appears in the image. The recheck question is underspecified relative to the generative task, and no control experiments are provided to show that \"No\" responses correspond to failures to recognize the object rather than to intentional reference to an imagined instance. Since IH is the numerator of Hal-I, this ambiguity directly threatens the central claim that most LVLMs hallucinate heavily during volunta","section":"Sec. 3.2, Eq. (2); Table 1"},{"comment":"All reported metrics are point estimates without sample sizes, confidence intervals, or significance tests. Differences of a few percentage points are used to support substantive conclusions (e.g., the claim that Hal-I remains \"largely unchanged\" under mitigation or that models rank differently). The paper should report per-task numbers of images and objects, bootstrap intervals across images, and multiple decoding seeds for open models; API-based models should be evaluated with temperature settings reported.","section":"Tables 1–3"},{"comment":"The second key finding, that existing hallucination-mitigation methods are ineffective in voluntary imagination tasks, is supported only for LLaVA1.5 with greedy decoding on this benchmark. VCD, OPERA, HALC, and VTI are decoding-time interventions whose behavior can depend on the base model; one base model is too narrow to support the general statement in the abstract and conclusion. The scope should either be restricted to LLaVA1.5 or the experiments extended to at least one additional model. The large Hal-I increase for VTI also needs explanation rather than being grouped under \"unchanged or even increases.\"","section":"Sec. 4.7, Table 3"},{"comment":"The claim that expressive tendency can be controlled \"without impacting the hallucination rate\" is not fully supported by the reported numbers. The delta notation in Table 2 is not defined, and the deltas are not negligible: e.g., for Qwen2.5-VL, Hal-I changes by +5.1 points at alpha=-1 relative to alpha=0, and Exp changes by -14.1 points. This is a secondary finding, but the current presentation overstates the invariance.","section":"Sec. 4.4, Table 2"}],"minor_comments":[{"comment":"The text says \"open-source models, such as Gemini2.0\", but Gemini2.0 is a closed-source model. This is a factual error.","section":"Abstract / Sec. 1"},{"comment":"The captioning task is described as requiring \"factual descriptions without voluntary imagination\", yet Table 1 reports positive Exp values for captioning (e.g., LLaVA1.5 13.6). The text should clarify that Exp measures actual behavior, not the task's intended level of imagination.","section":"Sec. 4.1 / Table 1"},{"comment":"A weighted kappa of 0.45 between GPT-4o and human annotators is moderate agreement; the statement that this is \"sufficient to demonstrate a clear tendency\" is overstrong. Report confidence intervals and treat the relevance assessment as approximate.","section":"Appendix E, Fig. 13"},{"comment":"The attention-weight calculation is described only in prose. Please specify the layer, head, aggregation over tokens, and how objects are aligned with token spans, so the analysis is reproducible.","section":"Sec. 4.6, Fig. 9"},{"comment":"The related-work section mentions NOPE but does not discuss the relationship between NOPE and the proposed presence-evaluation design, despite the similarity in the acronym and task. A brief comparison would help position the contribution.","section":"Sec. 2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real gap and the basic idea is likely worth publishing after substantial revision. The major issues are about evaluation validity rather than presentation: the missing object-extraction and ground-truth details and the ambiguity of the presence recheck are load-bearing for the headline claim. I would not reject on the basis of the current framing, but the authors need to add methodological detail and targeted control experiments before the numbers can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea of VOPE is worth engaging with: instead of penalizing every object not in the image, ask the model whether it believes its own mentioned objects are present, and score based on agreement with the image. The Hal-D/Hal-I split and the derivation in Eq. (4) showing CHAIRi as a special case are clean and genuinely new. The three-task benchmark is also sensibly built.\n\nBut the central empirical claim—that models hallucinate heavily during voluntary imagination (high Hal-I)—does not hold up under scrutiny, because the recheck question is underspecified. When the model writes 'a few books' in a story and is later asked 'Is there a book in the image?', a 'No' may mean 'the books I imagined are not visible' even if the image contains a bookshelf. The model is not necessarily failing to recognize the object; it may be answering a different question. The paper does not disambiguate this, and the captioning Hal-I numbers are a red flag: Qwen2.5-VL, a strong captioner, gets 43.6% Hal-I in the pure captioning task, where there is no voluntary imagination. That is far too high for a pure recognition failure and suggests the recheck is measuring something else, such as the model's tendency to treat its specific narrative instance as non-existent.\n\nThere are also reproducibility gaps. The object extraction method is never described, and the paper doesn't explain how ground-truth presence is assigned for objects that are not in COCO's 80 categories. This matters a lot: if the presence labels are COCO-only, the Hal-D numbers could be inflated and Hal-I deflated, or vice versa depending on the direction. No error bars or significance tests are reported, so the mitigation-methods conclusion rests on point estimates with small deltas (e.g., Hal-I going from 17.0 to 20.6 for VCD).\n\nThe relevance assessment is a nice add-on, but the GPT-4o/human weighted kappa is 0.45, which is moderate and lower than the human-human 0.62. They call that sufficient, but it is a weak basis for the claim that the judge is reliable.\n\nWho is this for? Anyone building benchmarks for creative LVLM evaluation. The conceptual framework is worth stealing, and the paper is clearly written. But the numbers should not be quoted until the recheck prompt and ground-truth pipeline are fixed and validated with control experiments. I'd send it to peer review, but I'd expect major revision.\n\nRecommendation: engage with the idea, but treat the measurements as preliminary. A serious referee should ask for: (1) a controlled study of recheck question phrasing (e.g., 'Is there any X visible in the image?' vs 'Is the particular X you mentioned visible?'), (2) full release of the extraction and ground-truth code, and (3) error bars or significance tests. If they provide those, the benchmark could be a standard tool.","headline":"VOPE has a genuinely useful idea—separating imagination from hallucination by rechecking the model's own presence judgments—but the headline numbers are not trustworthy until the recheck question is pinned down and the object-extraction/ground-truth pipeline is public.","tokens_in":16912,"tokens_out":5751,"would_cite":true,"duration_ms":49533,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model hallucinates when it misjudges the presence of the objects it mentions, not when it invents them.","keywords":["hallucination evaluation","vision-language models","voluntary imagination","presence checking","recheck-based evaluation","object grounding","story generation","LVLM"],"falsifier":"Give the VOPE recheck to human storytellers on the same set of images; if humans also produce high Hal-I scores, then the metric is systematically mislabeling valid imagination as hallucination, collapsing the paper's central measure.","tokens_in":16117,"feed_emoji":"🤖","tokens_out":5489,"duration_ms":53652,"temperature":0.7,"pith_summary":"The paper argues that in tasks where a model is invited to imagine—like story writing—inventing an object that is not in the image is not, by itself, a hallucination. A hallucination occurs when the model's own judgment about the presence of the objects it mentions is wrong: it asserts an absent object is present, or denies one that is present. To measure this, the paper introduces VOPE, a recheck in which the model answers presence questions about every object it just generated. Across mainstream models, the paper finds that factual descriptions are often accurate, but models frequently misjudge the very objects they made up—yielding high hallucination rates in voluntary imagination. Existing mitigation methods targeting factual hallucinations barely change these rates, pointing to a distinct failure mode.","feed_headline":"Most vision-language models misjudge the objects they imagine","feed_subtitle":"A recheck-based benchmark shows hallucination rates in story writing stay high even when factual description looks clean.","key_machinery":"The recheck-based presence evaluation. After a model generates text, each mentioned object is queried with 'Is there a X in the image?' and the model's answers sort the output into factual description (object claimed present) and voluntary imagination (object claimed absent). Ground-truth labels then classify each as true or hallucinated, producing Hal-D = |D_H|/(|D_T|+|D_H|) and Hal-I = |I_H|/(|I_T|+|I_H|). The key move is using the model's own interpretation as the boundary between description and imagination, so invented content is not penalized unless the model is wrong about it.","core_discovery":"On its own terms, the paper's central claim is that hallucination in generative vision-language tasks should be defined by the consistency between a model's presence judgments and the image, not by overlap between the generated text and a list of visible objects. VOPE extracts the objects a model mentions, rechecks each with a presence question, and splits the output into factual description (model says present) and voluntary imagination (model says absent). Comparing these judgments against ground-truth labels yields two rates: Hal-D for factual description and Hal-I for voluntary imagination. The reported empirical finding is that most models have low Hal-D but high Hal-I, and that the mod","pith_inferences":["If Hal-I is a true signal, then story-length outputs from current models are not reliably grounded in the visual scene, which matters for any application where the narrative should respect image content (e.g., children's books from a single picture).","The recheck design is gameable: a model that systematically denies every object's presence will appear to have perfect true imagination, so the metric should be paired with a separate check on whether the model's denial matches its initial mention.","The benchmark's dependence on a predefined 80-class taxonomy may be too coarse; extending to a larger or open-vocabulary detection system could change both Hal-I magnitudes and model rankings.","A direct test: run the same presence recheck on human storytellers; if human Hal-I is not near zero, the metric is biased against imagination."],"forward_implications":["A model can look strong on factual captioning benchmarks while being unreliable about the objects it invents in open-ended writing.","Counting every object that is not in the image as a hallucination (the CHAIR-style approach) is invalid for imaginative tasks and can rank better models as worse.","Hallucination mitigation methods developed for factual description do not transfer; none of the tested methods reduced Hal-I, and some increased it.","The proportion of imagined content (Exp) can be adjusted via contrastive decoding without changing hallucination rates, suggesting expressive tendency and grounding are separable."],"fun_headline_variants":["Vision-language models misjudge objects they imagine","New benchmark exposes hallucination in imagined content","Imagination tasks trigger more vision-language hallucinations","Presence tests reveal hidden hallucination in vision-language models","Vision-language models fail to ground their imagined objects"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark treats the dataset's predefined object list as the complete ground truth for what exists in an image, and trusts the model's yes/no presence answer as a faithful statement of what it believes.","fun_headline_variants_meta":{"raw":{"variants":["Vision-language models misjudge objects they imagine","New benchmark exposes hallucination in imagined content","Imagination tasks trigger more vision-language hallucinations","Presence tests reveal hidden hallucination in vision-language models","Vision-language models fail to ground their imagined objects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1057,"prompt_tokens":745,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":489,"tokens_out":312,"duration_ms":4036,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T21:46:42.691144+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the VOPE recheck to human storytellers on the same set of images; if humans also produce high Hal-I scores, then the metric is systematically mislabeling valid imagination as hallucination, collapsing the paper's central measure.","supporting_citations":[],"review_version":1}