{"id":"b3def6ee-d468-4364-8f23-e2ad8fcd549a","arxiv_id":"2412.11124","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A training-free bottom-up reasoning framework that verifies scene graphs and commonsense with external tools reduces hallucinations in multimodal LLMs.","lead":"This paper introduces a six-step reasoning pipeline that reduces hallucination in multimodal language models by checking visual details with external tools and verifying commonsense claims with web searches. The method needs no retraining and reports accuracy gains on several hallucination benchmarks across four popular models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 7 lists identical Hal/Cog values for three different base models under +Ours; the AMBER evidence needs reproduction before the headline improvement is accepted.","rationale":"The reader's weakest_assumption was that the external verification tools and self-checks are reliable, with no reported accuracy or thresholds. I agree that this is a real gap, but the more load-bearing issue is a concrete internal inconsistency in Table 7: three different base models, when augmented with +Ours, yield exactly the same Hal (25.83) and Cog (34.69) values. Because the DEHALL pipeline re-prompts the base MLLM at every module, the final outputs should inherit model-specific behavior; identical two-decimal values across all three models are not a plausible outcome of genuine evaluation. This is not an accusation of fabrication; it is an apparent data error or a copy-paste artifact that must be resolved before the AMBER-based part of the central claim is accepted. The paper also lacks code release, statistical significance testing, and error bars, so the reader cannot distinguish a true improvement from reporting noise. I do not move to REJECT because Tables 1, 2, and 3 report large gains that may well be real, and the anomaly is isolated to one table. But the anomaly is exactly the kind of concrete, checkable defect that keeps the verdict at CONDITIONAL. My agreement with the reader is partial because the reader flagged the Table 7 inconsistency in the rationale but did not make it the weakest assumption; I treat it as the primary load-bearing concern. The proposed test, obtaining per-sample AMBER outputs and recomputing the two columns, would settle whether the concern lands.","tokens_in":21845,"tokens_out":4060,"duration_ms":38573,"concrete_test":"Request from the authors the per-sample AMBER predictions for all three base models with +Ours, together with the exact evaluation script and prompts. Recompute Hal and Cog for each model. If the three sets of outputs do not all reproduce exactly 25.83 and 34.69, Table 7 is erroneous; the AMBER rows must be corrected and the claimed cross-benchmark consistency re-assessed. If the identical values do reproduce, require an explanation for the coincidence and run a bootstrap significance test over the AMBER and POPE subsets to see whether the reported gains survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirically load-bearing point is the internal inconsistency in Table 7 (Appendix F.1). For LLaVA-1.5+Ours, Qwen-VL+Ours, and GPT-4V+Ours, the AMBER generative-task columns Hal and Cog are all exactly 25.83 and 34.69, while the unmodified baselines differ (36.4/4.2, 23.6/1.9, 30.7/2.6). Since every module of DEHALL prompts the same base MLLM f_theta, the corrected outputs should depend on which model is being augmented; identical values to two decimals are not explained by the experimental design. The paper does not release code, raw outputs, or error bars, so this cannot be checked independently. Because Table 7 is one of the six benchmarks used to support the claim of 'significant improvements', this anomaly puts part of the empirical foundation in question. A secondary but related gap is that Module 5 (Commonsense Verification) uses web-search snippets and the same MLLM to label claims as Hallucination or Non-hallucination, with no reported accuracy, no thresholds, and no failure-mode analysis; if that self-verification is biased, the cognition-level gains could overstate real de-hallucination. Both concerns are resolvable by releasing data and exact prompts, but currently they make the central claim conditional rather than established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DEHALL, a training-free six-module bottom-up reasoning framework for reducing hallucination in multimodal LLMs. The modules are: (1) target identification and partial scene graph generation by an MLLM, (2) verification of objects, attributes, and relations using Grounding DINO and BLIP, (3) validation and adjustment of the input question against the verified scene graph, (4) commonsense claim induction, (5) commonsense verification via web search and MLLM-based labeling, and (6) final answer generation. The authors evaluate on PhD, POPE, WHOOPS!, VQAv2-IDK, AMBER, and Hal-Eval across LLaVA-1.5, Qwen-VL-Chat, MiniGPT-v2, and GPT-4V, reporting substantial gains such as GPT-4V+Ours reaching 80.5 on PhD Avg. versus 70.5 for GPT-4V alone, and POPE F1 of 93.9 versus 82.0.","tokens_in":22039,"tokens_out":6434,"duration_ms":54186,"significance":"If the reported results are reproducible, the contribution is significant: it broadens hallucination mitigation beyond object-level corrections to relation and attribute verification, introduces the under-studied problem of input-question conflicts, and combines perception-level verification with cognition-level commonsense checking in a training-free pipeline. The ablation in Table 4 and the case studies in Figures 5 and 7 are instructive, and the use of off-the-shelf tools makes the approach practical. However, the empirical foundation is currently weakened by an apparent data anomaly in Table 7, by the absence of error bars and significance tests, and by unreported thresholds and code that prevent independent reproduction. The idea is therefore promising but the evidence is not yet established at the level claimed.","major_comments":[{"comment":"The +Ours rows for LLaVA-1.5, Qwen-VL, and GPT-4V report identical generative-task values (Hal=25.83, Cog=34.69), while the baselines are different (36.4/4.2, 23.6/1.9, 30.7/2.6). Since every DEHALL module prompts the same base MLLM f_theta, the corrected outputs should depend on the base model; identical values to two decimal places are not explainable by the experimental design and indicate a transcription or processing error. Because AMBER is one of the six benchmarks supporting the headline claim of significant improvements, the generative results in Table 7 must be corrected or re-run before the claim can be accepted.","section":"Appendix F.1, Table 7"},{"comment":"The framework's behavior depends on several hyperparameters that are never specified: the Grounding DINO detection confidence threshold used to keep or remove objects and attributes, the BLIP similarity cutoff for relation verification, the number of in-context examples for question validation (Figure 4 varies this without giving the value used in the main experiments), the number of top web results in Module 5, and the confidence-score threshold appearing in the step-by-step example in Figure 8. Without these values, the method cannot be reproduced, and the comparison with single-pass baselines is not apples-to-apples because DEHALL receives substantially more inference compute and external tool access. Please report all thresholds and prompts, and ideally release code and exact prompts.","section":"Section 3, Modules 2 and 5; Appendix D"},{"comment":"In Commonsense Verification, the same MLLM f_theta that produced the commonsense claims also labels them as Hallucination or Non-hallucination based on web-search snippets. The paper reports no accuracy of this self-verification step, no threshold for accepting or rejecting a claim, and no failure-mode analysis. If the model's verification is systematically biased, the cognition-level gains (e.g., VQAv2-IDK +11.35 in Table 3) could overstate the true de-hallucination effect. Please provide a human- or metric-based evaluation of the verification step and quantify how often the verification changes the final answer.","section":"Section 3, Eq. (4)"},{"comment":"No error bars, standard deviations, or significance tests are reported for any result. Some improvements are small (e.g., Table 1: LLaVA-1.5 OR +1.6, MiniGPT-V2 OR +1.5), and without variance estimates the stated 'significant improvements' are not statistically supported. Please report multiple runs or significance tests for the main tables, and state the number of test samples in each benchmark split.","section":"Tables 1, 3, 4, 7"}],"minor_comments":[{"comment":"The 47.8% statistic is presented as a general property of user input texts, but Table 6 gives per-dataset proportions (70.15%, 31.75%, 41.50%) whose unweighted average is 47.8%; please report the aggregation method and avoid the overgeneralized phrasing.","section":"Appendix C.3"},{"comment":"The two values per model row for each PhD subtask are not clearly labeled in the rendered table; please make the Neu./Mis. split explicit in the column headers.","section":"Table 1"},{"comment":"The y-axis labels 'BLIP Score' and 'Delta Word across models' are not defined in the caption or text; please define the metrics and state the in-context example count used in the main experiments.","section":"Figure 4"},{"comment":"The bracket in 'Y = f_theta(I, [P6; ...' is missing a closing bracket, and the notation '[; ]' is defined only after Eq. (1).","section":"Eq. (5)"},{"comment":"The text says 'five Multimodal Large Language Models' but then lists four (LLaVA-1.5, MiniGPT-v2, Qwen-VL, GPT-4V); please correct the count.","section":"Section 4, Datasets and Baselines"},{"comment":"The table reports an 'AMBER Score' column as a single number with a delta in parentheses, but the preceding columns already include Acc/Pre/Rec/F1; please clarify how the AMBER Score is computed from these components.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The Table 7 anomaly is serious enough that I would want to see corrected numbers or a re-run before publication. The paper currently lacks code, exact prompts, thresholds, and raw outputs, so the authors should be asked to provide at least the full prompt set and the AMBER generative evaluation details. The novelty and scope are appropriate for the venue, but the empirical foundation needs repair before the central claim can be relied upon."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a sensible, training-free de-hallucination pipeline with real potential, and I think the main results are probably right—but there's one data anomaly that needs correction before I'd trust the numbers fully.\n\nThe contribution is a six-module reasoning chain: partial scene graph extraction, external verification with Grounding DINO and BLIP, question rectification via in-context learning, commonsense claim induction and web-search verification, and final answering. The question-rectification module is the most distinctive piece, and the combination itself is new. The paper does a good job of ablating the modules: Table 4 shows each one contributes, with the perception modules having the biggest impact. The writing is clear, with full prompts and examples.\n\nThe soft spot is real. Appendix Table 7 reports identical Hal and Cog values (25.83 and 34.69) for LLaVA-1.5, Qwen-VL, and GPT-4V under +Ours, even though their baselines differ widely. That cannot be explained by the experimental design—each module prompts the same base model, so outputs should differ. This is almost certainly a reporting error, but it sits in one of the six benchmark tables used to support the 'significant improvements' claim, so it has to be fixed. Also missing: error bars, significance tests, released code, and exact thresholds for the verification tools. The self-verification of commonsense claims by the same MLLM, using web snippets, is a fair concern; it's not circular in a fitting sense, but its accuracy is unreported, so the cognition-level gains might be optimistic. The 'first to highlight input-text hallucinations' claim is also a bit strong; earlier work has noted misleading questions, though not in exactly this framing.\n\nNone of this kills the paper. The architecture is plausible, the ablations are supportive, and the limitations section is honest. This deserves a serious referee, but the authors should be asked for a corrected Table 7, statistical reporting, and an artifact release before the central claim is accepted.\n\nRecommendation: engage with it. Send it to review, but with a clear request for repairs.","headline":"A solid training-free de-hallucination pipeline with strong reported gains, but a suspicious data anomaly in Table 7 and missing reproducibility details make the evidence conditional until cleaned up.","tokens_in":22654,"tokens_out":2635,"would_cite":true,"duration_ms":23812,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free six-module pipeline reduces multimodal LLM hallucination by verifying the scene graph, correcting conflicting questions, and checking commonsense claims against web evidence before answering.","keywords":["multimodal large language models","hallucination mitigation","scene graph","commonsense verification","chain-of-thought reasoning","visual question answering","input question conflict","training-free framework"],"falsifier":"Take a labeled set of image-question pairs with ground-truth scene graphs and ground-truth commonsense claims; run the verification modules in isolation and measure whether the object/attribute detector and the relation scorer delete true scene elements or keep false ones, and whether the model's Hallucination/Non-hallucination labels agree with human labels. Low precision or recall in these checks would indicate the gains come from prompt decomposition rather than from genuine verification, and this can be checked directly.","tokens_in":21571,"feed_emoji":"🧠","tokens_out":8991,"duration_ms":69272,"temperature":0.7,"pith_summary":"The paper argues that multimodal LLM hallucination cannot be cured by perception-level fixes alone, because two sources of error are routinely ignored: hallucinated premises in the user's own question, and missing commonsense knowledge needed for cognition-level answers. It proposes DEHALL, a training-free six-module reasoning pipeline that works bottom-up: extract a question-relevant scene graph, verify its objects, attributes, and relations with external tools, check the question against that verified scene and rewrite it if they conflict, induce the commonsense claims an answer needs, verify those claims against web search results, and only then answer. On six benchmarks the pipeline raises base-model performance consistently; for GPT-4V, PhD average accuracy goes from 70.5 to 80.5 and POPE F1 from 82.0 to 93.9. The intended lesson is that reliable answers require grounding at every layer—pixels, relationships, question wording, and commonsense—before the model speaks.","feed_headline":"Six-step pipeline cuts multimodal AI hallucination","feed_subtitle":"A training-free pipeline verifies the scene, repairs misleading questions, and checks commonsense before answering.","key_machinery":"The load-bearing object is the verified partial scene graph $\\{O, A, R\\}$—objects, attributes, and relationships—plus the six-module reasoning chain built around it. Module 1 prompts the MLLM to extract only the scene elements relevant to the question; Module 2 checks objects and attribute phrases with an open-set detector and relationships with an image-text similarity scorer, deleting low-fidelity entries; Module 3 uses in-context examples to detect and rewrite object, attribute, and relation conflicts between question and scene; Module 4 has the model state the commonsense claims a cognition-level answer requires; Module 5 labels each claim Hallucination or Non-hallucination using web-search snippets; Module 6 answers from verified scene plus verified claims. The partial scene graph is what turns a fuzzy image into checkable evidence, and the question-validation step is what stops a wrong premise in the prompt from propagating into the answer.","core_discovery":"The central discovery is that a holistic 'perception-then-cognition' decomposition removes hallucinations that either perception-only or output-only corrections miss. The paper identifies three under-addressed causes: shallow visual understanding, which it addresses with a partial scene graph; conflicting input text, which it estimates triggers up to 47.8% of hallucination cases and which it repairs by minimal question rewriting; and cognition-level errors, which it estimates account for about 51% of hard query problems and which it addresses by web-verifying commonsense claims. DEHALL sequences these fixes so that each reasoning stage consumes only verified evidence: the scene graph is checked by open-set detection and image-text similarity scoring, the rewritten question inherits that verification, and the final answer combines verified perception with verified commonsense. The paper reports gains across four MLLMs and six benchmarks, with the largest gains on misleading-question and commonsense-reasoning subsets.","pith_inferences":["The 47.8% and 51% figures come from the paper's own statistics on its chosen benchmarks; a fair test would re-measure them on diverse real-user VQA logs, since the pipeline's value depends on how often faulty questions actually occur.","The question-validation module is the most portable piece: it could be applied standalone to any instruction-following MLLM as a premise-checking front-end, even without scene graphs, though its accuracy would then depend on the model's own visual judgment.","One implied but untested boundary is that the method's ceiling is set by the external detectors and the model's self-verification on unusual or abstract images; the paper's own failure case of an ambiguous chimpanzee image points there.","A natural extension is to apply the same perception-verification-question-rectification loop to video or audio-grounded LLMs, where relationship and commonsense errors are likely even more frequent than in still images."],"forward_implications":["A user can wrap an existing MLLM in DEHALL with no training and get large hallucination reductions, e.g., GPT-4V POPE F1 rises from 82.0 to 93.9, which matters for applications where model weights cannot be fine-tuned.","Correcting the input question is itself a large part of the gain: the module changes on average about two words per conflict while keeping semantic similarity high, so premise errors in VQA prompts can be caught before they reach the answer.","Verifying scene graphs addresses relationship hallucinations, a type that object-detection-only methods overlook; the paper's ablation shows removing the visual perception and verification modules causes the biggest drops.","Commonsense verification via web retrieval lets the framework handle questions that cannot be answered from pixels alone, such as sentiment or 'unanswerable' cases; on VQAv2-IDK the total IDK score rises by 11–15 points depending on the base model.","As a CoT-style decomposition, DEHALL outperforms other CoT variants on hallucination benchmarks, suggesting that the verification steps—not the chain structure alone—are what reduce hallucinations."],"supporting_citations":[{"why":"Grounding DINO is the open-set object detector used in Module 2 to verify objects and attribute phrases in the scene graph.","marker":"Liu et al. 2023b"},{"why":"BLIP supplies the image–text similarity for verifying relationship triplets and for measuring semantic similarity of adjusted questions.","marker":"Li et al. 2022b"},{"why":"The PhD benchmark provides the perception-level accuracy numbers, including misleading questions, reported in Table 1.","marker":"Liu et al. 2024c"},{"why":"POPE provides the object-hallucination F1 evaluation that reports the largest perception-level gain for GPT-4V.","marker":"Li et al. 2023"},{"why":"WHOOPS! supplies the compositional VQA and explanation-generation tasks used to test commonsense-level hallucinations.","marker":"Guetta et al. 2023"},{"why":"VQAv2-IDK supplies the unanswerable and ambiguous question benchmark that measures cognition-level hallucination improvements.","marker":"Cha et al. 2024"},{"why":"Woodpecker is the post-hoc output-correction baseline that DEHALL is compared against on hallucination benchmarks.","marker":"Yin et al. 2023"},{"why":"HiKER-SGG is the closed-set scene-graph specialist used to benchmark the quality of MLLM-generated scene graphs.","marker":"Zhang et al. 2024"}],"fun_headline_variants":["Bottom-up reasoning trims LLM hallucinations in vision","Perception-first pipeline curbs multimodal hallucination","Fixing visual and textual inputs reduces AI falsehoods","Holistic check verifies scene, question, commonsense","New framework slashes MLLM hallucination on six benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the external detectors and the model's own web-search-based labeling are accurate enough that a 'verified' scene graph and claim set are actually true; the paper reports neither the verification steps' error rates nor the labeling accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Bottom-up reasoning trims LLM hallucinations in vision","Perception-first pipeline curbs multimodal hallucination","Fixing visual and textual inputs reduces AI falsehoods","Holistic check verifies scene, question, commonsense","New framework slashes MLLM hallucination on six benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000342,"raw_usage":{"total_tokens":1882,"prompt_tokens":944,"completion_tokens":938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":859}},"tokens_in":560,"tokens_out":938,"duration_ms":6321,"temperature":1.0,"reasoning_tokens":859,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:16:13.249432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a labeled set of image-question pairs with ground-truth scene graphs and ground-truth commonsense claims; run the verification modules in isolation and measure whether the object/attribute detector and the relation scorer delete true scene elements or keep false ones, and whether the model's Hallucination/Non-hallucination labels agree with human labels. Low precision or recall in these checks would indicate the gains come from prompt decomposition rather than from genuine verification, and this can be checked directly.","supporting_citations":[],"review_version":1}