{"id":"80bb5764-4aa9-4c3a-9025-47949264f0e3","arxiv_id":"2505.24007","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper reports lower hallucination scores when selecting the best of three filtered image variants, but the selection uses the ground truth, so the improvement is an artifact of choosing the minimum.","lead":"This paper tries to reduce visual hallucinations in language models by feeding the model either the original, noise-filtered, or edge-enhanced version of an image and then picking the answer that scores best on a hallucination check. The reported 44.3% improvement is essentially guaranteed by the selection rule, because the method always chooses the best of three scores by looking at the ground truth.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ensemble reduction is an oracle artifact: NLI scores are computed against ground truth, so selecting the lowest score requires the answer key, making the 44.3% reduction non-deployable.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing flaw: the ensemble uses ground truth to select the best variant, so the reported improvement is not attainable in practice. My analysis of the paper's methodology (§3.4, §4.2) confirms this: NLI scores are computed against the ground truth answer, and the ensemble takes the minimum. This invalidates the central claim of adaptive, input-level hallucination reduction. The concrete test proposed would settle whether any fixed, learnable routing rule can reproduce a meaningful fraction of the 44.3% reduction; if it cannot, the paper's contribution reduces to a post-hoc selection bound. No other concern is more fundamental: the misuse of SelfCheckGPT is related but secondary, and the absence of baselines, error bars, and code would not by itself overturn the central claim if the routing were deployable. Therefore, the reader's REJECT verdict is justified, and my read does not change it.","tokens_in":13876,"tokens_out":2951,"duration_ms":31511,"concrete_test":"Run a held-out evaluation with a fixed, ground-truth-free routing rule: split the 1000 samples into training/test halves. On the training half, learn a per-question-type preference (e.g., choose NR for object-identification and color questions, and org for quantity questions, based on Table 2's win counts) or fit a simple classifier on image statistics to predict the winning variant. Apply this fixed rule to the test half, generating one response per sample and computing NLI scores only for evaluation (not for selection). If the average NLI reduction over the original-image baseline falls substantially below 44.3%—or becomes negative—the ensemble claim is confirmed as an oracle artifact rather than a deployable method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract; §4.2) is that an ensemble 'adaptively selects' the best input variant—original, noise-reduced, or edge-enhanced—based on the type of question. However, §3.4 states that the NLI score for each variant is obtained by comparing the generated response to the ground truth answer, and §4.2 defines the ensemble as choosing the variant with the lowest NLI score. This is an oracle selection rule: in deployment, ground truth answers are unavailable, so the reported 44.3% reduction (0.307 → 0.171) is simply the minimum of the three per-sample scores, not the performance of any implementable routing mechanism. The abstract's promised question-type-adaptive selection is never implemented; Table 2 only tabulates which variant happened to win per category. Consequently, the claim that input-level preprocessing alone reduces hallucination is unsupported. Additionally, the scoring protocol itself diverges from SelfCheckGPT, which compares multiple sampled responses for self-consistency, not against ground truth; here the ground-truth comparison is a supervised oracle, not a black-box consistency check. Without a deployable selection rule, the headline result is an artifact of the evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an input-level preprocessing framework that reduces visual hallucination in multimodal LLMs by generating three variants of each input image—original, noise-reduced (median filter), and edge-enhanced (Laplacian)—and then selecting among them. Responses are generated by GPT-3.5 for each variant on 1000 HaloQuest samples, and a 'SelfCheckGPT NLI' score is computed for each generated response. The authors report average NLI scores of 0.334 (edge-enhanced), 0.307 (original), 0.301 (noise-reduced), and 0.171 (ensemble), claiming a 44.3% reduction because the ensemble selects the variant with the lowest NLI score. They also report per-category winner counts and provide qualitative examples.","tokens_in":14058,"tokens_out":7158,"duration_ms":56571,"significance":"If the headline claim were valid, it would be a notable practical result: cheap input transformations could reduce hallucination in multimodal LLMs without retraining or architectural changes. The paper has some credible elements: it evaluates on 1000 samples from the public HaloQuest benchmark, uses simple and transparent filters, provides qualitative examples, and explicitly acknowledges that validation on one benchmark and one model family is a limitation. However, the central contribution is not established. The ensembling rule uses ground-truth answers to choose the best variant, and the NLI protocol is a supervised comparison to the answer key rather than the cited SelfCheckGPT self-consistency check. The 44.3% reduction is therefore a minimum-of-three artifact rather than evidence for input-level preprocessing as a deployable mitigation. This is a correctness problem, not a presentation issue.","major_comments":[{"comment":"The ensemble selection rule is an oracle and cannot be deployed. §3.4 states that each generated response is 'evaluated against the ground truth answer' to obtain an NLI score, and §4.2 defines ensembling as selecting 'the optimal NLI score among the three' (original, noise-reduced, edge-enhanced) variants. In real use the ground-truth answer is unavailable, so the lowest-NLI variant cannot be identified. The reported drop from 0.307 to 0.171 is therefore the average of the per-sample minima of the three scores, which is mathematically no larger than the average of any single fixed variant; it is not the performance of an implementable routing mechanism. The abstract's claim of question-type-adaptive selection is not realized by any rule: Table 2 only reports, after the fact, how often each variant happened to win. This is the load-bearing issue: the headline 44.3% reduction is an artifact of the evaluation protocol.","section":"§3.4, §4.2"},{"comment":"The NLI protocol is not SelfCheckGPT as defined by the cited work. SelfCheckGPT (Ref. [41]) measures hallucination by sampling multiple responses to the same prompt and checking self-consistency; it is a zero-resource, black-box method that does not use reference answers. In §3.3.2 the equations are written in that framework (premise Sn, hypothesis ri), but §3.4 explicitly says the NLI score is obtained by comparing the LLM-generated response to the ground-truth answer. This replaces the intended self-consistency check with supervised oracle scoring, so the paper's claim that the method is 'zero-resource, black-box' and requires no external reference data is inaccurate. The reported hallucination rates are not comparable to standard SelfCheckGPT scores.","section":"§3.3.2, §3.4"},{"comment":"The per-category statistics are internally inconsistent. §3.1 reports 593 object-identification, 415 quantity, and 289 color questions and notes that the categories overlap because some 'what' questions are counted in both object-identification and color. Table 2 nevertheless presents these as three separate columns with fixed totals and counts such as 150, 111, and 75 in one row, as if they were disjoint. Moreover, the first three rows of the 'All Ques' column sum to 861 rather than 1000, with no explanation for the missing 139 cases. These inconsistencies affect the paper's secondary claim that preprocessing benefits differ by question type.","section":"Table 2, §3.1"},{"comment":"The model used for image-conditioned response generation is not clearly identified and appears to be text-only. The paper repeatedly states that images were presented to 'GPT-3.5' (e.g., §3.2: 'we used OpenAI's GPT-3.5 model for its image processing compatibility'; §3.4: 'the original image and its corresponding question were presented to GPT-3.5'). GPT-3.5 is not a vision-capable model. If a different model or API (e.g., a vision variant) was used, it must be named; without this, the reported image-conditioned responses cannot be reproduced or interpreted.","section":"§3.2, §3.4"}],"minor_comments":[{"comment":"There are many typos in the qualitative tables, including 'N LIorg = 0.0.998', 'N LIN R= 0.265', and 'Holoquest' in the Table 1 caption; these should be corrected.","section":"Tables 3–4"},{"comment":"Figure 4 plots 1000 NLI scores as lines and is likely unreadable; consider showing aggregate statistics or a small multiple by category.","section":"Figure 4"},{"comment":"The description of the consistency score in §3.4 ('proportion of supporting (entailment) versus contradicting or neutral responses') is inconsistent with Eq. (5), which ignores the neutral class and normalizes over entailment and contradiction only.","section":"Eq. (5), §3.4"},{"comment":"The paper does not specify how the 'first 1000 samples' were selected from HaloQuest or whether they are a random subset; this matters because the dataset's difficulty is not uniform.","section":"§3.1"},{"comment":"Reference [43] contains a typo ('media filtering' should be 'median filtering') and several references use placeholder arXiv IDs (e.g., [28], [29], [32], [34], [37]), making them hard to verify.","section":"References"}],"recommendation":"reject","confidential_remarks":"The oracle-selection problem is conclusive and, combined with the SelfCheckGPT misapplication, means the headline claim is not supported. The category-count inconsistencies and unresolved model-identity issue reinforce this. I do not see a revision path that preserves the paper's central claim; rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result doesn't survive contact with the method. The ensemble 'reduction' is just taking the per-sample minimum of three NLI scores, and those NLI scores are computed by comparing generated answers to ground truth answers (Section 3.4). So the 44.3% improvement is a mathematical consequence of the selection rule, not an empirical finding about input preprocessing. There is no deployable routing mechanism; the abstract's promise of question-type-adaptive selection is never implemented, and Table 2 only tabulates which variant happened to win.\n\nCredit where it's due: the paper is clearly written, and it does run a systematic comparison of two classical filters (median, Laplacian) on 1000 HaloQuest samples, breaking results down by question type. The observation that noise reduction helps most on color questions is a plausible descriptive result, and the qualitative examples are illustrative. That part is fine as a small exploratory study.\n\nThe soft spots are load-bearing. First, the SelfCheckGPT citation is wrong: SelfCheckGPT compares multiple sampled responses for self-consistency, not a generated answer against a ground-truth answer. What the authors actually use is a supervised NLI oracle, which is a different thing. Second, they say they feed images to GPT-3.5, which is text-only; there is no description of a visual front-end. Third, no error bars, no baselines, no comparison to prior hallucination-mitigation work, and no code or data release. The category counts are inconsistent (593+415+289=1297 vs. 1000), though a footnote explains the overlap.\n\nWho is this for? Possibly someone curious about whether classical preprocessing shifts model responses, but they would need to redo the experiment with a proper evaluation protocol and a deployable selection rule. As is, the central claim is unsupported, and the paper is not a serious candidate for publication in its current form. I would desk reject it rather than send to reviewers; the flaw is fundamental and would require a redesign, not a revision.","headline":"The headline 44.3% reduction is an oracle artifact: the ensemble picks the variant with the lowest NLI score, and NLI scores are computed against ground truth, so the result is circular and non-deployable.","tokens_in":599,"tokens_out":714,"would_cite":false,"duration_ms":27402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that adaptively choosing among three preprocessed versions of an input image can cut measured visual hallucination rates in a multimodal language model by 44.3%, with no retraining and no architectural changes.","keywords":["visual hallucination","multimodal large language models","input-level preprocessing","ensemble filtering","noise reduction","edge enhancement","SelfCheckGPT","HaloQuest dataset"],"falsifier":"Run the same three-variant pipeline on a held-out subset of HaloQuest where the image version is selected without access to the ground-truth answer, using question category, image statistics, or a learned router, and compare average NLI scores: if the router's score is close to 0.171 the selection is real, and if it stays near the best single variant (about 0.301) the reported gain depends on oracle access to the correct answer.","tokens_in":13654,"feed_emoji":"🖼️","tokens_out":6264,"duration_ms":61040,"temperature":0.7,"pith_summary":"The paper argues that visual hallucinations in multimodal language models can be reduced at the input stage, before the model ever sees the image. It reports a 44.3% reduction in hallucination rates, measured by Natural Language Inference (NLI) scores, on 1000 visually challenging samples from the HaloQuest dataset using GPT-3.5. The method compares three image variants—original, noise-reduced, and edge-enhanced—and selects the variant whose answer best matches the ground-truth answer. The claim matters because, if true, it offers a lightweight, model-agnostic way to improve factual grounding without fine-tuning or post-hoc correction.","feed_headline":"Pre-filtering images cuts hallucination scores by 44.3 percent","feed_subtitle":"An ensemble of original, noise-reduced, and edge-enhanced images reduces hallucinated answers without retraining the model.","key_machinery":"The ensemble of three input variants, scored by SelfCheckGPT-NLI. The variants are the raw image, a noise-reduced version produced by a 15x15 median filter, and an edge-enhanced version produced by a channel-wise Laplacian operator with weighted blending (α=1.5, β=-0.5). SelfCheckGPT-NLI uses a DeBERTa-v3-large model to assign contradiction probabilities between the generated answer and the reference answer, and the ensemble selects the variant with the lowest average contradiction score.","core_discovery":"The central claim is that an ensemble of preprocessing filters can substantially reduce hallucination in multimodal LLM responses. The pipeline applies a median filter for noise reduction and a Laplacian operator for edge enhancement, generating three image variants—original, noise-reduced, and edge-enhanced—and feeds each to GPT-3.5. The answer from the variant with the lowest SelfCheckGPT NLI score is selected, yielding an average NLI score of 0.171 compared to 0.307 for the original image, a 44.3% decrease. The paper also finds that the best variant varies by question type and even within a type, so no single fixed filter consistently wins.","pith_inferences":["The reported 44.3% reduction is an upper bound: the ensemble chooses the variant whose answer best matches the ground-truth answer, so the gain is an oracle-selection result unless a deployable routing rule can predict the best variant before generating an answer.","A routing rule based only on the three coarse question categories (object identification, quantity, color) would likely underperform the oracle, since within each category the best variant varies from sample to sample.","The same ensemble could be extended with additional image transforms such as contrast normalization or deblurring, and evaluated on other benchmarks to test whether input-level conditioning transfers across models and domains.","The finding indirectly suggests that hallucination is partly a function of how the image is presented, not only of model competence, which may open a complementary axis for reliability improvements."],"forward_implications":["If the claim holds, visual hallucination can be reduced without accessing model internals or retraining.","The per-question-type analysis suggests that different visual degradations call for different input representations, so a single fixed preprocessing would miss gains.","Because the best variant varies within every question category, per-input selection is necessary rather than a single global filter.","The approach is lightweight and model-agnostic, so it could be combined with training-time alignment or post-hoc correction methods."],"supporting_citations":[{"why":"Supplies the SelfCheckGPT zero-resource, black-box NLI method used to measure hallucination.","marker":"[41]"},{"why":"Provides the HaloQuest dataset with the 1000 image-question-answer samples used in the evaluation.","marker":"[42]"},{"why":"Defines the median filtering technique used for the noise-reduced image variant.","marker":"[43]"},{"why":"Provides the Laplacian edge enhancement and unsharp-masking blending formulas used to build the edge-enhanced variant.","marker":"[44]"},{"why":"Supplies the DeBERTa-v3-large NLI model used to compute contradiction probabilities.","marker":"[47]"},{"why":"Provides the MNLI training data underlying the NLI contradiction scoring.","marker":"[46]"}],"fun_headline_variants":["Adaptive image filters cut LLM hallucination by 44.3%","No retraining: filter ensemble reduces hallucination 44.3%","Pick best filtered image: hallucination drops 44.3%","Adaptive filter selection cuts hallucination 44.3% without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the system can choose the best image version during real use, because the reported reduction is obtained by picking the version whose answer matches the ground-truth answer, which is not available when the model is deployed.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive image filters cut LLM hallucination by 44.3%","No retraining: filter ensemble reduces hallucination 44.3%","Pick best filtered image: hallucination drops 44.3%","Adaptive filter selection cuts hallucination 44.3% without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001255,"raw_usage":{"total_tokens":5116,"prompt_tokens":890,"completion_tokens":4226,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":4146}},"tokens_in":506,"tokens_out":4226,"duration_ms":28450,"temperature":1.0,"reasoning_tokens":4146,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:36:55.200229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-variant pipeline on a held-out subset of HaloQuest where the image version is selected without access to the ground-truth answer, using question category, image statistics, or a learned router, and compare average NLI scores: if the router's score is close to 0.171 the selection is real, and if it stays near the best single variant (about 0.301) the reported gain depends on oracle access to the correct answer.","supporting_citations":[{"cited_title":"Prentice- Hall, Inc., USA (2006)","cited_arxiv_id":null,"evidence_quote":"Provides the Laplacian edge enhancement and unsharp-masking blending formulas used to build the edge-enhanced variant."}],"review_version":1}