{"id":"43f548ff-6fdf-4940-bdc4-0ffab34d6212","arxiv_id":"2502.01056","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"IFCD suppresses object and attribute hallucinations in large vision-language models by subtracting an internally edited, hallucination-amplified output distribution from the model's normal predictions.","lead":"This paper introduces a decoding trick, IFCD, that reduces hallucinations in image captioning models by comparing the model's normal output with a deliberately biased version of itself. If the reported gains are real, it offers a low-cost, inference-time fix for a well-known reliability problem in vision-language models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The direction assumption behind Eq. 5 is contradicted by the paper's own ablation: positive editing alone worsens hallucinations on InstructBLIP, and \"w/o POS\" (negative editing + original) beats full IFCD, so p- is not established as a hallucination distribution.","rationale":"The reader's weakest assumption identifies the internal representation edit direction as the most fragile premise, and the paper's own ablations and figure reinforce that concern. Table 4 provides a direct, quantitative contradiction: positive editing alone on InstructBLIP performs worse than regular decoding, and the \"w/o POS\" variant outperforms the full method. This is not merely an issue of reporting; it undermines the mechanistic interpretation of Eq. 5. If the direction of the edit is not consistently truthfulness-modulating, then subtracting p- from p+ is not a fact-based operation. Figure 3's numbers, while harder to parse, point in the same direction. Secondary problems—the abstract's 9%/8% gains not matching Table 1, hyperparameters tuned on evaluation benchmarks, and unreleased artifacts—further reduce confidence but are not needed for the rejection. The method may still produce useful empirical reductions in hallucination on some benchmarks, and the paper does show competitive results against VCD and ICD, but the central claim as stated is not supported by the evidence presented. The correct disposition is therefore to maintain the reader's REJECT verdict: the approach is plausible but the current write-up does not establish that the proposed mechanism, rather than a generic contrastive perturbation, is responsible for the observed improvements.","tokens_in":22476,"tokens_out":3858,"duration_ms":40104,"concrete_test":"On a held-out subset of POPE/MSCOCO (not used for tuning), generate responses under three decoding conditions: original, positive editing (+δ), and negative editing (−δ). For each question with a known ground-truth object set, measure whether the ground-truth token probability increases under positive editing and decreases under negative editing relative to the original. Compute the fraction of samples satisfying this assumed monotonicity. The central mechanism requires the fraction to be substantially above chance for both backbones; if it is not, the claim that p- is a hallucination distribution is unsupported. Also recompute Table 4 with the sign labels for δ swapped, and check whether the \"w/o POS\" advantage on InstructBLIP persists under the swapped convention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that IFCD calibrates outputs by subtracting hallucination logits from p- in Eq. 5. This requires negative editing (+δ) to produce a distribution that is reliably more hallucinated than the original, and positive editing (−δ) to produce a distribution that is reliably less hallucinated. The paper's own evidence contradicts this assumption. In Table 4, \"EDITING\" (positive editing alone) on InstructBLIP yields CHAIRs=57, CHAIRi=15, which is worse than Regular decoding (48, 13.9); positive editing therefore amplifies hallucinations rather than suppressing them. Moreover, \"IFCD w/o POS\" (contrasting negative editing with the original distribution) achieves 28 and 7.6, which is better than full IFCD (39.6, 11.2), so the component identified as the method's core mechanism is not the one producing the best result on that backbone. Figure 3 likewise shows the \"Hallucinated Editing\" panel with logits (3.2, 14.5, 2.4) favoring \"Black\" over \"Red\" for black strawberries, the opposite of the caption's claim that editing amplifies the language prior toward conventional colors. If p- is not demonstrated to be a hallucination-amplified distribution, Eq. 5 reduces to an arbitrary linear combination of two perturbed distributions, and the observed gains could stem from generic contrastive effects or benchmark-specific tuning rather than from fact-based internal editing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Internal Fact-based Contrastive Decoding (IFCD), an inference-time method for reducing object hallucinations in LVLMs. IFCD trains a TruthX-style probe to identify a truthful/untruthful editing direction in hidden representations, then forms two distributions p+ and p− by editing internal representations in opposite directions and decodes with Eq. (5): p_IFCD = σ((1+α)p+ − αp−). The authors evaluate on POPE, MME, MSCOCO captioning, and LLaVA-Bench with LLaVA 1.5 and InstructBLIP, reporting gains over regular decoding and the VCD/ICD baselines.","tokens_in":22760,"tokens_out":5699,"duration_ms":51903,"significance":"Hallucination mitigation at inference time with a small trained probe is a practically important and timely goal, and the paper has the right experimental skeleton: two backbone LVLMs, several benchmarks, and an ablation study. The central mechanistic claim, however, depends on the direction δ in Eq. (3) being a reliable hallucination amplifier; the paper's own Table 4 and Figure 3 contradict that premise. Because the advertised gains are also obtained under hyperparameters selected on the evaluation benchmarks themselves (Appendix A, Section 5), the paper does not currently establish that IFCD works through 'internal facts' rather than through generic contrastive effects or benchmark-specific tuning.","major_comments":[{"comment":"The ablation undermines the central premise of Eq. (5). The method assumes that negative editing produces a hallucination-amplified distribution p− whose subtraction removes hallucinatory logits. On InstructBLIP, however, 'EDITING' (positive editing alone) yields CHAIRs/CHAIRi = 57/15, which is worse than Regular decoding (48/13.9), so positive editing does not act as an anti-hallucination operation in this backbone. Moreover, 'IFCD w/o POS' (contrasting negative editing with the original distribution) achieves 28/7.6, better than full IFCD (39.6/11.2), so the component identified as the core contrast is also not the one responsible for the best result. The table therefore does not establish that p− is a hallucination distribution or that Eq. (5) calibrates outputs according to internal facts.","section":"Section 5, Table 4"},{"comment":"Figure 3 contradicts its own caption. For black strawberries, the panel labeled 'Hallucinated Editing' reports logits Red 3.2, Black 14.5, Gray 2.4, which still favor 'Black' over 'Red'; the caption instead claims that editing amplifies the language prior toward the conventional color 'red'. Since this figure is the direct evidence that internal representation editing amplifies hallucination, the direction assumption behind Eqs. (3)–(5) is not supported by the displayed numbers.","section":"Figure 3"},{"comment":"The headline results are not independent predictions because the hyperparameters are selected on the evaluation benchmarks. Appendix A selects the TruthX training size (300) by POPE accuracy on the MSCOCO subset, and Section 5 selects editing strength (0.5), the number of edited layers (15), and contrast strength α (0.1) through CHAIR-style sweeps on MSCOCO (Figures 6 and 7). The same POPE and CHAIR/MSCOCO settings are then reported as the main results in Tables 1 and 3. The paper needs a separate validation split or a sensitivity analysis showing that the gains are not artifacts of this selection.","section":"Appendix A and Section 5"},{"comment":"The abstract claims an average 9% accuracy improvement on POPE, but Table 1 does not contain an average gain of 9 points: the largest per-setting gain over 'Regular' is 7.44 points (InstructBLIP, GQA adversarial), and many gains are between 1.6 and 7.3 points. If the '9%' refers to a different statistic (for example, a relative improvement or a subset of settings), the text should state that explicitly rather than presenting it as the average POPE accuracy gain.","section":"Abstract and Table 1"}],"minor_comments":[{"comment":"The sentence 'As shown in the left part of Figure 5' appears to refer to the training-size comparison, but Figure 5 is the max-token CHAIR experiment; the reference should point to Table 5.","section":"Appendix A"},{"comment":"There are several typos: 'Regrading' should be 'Regarding', 'DIRTURBANCE' should be 'DISTURBANCE' (also in Figure 3), and Table 3's caption says 'resutls' instead of 'results'.","section":"Appendix B"},{"comment":"No standard errors, multiple seeds, or significance tests are reported for the POPE/CHAIR numbers. Given that temperatures are set to 1, the generation is stochastic and the reported differences between close configurations may be within noise.","section":"Section 4.1 and Tables 1–3"},{"comment":"The sign convention for editing is confusing: Eq. (3) defines δ = H_pos_truth − H_neg_truth, while Figure 2 labels −δ as 'Neg Editing hallucinated' and +δ as 'Pos Editing anti-hallucinated'; these two conventions should be reconciled in one place.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"Rejection is on internal-consistency grounds rather than on novelty or fit. The central direction assumption is contradicted by the paper's own ablation and figure, and the evaluation is affected by benchmark-based hyperparameter selection. If the authors can reframe the method as a generic contrastive-decoding variant and demonstrate out-of-sample gains on held-out validation, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe genuinely new piece here is using TruthX to edit hidden states in two directions and subtracting the 'hallucinated' distribution from the 'anti-hallucinated' one. That is a real extension of VCD/ICD, and the full IFCD does improve over plain decoding on most POPE rows and on CHAIR for LLaVA 1.5. The probe is cheap and the method is model-agnostic.\n\nThe problems start with the abstract: it claims an average 9% accuracy gain on POPE, but Table 1's largest gain is 7.44 points and typical gains are 4-6. That is not a small discrepancy. Second, the key hyperparameters — TruthX training size (300), editing strength (0.5), number of layers (15), and contrast strength (0.1) — are selected on the evaluation benchmarks (POPE and CHAIR on MSCOCO). So the reported numbers include tuning on the test sets.\n\nThe most serious issue is that the central mechanism is contradicted by the paper's own ablation. In Table 4, 'EDITING' — decoding with positive editing alone — is worse than regular decoding on InstructBLIP (CHAIRs 57 vs 48). The variant without the positive branch ('IFCD w/o POS') beats the full IFCD. If positive editing is supposed to be the anti-hallucinated distribution, the data say it amplifies hallucinations on that backbone. Figure 3 is also inconsistent: the 'hallucinated editing' panel shows higher logits for the correct color, not the language prior. So the directional claim behind Eq. 5 is not supported.\n\nI don't think the method is necessarily a dud; the gains could come from generic contrastive effects or from the specific tuning. But as written, the interpretation is unsupported and the headline numbers are inaccurate. No code is released.\n\nThis paper deserves a serious referee because the idea is novel and potentially useful, but the referee would need to demand held-out hyperparameter selection, corrected claims, a re-examination of the editing direction, and code. I would not cite it in its current form. Bring it to reading group if you want a good example of why ablation tables matter.","headline":"Clever internal-editing contrastive decoding, but the paper's own ablation contradicts the mechanism and the headline numbers don't match the tables.","tokens_in":23325,"tokens_out":3417,"would_cite":false,"duration_ms":32956,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Editing a vision-language model's internal representations to amplify its own hallucinations, then subtracting those hallucinatory logits, reduces object and attribute hallucination at inference time.","keywords":["large vision-language models","object hallucination","contrastive decoding","internal representation editing","truthfulness probe","POPE benchmark","MME benchmark","inference-time mitigation"],"falsifier":"Measure token-level probabilities from p^+ and p^- on a diverse set of images with known ground-truth objects: if p^- does not assign systematically higher probability than p^+ to hallucinated objects (the very tokens that should be suppressed), the contrastive subtraction cannot be the cause of the reported gains. A simpler check would be to run the reported ablation 'w/o POS' (negative editing contrasted against the original, unedited distribution) across all POPE and MME settings: on InstructBLIP's CHAIR_i the paper already reports it beating full IFCD, so if that pattern holds across benchmarks the positive direction is not doing the claimed work.","tokens_in":22218,"feed_emoji":"🧠","tokens_out":7082,"duration_ms":63452,"temperature":0.7,"pith_summary":"This paper tries to establish that a vision-language model's own hallucinations can be used against themselves: by editing the model's internal representations to deliberately amplify its hallucinatory tendencies, one can obtain a distribution of 'hallucinatory logits' that, when subtracted from the normal output, removes false objects and attributes at inference time. The method, called Internal Fact-based Contrastive Decoding (IFCD), requires only a small trained probe and no fine-tuning of the base model or external knowledge sources. On the POPE and MME object-hallucination benchmarks the authors report average accuracy gains of about 9% and 8% over direct decoding across two open models, LLaVA 1.5 and InstructBLIP. If correct, this offers a cheap, model-agnostic route to reducing hallucinations in deployed vision-language systems.","feed_headline":"Editing then cancelling the model's own hallucinations lifts accuracy 9%","feed_subtitle":"Inference-time fix needs only a small trained probe, no fine-tuning or external knowledge.","key_machinery":"The central mechanism is a learned truthfulness-direction edit combined with contrastive decoding. TruthX is an autoencoder with two multi-layer-perceptron encoders, TruthEnc and SemEnc, and a decoder; it maps an LVLM's internal hidden states into separate truthfulness and semantic latent spaces, reconstructs them, and the difference between reconstructed and original states gives an editing vector $\\Delta$. Editing the hidden state as $\\hat{x} = x + \\gamma \\Delta$ with positive $\\gamma$ is claimed to make the model's output more truthful, while negative $\\gamma$ amplifies language-prior hallucinations such as calling black strawberries red. Contrastive decoding then subtracts the hallucinated distribution from the positive one using the formula above, and an adaptive plausibility constraint restricts the penalty to tokens whose normal probability is above a fraction $\\beta$ of the maximum, keeping the contrast from distorting already-correct tokens.","core_discovery":"IFCD builds two token distributions from the same LVLM: a positive distribution $p^+$ obtained by editing the model's internal representations in a 'truthful' direction, and a negative distribution $p^-$ obtained by editing in the opposite, hallucination-amplifying direction. The final decoding distribution is computed as $p_{\\text{IFCD}}(y_t) = \\sigma((1+\\alpha)p^+(y_t) - \\alpha p^-(y_t))$, so tokens that are boosted by the hallucinated distribution are penalised. The editing direction comes from an autoencoder, TruthX, trained with contrastive learning on a few hundred image-caption pairs to separate truthful from untruthful internal states; the difference between the average truthful and untruthful latent positions defines the editing vector $\\delta$, and the edited representation is $\\hat{x} = x + \\gamma \\Delta$. The paper's experiments claim this reduces object-existence and attribute hallucinations on POPE, MME, and long-form captioning (CHAIR), and that the method degrades less than baselines when POPE's sampling becomes adversarial.","pith_inferences":["Because the method is model-agnostic and the probe is small, the same two-model contrastive decoding recipe could transfer to other modalities, such as editing internal states of audio or video-language models to amplify modality-specific priors.","The reported ablation that contrasting the original distribution with the negative-edited distribution alone ('w/o POS') matches or exceeds the full IFCD on some CHAIR scores suggests that the positive editing direction may not be strictly necessary; a focused study of when the positive edit helps would clarify the mechanism.","The performance peak at 300 training samples, with declines at larger sizes, hints that the truthfulness direction is learned from a narrow data distribution; testing on diverse domains (medical, scientific imagery) would reveal whether the direction generalises.","One could test the method's central premise directly by measuring, at the token level, whether $p^-$ assigns higher probability to known hallucinated objects than $p^+$; such a probe would make the internal mechanism falsifiable without a full benchmark run."],"forward_implications":["IFCD can be applied to any open-source LVLM with a small trained probe, requiring no fine-tuning of the base model and no external knowledge sources.","The method reduces both object-existence and attribute-level hallucinations, with reported average accuracy gains of 9% on POPE and 8% on the MME object-hallucination subset across LLaVA 1.5 and InstructBLIP.","In long-form captioning on MSCOCO, IFCD lowers the proportion of hallucinated objects (CHAIR_i) and hallucinated sentences (CHAIR_s) compared with direct decoding, while keeping BLEU roughly unchanged.","IFCD's accuracy drop from POPE's random to adversarial sampling settings is smaller than that of direct decoding, VCD, and ICD, suggesting it is more robust to statistical bias.","Training the TruthX probe requires only about 300 image-caption pairs; larger training sets did not improve POPE accuracy in the reported experiments."],"supporting_citations":[{"why":"Supplies TruthX, the contrastively trained autoencoder whose truthfulness latent direction defines the editing vector delta.","marker":"(Zhang et al., 2024)"},{"why":"Supplies the contrastive decoding framework and the adaptive plausibility constraint that IFCD adapts for vision-language models.","marker":"(Li et al., 2023c)"},{"why":"The Visual Contrastive Decoding baseline that IFCD compares against and whose visual-disturbance approach it contrasts with internal editing.","marker":"(Leng et al., 2024)"},{"why":"The Instruction Contrastive Decoding baseline using prompt-based disturbance, reproduced for comparison.","marker":"(Wang et al., 2024)"},{"why":"POPE, the polling-based object-probing benchmark used for the main hallucination accuracy numbers.","marker":"(Li et al., 2023d)"},{"why":"MME, the perception benchmark whose object-level and attribute-level subsets measure hallucination gains.","marker":"(Fu et al., 2023)"},{"why":"CHAIR, the caption-hallucination metric used to evaluate long-form generation.","marker":"(Rohrbach et al., 2018)"},{"why":"LLaVA 1.5, one of the two LVLM backbones on which IFCD is evaluated.","marker":"(Liu et al., 2024)"}],"fun_headline_variants":["Cancelling the model's own hallucination signals boosts accuracy 9%","Inference-time trick: amplify then subtract the model's hallucination bias","No fine-tuning: internal contrastive decoding cuts LVLM hallucinations","Self-correcting at inference: LVLMs cancel their own hallucinated tokens","From biased logits to truthful output: internal contrastive decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that editing a model's internal representation along the learned 'truthfulness' direction reliably makes its outputs more truthful, and editing in the opposite direction reliably makes them more hallucinated, so that the subtracted distribution p^- really consists of hallucinatory logits rather than random noise.","fun_headline_variants_meta":{"raw":{"variants":["Cancelling the model's own hallucination signals boosts accuracy 9%","Inference-time trick: amplify then subtract the model's hallucination bias","No fine-tuning: internal contrastive decoding cuts LVLM hallucinations","Self-correcting at inference: LVLMs cancel their own hallucinated tokens","From biased logits to truthful output: internal contrastive decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2778,"prompt_tokens":973,"completion_tokens":1805,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1711}},"tokens_in":589,"tokens_out":1805,"duration_ms":12059,"temperature":1.0,"reasoning_tokens":1711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T16:45:26.212881+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure token-level probabilities from p^+ and p^- on a diverse set of images with known ground-truth objects: if p^- does not assign systematically higher probability than p^+ to hallucinated objects (the very tokens that should be suppressed), the contrastive subtraction cannot be the cause of the reported gains. A simpler check would be to run the reported ablation 'w/o POS' (negative editing contrasted against the original, unedited distribution) across all POPE and MME settings: on InstructBLIP's CHAIR_i the paper already reports it beating full IFCD, so if that pattern holds across benchmarks the positive direction is not doing the claimed work.","supporting_citations":[],"review_version":1}