{"id":"6b3b507f-069e-41c8-9109-6c6efcd737b6","arxiv_id":"2505.21547","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CGC+VTD identifies co-occurring image token clusters as a source of hallucinated objects in discrete-token LVLMs and suppresses clusters' absent-token signals in latent space, cutting hallucination rates across Chameleon, Janus-Pro, and Emu3.","lead":"The paper proposes a two-step method that reduces hallucination in large vision-language models using discrete image tokens, by clustering tokens that co-occur and then subtracting hidden state contributions of absent, clustered tokens during generation. The approach beats existing mitigation methods on three benchmarks and three models while being faster and lighter.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Sec. 3.2 HitRate evidence is same-data confounded: the same COCO panoptic masks build CGC clusters and score token–object associations, so the dominant-cluster absent-token signal may be manufactured by the clustering objective rather than a genuine cause of hallucination.","rationale":"The reader's CONDITIONAL verdict already flags VTD under-specification and mentions the same-data issue; I agree with the verdict but locate the single most load-bearing weakness in the HitRate analysis rather than in Eq. 4. The VTD concern is genuine and should be fixed by documenting how g(l)(vhal) is computed for absent tokens, and the released code can settle it quickly. The same-data confound is more serious because it undermines the foundational claim that hallucination is closely linked to visual priors: the correlation that motivates VTD may be an artifact of building clusters and scoring associations on the same COCO masks. The empirical mitigation results, POPE/GQA transfer, and ablations are real evidence that the method reduces hallucination in the tested settings, so I would not reject the paper; however, without a non-confounded diagnostic or a mechanistic test, the causal story remains conditional. Re-running the HitRate analysis on disjoint segmentation data is the minimal check that would settle this.","tokens_in":22643,"tokens_out":8199,"duration_ms":81196,"concrete_test":"Rebuild CGC using only COCO 2017 Panoptic train images, then recompute the Sec. 3.2 HitRate@K curves (C1/C2/C3, K=1..10) using token–object associations counted on COCO 2017 Panoptic val images only, or stronger, on ADE20K masks, for all three models. If the reported 5–10 point C2 advantage over C1/C3 persists on fully disjoint data, the visual-prior diagnostic survives; if it shrinks to noise, the CGC+VTD motivation is an artifact of using the same masks for clustering and scoring.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 3.2 states that the segmentation masks used to score token–object associations are obtained from the COCO 2017 Panoptic dataset, which is also the dataset used for CGC. This is a same-data confound. CGC constructs graph edges from co-occurrence inside 3×3 neighborhoods or inside the same panoptic segment, and then optimizes clusters so that tokens co-occurring within those same masks are pulled together. Consequently, an absent token that belongs to the dominant cluster is, by construction, a token whose COCO-mask object associations match the objects that co-occur with the present tokens. Hallucinated objects in AMBER are precisely absent objects that co-occur with present ones, so the high HitRate for C2 may simply re-discover the graph-construction objective. The K-means comparison does not remove this confound, because K-means still operates on the same token statistics and is scored on the same masks. Thus the paper's central finding that hallucinations are closely linked to visual priors evoking absent tokens from dominant clusters is not yet established by the reported correlation. The separate VTD implementation issue (Eq. 4 does not define g(l)(vhal) for an absent token) is real but can be settled by code inspection; the same-data confound requires a new experiment and directly attacks the causal claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies object hallucination in LVLMs that use discrete image tokenizers. It hypothesizes that co-occurrence statistics among image tokens create visual priors, causing models to evoke visually absent tokens from dominant clusters. To test this, the authors propose Context-Guided Clustering (CGC), which builds a co-occurrence graph over codebook tokens using COCO panoptic segmentation and 3x3 spatial neighborhoods, learns graph embeddings with a GNN and contrastive loss, and clusters the tokens. They then propose Visual Token Decontamination (VTD), which during decoding subtracts a projection of the hidden-state representation of absent dominant-cluster tokens from the hidden states of present image tokens. Experiments on Chameleon-7B, Janus-Pro-7B, and Emu3-13B across AMBER, Object HalBench, MME, and POPE report consistent hallucination reductions with error bars, a favorable efficiency comparison, and gains when combined with OPERA and other baselines.","tokens_in":22964,"tokens_out":5562,"duration_ms":52533,"significance":"If the causal claim is established, the paper identifies a new, tokenizer-specific source of object hallucination and offers a lightweight, training-free mitigation that is complementary to existing decoding-time methods. Strong points include the breadth of evaluation (three discrete-token LVLMs, multiple benchmarks, error bars), the ablations of CGC design choices, the cross-dataset POPE results, the efficiency analysis, and the commitment to release code. However, the diagnostic evidence for the central claim is weakened by a same-data confound, and the core VTD operation is underspecified. The contribution is promising and potentially publishable, but the central claim needs additional validation before it can be accepted as stated.","major_comments":[{"comment":"The HitRate analysis is confounded by using the same COCO 2017 Panoptic segmentation source both to construct the CGC co-occurrence graph and to score token-object associations. In Sec. 3.2 the paper states that 'the segmentation masks are obtained from the COCO 2017 Panoptic dataset, which we also use for CGC,' while Sec. 3.1 and Algorithm 1 define graph edges precisely as token pairs that co-occur inside 3x3 grid cells or within the same panoptic mask. Since the contrastive objective pulls together tokens that co-occur inside those same masks, a high HitRate for absent tokens from dominant clusters may partly re-discover the graph-construction objective rather than independently demonstrate a visual prior that causes hallucination. The K-means baseline in Figure 3 (middle) does not remove this confound because it operates on the same token statistics and is scored on the same masks. To support the load-bearing claim that hallucinations are 'closely linked to visual priors that can evoke absent tokens from dominant clusters,' please provide a held-out validation: for example, train CGC on one segmentation source and score token-object associations on a different dataset or on a strictly disjoint split, and also report what happens when the co-occurrence graph is constructed from permuted or random co-occurrence statistics.","section":"Sec. 3.2 and Sec. 3.1"},{"comment":"VTD is underspecified in a way that blocks reproducibility. Equation (4) subtracts a projection of g(l)(vhal) from g(l)(vi), but vhal is by definition absent from the input token sequence, so g(l)(vhal) is not defined by the LVLM forward pass. The text says VTD 'projects these tokens into the model's latent space' but does not state how the hidden state of an absent token is obtained: is it computed by a separate forward pass with vhal inserted into the sequence, by taking the codebook embedding and projecting it to layer l, or by some other mapping? In addition, if multiple absent tokens are identified, the paper does not specify whether the subtraction is applied sequentially, summed over all vhal, or otherwise aggregated. Please specify the computation precisely and include an ablation that replaces g(l)(vhal) with a neutral baseline, such as the raw codebook embedding or the hidden state of a random absent token, to verify that the method acts on hallucination-related signal rather than on general visual content.","section":"Sec. 3.3, Eq. (4)"},{"comment":"The HitRate@K evidence is reported only as aggregate curves over 1,000 AMBER images, without the number of hallucinated objects per group, confidence intervals, or a statistical test of the 5-10 percentage-point gap between C2 and C1/C3. Given the same-data confound in the first comment, the aggregated comparison is not yet sufficient to establish that absent tokens in dominant clusters are 'more strongly associated with hallucinated objects.' Please report per-image variance, significance tests, and, ideally, a breakdown across object categories to show that the effect is not driven by a few frequent hallucinated objects.","section":"Sec. 3.2, Figure 3"},{"comment":"The combination experiments are presented with only CHAIR-s and CHAIR-i on Object HalBench, and the appendix tables contain apparent typographical errors and inconsistent signs (for example, Table 13 lists '1966' for PROJECTAWAY+CHAIR-s, and the VCD+ row in Table 12 shows a positive CHAIR-s change but a negative CHAIR-i change relative to the baseline). These issues make it difficult to assess the claim that CGC+VTD 'enhances the performance of prior methods in most cases.' Please correct the typos, report full evaluation sets with error bars, and clarify whether the combination results are averaged over multiple runs.","section":"Sec. 4.2, Tables 2 and 12-13"}],"minor_comments":[{"comment":"The heading 'Mitgate Hallucination via Visual Token Decontamination' contains a typo and should read 'Mitigate Hallucination via Visual Token Decontamination.'","section":"Sec. 3.3 heading"},{"comment":"The caption reads 'We report CHIAR score for generative tasks' and should read 'CHAIR score.'","section":"Figure 5 caption"},{"comment":"The value '1966' in the PROJECTAWAY+ row of Table 13 appears to be a typo for '19.66'; please correct it.","section":"Appendix B, Table 13"},{"comment":"The limitation section states 'we observe a slight drop in coverage scores (Cover ↑ in Table 1) with our proposed method,' but in Table 1 CGC+VTD's Cover is higher than Nucleus Sampling on all three models. Please clarify the intended comparison (e.g., relative to contrastive-decoding baselines) or correct the statement.","section":"Sec. 6"},{"comment":"The sentence 'We discuss how to tune other other performance-related hyperparameters' contains a duplicated word; please remove the second 'other.'","section":"Appendix B"},{"comment":"The HitRate@K metric refers to 'top-K(C)' but does not formally specify how the object ranking for a token group is obtained from the segmentation masks; please define the association score and the ranking procedure.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical study with a plausible and novel mechanism, and the benchmark results are encouraging. However, the same-data confound in the Sec. 3.2 HitRate analysis goes to the heart of the causal claim, and the VTD mechanism is insufficiently specified for reproduction. I would like to see a held-out validation of the HitRate result and a precise specification of how absent-token hidden states are computed before this paper is accepted. The code release will help, but the manuscript itself should state these details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the short version: this paper makes a real contribution by connecting discrete image token co-occurrence to hallucination in LVLMs, and it ships a lightweight, retraining-free fix that consistently beats baselines on Chameleon, Emu3, and Janus-Pro. I'd send it out, but I'd want the authors to tighten two things before I'd sign off on the causal story.\n\nWhat's new: CGC is the first attempt I know of to cluster discrete image tokens by spatial and semantic co-occurrence (building a graph from a segmentation dataset) and then use that structure to predict and suppress hallucinations. The paper also documents that contrastive decoding misbehaves on discrete-token models, which is useful for the community. The empirical work is solid: error bars, ablations, efficiency numbers, and cross-method combination all point the same direction. Releasing code helps.\n\nThe soft spots, in order of severity. First, the HitRate diagnostic in Sec. 3.2 uses the same COCO 2017 Panoptic masks to build the co-occurrence graph and to score token–object associations. That creates a partial confound: the clusters are constructed to pull together tokens that co-occur in those masks, so the high association of absent tokens with hallucinated objects is partly a restatement of the clustering objective. It's not fully circular, because the hallucinated objects come from AMBER, not COCO, but the token–object scoring is on the same masks that defined the graph. A cross-dataset validation—build clusters on COCO, score associations on a different segmentation dataset, or vice versa—would settle it. Second, VTD's Eq. (4) subtracts a projection of g(l)(vhal) for a token absent from the input, but the paper never says how that hidden state is computed. That's a reproducibility gap; code inspection may resolve it, but it needs to be stated. Third, the hyperparameters are tuned on an AMBER subset, which makes the AMBER numbers slightly optimistic; the fact that Object HalBench and MME also improve mitigates that concern.\n\nOverall, the central claim—that absent tokens from dominant co-occurrence clusters drive a meaningful share of hallucination in discrete-token LVLMs—is plausible and supported by consistent benchmark gains, but not yet proven to the level the paper claims. The method itself is useful regardless of the exact mechanism. I'd accept this for peer review and ask for the VTD specification and a cross-dataset check on the HitRate.","headline":"A plausible, well-evaluated first step on token co-occurrence and hallucination in discrete-token LVLMs, with two fixable gaps in the causal evidence.","tokens_in":23468,"tokens_out":3572,"would_cite":true,"duration_ms":32535,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hallucinations in discrete-tokenizer LVLMs are driven by visual priors from image-token co-occurrence, and can be mitigated by subtracting the hidden states of absent dominant-cluster tokens during generation.","keywords":["object hallucination","discrete image tokenizer","visual priors","token co-occurrence","graph neural network","latent editing","vision-language models","hallucination mitigation"],"falsifier":"Take a fixed image set and model, and compare three editing targets: (i) the absent tokens from the dominant cluster that CGC identifies, (ii) an equal number of randomly chosen absent tokens matched for frequency, and (iii) the present tokens from the dominant cluster. If targets (ii) or (iii) reduce CHAIR as much as (i), the hallucination drop is not specific to the co-occurrence-prior mechanism, and VTD is likely removing generic directional content.","tokens_in":22429,"feed_emoji":"🖼️","tokens_out":9158,"duration_ms":66779,"temperature":0.7,"pith_summary":"The paper tries to establish that a large share of object hallucination in vision-language models built on discrete image tokens is a statistical echo: tokens that often co-occur in training images form visual priors, and when a cluster of such tokens dominates an image, the model tends to mention objects tied to cluster tokens that are NOT in the image. The authors test this by building a co-occurrence graph from a panoptic segmentation dataset, embedding it with a graph neural network, and clustering the tokens; hallucinated objects are most strongly correlated with absent tokens from the most dominant clusters. They then propose Visual Token Decontamination, a latent-space edit that subtracts the weighted projection of these absent tokens' hidden states from the hidden states of all present image tokens during generation. On three discrete-tokenizer models, the edit reduces hallucination on standard benchmarks while preserving or improving general perception scores. If correct, the result offers a training-free, mechanistic explanation for a common failure mode and a cheap fix that composes with language-side methods.","feed_headline":"Absent image tokens drive hallucination in discrete-token LVLMs","feed_subtitle":"Co-occurrence clusters expose which missing tokens cause false objects; subtracting their latent signal cuts errors.","key_machinery":"Two components carry the argument: Context-Guided Clustering (CGC) and Visual Token Decontamination (VTD). CGC treats each codebook entry as a node in a co-occurrence graph, weights edges by spatial proximity (tokens within a $3\\times 3$ grid cell) and semantic coherence (tokens inside the same panoptic segment), keeps the top 10% of edges, and trains a two-layer GNN with a contrastive InfoNCE-style objective plus a positive-pair similarity loss; the resulting node embeddings are K-means clustered into groups of tokens that frequently appear together. VTD uses those clusters at inference: it identifies the most dominant clusters for an input image, finds the tokens in those clusters that are absent from the image, and applies the update $g^{(l)}(v_i) := g^{(l)}(v_i) - \\gamma \\cdot \\frac{\\hat{g}^{(l)}(v_i) \\cdot \\hat{g}^{(l)}(v_{\\text{hal}})}{\\|\\hat{g}^{(l)}(v_{\\text{hal}})\\|_2^2} \\cdot g^{(l)}(v_{\\text{hal}})$ to each present image token's hidden state, removing the absent token's direction from the representation before the model continues generating.","core_discovery":"The paper's central claim is that hallucinations in LVLMs with discrete image tokenizers are closely linked to visual priors encoded in token co-occurrence, and specifically that absent tokens from dominant clusters are the carriers of hallucination. Context-Guided Clustering groups codebook tokens by spatial and semantic co-occurrence, and the resulting dominant clusters of an input image contain tokens that are absent from the image yet strongly associated with hallucinated objects, as measured by HitRate@K. Visual Token Decontamination then suppresses those absent tokens during autoregressive decoding: at a chosen transformer layer, for each such token $v_{\\text{hal}}$ the hidden state $g^{(l)}(v_{\\text{hal}})$ is projected onto the direction of each present image token's hidden state and subtracted with a scaling coefficient $\\gamma$, so that the co-occurrence-driven prior is removed from the representations the language model reads. The paper reports that this reduces hallucination across Chameleon-7B, Janus-Pro-7B, and Emu3-13B on AMBER, Object HalBench, and POPE, while keeping or improving MME perception scores.","pith_inferences":["One consequence the authors leave implicit: if the co-occurrence-prior mechanism is right, hallucination should also be reducible at the source by retraining or masking the tokenizer to break dominant-cluster statistics; the paper does not attempt that.","A natural next experiment the paper does not run: apply the same latent-editing recipe to continuous-feature LVLMs by treating clusters of patch embeddings as pseudo-tokens.","The HitRate analysis could serve as a cheap pre-screening tool: run CGC on a new model and check whether a small sample's hallucinated objects are predominantly top-associated objects of absent dominant-cluster tokens.","A control the paper does not report would sharpen the causal claim: subtracting a random absent token's direction, matched for frequency, should not produce the same CHAIR drop as subtracting the identified absent dominant-cluster tokens."],"forward_implications":["Because CGC runs once per model and VTD only edits hidden states during decoding, the method adds no training step and is faster and lighter than contrastive decoding or two-pass revision baselines.","Hallucination and coverage can be decoupled: VTD lowers CHAIR and Hal while leaving MME perception scores flat or higher, so the reduction is not simply censorship of object mentions.","VTD composes with language-side methods such as OPERA, and the combined gain exceeds either method alone, implying that visual-prior and language-prior hallucinations are partly independent.","The transfer of CGC clusters from COCO to GQA in POPE suggests the learned co-occurrence structure generalizes beyond the segmentation dataset used to build it.","On discrete-token models, contrastive-decoding baselines can inflate response length and even increase hallucination; VTD avoids perturbing the visual input, which the analysis shows is unstable in this paradigm."],"supporting_citations":[{"why":"Defines the VQ-VAE discrete tokenizer paradigm that the target models rely on.","marker":"[23]"},{"why":"Provides the VQGAN tokenizer used by Chameleon and the codebook compression rationale.","marker":"[24]"},{"why":"Chameleon is one of the three evaluated LVLMs with a discrete image tokenizer.","marker":"[7]"},{"why":"Janus-Pro is the primary analysis model and a central evaluation target.","marker":"[13]"},{"why":"Emu3 is the third evaluated discrete-tokenizer LVLM.","marker":"[8]"},{"why":"COCO panoptic segmentation supplies the object-level regions and masks used to build the co-occurrence graph.","marker":"[26]"},{"why":"InfoNCE-style contrastive objective trains the GNN node embeddings that encode co-occurrence structure.","marker":"[28]"},{"why":"PROJECTAWAY supplies the projection-subtraction latent editing idea that VTD adapts in Eq. 4.","marker":"[6]"},{"why":"AMBER provides the benchmark and ground-truth annotations for the hallucination analysis and main evaluation.","marker":"[29]"},{"why":"Object HalBench is the second hallucination benchmark measuring response- and instance-level hallucination rates.","marker":"[30]"}],"fun_headline_variants":["Stop absent tokens from hallucinating LVLMs","Latent editing erases false objects in token-based vision LLMs","Curb hallucination by subtracting missing token signals","Find and quash hallucination clusters in image tokenizers","Token co-occurrence graph reveals and fixes LVLM hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a token that is not in the image still has a useful 'mental direction' inside the model, and that subtracting that direction removes false objects without also erasing real ones.","fun_headline_variants_meta":{"raw":{"variants":["Stop absent tokens from hallucinating LVLMs","Latent editing erases false objects in token-based vision LLMs","Curb hallucination by subtracting missing token signals","Find and quash hallucination clusters in image tokenizers","Token co-occurrence graph reveals and fixes LVLM hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1384,"prompt_tokens":1021,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":637,"tokens_out":363,"duration_ms":2930,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:23:19.556412+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed image set and model, and compare three editing targets: (i) the absent tokens from the dominant cluster that CGC identifies, (ii) an equal number of randomly chosen absent tokens matched for frequency, and (iii) the present tokens from the dominant cluster. If targets (ii) or (iii) reduce CHAIR as much as (i), the hallucination drop is not specific to the co-occurrence-prior mechanism, and VTD is likely removing generic directional content.","supporting_citations":[{"cited_title":"Chameleon: Mixed-modal early-fusion foundation models, 2024","cited_arxiv_id":null,"evidence_quote":"Chameleon is one of the three evaluated LVLMs with a discrete image tokenizer."},{"cited_title":"Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation, 2024","cited_arxiv_id":null,"evidence_quote":"AMBER provides the benchmark and ground-truth annotations for the hallucination analysis and main evaluation."}],"review_version":1}