Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hallucinations in discrete-tokenizer LVLMs are driven by visual priors from image-token co-occurrence, and can be mitigated by subtracting the hidden states of absent dominant-cluster tokens during generation.

desk verdict A plausible, well-evaluated first step on token co-occurrence and hallucination in discrete-token LVLMs, with two fixable gaps in the causal evidence. read the letter →

arxiv 2505.21547 v1 pith:WKEPV2Y7 submitted 2025-05-24 cs.CV cs.AI

Weixing Wang , Zifeng Ding , Jindong Gu , Rui Cao , Christoph Meinel , Gerard de Melo , Haojin Yang This is my paper · ORCID
classification cs.CVcs.AI
keywords objecthallucinationdiscreteimagetokenizervisualpriorstokenco-occurrencegraphneuralnetworklatenteditingvision-languagemodelsmitigation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a large share of object hallucination in vision-language models built on discrete image tokens is a statistical echo: tokens that often co-occur in training images form visual priors, and when a cluster of such tokens dominates an image, the model tends to mention objects tied to cluster tokens that are NOT in the image. The authors test this by building a co-occurrence graph from a panoptic segmentation dataset, embedding it with a graph neural network, and clustering the tokens; hallucinated objects are most strongly correlated with absent tokens from the most dominant clusters. They then propose Visual Token Decontamination, a latent-space edit that subtracts the weighted projection of these absent tokens' hidden states from the hidden states of all present image tokens during generation. On three discrete-tokenizer models, the edit reduces hallucination on standard benchmarks while preserving or improving general perception scores. If correct, the result offers a training-free, mechanistic explanation for a common failure mode and a cheap fix that composes with language-side methods.

What carries the argument

Two components carry the argument: Context-Guided Clustering (CGC) and Visual Token Decontamination (VTD). CGC treats each codebook entry as a node in a co-occurrence graph, weights edges by spatial proximity (tokens within a $3\times 3$ grid cell) and semantic coherence (tokens inside the same panoptic segment), keeps the top 10% of edges, and trains a two-layer GNN with a contrastive InfoNCE-style objective plus a positive-pair similarity loss; the resulting node embeddings are K-means clustered into groups of tokens that frequently appear together. VTD uses those clusters at inference: it identifies the most dominant clusters for an input image, finds the tokens in those clusters that are absent from the image, and applies the update $g^{(l)}(v_i) := g^{(l)}(v_i) - \gamma \cdot \frac{\hat{g}^{(l)}(v_i) \cdot \hat{g}^{(l)}(v_{\text{hal}})}{\|\hat{g}^{(l)}(v_{\text{hal}})\|_2^2} \cdot g^{(l)}(v_{\text{hal}})$ to each present image token's hidden state, removing the absent token's direction from the representation before the model continues generating.

What would settle it

Take a fixed image set and model, and compare three editing targets: (i) the absent tokens from the dominant cluster that CGC identifies, (ii) an equal number of randomly chosen absent tokens matched for frequency, and (iii) the present tokens from the dominant cluster. If targets (ii) or (iii) reduce CHAIR as much as (i), the hallucination drop is not specific to the co-occurrence-prior mechanism, and VTD is likely removing generic directional content.

Watch

Extended reading notes

Core claim

The paper's central claim is that hallucinations in LVLMs with discrete image tokenizers are closely linked to visual priors encoded in token co-occurrence, and specifically that absent tokens from dominant clusters are the carriers of hallucination. Context-Guided Clustering groups codebook tokens by spatial and semantic co-occurrence, and the resulting dominant clusters of an input image contain tokens that are absent from the image yet strongly associated with hallucinated objects, as measured by HitRate@K. Visual Token Decontamination then suppresses those absent tokens during autoregressive decoding: at a chosen transformer layer, for each such token $v_{\text{hal}}$ the hidden state $g^{(l)}(v_{\text{hal}})$ is projected onto the direction of each present image token's hidden state and subtracted with a scaling coefficient $\gamma$, so that the co-occurrence-driven prior is removed from the representations the language model reads. The paper reports that this reduces hallucination across Chameleon-7B, Janus-Pro-7B, and Emu3-13B on AMBER, Object HalBench, and POPE, while keeping or improving MME perception scores.

Load-bearing premise

The method assumes that a token that is not in the image still has a useful 'mental direction' inside the model, and that subtracting that direction removes false objects without also erasing real ones.

Editorial extensions

If this is right

  • Because CGC runs once per model and VTD only edits hidden states during decoding, the method adds no training step and is faster and lighter than contrastive decoding or two-pass revision baselines.
  • Hallucination and coverage can be decoupled: VTD lowers CHAIR and Hal while leaving MME perception scores flat or higher, so the reduction is not simply censorship of object mentions.
  • VTD composes with language-side methods such as OPERA, and the combined gain exceeds either method alone, implying that visual-prior and language-prior hallucinations are partly independent.
  • The transfer of CGC clusters from COCO to GQA in POPE suggests the learned co-occurrence structure generalizes beyond the segmentation dataset used to build it.
  • On discrete-token models, contrastive-decoding baselines can inflate response length and even increase hallucination; VTD avoids perturbing the visual input, which the analysis shows is unstable in this paradigm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the authors leave implicit: if the co-occurrence-prior mechanism is right, hallucination should also be reducible at the source by retraining or masking the tokenizer to break dominant-cluster statistics; the paper does not attempt that.
  • A natural next experiment the paper does not run: apply the same latent-editing recipe to continuous-feature LVLMs by treating clusters of patch embeddings as pseudo-tokens.
  • The HitRate analysis could serve as a cheap pre-screening tool: run CGC on a new model and check whether a small sample's hallucinated objects are predominantly top-associated objects of absent dominant-cluster tokens.
  • A control the paper does not report would sharpen the causal claim: subtracting a random absent token's direction, matched for frequency, should not produce the same CHAIR drop as subtracting the identified absent dominant-cluster tokens.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper studies object hallucination in LVLMs that use discrete image tokenizers. It hypothesizes that co-occurrence statistics among image tokens create visual priors, causing models to evoke visually absent tokens from dominant clusters. To test this, the authors propose Context-Guided Clustering (CGC), which builds a co-occurrence graph over codebook tokens using COCO panoptic segmentation and 3x3 spatial neighborhoods, learns graph embeddings with a GNN and contrastive loss, and clusters the tokens. They then propose Visual Token Decontamination (VTD), which during decoding subtracts a projection of the hidden-state representation of absent dominant-cluster tokens from the hidden states of present image tokens. Experiments on Chameleon-7B, Janus-Pro-7B, and Emu3-13B across AMBER, Object HalBench, MME, and POPE report consistent hallucination reductions with error bars, a favorable efficiency comparison, and gains when combined with OPERA and other baselines.

Significance. If the causal claim is established, the paper identifies a new, tokenizer-specific source of object hallucination and offers a lightweight, training-free mitigation that is complementary to existing decoding-time methods. Strong points include the breadth of evaluation (three discrete-token LVLMs, multiple benchmarks, error bars), the ablations of CGC design choices, the cross-dataset POPE results, the efficiency analysis, and the commitment to release code. However, the diagnostic evidence for the central claim is weakened by a same-data confound, and the core VTD operation is underspecified. The contribution is promising and potentially publishable, but the central claim needs additional validation before it can be accepted as stated.

major comments (4)
  1. [Sec. 3.2 and Sec. 3.1] The HitRate analysis is confounded by using the same COCO 2017 Panoptic segmentation source both to construct the CGC co-occurrence graph and to score token-object associations. In Sec. 3.2 the paper states that 'the segmentation masks are obtained from the COCO 2017 Panoptic dataset, which we also use for CGC,' while Sec. 3.1 and Algorithm 1 define graph edges precisely as token pairs that co-occur inside 3x3 grid cells or within the same panoptic mask. Since the contrastive objective pulls together tokens that co-occur inside those same masks, a high HitRate for absent tokens from dominant clusters may partly re-discover the graph-construction objective rather than independently demonstrate a visual prior that causes hallucination. The K-means baseline in Figure 3 (middle) does not remove this confound because it operates on the same token statistics and is scored on the same masks. To support the load-bearing claim that hallucinations are 'closely linked to visual priors that can evoke absent tokens from dominant clusters,' please provide a held-out validation: for example, train CGC on one segmentation source and score token-object associations on a different dataset or on a strictly disjoint split, and also report what happens when the co-occurrence graph is constructed from permuted or random co-occurrence statistics.
  2. [Sec. 3.3, Eq. (4)] VTD is underspecified in a way that blocks reproducibility. Equation (4) subtracts a projection of g(l)(vhal) from g(l)(vi), but vhal is by definition absent from the input token sequence, so g(l)(vhal) is not defined by the LVLM forward pass. The text says VTD 'projects these tokens into the model's latent space' but does not state how the hidden state of an absent token is obtained: is it computed by a separate forward pass with vhal inserted into the sequence, by taking the codebook embedding and projecting it to layer l, or by some other mapping? In addition, if multiple absent tokens are identified, the paper does not specify whether the subtraction is applied sequentially, summed over all vhal, or otherwise aggregated. Please specify the computation precisely and include an ablation that replaces g(l)(vhal) with a neutral baseline, such as the raw codebook embedding or the hidden state of a random absent token, to verify that the method acts on hallucination-related signal rather than on general visual content.
  3. [Sec. 3.2, Figure 3] The HitRate@K evidence is reported only as aggregate curves over 1,000 AMBER images, without the number of hallucinated objects per group, confidence intervals, or a statistical test of the 5-10 percentage-point gap between C2 and C1/C3. Given the same-data confound in the first comment, the aggregated comparison is not yet sufficient to establish that absent tokens in dominant clusters are 'more strongly associated with hallucinated objects.' Please report per-image variance, significance tests, and, ideally, a breakdown across object categories to show that the effect is not driven by a few frequent hallucinated objects.
  4. [Sec. 4.2, Tables 2 and 12-13] The combination experiments are presented with only CHAIR-s and CHAIR-i on Object HalBench, and the appendix tables contain apparent typographical errors and inconsistent signs (for example, Table 13 lists '1966' for PROJECTAWAY+CHAIR-s, and the VCD+ row in Table 12 shows a positive CHAIR-s change but a negative CHAIR-i change relative to the baseline). These issues make it difficult to assess the claim that CGC+VTD 'enhances the performance of prior methods in most cases.' Please correct the typos, report full evaluation sets with error bars, and clarify whether the combination results are averaged over multiple runs.
minor comments (6)
  1. [Sec. 3.3 heading] The heading 'Mitgate Hallucination via Visual Token Decontamination' contains a typo and should read 'Mitigate Hallucination via Visual Token Decontamination.'
  2. [Figure 5 caption] The caption reads 'We report CHIAR score for generative tasks' and should read 'CHAIR score.'
  3. [Appendix B, Table 13] The value '1966' in the PROJECTAWAY+ row of Table 13 appears to be a typo for '19.66'; please correct it.
  4. [Sec. 6] The limitation section states 'we observe a slight drop in coverage scores (Cover ↑ in Table 1) with our proposed method,' but in Table 1 CGC+VTD's Cover is higher than Nucleus Sampling on all three models. Please clarify the intended comparison (e.g., relative to contrastive-decoding baselines) or correct the statement.
  5. [Appendix B] The sentence 'We discuss how to tune other other performance-related hyperparameters' contains a duplicated word; please remove the second 'other.'
  6. [Sec. 3.2] The HitRate@K metric refers to 'top-K(C)' but does not formally specify how the object ranking for a token group is obtained from the segmentation masks; please define the association score and the ranking procedure.

Circularity Check

1 steps flagged · score 5.0 of 10

The Sec. 3.2 HitRate diagnostic is partially circular: token–object association scores use the same COCO panoptic masks that define the CGC co-occurrence graph, so the high HitRate of absent dominant-cluster tokens partly re-discovers the clustering objective.

  1. fitted input called prediction [Sec. 3.1 (Context-Guided Clustering, graph construction) and Sec. 3.2 (Visual Priors Induce Hallucinations, HitRate analysis)]
    "CGC first computes the co-occurrence strength between each pair of image tokens based on two complementary contexts: ... (2) semantic coherence, where tokens located within the same object segmentation mask are given greater strength. ... To quantify the association between a token group and an object, we count how often tokens from that group appear within the object’s segmentation masks. The segmentation masks are obtained from the COCO 2017 Panoptic dataset, which we also use for CGC."

    The CGC co-occurrence graph is constructed from COCO 2017 Panoptic masks: edges connect tokens co-occurring inside the same object segmentation mask, and the GNN contrastive objective pulls such connected tokens together before K-means clustering. The HitRate analysis in Sec. 3.2 then scores every token group by counting how often its tokens fall inside object masks from the very same COCO 2017 Panoptic dataset. Consequently, the top-K objects 'associated' with a cluster are approximately the mask labels that generated the cluster in the first place; the high HitRate of C2 (absent tokens from the dominant cluster) is in part a re-description of the graph-construction objective rather than independent evidence that absent tokens drive hallucination.

full rationale

The paper's main diagnostic evidence for the visual-prior hypothesis is partially self-referential. CGC builds its co-occurrence graph from COCO 2017 Panoptic segmentation masks, and the HitRate metric used to link token groups to hallucinated objects counts token appearances inside those same masks. Thus the correlation between absent tokens in dominant clusters and hallucinated object categories is partly manufactured by the data feeding both the clustering and the evaluation. However, the paper does not rely solely on this correlation: VTD's effectiveness is measured on Object HalBench, MME, and POPE with three discrete-token LVLMs, and those benchmarks are external to the CGC training data. The same-data confound weakens the causal interpretation of Sec. 3.2 but does not reduce the entire contribution to a tautology. No self-citation chains or ansatz-smuggling via citation were found. Overall, the circularity is real but partial, warranting a score of 5 rather than a higher score.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method relies on several hand-tuned parameters (cluster size, number of dominant clusters, editing layer, editing magnitude) that are fitted on the AMBER subset, and on the untested assumption that COCO-derived co-occurrence clusters generalize. The VTD suppression rule is introduced without a mechanistic justification, making it an ad hoc intervention rather than a derived consequence. No new physical or conceptual entities are postulated.

free parameters (5)
  • cluster size = 10 for all models
    Tuned via grid search on an AMBER subset (Sec. B.1); cluster size directly determines how tokens are grouped and thus which absent tokens are suppressed.
  • number of selected dominant clusters = 2 for Chameleon/Janus-Pro, 4 for Emu3
    Tuned on an AMBER subset (Sec. B.1); determines how many clusters contribute absent tokens for VTD.
  • editing layer l = 25 (Chameleon), 27 (Janus-Pro), 21 (Emu3)
    Tuned via layer sweep on an AMBER subset (Sec. B.1); controls where in the transformer the hidden-state subtraction is applied.
  • editing magnitude coefficient gamma = 0.5 (Chameleon), 0.2 (Janus-Pro), 0.6 (Emu3)
    Tuned via grid search on an AMBER subset (Sec. B.1); scales the strength of the decontamination subtraction in Eq. (4).
  • graph construction thresholds = 3x3 grid, top 10% connections
    Chosen by hand (Sec. 3.1, Algorithm 1); affects which edges enter the co-occurrence graph and hence the clusters.
assumptions (3)
  • domain assumption Image token co-occurrence statistics in COCO panoptic capture visual priors that generalize to the target LVLMs and benchmarks.
    The entire CGC step assumes that co-occurrence patterns measured on a single segmentation dataset are representative of the visual priors inside Chameleon, Emu3, and Janus-Pro. The paper provides no direct evidence that the tokenizer's training data aligns with COCO statistics.
  • ad hoc to paper Subtracting the hidden-state projection of an absent token from present-token hidden states removes hallucination-related signal without removing useful visual content.
    Eq. (4) is the core VTD intervention, but no derivation or causal proof is provided. It is an asserted mechanism.
  • ad hoc to paper Tokens that co-occur in a 3x3 spatial grid or within the same segmentation mask are the right notion of 'context' for hallucination.
    The graph construction in Sec. 3.1 defines co-occurrence by these two rules; other choices (e.g., larger neighborhoods, global co-occurrence) are not compared except in ablation of removing each signal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing." pith.science (2026). https://pith.science/paper/WKEPV2Y7

@misc{pith2026250521547,
  author       = {Pith},
  title        = {Pith review of: Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WKEPV2Y7}},
  note         = {Machine review of arXiv:2505.21547}
}
read the original abstract

Large Vision-Language Models (LVLMs) with discrete image tokenizers unify multimodal representations by encoding visual inputs into a finite set of tokens. Despite their effectiveness, we find that these models still hallucinate non-existent objects. We hypothesize that this may be due to visual priors induced during training: When certain image tokens frequently co-occur in the same spatial regions and represent shared objects, they become strongly associated with the verbalizations of those objects. As a result, the model may hallucinate by evoking visually absent tokens that often co-occur with present ones. To test this assumption, we construct a co-occurrence graph of image tokens using a segmentation dataset and employ a Graph Neural Network (GNN) with contrastive learning followed by a clustering method to group tokens that frequently co-occur in similar visual contexts. We find that hallucinations predominantly correspond to clusters whose tokens dominate the input, and more specifically, that the visually absent tokens in those clusters show much higher correlation with hallucinated objects compared to tokens present in the image. Based on this observation, we propose a hallucination mitigation method that suppresses the influence of visually absent tokens by modifying latent image embeddings during generation. Experiments show our method reduces hallucinations while preserving expressivity. Code is available at https://github.com/weixingW/CGC-VTD/tree/main

Figures

Figures reproduced from arXiv: 2505.21547 by the authors.

Figure 1
Figure 1. Overview of our proposed method, which consists of two key components: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the CGC pipeline. Code￾book entries are first used to help construct a co￾occurrence graph over image tokens, followed by a learning process to obtain graph-based embed￾dings, which are then used to cluster image tokens. To capture visual priors, we cluster image tokens based on their co-occurrence patterns. We per￾form clustering based on codebook embeddings rather than directly on image token embed… view at source ↗
Figure 3
Figure 3. Left: Hallucination HitRate for K from 1-10. Middle: Comparison between C2 tokens identified by our CGC method versus naive K-means. Right: HitRate@5 of C2 tokens from the ten most dominant clusters. We plot HitRate@K for K = 1 to 10 for the three token groups C1, C2, and C3 in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The three heatmaps show CHAIR score difference between our method with the Nucleus [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: We show how editing weight γ, cluster size, number of GNN layers and editing layer affect our method. We report CHIAR score for generative tasks and F1 for discriminative tasks using Janus-Pro-7B. Overall, our method is relatively robust to hyperparameters. 16 [PITH_F…
Figure 6
Figure 6. Figure 6: Left: Hallucination HitRate for K from 1-10. Middle: Comparison between C2 tokens identified by our CGC method versus naive K-means. Right: HitRate@5 of C2 tokens from the ten most dominant clusters. The analysis is based on Chameleon-7B. D Additional results D.1 Combi…
Figure 7
Figure 7. Figure 7: Left: Hallucination HitRate for K from 1-10. Middle: Comparison between C2 tokens identified by our CGC method versus naive K-means. Right: HitRate@5 of C2 tokens from the ten most dominant clusters. The analysis is based on Emu3-13B [PITH_FULL_IMAGE:figures/full_fig_…
Figure 8
Figure 8. Figure 8: Cases as qualitative results. Comparison of applying different types of hallucination mitiga [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Change of answering length when applying VCD. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

    cs.AI 2026-04 unverdicted novelty 8.0 of 10

    FeynmanBench is the first benchmark for evaluating multimodal LLMs on diagrammatic reasoning with Feynman diagrams, revealing systematic failures in enforcing physical constraints and global topology.

  2. The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook

    cs.AI 2026-04 accept novelty 5.0 of 10

    A large survey organizes latent-space work in language-based models by foundation, evolution, four mechanisms, seven abilities, and open challenges.

Reference graph

Works this paper leans on

45 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    Mitigating object hallucinations in large vision-language models through visual contrastive decoding.arXiv preprint arXiv:2311.16922, 2023

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding.arXiv preprint arXiv:2311.16922, 2023

    arXiv 2023

  2. [2]

    Self-introspective decoding: Alleviating hallucinations for large vision-language models, 2024

    Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self-introspective decoding: Alleviating hallucinations for large vision-language models, 2024

    2024

  3. [3]

    Code: Contrasting self-generated description to combat hallucination in large multi-modal models, 2024

    Junho Kim, Hyunjun Kim, Yeonju Kim, and Yong Man Ro. Code: Contrasting self-generated description to combat hallucination in large multi-modal models, 2024

    2024

  4. [4]

    Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding, 2024

    Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding, 2024

    work page 2024

  5. [5]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation, 2024

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation, 2024

    work page 2024

  6. [6]

    Interpreting and editing vision-language representations to mitigate hallucinations.arXiv preprint arXiv:2410.02762, 2025

    Jiang Nick, Kachinthaya Anish, Petryk Suzie, and Gandelsman Yossi. Interpreting and editing vision-language representations to mitigate hallucinations.arXiv preprint arXiv:2410.02762, 2025

    arXiv 2025

  7. [7]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    ChameleonTeam. Chameleon: Mixed-modal early-fusion foundation models, 2024

    work page 2024

  8. [8]

    Emu3: Next-token prediction is all you need, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need, 2024

    2024

Show all 45 references
  1. [9]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024

    2024

  2. [10]

    Jian Jia, Jingtong Gao, Ben Xue, Junhao Wang, Qingpeng Cai, Quan Chen, Xiangyu Zhao, Peng Jiang, and Kun Gai. From principles to applications: A comprehensive survey of discrete tokenizers in generation, comprehension, recommendation, and information retrieval.arXiv preprint a...

    2025 arXiv

  3. [11]

    Show-o: One single transformer to unify multimodal understanding and generation, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation, 2024

    2024

  4. [12]

    Anygpt: Unified multimodal llm with discrete sequence modeling, 2024

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal llm with discrete sequence modeling, 2024

    2024

  5. [13]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025

    2025

  6. [14]

    Next token prediction towards multimodal intelligence: A comprehensive survey.arXiv preprint arXiv:2412.18619, 2024

    Liang Chen, Zekun Wang, Shuhuai Ren, Lei Li, Haozhe Zhao, Yunshui Li, Zefan Cai, Hongcheng Guo, Lei Zhang, Yizhe Xiong, et al. Next token prediction towards multimodal intelligence: A comprehensive survey.arXiv preprint arXiv:2412.18619, 2024

    2024 arXiv

  7. [15]

    Analyzing and mitigating object hallucination in large vision-language models, 2024

    Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models, 2024

    2024

  8. [16]

    Visual commonsense r-cnn, 2020

    Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-cnn, 2020

    2020

  9. [17]

    MacQueen

    J. MacQueen. Some methods for classification and analysis of multivariate observations. 1967

    1967

  10. [18]

    Object hallucination in image captioning, 2019

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning, 2019

    2019

  11. [19]

    Evaluating object hallucination in large vision-language models, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023

    2023

  12. [20]

    Contrastive decoding: Open-ended text generation as optimization, 2023

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization, 2023

    2023

  13. [21]

    Improved baselines with visual instruction tuning, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024

    2024

  14. [22]

    Investigating and mitigating object hallucinations in pretrained vision-language (clip) models, 2024

    Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. Investigating and mitigating object hallucinations in pretrained vision-language (clip) models, 2024

    2024

  15. [23]

    Neural discrete representation learning, 2018

    Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018

    2018

  16. [24]

    Taming transformers for high-resolution image synthesis, 2021

    Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2021

    2021

  17. [25]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023

    2023

  18. [26]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015

    2015

  19. [27]

    How attentive are graph attention networks?, 2022

    Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks?, 2022

    2022

  20. [28]

    Representation learning with contrastive predictive coding, 2019

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019

    2019

  21. [29]

    Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation, 2024

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang. Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation, 2024. 11

    2024

  22. [30]

    Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness, 2024

    Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness, 2024

    2024

  23. [31]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    2024

  24. [32]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback, 2024

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback, 2024

    2024

  25. [33]

    Detecting and preventing hallucinations in large vision language models.arXiv preprint arXiv:2308.06394, 2024

    Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models.arXiv preprint arXiv:2308.06394, 2024

    2024 arXiv

  26. [34]

    Improving autoregressive visual generation with cluster-oriented token prediction, 2025

    Teng Hu, Jiangning Zhang, Ran Yi, Jieyu Weng, Yabiao Wang, Xianfang Zeng, Zhucun Xue, and Lizhuang Ma. Improving autoregressive visual generation with cluster-oriented token prediction, 2025

    2025

  27. [35]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  28. [36]

    xformers: A modular and hack- able transformer modelling library

    Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hack- able transforme...

    2022

  29. [37]

    Towards understanding how knowledge evolves in large vision-language models, 2025

    Sudong Wang, Yunjian Zhang, Yao Zhu, Jianing Li, Zizhe Wang, Yanwei Liu, and Xiangyang Ji. Towards understanding how knowledge evolves in large vision-language models, 2025

    2025

  30. [38]

    Eliciting latent predictions from transformers with the tuned lens, 2023

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023

    2023

  31. [39]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  32. [40]

    Describe this image

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019. A Benchmark Details A.1 AMBER AMBER consists of 1,004 high-quality images collected from MS-COCO 2014 test set [ 26] and Unsplash4. The dat...

    2019

  33. [41]

    You should only mention objects that are explicitly mentioned in the examiner’s answer

  34. [42]

    You should only extract the object names without the attributes of the objects

  35. [43]

    as part of the object name in your result

    You should not include the properties of the object, like the color, material, etc. as part of the object name in your result

  36. [44]

    object 1

    Make your answer precise. Present the results in a JSON list format: ["object 1", ..., "object n"]

  37. [45]

    The image displays

    You should return an empty JSON list ([]) if no visible objects can be found. A.3 MME The Multimodal Large Language Model Evaluation Benchmark (MME) [31] represents a pioneering effort to comprehensively assess the capabilities of LVLMs. This benchmark evaluates models across ...

    2000

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.