Pith. sign in

REVIEW 3 major objections 4 minor 51 references

CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that multilingual object hallucination in LVLMs can be mitigated at inference time by shifting the attention outputs of language-specific cross-modal heads toward English, with average gains of 13.56 points on POPE and…

desk verdict Genuinely new inference-time method with a plausible mechanism, but the headline POPE gain is partly in-sample via hyperparameter tuning, so the numbers are promising rather than proven. read the letter →

arxiv 2506.11073 v1 pith:42L6UZYC submitted 2025-06-03 cs.CL cs.AIcs.CV

Zekai Ye , Qiming Li , Xiaocheng Feng , Libo Qin , Yichong Huang , Baohang Li , Kui Jiang , Yang Xiang
show 5 more authors
Zhirui Zhang Yunfei Lu Duyu Tang Dandan Tu Bing Qin
This is my paper
classification cs.CLcs.AIcs.CV
keywords multilingualobjecthallucinationlargevision-languagemodelsattentioninterventiontraining-freemitigationcross-lingualalignmentPOPEbenchmarkMMEheads
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a large share of multilingual object hallucination in vision-language models comes from non-English queries failing to route visual attention the way English queries do, and that this can be corrected without retraining. Its proposed method, CLAIM, finds the attention heads whose cross-modal outputs differ most between English and a target language, estimates a per-head language shift vector from 1,000 captioning prompts, and adds that shift at inference time. On two LVLMs and six non-English languages it reports average accuracy gains of 13.56 percentage points on POPE and 21.75 percentage points on MME's hallucination subsets. If true, this offers a lightweight, plug-and-play alternative to supervised fine-tuning and preference optimization for multilingual hallucination.

What carries the argument

The central object is the language shift vector $S^l_h$, a per-head, per-layer mean difference in attention outputs between English and target-language caption queries over 1,000 COCO images, computed by Equation (9) with the single template “What is it in the image?”. During inference, Equation (10) adds the scaled shift $\alpha S^l_h$ to the attention outputs of the heads selected by probe classification accuracy, so that non-English queries are processed with English-like visual perception.

What would settle it

Compute the shift from the caption template, then evaluate on Spanish questions whose syntax differs sharply from the template—for example attribute or spatial-relation questions on images outside COCO; if accuracy does not improve over the baseline, the transfer assumption fails.

Watch

Extended reading notes

Core claim

CLAIM establishes that cross-modal attention outputs carry a language-specific offset: for semantically identical queries, the attention output of certain heads differs systematically between English and other languages, and this offset can be measured once and then applied during generation. The paper identifies language-specific cross-modal attention heads by training linear probes to predict query language from masked attention outputs, estimates the shift as the mean English-minus-target difference over COCO caption queries, and intervenes by adding a scaled shift to the attention outputs of selected heads. This realigns non-English attention toward English-like perception, reducing object-level and attribute-level hallucination without updating any model weights.

Load-bearing premise

The load-bearing premise is that a shift vector averaged over 1,000 captioned COCO images with the single template “What is it in the image?” transfers to arbitrary unseen questions, images, and tasks at inference.

Editorial extensions

If this is right

  • CLAIM works without supervised fine-tuning or preference optimization, so it avoids the cost of building annotated multilingual image-text datasets.
  • The method improves both low-resource and high-resource languages, and a shift estimated from one language pair transfers partially to unseen languages.
  • The intervention reduces not only object-level hallucination in existence and count questions but also attribute-level hallucination in color and position questions.
  • Attention divergence between languages is strongest in intermediate layers, indicating where cross-modal integration and language mapping happen in LVLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same shift-based intervention could be applied to other token divisions, such as separating image regions or audio chunks, to steer multimodal attention without training.
  • A fixed shift averaged over 1,000 images could be replaced by a per-query adaptive shift computed on the fly, which may improve robustness on out-of-distribution images and non-caption-style questions.
  • Because CLAIM aligns non-English attention toward English patterns, it may reinforce English-centric biases, as the paper itself notes; a natural extension is aligning toward a multilingual consensus representation instead.
  • A testable extension is estimating shifts from task-specific query distributions and checking whether the resulting vectors transfer across languages and benchmarks better than the single caption-template estimate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CLAIM, an inference-time intervention that mitigates multilingual object hallucination in LVLMs by aligning cross-modal attention patterns of non-English queries to those of English. The method first identifies language-specific cross-modal attention heads using linear probes, then estimates language shift vectors as the mean English-minus-target difference in attention outputs over 1,000 COCO images, and finally adds scaled shift vectors to the selected heads during inference. Experiments on POPE and MME with LLaVA-1.5 and Qwen-VL-Chat report consistent improvements across five to six non-English languages, with an average improvement of 13.56% on POPE and 21.75% on MME hallucination subsets claimed in the abstract.

Significance. If the central claim holds, the paper offers a valuable, near training-free alternative to fine-tuning approaches for multilingual hallucination, with a mechanistic explanation in terms of cross-modal attention divergence. The evaluation is external (POPE, MME), uses multiple models and languages, and includes comparisons to VCD and PAI; the method is falsifiable and relatively cheap to apply. The attention-level analysis and intervention analysis are thoughtful and help interpret why the method works. The main weakness is that the headline POPE number is partly in-sample because the two hyperparameters are tuned on the POPE-COCO popular subset, which is part of the reported average; this needs to be addressed before the quantitative claims can be taken at face value.

major comments (3)
  1. [Appendix A, Table 1, Abstract] The hyperparameters α and K are selected by sequential search on the POPE-COCO popular subset (Appendix A), and this same subset is included in the POPE average reported in Table 1 and used for the 13.56% improvement in the abstract. Because the tuning subset is part of the reported evaluation distribution, the headline POPE gain is partly in-sample. Please report the average excluding the COCO-popular rows, or perform hyperparameter selection on a held-out validation split disjoint from all evaluation benchmarks, and verify that the improvement is not concentrated in the tuned subset.
  2. [§3.2–§3.3] Head selection uses masked attention outputs bO (Eq. 6) that exclude text tokens, whereas the language shift vectors (Eq. 9) and the intervention (Eq. 10) act on the unmasked outputs O. The paper justifies this by preserving the original representation space, but it does not address whether the heads selected under masking are the right heads to intervene on with shifts computed from the unmasked distribution. Please provide an analysis (e.g., head-selection agreement between masked and unmasked outputs) or a theoretical argument that the mismatch does not undermine the intervention's mechanism.
  3. [§4, Table 1, Figure 8] No variance estimates or repeated-run results are reported, so it is unclear whether the performance gaps (which can be as small as a few points) are significant relative to run-to-run variation. Please report standard deviations over at least three seeds, or provide bootstrap confidence intervals, particularly for the hyperparameter robustness analysis in Figure 8, which currently shows only the tuning subset.
minor comments (4)
  1. [Abstract and §4.3] The reported improvements (13.56%, 21.75%, 17.5%, 9.8%) should be clarified as absolute percentage points or relative improvements, and the exact set of languages and rows over which each average is computed should be specified.
  2. [Throughout] The text contains typos: 'Rencent' (Introduction), 'Perfomance' (Table 2 caption), 'theses' (§5.4), and 'a issue' (§2); please proofread.
  3. [Table 1] The Avg columns in Table 1 exclude the English column; please state this explicitly in the table caption or in the text.
  4. [Figure 3] The x-axis labels (e.g., 'Total Average') and y-axis units should be defined in the caption so the reader can interpret the scale.

Circularity Check

1 steps flagged · score 3.0 of 10

POPE headline gain is partly in-sample: α and K are tuned on the POPE-COCO popular subset, which is then included in the reported POPE average.

  1. fitted input called prediction [Appendix A; Table 1; Section 4.3]
    "We conduct hyperparameter tuning exclusively on the popular subset of POPE-COCO, with the search space for K defined as {50, 100, 150, 200, 250, 300} and for α as {0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5}."

    The hyperparameters α and K are selected by maximizing accuracy on the POPE-COCO popular subset, and Table 1 reports the POPE-COCO popular rows as part of the aggregate that supports the abstract's 13.56% POPE improvement and Section 4.3's 17.5%/9.8% average gains. The improvement on that subset is therefore in-sample: the hyperparameters were chosen using the very rows that are then reported, so the headline POPE number is not a fully out-of-sample prediction. The MME numbers are not directly tuned, but they inherit the same α and K, so they are not fully independent of the selection.

full rationale

The core derivation is not circular: language shift vectors (Eq. 9) are the mean English-minus-target attention-output difference over 1000 COCO-2017 training captions, and the intervention (Eq. 10) is applied to translated POPE/MME presence, attribute, and count questions whose labels are never used to estimate the shift or select heads. Probe-based head identification (Eqs. 7-8) uses only language labels, not hallucination labels, so the benchmark outcomes are external evidence rather than constructed outputs. The limitations section correctly notes the method requires white-box access and may reinforce English-centric bias, neither of which reveals a definitional dependence. The one concrete circularity-like concern is that hyperparameter selection is performed on the POPE-COCO popular subset and that subset is included in the reported POPE average, making that component in-sample; the MME result is mostly independent but shares the same hyperparameters. Overall this is a modest selection-bias concern, not an equation-level circularity, so the score is 3.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central method rests on two calibrated scalar hyperparameters (alpha and K) and four domain assumptions about the safety and transferability of attention-space shifts. No new physical or conceptual entities are introduced; the language shift vector is an estimated statistic, not a postulated entity.

free parameters (2)
  • alpha (intervention intensity) = not stated numerically; tuned on POPE-COCO popular, range {0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5}
    Scales the language shift vector added to attention outputs (Eq. 10); chosen by sequential search on POPE-COCO popular subset.
  • K (number of intervened heads) = not stated numerically; tuned on POPE-COCO popular, range {50, 100, 150, 200, 250, 300}
    Number of language-specific cross-modal attention heads selected by Top-K probe accuracy (Eq. 8); tuned on POPE-COCO popular subset.
assumptions (4)
  • domain assumption Additive intervention in attention output space (Eq. 10) is a safe, semantics-preserving transformation.
    The method adds a constant shift to selected heads' outputs and expects generation quality to remain; no normalization or reweighting is applied.
  • domain assumption Probe classification accuracy identifies heads causally relevant to cross-lingual visual perception.
    Section 3.2 uses Top-K probe accuracy to select intervention heads; this equates correlation with language discriminability to causal role in perception.
  • domain assumption English cross-modal attention is a desirable target; non-English divergence is a defect, not information.
    Section 3.4 and the Limitations state that aligning non-English to English is the goal; the paper assumes English perceptual patterns are superior or at least less hallucination-prone.
  • domain assumption A single shift vector per language, estimated from one query template over 1000 images, transfers across query types and images.
    Eq. (9) averages only caption-query attention differences; at inference it is applied to yes/no presence queries, attribute questions, and other tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention." pith.science (2026). https://pith.science/paper/42L6UZYC

@misc{pith2026250611073,
  author       = {Pith},
  title        = {Pith review of: CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/42L6UZYC}},
  note         = {Machine review of arXiv:2506.11073}
}
read the original abstract

Large Vision-Language Models (LVLMs) have demonstrated impressive multimodal abilities but remain prone to multilingual object hallucination, with a higher likelihood of generating responses inconsistent with the visual input when utilizing queries in non-English languages compared to English. Most existing approaches to address these rely on pretraining or fine-tuning, which are resource-intensive. In this paper, inspired by observing the disparities in cross-modal attention patterns across languages, we propose Cross-Lingual Attention Intervention for Mitigating multilingual object hallucination (CLAIM) in LVLMs, a novel near training-free method by aligning attention patterns. CLAIM first identifies language-specific cross-modal attention heads, then estimates language shift vectors from English to the target language, and finally intervenes in the attention outputs during inference to facilitate cross-lingual visual perception capability alignment. Extensive experiments demonstrate that CLAIM achieves an average improvement of 13.56% (up to 30% in Spanish) on the POPE and 21.75% on the hallucination subsets of the MME benchmark across various languages. Further analysis reveals that multilingual attention divergence is most prominent in intermediate layers, highlighting their critical role in multilingual scenarios.

Figures

Figures reproduced from arXiv: 2506.11073 by the authors.

Figure 1
Figure 1. A comparison of attention weights map be [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed CLAIM method. A block of MHA in the figure represents a attention head. CLAIM intervene in identified language-specific cross-modal attention heads using estimated language shift vectors. (1) Identification of Language-Specific Cross-Modal Attention Heads §3.2: We train probes to identify the language-specific cross-modal attention heads, which exhibit significantly different behavior across… view at source ↗
Figure 3
Figure 3. Average scores for LLaVA-1.5 across five languages on the MME full dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Logit lens observation for interpreting LLaVA-1.5 in multilingual scenarios. The depth of block color for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Ab per layer of LLaVA-1.5 across four lan￾guages, and the per-layer average change rate of non￾English languages Ab relative to English. As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Kernel density estimate plot of cross-modal [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of (a) sum attention weights of the [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Impact of hyperparameters α and K on the Accuracy for LLaVA-1.5 and Qwen-VL-Chat on the POPE-COCO popular subset. The "x" symbol indicates a value extraordinarily lower than normal. 6 Conclusion In this paper, we propose Cross-Lingual Attention Intervention for Mitigat…
Figure 9
Figure 9. Figure 9: Impact of training data size on the Accuracy [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Case study [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 11 canonical work pages

  1. [1]

    Michael Andersland. 2024. Amharic llama and llava: Multimodal llms for low resource languages. arXiv preprint arXiv:2403.06354

    work page Pith review arXiv 2024

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023 a . Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023 b . Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  4. [4]

    Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, and Chenliang Xu. 2024. Unveiling visual perception in language models: An attention head analysis approach. arXiv preprint arXiv:2412.18108

    arXiv 2024

  5. [5]

    Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. 2024 a . https://arxiv.org/abs/2405.15356 Alleviating hallucinations in large vision-language models through hallucination-induced optimization . Preprint, arXiv:2405.15356

    arXiv 2024

  6. [6]

    Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. 2024 b . Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2411.15268

    arXiv 2024

  7. [7]

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024 c . Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.00425

    arXiv 2024

  8. [8]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

Show all 51 references
  1. [9]

    Corinna Cortes. 1995. Support-vector networks. Machine Learning

    1995

  2. [10]

    Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  3. [11]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394

    2023 arXiv

  4. [12]

    Gregor Geigle, Abhay Jain, Radu Timofte, and Goran Glavas. 2023. https://doi.org/10.48550/ARXIV.2307.06930 mBLIP : Efficient Bootstrapping of Multilingual Vision - LLMs . CoRR, abs/2307.06930. ArXiv: 2307.06930

    2023 doi arXiv

  5. [13]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. https://arxiv.org/abs/2311.05232 A survey on hallucination in large language models: Principles, taxonomy, challenges, and ...

    2023 arXiv

  6. [14]

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Con...

    2024

  7. [15]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709

    2019

  8. [16]

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. 2024. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724

    2024 arXiv

  9. [17]

    Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2024

  10. [18]

    Kenneth Li, Oam Patel, Fernanda Vi \'e gas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36

    2024

  11. [19]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  12. [20]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

    2014

  13. [21]

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations

    2023

  14. [22]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  15. [23]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 b . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  16. [24]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 c . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  17. [25]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2025. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pages 125--140. Springer

    2025

  18. [26]

    Shaker, Salman H

    Muhammad Maaz, Hanoona Abdul Rasheed, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Tim Baldwin, Michael Felsberg, and Fahad Shahbaz Khan. 2024. https://doi.org/10.48550/ARXIV.2402.14818 PALO : A Polyglot Large Multimodal Model for 5B People . Co...

    2024 doi arXiv

  19. [27]

    Nostalgebraist. 2020. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens Interpreting gpt: The logit lens . LessWrong

    2020

  20. [28]

    OpenAI. 2023. GPT-4. https://openai.com/gpt-4

    2023

  21. [29]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in P ython. Journal of Machine L...

    2011

  22. [30]

    Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2025. A survey of multilingual large language models. Patterns, 6(1)

    2025

  23. [31]

    Xiaoye Qu, Mingyang Song, Wei Wei, Jianfeng Dong, and Yu Cheng. 2024. Mitigating multilingual hallucination in large vision-language models. arXiv preprint arXiv:2408.00550

    2024 arXiv

  24. [32]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  25. [33]

    David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, et al. 2024. https://doi.org/10.48550/ARXIV.2406.05967 CVQA : Culturally -diverse Multi...

    2024 doi arXiv

  26. [35]

    Florian Schneider and Sunayana Sitaram. 2024. https://aclanthology.org/2024.findings-emnlp.250 M5 - A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision - Language Tasks . In Findings of the Association for Compu...

    2024

  27. [36]

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146--162. Springer

    2022

  28. [37]

    Dhruv Shah, B a \.z ej Osi \'n ski, Sergey Levine, et al. 2023. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning, pages 492--504. PMLR

    2023

  29. [38]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . https://arxiv.org/abs/2302.13971 Llama: Open and efficient foundation language models . arXiv preprint

    2023 arXiv

  30. [39]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  31. [40]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  32. [41]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. arXiv preprint arXiv:2402.10588

    2024 arXiv

  33. [42]

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. 2023. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247--2256. IEEE

    2023

  34. [43]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 2023. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257

    2023 arXiv

  35. [44]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549

    2023 arXiv

  36. [45]

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. 2024. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220

    2024

  37. [46]

    Bai Z., Wang P., Xiao T., He T., Han Z., Zhang Z., and Shou MZ. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930

    2024 arXiv

  38. [47]

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  39. [48]

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. https://arxiv.org/abs/2311.16839 Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization . Preprint, arXiv:2311.16839

    2023 arXiv

  40. [49]

    Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. 2024. Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. arXiv preprint arXiv:2407.00569

    2024 arXiv

  41. [50]

    Peipei Zhu, Xiao Wang, Lin Zhu, Zhenglong Sun, Wei-Shi Zheng, Yaowei Wang, and Changwen Chen. 2023. Prompt-based learning for unpaired image captioning. IEEE Transactions on Multimedia, 26:379--393

    2023

  42. [51]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.