Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs

Anirudh Phukan; Apoorv Saxena; Divyansh; Harshit Kumar Morj; Koustava Goswami; Vaishnavi

REVIEW 3 major objections 5 minor 46 references

Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ContextualLens replaces logit-lens probing with middle-layer contextual embeddings to detect VLM hallucinations, including attributes, relations, comparisons, actions, and OCR, and to ground answers as bounding boxes.

desk verdict A simple, training-free extension of the logit lens that plausibly improves VLM hallucination detection and grounding, but the headline significance claims outrun the statistics. read the letter →

arxiv 2411.19187 v2 pith:5XMO6D65 submitted 2024-11-28 cs.CL

Anirudh Phukan , Divyansh , Harshit Kumar Morj , Vaishnavi , Apoorv Saxena , Koustava Goswami This is my paper · ORCID

classification cs.CL

keywords hallucinationdetectionvision-languagemodelscontextualembeddingslogitlensgroundedvisualquestionansweringmultimodalattributiontraining-freeinterpretabilityembeddingsimilarity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a training-free method using contextual token embeddings from middle layers of a vision-language model can detect hallucinations and ground answers more reliably than the logit lens, which reads uncontextualized logits from the unembedding layer. On the HQH benchmark, ContextualLens is reported to move hallucination detection in attribute, comparison, and relation categories from near-random to meaningful accuracy, and to outperform the logit lens on actions and OCR. The same similarity signal supports a grounding technique that returns precise bounding boxes, extending zero-shot object segmentation to grounded visual question answering. If the central claim holds, the model internals already used to generate an answer can also verify it and locate its visual evidence without any added training.

What carries the argument

The central object is the contextual embedding vector of answer tokens and image patches taken from intermediate transformer layers. Given answer tokens, let $h_{\mathrm{Ans}, l_T}$ be the average of token embeddings at text layer $l_T$; for each image patch $p_j$, let $h_{l_I}(p_j)$ be its embedding at image layer $l_I$; the patch score is $\mathrm{CosineSim}(h_{\mathrm{Ans}, l_T}, h_{l_I}(p_j))$, and $\mathrm{Confidence}_{\max} = \max_j \mathrm{Score}(p_j)$ is the hallucination score. For grounding, the paper averages patch embeddings inside every candidate bounding box and selects the box maximizing cosine similarity to the answer embedding. The mechanism works by replacing uncontextualized vocabulary-space logits with contextual embeddings that are known to represent multi-token concepts and relations.

What would settle it

On a curated set of images where the answer word appears somewhere in the image but not as the property of the object the question asks about, ContextualLens should give a low Confidence_max for a hallucinated answer; if Confidence_max stays high in that setting, the cosine-similarity signal is not actually checking the relation between the answer and the correct object.

Watch

Extended reading notes

Core claim

ContextualLens detects hallucinations by averaging the contextual embeddings of the generated answer tokens at a chosen text layer, then computing the cosine similarity of that average against each image patch embedding at a chosen image layer; the maximum over patches, $\mathrm{Confidence}_{\max}$, is taken as the model's confidence that the answer is visually supported. The paper argues that because these middle-layer embeddings are contextual, they capture multi-token concepts and relations that the logit lens cannot, which is why the method succeeds on attributes, spatial relations, comparisons, actions, and OCR. For grounding, ContextualLens either resizes per-patch cosine scores to the image or searches over all candidate bounding boxes for the box whose average patch embedding is most similar to the answer embedding, yielding high-precision regions for grounded visual question answering. The paper's central claim is that this contextual-similarity reading of the model's own representations is a general, training-free route to hallucination detection and multimodal attribution.

Load-bearing premise

The load-bearing premise is that cosine similarity between an averaged answer-token embedding and image-patch embeddings, at one fixed pair of middle layers chosen on validation data, reliably indicates whether the answer is visually grounded.

Editorial extensions

If this is right

Hallucination detection in VLM categories that used to be near random—attribute, comparison, and relation—becomes feasible without fine-tuning, using only the model's internal embeddings.
Grounding is not limited to object segmentation: answers involving actions, OCR text, and chart content can be localized to bounding boxes in a training-free way.
Because the method uses the same model that generated the answer, it can run at inference time without external detectors or retrievers.
The method is still weaker than output probabilities on counting, a limitation the paper explicitly acknowledges.
Layer selection is part of the procedure: validation-based or adversarial validation can pick the image/text layer pair, with only small reported drops when the tested category is excluded.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension, left implicit by the paper, is to decompose longer or multi-span answers and ground each span separately, which would move the method beyond short factual VQA.
The bounding-box search over all candidate boxes suggests a cheap proposal mechanism for trained segmentation models, using ContextualLens scores as a prior rather than a final mask.
The reported robustness of the chosen layer pair implies the method might transfer across models of similar depth, but models with different layer counts or vision encoders would likely need a lightweight per-model calibration step; this is directly testable.
The same confidence signal could plausibly serve as an abstention criterion, telling the system when not to answer at all, which the paper does not explore.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, this is a genuine, simple idea with real promise: swap the unembedding logits for middle-layer contextual embeddings when comparing answer tokens to image patches, and you get better hallucination detection on categories like attributes, comparisons, and relations, plus a bounding-box grounding method that actually localizes evidence. Second, the quantitative support is thinner than the abstract suggests: the headline "significant improvements" mostly rests on a single validation-tuned layer pair per model, 250 test examples per category, and no error bars or significance tests. The stress-test note is right about that.

What's actually new: the paper identifies a clear failure mode of Jiang et al.'s logit lens — it can verify presence but not contextual relationships — and fixes it with a parameter change that is almost trivial to implement. The bounding-box version that averages patch embeddings within candidate boxes and picks the max cosine is a sensible, training-free way to do grounded VQA. The qualitative examples look good, and the adversarial validation in Appendix C, at least for InternLM, suggests the layer choice is not fragile across categories. That's credit where it's due.

The load-bearing issue is inferential. Table 1 margins like InternLM Action 0.796 vs 0.795 and OCR 0.772 vs 0.769 are inside noise; the broad claim for actions and OCR leans heavily on Qwen2-VL. With 250 test points, a 0.02 mAP gap is not meaningful without a bootstrap or a test. The layer indices are selected on a validation split, and the robustness check covers only one model and checks rank, not margin stability. Also, "significantly" in the abstract is doing too much work. The Count category is honestly conceded as a failure. These are fixable: report bootstrap CIs, run a proper significance test, release the code and exact layer choices. The method is so simple that reproducibility is a realistic ask.

Who's it for: people working on training-free interpretability for VLMs, hallucination detection, or grounded VQA. It deserves a serious referee — the idea is worth engaging, and the weaknesses are empirical, not conceptual. I'd send it to peer review with a clear request for more statistical rigor. I wouldn't cite the numbers yet, but I'd cite the approach as a useful baseline and a clean extension of the logit lens.

Referee Report

3 major / 5 minor

Summary. The paper proposes ContextualLens, a training-free method for hallucination detection and grounding in vision-language models. Instead of the logit lens, which maps intermediate activations through the unembedding matrix, ContextualLens uses cosine similarity between average contextual embeddings of answer tokens at a middle text layer and image patch embeddings at a middle image layer. A high maximum patch similarity is interpreted as the answer being visually grounded. The paper also proposes a bounding-box variant that searches over all rectangular boxes for the one most similar to the answer embedding, enabling grounded visual question answering. Experiments on the HQH hallucination benchmark compare ContextualLens with output probabilities and the logit lens across seven categories on InternLM-VL and Qwen2-VL, and grounding is evaluated with PR curves on TextVQA-X and VizWiz-G. The paper claims significant improvements over the logit lens, especially in attribute, comparison, and relation categories, and also demonstrates a capability for grounding answers in images.

Significance. If the empirical claims are reliable, the work is a useful step toward training-free, model-internal verification and grounding for VLMs. The failure analysis of the logit lens on contextual categories is plausible and the idea of using middle-layer contextual embeddings is well motivated by prior work. The bounding-box grounding technique is a reasonable extension and the qualitative examples are suggestive. However, the central quantitative evidence is currently undermined by the absence of uncertainty quantification, the small per-category test sets, and validation-based layer selection whose cross-model robustness is not established. The paper's contribution is potentially valuable, but the evidence offered does not yet support the headline claim of 'significant improvement'.

major comments (3)

[§6.1, Table 1] The central claim that ContextualLens 'significantly improves' hallucination detection is not substantiated by the reported statistics. Each mAP is computed on only 250 test examples after validation-based layer selection, yet no error bars, bootstrap confidence intervals, or significance tests are provided. Several headline margins are within noise: on InternLM-VL, Action is 0.796 vs. 0.795 (logit lens) and OCR is 0.772 vs. 0.769, while output probabilities reach 0.856; on both models, Count is worse than output probabilities (0.885 vs. 0.946 and 0.889 vs. 0.956). The Table 1 caption's bold/underline criterion for 'significantly better' is therefore undefined and unverified.
[§4.2.1 and Appendix C] The method's free parameters are the layer indices lT and lI for detection and lb for grounding. Table 1 reports test-split results after selecting these layers on 250-example per-category validation splits. The adversarial validation in Appendix C is performed only for InternLM-VL and only checks that the layer pair (13,27) ranks highly; it does not test whether the chosen layers are stable under resampling or whether the same choice transfers to Qwen2-VL. The layer lb used for the bounding-box technique in §4.2.2 is not specified at all. Without a fixed-layer analysis or cross-model layer-transfer results, the generalization of the validation-selected layers remains an unverified assumption.
[§6.2, Figures 4 and 5] The grounding evaluation is reported only as PR curves, with no aggregate metric or uncertainty quantification. The text claims that ContextualLens 'consistently leads to better grounding' and that the bounding-box method's precision is 'often significantly higher,' but these claims cannot be assessed from the figures alone. The authors should report a summary statistic such as AUC or precision at a fixed recall, with bootstrap confidence intervals, for both datasets and both models.

minor comments (5)

[Appendix C] The adversarial validation set size is given as 1,500 examples, stated as 250 × 6, but Table 1 lists seven categories (Action, Attribute, Comparison, Count, Environment, Relation, OCR); either a category is omitted from the count or the arithmetic is incorrect.
[Table 1 caption] The caption defines bold and underline as 'significantly better' and 'marginal,' respectively, but no significance test is described anywhere in the paper; please define the criterion or remove the implication.
[Abstract and §6.1] The abstract's claim of improvement 'across diverse categories' is too broad given that Count is explicitly worse than output probabilities in both models; the authors should temper the wording to reflect the category-dependent results.
[§4.2.2] The bounding-box search enumerates all W^2 × H^2 boxes, but the patch grid resolution for the two tested models is not stated; please provide the grid sizes and a note on computational cost.
[Appendix D] The qualitative examples are helpful, but the top-k patch selection thresholds used in Figures 6–9 are not specified, which makes it hard to interpret the visualizations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the empirical comparison is self-contained and the self-citations are motivational rather than load-bearing.

full rationale

ContextualLens's scoring rule (average answer-token embedding, patch-wise cosine similarity, max confidence) is stated in Section 4.2.1 as a proposed detector and is evaluated against external ground truth on the HQH benchmark; it is not derived from the benchmark labels by construction. The layer pair is selected on a separate 250-example validation split and then reported on a held-out 250-example test split (Section 6.1, Appendix C), which is hyperparameter tuning rather than a fitted quantity that forces the reported mAP. The only self-citation with overlapping authors, Phukan et al. (2024), is used to motivate intermediate-layer contextual embeddings and the cosine-similarity scoring recipe; it is not invoked as a proof of the hallucination-detection result, and the paper's own experiments (Table 1, Figures 4-5) plus an adversarial layer-selection check (Appendix C) carry the load. The limitations section honestly concedes that output probabilities outperform ContextualLens on Count, which is inconsistent with a forced result. Concerns about missing error bars or small margins are statistical-evidence issues, not circularity, and per the review rules belong under correctness risk rather than a circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on several empirical assumptions rather than derived laws. The only numeric degrees of freedom are layer indices, selected on validation splits; their values are not fully disclosed for both models and datasets. No new theoretical entities are introduced.

free parameters (3)

Text layer lT (hallucination detection) = InternLM-VL: 27 (Appendix C)
Selected per category on HQH validation splits; results in Table 1 depend on this choice. No complete layer list for Qwen is given.
Image layer lI (hallucination detection) = InternLM-VL: 13 (Appendix C)
Selected on validation splits; sensitivity is only partially tested.
Bounding box layer lb (grounding) = Not specified
The bounding box method uses an unspecified layer lb; the paper does not report how this layer is chosen or its sensitivity.

assumptions (4)

domain assumption Middle-layer embeddings are contextual and represent multi-token concepts better than unembedding-layer tokens.
Relies on Phukan et al. (2024) and Wendler et al. (2024); not re-validated inside this paper for VLMs.
domain assumption Cosine similarity between averaged answer token embeddings and image patch embeddings is a valid proxy for visual grounding and hallucination confidence.
Central geometric assumption of the method; no calibration or analysis of embedding geometry is provided.
domain assumption The HQH benchmark labels and GPT-3.5-based evaluation are reliable ground truth for hallucination.
Adopted from Yan et al. (2024) without independent verification.
domain assumption The adapted logit lens baseline faithfully represents Jiang et al. (2024) for the categories tested.
The authors adapt the baseline themselves; minor implementation differences could affect the comparison.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs." pith.science (2026). https://pith.science/paper/5XMO6D65

@misc{pith2026241119187,
  author       = {Pith},
  title        = {Pith review of: Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5XMO6D65}},
  note         = {Machine review of arXiv:2411.19187}
}

read the original abstract

The rapid development of Large Multimodal Models (LMMs) has significantly advanced multimodal understanding by harnessing the language abilities of Large Language Models (LLMs) and integrating modality-specific encoders. However, LMMs are plagued by hallucinations that limit their reliability and adoption. While traditional methods to detect and mitigate these hallucinations often involve costly training or rely heavily on external models, recent approaches utilizing internal model features present a promising alternative. In this paper, we critically assess the limitations of the state-of-the-art training-free technique, the logit lens, in handling generalized visual hallucinations. We introduce ContextualLens, a refined method that leverages contextual token embeddings from middle layers of LMMs. This approach significantly improves hallucination detection and grounding across diverse categories, including actions and OCR, while also excelling in tasks requiring contextual understanding, such as spatial relations and attribute comparison. Our novel grounding technique yields highly precise bounding boxes, facilitating a transition from Zero-Shot Object Segmentation to Grounded Visual Question Answering. Our contributions pave the way for more reliable and interpretable multimodal models.

Figures

Figures reproduced from arXiv: 2411.19187 by the authors.

**Figure 1.** Visual Hallucination Detection. Logit lens, which only verifies the presence of the token "Green" in the image, mistakenly considers the answer correct. In contrast, ContextualLens assigns a low score to "Green" as it correctly contextualizes the color to the jacket rather than the hair. transferred to LMMs (Bai et al., 2024). Moreover, LMMs introduce additional hallucinations specific to the integrated modalities (… view at source ↗

**Figure 2.** (a) We extract latent representations of image patches and answer tokens from intermediate layers, (b) [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Qualitative examples for Grounded Visual Question Answering. Here is a GUI where we pass an image [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: PR Curves on the TextVQA-X dataset [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: PR Curves on the VizWiz-G dataset. general location of the evidence within the image. This need parallels the concept of attribution explored in LLMs and textual domains. We term this broader application in VLMs as multimodal attribution, where precision is more criti…

**Figure 6.** Figure 6: Top 20 image patches while detecting hallucination in [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Top 5 image patches while detecting hallucination in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Top 100 image patches while detecting hallucination in [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

**Figure 9.** Figure 9: Top 20 image patches while detecting hallucination in [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Additional qualitative examples for Grounded Visual Question Answering. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 9 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it's lying. arXiv preprint arXiv:2304.13734

arXiv 2023
[4]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023
[5]

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930

arXiv 2024
[6]

Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037

arXiv 2022
[7]

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024 a . Inside: Llms' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744

arXiv 2024
[8]

Chongyan Chen, Samreen Anjum, and Danna Gurari. 2022. Grounding answers for visual questions asked by visually impaired people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19098--19107

work page 2022

Show all 46 references

[9]

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Com...

2024
[10]

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883

2023 arXiv
[11]

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. 2024. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420

2024 arXiv
[12]

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2022. Rarr: Researching and revising what language models say, using language models. arXiv preprint arXiv:2210.08726

2022 arXiv
[13]

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627

2023 arXiv
[14]

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Con...

2024
[15]

Siqing Huo, Negar Arabzadeh, and Charles Clarke. 2023. Retrieving supporting evidence for generative question answering. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 11--20

2023
[16]

Jitesh Jain, Jianwei Yang, and Humphrey Shi. 2024. Vcoder: Versatile vision encoders for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27992--28002

2024
[17]

Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06271

2023 arXiv
[18]

Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. 2024. Interpreting and editing vision-language representations to mitigate hallucinations. arXiv preprint arXiv:2410.02762

2024 arXiv
[19]

Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. 2024. Efficient multimodal large language models: A survey. arXiv preprint arXiv:2405.10739

2024
[20]

Seyedalireza Khoshsirat and Chandra Kambhamettu. 2023. Sentence attention blocks for answer grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6080--6090

2023
[21]

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300

2019 arXiv
[22]

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. 2024. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

2024
[23]

Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024 a . A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253

2024 arXiv
[24]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

2024
[25]

nostalgebraist. 2020. Interpreting gpt: The logit lens. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. Aug 2020

2020
[26]

Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023 a . Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813

2023 arXiv
[27]

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023 b . Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824

2023 arXiv
[28]

Anirudh Phukan, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. 2024. Peering into the mind of language models: An approach for attribution in contextual question answering. arXiv preprint arXiv:2405.17980

2024 arXiv
[29]

Jirui Qi, Gabriele Sarti, Raquel Fern \'a ndez, and Arianna Bisazza. 2024. Model internals-based answer attribution for trustworthy retrieval-augmented generation. arXiv preprint arXiv:2406.13663

2024 arXiv
[30]

Pritika Ramu, Koustava Goswami, Apoorv Saxena, and Balaji Vasan Srinivavsan. 2024. Enhancing post-hoc attributions in long document comprehension via coarse grained answer decomposition. arXiv preprint arXiv:2409.17073

2024 arXiv
[31]

Varun Nagaraj Rao, Xingjian Zhen, Karen Hovsepian, and Mingwei Shen. 2021. A first look: Towards explainable textvqa models via visual and textual explanations. arXiv preprint arXiv:2105.02626

2021 arXiv
[32]

Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. 2024. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

2024
[33]

Abhilasha Sancheti, Koustava Goswami, and Balaji Vasan Srinivasan. 2024. Post-hoc answer attribution for grounded and trustworthy long document comprehension: Task, insights, and challenges. arXiv preprint arXiv:2406.06938

2024 arXiv
[34]

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih. 2023. Trusting your evidence: Hallucinate less with context-aware decoding. arXiv preprint arXiv:2305.14739

2023 arXiv
[35]

Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. 2024. Unsupervised real-time hallucination detection based on the internal states of large language models. arXiv preprint arXiv:2403.06448

2024 arXiv
[36]

Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. arXiv preprint arXiv:2210.01296

2022 arXiv
[37]

Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490

2019 arXiv
[38]

Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2023. Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08401

2023 arXiv
[39]

SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313

2024 arXiv
[40]

Aisha Urooj, Hilde Kuehne, Kevin Duarte, Chuang Gan, Niels Lobo, and Mubarak Shah. 2021. Found a reason for me? weakly-supervised grounded visual question answering using capsules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8465--8474

2021
[41]

Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. arXiv preprint arXiv:2307.03987

2023 arXiv
[42]

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

2024 arXiv
[43]

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. arXiv preprint arXiv:2402.10588

2024 arXiv
[44]

Bei Yan, Jie Zhang, Zheng Yuan, Shiguang Shan, and Xilin Chen. 2024. Evaluating the quality of hallucination benchmarks for large vision-language models. arXiv preprint arXiv:2406.17115

2024
[45]

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579--5588

2021
[46]

Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. 2024. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14227--14238

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it's lying. arXiv preprint arXiv:2304.13734

arXiv 2023

[4] [4]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023

[5] [5]

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930

arXiv 2024

[6] [6]

Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint arXiv:2212.08037

arXiv 2022

[7] [7]

Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024 a . Inside: Llms' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744

arXiv 2024

[8] [8]

Chongyan Chen, Samreen Anjum, and Danna Gurari. 2022. Grounding answers for visual questions asked by visually impaired people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19098--19107

work page 2022