AggTruth: Contextual Hallucination Detection using Aggregated Attention Scores in LLMs

Jan Eliasz; Jan Koco\'n; Konrad Kie{\l}czy\'nski; Miko{\l}aj Langner; Piotr Matys; Przemys{\l}aw Kazienko; Teddy Ferdinan

REVIEW 2 major objections 5 minor 20 references

AggTruth: Contextual Hallucination Detection using Aggregated Attention Scores in LLMs

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AggTruth detects contextual hallucinations in LLM outputs online by aggregating attention scores over the source passage, and it transfers across QA and summarization tasks and across four LLMs.

desk verdict Useful passage-only attention aggregation for hallucination detection, but the headline AUROCs are likely optimistic because best configurations were selected on test-set Gap. read the letter →

arxiv 2506.18628 v1 pith:X2EBIQY6 submitted 2025-06-23 cs.AI cs.CL

Piotr Matys , Jan Eliasz , Konrad Kie{l}czyński , Miko{l}aj Langner , Teddy Ferdinan , Jan Kocoń , Przemys{l}aw Kazienko This is my paper · ORCID

classification cs.AIcs.CL

keywords hallucinationdetectionattentionmapretrieval-augmentedgenerationcontextuallargelanguagemodelsfeatureselectiononlineaggregation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces AggTruth, a detector that watches the attention an LLM pays to the provided context passage while generating each token, and turns that attention into a small set of features for classifying tokens as hallucinated or faithful. It claims this passage-only signal is enough to detect contextual hallucinations in real time, during decoding, and that the method stays accurate when moving between question answering and summarization and across four different LLMs. The authors test four aggregation techniques: summing the attention placed on the passage, cosine similarity between heads, entropy of the attention distribution, and Jensen-Shannon distance from the layer's average attention. They report that AggTruth matches or beats the previous attention-map baseline on most settings while being cheaper, and that selecting a subset of attention heads can improve performance. A reader should care because online hallucination detection is a prerequisite for interventions like stopping, correcting, or regenerating untruthful output in retrieval-augmented generation systems.

What carries the argument

The mechanism is the passage-only attention aggregate: for each layer $l$, head $h$, and generated token $t$, the attention scores over the context tokens are reduced to one scalar by one of four formulas. Sum simply adds the attention placed on passage tokens; CosSim averages the cosine similarity between a head and all other heads in the same layer; Entropy treats the attention vector as a pseudo-probability distribution; and JS-Div measures the Jensen-Shannon distance between a head's attention and the layer's average head, with an extra appended value so the pseudo-distribution sums to one. These scalars become features for an eight-token sliding window, and the window label is positive if GPT-4o judged any token in it hallucinated; a logistic-regression classifier then makes the final decision. The paper's additional machinery is head selection: it reduces the feature set via Spearman correlation with the target, a simplified Boruta-style 'above random' selector, Lasso, or a center-ratio baseline.

What would settle it

Re-annotate a random sample of, say, 400 responses with token-level human labels, ideally with two or more annotators per token, and recompute the AUROC of the best AggTruth configuration against these human labels instead of GPT-4o labels. If the AUROC drops to near random, the method is detecting the judge's labeling style rather than hallucination; if it stays high, the central claim survives.

Watch

Extended reading notes

Core claim

The central discovery is that you do not need the full attention map, hidden states, or multiple sampled generations to detect contextual hallucinations: the attention scores the model assigns to the provided passage, aggregated per head per generated token, carry enough signal. AggTruth reduces the attention tensor to a single scalar per layer-head for each generated token, by summing the scores over passage tokens, by measuring how similar each head is to its layer-mates, or by treating the attention distribution as a pseudo-probability and measuring its entropy or Jensen-Shannon distance to the layer average. These per-token features are then windowed into overlapping chunks of eight tokens, labeled by GPT-4o as containing a hallucination or not, and fed to a logistic-regression classifier. In same-task and cross-task tests across four LLMs on Natural Questions, HotPotQA, CNN/Daily Mail, and XSum, the authors find that AggTruth variants achieve small gaps between source and target test performance and outperform hidden-state classifiers and, where comparable, the Lookback Lens baseline. The paper also finds that selecting a subset of attention heads, often 10 to 50 percent, can improve or preserve performance compared with using all heads.

Load-bearing premise

Every reported accuracy figure assumes GPT-4o's token-level hallucination labels are trustworthy; the only human check covered 75 samples and reached moderate agreement (a Cohen's Kappa of 0.7), so if GPT-4o's judgment is biased, the detector is really predicting GPT-4o's notion of hallucination rather than ground truth.

Editorial extensions

If this is right

A retrieval-augmented generation system using AggTruth could detect hallucinated tokens while the answer is still being generated, making post-hoc correction unnecessary.
Because the features come only from attention on the source passage, the detector transfers between question answering and summarization and across four different LLMs without target-task retraining.
Selecting a subset of attention heads, often between 10 and 50 percent, can match or slightly beat using all heads, so the classifier can be trained and run more cheaply.
The Sum aggregation is the recommended default: it is the fastest, most interpretable, and most stable variant across the tested settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: because AggTruth only needs attention onto a reference passage, the same recipe could apply to other grounded-generation tasks such as dialogue grounded in retrieved documents, clinical note generation, or legal citation, but the paper only tests QA and summarization, so this is an open question.
Beyond the paper: all responses were generated with greedy decoding; under sampling or higher temperatures the attention distributions would shift, so thresholds and head subsets would likely need recalibration, a setting the paper does not evaluate.
Beyond the paper: the paper itself notes that AggTruth requires access to attention scores for context tokens, which flash-attention and sparse or window attention implementations do not offer, so deployment is restricted to models that expose raw attention maps, and those regimes are left untested.
Beyond the paper: since GPT-4o supplied the training labels, a fully human-annotated benchmark would clarify whether AggTruth detects unfaithfulness itself or merely predicts the judge's labeling style.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You should know two things about this one. First, the core idea is genuinely useful: AggTruth restricts attention maps to the provided passage and aggregates them with four simple functions, and it works surprisingly well as a windowed, online hallucination detector across four LLMs and four datasets. Second, the reported “high and stable” results are probably optimistic, because the authors picked the best feature selector and head percentage per model/task based on the test-set Gap value. That is selection-on-test, and it undermines the central outperformance claim.

What’s new: the passage-only attention restriction, the four aggregation methods (Sum, CosSim, Entropy, JS-Div), and the feature selection comparison across four LLMs. The Lookback Lens baseline is honestly cited, and the paper clearly positions itself as a modification. The head-selection analysis is also useful: they show that using all heads is not always best, which contradicts Lookback Lens’s finding. The experiments cover more models than most papers in this area.

Where it goes soft. Label quality is a concern: GPT-4o produces token-level labels, with only 75 human-checked samples and Kappa 0.7. The classifier learns GPT-4o’s notion of hallucination, not necessarily truth. That alone should make you cautious. But the bigger problem is the selection protocol. Tables 5 and 6 report “best obtained results w.r.t. Gap value,” and Gap is computed on the test sets. They explore multiple selectors (Center, Random, Random+, Lasso, Spearman) and head percentages, then present the minimum-Gap configuration as the method’s performance. Without validation-based selection or multiple-comparison correction, those numbers are expected to be optimistic. The stress-test note gives a concrete example: for Llama-2 QA→SUM, Random+3,3 lowers Gap from 6.836 to 2.048. That is a large effect. There are also no error bars or significance tests anywhere, and the Lookback Lens baseline is only run on Llama-2 because the released code doesn’t generalize.

The paper is worth a serious referee, because the method is novel and the experiments are extensive. But it needs major revision: release code and data, pre-register selectors or select them on validation folds, report variance across runs, and validate a larger sample of GPT-4o labels. With those fixes, the central claim might hold. As written, take the headline numbers with a grain of salt.

Recommendation: send to peer review, but expect heavy revision.

Referee Report

2 major / 5 minor

Summary. The paper proposes AggTruth, an online, windowed hallucination detector for RAG settings that uses attention scores restricted to the provided passage, aggregated via four techniques (Sum, CosSim, Entropy, JS-Div). A Logistic Regression classifier is trained on token-level labels produced by GPT-4o, with features optionally selected by several head-selection methods (Center, Random/Random+, Lasso, Spearman). The method is evaluated on QA and summarization datasets across four LLMs, comparing against Lookback Lens and hidden-state baselines in same-task and cross-task settings. The authors report that AggTruth achieves high and stable AUROC values and outperforms baselines in multiple scenarios, and that head selection can improve performance while reducing the number of heads.

Significance. If the reported results hold, AggTruth offers a practical, interpretable, and computationally light approach to detecting contextual hallucinations during decoding, with the passage-only attention aggregation being a simple and reusable feature. The strengths of the paper include evaluation across four LLMs and four datasets, comparison against a recent SOTA (Lookback Lens) and hidden-state baselines, and a systematic exploration of feature/head selection. However, the central claim of 'high and stable results' rests on two assumptions that need tightening: the labels are generated by GPT-4o with only a small human check, and the reported configurations are selected based on test-set Gap values, which risks optimistic bias. The paper's contribution is valuable, but these issues must be addressed before the headline claim is secure.

major comments (2)

[Section 5, Eq. (5), Table 5, Table 6] The central outperformance claim in Section 7 is based on Tables 5 and 6, which report 'best obtained results w.r.t. Gap value.' Since Gap (Eq. 5) is computed on the three test sets, the selection of the aggregation variant, selector, and head percentage is performed using test-set performance. This is selection on the test set, which inflates expected AUROC. For example, in Table 5, for Llama-2 QA→SUM, Random+3,3 reduces Gap from 6.836 to 2.048 relative to no selection; for Gemma-2 SUM→QA, Spearman0.1 reduces Gap from 8.642 to 5.901. No validation-based selection protocol or multiple-comparison correction is described. The paper should either fix the configuration using only training/validation data before computing test AUROC, or use nested cross-validation, and then report the resulting test numbers as the primary results.
[Section 4.1] All token-level labels are produced by GPT-4o, with human agreement checked on only 75 samples (Cohen's Kappa 0.7). If GPT-4o's judgments are biased or noisy, the classifier learns to predict GPT-4o's notion of hallucination rather than true unfaithfulness, and every reported AUROC inherits this label assumption. The paper should provide evidence that the GPT-4o labels are reliable on a larger human-annotated sample, or use human labels as ground truth for at least one full dataset to confirm the ranking of methods.

minor comments (5)

[Section 3.2] The description of the Center selector is ambiguous: 'keep the top r/2 fraction of heads with the highest ratio and r/2 fraction with the lowest ratio' should clarify whether r is the total fraction of selected heads and how ties are broken.
[Table 1] The column labeled 'truthfulness evaluation results' reports percentages that look like hallucination rates, but the caption does not state this explicitly. Please clarify whether higher percentages mean more hallucinations or more truthful responses.
[Figure 4 caption] The caption says 'best obtained hidden states-based method w.r.t. Gap value,' which again implies selection on the test set; this should be reworded to avoid reinforcing the impression of test-set selection in the main results.
[Section 4, Experimental Setup] The windowing procedure is described as 'overlapping chunks of size 8, sliding token by token,' but the degree of overlap and the label assignment for a chunk that contains both hallucinated and non-hallucinated tokens are not fully specified; please define this precisely.
[Throughout] There are numerous formatting typos (e.g., 'T able 1', 'AggT ruth' in Table 5, 'AUCROC' instead of 'AUROC') that should be corrected in a revision.

Circularity Check

0 steps flagged · score 1.0 of 10

No derivation-to-fit circularity; central claim rests on an empirical classifier comparison, with test-set-based selector choice and GPT-4o labels as validity (not circularity) concerns.

full rationale

AggTruth is a supervised classifier: the four aggregation features (Eqs. 1-4) and the passage-percentage feature are computed from attention maps, while the token-level labels are produced by GPT-4o independently of those features. There is therefore no equation whose output is defined by a fitted parameter or by the target label; the training/evaluation loop is a standard fit-and-test design against external baselines (Lookback Lens, hidden-state classifiers, and an NLI-based SOTA). The paper's self-citations ([4], [10]) are contextual and do not carry the derivation. The two genuine concerns are statistical rather than circular: (i) Tables 5-6 report 'best obtained results w.r.t. Gap value,' meaning the selector and head percentage are chosen by minimizing a Gap defined on the three test sets (Eq. 5), which can bias reported AUROCs upward; and (ii) token labels come from GPT-4o with only 75 human samples (Cohen's Kappa 0.7), so the classifier may learn GPT-4o's notion of hallucination rather than an independent ground truth. Neither concern makes AggTruth's output equivalent to its inputs by construction, so no circular step is identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities, only new feature aggregation functions. The load-bearing assumptions are the GPT-4o labeling, the token-alignment shift, the comparability of attention scores across heads, and the 4096-token context restriction. Free parameters are limited to window size, selector hyperparameters, and significance thresholds.

free parameters (3)

window size = 8
Chosen from Lookback Lens [2], not optimized; features are averaged over overlapping windows of 8 tokens in Section 4.
selector hyperparameters (r, n, k) = various, e.g., Center0.2, Random+3,3, Spearman1.0
Feature selection fractions and repetition counts are tuned per LLM and task; the best values in Table 5 are selected using test-set Gap values.
significance thresholds = p=0.001 for Spearman, p=0.01 for Welch's t-test
Hand-chosen thresholds for head selection and for the head analysis in Section 6.

assumptions (4)

domain assumption GPT-4o token-level labels are a valid ground truth for contextual hallucination
Used in Section 4.1 to create training and evaluation labels for all examples; only 75 samples were checked by a human, with Cohen's Kappa 0.7.
domain assumption Attention map from generating token t+1 describes token t
Section 3.3 hypothesizes this alignment correction; no experiment validates the shift, and all token-level features and labels depend on it.
domain assumption Attention scores can be aggregated arithmetically across context tokens and heads
Sum, cosine, entropy, and JS-divergence in Section 3.1 treat attention scores as comparable pseudo-probabilities, ignoring scale and positional differences.
domain assumption Sliding window attention can be handled by restricting context to 4096 tokens
Section 4.1 fixes context length at 4096 to allow attention extraction; results may not transfer to longer contexts or flash-attention inference.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AggTruth: Contextual Hallucination Detection using Aggregated Attention Scores in LLMs." pith.science (2026). https://pith.science/paper/X2EBIQY6

@misc{pith2026250618628,
  author       = {Pith},
  title        = {Pith review of: AggTruth: Contextual Hallucination Detection using Aggregated Attention Scores in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X2EBIQY6}},
  note         = {Machine review of arXiv:2506.18628}
}

read the original abstract

In real-world applications, Large Language Models (LLMs) often hallucinate, even in Retrieval-Augmented Generation (RAG) settings, which poses a significant challenge to their deployment. In this paper, we introduce AggTruth, a method for online detection of contextual hallucinations by analyzing the distribution of internal attention scores in the provided context (passage). Specifically, we propose four different variants of the method, each varying in the aggregation technique used to calculate attention scores. Across all LLMs examined, AggTruth demonstrated stable performance in both same-task and cross-task setups, outperforming the current SOTA in multiple scenarios. Furthermore, we conducted an in-depth analysis of feature selection techniques and examined how the number of selected attention heads impacts detection performance, demonstrating that careful selection of heads is essential to achieve optimal results.

Figures

Figures reproduced from arXiv: 2506.18628 by the authors.

**Figure 1.** Selection of tokens from the LLM input and aggregation of attention scores. The darker subregion indicates the attention scores of generated tokens on the provided passage. The aggregation of each such region provides one feature for the hallucination detection. Here, we have three regions resulting in three features. 3.1 Attention maps aggregation techniques For every response generated by a transformer, a four-dim… view at source ↗

**Figure 2.** The whole end-to-end pipeline. It begins with a Prompt passed to an LLM and eventually results in Selected Features based on which the final Classifier detects potential hallucinations in the obtained answer. 4.1 Data We first established a comprehensive dataset of language model responses with token-level annotations to train classifiers for hallucination detection. We restricted our analysis to 4, 096 LLM’s conte… view at source ↗

**Figure 3.** Percentage of heads for which the mean features (i.e. AggTruth Sum/JS-Div) are statistically higher for non-hallucinated examples than for the hallucinated ones. The comparison was conducted using Welch’s t–test with p–value = 0.01 Based on our research findings, we recommend the AggTruth Sum as the preferred default method. It is the most stable and robust approach that resulted in the lowest overall gap across al… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: AUCROC hallucination detection results grouped by LLMs and tasks for best obtained hidden states-based method w.r.t. Gap value, Lookback Lens (only for Llama2), AggTruth Sum and best obtained AggTruth method for a specific dataset and LLM. all the proposed aggregation…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

[1]

In: EMNLP

Azaria, A., et al.: The internal state of an LLM knows when it‘s lying. In: EMNLP

work page
[2]

Chuang, Y.S., et al.: Lookback lens: Detecting and mitigating contextual halluci- nations in large language models using only attention maps (2024)

work page 2024
[3]

Nature630(8017), 625–630 (Jun 2024)

Farquhar, S., et al.: Detecting hallucinations in large language models using se- mantic entropy. Nature630(8017), 625–630 (Jun 2024)

work page 2024
[4]

In: SENTIRE at ICDM’2024

Ferdinan, T., et al.: Into the unknown: Self-learning large language models. In: SENTIRE at ICDM’2024. pp. 423–432. IEEE (2024)

work page 2024
[5]

Gekhman, Z., et al.: Does fine-tuning llms on new knowledge encourage hallucina- tions? (2024), https://arxiv.org/abs/2405.05904

arXiv 2024
[6]

Gemma, T., et al.: Gemma 2: Improving open language models at a practical size (2024), https://arxiv.org/abs/2408.00118

arXiv 2024
[7]

ACM Trans

Huang, L., et al.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. on Inform. Sys. (2024)

work page 2024
[8]

ACM Com- puting Surveys 55(12), 1–38 (Mar 2023)

Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Com- puting Surveys 55(12), 1–38 (Mar 2023)

work page 2023

Show all 20 references

[9]

Ke, Z., et al.: Continual training of language models for few-shot learning (2022), https://arxiv.org/abs/2210.05549

2022 arXiv
[10]

Information Fusion 99, 101861 (2023)

Kocoń, J., et al.: Chatgpt: Jack of all trades, master of none. Information Fusion 99, 101861 (2023)

2023
[11]

Journal of statistical software36, 1–13 (2010)

Kursa, M.B., Rudnicki, W.R.: Feature selection with the boruta package. Journal of statistical software36, 1–13 (2010)

2010
[12]

Transactions of the ACL7, 452–466 (2019)

Kwiatkowski, T., et al.: Natural questions: A benchmark for question answering research. Transactions of the ACL7, 452–466 (2019)

2019
[13]

Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks (2021), https://arxiv.org/abs/2005.11401

2021 arXiv
[14]

Manakul, P., et al.: Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models (2023), https://arxiv.org/abs/2303.08896

2023 arXiv
[15]

Narayan, S., et al.: Don’t give me the details, just the summary! topic-aware con- volutional neural networks for extreme summarization (2018)

2018
[16]

In: Findings of the Association for Computational Linguistics: EMNLP 2023 (2023)

Peng, B., Alcaide, E., et al.: Rwkv: Reinventing rnns for the transformer era. In: Findings of the Association for Computational Linguistics: EMNLP 2023 (2023)

2023
[17]

In: ACL 2017

See, A., et al.: Get to the point: Summarization with pointer-generator networks. In: ACL 2017. pp. 1073–1083. ACL (2017)

2017
[18]

Yang, Z., et al.: Hotpotqa: A dataset for diverse, explainable multi-hop question answering (2018), https://arxiv.org/abs/1809.09600

2018 arXiv
[19]

0.858 0.853 0.731 0.587 0.713 5.639 Spearman1.0 SUM

Zhao, W.X., et al.: A survey of large language models (2024), https://arxiv.org/abs/2303.18223 A Appendix LLM AggT ruth Selector Source T arget Source T arget Gap [%] T rain V al T est T est (1) T est (2) llama-2-7b- chat-hf CosSim Lasso QA SUM. 0.858 0.853 0.731 0.587 0.713 5...

2024 arXiv
[2023]

pp. 967–976. ACL, Singapore (Dec 2023)

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

In: EMNLP

Azaria, A., et al.: The internal state of an LLM knows when it‘s lying. In: EMNLP

work page

[2] [2]

Chuang, Y.S., et al.: Lookback lens: Detecting and mitigating contextual halluci- nations in large language models using only attention maps (2024)

work page 2024

[3] [3]

Nature630(8017), 625–630 (Jun 2024)

Farquhar, S., et al.: Detecting hallucinations in large language models using se- mantic entropy. Nature630(8017), 625–630 (Jun 2024)

work page 2024

[4] [4]

In: SENTIRE at ICDM’2024

Ferdinan, T., et al.: Into the unknown: Self-learning large language models. In: SENTIRE at ICDM’2024. pp. 423–432. IEEE (2024)

work page 2024

[5] [5]

Gekhman, Z., et al.: Does fine-tuning llms on new knowledge encourage hallucina- tions? (2024), https://arxiv.org/abs/2405.05904

arXiv 2024

[6] [6]

Gemma, T., et al.: Gemma 2: Improving open language models at a practical size (2024), https://arxiv.org/abs/2408.00118

arXiv 2024

[7] [7]

ACM Trans

Huang, L., et al.: A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. on Inform. Sys. (2024)

work page 2024

[8] [8]

ACM Com- puting Surveys 55(12), 1–38 (Mar 2023)

Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Com- puting Surveys 55(12), 1–38 (Mar 2023)

work page 2023