Pith. sign in

REVIEW 4 major objections 7 minor 78 references

Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hidden states that shake hardest flag truthful answers

desk verdict A solid supervised hallucination detector with a plausible story, but the paper overstates the intrinsic-sensitivity claim and needs a frozen-baseline check. read the letter →

arxiv 2506.02696 v1 pith:6HGETMJU submitted 2025-06-03 cs.AI

Jinyuan Luo , Zhen Fang , Yixuan Li , Seongheon Park , Ling Chen This is my paper
classification cs.AI
keywords hallucinationdetectionself-assessmentintermediaterepresentationsperturbationsensitivitypromptlearningcontrastivelargelanguagemodelsAUROCevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a large language model's intermediate representations react differently to small input perturbations depending on whether the answer it is processing is factually grounded: truthful answers show larger shifts, hallucinated ones stay comparatively stable. It builds Sample-Specific Prompting (SSP) on this idea: for each question–answer pair, a small network generates a noise prompt, and a lightweight encoder amplifies the cosine gap between the original and perturbed hidden states. This gap is the truthfulness score, so detection uses no output confidence, no repeated sampling, and no editing of the model itself. The paper reports AUROC scores above the compared baselines, for example 73.43 on the TruthfulQA benchmark with an 8-billion-parameter instruction-tuned model, and shows the learned prompts and encoder transfer across datasets. If the claim is right, dynamic sensitivity of internal states is a practical signal for hallucination detection.

What carries the argument

The load-bearing object is the perturbation-induced representation gap $\Delta E_\theta(Q, A, N, T) = \mathrm{Disc}(E_\theta(x=\mathrm{True}|Q,A,T), E_\theta(x=\mathrm{True}|Q,A,N,T))$, where the noise prompt $N$ is a short, semantically neutral sentence appended to the answer and $\mathrm{Disc}(z,\tilde z)=1-\cos(z,\tilde z)$ is the cosine discrepancy. The machinery is the joint learning of the sample-specific prompt generator $M_\varphi(h)$, which adds a learned embedding to a seed noise prompt, and the shared encoder $f_\phi(\cdot)$; both are trained by the contrastive objective of Eq. (16), while the LLM backbone stays frozen. The generator tailors the perturbation to each input, the encoder amplifies the separation, and the cosine distance converts hidden-state changes into a scalar score.

What would settle it

Re-run SSP's training and evaluation on the same questions and answers but with truthfulness labels assigned by human annotators rather than by an external similarity-scoring model; if the AUROC over a large labeled set is no better than the static linear-probe baseline (68.65 on TruthfulQA in the paper's table), the perturbation-sensitivity signal is an artifact of the teacher's labels rather than a property of factual answers.

Watch

Extended reading notes

Core claim

The central claim is inequality (7): there exist noise prompts $N$ such that the perturbation-induced representation gap satisfies $\Delta E_\theta(Q, A_\mathrm{Truth}, N, T) > \Delta E_\theta(Q, A_\mathrm{Hallu}, N, T)$. In words, factual answers are more sensitive than fabricated ones to an input perturbation at intermediate layers. SSP operationalizes the inequality by learning, for every sample, a noise prompt embedding through a two-layer MLP conditioned on the sentence embedding of the question–answer pair, then computing $\mathrm{Disc}(z, \tilde z) = 1 - \cos(z, \tilde z)$ between the hidden states before and after the perturbation. A shared encoder $f_\phi$ maps both states into a latent space, and a contrastive loss maximizes the gap for truthful samples while minimizing it for hallucinated ones. The paper reports that this scoring function yields average AUROC of 75.38 on the compared benchmarks with one instruction-tuned model, besting all twelve baselines, and that the learned prompts transfer across datasets.

Load-bearing premise

The labels that tell the method which answers are truthful and which are hallucinated are produced by an external language model scoring semantic similarity to a reference at a fixed threshold, so every reported accuracy is relative to that judge's notion of truth rather than to independently verified factuality.

Editorial extensions

If this is right

  • SSP detects hallucinations with a single forward pass before and after perturbation, avoiding the cost of sampling multiple responses per question that consistency-based methods require.
  • Because the signal lives in intermediate representations, it remains usable even when the model's final output distribution is miscalibrated or biased.
  • The learned sample-specific prompts transfer across datasets, so a detector trained on one domain can be applied to another without retraining.
  • The middle layers of the network carry the most discriminative signal; performance degrades when representations are taken from the earliest or final layers.
  • An ablation reversing the contrastive direction (encouraging small shifts for truthful answers) cuts average AUROC from 75.38 to 67.57, confirming the directional assumption is load-bearing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not pursue is applying the same discrepancy gap to sentence-level units within a long generation, which could localize the hallucinated span; the paper states it cannot localize incorrect tokens.
  • Because the teacher labels come from an external similarity model, a natural robustness check would be to re-run training and evaluation with human-verified factuality labels; the paper's current numbers cannot distinguish signal from label artifact.
  • The perturbation gap could be combined with output-level confidence into a two-signal detector, which might improve calibration in settings where representation sensitivity and verbalized confidence disagree.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes Sample-Specific Prompting (SSP), a hallucination detector that, for each question-answer pair, generates a sample-specific noise prompt (initialized by the LLM from a SeedPrompt and refined by an MLP applied to token embeddings), appends it to the input, and scores truthfulness by the cosine discrepancy between the LLM's intermediate representation before and after perturbation, using a lightweight trained encoder to amplify the difference. The encoder and prompt generator are trained on 100 labeled QA pairs with a contrastive hinge loss. Experiments on four QA datasets and two LLM families report average AUROC values of 75.38 (LLaMA-3-8B-Instruct) and 72.72 (Qwen2.5-7B-Instruct), outperforming 12 baselines, together with ablations, cross-dataset transfer experiments, and an inference-time comparison.

Significance. If the results hold, the paper offers a practical supervised hallucination detector and a useful empirical finding that learned sample-specific perturbations can make intermediate representations more separable. The strengths are the breadth of evaluation (12 baselines, two LLM families, four datasets), the reversed-objective ablation in Appendix F, which supports the training direction, the cross-dataset transfer study in Figure 2, and the reported inference-time analysis. The main caveat is interpretative: the training objective explicitly optimizes the same discrepancy that is later reported as a detection score, so the AUROC values do not by themselves validate Eq. (7) as a pre-existing property of LLM representations. The external-teacher labeling and the absence of variance or significance reporting further qualify the absolute performance claims. These issues are addressable with additional experiments and rewording rather than being fundamental to the proposed training scheme.

major comments (4)
  1. [§4.3, Eq. (16) and §3.2, Eq. (7)] The training objective in Eq. (16) directly optimizes the inequality that Eq. (7) posits as an intrinsic phenomenon: for truthful samples it pushes cos(z, z̃) below τT, and for hallucinated samples it pushes cos(z, z̃) above τH. The reported held-out AUROC therefore demonstrates that a per-sample prompt generator and encoder can learn to separate the training labels, but it does not by itself show that truthful and hallucinated answers exhibit different perturbation sensitivity before any training. The ablations in Table 2 do not isolate this: the 'Static prompt' row (70.93) still uses the label-trained encoder, and 'SSP w/o Encoder' (65.82) still uses the label-trained prompt generator. I request a fully frozen baseline in which the SeedPrompt-generated noise is used with no learned Mφ and no learned fφ (raw cosine distance on a frozen representation), plus an analysis of the semantic change introduced by the learned perturbation. If the frozen baseline is weak, the novelty claim in Section 1 ('first to leverage the sensitivity of LLMs to input perturbations') and the motivation grounded in Eq. (7) should be restated as a supervised method that learns sample-specific perturbations, not as evidence of a pre-existing intrinsic signal.
  2. [§5.1 and Appendix C] Both the training labels and the evaluation labels are assigned by DeepSeek-V3 using a semantic-similarity score thresholded at 0.5. Every AUROC in Tables 1-3 and Figures 2-3 is therefore a measure of agreement with this teacher's notion of truthfulness rather than with verified factuality, and the same teacher is used for generating the training labels and for scoring the test set. There is no human-annotation sanity check or error analysis of the teacher labels. I request a manually annotated subset (e.g., 100-200 examples per dataset) with reported agreement between DeepSeek-V3 and human labels, and ideally an evaluation where the teacher used for training differs from the teacher used for testing. Without this, the absolute detection accuracy claims are not established.
  3. [§5.2, Table 1] The paper reports single-run AUROC values without variance, confidence intervals, or significance tests, yet repeatedly claims that SSP 'significantly outperforms' baselines (Sections 1 and 5.2). The margins over Linear probe (75.38 vs 71.66 for LLaMA-3-8B-Instruct; 72.72 vs 67.98 for Qwen2.5-7B-Instruct) are plausible but need support. Please report mean and standard deviation over at least five random initializations of Mφ and fφ, and include paired significance tests or bootstrap confidence intervals on the AUROC differences for the main comparisons.
  4. [§4.2, Eq. (12) and §4.3] The semantic-preservation constraint is only an instruction in the SeedPrompt (Appendix D); nothing in the loss in Eq. (16) prevents Mφ(h) from moving the noise-prompt embedding in arbitrary directions that the label-trained encoder then amplifies. The claim that SSP measures sensitivity to 'stylistic' or 'semantically neutral' perturbations (Section 4.2) is therefore untested. Please add an analysis of the learned perturbations, for example by decoding the updated embedding or measuring the semantic similarity between the SeedPrompt-generated noise prompt before and after optimization, and include a variant in which Mφ is constrained (e.g., by a norm bound or a semantic-similarity regularizer) to verify that the performance does not rely on arbitrary adversarial embeddings.
minor comments (7)
  1. [§4.1, Eq. (8)] The notation Eθ(x=True|Q,A,T) is confusing, as the representation is not conditioned on the answer being true; please use a neutral notation such as h(Q,A,T) for the intermediate representation.
  2. [§5.1 and §5.4] The layer index used for feature extraction in the main experiments is not specified; Figure 3(a) shows a layer sweep, but Table 1 does not state which layer was used for the reported numbers. Please state the exact layer.
  3. [§5.3, Figure 2] The text cites an in-domain TruthfulQA AUROC of 78.64%, but Table 1 and Figure 2(a) report 73.43% for the same configuration; please correct this inconsistency.
  4. [Figure 2 caption] The caption should state the matrix orientation explicitly (source dataset rows versus target dataset columns) and note that all values are AUROC percentages.
  5. [Table 2] Please clarify what replaces the SeedPrompt in the 'SSP w/o SeedPrompt' row (e.g., random initialization) and what the 'Static prompt' row uses as its fixed perturbation; the current caption leaves both ambiguous.
  6. [Appendix K] The Limitations section should acknowledge the dependence on external teacher labels and the requirement of labeled training pairs for the prompt generator and encoder.
  7. [General] Please include a reproducibility statement indicating whether the code, learned prompts, and human-annotation data will be released.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the supervised training objective matches the test score by design, but held-out evaluation and external baselines keep the central empirical claim independent.

full rationale

The paper's load-bearing claim is Equation (7), an existential expectation that some noise prompts produce larger representational shifts for truthful than for hallucinated answers. This is stated as a motivating hypothesis ('we expect'), not derived from the training labels; the method then trains the prompt generator M_phi and encoder f_phi with the contrastive loss (Eq. 16) whose terms are exactly the cosine discrepancy (Eq. 9) used as the test score. This means the training objective and scoring function coincide, but that is a standard supervised design rather than a circular reduction: the reported AUROC numbers are computed on held-out test samples whose labels are not used to fit M_phi or f_phi, and all baselines are evaluated under the same label protocol, so the comparison is external evidence. The only self-citations are HaloScope [9] and the latent-overlap study [11] by two of the present authors; both are used for experimental setup or motivation, not as a uniqueness theorem or as the justification for Equation (7). The DeepSeek-V3 label assignment is a shared evaluation protocol and a potential validity risk, but it applies equally to SSP and baselines and is not a by-construction equivalence between an input and a prediction. No quoted reduction of a result to its own definition, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain were found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

SSP is a supervised method: the target relationship (Eq. 7) is enforced by the contrastive objective, so the only external evidence is the held-out AUROC. The free parameters are the thresholds, the layer choice, the MLP weights, and the SeedPrompt template. No new physical entities are introduced; the noise prompt is an internal, learned artifact without independent falsifiable evidence.

free parameters (4)
  • Discrepancy thresholds (τT, τH) = 0.3 and 0.7
    Chosen from a sweep on TriviaQA (Figure 3b); the paper does not describe a validation split, and these values are applied across all datasets.
  • Representation layer index for scoring = Not reported
    Figure 3a shows AUROC varies from about 55 to 79 depending on layer, but the main experiments never state which layer is used; this choice materially affects results.
  • Weights of encoder fϕ and prompt generator Mϕ = Not reported
    Trained on 100 labeled QA pairs per dataset with SGD for 40 epochs; these learned parameters determine the detector and are not released.
  • SeedPrompt template = Hand-written instruction in Appendix D
    The nature of the generated noise prompt depends on this template, which is a design choice not derived from theory.
assumptions (4)
  • domain assumption Truthful and hallucinated responses exhibit distinct perturbation-induced representation shifts: ΔEθ(Q,ATruth,N,T) > ΔEθ(Q,AHallu,N,T) for some noise prompt N (Eq. 7).
    This is the central hypothesis of the method; the paper motivates it with references to factual knowledge being structured, but provides no proof or mechanistic test.
  • domain assumption Intermediate representations are less influenced by model bias than output distributions.
    Section 3.1 argues bias accumulates across layers, so middle layers are more faithful; this is a qualitative claim cited from prior work, not verified here.
  • domain assumption DeepSeek-V3 similarity score at least 0.5 correctly identifies a generation as truthful.
    Appendix C and Section 5.1 use this threshold to build training labels and evaluation labels; no human validation is reported.
  • domain assumption The generated noise prompt preserves semantics and does not introduce contradictions.
    Appendix D instructs the LLM to avoid changing meaning, but the paper does not measure whether the constraint holds or whether violations affect the discrepancy score.
invented entities (1)
  • Sample-specific noise prompt N (text plus learned embedding adjustment)
    purpose: Injected into the input to induce a representational shift that separates truthful from hallucinated answers.
    The optimal form of N is a learned artifact of the trained prompt generator; the paper provides no external validation that N is semantically neutral or that the shift is caused by perturbation sensitivity rather than added-token artifacts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations." pith.science (2026). https://pith.science/paper/6HGETMJU

@misc{pith2026250602696,
  author       = {Pith},
  title        = {Pith review of: Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6HGETMJU}},
  note         = {Machine review of arXiv:2506.02696}
}
read the original abstract

Hallucination remains a key obstacle to the reliable deployment of large language models (LLMs) in real-world question answering tasks. A widely adopted strategy to detect hallucination, known as self-assessment, relies on the model's own output confidence to estimate the factual accuracy of its answers. However, this strategy assumes that the model's output distribution closely reflects the true data distribution, which may not always hold in practice. As bias accumulates through the model's layers, the final output can diverge from the underlying reasoning process, making output-level confidence an unreliable signal for hallucination detection. In this work, we propose Sample-Specific Prompting (SSP), a new framework that improves self-assessment by analyzing perturbation sensitivity at intermediate representations. These representations, being less influenced by model bias, offer a more faithful view of the model's latent reasoning process. Specifically, SSP dynamically generates noise prompts for each input and employs a lightweight encoder to amplify the changes in representations caused by the perturbation. A contrastive distance metric is then used to quantify these differences and separate truthful from hallucinated responses. By leveraging the dynamic behavior of intermediate representations under perturbation, SSP enables more reliable self-assessment. Extensive experiments demonstrate that SSP significantly outperforms prior methods across a range of hallucination detection benchmarks.

Figures

Figures reproduced from arXiv: 2506.02696 by the authors.

Figure 1
Figure 1. Overview of Sample-Specific Prompting (SSP) framework for hallucination detection. Given a question-answer (QA) pair, a noise prompt generator produces a perturbation adapted to the input. The noise prompt is appended to the original answer and passed through a shared LLM backbone to induce representational shifts. The encoder then maps the intermediate representations to a discriminative space and maximize the disc… view at source ↗
Figure 2
Figure 2. Generalization performance comparison across SSP (Ours), EGH [ [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Visualization of performance under different layers (left) and threshold settings (right). [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of discrepancy scores assigned by different prompting strategies. [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: AUROC and inference time. I Compute Resources and Time Software and hardware. We conducted all experiments using Python 3.9.20 and PyTorch 1.13.1 on NVIDIA A40 GPUs. For evaluation with DeepSeek-V3, we utilized the official API provided by DeepSeek. Inference time. To …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 16 canonical work pages

  1. [1]

    Language models (mostly) know what they know,

    S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield- Dodds, N. DasSarma, E. Tran-Johnsonet al., “Language models (mostly) know what they know,”arXiv preprint arXiv:2207.05221, 2022

    arXiv 2022

  2. [2]

    Do llms know about hallucination? an empirical investigation of llm’s hidden states,

    H. Duan, Y . Yang, and K. Y . Tam, “Do llms know about hallucination? an empirical investigation of llm’s hidden states,”arXiv preprint arXiv:2402.09733, 2024

    arXiv 2024

  3. [3]

    Learning to trust your feelings: Leveraging self-awareness in llms for hallucination mitigation,

    Y . Liang, Z. Song, H. Wang, and J. Zhang, “Learning to trust your feelings: Leveraging self-awareness in llms for hallucination mitigation,”arXiv preprint arXiv:2401.15449, 2024

    arXiv 2024

  4. [4]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,

    P. Manakul, A. Liusie, and M. J. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,”arXiv preprint arXiv:2303.08896, 2023

    arXiv 2023

  5. [5]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  6. [6]

    Discovering latent knowledge in language models without supervision,

    C. Burns, H. Ye, D. Klein, and J. Steinhardt, “Discovering latent knowledge in language models without supervision,”arXiv preprint arXiv:2212.03827, 2022

    arXiv 2022

  7. [7]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inInternational conference on machine learning. PMLR, 2017, pp. 1321–1330

    2017

  8. [8]

    Large language models as reliable knowledge bases?

    D. Zheng, M. Lapata, and J. Z. Pan, “Large language models as reliable knowledge bases?” arXiv preprint arXiv:2407.13578, 2024

    arXiv 2024

Show all 78 references
  1. [9]

    Haloscope: Harnessing unlabeled llm generations for hallucination detection,

    X. Du, C. Xiao, and S. Li, “Haloscope: Harnessing unlabeled llm generations for hallucination detection,”Advances in Neural Information Processing Systems, vol. 37, pp. 102 948–102 972, 2024

    2024

  2. [10]

    Inside: Llms’ internal states retain the power of hallucination detection,

    C. Chen, K. Liu, Z. Chen, Y . Gu, Y . Wu, M. Tao, Z. Fu, and J. Ye, “Inside: Llms’ internal states retain the power of hallucination detection,”arXiv preprint arXiv:2402.03744, 2024

    2024 arXiv

  3. [11]

    How to steer llm latents for hallucination detection?

    S. Park, X. Du, M.-H. Yeh, H. Wang, and Y . Li, “How to steer llm latents for hallucination detection?”arXiv preprint arXiv:2503.01917, 2025

    2025 arXiv

  4. [12]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,”ACM computing surveys, vol. 55, no. 12, pp. 1–38, 2023

    2023

  5. [13]

    Hallushift: Measuring distribution shifts towards hallucination detection in llms,

    S. Dasgupta, S. Nath, A. Basu, P. Shamsolmoali, and S. Das, “Hallushift: Measuring distribution shifts towards hallucination detection in llms,”arXiv preprint arXiv:2504.09482, 2025

    2025 arXiv

  6. [14]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2. 5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  8. [16]

    Truthfulqa: Measuring how models mimic human falsehoods,

    S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  9. [17]

    Biases in large language models: origins, inventory, and discussion,

    R. Navigli, S. Conia, and B. Ross, “Biases in large language models: origins, inventory, and discussion,”ACM Journal of Data and Information Quality, vol. 15, no. 2, pp. 1–21, 2023

    2023

  10. [18]

    Do large language models know how much they know?

    G. Prato, J. Huang, P. Parthasarathi, S. Sodhani, and S. Chandar, “Do large language models know how much they know?”arXiv preprint arXiv:2502.19573, 2025

    2025

  11. [19]

    Self-evaluation improves selective generation in large language models,

    J. Ren, Y . Zhao, T. Vu, P. J. Liu, and B. Lakshminarayanan, “Self-evaluation improves selective generation in large language models,” inProceedings on. PMLR, 2023, pp. 49–64

    2023

  12. [20]

    Deep information propagation,

    S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, “Deep information propagation,” arXiv preprint arXiv:1611.01232, 2016. 10

    2016 arXiv

  13. [21]

    Universal adversarial triggers for attacking and analyzing nlp,

    E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh, “Universal adversarial triggers for attacking and analyzing nlp,”arXiv preprint arXiv:1908.07125, 2019

    1908 arXiv

  14. [22]

    Gradient-based adversarial attacks against text transformers,

    C. Guo, A. Sablayrolles, H. Jégou, and D. Kiela, “Gradient-based adversarial attacks against text transformers,”arXiv preprint arXiv:2104.13733, 2021

    2021 arXiv

  15. [23]

    Identifying and controlling important neurons in neural machine translation,

    A. Bau, Y . Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. Glass, “Identifying and controlling important neurons in neural machine translation,”arXiv preprint arXiv:1811.01157, 2018

    2018 arXiv

  16. [24]

    A primer in bertology: What we know about how bert works,

    A. Rogers, O. Kovaleva, and A. Rumshisky, “A primer in bertology: What we know about how bert works,”Transactions of the association for computational linguistics, vol. 8, pp. 842–866, 2021

    2021

  17. [25]

    How can we know when language models know? on the calibration of language models for question answering,

    Z. Jiang, J. Araki, H. Ding, and G. Neubig, “How can we know when language models know? on the calibration of language models for question answering,”Transactions of the Association for Computational Linguistics, vol. 9, pp. 962–977, 2021

    2021

  18. [26]

    Reducing negative effects of the biases of language models in zero-shot setting,

    X. Wang, Y . Xiong, B. Kang, Y . Zhang, P. S. Yu, and Y . Zhu, “Reducing negative effects of the biases of language models in zero-shot setting,” inProceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, 2023, pp. 904–912

    2023

  19. [27]

    Understanding the difficulty of training deep feedforward neural networks,

    X. Glorot and Y . Bengio, “Understanding the difficulty of training deep feedforward neural networks,” inProceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010, pp. 249–256

    2010

  20. [28]

    A simple framework for contrastive learning of visual representations,

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” inInternational conference on machine learning. PmLR, 2020, pp. 1597–1607

    2020

  21. [29]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,”arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  22. [30]

    Momentum contrast for unsupervised visual representation learning,

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738

    2020

  23. [31]

    Euclidean distance mapping,

    P.-E. Danielsson, “Euclidean distance mapping,”Computer Graphics and image processing, vol. 14, no. 3, pp. 227–248, 1980

    1980

  24. [32]

    Analysis of euclidean distance and manhattan distance measure in face recognition,

    M. Malkauthekar, “Analysis of euclidean distance and manhattan distance measure in face recognition,” inThird International Conference on Computational Intelligence and Information Technology (CIIT 2013). IET, 2013, pp. 503–507

    2013

  25. [33]

    Coqa: A conversational question answering challenge,

    S. Reddy, D. Chen, and C. D. Manning, “Coqa: A conversational question answering challenge,” Transactions of the Association for Computational Linguistics, vol. 7, pp. 249–266, 2019

    2019

  26. [34]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,

    M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension,”arXiv preprint arXiv:1705.03551, 2017

    2017 arXiv

  27. [35]

    Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages,

    J. H. Clark, E. Choi, M. Collins, D. Garrette, T. Kwiatkowski, V . Nikolaev, and J. Palomaki, “Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages,”Transactions of the Association for Computational Linguistics, vol. 8, pp. 454–470, 2020

    2020

  28. [36]

    Generating with confidence: Uncertainty quantification for black-box large language models,

    Z. Lin, S. Trivedi, and J. Sun, “Generating with confidence: Uncertainty quantification for black-box large language models,”arXiv preprint arXiv:2305.19187, 2023

    2023 arXiv

  29. [37]

    Out-of- distribution detection and selective generation for conditional language models,

    J. Ren, J. Luo, Y . Zhao, K. Krishna, M. Saleh, B. Lakshminarayanan, and P. J. Liu, “Out-of- distribution detection and selective generation for conditional language models,”arXiv preprint arXiv:2209.15558, 2022

    2022 arXiv

  30. [38]

    Inference-time intervention: Eliciting truthful answers from a language model,

    K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg, “Inference-time intervention: Eliciting truthful answers from a language model,”Advances in Neural Information Processing Systems, vol. 36, pp. 41 451–41 530, 2023. 11

    2023

  31. [39]

    Deepseek llm: Scaling open-source language models with longtermism,

    X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu et al., “Deepseek llm: Scaling open-source language models with longtermism,”arXiv preprint arXiv:2401.02954, 2024

    2024 arXiv

  32. [40]

    Uncertainty estimation in autoregressive structured prediction,

    A. Malinin and M. Gales, “Uncertainty estimation in autoregressive structured prediction,” arXiv preprint arXiv:2002.07650, 2020

    2002 arXiv

  33. [41]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,

    L. Kuhn, Y . Gal, and S. Farquhar, “Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation,”arXiv preprint arXiv:2302.09664, 2023

    2023 arXiv

  34. [42]

    Teaching models to express their uncertainty in words,

    S. Lin, J. Hilton, and O. Evans, “Teaching models to express their uncertainty in words,”arXiv preprint arXiv:2205.14334, 2022

    2022 arXiv

  35. [43]

    The internal state of an llm knows when it’s lying,

    A. Azaria and T. Mitchell, “The internal state of an llm knows when it’s lying,”arXiv preprint arXiv:2304.13734, 2023

    2023 arXiv

  36. [44]

    Alignscore: Evaluating factual consistency with a unified alignment function,

    Y . Zha, Y . Yang, R. Li, and Z. Hu, “Alignscore: Evaluating factual consistency with a unified alignment function,”arXiv preprint arXiv:2305.16739, 2023

    2023 arXiv

  37. [45]

    On early detection of hallucinations in factual question answering,

    B. Snyder, M. Moisescu, and M. B. Zafar, “On early detection of hallucinations in factual question answering,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 2721–2732

    2024

  38. [46]

    Embedding and gradient say wrong: A white-box method for hallucination detection,

    X. Hu, Y . Zhang, R. Peng, H. Zhang, C. Wu, G. Chen, and J. Zhao, “Embedding and gradient say wrong: A white-box method for hallucination detection,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 1950–1959

    2024

  39. [47]

    Bleurt: Learning robust metrics for text generation,

    T. Sellam, D. Das, and A. P. Parikh, “Bleurt: Learning robust metrics for text generation,”arXiv preprint arXiv:2004.04696, 2020

    2004 arXiv

  40. [48]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText summarization branches out, 2004, pp. 74–81

    2004

  41. [49]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  42. [50]

    Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps,

    Y .-S. Chuang, L. Qiu, C.-Y . Hsieh, R. Krishna, Y . Kim, and J. Glass, “Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps,” arXiv preprint arXiv:2407.07071, 2024

    2024 arXiv

  43. [51]

    Navigating the grey area: How expressions of uncertainty and overconfidence affect language models,

    K. Zhou, D. Jurafsky, and T. Hashimoto, “Navigating the grey area: How expressions of uncertainty and overconfidence affect language models,”arXiv preprint arXiv:2302.13439, 2023

    2023 arXiv

  44. [52]

    I-divergence geometry of probability distributions and minimization problems,

    I. Csiszár, “I-divergence geometry of probability distributions and minimization problems,”The annals of probability, 1975

    1975

  45. [53]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  46. [54]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  47. [55]

    A survey on hallucination in large vision-language models,

    H. Liu, W. Xue, Y . Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision-language models,”arXiv preprint arXiv:2402.00253, 2024

    2024 arXiv

  48. [56]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets,

    S. Marks and M. Tegmark, “The geometry of truth: Emergent linear structure in large language model representations of true/false datasets,”arXiv preprint arXiv:2310.06824, 2023

    2023 arXiv

  49. [57]

    Characterizing truthfulness in large language model generations with local intrinsic dimension,

    F. Yin, J. Srinivasa, and K.-W. Chang, “Characterizing truthfulness in large language model generations with local intrinsic dimension,”arXiv preprint arXiv:2402.18048, 2024. 12

    2024 arXiv

  50. [58]

    Visual prompt tuning,

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” inEuropean conference on computer vision. Springer, 2022, pp. 709–727

    2022

  51. [59]

    Locating and editing factual associations in gpt,

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov, “Locating and editing factual associations in gpt,”Advances in neural information processing systems, vol. 35, pp. 17 359–17 372, 2022

    2022

  52. [60]

    Rebuilding rome: Resolving model collapse during sequential model editing,

    A. Gupta, S. Baskaran, and G. Anumanchipalli, “Rebuilding rome: Resolving model collapse during sequential model editing,”arXiv preprint arXiv:2403.07175, 2024

    2024 arXiv

  53. [61]

    Adversarial representation engineering: A general model editing framework for large language models,

    Y . Zhang, Z. Wei, J. Sun, and M. Sun, “Adversarial representation engineering: A general model editing framework for large language models,”arXiv preprint arXiv:2404.13752, 2024

    2024 arXiv

  54. [62]

    Attack and de- fense techniques in large language models: A survey and new perspectives,

    Z. Liao, K. Chen, Y . Lin, K. Li, Y . Liu, H. Chen, X. Huang, and Y . Yu, “Attack and de- fense techniques in large language models: A survey and new perspectives,”arXiv preprint arXiv:2505.00976, 2025

    2025 arXiv

  55. [63]

    Looking for a needle in a haystack: A comprehen- sive study of hallucinations in neural machine translation,

    N. M. Guerreiro, E. V oita, and A. F. Martins, “Looking for a needle in a haystack: A comprehen- sive study of hallucinations in neural machine translation,”arXiv preprint arXiv:2208.05309, 2022

    2022 arXiv

  56. [64]

    Look before you leap: An exploratory study of uncertainty measurement for large language models,

    Y . Huang, J. Song, Z. Wang, S. Zhao, H. Chen, F. Juefei-Xu, and L. Ma, “Look before you leap: An exploratory study of uncertainty measurement for large language models,”arXiv preprint arXiv:2307.10236, 2023

    2023 arXiv

  57. [65]

    Alleviating hallucinations of large language models through induced hallucinations,

    Y . Zhang, L. Cui, W. Bi, and S. Shi, “Alleviating hallucinations of large language models through induced hallucinations,”arXiv preprint arXiv:2312.15710, 2023

    2023 arXiv

  58. [66]

    Hallucination is inevitable: An innate limitation of large language models,

    Z. Xu, S. Jain, and M. Kankanhalli, “Hallucination is inevitable: An innate limitation of large language models,”arXiv preprint arXiv:2401.11817, 2024

    2024 arXiv

  59. [67]

    Enhancing uncertainty-based hallucination detection with stronger focus,

    T. Zhang, L. Qiu, Q. Guo, C. Deng, Y . Zhang, Z. Zhang, C. Zhou, X. Wang, and L. Fu, “Enhancing uncertainty-based hallucination detection with stronger focus,”arXiv preprint arXiv:2311.13230, 2023

    2023 arXiv

  60. [68]

    Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios,

    I. Chern, S. Chern, S. Chen, W. Yuan, K. Feng, C. Zhou, J. He, G. Neubig, P. Liuet al., “Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios,”arXiv preprint arXiv:2307.13528, 2023

    2023 arXiv

  61. [69]

    Factscore: Fine-grained atomic evaluation of factual precision in long form text generation,

    S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi, “Factscore: Fine-grained atomic evaluation of factual precision in long form text generation,”arXiv preprint arXiv:2305.14251, 2023

    2023 arXiv

  62. [70]

    Shifting attention to relevance: Towards the uncertainty estimation of large language models,

    J. Duan, H. Cheng, S. Wang, A. Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu, “Shifting attention to relevance: Towards the uncertainty estimation of large language models,” 2023

    2023

  63. [71]

    Do language models know when they’re hallucinating references?

    A. Agrawal, M. Suzgun, L. Mackey, and A. T. Kalai, “Do language models know when they’re hallucinating references?”arXiv preprint arXiv:2305.18248, 2023

    2023 arXiv

  64. [72]

    Lm vs lm: Detecting factual errors via cross examination,

    R. Cohen, M. Hamri, M. Geva, and A. Globerson, “Lm vs lm: Detecting factual errors via cross examination,”arXiv preprint arXiv:2305.13281, 2023

    2023 arXiv

  65. [73]

    Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation,

    N. Mündler, J. He, S. Jenko, and M. Vechev, “Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation,”arXiv preprint arXiv:2305.15852, 2023

    2023 arXiv

  66. [74]

    Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,

    K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning, “Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback,”arXiv preprint arXiv:2305.14975, 2023

    2023 arXiv

  67. [75]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,”arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  68. [76]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,”arXiv preprint arXiv:2101.00190, 2021. 13

    2021 arXiv

  69. [77]

    P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,

    X. Liu, K. Ji, Y . Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,”arXiv preprint arXiv:2110.07602, 2021

    2021 arXiv

  70. [78]

    Sample-specific masks for visual reprogramming-based prompting,

    C. Cai, Z. Ye, L. Feng, J. Qi, and F. Liu, “Sample-specific masks for visual reprogramming-based prompting,”arXiv preprint arXiv:2406.03150, 2024. 14 Appendix A Related Works Hallucination detectionhas become an increasingly important research topic, aiming to address the safe...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.