Pith. sign in

REVIEW 5 major objections 5 minor 79 references

Evaluating Evaluation Metrics -- The Mirage of Hallucination Detection

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most hallucination-detection metrics fail to track human judgments, while GPT-4 as judge emerges as the most reliable detector.

desk verdict A broad and useful empirical audit of hallucination detection metrics; the central claim is plausible but the GPT-4 result is actually gpt-4o-mini and the ensemble needs out-of-sample validation. read the letter →

arxiv 2504.18114 v2 pith:3RRJFQM5 submitted 2025-04-25 cs.CL cs.AIcs.LG

Atharva Kulkarni , Yuan Zhang , Joel Ruben Antony Moniz , Xiou Ge , Bo-Hsiang Tseng , Dhivya Piraviperumal , Swabha Swayamdipta , Hong Yu This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords hallucinationdetectionevaluationmetricsLLM-as-judgefaithfulnessfactualityknowledge-groundeddialogueparameterscalingdecodingmethods
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a large-scale comparison of six families of hallucination-detection metrics applied to four datasets, 37 language models, and five decoding methods. Its central claim is that current automatic metrics are unreliable: they disagree with one another, often fail to match human judgments, and do not improve consistently as models grow. The exception is LLM-based evaluation, especially GPT-4, which aligns best with human labels; an ensemble of metrics is a close second. The paper also argues that instruction-tuning and mode-seeking decoding (greedy and beam) reduce hallucinations, particularly in knowledge-grounded dialogue. The practical stake is that if detection metrics are untrustworthy, efforts to measure and mitigate hallucination are built on shaky ground.

What carries the argument

The central object under test is the hallucination-detection metric itself, organized into six families: n-gram overlap, semantic similarity, pretrained UniEval evaluators, Q2's question-answering-based detection, Critic's NLI classifier, GPT-4-as-judge, and one ensemble built by Factor Analysis of Mixed Data. The argument is carried by a factorial comparison grid crossing these metric families with four datasets, 37 models from five families, and five decoding methods. Human-alignment claims rest on the Begin and HaluEval human labels, while generality claims rest on paired and unpaired significance tests across training type, model size, and decoding method.

What would settle it

Collect human hallucination labels on a random sample of the TruthfulQA and FaithDial outputs used in Findings 3 and 4 and rerun the instruction-tuning and parameter-scaling significance tests; if the apparent gains of GPT-4 and Critic disappear against human labels, those findings collapse.

Watch

Extended reading notes

Core claim

The paper's central claim is that current automatic hallucination-detection metrics are not measuring what they claim to measure. Across six metric families, four datasets, 37 models, and five decoding methods, most metrics show weak inter-correlation and inconsistent agreement with human annotations; the UniEval consistency evaluator often performs at or below random and classifies nearly everything as hallucinated, which the authors call a mirage of success on skewed labels. GPT-4 as a judge shows the strongest and most consistent alignment with humans, and an ensemble of metrics built with factor analysis is second best. In addition, the paper claims that instruction-tuned models and mode-seeking decoding (greedy, beam) reduce hallucinations, especially in knowledge-grounded dialogue, while most metrics fail to show monotonic gains with parameter scaling; only GPT-4 consistently improves with model size.

Load-bearing premise

The load-bearing premise is that automated metric scores on TruthfulQA and FaithDial reflect real hallucination rates and that trends observed on FaithDial transfer to the human-labeled Begin benchmark because FaithDial derives from the Wizard of Wikipedia subset of Begin.

Editorial extensions

If this is right

  • Published hallucination results that rely on a single automatic metric, especially UniEval's consistency evaluator, may be measuring label skew rather than hallucination.
  • For benchmark comparisons, GPT-4-as-judge or an ensemble of diverse metrics is the safer choice; a single overlap or NLI metric is not trustworthy across tasks.
  • Instruction-tuning and greedy or beam decoding can be treated as evidence-based ways to reduce hallucination in knowledge-grounded dialogue, while sampling methods should be used with caution.
  • Parametric scaling claims about hallucination should specify which metric was used, since most metrics show no monotonic improvement and can even reverse, as with Gemma models abstaining from answers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to obtain human labels on a sample of the TruthfulQA and FaithDial generations and re-run Findings 3 and 4; the paper's transfer assumption from FaithDial to Begin would then be testable directly.
  • Uncertainty-based detectors excluded here, such as semantic entropy and SAR, could be coupled to the same human-labeled benchmarks to see whether they close the gap with GPT-4 at lower inference cost.
  • The weak inter-metric correlations suggest that hallucination is not a single construct; future work may need separate faithfulness and factuality metrics rather than one universal score.
  • The fact that an ensemble of imperfect metrics rivals GPT-4 suggests a cheap direction: replacing the GPT-4 component with a smaller instruction-tuned judge could preserve ensemble gains while cutting cost.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a meta-evaluation of six families of hallucination detection metrics (n-gram overlap, semantic similarity, trained evaluators, QA-based NLI, LLM-as-judge, and a FAMD-based ensemble) across four datasets, 37 models from five families, and five decoding methods. The authors report four findings: (1) except for GPT-4-based evaluation, metrics align poorly with human labels on the two datasets with human annotations (Begin and HaluEval); (2) inter-metric correlations are weak; (3) instruction-tuning and mode-seeking decoding reduce hallucination according to significance tests on TruthfulQA and FaithDial; and (4) most metrics do not improve monotonically with parameter scaling. The abstract concludes that LLM-based evaluation, particularly GPT-4, is the most reliable, and that an ensemble of metrics is a good alternative.

Significance. If the central claims were fully supported, the paper would be a valuable reference for practitioners choosing hallucination detection metrics. The study covers a broad and representative set of models and metrics, and the human-alignment analysis on Begin and HaluEval is a concrete contribution. The paper is also transparent about its own limitations, and the failure-mode analysis of metric behavior with respect to response length is a useful addition. However, the significance is currently limited by three concerns: the LLM-judge is actually gpt-4o-mini rather than GPT-4; the ensemble metric is fit and evaluated in-sample without cross-validation; and Findings 3 and 4 rely on automated metric scores on datasets without human hallucination labels, with an explicit but untested extrapolation from FaithDial to Begin. These issues affect the weight of the positive claims in the abstract and conclusion.

major comments (5)
  1. [§6 Limitations] The extrapolation from FaithDial to Begin is load-bearing but unsupported. The paper states that 'since FaithDial is a modified version of the WoW dataset, which is already included in begin, we can reasonably assume that the Findings 3 and 4 results for begin will follow similar trends to those observed for FaithDial.' However, Findings 3 and 4 are computed exclusively on TruthfulQA and FaithDial, which have no human hallucination labels. The automated metrics used there (Critic, GPT-4, etc.) are not validated on these two datasets; Table 1 shows Critic at chance level on HaluEval QA (weighted-F1 0.53), so using Critic on TruthfulQA, also a QA task, as evidence that instruction-tuning reduces hallucinations is questionable. To sustain the abstract's claim that mode-seeking decoding reduces hallucinations 'especially in knowledge-grounded settings,' the authors should validate the metrics on a human-labeled subset of TruthfulQA/FaithDial, or add human annotations for a sample, or explicitly reframe Findings 3 and 4 as statements about automated metric scores rather than about true hallucination rates.
  2. [Appendix A.3] The paper's central positive finding is that 'GPT-4' yields the best hallucination detection, but the evaluation actually uses the gpt-4o-mini model, as stated in Appendix A.3. This is misleading: gpt-4o-mini is a different and generally weaker model than GPT-4, and the abstract, Introduction, Finding 1, and Conclusion all attribute the result to GPT-4. The authors should either run the judgment with the actual GPT-4 model or consistently report gpt-4o-mini (e.g., 'GPT-4-class' or 'gpt-4o-mini') and temper the claims accordingly. This is not merely a naming issue: the headline recommendation for practitioners depends on which specific model was used.
  3. [§2, Table 1 (Ensemble metric)] The Ensemble metric is constructed using Factor Analysis of Mixed Data (FAMD) on the same datasets (Begin and HaluEval) on which its alignment with human labels is subsequently evaluated. The paper does not describe any cross-validation or held-out procedure for fitting the FAMD weights, so the ensemble's strong performance in Table 1 and the claim that 'an ensemble of metrics also seems to be a good choice' may be inflated by in-sample fitting. I recommend fitting the FAMD components on a subset of the data and evaluating on held-out subsets, or at least reporting the stability of the loadings across datasets, before drawing conclusions about ensemble usefulness.
  4. [Table 1, Finding 1] The agreement scores in Table 1 are reported without confidence intervals or significance tests. With only five datasets and several metrics, the differences between the top performers (e.g., GPT-4 average 0.74 vs. Ensemble average 0.72) may be within sampling noise. The paper would be much stronger with bootstrap confidence intervals or a paired significance test across datasets for the metric-versus-human agreement. Without such quantification, the ranking of metrics in Finding 1 is not as solid as the text suggests.
  5. [§3, Finding 1 text] The sentence 'The UniEval suite's factual consistency evaluator performed just about at or below random chance across all the six data subsets' is contradicted by Table 1: the Consistency scores for BeginCMU and BeginTC are 0.65, clearly above the random baseline of 0.50. The statement should be corrected to say that the consistency evaluator is at or below chance on most subsets (BeginWoW, HaluEval Dial, HaluEval QA) but above chance on two. This is a factual misstatement that could mislead readers about the metric's performance profile.
minor comments (5)
  1. [Table 2 caption] The caption states 'Red cells (p > 0.05) indicate failure to reject the null hypothesis,' but the table is printed in black and white with no visible color indicators. Please use shading or an asterisk convention to mark non-significant p-values.
  2. [Appendix A.4] The hypothesis testing section would benefit from a more explicit description of how the metric scores were aggregated per model (e.g., per-example scores vs. per-model scores) and whether multiple-comparison corrections were applied to the p-values in Table 2.
  3. [§2, Metrics] The paper introduces six 'types' of metrics but Table 1 and the text sometimes refer to specific components (Consistency, K-BertScore, Q2NLI, Critic, GPT-4) without clearly mapping them to the six types. A table or legend that lists the exact measure used for each row would improve readability.
  4. [§3, Finding 2] The sentence 'the latter also demonstrate high overlap with each other on the Begin datasets' is ambiguous; it is unclear whether 'the latter' refers to Critic and GPT-4 or to the continuous metrics. Please rephrase for clarity.
  5. [Abstract and Introduction] The abstract and introduction repeatedly use 'hallucination' without noting that the study focuses on faithfulness and factuality in knowledge-grounded dialogue and QA; adding this qualification would set accurate expectations for the reader.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline metric-vs-human alignment results are measured against external human labels, and the remaining internal choices are methodological assumptions or in-sample fitting, not definitional reductions.

full rationale

The paper's central claim that most hallucination detection metrics fail to align with human judgments, while GPT-4 performs best, is an empirical evaluation against externally provided human annotations on Begin and HaluEval. These agreement scores are not derived from the metrics' own definitions or from parameters fit to the same labels, so the main result is not circular. Finding 2's inter-metric correlation claim rests on threshold-free Spearman rank correlations (Figure 1); the threshold-optimized overlap analysis in Figure 2 is an illustrative secondary analysis and does not drive the conclusion. Findings 3 and 4 use automated metric scores on TruthfulQA and FaithDial, which lack human hallucination labels, and the paper explicitly acknowledges the transfer assumption to Begin in the Limitations section. That is a genuine external-validity and construct-validity limitation, but it is not a circular reduction: the paper does not define hallucination reduction as 'what these metrics measure' in a way that makes the conclusion true by construction. The Ensemble metric's FAMD construction is evaluated on the same data without a described held-out split, which may introduce optimistic bias, but FAMD is an unsupervised combination of metric scores and the paper does not claim to predict human labels from its own fitted weights. There is no load-bearing self-citation chain: the cited prior results are from external groups, and the paper adds its own paired significance tests rather than relying solely on those citations. No step in the derivation chain is equivalent to its inputs by definition, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim is an empirical comparison, so the main ledger entries are analysis choices (FAMD weights, binarization thresholds) and domain assumptions (human labels are ground truth, gpt-4o-mini stands for GPT-4, metrics on unlabeled data are meaningful). The extrapolation from FaithDial to Begin is the most fragile assumption.

free parameters (2)
  • FAMD ensemble component loadings = not reported; learned from data
    The Ensemble metric is constructed by factor analysis of mixed data on the same datasets where it is later scored (Table 1), so the weighting is fitted in-sample.
  • Binary threshold for continuous metrics = per-metric and per-dataset, chosen to maximize weighted F1
    Used in Figure 2 and label-distribution analysis; fitting the threshold on the test labels can inflate apparent agreement.
assumptions (4)
  • domain assumption Human hallucination labels in Begin and HaluEval are valid ground truth.
    Finding 1's alignment scores treat these annotations as correct; if the annotations are noisy or biased, the ranking of metrics shifts.
  • ad hoc to paper gpt-4o-mini behavior is representative of GPT-4 for judging hallucination.
    Appendix A.3 says the judge is gpt-4o-mini, but abstract and findings call it GPT-4; the conclusion is stated for GPT-4.
  • domain assumption Metrics applied to TruthfulQA and FaithDial generations measure hallucination adequately for Findings 3 and 4.
    These datasets have no human labels, so decoding and scaling conclusions rely on the very metrics whose validity the paper questions.
  • ad hoc to paper Findings 3 and 4 for FaithDial transfer to Begin because FaithDial is derived from WoW.
    Stated in Limitations; this extrapolation is load-bearing for generalizing to Begin.
invented entities (1)
  • Ensemble metric (FAMD combination)
    purpose: Combine consistency, K-BertScore, Q2 NLI, Critic, and GPT-4 scores into one hallucination score.
    The ensemble is fit and evaluated on the same data; no held-out or external validation is provided, so its reported strength may be in-sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Evaluation Metrics -- The Mirage of Hallucination Detection." pith.science (2026). https://pith.science/paper/3RRJFQM5

@misc{pith2026250418114,
  author       = {Pith},
  title        = {Pith review of: Evaluating Evaluation Metrics -- The Mirage of Hallucination Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3RRJFQM5}},
  note         = {Machine review of arXiv:2504.18114}
}
read the original abstract

Hallucinations pose a significant obstacle to the reliability and widespread adoption of language models, yet their accurate measurement remains a persistent challenge. While many task- and domain-specific metrics have been proposed to assess faithfulness and factuality concerns, the robustness and generalization of these metrics are still untested. In this paper, we conduct a large-scale empirical evaluation of 6 diverse sets of hallucination detection metrics across 4 datasets, 37 language models from 5 families, and 5 decoding methods. Our extensive investigation reveals concerning gaps in current hallucination evaluation: metrics often fail to align with human judgments, take an overtly myopic view of the problem, and show inconsistent gains with parameter scaling. Encouragingly, LLM-based evaluation, particularly with GPT-4, yields the best overall results, and mode-seeking decoding methods seem to reduce hallucinations, especially in knowledge-grounded settings. These findings underscore the need for more robust metrics to understand and quantify hallucinations, and better strategies to mitigate them.

Figures

Figures reproduced from arXiv: 2504.18114 by the authors.

Figure 1
Figure 1. Spearman rank correlation between halluci [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Percentage of correct matching labels shows [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Hallucination detection metric scores for greedy decoding on various model sizes. Circles and hexagons [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: P-values for different model size bins from the [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Metric accuracy across varying response lengths on the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Spearman rank correlation between halluci [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Distribution of hallucinated and non￾hallucinated responses in Begin and HaluEvl. Dataset Critic GPT4 Consistency K-BertScore Q 2 NLI Ensemble Begin CMU 2843 / 107 2159 / 791 2949 / 1 22947 / 3 1066 / 1884 2949 / 1 Begin TC 3704 / 101 1993 / 1812 3804 / 1 3804 / 1 2069…
Figure 8
Figure 8. Figure 8: Comparison of factual hallucination metrics across decoding techniques. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Comparison of traditional NLG metrics across decoding techniques. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Comparison of uncertainty and token-overlap based hallucination metrics across decoding techniques. [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Per-group p-values for decoding techniques using pairwise T-test with Bonferroni correction. [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Per-group p-values for different decoding techniques using pairwise T-test with Bonferroni correction. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: NLG-based hallucination detection metric scores for greedy decoding as model size increases. Circles [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Uncertainty and token-overlap based hallucination detection metric scores for greedy decoding as model [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Per-group p-values for different model size bins using the pairwise Mann-Whitney rank test. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 11 canonical work pages

  1. [1]

    Marah Abdin et al. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . Preprint, arXiv:2404.14219

    arXiv 2024

  2. [2]

    Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2024. https://doi.org/10.1162/tacl_a_00667 Evaluating correctness and faithfulness of instruction-following models for question answering . Transactions of the Association for Computational Linguistics, 12:681--699

    doi:10.1162/tacl_a_00667 2024

  3. [3]

    Shayan Ali Akbar, Md Mosharaf Hossain, Tess Wood, Si-Chi Chin, Erica M Salinas, Victor Alvarez, and Erwin Cornejo. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.837 H allu M easure: Fine-grained hallucination measurement using chain-of-thought reasoning . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages ...

    doi:10.18653/v1/2024.emnlp-main.837 2024

  4. [4]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2024. http...

    arXiv 2024

  5. [5]

    Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. 2023. https://openreview.net/forum?id=sckjveqlCZ Broken neural scaling laws . In The Eleventh International Conference on Learning Representations

    work page 2023

  6. [6]

    Meng Cao, Yue Dong, and Jackie Cheung. 2022. https://doi.org/10.18653/v1/2022.acl-long.236 Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3340--3354, Dublin, Ireland. Association fo...

    doi:10.18653/v1/2022.acl-long.236 2022

  7. [7]

    Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, and Yanghua Xiao. 2023. https://doi.org/10.1145/3583780.3614905 Hallucination detection: Robustly discerning reliable answers in large language models . In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM '23, pag...

    arXiv 2023

  8. [8]

    Glass, and Pengcheng He

    Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. https://openreview.net/forum?id=Th6NyL07na Dola: Decoding by contrasting layers improves factuality in large language models . In The Twelfth International Conference on Learning Representations

    2024

Show all 79 references
  1. [9]

    Souvik Das, Sougata Saha, and Rohini K Srihari. 2022. Diving deep into modes of fact hallucinations in dialogue systems. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 684--699

    2022

  2. [10]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. https://arxiv.org/abs/2309.11495 Chain-of-verification reduces hallucination in large language models . Preprint, arXiv:2309.11495

    2023 arXiv

  3. [11]

    Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. https://openreview.net/forum?id=r1l73iRqKm Wizard of wikipedia: Knowledge-powered conversational agents . In International Conference on Learning Representations

    2019

  4. [12]

    Xuefeng Du, Chaowei Xiao, and Yixuan Li. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/ba92705991cfbbcedc26e27e833ebbae-Paper-Conference.pdf Haloscope: Harnessing unlabeled llm generations for hallucination detection . In Advances in Neural Information Proce...

    2024

  5. [13]

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. https://doi.org/10.18653/v1/2024.acl-long.276 Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models ...

    2024 doi

  6. [14]

    Abhimanyu Dubey et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  7. [15]

    Esin Durmus, He He, and Mona Diab. 2020. https://doi.org/10.18653/v1/2020.acl-main.454 FEQA : A question answering evaluation framework for faithfulness assessment in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Lin...

    2020 doi

  8. [16]

    Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. 2022. https://doi.org/10.18653/v1/2022.acl-long.102 Spurious correlations in reference-free evaluation of text generation . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2022 doi

  9. [17]

    Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar Zaiane. 2019. https://doi.org/10.18653/v1/N19-1381 Evaluating coherence in dialogue systems using entailment . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Lingu...

    2019 doi

  10. [18]

    Ponti, and Siva Reddy

    Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022 a . https://doi.org/10.1162/tacl_a_00529 F aith D ial: A faithful benchmark for information-seeking dialogue . Transactions of the Association for Computational Linguistics, ...

    2022 doi

  11. [19]

    Nouha Dziri, Andrea Madotto, Osmar Za \"i ane, and Avishek Joey Bose. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.168 Neural path hunter: Reducing hallucination in dialogue systems via path grounding . In Proceedings of the 2021 Conference on Empirical Methods in Natural...

    2021 doi

  12. [20]

    Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022 b . https://doi.org/10.1162/tacl_a_00506 Evaluating attribution in dialogue systems: The BEGIN benchmark . Transactions of the Association for Computational Linguistics, 10:1066--1083

    2022 doi

  13. [21]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. https://doi.org/10.18653/v1/P18-1082 Hierarchical neural story generation . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889--898, Melbourne, Australia....

    2018 doi

  14. [22]

    Team Gemma. 2024. https://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technology . Preprint, arXiv:2403.08295

    2024 arXiv

  15. [23]

    Gaurav Rohit Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. 2024. https://openreview.net/forum?id=cPsn9AcOYh Understanding finetuning for factual knowledge extraction . In Forty-first International Conference on Machine Learning

    2024

  16. [24]

    Ameya Godbole and Robin Jia. 2025. https://arxiv.org/abs/2501.14883 Verify with caution: The pitfalls of relying on imperfect factuality metrics . Preprint, arXiv:2501.14883

    2025 arXiv

  17. [25]

    Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. https://doi.org/10.21437/Interspeech.2019-3079 Topical-chat: Towards knowledge-grounded open-domain conversations . In Interspeec...

    2019 doi

  18. [26]

    Alex Graves. 2012. https://arxiv.org/abs/1211.3711 Sequence transduction with recurrent neural networks . Preprint, arXiv:1211.3711

    2012 arXiv

  19. [27]

    Dirk Groeneveld et al. 2024. https://doi.org/10.18653/v1/2024.acl-long.841 OLM o: Accelerating the science of language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789--15809, Bangkok, Thai...

    2024 doi

  20. [28]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Y...

    2023 arXiv

  21. [29]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  22. [30]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH The curious case of neural text degeneration . In International Conference on Learning Representations

    2020

  23. [31]

    Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.619 q^ 2 : E valuating factual consistency in knowledge-grounded dialogues via question generation and question answering . In Proceedings o...

    2021 doi

  24. [32]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. https://doi.org/10.1145/3703155 A survey on hallucination in large language models: Principles, taxonomy, challenges, and o...

    2025 doi

  25. [33]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. https://doi.org/10.1145/3571730 Survey of hallucination in natural language generation . ACM Comput. Surv., 55(12)

    2023 doi

  26. [34]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. https://arxiv.org/abs/2307.10169 Challenges and applications of large language models . Preprint, arXiv:2307.10169

    2023 arXiv

  27. [35]

    Adam Tauman Kalai and Santosh S. Vempala. 2024. https://doi.org/10.1145/3618260.3649777 Calibrated language models must hallucinate . In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, STOC 2024, page 160–171, New York, NY, USA. Association for Computing Machinery

    2024

  28. [36]

    Haoqiang Kang, Terra Blevins, and Luke Zettlemoyer. 2024. https://arxiv.org/abs/2402.10496 Comparing hallucination detection metrics for multilingual generation . Preprint, arXiv:2402.10496

    2024 arXiv

  29. [37]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  30. [38]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. https://openreview.net/forum?id=VD-AYtP0dve Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation . In The Eleventh International Conference on Learning Representations

    2023

  31. [39]

    Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024. https://arxiv.org/abs/2401.03205 The dawn after the dark: An empirical study on factuality hallucination in large language models . Preprint, arXiv:2401.03205

    2024 arXiv

  32. [40]

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.397 H alu E val: A large-scale hallucination evaluation benchmark for large language models . In Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023 doi

  33. [41]

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023 b . https://arxiv.org/abs/2309.05463 Textbooks are all you need ii: phi-1.5 technical report . Preprint, arXiv:2309.05463

    2023 arXiv

  34. [42]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  35. [43]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  36. [44]

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. https://arxiv.org/abs/2404.00971 Exploring and evaluating hallucinations in llm-powered code generation . Preprint, arXiv:2404.00971

    2024

  37. [45]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  38. [46]

    Junliang Luo, Tianyu Li, Di Wu, Michael Jenkin, Steve Liu, and Gregory Dudek. 2024. https://arxiv.org/abs/2401.08358 Hallucination detection and hallucination mitigation: An investigation . Preprint, arXiv:2401.08358

    2024 arXiv

  39. [47]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. https://doi.org/10.18653/v1/2020.acl-main.173 On faithfulness and factuality in abstractive summarization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1...

    2020 doi

  40. [48]

    Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. https://doi.org/10.18653/v1/P19-1081 O pen D ial KG : Explainable conversational reasoning with attention-based walks over knowledge graphs . In Proceedings of the 57th Annual Meeting of the Association for Compu...

    2019 doi

  41. [49]

    Team OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  42. [50]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  43. [51]

    J \'e r \^o me Pag \`e s. 2014. Multiple factor analysis by example using R. CRC Press

    2014

  44. [52]

    Matt Post. 2018. https://doi.org/10.18653/v1/W18-6319 A call for clarity in reporting BLEU scores . In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186--191, Brussels, Belgium. Association for Computational Linguistics

    2018 doi

  45. [53]

    Shrimai Prabhumoye, Kazuma Hashimoto, Yingbo Zhou, Alan W Black, and Ruslan Salakhutdinov. 2021. https://doi.org/10.18653/v1/2021.naacl-main.338 Focused attention improves document-grounded generation . In Proceedings of the 2021 Conference of the North American Chapter of the...

    2021 doi

  46. [54]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  47. [55]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  48. [56]

    Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021. https://doi.org/10.18653/v1/2021.acl-long.58 Increasing faithfulness in knowledge-grounded dialogue with controllable features . In Proceedings of the 59th Annual Meeting of the Association for Computat...

    2021 doi

  49. [57]

    Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. https://doi.org/10.18653/v1/2021.naacl-main.92 The curious case of hallucinations in neural machine translation . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computatio...

    2021 doi

  50. [58]

    Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. https://doi.org/10.18653/v1/2024.naacl-short.69 Trusting your evidence: Hallucinate less with context-aware decoding . In Proceedings of the 2024 Conference of the North American C...

    2024 doi

  51. [59]

    Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2025. https://arxiv.org/abs/2406.12624 Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges . Preprint, arXiv:2406.12624

    2025 arXiv

  52. [60]

    Hugo Touvron et al. 2023. https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . Preprint, arXiv:2307.09288

    2023 arXiv

  53. [61]

    Saad Obaid ul Islam, Anne Lauscher, and Goran Glavaš. 2025. https://arxiv.org/abs/2502.12769 How much do llms hallucinate across languages? on multilingual estimation of llm hallucination in the wild . Preprint, arXiv:2502.12769

    2025

  54. [62]

    Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. https://arxiv.org/abs/2307.03987 A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation . Preprint, arXiv:2307.03987

    2023 arXiv

  55. [63]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. https://doi.org/10.18653/v1/2020.acl-main.450 Asking and answering questions to evaluate the factual consistency of summaries . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 500...

    2020 doi

  56. [64]

    Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, and David Mortensen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.401 Counting th...

    2023 doi

  57. [65]

    Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. 2024 a . Sayself: Teaching llms to express confidence with self-reflective rationales. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5985--5998

    2024

  58. [66]

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. 2024 b . https://arxiv.org/abs/2401.11817 Hallucination is inevitable: An innate limitation of large language models . Preprint, arXiv:2401.11817

    2024 arXiv

  59. [67]

    Ziyou Yan. 2024. https://eugeneyan.com/writing/llm-evaluators/ Evaluating the effectiveness of llm-evaluators (aka llm-as-judge) . eugeneyan.com

    2024

  60. [68]

    Joonho Yang, Seunghyun Yoon, Hwan Chang, Byeongjeong Kim, and Hwanhee Lee. 2025. https://arxiv.org/abs/2505.15291 Hallucinate at the last in long response generation: A case study on long document summarization . Preprint, arXiv:2505.15291

    2025

  61. [69]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  62. [70]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://doi.org/10.18653/v1/2023.acl-long.634 A lign S core: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ...

    2023 doi

  63. [71]

    Shaolei Zhang, Tian Yu, and Yang Feng. 2024 a . Truthx: Alleviating hallucinations by editing large language models in truthful space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8908--8949

    2024

  64. [72]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  65. [73]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  66. [74]

    Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. 2024 b . https://arxiv.org/abs/2312.15710 Alleviating hallucinations of large language models through induced hallucinations . Preprint, arXiv:2312.15710

    2024 arXiv

  67. [75]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. https://arxiv.org/abs/2309.01219 Siren's song in the ai ocean: A survey on hallucination in ...

    2023 arXiv

  68. [76]

    Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.131 Towards a unified multi-dimensional evaluator for text generation . In Proceedings of the 2022 Conference on Empiric...

    2022 doi

  69. [77]

    Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. https://doi.org/10.18653/v1/D18-1076 A dataset for document grounded conversations . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708--713, Brussels, Belgium. Associat...

    2018 doi

  70. [78]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  71. [79]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.