Pith. sign in

REVIEW 3 major objections 7 minor 30 references

Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that five leading automated factuality metrics, despite similar leaderboard scores, disagree sharply on individual examples and misestimate how often systems produce unsupported claims.

desk verdict Careful, honest meta-evaluation showing AutoAIS metrics disagree with each other and misestimate system error rates; main caveat is the untested quality of the human labels, which the authors openly acknowledge. read the letter →

arxiv 2501.14883 v2 pith:L62TIQMY submitted 2025-01-24 cs.CL cs.LG

Ameya Godbole , Robin Jia This is my paper · ORCID
classification cs.CLcs.LG
keywords factualityevaluationattributiontoidentifiedsourcesAutoAISLLM-as-judgebalancedaccuracyhallucinationrateestimationmetricbiasLLM-AggreFact
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test the optimistic view that large language models can serve as reliable automated judges of whether a generated summary or answer is faithful to its source document. It re-evaluates five leading attribution-checking metrics across fourteen dataset splits spanning summarization, retrieval-augmented generation, and question answering. The central finding is that these metrics disagree with one another on individual examples, hide large differences in how they trade off false alarms against missed errors, and often over- or underestimate how frequently a system produces unsupported claims. The paper argues that error analyses, headroom estimates, and system rankings built on such metrics alone are therefore open to misleading conclusions until the metric is validated on the user's own data. This matters because automated factuality checking is increasingly used as a cheaper stand-in for human evaluation in settings where trustworthiness is the central concern.

What carries the argument

The argument is carried by a small set of diagnostic measurements rather than by a single theorem or identity. The first is balanced accuracy (BAcc), the average of true-positive and true-negative rates, which the paper decomposes to show that equal BAcc can hide opposite error trade-offs. The second is instance-level intersection-over-union of the sets of examples that different evaluators predict as unattributable, which measures whether two metrics that score alike are actually agreeing about the same cases. The third is the quantification bias of an evaluator, defined as the difference between the human-labeled error rate and the metric-predicted error rate for a generation system, computed at both claim level and response level. The fourth is ROUGE-2 precision, the fraction of claim bigrams that also appear in the document, used to group examples by surface similarity, and the derived R2-diff score, which measures how much of that bigram overlap is lost when a long document is chunked; R2-diff isolates the mechanism by which chunked evaluators make attributable claims unverifiable. The paper also applies threshold tuning and an adjusted-counts correction as calibration procedures that can reduce system-level estimation bias.

What would settle it

Have two or more independent human annotators re-label a stratified random sample of claim-document pairs from each of the fourteen dataset splits, then compute inter-annotator agreement and compare it with the agreement between the top two AutoAIS evaluators; if human-human agreement is as low as evaluator-evaluator agreement, the ground truth itself is the weak link and the paper's metric-failure conclusion would need to be restated as joint human-and-metric uncertainty.

Watch

Extended reading notes

Core claim

The central discovery is that choosing an AutoAIS (attribution-to-identified-sources) evaluator by balanced accuracy on the LLM-AGGRE FACT leaderboard does not tell you how the evaluator will behave. The two top-ranked evaluators, gpt-4-turbo and Bespoke-7B, have nearly equal balanced accuracy (76.2% versus 77.4%) yet an intersection-over-union below 50% on five of fourteen datasets for the set of examples they flag as unattributable, so they point to different failures in error analysis. The same score can hide a true-positive/true-negative trade-off, as on ExpertQA where the two evaluators have essentially swapped those rates. At the system level, the metrics misestimate hallucination rates in both directions: on TofuEval-MediaSum, gpt-4-turbo underestimates response-level error by roughly 18 to 30 percentage points while MiniCheck-RoBERTa overestimates it by up to about 21 points, and on 8 of 14 datasets the evaluators order a meaningful fraction of system pairs incorrectly relative to human labels. The paper also finds two systematic biases: the metrics miss unattributable claims that copy much of the source text, and they penalize attributable claims that are heavily paraphrased or that draw on evidence spread across distant chunks of a long document. It concludes that users should validate any metric's predictions on their own domain and that metric developers should report true-positive/true-negative breakdowns, quantification bias, and ranking quality rather than a single averaged number.

Load-bearing premise

The analysis assumes that the human annotations in the underlying datasets are highly accurate and unambiguous ground truth; if those labels are noisy, or give evaluators less information than human annotators had, the measured metric failures and bias figures would be partly artifacts of label uncertainty.

Editorial extensions

If this is right

  • Users of factuality metrics should not infer reliability from a leaderboard balanced-accuracy score; they should validate metric predictions against human labels on a sample from their own dataset before drawing conclusions.
  • Error analysis performed with a single evaluator is likely to highlight a biased subset of the actual failures, because the top evaluators agree on fewer than half the flagged examples on several datasets.
  • Headroom estimates on a benchmark are not trustworthy until the chosen metric's quantification bias is measured, since the same evaluator can understate headroom on one dataset and overstate it on another by more than ten percentage points.
  • System rankings produced by these metrics are not reliable as-is: on datasets with six or more systems, gpt-4-turbo orders roughly a quarter of system pairs incorrectly relative to the human-labeled ranking.
  • Developers of new AutoAIS metrics should report true-positive and true-negative rates separately, system-level quantification bias, and ranking consistency, because balanced accuracy alone does not distinguish evaluators with opposite error profiles.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same diagnostic suite of TPR/TNR decomposition, prediction-set overlap, and quantification bias could be applied to other LLM-as-judge settings such as reward models and preference judges, where averaged leaderboard scores may similarly hide who the judge errs on.
  • Beyond the paper, the R2-diff score could be used as a cheap pre-screening tool: when a claim's evidence is spread across chunks under an evaluator's context window, a long-context evaluator or full-document processing should be preferred, since chunked evaluation is shown to over-predict the unattributable label.
  • Beyond the paper, the surface-similarity bias suggests a concrete test: if an evaluator's true-positive rate drops mainly on paraphrase-heavy attributable claims, then augmenting its training data with synthetic paraphrases might raise TPR without new human labels; the paper does not test this.
  • Beyond the paper, if the human labels themselves are ambiguous, part of the measured disagreement between evaluators may reflect evaluators choosing different plausible readings of the same claim, so paired human-human disagreement data would be needed to separate metric error from label uncertainty.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper re-evaluates five state-of-the-art automated attribution-to-identified-sources (AutoAIS) factuality metrics on the LLM-AggreFact benchmark, which aggregates 11 datasets (split here into 14 sub-datasets) of claim–document pairs with human attributability labels. The authors measure instance-level agreement (IoU), the TPR/TNR decomposition of balanced accuracy, system-level quantification bias, consequent headroom estimation errors, and system-ranking errors. They further identify two biases: a dependence on surface-level similarity (ROUGE-2 precision) and a degradation caused by document chunking for long-context evaluators. The paper concludes that AutoAIS evaluators are inconsistent with each other, often misestimate system-level hallucination rates, and can mislead error analysis, headroom estimates, and system rankings; it urges domain-specific validation and proposes threshold tuning as a partial remedy for quantification bias.

Significance. If the findings hold, this is a timely and valuable meta-evaluation. The paper moves beyond leaderboard balanced accuracy to system-level consequences and provides a clear operational definition of bias, a TPR/TNR breakdown, and an explicit analysis of headroom and ranking pitfalls. The empirical design is largely sound: it uses a public benchmark, multiple evaluators from different developers, and direct computation of outcome variables from human labels. The code and data appear to be standard and the analysis is reproducible in principle. The paper also gives credit to and engages with concurrent work on evaluator surface-level heuristics. The main significance is that it provides concrete, actionable evidence that researchers should not trust AutoAIS metrics without per-domain validation, and it proposes concrete reporting standards (error breakdowns, quantification bias, ranking validation) that could improve future benchmark development.

major comments (3)
  1. [Section 3.3 and Limitations] The central bias results are defined as the difference between human-labeled and predicted error rates, so their validity rests entirely on the accuracy of the LLM-AggreFact human annotations. The Limitations section explicitly concedes that these annotations may be noisy, ambiguous, or information-asymmetric relative to what the evaluators can access. A robustness check that restricts the analysis to examples with high annotator agreement or adjudicated labels is needed to show that the headline findings (e.g., the 12.9% claim-level and 29.8% response-level underestimates for TofuEval-MediaSum) are not partly artifacts of label noise. Without such a check, the strength of the claim that the metrics 'often misestimate system-level performance' is not fully supported.
  2. [Section 3.3 and Tables 11–29] No confidence intervals or significance tests are reported for the quantification-bias estimates, many of which are computed on small per-system samples (e.g., AggreFact-CNN in Table 9, where several systems have error rates in the 4–8% range). The paper uses a two-proportion z-test for ranking significance in §3.5, but the equally load-bearing bias percentages in §3.3 and §3.4 are presented without uncertainty. Bootstrap confidence intervals for the key bias estimates would clarify whether differences such as the TofuEval-MediaSum underestimates are statistically distinguishable from zero or from each other.
  3. [Section 4.2 and Table 30] The chunking simulation uses Bespoke-7B with a context window simulated at '500 document tokens' (§2.2) but Table 30's caption says '500 words'; this inconsistency should be resolved. More importantly, the paper presents the chunking result as showing that chunking 'inherently' disadvantages evaluators (§6.1), but this conclusion is based on a single chunk size and an unspecified chunking procedure (e.g., whether chunks are sequential, overlapping, or sentence-aligned). A sensitivity analysis varying chunk size and chunking strategy is needed to support the strong wording, or the claim should be softened to state that the tested configuration exhibits this effect.
minor comments (7)
  1. [Section 1 and Throughout] The dataset name 'TofuEval-MediaSum' is abbreviated inconsistently as 'TofuEval-MediaS' in several places (e.g., §3.3); please standardize.
  2. [Related Work] There is a typo, 'Cooroborating', which should be 'Corroborating'.
  3. [Limitations] 'inbalances' should be 'imbalances'.
  4. [Table captions (Tables 9–29)] The word 'paratheses' appears in many captions and should be 'parentheses'.
  5. [Section A.3] The sentence 'The value of τ indicates that the evaluators make one-three ranking errors in each ranking of the 6 systems' is unclear; please specify the number of pairs (15) and interpret τ against that baseline.
  6. [Section 5 and Table 1] Table 1's 'Thres. tuning for ↑BAcc' column is cryptic; please explain in the caption that it refers to tuning the evaluator threshold to maximize balanced accuracy on the calibration set.
  7. [Section 4.2] The R2-diff definition in the main text uses ROUGE-2 precision but the notation in the equation mixes 'd' and 'd(k)' without explicitly stating that these are tokenized representations; a brief clarification would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's headline findings are direct comparisons between external human labels and published third-party evaluators, with no fitted parameter or self-citation chain forcing the conclusions.

full rationale

The paper's load-bearing quantities—evaluator consistency (IoU), TPR/TNR breakdowns, system-level quantification bias, headroom misestimation, and system-pair ranking errors—are all defined as direct measurements of the difference between human annotations in the external LLM-AggreFact benchmark and the predictions of five published AutoAIS evaluators (gpt-4-turbo, gpt-3.5-turbo, Bespoke-7B, MiniCheck-FT5, MiniCheck-Rbta). No outcome variable is defined in terms of the conclusion it is used to support, and no parameter is fitted to the reported headline results. The Section 5 threshold-tuning experiments calibrate on one system and evaluate on held-out systems, with the calibration labels explicitly disclosed; this is cross-validation rather than a fitted input renamed as a prediction. The only explicit caveat, in the Limitations section, states: 'Our analysis assumes that the datasets underlying LLM-AggreFact have highly accurate human annotations with little ambiguity.' This is a genuine external-validity assumption—if human labels are noisy or information-asymmetric, the measured biases could partly reflect label artifacts—but it is not circularity, because the paper's target quantities are defined against those labels and the analysis does not rely on the authors' own prior results. The MiniCheck evaluators and the benchmark were both developed by Tang et al., but those are prior external artifacts studied here rather than self-citations used to justify the conclusions; no load-bearing argument reduces to a self-citation or to a definitional identity. The paper does not present a derivation that is equivalent to its inputs by construction, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

All central claims are empirical measurements against external human labels; no free parameters are fitted. The main unproved premises are the quality of those labels, the unitization of claims into sentences, and the strict response-level aggregation rule, all stated in Sections 2.1 and the Limitations.

assumptions (5)
  • domain assumption Human annotations in LLM-AggreFact are treated as high-quality ground truth for attributability.
    Stated in Limitations: 'Our analysis assumes that the datasets underlying LLM-AggreFact have highly accurate human annotations with little ambiguity.' If labels are noisy, measured metric bias is confounded.
  • domain assumption Each sentence is treated as a standalone claim; decontextualization is assumed to be handled separately.
    Section 2.1, following Tang et al. (2024b), uses sentence-level claims and treats decontextualization as a separate step; AggreFact-CNN is an exception.
  • domain assumption A response is attributable iff all its claims are attributable (strict aggregation).
    Section 2.1; this strict rule drives response-level bias and headroom estimates.
  • standard math Evaluator scores are binarized at threshold 0.5.
    Section 2.1, following Tang et al. (2024a); later threshold tuning is explicitly an intervention, not a hidden fit.
  • domain assumption ROUGE-2 precision is a meaningful proxy for surface-level claim-document overlap.
    Section 4.1 uses ROUGE-2 precision bins to demonstrate surface-similarity bias; the grouping is descriptive, not causal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics." pith.science (2026). https://pith.science/paper/L62TIQMY

@misc{pith2026250114883,
  author       = {Pith},
  title        = {Pith review of: Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L62TIQMY}},
  note         = {Machine review of arXiv:2501.14883}
}
read the original abstract

Improvements in large language models have led to increasing optimism that they can serve as reliable evaluators of natural language generation outputs. In this paper, we challenge this optimism by thoroughly re-evaluating five state-of-the-art factuality metrics on a collection of 11 datasets for summarization, retrieval-augmented generation, and question answering. We find that these evaluators are inconsistent with each other and often misestimate system-level performance, both of which can lead to a variety of pitfalls. We further show that these metrics exhibit biases against highly paraphrased outputs and outputs that draw upon faraway parts of the source documents. We urge users of these factuality metrics to proceed with caution and manually validate the reliability of these metrics in their domain of interest before proceeding.

Figures

Figures reproduced from arXiv: 2501.14883 by the authors.

Figure 1
Figure 1. Selecting an AutoAIS evaluator based solely [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. TPR/TNR/BAcc of evaluators across datasets. Visualizing the breakdown of BAcc shows that AutoAIS evaluators can have a large gap between TPR and TNR. Moreover, evaluators with the same BAcc can have different TPR and TNR trade-offs. In the extreme case of ExpertQA, GPT-4-turbo has a TPR of 68% and TNR of 53%, while Bespoke-7B has nearly the opposite performance. the ability of metrics to detect unattributable claims… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: TPR/TNR vs ROUGE-2 precision of AutoAIS evaluators: ROUGE-2-precision is (anti-)correlated with true (negative)positive rate, i.e. metrics mislabel attributable generations with low ROUGE and unattributable generations with high ROUGE-2 precision [PITH_FULL_IMAGE:figu…
Figure 5
Figure 5. Figure 5: R2-diff vs rate of change in prediction with chunking: The figure shows the change in predictions of the Bespoke-7B evaluator to the same evaluator with documents chunked to 500 tokens. When chunking causes the overlap between the claim and the document to decrease (R2…
Figure 6
Figure 6. Figure 6: Predicted system-level error rate on RAGTruth (claim-level). Inconsistent predictions be￾tween different metrics lead to discrepancies in the quan￾tification of the system error rate. dataset of LLM-AGGREFACT [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 8
Figure 8. Figure 8: Intersection-over-Union of "unattributable" predictions by gpt-4-turbo and Bespoke-7B. IoU less than 50% on 5 of 14 datasets shows that the top-performing models (with very similar balanced accuracy of 76.2% and 77.4% respectively) have low consistency on what examples…
Figure 9
Figure 9. Figure 9: Pairwise Intersection-over-Union of "unattributable" predictions by AutoAIS metrics. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Pairwise Intersection-over-Union of "attributable" predictions by AutoAIS metrics. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 2 canonical work pages

  1. [1]

    Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. 2023 a . https://arxiv.org/abs/2305.14908 Purr: Efficiently editing language model hallucinations by denoising language model corruptions . Preprint, arXiv:2305.14908

    arXiv 2023

  2. [2]

    Hung-Ting Chen, Fangyuan Xu, Shane Arora, and Eunsol Choi. 2023 b . https://arxiv.org/abs/2310.12150 Understanding retrieval augmentation for long-form question answering . Preprint, arXiv:2310.12150

    arXiv 2023

  3. [3]

    Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.196 Complex claim verification with evidence retrieved in the wild . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Paper...

    doi:10.18653/v1/2024.naacl-long.196 2024

  4. [4]

    George Forman. 2006. https://doi.org/10.1145/1150402.1150423 Quantifying trends accurately despite classifier error and class imbalance . In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, page 157–166, New York, NY, USA. Association for Computing Machinery

    arXiv 2006

  5. [5]

    Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. https://doi.org/10.18653/v1/2023.acl-long.910 RARR : Researching and revising what language models say, using language models . In Proceedings of the 61st Annual Meeting of the Association for Co...

    doi:10.18653/v1/2023.acl-long.910 2023

  6. [6]

    Chawla, and Juan Jos\' e Del Coz

    Pablo Gonz\' a lez, Alberto Casta\ n o, Nitesh V. Chawla, and Juan Jos\' e Del Coz. 2017. https://doi.org/10.1145/3117807 A review on quantification learning . ACM Comput. Surv., 50(5)

    doi:10.1145/3117807 2017

  7. [7]

    Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. https://doi.org/10.18653/v1/2022.dialdoc-1.19 TRUE : Re-evaluating factual consistency evaluation . In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversatio...

    doi:10.18653/v1/2022.dialdoc-1.19 2022

  8. [8]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.248 Prometheus 2: An open source language model specialized in evaluating other language models . In Proceedings of the 2024 Conference on Empirical Methods in Na...

    doi:10.18653/v1/2024.emnlp-main.248 2024

Show all 30 references
  1. [9]

    Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. https://aclanthology.org/2021.wmt-1.57/ To ship or not to ship: An extensive evaluation of automatic metrics for machine translation . In Proceedings of the...

    2021

  2. [10]

    Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. https://doi.org/10.18653/v1/2023.eacl-main.121 L ong E val: Guidelines for human evaluation of faithfulness in long-form summarization . In Proceedings of the 17th Confere...

    2023 doi

  3. [11]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. https://arxiv.org/abs/2403.13787 Rewardbench: Evaluating reward models for language mo...

    2024 arXiv

  4. [12]

    Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. 2024. https://doi.org/10.18653/v1/2024.findings-acl.886 A ttribution B ench: How hard is automatic attribution evaluation? In Findings of the Association for Computational Linguistics: ACL 2024, pages 14919--14935, Bangkok, Thailan...

    2024 doi

  5. [13]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  6. [14]

    Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. https://doi.org/10.18653/v1/2020.acl-main.448 Tangled up in BLEU : Reevaluating the evaluation of automatic machine translation evaluation metrics . In Proceedings of the 58th Annual Meeting of the Association for Computat...

    2020 doi

  7. [15]

    Iftitahu Nimah, Meng Fang, Vlado Menkovski, and Mykola Pechenizkiy. 2023. https://doi.org/10.18653/v1/2023.acl-long.69 NLG evaluation metrics beyond correlation analysis: An empirical metric preference checklist . In Proceedings of the 61st Annual Meeting of the Association fo...

    2023 doi

  8. [16]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.585 RAGT ruth: A hallucination corpus for developing trustworthy retrieval-augmented language models . In Proceedings of the 62n...

    2024 doi

  9. [17]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  10. [18]

    Sanjana Ramprasad and Byron C. Wallace. 2024. https://arxiv.org/abs/2411.16638 Do automatic factuality metrics measure factuality? a critical evaluation . Preprint, arXiv:2411.16638

    2024

  11. [19]

    Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. https://doi.org/10.1162/coli_a_00486 Measuring attribution in natural language generation models . Computational Ling...

    2023 doi

  12. [20]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. https://doi.org/10.18653/v1/2020.acl-main.442 Beyond accuracy: Behavioral testing of NLP models with C heck L ist . In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu...

    2020 doi

  13. [21]

    Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M

    Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, and Mitesh M. Khapra. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.575 Perturbation C heck L ists for evaluating NLG evaluation metrics . In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan...

    2021 doi

  14. [22]

    Melanie Subbiah, Faisal Ladhak, Akankshya Mishra, Griffin Thomas Adams, Lydia Chilton, and Kathleen McKeown. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.557 STORYSUMM : Evaluating faithfulness in story summarization . In Proceedings of the 2024 Conference on Empirical Me...

    2024 doi

  15. [23]

    Liyan Tang, Philippe Laban, and Greg Durrett. 2024 a . https://arxiv.org/abs/2404.10774 Minicheck: Efficient fact-checking of llms on grounding documents . Preprint, arXiv:2404.10774

    2024 arXiv

  16. [24]

    Liyan Tang, Philippe Laban, and Greg Durrett. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.499 M ini C heck: Efficient fact-checking of LLM s on grounding documents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818-...

    2024 doi

  17. [25]

    Liyan Tang, Igor Shalyminov, Amy Wong, Jon Burnsky, Jake Vincent, Yu ' an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKeown. 2024 c . https://doi.org/10.18653/v1/2024.naacl-long.251 T ofu E val: Evaluating hallucinatio...

    2024 doi

  18. [26]

    Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.949 Foundational autoraters: Taming large language models for better automatic evaluation . In Proceedings of the 2024 Conference on Empi...

    2024 doi

  19. [27]

    David Wan, Jesse Vig, Mohit Bansal, and Shafiq Joty. 2024. https://arxiv.org/abs/2410.23609 On positional bias of faithfulness for long-form summarization . Preprint, arXiv:2410.23609

    2024 arXiv

  20. [28]

    Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.307 Automatic evaluation of attribution by large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4615--4635...

    2023 doi

  21. [29]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  22. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.