Pith. sign in

REVIEW 3 major objections 6 minor 40 references

FActBench: A Benchmark for Fine-grained Automatic Evaluation of LLM-Generated Text in the Medical Domain

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FActBench argues that a unanimous vote of NLI and chain-of-thought fact-checkers tracks medical-expert judgment more closely than either method alone, except for open-ended generation.

desk verdict The benchmark is a genuine resource, but the central claim that UnVot 'correlates best' with expert evaluation is not supported by the analysis—only task-level means are compared, no per-generation correlation is computed. read the letter →

arxiv 2509.02198 v1 pith:H5MSPAPP submitted 2025-09-02 cs.CL

Anum Afzal , Juraj Vladika , Florian Matthes This is my paper
classification cs.CL
keywords fact-checkingbenchmarkmedicalLLMevaluationunanimousvotingnaturallanguageinferencechain-of-thoughtpromptingFActScorehallucinationdetectionretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FActBench is a benchmark for automatic fact-checking of medical text generated by large language models. It tests six LLMs across four tasks: abstractive summarization, lay summarization, retrieval-augmented question answering, and open-ended generation. The paper's central claim is that an 'Unanimous Voting' score—counting an atomic fact as true only when both natural-language-inference (NLI) and chain-of-thought (CoT) fact-checkers support it—matches domain-expert scores better than the standard FActScore baseline, for tasks where a grounding document exists. The exception is open-ended generation, where the plain FActScore baseline tracks the experts best. The authors present this as evidence that cheap automatic fact-checking can substitute for expensive expert annotation in grounded medical NLP evaluation.

What carries the argument

Unanimous Voting (UnVot): the decision rule that an atomic fact is factually correct only if both components agree—the NLI model (Tasksource DeBERTa-v3-long-nli, using entailment vs contradiction) and the chain-of-thought fact-checker (an adaptation of FActScore that accepts user-provided grounding documents and uses GPT-4o mini with an automatic topic generator). The hybrid intrinsic-extrinsic pipeline is the carrier: first check each atomic fact against the supplied document; send only rejected facts to Wikipedia for a second opinion. This two-stage design is what lets UnVot avoid the baseline's underestimation and CoT's overestimation.

What would settle it

Take the 80 human-annotated generations (20 per task) and compute Spearman or Pearson correlations between each automatic method's scores and the expert scores at the per-generation level. If UnVot's per-generation correlation is not consistently higher than FActScore's on summarization, lay summarization, and RAG—or if baseline FActScore beats it on RAG as it does on open-ended generation—the paper's central claim fails.

Watch

Extended reading notes

Core claim

The core discovery the authors seek to establish is that combining two independent fact-checking signals by unanimous voting yields the best automatic approximation of domain-expert factuality judgments in the medical domain, on grounded generation tasks. They break every generated text into atomic facts following Min et al. (2023), check each fact first against the source grounding document (intrinsic) and then, for facts marked unsupported, against Wikipedia (extrinsic). An atomic fact is certified only if both the NLI engine (DeBERTa-v3 fine-tuned for long inputs) and the adapted FActScore-style CoT engine (GPT-4o mini) judge it supported. Aggregating these votes into sentence-level factu

Load-bearing premise

The paper's claim that UnVot 'correlates best' rests on comparing average scores per task (Table 3) rather than computing per-generation correlation coefficients; if per-output correlations were used, the ranking of unanimous voting versus NLI or FActScore could change.

Editorial extensions

If this is right

  • For grounded medical tasks, automatic UnVot scores could replace expensive, slow expert annotation in routine LLM evaluation.
  • Text summarization, lay summarization, and RAG outputs can be fact-checked with one combined pipeline instead of choosing between NLI and CoT.
  • Open-ended medical generation should be evaluated with the original FActScore baseline, not the unanimous rule, since that is where the vote performs worst.
  • The hybrid intrinsic-then-extrinsic check raises factuality scores for grounded tasks; benchmark designers should include both a source document and a world-knowledge source.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper compares only task-level means, its 'correlates best' claim would be tested more directly by per-generation correlations; those might rank the methods differently on the 80 human-annotated outputs.
  • UnVot is essentially a high-precision gate: requiring both methods to agree will discard facts that only one method recognizes, which likely explains the poor open-ended result, where NLI alone scores near 30.
  • The same intrinsic-then-extrinsic UnVot recipe should transfer to other high-stakes domains such as legal or financial text, where grounded and ungrounded outputs coexist; the benchmark's prompts and pipeline are domain-agnostic.
  • Since each technique uses a single backbone (one NLI model, one CoT LLM), the measured agreement with experts may be a property of those particular models rather than of the voting rule itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. FActBench presents a benchmark for fine-grained automatic fact-checking of LLM-generated medical text across four generation tasks (text summarization, lay summarization, RAG, and open-ended generation) and six LLMs. The authors compare two fact-checking techniques—Chain-of-Thought (CoT) prompting (an adaptation of FActScore called FActScore+) and Natural Language Inference (NLI)—plus their combination via Unanimous Voting (UnVot), against a FActScore baseline. For grounding, they use a hybrid intrinsic/extrinsic method: first checking against the source document, then checking remaining claims against Wikipedia. They report domain-expert human evaluation on 80 generations (20 per task) and claim that UnVot scores correlate best with human judgment, while also noting that for pure generation the baseline correlates best. The paper also reports factuality scores across models and tasks and discusses the effect of grounding on hallucination rates.

Significance. If the central claim were adequately supported, FActBench would be a useful resource for evaluating factuality in medical LLM outputs: it covers four realistic tasks, six models, and two knowledge sources, and UnVot is a simple, parameter-free aggregation rule that is not fitted to human scores. I see no circularity in the human evaluation design: the UnVot rule is fixed and the human scores are used only for validation. The shared use of GPT-4o mini as both generator and CoT fact-checker is a potential confound but not circular. The benchmark itself and the code release would be valuable to the community. However, the headline claim of correlation with expert judgment is currently not supported by the statistical evidence presented, which is a load-bearing gap.

major comments (3)
  1. [Section 4.1, Table 3] The claim that UnVot 'correlates best with Domain Expert Evaluation' is not established. The evidence is a comparison of task-level means: for each task, the average human score is compared with average CoT, NLI, UnVot, and baseline scores. 'Correlates best' is an association claim that requires paired per-generation comparisons between each automatic method and the human score (e.g., Spearman or Pearson), with significance testing. With only four task-level points, no meaningful correlation can be computed, and closeness of means measures calibration/bias, not correlation. The Limitation section itself concedes that the techniques are 'not perfect' and that there 'will always be some mishaps,' which makes the statistical support more, not less, necessary. Please release per-output scores and report correlation coefficients with confidence intervals and significance tests.
  2. [Abstract and Section 4.1] The abstract states without qualification that 'fact-checking scores acquired through the Unanimous Voting of both techniques correlate best with Domain Expert Evaluation.' Section 4.1 explicitly concedes that for pure generation, the baseline FActScore system correlates best. The claim should be qualified to summarization, lay summarization, and RAG tasks, or else supported by a pooled analysis across all tasks. As written, the abstract overstates the findings and is inconsistent with the body of the paper.
  3. [Section 3.1, Human Evaluation] The human evaluation consists of 80 generations (20 per task), each scored by two annotators. Cohen's kappa 0.75 is reported, but no per-generation human scores or automatic scores are provided or analyzed. With n=80 overall and n=20 per task, the comparison is vulnerable to outliers and lacks statistical power. Please report per-output correlation coefficients with confidence intervals or bootstrap intervals, and ideally scatterplots, so that the reader can judge whether the ranking of methods is stable. The current aggregate means alone do not support the central contribution.
minor comments (6)
  1. [Table 2] Some entries have inconsistent decimal precision (e.g., '85.28', '82.84', '80.38' appear with two decimals while most entries have one). Please use a consistent number of decimal places throughout the table.
  2. [Section 3.1] The annotation design should be clarified: the text says '8 in-house employed individuals' were recruited, but also that 'each generation was evaluated by two annotators.' Please specify how the 80 generations were assigned across the 8 annotators (e.g., number of annotations per annotator, overlap structure).
  3. [Section 3.1] The term 'FActScore+' is used but not explicitly defined. Please define it at first use as the adapted CoT fact-checking method with grounding-document support.
  4. [Table 3] Consider reporting per-task standard deviations or ranges for the human scores. The means alone mask variability, which is especially relevant given the small sample size.
  5. [References] The G-Eval reference appears twice (Liu et al., 2023a and 2023b) with the same title; please consolidate into a single reference.
  6. [Throughout] The word 'correlates' is used in the abstract and Section 4.1 to describe comparisons of aggregate means. Since no correlation coefficient is computed, consider using 'agrees with' or 'is closest to' until paired correlation analysis is provided.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the benchmark methods are fixed rules and human evaluation is independent.

full rationale

The paper's central claim that Unanimous Voting (UnVot) of CoT and NLI correlates best with domain expert evaluation is an empirical comparison, not a derivation that reduces to its inputs. UnVot is defined as a fixed conjunction rule (both NLI and CoT must support an atomic fact), with no parameters fitted to the human scores. The human evaluation is conducted separately on 80 generations and is not used to tune, select, or define any automatic metric. There are no self-citations that carry load-bearing arguments, and no 'uniqueness theorem' or ansatz is imported from the authors' prior work. The only mild concern—GPT-4o mini serving both as a generator and as the CoT fact-checker—is a potential methodological bias, not a circularity. The paper's 'correlates best' is supported only by comparing task-level mean scores (Table 3), which is a statistical weakness (no per-generation correlation coefficients), but this is an issue of evidence strength, not of self-referential reasoning. Therefore, no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on the trustworthiness of its automatic evaluators and on the representativeness of the human gold standard. No free parameters are fitted to the human scores; the UnVot rule is a pre-defined AND. The main risk is that the NLI and CoT judges may be systematically wrong in the medical domain, which would invalidate all benchmark scores.

assumptions (4)
  • domain assumption DeBERTa-v3 (Tasksource) NLI predictions reliably distinguish supported from hallucinated atomic facts in medical text.
    Used as the NLI judge in Section 3.1; no validation is provided for this specific medical setting.
  • domain assumption GPT-4o mini with CoT prompting (FActScore+) reliably fact-checks atomic facts against grounding documents and Wikipedia.
    Used as the CoT judge in Section 3.1; the paper notes limitations that the model may make incorrect verdicts.
  • domain assumption The 80 human-annotated generations (20 per task) with two annotators each are an adequate gold standard for measuring agreement.
    Section 3.1 describes the human evaluation; the sample size is small and no analysis of per-task reliability other than Cohen's kappa = 0.75 is given.
  • domain assumption The four chosen datasets and prompts are representative of medical LLM generation tasks.
    Section 3.2 selects PubMed, PLOS, and BioASQ; the paper concedes these may not capture the entirety of LLM performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FActBench: A Benchmark for Fine-grained Automatic Evaluation of LLM-Generated Text in the Medical Domain." pith.science (2026). https://pith.science/paper/H5MSPAPP

@misc{pith2026250902198,
  author       = {Pith},
  title        = {Pith review of: FActBench: A Benchmark for Fine-grained Automatic Evaluation of LLM-Generated Text in the Medical Domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5MSPAPP}},
  note         = {Machine review of arXiv:2509.02198}
}
read the original abstract

Large Language Models tend to struggle when dealing with specialized domains. While all aspects of evaluation hold importance, factuality is the most critical one. Similarly, reliable fact-checking tools and data sources are essential for hallucination mitigation. We address these issues by providing a comprehensive Fact-checking Benchmark FActBench covering four generation tasks and six state-of-the-art Large Language Models (LLMs) for the Medical domain. We use two state-of-the-art Fact-checking techniques: Chain-of-Thought (CoT) Prompting and Natural Language Inference (NLI). Our experiments show that the fact-checking scores acquired through the Unanimous Voting of both techniques correlate best with Domain Expert Evaluation.

Figures

Figures reproduced from arXiv: 2509.02198 by the authors.

Figure 1
Figure 1. Block Diagram depicting how different fact-checking techniques interact with different data sources. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 9 canonical work pages

  1. [1]

    Mubashara Akhtar, Michael Schlichtkrull, and Andreas Vlachos. 2024. https://arxiv.org/abs/2411.05375 Ev2r: Evaluating evidence retrieval in automated fact-checking . Preprint, arXiv:2411.05375

    arXiv 2024

  2. [2]

    Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. 2023. https://arxiv.org/abs/2310.05189 Factuality challenges in the...

    arXiv 2023

  3. [3]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1--45

    2024

  4. [4]

    Xiang Chen, Duanzheng Song, Honghao Gui, Chenxi Wang, Ningyu Zhang, Yong Jiang, Fei Huang, Chengfei Lv, Dan Zhang, and Huajun Chen. 2024. https://arxiv.org/abs/2310.12086 Factchd: Benchmarking fact-conflicting hallucination detection . Preprint, arXiv:2310.12086

    arXiv 2024

  5. [5]

    I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023. Factool: Factuality detection in generative ai--a tool augmented framework for multi-task and multi-domain scenarios. arXiv preprint arXiv:2307.13528

    arXiv 2023

  6. [6]

    Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia L \"o ffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. 2023. The future landscape of large language models in medicine. Communications medicine, 3(1):141

    work page 2023

  7. [7]

    Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. https://doi.org/10.18653/v1/N18-2097 A discourse-aware attention model for abstractive summarization of long documents . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human...

    doi:10.18653/v1/n18-2097 2018

  8. [8]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. https://doi.org/10.18653/v1/2024.findings-acl.212 Chain-of-verification reduces hallucination in large language models . In Findings of the Association for Computational Linguistics: ACL 2024, pages 3563--3578, Bangkok, Thailand. Association...

    doi:10.18653/v1/2024.findings-acl.212 2024

Show all 40 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  2. [10]

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. 2024. https://doi.org/10.18653/v1/2024.findings-acl.558 Fact-checking th...

    2024 doi

  3. [11]

    Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.724 Making science simple: Corpora for the lay summarisation of scientific literature . In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan...

    2022 doi

  4. [12]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://openreview.net/forum?id=sE7-XhLxHA De BERT av3: Improving de BERT a using ELECTRA -style pre-training with gradient-disentangled embedding sharing . In The Eleventh International Conference on Learning Representations

    2023

  5. [13]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . Preprint, arXiv:2006.03654

    2021 arXiv

  6. [14]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Info...

    2025

  7. [15]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. https://doi.org/10.1145/3571730 Survey of hallucination in natural language generation . ACM Comput. Surv., 55(12)

    2023 doi

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  9. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  10. [18]

    Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. 2024 b . https://openreview.net/forum?id=EE1CBKC0SZ TIGERS core: Towards building explainable metric for all text generation tasks . Transactions on Machine Learning Research

    2024

  11. [19]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. https://openreview.net/forum?id=8euJaTveKw Prometheus: Inducing fine-grained evaluation capability in language models . In...

    2024

  12. [20]

    Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. https://doi.org/10.1038/s41597-023-02068-4 Bioasq-qa: A manually curated corpus for biomedical question answering . Scientific Data, 10(1)

    2023 doi

  13. [21]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. https://doi.org/10.1162/tacl_a_00453 S umma C : Re-visiting NLI -based models for inconsistency detection in summarization . Transactions of the Association for Computational Linguistics, 10:163--177

    2022 doi

  14. [22]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 a . G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511--2522

    2023

  15. [23]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  16. [24]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  17. [25]

    Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. 2021. https://doi.org/10.18653/v1/2021.naacl-main.104 Looking beyond sentence-level natural language inference for question answering and text summarization . In...

    2021 doi

  18. [26]

    Guy Mor-Lan and Effi Levi. 2024. https://doi.org/10.18653/v1/2024.starsem-1.15 Exploring factual entailment with NLI : A news media study . In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 190--199, Mexico City, Mexico. Asso...

    2024 doi

  19. [27]

    Hithesh Sankararaman, Mohammed Nasheed Yasin, Tanner Sorensen, Alessandro Di Bari, and Andreas Stolcke. 2024. https://doi.org/10.18653/v1/2024.emnlp-industry.97 Provenance: A light-weight fact-checker for retrieval augmented LLM generation output . In Proceedings of the 2024 C...

    2024 doi

  20. [28]

    Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari, et al. 2021. Questeval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processin...

    2021

  21. [29]

    Damien Sileo. 2023. https://arxiv.org/abs/2301.05948 tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation . arXiv preprint arXiv:2301.05948

    2023 arXiv

  22. [30]

    Julius Steen, Juri Opitz, Anette Frank, and Katja Markert. 2023. https://doi.org/10.18653/v1/2023.acl-short.79 With a little push, NLI models can robustly and efficiently predict faithfulness . In Proceedings of the 61st Annual Meeting of the Association for Computational Ling...

    2023 doi

  23. [31]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  24. [32]

    Prasetya Utama, Joshua Bambrick, Nafise Sadat Moosavi, and Iryna Gurevych. 2022. Falsesum: Generating document-level nli examples for recognizing factual inconsistency in summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for ...

    2022

  25. [33]

    Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. https://arxiv.org/abs/2307.03987 A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation . Preprint, arXiv:2307.03987

    2023 arXiv

  26. [34]

    Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966

    2023 arXiv

  27. [35]

    Chi, Quoc Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. https://arxiv.org/abs/2201.11903 Chain of thought prompting elicits reasoning in large language models . CoRR, abs/2201.11903

    2022 arXiv

  28. [36]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023. https://arxiv.org/abs/2309.01219 Siren's song in the ai ocean: A survey on hallucination in ...

    2023 arXiv

  29. [37]

    Yiran Zhao, Jinghan Zhang, I Chern, Siyang Gao, Pengfei Liu, Junxian He, et al. 2024. Felm: Benchmarking factuality evaluation of large language models. Advances in Neural Information Processing Systems, 36

    2024

  30. [38]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  31. [39]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.