Pith. sign in

REVIEW 3 major objections 4 minor 14 cited by

The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FACTS Grounding is an online benchmark and leaderboard that measures whether LLM long-form responses are fully grounded in a supplied context document while still answering the user request, using a validated ensemble of judge models.

desk verdict Useful and transparent benchmark, but the judge calibration is too weak to trust absolute scores or close rankings. read the letter →

arxiv 2501.03200 v1 pith:T44GF7MR submitted 2025-01-06 cs.CL

Alon Jacovi , Andrew Wang , Chris Alberti , Connie Tao , Jon Lipovetz , Kate Olszewska , Lukas Haas , Michelle Liu
show 18 more authors
Nate Keating Adam Bloniarz Carl Saroufim Corey Fry Dror Marcus Doron Kukliansky Gaurav Singh Tomar James Swirhun Jinwei Xing Lily Wang Madhu Gurumurthy Michael Aaron Moran Ambar Rachana Fellinger Rui Wang Zizhao Zhang Sasha Goldshtein Dipanjan Das
This is my paper · ORCID
classification cs.CL
keywords factualitygroundinglong-formgenerationLLM-as-a-judgeevaluationbenchmarkleaderboardinstructionfollowinghallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces FACTS Grounding, an online benchmark and leaderboard for a specific failure mode of large language models: generating confident long-form answers that drift away from the source document in the prompt. Each of the 1,719 human-written test items pairs a non-trivial user request with a document of up to 32k tokens, and a model's response is scored by whether every information-bearing claim can be verified against that document. Scoring runs in two phases: a response that fails to address the request is first disqualified and counted as inaccurate, and the remaining responses are judged by three different LLM judges whose prompt templates were selected on a human-annotated held-out set. The paper's claim is that this pipeline gives a usable, continuously maintained measurement of grounding ability, and it reports a fused ranking of current models on that measure.

What carries the argument

The load-bearing mechanism is the two-phase judge pipeline. In the eligibility phase, three prompted LLM judges (Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet) each compare the response against the user request, and a response is declared ineligible only by consensus, so only the worst instruction-following failures are filtered out. In the factuality phase, the same three judge models each classify the full response as 'accurate' or 'inaccurate' against the context, using prompt templates chosen for the highest Macro-F1 (balanced class-averaged F1) against human labels on a private 402-response test set; the unadjusted and final factuality scores are averages of the judges' accuracy percentages. The benchmark itself supplies the task: 860 public and 859 private human-written prompt-document pairs with contexts up to 32k tokens, curated to avoid creative writing, expert knowledge, and heavy reasoning.

What would settle it

Take a few hundred responses sampled from the public leaderboard, have human raters label each response as fully grounded or not under the paper's own rubric, and compare those labels with the aggregate verdicts of the three judge models; if the macro-F1 of the judges against human labels on this sample falls materially below the 68.9–71.5 range found on the validation set, the leaderboard's scores and rankings are not measuring grounding as claimed.

Watch

Extended reading notes

Core claim

The paper's central claim is a measurement recipe: long-form grounding can be scored at scale by combining an instruction-following filter with three independent LLM judges whose prompts were tuned to match human judgments. Each judge labels a response 'accurate' only if every information-bearing claim is supported by the supplied document or needs no grounding, and the unadjusted factuality score is the average of the three judges' accuracy rates; a response is disqualified and counted as inaccurate only when all three judges agree that it fails the user request. The judge prompts were selected by Macro-F1 (the average of the F1 scores for accurate and inaccurate classes) on a held-out set of 402 human-annotated responses, and the final score is the average of judge percentages after disqualification. On this measure, the paper reports Gemini 2.0 Flash Experimental first and Gemini 1.5 Flash second, with scores in the low-to-mid 80s on both the public and blind splits.

Load-bearing premise

The rankings are trustworthy only if the three judge models, taken together, judge grounding the way human raters would on the leaderboard responses themselves; the paper validates the judges on only one held-out set of 402 responses.

Editorial extensions

If this is right

  • Models that answer strictly from the supplied document, even when doing so conflicts with knowledge stored during pretraining, will score higher on FACTS Grounding.
  • Short, evasive responses that would otherwise dodge factuality checks are penalized by the eligibility filter, which treats them as inaccurate and lowers the score.
  • Because the benchmark spans domains such as medical, legal, financial, and retail text and request types from fact-finding to summarization, a high score indicates general grounding skill rather than task-specific summarization ability.
  • The public and blind splits allow external model submissions to be compared while keeping a private set that resists overfitting to the leaderboard.
  • The reported fused ranking, produced by Condorcet aggregation of six judge-split rankings, is the paper's answer to which current models are best at long-form grounded generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the selected judge prompts show false positive rates of 32–56% on the paper's own validation set, a natural next step is to re-measure human agreement on live leaderboard responses; if the same leniency persists, the leaderboard scores would overstate true grounding accuracy.
  • The reported self-preference bias of +3.23% is not removed before ranking; recomputing scores with judges that are not themselves among the evaluated models would show how much of the final order depends on judge family.
  • The web-sourced context documents may already be memorized by many models; constructing a parallel split with synthetic or fresh documents would separate true grounding from retrieval from parametric memory.
  • The same public/private split and multi-judge scoring scheme could be applied to other long-context abilities, such as following complex instructions or ignoring misleading context, since the infrastructure is task-agnostic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces FACTS Grounding, an online leaderboard and benchmark for evaluating whether LLMs generate long-form responses that are grounded in a provided context document of up to 32k tokens while satisfying a user request. The dataset consists of 860 public and 859 private human-written examples spanning several domains and task types. Responses are evaluated in two stages: an eligibility filter disqualifies responses that do not sufficiently fulfill the user request, and three judge models (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet) then produce binary grounding verdicts, with the final factuality score being the average of the three judges' accuracy rates. The judge prompts were selected on a private N=402 human-labeled set using Macro-F1, and the eligibility prompt was selected on a separate N=450 set. The paper reports unadjusted and final factuality scores for nine models across Open and Blind splits, and presents a fused ranking derived from Condorcet aggregation.

Significance. If the measurement pipeline is trustworthy, FACTS Grounding would be a valuable community resource: it is one of the first benchmark-plus-leaderboard efforts for long-form grounding at 32k-token context, it uses fresh human-written requests rather than repurposed academic tasks, it includes public and private splits to limit overfitting, and it explicitly validates judge prompts against human labels. The paper also reports self-preference bias rather than ignoring it, and the decision to average three judge models is a reasonable step toward robustness. However, the central measurement claim rests on judge models whose validation shows high false positive rates and for which no human validation is reported on the actual leaderboard responses. Because the reported absolute scores and close ranking gaps in Tables 5 and 6 depend on these judges, the paper's central contribution is not yet fully established.

major comments (3)
  1. [Section 3.1 and Table 2] The chosen judge templates have false positive rates of 32.26% (GPT-4o JSON), 45.16% (Claude 3.5 Sonnet implicit span-level), and 56.45% (Gemini 1.5 Pro JSON). Since the final factuality score is the average of these three judges, a large fraction of unfaithful responses are labelled accurate, which likely inflates the absolute scores in Tables 5 and 6. The validation set has only 13% negative examples, so these FPR estimates rest on roughly 52 cases and are quite noisy, and no human validation is reported on the public or blind leaderboard responses. This missing calibration check is load-bearing for the claim that FACTS Grounding scores reflect human judgments of grounding.
  2. [Section 3.2 and Table 4] The eligibility filter selects prompt templates with false negative rates of 62.67% (Claude, user request only), 74.00% (Gemini, user request only), and 80.67% (GPT-4o, user request only), and then applies a consensus rule that requires all three judges to label a response ineligible. The combined effect is that most responses that fail to fulfill the user request are not disqualified, so the 'while fulfilling the user request' component of the benchmark is not effectively enforced. The 1%–5% score reduction reported in Section 4 is consistent with this concern, but the benchmark would be more convincing if it also reported the disqualification rate and the judges' agreement on the final split.
  3. [Section 4, 'On aggregating multiple judge models'] The paper reports a mean self-preference bias of +3.23% but does not correct for it, and the judge set includes the same model families that are ranked (Gemini, GPT-4o, and Claude). Since the top two models in Table 6 differ by only 0.7 points, the reported ranking may be sensitive to this bias. The authors should either provide a bias-corrected leaderboard, or analyze how the fused ranking changes when each judge is dropped or when self-preference is removed, so readers can assess whether the ordering in Tables 5 and 6 is robust.
minor comments (4)
  1. [Abstract and Section 1] There are several spacing and capitalization errors in the text, for example 'TheFACTS Grounding leaderboard' in the abstract and 'theF ACTS Groundingleaderboard' in Section 1; these should be corrected.
  2. [Section 2.2, data contamination point 3] The sentence 'As all frontier language models models were trained on large corpora of web data' contains a duplicated word 'models'; please remove the repetition.
  3. [Table 2 and Section 3.1] The selection criterion is reported as Macro-F1, but the chosen templates have very different FPR/FNR trade-offs; a short explanation of why Macro-F1, rather than a false-positive-sensitive criterion, is appropriate for this benchmark would help readers interpret the chosen judges.
  4. [Table 6] The final factuality score is described as the average of the three judge scores, but the table presents the three individual judge columns and an 'Average Fused' column; the caption should clarify that the 'Average Fused' column is the unweighted mean of the three judge-specific percentages.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FACTS Grounding anchors its judge models to human-annotated held-out sets, so leaderboard scores are not equivalent to their own inputs.

full rationale

The paper constructs a benchmark and leaderboard rather than deriving a theoretical quantity. Its central measurement chain is: human raters write prompts and human annotators provide golden labels on held-out sets; judge prompt templates are selected by Macro-F1 against those human labels (Table 2 for N=402 factuality labels, Table 4 for N=450 eligibility labels); final scores are then the average of three judge verdicts, with ineligible responses treated as inaccurate. This is an externally anchored operational metric, not a self-defined prediction. No fitted parameter is renamed as a predicted outcome: the judge templates are validated, not used to predict the human labels on the leaderboard responses, and the disclosed high false positive rates and +3.23% self-preference bias are measurement-validity concerns, not circular reductions. The fused Condorcet ranking is an aggregation of the same judge scores and is explicitly shown to align with the final factuality score, so it introduces no independent circular claim. Citations to the authors' prior work (e.g., Jacovi et al. 2024 on Coverbench, Sainz et al. 2024 on data contamination) are contextual and not load-bearing for the benchmark's validity. No step in the paper reduces by construction to its own input, and no uniqueness or ansatz is imported from self-citations. Therefore no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on the assumption that LLM judges can reliably reproduce human grounding judgments and that ensembling removes self-preference. The paper provides one small human-labeled set for prompt selection and acknowledges self-preference without correcting it. No new physical or conceptual entities are introduced.

assumptions (4)
  • domain assumption LLM judge verdicts are a valid proxy for human judgments of grounding on the FACTS benchmark.
    The paper validates judge prompts on N=402 human-labeled responses and N=450 for eligibility, but the final public and blind splits are not human-validated. The selected judges have Macro-F1 from 68.85 to 71.47 and false positive rates from 32.26% to 56.45% (Table 2).
  • domain assumption Averaging three judge models is sufficient to mitigate self-preference and evaluator bias.
    The paper reports a mean +3.23% self-preference boost (Section 4) but does not correct for it. Models without a judge in the panel, such as o1-mini and o1-preview, may be systematically disadvantaged.
  • domain assumption A response can be reduced to a binary grounding label, where one unsupported claim makes the whole response inaccurate.
    Used in Section 3.1: 'a single claim that bears information is deemed to be not grounded in the contents of the prompt.' This simplifies long-form evaluation but may be harsh for partially grounded responses.
  • domain assumption Web-sourced context documents being present in pretraining corpora does not invalidate the grounding measurement.
    The paper argues this in Section 2.2 on data contamination. User requests are novel, but models could still rely on memorized document content, so the score may partly measure retrieval from memory rather than online grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input." pith.science (2026). https://pith.science/paper/T44GF7MR

@misc{pith2026250103200,
  author       = {Pith},
  title        = {Pith review of: The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T44GF7MR}},
  note         = {Machine review of arXiv:2501.03200}
}
read the original abstract

We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models' ability to generate text that is factually accurate with respect to given context in the user prompt. In our benchmark, each prompt includes a user request and a full document, with a maximum length of 32k tokens, requiring long-form responses. The long-form responses are required to be fully grounded in the provided context document while fulfilling the user request. Models are evaluated using automated judge models in two phases: (1) responses are disqualified if they do not fulfill the user request; (2) they are judged as accurate if the response is fully grounded in the provided document. The automated judge models were comprehensively evaluated against a held-out test-set to pick the best prompt template, and the final factuality score is an aggregate of multiple judge models to mitigate evaluation bias. The FACTS Grounding leaderboard will be actively maintained over time, and contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard. It can be found at https://www.kaggle.com/facts-leaderboard.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Hallucination Self-Play co-evolves a generator and detector from one base LLM via RLAIF and RLVR, lifting a 7B model to match advanced LLMs on RAGTruth faithfulness detection.

  2. DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs

    cs.CL 2025-06 conditional novelty 7.0 of 10

    The paper introduces a taxonomy and benchmark for knowledge conflicts in search-augmented LLMs, and experiments show that prompting for conflict type improves response quality.

  3. Counterfactual Benchmarking and Training for Factuality Consistency and Order-Robust Grounded Reasoning in LLMs over Heterogeneous Knowledge

    cs.AI 2026-08 conditional novelty 6.0 of 10

    TKFQA evaluates factuality and input-order robustness in multi-hop grounded reasoning over tables, texts, and knowledge graphs, and ORLF improves performance and order robustness on this benchmark.

  4. When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

    cs.AI 2026-02 reject novelty 6.0 of 10

    Nearly half of 60 widely used LLM benchmarks show saturation, which increases with age; private test access does not prevent saturation, while expert curation does.

  5. A Neurosymbolic Approach to Natural Language Formalization and Verification

    cs.CL 2025-11 conditional novelty 6.0 of 10

    A neurosymbolic guardrail reports 99.2% soundness on a 522-item policy QA benchmark, mainly by rejecting 84% of correct answers; unvetted real-world policies score 96.8%.

  6. StructText: A Synthetic Table-to-Text Approach for Benchmark Generation with Multi-Dimensional Evaluation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A synthetic table-to-text pipeline that generates and validates key-value extraction benchmarks, revealing that LLM-generated reports keep numerical facts intact but are poorly machine-extractable.

  7. How Does Response Length Affect Long-Form Factuality

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Longer LLM responses have lower factual precision, and the main cause appears to be the model exhausting reliable knowledge on a topic, not error propagation or long context.

  8. Towards Large Reasoning Models for Agriculture

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A new 100-question agricultural reasoning benchmark and a 44.6K-question training dataset show current AI models score at most 36%, and fine-tuning small models on the dataset lifts them from 0-1% to 3-5%.

  9. LIFEBench: Evaluating Length Instruction Following in Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    LIFEBench's evaluation of 26 LLMs shows most follow short length instructions but degrade sharply beyond a few hundred words, and none reliably hit vendor-claimed maximum output lengths.

  10. AI and Authenticity in Islamic Research: A Critical Evaluation of Generative AI Reliability, Hallucination, and Source Fidelity in Quranic, Hadith, and Fiqh Knowledge

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Leading generative AIs are useful for introductory Islamic learning but unreliable as authorities on Fiqh, citations, and madhhab-sensitive rulings without human verification.

  11. Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models

    cs.CL 2026-02 conditional novelty 5.0 of 10

    A step-level reinforcement-learning reward combining a process reward model with truncated resampling reduces chain-of-thought faithfulness hallucinations in small reasoning models.

  12. ConSens: Assessing context grounding in open-book question answering

    cs.CL 2025-04 conditional novelty 5.0 of 10

    ConSens measures context grounding in open-book QA as a sigmoid-transformed log ratio of answer perplexity without context to perplexity with context, reaching ROC AUC 0.88 to 0.93.

  13. Evaluating LLM Metrics Through Real-World Capabilities

    cs.AI 2025-05 reject novelty 4.0 of 10

    An analysis of survey and usage data yields six LLM capabilities, a benchmark-gap map, and a model comparison that ranks Gemini first, but the ranking is built on subjectively selected and self-reported benchmarks.

  14. Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation

    cs.AI 2025-05 conditional novelty 4.0 of 10

    The authors argue that time-bound AI competitions with hidden test data are the gold standard for generative AI evaluation, and that public static benchmarks should be considered contaminated once published.

Reference graph

Works this paper leans on

39 extracted references · 11 canonical work pages · cited by 14 Pith papers

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The Claude 3 model family: Opus , Sonnet , Haiku , 2024

    Anthropic . The Claude 3 model family: Opus , Sonnet , Haiku , 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf

    work page 2024

  3. [3]

    J. A. Bishop, Q. Xie, and S. Ananiadou. LongDocFACTScore : Evaluating the factuality of long document abstractive summarisation. arXiv preprint arXiv:2309.12455, 2023

    arXiv 2023

  4. [4]

    Chang, K

    Y. Chang, K. Lo, T. Goyal, and M. Iyyer. Booookscore: A systematic exploration of book-length summarization in the era of LLMs . arXiv preprint arXiv:2310.00785, 2023

    arXiv 2023

  5. [5]

    Gekhman, J

    Z. Gekhman, J. Herzig, R. Aharoni, C. Elkind, and I. Szpektor. Trueteacher: Learning factual consistency evaluation with large language models. arXiv preprint arXiv:2305.11171, 2023

    arXiv 2023

  6. [6]

    Introducing gemini 2.0: our new ai model for the agentic era, 2024

    Gemini Team . Introducing gemini 2.0: our new ai model for the agentic era, 2024. URL https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024. Accessed: 2024-12-11

    work page 2024

  7. [7]

    Anil , S

    Gemini Team: R. Anil , S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  8. [8]

    Honovich, R

    O. Honovich, R. Aharoni, J. Herzig, H. Taitelbaum, D. Kukliansy, V. Cohen, T. Scialom, I. Szpektor, A. Hassidim, and Y. Matias. TRUE : Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991, 2022

    arXiv 2022

Show all 39 references
  1. [9]

    Huang and Y.-N

    C.-W. Huang and Y.-N. Chen. FactAlign : Long-form factuality alignment of large language models. arXiv preprint arXiv:2410.01691, 2024

    2024 arXiv

  2. [10]

    Jacovi, M

    A. Jacovi, M. Ambar, E. Ben-David, U. Shaham, A. Feder, M. Geva, D. Marcus, and A. Caciularu. Coverbench: A challenging benchmark for complex claim verification. arXiv preprint arXiv:2408.03325, 2024

    2024 arXiv

  3. [11]

    A. T. Kalai and S. S. Vempala. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 160--171, 2024

    2024

  4. [12]

    Karpinska, K

    M. Karpinska, K. Thai, K. Lo, T. Goyal, and M. Iyyer. One thousand and one pairs: A "novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024

    2024 arXiv

  5. [13]

    Y. Kim, Y. Chang, M. Karpinska, A. Garimella, V. Manjunatha, K. Lo, T. Goyal, and M. Iyyer. FABLES : Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261, 2024

    2024 arXiv

  6. [14]

    Krishna, E

    K. Krishna, E. Bransom, B. Kuehl, M. Iyyer, P. Dasigi, A. Cohan, and K. Lo. LongEval : Guidelines for human evaluation of faithfulness in long-form summarization. arXiv preprint arXiv:2301.13298, 2023

    2023 arXiv

  7. [15]

    Z. Lan, W. Li, J. Su, X. Xiao, J. Liu, W. Wu, and Y. Lyu. Factgen: Faithful text generation by factuality-aware pre-training and contrastive ranking fine-tuning. Journal of Artificial Intelligence Research, 76: 0 1281--1303, 2023

    2023

  8. [16]

    S. Lee, H. Hsu, and C.-F. Chen. LLM hallucination reasoning with zero-shot knowledge test. arXiv preprint arXiv:2411.09689, 2024

    2024

  9. [17]

    J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen. HaluEval : A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747, 2023

    2023 arXiv

  10. [18]

    Y. Liu, N. Moosavi, and C. Lin. LLM s as narcissistic evaluators: When ego inflates evaluation scores. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12688--12701, Bangkok, Thailand, Aug. 2024. Asso...

    2024 doi

  11. [19]

    S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023

    2023 arXiv

  12. [20]

    Learning to reason with LLMs , 2024

    OpenAI . Learning to reason with LLMs , 2024. URL https://openai.com/index/learning-to-reason-with-llms

    2024

  13. [21]

    L. Pan, X. Wu, X. Lu, A. T. Luu, W. Y. Wang, M.-Y. Kan, and P. Nakov. Fact-checking complex claims with program-guided reasoning. arXiv preprint arXiv:2305.12744, 2023

    2023 arXiv

  14. [22]

    Ramprasad and B

    S. Ramprasad and B. C. Wallace. Do automatic factuality metrics measure factuality? A critical evaluation. arXiv preprint arXiv:2411.16638, 2024

    2024

  15. [23]

    Ramprasad, K

    S. Ramprasad, K. Krishna, Z. C. Lipton, and B. C. Wallace. Evaluating the factuality of zero-shot summarizers across varied domains. arXiv preprint arXiv:2402.03509, 2024

    2024 arXiv

  16. [24]

    Rashkin, V

    H. Rashkin, V. Nikolaev, M. Lamm, L. Aroyo, M. Collins, D. Das, S. Petrov, G. S. Tomar, I. Turc, and D. Reitter. Measuring attribution in natural language generation models. Computational Linguistics, 49 0 (4): 0 777--840, 2023

    2023

  17. [25]

    P. Roit, J. Ferret, L. Shani, R. Aharoni, G. Cideron, R. Dadashi, M. Geist, S. Girgin, L. Hussenot, O. Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00186, 2023

    2023 arXiv

  18. [26]

    Sainz, I

    O. Sainz, I. Garc \' a-Ferrero, A. Jacovi, J. A. Campos, Y. Elazar, E. Agirre, Y. Goldberg, W.-L. Chen, J. Chim, L. Choshen, et al. Data contamination report from the 2024 CONDA shared task. arXiv preprint arXiv:2407.21530, 2024

    2024 arXiv

  19. [27]

    Y. Song, Y. Kim, and M. Iyyer. VERISCORE : Evaluating the factuality of verifiable claims in long-form text generation. arXiv preprint arXiv:2406.19276, 2024

    2024 arXiv

  20. [28]

    W. Su, C. Wang, Q. Ai, Y. Hu, Z. Wu, Y. Zhou, and Y. Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. arXiv preprint arXiv:2403.06448, 2024

    2024 arXiv

  21. [29]

    L. Tang, P. Laban, and G. Durrett. MiniCheck : Efficient fact-checking of LLMs on grounding documents. arXiv preprint arXiv:2404.10774, 2024

    2024 arXiv

  22. [30]

    Hallucination evaluation model (revision 7437011), 2024

    Vectara . Hallucination evaluation model (revision 7437011), 2024. URL https://huggingface.co/vectara/hallucination_evaluation_model

    2024

  23. [31]

    Wataoka, T

    K. Wataoka, T. Takahashi, and R. Ri. Self-preference bias in LLM -as-a-judge. arXiv preprint arXiv:2410.21819, 2024

    2024 arXiv

  24. [32]

    J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024 a

    2024 arXiv

  25. [33]

    J. Wei, C. Yang, X. Song, Y. Lu, N. Hu, D. Tran, D. Peng, R. Liu, D. Huang, C. Du, et al. Long-form factuality in large language models. arXiv preprint arXiv:2403.18802, 2024 b

    2024 arXiv

  26. [34]

    W. Xu, G. Zhu, X. Zhao, L. Pan, L. Li, and W. Wang. Pride and prejudice: LLM amplifies self-bias in self-refinement. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024 doi

  27. [35]

    J. Ye, Y. Wang, Y. Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y. Chen, N. V. Chawla, and X. Zhang. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024

    2024 arXiv

  28. [36]

    W. Zhao, T. Goyal, Y. Y. Chiu, L. Jiang, B. Newman, A. Ravichander, K. Chandu, R. L. Bras, C. Cardie, Y. Deng, et al. WildHallucinations : Evaluating long-form factuality in LLMs with real-world entity queries. arXiv preprint arXiv:2407.17468, 2024 a

    2024 arXiv

  29. [37]

    Y. Zhao, J. Zhang, I. Chern, S. Gao, P. Liu, J. He, et al. FELM : Benchmarking factuality evaluation of large language models. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  30. [38]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advan...

    2023

  31. [39]

    Z. Zhu, Y. Yang, and Z. Sun. HaluEval-Wild : Evaluating hallucinations of language models in the wild. arXiv preprint arXiv:2403.04307, 2024

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.