Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Long-form answers should be judged on whether their verified facts cover all expected aspects, and the paper introduces ICAT, a retrieval-based framework that measures exactly that by decomposing outputs into atomic claims, grounding each…

desk verdict A well-engineered coverage metric whose best variant correlates moderately with human judgments on a shared TREC checklist; the framework is solid and reusable, but the broader 'diverse factual coverage' claim rests on that checklist's completeness. read the letter →

arxiv 2501.03545 v4 pith:CVQOO53G submitted 2025-01-07 cs.CL

Chris Samarinas , Alexander Krubner , Alireza Salemi , Youngwoo Kim , Hamed Zamani This is my paper · ORCID
classification cs.CL
keywords long-formtextgenerationfactualcoverageevaluationmetricsatomicclaimsaspectLLMretrieval-baseddiversification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that evaluating long-form text generation requires more than checking factual accuracy: a good answer must also cover the diverse aspects a user or topic implies. It introduces ICAT, which splits an LLM output into atomic claims, verifies each claim by retrieving supporting evidence from a knowledge source, and then scores how many of the expected aspects are covered by the verified claims. Across three implementations that differ in whether aspects and claim-aspect alignment come from humans or LLMs, the variant with human-provided aspects and LLM-based alignment (ICAT-S) correlates with human coverage judgments at Pearson's rho = 0.422 and Spearman's rho = 0.446 on ClueWeb/TREC queries. Traditional lexical metrics such as BLEU and ROUGE show negative or near-zero correlation with the same human judgments, which the paper reads as evidence that coverage of diverse information is a distinct quality that needs its own metric.

What carries the argument

The load-bearing object is the ICAT score itself, a pair of ratios combined by a beta-weighted harmonic mean. Atomic claim generation turns each long output into decontextualized standalone facts; claim grounding uses dense retrieval followed by a natural language inference model to keep only claims supported by the knowledge source; aspect coverage maps those grounded claims to a query aspect list, either through aspect-annotated documents (ICAT-M), an LLM aligner (ICAT-S), or an LLM-generated aspect set (ICAT-A). The machinery's power is that it decomposes completeness into a checkable claim-level pipeline, making both factuality and coverage inspectable claim by claim.

What would settle it

Re-run the human coverage study on the same queries but with an independently constructed aspect list, such as aspects free-listed by annotators or generated without the TREC subtopic set, and compare ICAT-S against human scores under both aspect sets. If the correlation with human judgments drops substantially when the aspect set changes, the reported alignment depends on the particular TREC subtopics rather than on coverage of diverse information itself.

Watch

Extended reading notes

Core claim

The central claim is that coverage of diverse factual information can be measured automatically by decomposing a response into atomic claims, grounding each claim through retrieval and natural language inference, and aligning the grounded claims with a list of expected aspects. The paper defines a factuality score $S_{\mathrm{fact}} = |C_T|/|C|$ and a coverage score $S_{\mathrm{coverage}} = |\{T_O(c,K): c \in C_T\} \cap T_Q(x)| / |T_Q(x)|$, combined as a weighted harmonic mean $ICAT_\beta$. Its experiments show ICAT-S, using ground-truth TREC subtopics and Llama-3.1-70B for claim-aspect alignment, correlates strongly with human annotators (Pearson's rho = 0.422, p < 0.01; Spearman's rho = 0.446, p < 0.01). By contrast, BLEU, ROUGE, METEOR, and most BERTScore variants fail to track human coverage judgments. The paper also reports that web-based retrieval raises factuality scores across all tested LLMs but that coverage improvements are modest, and that the $\beta$ parameter trades factuality against coverage in the combined score.

Load-bearing premise

The claim rests on the assumption that the TREC Web Track subtopics, originally designed for web search result diversification, are a complete and valid set of aspects for a long-form answer; if the true space of aspects differs, both the human gold standard and the ICAT scores are anchored to an incomplete target.

Editorial extensions

If this is right

  • Because ICAT-S tracks human coverage judgments at moderate-to-strong correlation, it can replace expensive human annotation when comparing LLMs on long-form answers.
  • BLEU, ROUGE, and METEOR are unsuitable for coverage evaluation: their correlations with human coverage judgments are negative or near zero, so they should not be used as proxies for completeness.
  • The beta parameter lets an evaluator weight factuality against coverage; for beta less than 1, GPT-4 and Mixtral lead, while for beta greater than 1, GPT-4 and Llama lead on coverage.
  • Switching from corpus-based to web-based retrieval increases factuality scores for every tested LLM, while coverage gains are smaller, indicating that coverage depends more on the generator than on the retrieval source.
  • The modular design supports adaptation to domains where aspect lists or evidence corpora exist, and ICAT's scores could serve as a reward signal for training models to produce more complete outputs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own observation that ICAT-A's automatically generated aspects look more comprehensive than the TREC subtopics suggests the reported correlations may underestimate how well the framework could perform with a better aspect target; testing ICAT-A against human judgment on richer aspect sets would settle this.
  • Because ICAT-A relies on the same or similar LLMs for aspect generation and response evaluation, responses sharing that model's biases could be scored as more complete; the paper flags this circularity but does not measure its size.
  • A natural extension is to apply the same claim-level coverage pipeline to domain-specific long-form tasks, such as clinical or policy writing, where the aspect list is defined by guidelines rather than search subtopics and the evidence corpus is a trusted authority.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ICAT, a reference-free evaluation framework for long-form text generation. ICAT decomposes a generated response into atomic claims, verifies each claim by retrieving supporting evidence from a corpus (ClueWeb09) or the web and applying an NLI model, and then computes a coverage score as the fraction of query aspects supported by factually grounded claims. Three variants are instantiated: ICAT-M (manual aspects and manual document-aspect alignment), ICAT-S (manual aspects with LLM-based claim-aspect alignment), and ICAT-A (LLM-generated aspects with LLM-based alignment). The framework is evaluated on 50 TREC Web Track queries, comparing ICAT coverage scores against AMT annotators' judgments on the TREC subtopic lists, and on four LLM generators (GPT-4, Llama-3-70B, Mixtral-8x22B, Openchat 3.5). The best reported correlation is ICAT-S with Llama-3.1-70B (Pearson 0.422, Spearman 0.446 in the corpus setting); BLEU, ROUGE, and METEOR show near-zero correlation, while BERTScore-recall reaches 0.291. The paper also presents a beta-weighted harmonic combination of factuality and coverage and component-level analyses of claim generation, topic generation, and coverage alignment.

Significance. If the central validation were independent of the fixed TREC subtopic checklist, ICAT would be a practically useful, modular reference-free evaluator for long-form generation. The paper's strengths include a clean decomposition into atomic claims with retrieval-based grounding, three clearly specified variants that separate the effect of manual versus automatic aspects and alignment (Table 1), a released open-source implementation, component-level human evaluations (Tables 6-8), and a fair comparison showing that lexical overlap metrics carry near-zero signal for coverage. The reported coverage-prediction precision and recall against human judgments for ICAT-S (Table 8: 0.903 and 0.798) are encouraging. However, the contribution's central claim hinges on a validation whose human gold standard shares the same fixed aspect list as the metric, and the fully automatic variant's correlations are weak; these issues need to be addressed before the claim that ICAT measures coverage of diverse factual information can be accepted.

major comments (4)
  1. [Section 5 and Section 6, Table 2] The validation protocol computes both the human coverage judgments and the ICAT coverage scores against the same fixed TREC Web Track subtopic list. The AMT task (Section 6, Figure 4) asked annotators to mark whether each listed aspect is present and to highlight evidence; it did not ask them to judge overall completeness or propose missing aspects. ICAT-S aligns claims to exactly the same list. The reported Pearson rho = 0.422 and Spearman rho = 0.446 can therefore be interpreted as human-LLM agreement on a shared checklist rather than as evidence that ICAT recovers the aspects a good long-form answer should cover. The TREC subtopics were designed for web-search result diversification, not for long-form generation, and the paper's own note under Table 2 states that auto-generated topics were 'more comprehensive and higher quality' than the TREC subtopics. Because both sides of the correlation use the same incomplete denominator, the result is not informative about coverage of unlisted aspects.
  2. [Table 2] The fully automatic variant ICAT-A, which is the only one that does not require manually supplied aspects, reaches only Pearson rho = 0.246 in the corpus setting and rho = 0.055 in the web setting with Llama-3.1-70B as the coverage model; the web correlation is effectively zero and no significance test is reported for it. The Section 7 statement that automatic evaluation approaches 'maintain comparable performance' is therefore supported only by ICAT-S, which assumes ground-truth aspects are available. Since the abstract and introduction motivate ICAT as an automated evaluator of coverage, the authors should either qualify the claim or show that ICAT-A's low correlation is an artifact of the incomplete TREC aspect set rather than a property of the metric.
  3. [Section 6] All correlations are computed on 50 queries, yet no confidence intervals are reported. With n = 50, the best Pearson rho = 0.422 has a wide confidence interval, and the difference between ICAT-S (0.422) and ICAT-A (0.246) may not be statistically significant. Report bootstrap confidence intervals for every row of Table 2 and a significance test for the difference between ICAT-S and the best traditional metric (BERTScore-recall, 0.291).
  4. [Section 4.2 and Limitations] The aspect-generation and aspect-claim alignment components use Llama-3.1 models, while the set of evaluated generators includes Llama-3-70B; the paper itself acknowledges that using the same or similar LLM for aspect generation and response evaluation can inflate performance. The near-zero web correlation of ICAT-A (0.055) is consistent with this bias rather than with robust coverage measurement. The authors should test at least one configuration where the aspect and alignment model is from a different model family than the evaluated generator, or where the aspects are curated independently of both the generator and the TREC list.
minor comments (4)
  1. [Introduction] The phrase 'corelation' in the third paragraph should be corrected to 'correlation'.
  2. [Section 6] The phrase 'rank-bsaed' in the first paragraph should be corrected to 'rank-based'.
  3. [Table 3] The header notation 'ICAT-M1', 'ICAT-S1', and 'ICAT-A1' is not defined in the caption or the text; clarify that the subscript denotes the default beta = 1 value.
  4. [Section 3.3] The symbols T_O and T_Q are introduced with subscript-like notation but are rendered ambiguously in the coverage-score equation; use a consistent mathematical notation for the claim-to-aspect and query-to-aspect functions.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: ICAT is validated against external human judgments, and the shared TREC checklist is a validity limitation, not a circular derivation.

full rationale

The central claim is that ICAT-S with Llama-3.1-70B alignment correlates with human coverage judgments (Pearson rho=0.422, Spearman rho=0.446 in Table 2). The human judgments were collected independently via AMT and were not used to fit or tune the ICAT components: the coverage model is a prompted LLM, the NLI model is a standard DeBERTa model, and the claim-generation model is fine-tuned on synthetic data from Llama-3.1-405B, not on the human coverage labels. The correlation is therefore an empirical external validation rather than a quantity forced by construction. Both human scores and ICAT-S use the TREC subtopics as the aspect denominator, so the validation measures agreement on a fixed checklist rather than the completeness of the checklist; however, this is a construct-validity limitation, not circularity, because ICAT-S explicitly assumes a ground-truth aspect set and the paper does not claim to discover that set. The paper even observes that auto-generated aspects were 'more comprehensive and higher quality' than TREC subtopics, which means the fixed checklist is if anything an incomplete target, not an input that guarantees the reported correlation. The Limitations section acknowledges a potential 'circular dependency' when the same LLM family generates aspects and responses; this applies only to ICAT-A, which shows weaker correlation (0.246 corpus-based, 0.055 web-based), so the acknowledged bias is not load-bearing for the headline result. The self-citations (Samarinas et al., 2022, 2024) are ordinary references, and the 2022 claim about LLM subtopic generation is independently supported by the paper's own human evaluation (subtopic precision 0.966 and 0.958, Table 7). No step in the derivation reduces to its own inputs, and no fitted parameter is renamed as a prediction. Overall, the paper is self-contained against an external human benchmark, and the appropriate circularity score is near zero.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The framework rests on four domain assumptions: lossless atomic claim decomposition by an LLM, reliable NLI-based grounding, TREC subtopics as a valid aspect gold standard, and AMT majority-vote as ground truth for coverage. The only hand-chosen parameter is beta, which is not used in the headline correlation experiments. No new physical or conceptual entities are introduced.

free parameters (1)
  • beta (β) = 1 (default)
    Weight in the ICAT_beta harmonic mean that trades off factuality vs coverage. It is hand-chosen, not fitted to the human data, and the Table 2 correlations use coverage score alone, so beta does not directly drive the headline result.
assumptions (4)
  • domain assumption An LLM can decompose a long response into a complete set of atomic, self-contained factual claims without losing or adding information.
    Section 3.1 defines AC(y) via an LLM prompt with a specific decomposition prompt. If the model misses claims, S_coverage is underestimated; the paper evaluates claim precision but does not measure how decomposition errors affect coverage scores.
  • domain assumption The NLI model's entailment decision reliably indicates whether a claim is supported by a retrieved document.
    Sections 3.2 and 4.3 ground claims using a DeBERTa-based NLI model with ANLI accuracy 0.579. False accepts or false rejects propagate into both the factuality score and the coverage score.
  • ad hoc to paper TREC Web Track subtopics are a valid and complete set of aspects for evaluating long-form text generation.
    Section 5 adopts TREC subtopics as ground-truth aspects. These subtopics were created for web search diversification, not for long-form answers; an incomplete or mismatched aspect set biases both human and ICAT coverage scores.
  • domain assumption Human annotations from three AMT workers, combined by majority voting, provide a reliable gold standard for aspect coverage.
    Section 6 uses 50 queries with three annotators each and reports Fleiss's kappa 0.829. The small sample and annotator recruitment criteria may limit the reliability and generality of the gold standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation." pith.science (2026). https://pith.science/paper/CVQOO53G

@misc{pith2026250103545,
  author       = {Pith},
  title        = {Pith review of: Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CVQOO53G}},
  note         = {Machine review of arXiv:2501.03545}
}
read the original abstract

This paper presents ICAT, an evaluation framework for measuring coverage of diverse factual information in long-form text generation. ICAT breaks down a long output text into a list of atomic claims and not only verifies each claim through retrieval from a (reliable) knowledge source, but also computes the alignment between the atomic factual claims and various aspects expected to be presented in the output. We study three implementations of the ICAT framework, each with a different assumption on the availability of aspects and alignment method. By adopting data from the diversification task in the TREC Web Track and the ClueWeb corpus, we evaluate the ICAT framework. We demonstrate strong correlation with human judgments and provide comprehensive evaluation across multiple state-of-the-art LLMs. Our framework further offers interpretable and fine-grained analysis of diversity and coverage. Its modular design allows for easy adaptation to different domains and datasets, making it a valuable tool for evaluating the qualitative aspects of long-form responses produced by LLMs.

Figures

Figures reproduced from arXiv: 2501.03545 by the authors.

Figure 1
Figure 1. Retrieval-based evaluation of LLM responses with ICAT. Topic generation and coverage models are [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of atomic claim generation illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. ICAT-Aβ for various LLMs using Llama-3.1- 70B as coverage model and the corpus as knowledge source. For low β values where factuality has higher weight, GPT-4 and Mixtral have better performance while for higher values of β, GPT-4 and Llama have higher scores due to higher Scoverage. ness of aspect coverage is more dependent on the model’s capabilities than the retrieval source. Controlling the trade-off between fac… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Human annotation interface showing the supporting evidence highlights from 3 annotators for the given [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Subtopic coverage prompt used to identify which subtopics are covered in a given text. The prompt [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Claim generation prompt designed to extract atomic factual statements from input text. The prompt [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Topic generation prompt used to automatically generate diverse subtopics for a given query. The prompt [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation

    cs.CL 2025-01 conditional novelty 6.0 of 10

    ExPerT is a reference-based LLM evaluation metric that extracts and matches atomic aspects, scores content and style, and reports 0.74 human alignment on LongLaMP, a 7.2% relative gain over GEMBA and G-Eval.

  2. CLAImate: AI-Enabled Climate Change Communication through Personalized and Localized Narrative Visualizations

    cs.HC 2025-07 conditional novelty 5.0 of 10

    A personalized, localized AI conversation system for climate communication shows modest factual accuracy and positive early feedback from 10 UK users.

Reference graph

Works this paper leans on

38 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sweta Agrawal, George Foster, Markus Freitag, and Colin Cherry. 2021. https://doi.org/10.18653/v1/2021.naacl-main.91 Assessing reference-free peer evaluation for machine translation . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1158--1171, Online. ...

    work page doi:10.18653/v1/2021.naacl-main.91 2021

  4. [4]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65--72

    work page 2005

  5. [5]

    Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009. https://api.semanticscholar.org/CorpusID:13369675 Overview of the trec 2009 web track . In Text Retrieval Conference

    work page 2009

  6. [6]

    Charles L. A. Clarke, Nick Craswell, Ian Soboroff, and Gordon V. Cormack. 2010. https://api.semanticscholar.org/CorpusID:16213318 Overview of the trec 2010 web track . In Text Retrieval Conference

    work page 2010

  7. [7]

    Charles L. A. Clarke, Nick Craswell, Ian Soboroff, and Ellen M. Voorhees. 2011. https://api.semanticscholar.org/CorpusID:30284167 Overview of the trec 2011 web track . In Text Retrieval Conference

    work page 2011

  8. [8]

    Charles L. A. Clarke, Nick Craswell, and Ellen M. Voorhees. 2012. https://api.semanticscholar.org/CorpusID:11517775 Overview of the trec 2012 web track . In Text Retrieval Conference

    work page 2012

Show all 38 references
  1. [9]

    Charles L. A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan B \"u ttcher, and Ian MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st Annual International ACM SIGIR Conference, pages 659--666

    2008

  2. [10]

    Cormack, Mark D

    Gordon V. Cormack, Mark D. Smucker, and Charles L. A. Clarke. 2011. https://doi.org/10.1007/s10791-011-9162-z Efficient and effective spam filtering and re-ranking for large web datasets . Inf. Retr., 14(5):441–465

    2011 doi

  3. [11]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314

    2023 arXiv

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and Akhil Mathur et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  6. [14]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . Preprint, arXiv:2111.09543

    2021 arXiv

  7. [15]

    Jeff Johnson, Matthijs Douze, and Herv \'e J \'e gou. 2019. Billion-scale similarity search with GPUs . IEEE Transactions on Big Data, 7(3):535--547

    2019

  8. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  9. [17]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. https://doi.org/10.18653/v1/N16-1014 A diversity-promoting objective function for neural conversation models . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association f...

    2016 doi

  10. [18]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81

    2004

  11. [19]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  12. [20]

    Shikib Mehri and Maxine Eskenazi. 2020. https://doi.org/10.18653/v1/2020.acl-main.64 USR : An unsupervised and reference free evaluation metric for dialog generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681--707, O...

    2020 doi

  13. [21]

    Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos. 2024. https://api.semanticscholar.org/CorpusID:269635319 Arctic-embed: Scalable, efficient, and accurate text embedding models . ArXiv, abs/2405.05374

    2024 arXiv

  14. [22]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  15. [23]

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. https://doi.org/10.18653/v1/2020.acl-main.441 Adversarial NLI : A new benchmark for natural language understanding . In Proceedings of the 58th Annual Meeting of the Association for Comp...

    2020 doi

  16. [24]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--318

    2002

  17. [25]

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. https://arxiv.org/abs/2411.09607 Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework . Preprint, arXiv:2411.09607

    2024 arXiv

  18. [26]

    Chris Samarinas, Arkin Dharawat, and Hamed Zamani. 2022. https://doi.org/10.1145/3539813.3545138 Revisiting open domain query facet extraction and generation . In Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '22, page 43–...

    2022

  19. [27]

    Chris Samarinas, Pracha Promthaw, Atharva Nijasure, Hansi Zeng, Julian Killingback, and Hamed Zamani. 2024. https://arxiv.org/abs/2404.14772 Simulating task-oriented dialogues with state transition graphs and large language models . Preprint, arXiv:2404.14772

    2024 arXiv

  20. [28]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892

    2020

  21. [29]

    Yixiao Song, Yekyung Kim, and Mohit Iyyer. 2024. https://aclanthology.org/2024.findings-emnlp.552 V eri S core: Evaluating the factuality of verifiable claims in long-form text generation . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9447--9...

    2024

  22. [30]

    The Lemur Project . 2009. https://lemurproject.org/clueweb09/ The ClueWeb09 dataset . Accessed: 2024-06-24

    2009

  23. [31]

    Brian Thompson and Matt Post. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.8 Automatic machine translation evaluation in many languages via zero-shot paraphrasing . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90...

    2020 doi

  24. [32]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/v1/N18-1074 FEVER : a large-scale dataset for fact extraction and VER ification . In Proceedings of the 2018 Conference of the North A merican Chapter of the Associatio...

    2018 doi

  25. [33]

    Cornelius Joost Van Rijsbergen. 1979. Information retrieval. 2nd. newton, ma

    1979

  26. [34]

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235

    2023 arXiv

  27. [35]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  28. [36]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations

    2019

  29. [37]

    Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.131 Towards a unified multi-dimensional evaluator for text generation . In Proceedings of the 2022 Conference on Empiric...

    2022 doi

  30. [38]

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. https://doi.org/10.1145/3209978.3210080 Texygen: A benchmarking platform for text generation models . In The 41st International ACM SIGIR Conference on Research & Development in Informatio...

    2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.