Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Evaluating the Evaluation of Diversity in Commonsense Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that content-based diversity metrics, built on sentence embeddings, match human and LLM judgments of commonsense diversity better than form-based n-gram overlap metrics, which overestimate diversity in low-quality outputs.

desk verdict A useful, systematic meta-evaluation of diversity metrics for commonsense generation, but the headline recommendation is only partly supported: the low-quality regime, where content-based metrics look best, rests on an LLM gold standard that was human-validated only on high-quality pairs. read the letter →

arxiv 2506.00514 v1 pith:3ZFI6LZM submitted 2025-05-31 cs.CL

Tianhui Zhang , Bei Peng , Danushka Bollegala This is my paper
classification cs.CL
keywords commonsensegenerationdiversityevaluationmeta-evaluationVendiScoreself-BLEUsentenceembeddingsLLM-as-judgeform-basedvscontent-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which automatic metrics best measure the diversity of commonsense-generated sentence sets. It builds a large dataset of pairwise diversity preferences annotated by an LLM, validates a sample against five human annotators with 80.6 percent agreement, and compares twelve diversity metrics on outputs from three generator LLMs across three commonsense datasets. The paper finds that n-gram-overlap form-based metrics such as self-BLEU and distinct-n systematically overestimate diversity, especially for nonsensical or randomly shuffled sentences, while embedding-based content-based metrics such as the Vendi Score over embeddings and Chamfer distance track the reference judgments consistently. The recommendation is that future commonsense generation work should evaluate diversity with content-based metrics.

What carries the argument

The paper's central instrument is an LLM-based pairwise diversity annotation protocol: the annotator LLM scores each sentence set on a five-point scale across five runs, with few-shot human-labelled examples and prompts that require commonsense adherence. Against these ratings, each target diversity metric is treated as an annotator that orders pairs of sentence sets, and accuracy is the percentage of pairwise decisions matching the LLM. The content-based metrics share the mechanism of sentence embeddings followed by a set-level aggregation, such as average pairwise cosine similarity, Chamfer distance, or the Vendi Score computed from the eigenvalues of a similarity kernel, whereas form-based metrics operate on n-gram overlap.

What would settle it

Collect human pairwise diversity judgments on a sample of the low-quality sets and measure whether human choices agree with the annotator LLM and with each of the twelve metrics; if humans side with form-based metrics on those sets, or the LLM diverges from humans there, the paper's central ranking collapses.

Watch

Extended reading notes

Core claim

The central claim is that diversity in commonsense generation is a semantic property, not a lexical one, and evaluation metrics should reflect that. Across CommonGen, ComVE, and DimonGen, and across three generator LLMs, content-based metrics obtain higher pairwise agreement with the LLM-based diversity ratings than form-based metrics, with VS-Embed variants leading. Form-based metrics are shown to assign high diversity scores to randomly shuffled or nonsensical sentence sets, so they conflate lexical variation with meaningful diversity. The paper also constructs a controlled candidate-set generation process that separates diversity from quality, and shows the content-based advantage persists in both high- and low-quality regimes.

Load-bearing premise

The entire metric ranking is anchored to the annotator LLM's diversity ratings, and human validation of those ratings covers only 70 pairs of high-quality sentence sets, so for the low-quality sets where content metrics show their biggest edge the gold standard is unvalidated.

Editorial extensions

If this is right

  • Future commonsense generation papers should report content-based diversity metrics such as VS-Embed or Chamfer rather than relying on self-BLEU and distinct-n.
  • Rankings of generation models by diversity may change under content-based metrics, because form-based metrics can reward paraphrase-heavy outputs that humans see as redundant.
  • Diversity should not be evaluated in isolation from quality; low-quality sets inflate diversity scores for all metrics, though content-based ones remain closer to human-like judgment.
  • The content-based advantage holds across three datasets and three generator LLMs, suggesting the result generalizes within English commonsense generation.
  • The LLM-annotated dataset itself is a reusable resource for future meta-evaluations of diversity metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the annotator LLM encodes a particular stylistic or thematic notion of what counts as diverse, metrics tuned to its ratings could inherit that bias; a human study on low-quality sets would test whether the content-based advantage is genuinely human-aligned in that regime.
  • The same form-versus-content split likely applies to other text generation tasks, such as storytelling or question generation, where n-gram diversity metrics are still common; the paper's annotation protocol could be transplanted directly.
  • Because all metrics in the paper's score table assign higher diversity to low-quality sets, no single metric fully separates diversity from quality, so a combined quality-diversity metric is a natural but untested next step.
  • The recommendation is currently English-only; for languages with different morphology or word boundaries, the gap between form-based and content-based metrics may shrink or invert.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper performs a meta-evaluation of twelve diversity metrics for commonsense generation, comparing form-based metrics (self-BLEU, distinct-n, entropy, VS over n-grams) with content-based metrics (self-CosSim, Chamfer distance, VS over sentence embeddings). The authors construct a new evaluation set by asking GPT-4o to rate pairwise diversity of sentence sets generated from CommonGen, ComVE, and DimonGen, with five human annotators validating a subset of 70 high-quality CommonGen pairs. They report that content-based metrics agree with GPT-4o much more often than form-based metrics, especially on low-quality sentence sets, and recommend that future commonsense generation work adopt content-based diversity metrics.

Significance. If the central claim holds, the paper provides a practically useful recommendation for the growing literature on diverse commonsense generation: replace n-gram overlap metrics with embedding-based metrics. The paper has clear strengths: it covers three datasets and three generator LLMs, reports bootstrap confidence intervals for the main tables, makes code and data available, and systematically varies generation quality. The main limitation is that the gold standard for the low-quality regime, where the content-versus-form gap is largest, is GPT-4o alone, with human validation restricted to a small high-quality subset. The prompt also explicitly instructs the annotator that nonsensical outputs are not diverse, which may encode the paper's conclusion. These issues make the contribution valuable but conditional on additional validation.

major comments (4)
  1. [§4.3, §5.3, Table 3] The human validation of the GPT-4o annotator is restricted to 70 pairs of high-quality sentence sets (§4.3). The central evidence for the paper's recommendation is Table 3, where the largest content-versus-form gap occurs in the low-quality columns (e.g., self-BLEU-3 at 27.6% vs. VS-Embed-0.5 at 81.1% for GPT-4-turbo), but those columns are validated only against GPT-4o ratings. The reported 80.6% human-LLM accuracy therefore does not cover the regime that carries the paper's conclusion; human evaluation on a sample of Nonsensical, NounShuff, and RndShuff pairs is needed before the low-quality claim can be accepted.
  2. [§4.1.1, Appendix D, Fig. 8] The annotation prompt explicitly instructs GPT-4o that nonsensical outputs should not be considered genuinely diverse and that diversity should not compromise realism and commonsense. Because content-based embeddings trained on semantic similarity will naturally assign low diversity to word-shuffled or nonsensical sentences, the higher agreement of content-based metrics with GPT-4o on low-quality sets may reflect the prompt's instructed prior rather than human diversity judgments. The authors should provide an ablation without the quality/commonsense instruction, or human labels for the low-quality pairs, to demonstrate that the advantage is not an artifact of the annotation protocol.
  3. [§5.1] The 0.5 rating-difference filter is applied without reporting how many pairs are removed in each quality condition or testing alternative thresholds. If the filter disproportionately removes easy or hard pairs in the low-quality condition, the accuracy gaps in Table 3 could be sensitive to this choice; a sensitivity analysis over thresholds (e.g., 0.25 and 0.75) and per-condition retention counts is needed.
  4. [§6, §4.3] The conclusion states that content-based metrics 'consistently align with human judgments,' but the only human-judgment comparison is on 70 high-quality CommonGen pairs. For ComVE and DimonGen (Table 2) and for all low-quality sets, the gold standard is GPT-4o alone; the wording should be limited to alignment with LLM ratings unless additional human validation is supplied.
minor comments (5)
  1. [Abstract] The abstract contains the typo 'an Large Language Model' and should read 'a Large Language Model'.
  2. [§5.3, Fig. 4] The text refers to 'self-BLUE-3' in the discussion of Figure 4a; this should be 'self-BLEU-3'.
  3. [§4.1.1] The sentence 'the LLM exhibits ordering sensitivity problem that the ordering of choices would affect the quality ranking' is grammatically unclear; consider rephrasing to 'the LLM is sensitive to the ordering of choices, which affects the quality ranking'.
  4. [Table 3] Unlike Tables 1 and 2, Table 3 does not report confidence intervals for the per-quality-level accuracies; adding bootstrap intervals or Clopper-Pearson intervals for each cell would strengthen the comparison.
  5. [§4.1.2] The sentence 'An example of an LLM-based diversity judgement byGPT-4ois shown in Figure 2' lacks spaces around 'by' and 'is'; this appears to be a formatting issue.

Circularity Check

1 steps flagged · score 5.0 of 10

Low-quality-regime conclusion is partly built into the GPT-4o prompt used as the gold standard; high-quality claim retains independent human support.

  1. self definitional [Section 4.1.1 (Prompt Engineering), Figure 8; applied in Section 5.3 (Table 3)]
    "This prompt instructs the LLM diversity annotator to adhere to commonsense constraints (i.e. nonsensical outputs should not be interpreted to be genuinely diverse). ... We define the accuracy of a target diversity metric as the percentage of pairwise decisions that agree with those of the annotator LLM."

    The meta-evaluation's target variable is the GPT-4o diversity rating, and the prompt defining that target explicitly encodes a content-based, quality-aware notion of diversity. Human validation in Section 4.3 covers only '70 pairs of high-quality sentence sets,' so the low-quality comparisons in Table 3, where content-based metrics show their largest advantage, are adjudicated solely by LLM labels that were instructed to discount nonsensical outputs. Content-based metrics use sentence embeddings that capture semantic similarity, so their agreement with these instructed labels is substantially built in; form-based n-gram metrics count token overlap and cannot access the semantic instruction.

full rationale

The paper is a meta-evaluation of diversity metrics against a GPT-4o-created gold standard, not an unmediated human judgment. The prompt-engineering step explicitly tells the annotator that nonsensical outputs are not genuinely diverse and asks for low semantic similarity without compromising commonsense, which is precisely the content-based notion of diversity. Because the human evaluation is restricted to 70 high-quality pairs, the low-quality columns of Table 3—where content-based metrics outperform form-based metrics most dramatically—are supported only by LLM labels constructed in content-based terms. This makes part of the conclusion self-fulfilling. However, the paper also provides independent evidence: 80.6% human-LLM agreement on high-quality pairs, and content-based metrics also lead on high-quality and all-pair comparisons even before the low-quality regime is considered. The self-citations to Zhang et al. (2024) are not load-bearing because the paper runs its own human evaluation. Overall, the circularity is partial rather than a forced equivalence, so a score of 5 reflects the constructed gold standard in the decisive low-quality regime without dismissing the paper's genuine high-quality evidence.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the reliability of the LLM gold standard and on the choice of SimCSE embeddings; both are domain assumptions. The temperature and filtering threshold are hand-tuned choices that influence the reported accuracy.

free parameters (2)
  • LLM annotation temperature = 1.0
    Selected as the temperature that maximizes agreement with human judgments on 70 high-quality pairs (Appendix E). The main accuracy numbers depend on this choice.
  • Pair filtering threshold = 0.5
    Pairs with LLM average rating difference less than 0.5 are removed before computing accuracy (Section 5.1). This hand-chosen threshold could bias the evaluation toward easier, more decision-relevant pairs.
assumptions (4)
  • domain assumption GPT-4o provides reliable diversity ratings for both high- and low-quality sentence sets in GCR
    Validated only on 70 high-quality pairs with 80.6% average accuracy against five human annotators (Fleiss kappa 0.45). Extrapolated to low-quality sets without direct human validation (Section 4.3).
  • domain assumption SimCSE sentence embeddings capture semantic content well enough for diversity comparisons
    Used as the embedding for all content-based metrics including VS-Embed and self-cosSim (Section 3). The conclusion that content-based metrics are superior depends on this embedding choice.
  • domain assumption The generated candidate sets achieve their intended quality levels
    High-quality (Default, Para-1/2/3) and low-quality (Nonsensical, NounShuff, RndShuff) sets are assumed to reflect the specified diversity levels, checked via quality metrics in Appendix C.1.
  • standard math Standard metric definitions (Vendi Score, Chamfer distance, self-BLEU) are correctly applied
    The metrics are taken from prior literature and implemented as described in Section 3 and appendices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating the Evaluation of Diversity in Commonsense Generation." pith.science (2026). https://pith.science/paper/3ZFI6LZM

@misc{pith2026250600514,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Evaluation of Diversity in Commonsense Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ZFI6LZM}},
  note         = {Machine review of arXiv:2506.00514}
}
read the original abstract

In commonsense generation, given a set of input concepts, a model must generate a response that is not only commonsense bearing, but also capturing multiple diverse viewpoints. Numerous evaluation metrics based on form- and content-level overlap have been proposed in prior work for evaluating the diversity of a commonsense generation model. However, it remains unclear as to which metrics are best suited for evaluating the diversity in commonsense generation. To address this gap, we conduct a systematic meta-evaluation of diversity metrics for commonsense generation. We find that form-based diversity metrics tend to consistently overestimate the diversity in sentence sets, where even randomly generated sentences are assigned overly high diversity scores. We then use an Large Language Model (LLM) to create a novel dataset annotated for the diversity of sentences generated for a commonsense generation task, and use it to conduct a meta-evaluation of the existing diversity evaluation metrics. Our experimental results show that content-based diversity evaluation metrics consistently outperform the form-based counterparts, showing high correlations with the LLM-based ratings. We recommend that future work on commonsense generation should use content-based metrics for evaluating the diversity of their outputs.

Figures

Figures reproduced from arXiv: 2506.00514 by the authors.

Figure 1
Figure 1. An example from the CommonGen (Lin et al., 2020) dataset comparing two sets of generated sen￾tences. self-BLEU-3 indicates Set-2 to be more diverse, which simply repeats near-identical paraphrases. In con￾trast, Vendi Score (VS)-embed-0.5 aligns well with the notion of meaningful textual diversity. While recent neural architectures have signifi￾cantly improved the quality of commonsense gener￾ation, reliably evaluat… view at source ↗
Figure 2
Figure 2. An example of annotating for diversity using [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Inter-annotator agreement (measured using Cohen’s Kappa) between two diversity metrics when used to [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Distribution of diversity scores for self-BLEU-3 (form-based) and Chamfer Distance (content-based) for [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The prompt used to instruct generator LLMs [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 7
Figure 7. Figure 7: The prompt used to instruct generator LLMs [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Prompt provided to GPT-4o for scoring and comparing two sentence sets. The instructions specify a five-point diversity scale ranging from highly redundant (score = 1) to explore a wide range of aspects of the theme (score = 5). We also emphasise commonsense consistency…
Figure 9
Figure 9. Figure 9: GPT-4o’s diversity annotations and accompanying explanation for two sentence sets generation for the [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: GPT-4o’s diversity annotations and accompanying explanation for two sentence sets in the ComVE [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: GPT-4o’s diversity annotations and accompanying explanation for two sentence sets in the DimonGen [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 13
Figure 13. Figure 13: Binomial confidence intervals are super [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 12
Figure 12. Figure 12: Binomial confidence intervals are super [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 28 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382--398. Springer

    2016

  3. [3]

    Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2024. Benchmarking foundation models with language-model-as-an-examiner. In Advances in Neural Information Processing Systems, volume 36

    work page 2024

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877--1901

    work page 2020

  5. [5]

    C J Clopper and E S Pearson. 1934. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404--413

    work page 1934

  6. [6]

    Samuel Rhys Cox, Yunlong Wang, Ashraf Abdul, Christian Von Der Weth, and Brian Y. Lim. 2021. Directed diversity: Leveraging language embedding distances for collective creativity in crowd ideation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1--35

    work page 2021

  7. [7]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234

    arXiv 2022

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 44 references
  1. [9]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics

    2018

  2. [10]

    Dan Friedman and Adji Bousso Dieng. 2023. The vendi score: A diversity evaluation metric for machine learning. Transactions on Machine Learning Research

    2023

  3. [11]

    Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6556--6576

    2024

  4. [12]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894--6910

    2021

  5. [13]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751

    2019 arXiv

  6. [14]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  7. [15]

    EunJeong Hwang, Veronika Thost, Vered Shwartz, and Tengfei Ma. 2023. Knowledge graph compression enhances diverse commonsense generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 558--572, Singapore. Association for Compu...

    2023

  8. [16]

    Mark W Jones, J Andreas Baerentzen, and Milos Sramek. 2006. 3d distance fields: A survey of techniques and applications. IEEE Transactions on visualization and Computer Graphics, 12(4):581--599

    2006

  9. [17]

    Masahiro Kaneko and D Bollegala. 2021. Unmasking the mask - evaluating social biases in masked language models. National Conference on Artificial Intelligence, pages 11954--11962

    2021

  10. [18]

    Masahiro Kaneko, Aizhan Imankulova, Danushka Bollegala, and Naoaki Okazaki. 2022. Gender bias in masked language models for multiple languages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language ...

    2022

  11. [19]

    Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In 24th Annual Conference of the European Association for Machine Translation, page 193

    2023

  12. [20]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

    2016

  13. [21]

    Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Generating reasonable and diversified story ending using sequence to sequence model with adversarial training. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1033--1043

    2018

  14. [22]

    Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. C ommon G en: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, p...

    2020

  15. [23]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  16. [24]

    Chenzhengyi Liu, Jie Huang, Kerui Zhu, and Kevin Chen-Chuan Chang. 2023 a . D imon G en: Diversified generative commonsense reasoning for explaining concept relationships. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 4719--4...

    2023

  17. [25]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511--2522

    2023

  18. [26]

    Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. 2020. C row S -pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953--196...

    2020

  19. [27]

    Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent advances in neural question generation. arXiv preprint arXiv:1905.08949

    2019 arXiv

  20. [28]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  21. [29]

    Amey P Pasarkar and Adji Bousso Dieng. 2024. Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning. In International Conference on Artificial Intelligence and Statistics, pages 3808--3816. PMLR

    2024

  22. [30]

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is temperature the creativity parameter of large language models? arXiv preprint arXiv:2405.00492

    2024 arXiv

  23. [31]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006--2017

    2024

  24. [32]

    Murray Rosenblatt. 1956. Remarks on some nonparametric estimates of a density function. Ann. Math. Stat., 27(3):832--837

    1956

  25. [33]

    Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR), 55(2):1--39

    2022

  26. [34]

    Tianxiao Shen, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In International conference on Machine Learning, pages 5719--5728. PMLR

    2019

  27. [35]

    Guy Tevet and Jonathan Berant. 2021. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326--346

    2021

  28. [36]

    Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang. 2020. Semeval-2020 task 4: Commonsense validation and explanation. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 307--321

    2020

  29. [37]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. In Proceedings of EMNLP Workshop, page 1

    2023

  30. [38]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 9440--9450

    2024

  31. [39]

    Wenhao Yu, Chenguang Zhu, Lianhui Qin, Zhihan Zhang, Tong Zhao, and Meng Jiang. 2022. Diversifying content generation for commonsense reasoning with mixture of knowledge graph experts. In Proceedings of the 2nd Workshop on Deep Learning on Graphs for Natural Language Processin...

    2022

  32. [40]

    Tianhui Zhang, Bei Peng, and Danushka Bollegala. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.540 Improving diversity of commonsense generation by large language models via in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2024,...

    2024 doi

  33. [41]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  34. [42]

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1097--1100

    2018

  35. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.