Pith. sign in

REVIEW 4 major objections 4 minor 47 references

The paper claims the NLG field's rapid shift to LLM-as-a-judge rests on unvalidated instruments: LaaJ and human evaluation are nearly uncorrelated in how they associate with metrics, and fewer than 8% of papers compare the two.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:02 UTC pith:VTDOT6BY

load-bearing objection The largest quantitative map of NLG evaluation practice to date, with qualitative conclusions that are probably right — but the headline percentages rest on an unvalidated extraction stage and arithmetic errors that need fixing before they are citable. the 4 major comments →

arxiv 2601.07648 v2 pith:VTDOT6BY submitted 2026-01-12 cs.CL

What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025

Jing Yang , Nils Feldhus , Salar Mohtaj , Leonhard Hennig , Qianli Wang , Eleni Metheniti , Sherzod Hakimov , Charlott Jakob
show 5 more authors
Veronika Solopova Konrad Rieck David Schlangen Sebastian M\"oller Vera Schmitt
This is my paper
classification cs.CL
keywords NLG evaluationLLM-as-a-judgehuman evaluationmeta-analysismetric inertiaevaluation validationquality dimensionstext generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that the field's rapid, post-2023 adoption of LLM-as-a-judge (LaaJ) is not supported by evidence of agreement with human judgment. Mining 14,171 papers from four major NLP conferences, the authors extract which evaluation methods each paper used and how metrics associate with LaaJ versus human evaluation. They find the two are essentially uncorrelated (Spearman ρ = 0.007) at the metric-association level, that fewer than 8% of NLG papers explicitly compare LaaJ to humans, and that the comparisons that do exist show moderate correlations on overall quality but weak ones on fluency. A sympathetic reader would care because this challenges the default assumption that LLMs are cheap proxies for human judgment in text evaluation.

Core claim

The central claim is that LaaJ and human evaluation prioritize different quality signals and do not agree closely. The evidence: across metrics used in more than ten papers, the likelihood of a metric appearing in LaaJ papers versus human-evaluation papers is essentially uncorrelated (Spearman ρ = 0.007); of the 3,334 NLG papers analyzed, only 433 used both LaaJ and human evaluation, and only 254 of those (less than 8% of the corpus) explicitly compared the two; among those comparisons, the mean correlation with human judgment is moderate for overall quality (μ = 0.47–0.71 depending on statistic) but weak for fluency (μ = 0.35). The authors conclude that the field's shift toward LaaJ rests o

What carries the argument

The carrying mechanism is a large-scale extraction and association-analysis pipeline. Three open-source LLMs read each paper and answered four binary questions (Is it NLG? Automatic metrics? LaaJ? Human eval?), with a fourth LLM verifying and normalizing the resulting metadata. On a human-validated subset (110 papers), the harmonized annotations agreed with humans on roughly 74–95% of the binary answers. The analysis then uses likelihood ratios — how much more likely a metric appears in LaaJ papers than in non-LaaJ papers — and the correlation between those ratios to show that metric–method associations for LaaJ and human evaluation are orthogonal. A third extraction stage classifies whether

Load-bearing premise

The claim rests on the accuracy of the automated extraction step that decides whether a paper 'explicitly compares' LaaJ to human evaluation — that step was never validated against human annotation, and even the best-validated binary questions (task, metrics) are wrong on roughly one in four papers; if that classification is off at a similar rate, the headline numbers (<8% validation, the correlation means) could shift materially.

What would settle it

Take a random sample of, say, 200 of the 433 papers the pipeline says used both LaaJ and human evaluation, have human readers independently judge whether the paper explicitly compares the two (same criterion as the prompt), and compute the error rate of the third extraction stage. If the disagreement matches the ~25% rate seen on the binary questions, the 'fewer than 8%' validation figure and the mean correlation estimates for fluency and overall quality would need revision.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, the default practice of treating LaaJ as a cheaper substitute for human evaluation is unjustified for criteria like fluency, where agreement is weak.
  • Leaderboards and evaluation suites that rely on LaaJ without human validation are measuring something other than human-perceived quality; the two signals diverge by task and metric.
  • The persistence of BLEU/ROUGE in machine translation and exact match in QA means many published results may not capture the semantic dimensions these tasks require; authors should state which criterion each metric targets.
  • The near-zero correlation between metric–LaaJ and metric–human associations suggests that papers using LaaJ tend to pair it with a different set of metrics than papers using human evaluation, so replicating results across evaluation paradigms is not straightforward.
  • Adopting the paper's Evaluation Checklist (state target criterion per metric, validate LaaJ against humans, use human evaluation for high-level dimensions) would concretely change how new NLG systems are reported.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the authors do not run: apply the same extraction and association analysis to papers from 2026 onward to see whether the validation gap closes as community pressure increases; if LaaJ validation rates stay below 10% while usage keeps growing, the gap is systemic rather than a lag.
  • A plausible consequence of the fluency finding: LLM judges may be acceptable for coarse overall-quality ratings but systematically unreliable for fine-grained perceptual criteria; benchmark designers could quantify a per-criterion 'validity budget' and require a minimum correlation before accepting a judge.
  • The pipeline's binary-question error rate (~25% on task and metric questions) implies the reported task-level and metric-level percentages contain noise; re-annotating a stratified sample of papers and applying measurement-error correction would bound the headline percentages.
  • The orthogonal clusters in the metric-association figure suggest a way to build better judges: use human evaluation data to train metric-specific calibration for LaaJ, or explicitly report both sets of scores, rather than treating one as a drop-in for the other.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a large-scale meta-analysis of NLG evaluation practices, using a multi-LLM information extraction pipeline over 14,171 papers from ACL, EMNLP, NAACL, and INLG (2020–2025). After filtering to 3,334 NLG papers, it reports three main findings: metric inertia (BLEU/ROUGE persist despite shifts to open-ended generation), a metric–criteria mapping problem, and a validation gap for LLM-as-a-judge (LaaJ), with fewer than 8% of papers explicitly validating LaaJ against human judgments and only moderate-to-low aggregate correlations. The paper also derives practical recommendations and an advertised Evaluation Checklist. The central claim is that LaaJ and human evaluation are not interchangeable proxies, as they associate with different metrics and show weak direct correlations on fine-grained criteria.

Significance. If the quantitative claims are reliable, this would be an important, field-level evidence base for ongoing debates about NLG evaluation. The authors are to be credited for publishing the full prompts, the human-annotation guideline, normalization rules, and per-conference statistics, which makes the pipeline inspectable and partially reproducible. The use of three open-weight LLMs with majority voting and a 110-paper human validation of the four binary questions is a reasonable design. However, the paper's headline numbers — especially the <8% validation-gap and the Figure 7 correlation aggregates — rest on a third extraction stage that is not included in the human validation, and the reported arithmetic inconsistencies further undermine confidence in the precision of the quantitative claims. The qualitative direction of the argument is plausible and consistent with prior work, but the quantitative anchors need additional validation before they can be cited as definitive.

major comments (4)
  1. [§4.1 / Table 12 / Figure 8] The headline percentages in §4.1 are internally inconsistent. The four top tasks are reported as 78.1% of the 3,334 filtered papers, but the stated counts (871+847+734+462 = 2,914) give 2,914/3,334 = 87.4%. Similarly, '182 papers (26%)' for MT human evaluation is 182/847 = 21.5%, not 26%. In addition, §3.2 reports 8,665 initial NLG papers after filtering on A1, while Table 12 sums to 8,019 NLG papers. These discrepancies affect multiple percentages that are used as evidence, and they must be reconciled — either the counts, the denominators, or the percentages are wrong.
  2. [§4.3 / Appendix D.3 / Limitations] The paper's central quantitative claim — that fewer than 8% of NLG papers explicitly validate LaaJ against humans, and that aggregate validation is moderate (Overall μ=0.47–0.71) but weak for Fluency (μ=0.35) — is produced entirely by the third extraction stage described in §4.3 and Appendix D.3. This stage uses a single LLM (DeepSeek V3.1 Terminus) and introduces a new binary decision ('explicit comparison') plus extraction of correlation metrics, criteria, and sample sizes. The Limitations section explicitly states that agreements 'were only computed for the four binary questions,' so D.3 is not validated. Given that the validated binary questions A1/A2 have human agreement of only 74.6–77.3% (Table 1), a comparable error rate in D.3 would materially change the 254/433 count and the Figure 7 distributions. A human audit of D.3 on a representative sample, with agreement and error rates
  3. [Abstract / §5 / Appendix] The abstract and the contribution list promise a 'minimal Evaluation Checklist' to guide metric selection, construct validity, and LaaJ deployment. However, no such checklist appears anywhere in the manuscript — not in §5, not as a table, and not in the appendix. This is a missing deliverable. The authors should either include the checklist explicitly (e.g., as a table) or revise the contribution claims to match what is actually presented.
  4. [§4.2 / §6 / Figure 1] The Spearman ρ = 0.007 reported in §4.2 is the correlation between two sets of likelihood ratios (metric association with LaaJ vs. metric association with human evaluation). It is not a direct measure of agreement between LaaJ scores and human scores. The wording in §6 — 'lack strong correlation with human evaluators' — risks conflating this co-occurrence-level association with direct validity evidence. The direct evidence is in §4.3/Figure 7, which currently suffers from the D.3 validation gap noted above. Please clarify what ρ = 0.007 does and does not measure, and adjust the conclusion accordingly.
minor comments (4)
  1. [§3.2] Typo/garbled phrase: 'We filter out papers that havenoto Answer 1 (A1).' Should likely read 'that have no to Answer 1' or 'that answer No to A1.'
  2. [§4.2] Typo: 'Notebly' should be 'Notably.'
  3. [§3.1 / B.1] Temperature is set to 1 for all LLM extractions, but the manuscript does not state whether each model was run once or multiple times. With temperature > 0, repeated runs can yield different extractions. Please specify the number of runs per paper and whether any stability checks were performed.
  4. [Appendix B.4] The excluded task list includes 'Automatic Speech Recognition' and 'Text-To-Speech Generation.' These are not always considered NLG, but the exclusion criteria could be stated more explicitly, especially since the top-30 retention threshold is a free parameter that affects all subsequent analyses.

Circularity Check

0 steps flagged

No formal circularity; the unvalidated D.3 extraction is a validity limitation, not a definitional reduction.

full rationale

The paper's central claims are empirical corpus statistics, not derivations from its own definitions or from fitted parameters. The 'validation gap' (<8%) and Figure 7 mu-values are produced by the third-stage extraction prompt (Appendix D.3), and the Limitations explicitly concede: 'Our agreements of LLM annotations were only computed for the four binary questions.' This creates a genuine self-referential measurement concern—an unvalidated LLM judge is used to count how rarely researchers validate LLM judges—and a real external-validity risk for the quantitative anchors. But under the circularity rubric this is not a circular derivation: no equation is solved from its own output, no fitted parameter is renamed as a prediction, and no load-bearing claim reduces to a self-authored citation. The only author-overlapping citation (Bavaresco et al. 2025, co-authored by D. Schlangen) is background support, not the basis for the paper's quantitative findings. The likelihood-ratio and Spearman analyses (Eqs. 1-5, Sec. 4.2) are straightforward co-occurrence statistics over extracted metadata; the conclusion that LaaJ and human evaluation are 'orthogonal' is a reading of those statistics, not a tautology. The D.3 issue should be weighed as a methodological correctness risk requiring a human audit, not as circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The paper contributes an empirical measurement; its 'parameters' are hand-set thresholds that shape every downstream number (task exclusions, top-30 cutoff, >10-paper metric filter, 0.9 fuzzy-match threshold, temperature=1). The load-bearing domain assumptions are: (i) LLM extraction is accurate enough for trend statistics — only partially tested (n=110; 74-95% agreement); (ii) paper-level metric-criterion co-occurrence licenses the 'metric applied without specified criterion' finding — downgraded by the authors themselves in footnote 4; (iii) second-hand correlation values from 254 optionally-validating papers are unbiased — they are subject to publication and protocol-selection bias; (iv) four main conferences represent 'the field.' No invented entity with independent evidence is introduced; the Evaluation Checklist is a recommendation without a falsifiable handle.

free parameters (5)
  • NLG task exclusion list = 15 tasks (Code Gen, Math Reasoning, Semantic Parsing, Classification, Text-To-SQL, ASR, NER, Sentiment, TTS, etc.)
    Hand-chosen boundary between NLG and non-NLG, motivated by observed human-annotation disagreements; determines the 3,334-paper denominator and every downstream percentage.
  • Top-30 task retention cutoff = 30 tasks
    Post-hoc retention of high-frequency tasks only; low-frequency NLG tasks are dropped from analysis.
  • Metric frequency threshold = >10 papers
    Restricts the §4.2 metric-method association analysis and changes the metric set shown in Figure 1.
  • Fuzzy matching threshold = SequenceMatcher ratio 0.9
    Term-normalization threshold; affects all normalized counts (BLUE variants, 'Overall' criteria variants) and hence the reported frequencies.
  • Extraction sampling temperature = 1.0
    Non-deterministic LLM sampling (Appendix B.1); reruns will differ and no seed is reported.
axioms (5)
  • domain assumption LLM extraction is accurate enough for aggregate trend statistics
    §3.1-3.4: validated on only 110 papers; A1/A2 human agreement of 74.6-77.3% and Krippendorff's α 0.69-0.81 mean roughly one-in-four label noise on the questions feeding task and metric counts.
  • domain assumption Paper-level co-occurrence of a metric and a criterion implies the metric was used for that criterion
    Load-bearing for the metric-criteria mapping finding (§4.2, Figure 6); footnote 4 concedes this is 'less accurate than task-specific LR' because there is no exact metric-to-criterion map.
  • domain assumption Self-reported LaaJ-human correlations in 254 papers are unbiased estimates of true alignment
    Figure 7 aggregates second-hand values with n=1-36 per criterion and inherits publication and protocol-selection bias from the source papers.
  • domain assumption ACL/EMNLP/NAACL/INLG main-conference papers represent NLG research practice
    The coverage claim about 'the field' rests on four venues and main track only; excludes arXiv-only work, ICLR/NeurIPS, workshops, and findings papers.
  • standard math Standard frequency and likelihood-ratio definitions (Eq. 1-2)
    Background statistics used for all association measures; uncontroversial.
invented entities (1)
  • Evaluation Checklist no independent evidence
    purpose: Minimal guideline for metric selection, construct validity, and LaaJ deployment distilled from the findings
    A recommendation artifact named in the arXiv abstract; the full text contains three recommendations but no concrete checklist artifact and no falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 23402 in / 23548 out tokens · 209776 ms · 2026-08-03T11:02:33.406549+00:00 · methodology

0 comments
read the original abstract

As Natural Language Generation (NLG) dominates modern NLP, scalable evaluation remains a critical bottleneck. Consequently, LLM-as-a-judge (LaaJ) adoption has accelerated rapidly, appearing in more papers than human evaluation in 2025. This pivotal shift motivates a critical analysis of current evaluation practices. Overcoming the limits of rigid keyword filtering and manual review, we employ a multi-LLM information extraction pipeline to gather structured metadata from 14,171 papers across four major NLP conferences (2020-2025). Analyzing 3,334 filtered NLG papers, we identify three systemic challenges. (1) Metric inertia: despite the shift toward open-ended generation, legacy lexical metrics (BLEU, ROUGE) persist as primary indicators, typically used alongside rather than replaced by semantic alternatives. (2) Metric-criteria mapping problem: our paper-level co-occurrence data reveals that general-purpose automatic metrics are applied as broad proxies for quality, without specifying which dimension of text generation they are intended to evaluate. (3) Validation gap: LaaJ has grown rapidly without commensurate human validation (fewer than 8% of papers). Crucially, while LaaJ correlates with aggregate quality, alignment collapses on fine-grained criteria like fluency. To address these gaps, we distill our findings into a minimal Evaluation Checklist to guide metric selection, construct validity, and LaaJ deployment.

Figures

Figures reproduced from arXiv: 2601.07648 by Charlott Jakob, David Schlangen, Eleni Metheniti, Jing Yang, Konrad Rieck, Leonhard Hennig, Nils Feldhus, Qianli Wang, Salar Mohtaj, Sebastian M\"oller, Sherzod Hakimov, Vera Schmitt, Veronika Solopova.

Figure 1
Figure 1. Figure 1: The fractured landscape of NLG evaluation, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Our paper annotation pipeline (§3), including converting PDF to text (§3.1), extraction of metadata based on the NLG evaluation questionnaire ( [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of different terms across years, numbers are counted are after normalization. The only [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of papers (2020 – 2025) with [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Bump charts of task-specific NLG evaluation trends across four tasks (left-to-right) and three paradigms [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Metric-criteria association heatmaps for LaaJ (left) and human evaluation (right). Stronger color indicates [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of LaaJ vs. human evaluation scores by metric and criterion. Individual observations are [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Heatmap of task-year distribution of the top-30 tasks. Both the total number and percentage per year are [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of metadata by tasks, columns include top-four and all-tasks, and rows are the metadata [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 5 linked inside Pith

  1. [1]

    Does the paper address NLG tasks?

  2. [2]

    Does the paper use automatic metrics to evaluate the generated outputs?

  3. [3]

    InFindings of the As- sociation for Computational Linguistics: ACL 2025, pages 5880–5895, Vienna, Austria

    An empirical study of LLM-as-a-judge for LLM evaluation: Fine-tuned judge model is not a general substitute for GPT-4. InFindings of the As- sociation for Computational Linguistics: ACL 2025, pages 5880–5895, Vienna, Austria. Association for Computational Linguistics. Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-Kiu Lo, Vilém Zouhar, Fre...

    2025

  4. [4]

    answer_1

    Does the paper conduct *human* evaluations of the generated outputs? ANSWER FORMAT (strict) { "answer_1": { "answer": "Yes|No", "quote": "...", "tasks": ["Text Summarization", "Machine Translation", "Other:<task>"], "datasets": ["..."], "languages": ["English","Chinese","German","..."], "models": ["..."], "outputs": "..." }, "answer_2": { "answer": "Yes|N...

  5. [5]

    **Remove** any irrelevant or incorrect items ## Paper Information **Paper ID:** {paper_id} **Title:** {title} **Abstract:** {abstract} **Full Paper Text:** {full_text} --- ## Extracted Metadata to Review ### Question 1: Does the paper address NLG tasks? **Extracted Answer:** {answer_1_answer} **Extracted Metadata:** - **Tasks:** {answer_1_tasks} - **Datas...

  6. [6]

    multilingual

    Large language models for data annotation and synthesis: A survey. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing, pages 930–957, Miami, Florida, USA. Association for Computational Linguistics. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the hu...

    Pith/arXiv arXiv 2024

  7. [7]

    Review your annotations for completeness and consistency

  8. [8]

    Add any notes about difficult decisions or uncertainties Questions or Issues? If you encounter any problems during annotation: • Document unclear cases in the notes section • Flag papers that are ambiguous or difficult to categorize • Ask for clarification on systematic issues 33

  9. [9]

    Does the paper use Large-Language Models (LLMs) as judges (i.e., *after* generation, an LLM is used to judge/assess the outputs)?

  10. [11]

    Answer **Yes** only if an LLM is used *after* generation to assess the outputs. 18

  11. [12]

    **methods** - short name/description of the evaluation procedure or prompt

  12. [13]

    fluency",

    **criteria** - list the rubric properties the LLM is asked to score (e.g., ,→"fluency","relevance","helpfulness"). If the prompt does not specify criteria, leave as an empty list ,→[]. * **Answer-4 guidance (human evaluation)** * The **quote** must mention humans, annotators, raters, a crowdsourcing platform, or a similar ,→human-evaluation indicator. * T...

  13. [14]

    **Verify** the extracted yes/no answers are correct

  14. [15]

    BLEU" instead of

    **Normalize** metadata to use canonical forms (e.g., "BLEU" instead of "bleu")

  15. [16]

    **Correct** any incorrect items

  16. [17]

    **Add** any missing important items

  17. [19]

    **Verify the Yes/No answer** - Is it correct based on the full paper text?

  18. [20]

    BLEU" vs

    **Review the metadata lists** - For each item: - Is it correctly extracted from the paper? - Is it relevant to the specific question? - Should it be normalized? (e.g., "BLEU" vs "bleu", "GPT-3" vs "gpt-3")

  19. [21]

    **Add missing items** - Are there important items mentioned in the paper that are missing?

  20. [22]

    BLEU" not

    **Remove incorrect items** - Are there items that shouldn't be there? ## Guidelines ### Normalization Rules - Use canonical/standard forms (e.g., "BLEU" not "bleu", "GPT-3" not "gpt-3") - Use consistent capitalization for metrics, models - Use title case for tasks (e.g., "Machine Translation") - **For metrics**: Simplify to base form (e.g., "BLEU-1", "BLE...

  21. [23]

    **Full validation**: Paper uses LLM to evaluate all outputs, validates on human-annotated subset, ,→reports correlation

  22. [24]

    **Parallel evaluation**: Both LLM and human evaluate the same outputs, direct comparison

  23. [25]

    **Sequential validation**: Human labels used as ground truth, LLM accuracy measured

  24. [26]

    **Independent streams**: Both methods used but never compared (answer "No")

  25. [27]

    Yes” answer to Question 1). However, this filtering may not be perfect. You should verify this classification and change the answer to “No

    **Qualitative only**: Paper discusses differences but no quantitative comparison (answer "No") **Read the full paper text carefully** and extract all validation-related information accurately. 25 E Human Annotation Guideline Overview This document provides detailed instructions for manually annotating research papers about natural language generation (NLG...

  26. [28]

    Download and read the paper using the PDF link provided in the spreadsheet

  27. [29]

    Yes”). However, the automatic filtering may have made mistakes. Your job is to verify this classification by carefully reading the paper. How to Answer: Answer “Yes

    Take notes as you read to identify relevant information for each question Step 2: Answer Four Main Questions For each paper, you will answer four yes/no questions and extract relevant metadata. Question 1: Does the paper address NLG tasks? Definition: Natural Language Generation (NLG) refers to tasks where a system produces/generates natural language text...

  28. [30]

    Other: [specific task name]

    Tasks (List of NLG task types) What to include: • The main NLG task(s) that the paper addresses • Use standardized task names from this list: –Text Summarization –Dialogue Generation –Paraphrase Generation –Machine Translation –Image Captioning –Code Generation • If the task doesn’t fit any category, use: “Other: [specific task name]” Examples: • [GOOD] “...

  29. [31]

    CNN/DailyMail

    Datasets (List of dataset names) What to include: • Names of NLG datasets used for experiments or evaluation • Include datasets that are central to the paper’s contribution • Use the official dataset name as cited in the paper Examples: • [GOOD] “CNN/DailyMail”, “XSum”, “WMT14”, “MultiWOZ” • [BAD] Don’t include: Generic terms like “news articles”, “dialog...

  30. [32]

    English”, “Chinese

    Languages (List of languages) What to include: • The language(s) of the generated outputs • Use standard language names in English Examples: • [GOOD] “English”, “Chinese”, “German”, “French” • [BAD] Don’t use: ISO codes like “en”, “zh” (use full names) Instructions: • Include all target languages for generation • For multilingual papers, list all language...

  31. [33]

    GPT-3”, “BART

    Models (List of model names) What to include: • Names of NLG models used or proposed for GENERATION (not evaluation) • These are models that produce/generate the text outputs 27 • Include both models proposed by the authors and baseline generation models IMPORTANT: This is for generation models only, NOT evaluation models: • [GOOD] Include: Models that ge...

  32. [34]

    News article summaries

    Outputs (List of output descriptions) What to include: • Brief descriptions of what text is being generated • Focus on the actual output artifacts, not the process Examples: • [GOOD] “News article summaries”, “Task-oriented dialogue responses”, “English-to-German translations”, “Image captions” • [BAD] Don’t use long sentences: “The approach generates tas...

  33. [35]

    GPT-4”, “GPT-3.5

    Models (List of LLM names used as judges) What to include: • Names of specific LLMs used for evaluation 29 • Include version numbers when specified Examples: • [GOOD] “GPT-4”, “GPT-3.5”, “Claude-3”, “PaLM-2”, “Llama-2-70B” • [BAD] Don’t use: “ChatGPT” (use “GPT-3.5-Turbo” or “GPT-4” if version is known) Normalization rules: • Use official model names with...

  34. [36]

    Pairwise comparison

    Methods (List of evaluation methods/approaches) What to include: • Brief description or name of the evaluation procedure • How the LLM is prompted or used Examples: • [GOOD] “Pairwise comparison”, “Direct scoring”, “Likert scale rating”, “Binary preference”, “Multi-aspect scoring” • [GOOD] “Chain-of-thought evaluation”, “Self-consistency” • [BAD] Don’t in...

  35. [37]

    Fluency”, “Relevance

    Criteria (List of evaluation criteria) What to include: • The specific aspects or dimensions that the LLM is asked to evaluate • The rubric properties being scored Examples: • [GOOD] “Fluency”, “Relevance”, “Coherence”, “Factuality”, “Helpfulness”, “Safety” • [BAD] Don’t include: The scores themselves (like “1-5 scale”) If criteria are not specified: • Le...

  36. [38]

    Pairwise comparison

    Methods (List of evaluation methods/approaches) What to include: • Brief description or name of the evaluation procedure • How human evaluators are asked to assess the outputs • The type of evaluation task (rating, ranking, comparison, etc.) Examples: • [GOOD] “Pairwise comparison”, “Direct scoring”, “Likert scale rating”, “Binary preference”, “Multi-aspe...

  37. [39]

    Fluency”, “Adequacy

    Criteria (List of evaluation criteria) What to include: • The specific aspects or dimensions that humans are asked to evaluate • Evaluation categories or rubric items Examples: • [GOOD] “Fluency”, “Adequacy”, “Coherence”, “Informativeness”, “Naturalness”, “Relevance”, “Grammaticality”, “Readability”, “Factuality” • [BAD] Don’t include: The scores themselv...

  38. [40]

    Read the paper (especially abstract, methodology, and evaluation sections)

  39. [41]

    Yes” • Change to “No

    Answer Question 1: Does it address NLG tasks? • Verify the pre-filtered classification - the paper was initially classified as “Yes” • Change to “No” if it doesn’t actually address NLG tasks • If No: Skip to the next paper (we only annotate NLG papers) • If Yes: Continue to extract all metadata

  40. [42]

    Extract Q1 metadata: tasks, datasets, languages, models, outputs 32

  41. [43]

    Answer Question 2: Does it use automatic metrics? • If Yes: Extract and normalize metric names

  42. [44]

    Answer Question 3: Does it use LLMs as judges? • If Yes: Extract LLM models, methods, and criteria

  43. [45]

    Answer Question 4: Does it conduct human evaluation? • If Yes: Extract evaluation methods and criteria

  44. [2020]

    InProceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4984–4997, Online

    Tangled up in BLEU: Reevaluating the eval- uation of automatic machine translation evaluation metrics. InProceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computa- tional Linguistics. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus,...

    Pith/arXiv arXiv 2025

  45. [2021]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

    Evaluation of text generation: A survey.arXiv, abs/2006.14799. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao, and 181 others

    Pith/arXiv arXiv 2006

  46. [2024]

    Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, and Amy X Zhang

    Llms-as-judges: A comprehensive sur- vey on llm-based evaluation methods.Preprint, arXiv:2412.05579. Zhehui Liao, Maria Antoniak, Inyoung Cheong, Evie Yu-Yen Cheng, Ai-Heng Lee, Kyle Lo, Joseph Chee Chang, and Amy X Zhang. 2025. LLMs as research tools: A large scale survey of researchers’ usage and perceptions. InSecond Conference on Language Modeling. Ni...

    Pith/arXiv arXiv 2025

  47. [2025]

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning.Preprint, arXiv:2501.12948. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Sri- nath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao...

    Pith/arXiv arXiv 2024