Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

Adarsa Sivaprasad; Albert Gatt; David M. Howcroft; Dimitra Gkatzia; Ond\v{r}ej Du\v{s}ek; Ond\v{r}ej Pl\'atek; Patr\'icia Schmidtov\'a; Saad Mahamood; Simone Balloccu

Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.09169 v1 pith:4BXVANW4 submitted 2024-08-17 cs.CL

Patrícia Schmidtová , Saad Mahamood , Simone Balloccu , Ondřej Dušek , Albert Gatt , Dimitra Gkatzia , David M. Howcroft , Ondřej Plátek

show 1 more author

Adarsa Sivaprasad

This is my paper

classification cs.CL

keywords metricsautomaticlanguagenaturalsurveyusedfieldgeneration

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use of automatic metrics, focusing particularly on natural language generation (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, including inappropriate metric usage, lack of implementation details and missing correlations with human judgements. We conclude with recommendations that we believe authors should follow to enable more rigour within the field.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach
cs.CV 2025-07 conditional novelty 5.0 of 10

The paper presents GEST, an event-graph representation of videos that is converted automatically into natural language and is also used as a teacher to pre-train end-to-end video captioning models.

Pith tools