Pith. sign in

REVIEW 1 cited by

Evaluating the Efficacy of Summarization Evaluation across Languages

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2106.01478 v1 pith:I6NGPABD submitted 2021-06-02 cs.CL

Fajri Koto , Jey Han Lau , Timothy Baldwin This is my paper
classification cs.CL
keywords languagessummarizationevaluationacrossefficacyenglishaboveannotate
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Across eight languages, n-gram metrics such as ROUGE correlate less with human ratings in fusional languages than in isolating and agglutinative ones, while the neural metric COMET correlates better, especially in low...

Pith tools