Fine-tuning Whisper on Estonian subtitles with iterative pseudo-labeling and test-time LLM editing improves subtitle quality, while LLM editing during training yields no gain.
SubER: A Metric for Automatic Evaluation of Subtitle Quality
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
This paper addresses the problem of evaluating the quality of automatically generated subtitles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and subtitle quality. A careful human evaluation in a post-editing scenario shows that the new metric has a high correlation with the post-editing effort and direct human assessment scores, outperforming baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs
Fine-tuning Whisper on Estonian subtitles with iterative pseudo-labeling and test-time LLM editing improves subtitle quality, while LLM editing during training yields no gain.