REVIEW 5 major objections 4 minor 26 references
Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs
T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Fine-tuning Whisper on human Estonian subtitles, then retraining on two rounds of model-generated pseudo-labels, improves subtitle quality across all metrics; test-time LLM editing adds further gains.
desk verdict Useful engineering result for Estonian subtitles, but the paper overclaims pseudo-labeling gains across all metrics when its own Table 2 shows t-BLEURT dropping. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is iterative pseudo-labeling: the current model transcribes unlabeled audio, those transcriptions are added to the supervised training set, and the model is retrained for two rounds. Training uses speed perturbation, SpecAugment, and a weighted loss $L_{\text{total}} = (1-\lambda)L_{\text{supervised}} + \lambda L_{\text{pseudo}}$ with $\lambda = 0.35$ chosen by Optuna. Quality is measured with SubER, an edit-distance metric that accounts for subtitle timing and segmentation shifts, plus two BLEURT-based semantic similarity variants (t-BLEURT and AS-BLEURT). LLM post-editing operates on chunks of 40 subtitle blocks, with a verification step that re-requests the LLM when block numbers or timestamps are corrupted.
What would settle it
Take a held-out subset of the 3,923-hour unlabeled set, transcribe it with the baseline model, and compare the pseudo-labels against human subtitles for the same broadcasts; then train a control model on an equally sized randomly sampled subset of that unlabeled audio without pseudo-labels—if the pseudo-label-trained model does not beat this control on SubER, the iterative gain claim collapses.
Extended reading notes
Core claim
The central discovery is that iterative pseudo-labeling of a large unlabeled dataset improves subtitle quality across all tested metrics, and that test-time LLM post-editing adds a further gain, in the specific setting of Estonian TV subtitles. Starting from a Whisper large-v3 model fine-tuned on 778 hours of human subtitles (SubER 35.1), two training iterations on model-generated pseudo-labels from 3,923 hours of unlabeled audio lower SubER to 33.4 and raise AS-BLEURT from 0.799 to 0.853. Applying GPT-4o post-editing at test time improves the best combined system to SubER 33.1 and AS-BLEURT 0.858, whereas applying GPT-4o to pseudo-labels during training gives no further gain. The authors also show that fine-tuning on subtitles rather than verbatim transcripts matters greatly, cutting SubER from 51.5 to 35.1.
Load-bearing premise
The model's own transcriptions of thousands of hours of unlabeled Estonian broadcasts are accurate enough to serve as training targets, and retraining on them without any confidence filtering will not reinforce systematic errors.
Editorial extensions
If this is right
- The full pipeline—supervised fine-tuning, two rounds of pseudo-labeling, and test-time GPT-4o editing—yields the best subtitle quality, with SubER 33.1 and AS-BLEURT 0.858.
- Subtitle-specific fine-tuning matters: switching from verbatim transcripts to human subtitles cuts SubER from 51.5 to 35.1.
- Pseudo-labeling gains accumulate over iterations: iteration 1 gives SubER 34.5, and iteration 2 gives 33.4 without LLM editing.
- LLM post-editing is only useful at inference: editing pseudo-labels during training does not improve the final model.
Reading between the lines
- The SubER gains from pseudo-labeling may partly reflect simply training on more in-domain audio; a control experiment that trains on an equal number of hours of unlabeled audio without pseudo-labels would separate the two effects.
- The recipe should transfer to other languages that have professionally subtitled broadcast corpora but limited verbatim transcripts, since it uses only unlabeled audio plus existing subtitles.
- Real-time subtitling would require replacing GPT-4o with a faster distilled or on-device editor; the paper's chunked 40-block post-editing is designed for parallelization and could feed a streaming pipeline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a system for generating same-language Estonian TV subtitles by fine-tuning Whisper large-v3 on human subtitles, then applying two iterations of pseudo-labeling on 3,923 hours of unlabeled broadcast audio, optionally with GPT-4o-based post-editing of pseudo-labels at training time and/or of test hypotheses at inference. Experiments report SubER, t-BLEURT, and AS-BLEURT on a 17-recording test set, with Wilcoxon signed-rank tests comparing three highlighted models. The authors claim that iterative pseudo-labeling improves subtitle quality across all metrics and that test-time LLM post-editing further helps, while training-time LLM post-editing does not.
Significance. If the empirical findings were fully supported, the paper would offer a practical and reproducible recipe for under-resourced language subtitling: fine-tune Whisper on a modest set of human subtitles, retrain on model-generated pseudo-labels, and post-edit with a strong LLM at inference. The authors ship the fine-tuned model and a demo, which is a concrete contribution to the subtitling and ASR community. However, the central claim is overstated relative to the paper's own data (t-BLEURT decreases with pseudo-labeling alone), the statistical tests do not isolate the pseudo-labeling effect, and several methodological gaps (test-set reuse, absent pseudo-label quality checks, no human evaluation) prevent the paper from being accepted as is. The core recipe may still be valid, but the manuscript needs substantial revision to align claims with evidence.
major comments (5)
- [§1, §4.6, Table 2] The paper claims in the introduction and in §4.6 that iterative pseudo-labeling improves subtitle quality 'across all metrics'. This is contradicted by Table 2: the model with two iterations of pseudo-labeling and no LLM (SubER 33.4, t-BLEURT .529, AS-BLEURT .853) has lower t-BLEURT than the supervised baseline model A (SubER 35.1, t-BLEURT .545, AS-BLEURT .799). The same holds for iteration 1 (t-BLEURT .526). Since t-BLEURT is one of the paper's three primary metrics, the statement as written is false. The authors should either revise the claim to name only SubER and AS-BLEURT, or provide a substantive argument why the t-BLEURT drop is irrelevant to the conclusion.
- [§4.6] The Wilcoxon signed-rank tests are computed only between models A, B, and C. Model B is the baseline plus test-time LLM, and model C adds pseudo-labeling on top of that, so these tests do not isolate the effect of pseudo-labeling. They cannot support the claim that pseudo-labeling alone improves metrics. Pairwise tests between model A and the pseudo-label-only models (e.g., the iteration-2 row without LLM) should be reported, or the claim should be explicitly restricted to the full pipeline.
- [§4.4, §4.6] The pseudo-labeling setup has no confidence filtering and no report of pseudo-label quality on the unlabeled set. Because the unlabeled set is roughly five times larger than the supervised set and contains news programs not present in the supervised data, the observed SubER and AS-BLEURT gains could be due to simply adding more in-domain audio rather than to the iterative refinement mechanism. The authors should measure pseudo-label accuracy (e.g., by running the pseudo-labeling model on a held-out subset of human subtitles and computing WER/SubER) or compare against a control that trains on the same amount of audio with a fixed teacher. At the very least, this confound should be discussed explicitly.
- [§4.5, Table 1] The selection of GPT-4o as the post-editing LLM is based on SubER scores computed on the test set ('Table 1 shows the SubER results on test data'). The same test set is then used to report final results, making the reported test-time LLM gains optimistic. The authors should reserve a development set for such model-selection decisions or otherwise qualify the reported gains. Similarly, the Optuna-based choice of λ in §4.4 should state which data split was used for optimization.
- [§4.6, Abstract] The abstract's statement that the approach 'holds promise for creating subtitle quality close to human standard' and the §4.6 assertion that 'minimal manual post-editing would be required to achieve error-free subtitles' are based only on the authors' subjective assessment; no human evaluation was conducted. This is not a minor omission but a missing piece of evidence for a quality claim directed at human viewers. A small-scale human rating study or an explicit removal of this claim from the abstract and discussion is needed.
minor comments (4)
- [Table 2] The meaning of the '!' symbol in the two LLM-post-editing columns is not defined in the table caption. Also, the column headers 'Pseudo-label LLM-post-editing?' and 'Test-time LLM-post-editing?' are easy to confuse; consider renaming them 'LLM post-editing of pseudo-labels (training time)' and 'LLM post-editing of test hypotheses'.
- [§4.6] Report p-values as '<0.001' rather than '0.000', and specify whether the Wilcoxon test was one-sided or two-sided and that it was computed over 17 test recordings.
- [§1] There is a typo in the introduction: 'it’s use at training time' should be 'its use at training time'.
- [§4.2] The phrase 'BLEURT outputs scores that usually in the range of 0..1' is awkward and vague; rephrase to state the actual score range and that higher is better.
Circularity Check
No circularity: all key claims are evaluated against held-out human references with standard external metrics.
full rationale
The paper is an empirical study, not a derivation, and its central claims are not equivalent to its inputs by construction. The training pipeline uses human-generated Estonian subtitles as supervised data, model-generated pseudo-labels from a large unlabeled set, and LLM post-editing; the reported improvements are measured with held-out human reference subtitles using external metrics (SubER, t-BLEURT, AS-BLEURT). Because the reference subtitles are human-authored and independent of the model's own outputs, the pseudo-labeling results are not forced: the model could have failed to improve or could have drifted toward its own errors, and the paper's own Table 2 shows mixed results across metrics. The self-citation to Alumäe et al. (2023) is used only to provide a verbatim-transcript baseline model for comparison, not to justify the target result or to forbid alternatives. No uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result are present. The abstract's wording that iterative pseudo-labeling 'improves subtitle quality across all metrics' is internally inconsistent with Table 2, where the two-iteration pseudo-labeling model without LLM post-editing shows t-BLEURT dropping from .545 to .529 while SubER improves from 35.1 to 33.4; however, this is a correctness or consistency concern, not circularity. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- lambda (pseudo-label loss weight) =
0.35
- reference check failure threshold =
3
- LLM chunk size =
40 subtitle blocks
- number of pseudo-label iterations =
2
assumptions (4)
- domain assumption SubER and BLEURT scores reflect subtitle quality as judged by humans.
- domain assumption Pseudo-labels from the current Whisper model are accurate enough to serve as training targets.
- domain assumption Human-generated subtitles are a valid gold standard for the subtitle generation task.
- standard math Wilcoxon signed-rank test assumptions hold for the paired metric differences.
Cite this review
Pith. "Pith review of Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs." pith.science (2026). https://pith.science/paper/LYI47JO4
@misc{pith2026250105234,
author = {Pith},
title = {Pith review of: Optimizing Estonian TV Subtitles with Semi-supervised Learning and LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/LYI47JO4}},
note = {Machine review of arXiv:2501.05234}
}
read the original abstract
This paper presents an approach for generating high-quality, same-language subtitles for Estonian TV content. We fine-tune the Whisper model on human-generated Estonian subtitles and enhance it with iterative pseudo-labeling and large language model (LLM) based post-editing. Our experiments demonstrate notable subtitle quality improvement through pseudo-labeling with an unlabeled dataset. We find that applying LLM-based editing at test time enhances subtitle accuracy, while its use during training does not yield further gains. This approach holds promise for creating subtitle quality close to human standard and could be extended to real-time applications.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Tanel Alum \"a e, Joonas Kalda, K \"u lliki Bode, and Martin Kaitsa. Automatic closed captioning for E stonian live broadcasts. In Proc. NoDaLiDa, T \'o rshavn, Faroe Islands, May 2023. URL: https://aclanthology.org/2023.nodalida-1.49
work page 2023
-
[4]
Data augmentation and teacher-student training for LF-MMI based robust speech recognition
Asadullah and Tanel Alum \"a e. Data augmentation and teacher-student training for LF-MMI based robust speech recognition. In Proc. TSD, pages 403--410. Springer, 2018
work page 2018
-
[5]
Subtitle translation as markup translation
Colin Cherry, Naveen Arivazhagan, Dirk Padfield, and Maxim Krikun. Subtitle translation as markup translation. In Proc. Interspeech, pages 2237--2241, 2021
work page 2021
-
[6]
Semi-supervised Learning for Code-Switching ASR with Large Language Model Filter
Yu Xi, Wen Ding, Kai Yu, and Junjie Lai. Semi-supervised learning for code-switching ASR with large language model filter. arXiv preprint arXiv:2407.04219, 2024
work page Pith review arXiv 2024
-
[7]
Averaging weights leads to wider optima and better generalization
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence (UAI), pages 876--885. AUAI Press, 2018
work page 2018
-
[8]
Comparing the impact of professional and automatic closed captions on video-watching experience
Hyunju Kim, Yan Tao, Chuanrui Liu, Yuzhuo Zhang, and Yuxin Li. Comparing the impact of professional and automatic closed captions on video-watching experience. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, 2023
work page 2023
Show all 26 references
-
[9]
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019
2019
-
[10]
Can generative large language models perform asr error correction? arXiv preprint arXiv:2307.04172, 2023
Rao Ma, Mengjie Qian, Potsawee Manakul, Mark Gales, and Kate Knill. Can generative large language models perform asr error correction? arXiv preprint arXiv:2307.04172, 2023
2023 arXiv
-
[11]
Anantha, Daben Liu, and Sashank Gondala
Ernest Pusateri, Anmol Walia, Anirudh Kashi, Bortik Bandyopadhyay, Nadia Hyder, Sayantan Mahinder, R. Anantha, Daben Liu, and Sashank Gondala. Retrieval augmented correction of named entity speech recognition errors. arXiv preprint arXiv:2409.06062, 2024
2024 arXiv
-
[12]
Multi-stage large language model correction for speech recognition
Jie Pu, Thai-Son Nguyen, and Sebastian St \"u ker. Multi-stage large language model correction for speech recognition. arXiv preprint arXiv:2310.11532, 2023
2023 arXiv
-
[13]
Evaluating machine translation output with automatic sentence segmentation
Evgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. Evaluating machine translation output with automatic sentence segmentation. In Proc. IWSLT, 2005
2005
-
[14]
Optuna: A next-generation hyperparameter optimization framework
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2019
2019
-
[15]
Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. SpecAugment: A simple data augmentation method for automatic speech recognition. In Interspeech 2019. ISCA, September 2019. URL: http://dx.doi.org/10.21437/Interspeech.2019-26...
2019 doi
-
[16]
Survey: Why America is obsessed with subtitles
Nadiia Mykhalevych and Preply. Survey: Why America is obsessed with subtitles. 2024. URL: https://preply.com/en/blog/americas-subtitles-use/
2024
-
[17]
Robust speech recognition via large-scale supervision
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale supervision. arXiv preprint arXiv:2212.04356, 2022
2022 arXiv
-
[18]
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. BLEURT : Learning robust metrics for text generation. In Proc. ACL, 2020
2020
-
[19]
Silero VAD : Pre-trained enterprise-grade voice activity detector
Silero Team. Silero VAD : Pre-trained enterprise-grade voice activity detector. 2021. URL: https://github.com/snakers4/silero-vad
2021
-
[20]
SubER : A metric for automatic evaluation of subtitle quality
Patrick Wilken, Panayota Georgakopoulou, and Evgeny Matusov. SubER : A metric for automatic evaluation of subtitle quality. arXiv preprint arXiv:2205.05805, 2022
2022 arXiv
-
[21]
Iterative pseudo-labeling for speech recognition
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert. Iterative pseudo-labeling for speech recognition. arXiv preprint arXiv:2005.09267, 2020
2005 arXiv
-
[22]
Semi-supervised training of deep neural networks
Karel Vesel \`y , Mirko Hannemann, and Luk \'a s Burget. Semi-supervised training of deep neural networks. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 267--272. IEEE, 2013
2013
-
[23]
Parikh, Sebastian Gehrmann, and Thibault Sellam
Amy Pu, Hyung Won Chung, Ankur P. Parikh, Sebastian Gehrmann, and Thibault Sellam. Learning compact metrics for MT . In Proc. EMNLP, 2021
2021
-
[24]
Using untranscribed training data to improve performance
George Zavaliagkos, Man-Hung Siu, Thomas Colthurst, and Jayadev Billa. Using untranscribed training data to improve performance. In Proc. ICSLP, volume 1998, 1998
1998
-
[25]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clément Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...
1910 arXiv
-
[26]
Individual comparisons by ranking methods
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80--83, 1945. URL: https://doi.org/10.2307/3001968, DOI: 10.2307/3001968
1945 doi
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.