{"id":"972a13da-8f2b-413a-bcd5-11f5e0691eff","arxiv_id":"2501.05234","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning Whisper on Estonian subtitles with iterative pseudo-labeling and test-time LLM editing improves subtitle quality, while LLM editing during training yields no gain.","lead":"This paper fine-tunes Whisper on 778 hours of human-generated Estonian TV subtitles, then adds iterative self-training on 3,923 hours of unlabeled audio and GPT-4o post-editing. Subtitle edit rate drops from 35.1 to 33.4, and the surprising result is that using GPT-4o to clean pseudo-labels during training does not help.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 contradicts 'improves across all metrics': iter-2 pseudo-labeling alone lowers t-BLEURT from .545 to .529, so the abstract/intro overclaim is unsupported by the paper's own data.","rationale":"The reader's weakest assumption (unfiltered pseudo-label accuracy, confound with added data) is a reasonable external-validity concern, but it is speculative: the reported improvements could indeed come from effective pseudo-labels. The directly observable problem is stronger: the paper's own table contains a metric that moves in the wrong direction for the stated claim. Because the strongest claim explicitly says 'across all metrics', and t-BLEURT decreases from .545 to .529 in the key comparison, the paper as written is internally inconsistent. This does not invalidate the entire contribution; the SubER and AS-BLEURT gains remain, and the LLM post-editing results are promising. Hence the conditional verdict is appropriate, pending a correction of the claim or additional evidence for t-BLEURT. I therefore keep the reader's CONDITIONAL verdict unchanged but flag a different, more decisive weakness.","tokens_in":122,"tokens_out":5463,"duration_ms":56587,"concrete_test":"Re-extract per-recording t-BLEURT scores for model A and the iteration-2 pseudo-label-only model from the evaluation script, and compute a paired bootstrap or Wilcoxon signed-rank test on the differences. If the median or mean t-BLEURT remains lower for the pseudo-label model (as Table 2 suggests), the sentence 'improves subtitle quality across all metrics' must be revised. Also verify the column alignment in Table 2 to ensure the 0.529 entry is not from a different configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract; §1) that iterative pseudo-labeling improves subtitle quality across all metrics is directly contradicted by its own Table 2. Compare model A ('Subtitles', SubER 35.1, t-BLEURT .545, AS-BLEURT .799) with the two-iteration pseudo-labeling model without LLM ('Subtitles + pseudo-labels', SubER 33.4, t-BLEURT .529, AS-BLEURT .853). While SubER improves (35.1→33.4) and AS-BLEURT improves (.799→.853), t-BLEURT degrades (.545→.529). Thus the claim 'across all metrics' is false for one of the paper's own evaluation metrics. The Wilcoxon test reported in §4.6 compares only models A, B and C and does not address this pairwise comparison, nor is any significance or error bar given for the t-BLEURT drop. The weaker claim that pseudo-labeling improves SubER and AS-BLEURT may survive, but the central statement as written does not. This is an internal inconsistency, not a matter of external consensus.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a system for generating same-language Estonian TV subtitles by fine-tuning Whisper large-v3 on human subtitles, then applying two iterations of pseudo-labeling on 3,923 hours of unlabeled broadcast audio, optionally with GPT-4o-based post-editing of pseudo-labels at training time and/or of test hypotheses at inference. Experiments report SubER, t-BLEURT, and AS-BLEURT on a 17-recording test set, with Wilcoxon signed-rank tests comparing three highlighted models. The authors claim that iterative pseudo-labeling improves subtitle quality across all metrics and that test-time LLM post-editing further helps, while training-time LLM post-editing does not.","tokens_in":6492,"tokens_out":6810,"duration_ms":58492,"significance":"If the empirical findings were fully supported, the paper would offer a practical and reproducible recipe for under-resourced language subtitling: fine-tune Whisper on a modest set of human subtitles, retrain on model-generated pseudo-labels, and post-edit with a strong LLM at inference. The authors ship the fine-tuned model and a demo, which is a concrete contribution to the subtitling and ASR community. However, the central claim is overstated relative to the paper's own data (t-BLEURT decreases with pseudo-labeling alone), the statistical tests do not isolate the pseudo-labeling effect, and several methodological gaps (test-set reuse, absent pseudo-label quality checks, no human evaluation) prevent the paper from being accepted as is. The core recipe may still be valid, but the manuscript needs substantial revision to align claims with evidence.","major_comments":[{"comment":"The paper claims in the introduction and in §4.6 that iterative pseudo-labeling improves subtitle quality 'across all metrics'. This is contradicted by Table 2: the model with two iterations of pseudo-labeling and no LLM (SubER 33.4, t-BLEURT .529, AS-BLEURT .853) has lower t-BLEURT than the supervised baseline model A (SubER 35.1, t-BLEURT .545, AS-BLEURT .799). The same holds for iteration 1 (t-BLEURT .526). Since t-BLEURT is one of the paper's three primary metrics, the statement as written is false. The authors should either revise the claim to name only SubER and AS-BLEURT, or provide a substantive argument why the t-BLEURT drop is irrelevant to the conclusion.","section":"§1, §4.6, Table 2"},{"comment":"The Wilcoxon signed-rank tests are computed only between models A, B, and C. Model B is the baseline plus test-time LLM, and model C adds pseudo-labeling on top of that, so these tests do not isolate the effect of pseudo-labeling. They cannot support the claim that pseudo-labeling alone improves metrics. Pairwise tests between model A and the pseudo-label-only models (e.g., the iteration-2 row without LLM) should be reported, or the claim should be explicitly restricted to the full pipeline.","section":"§4.6"},{"comment":"The pseudo-labeling setup has no confidence filtering and no report of pseudo-label quality on the unlabeled set. Because the unlabeled set is roughly five times larger than the supervised set and contains news programs not present in the supervised data, the observed SubER and AS-BLEURT gains could be due to simply adding more in-domain audio rather than to the iterative refinement mechanism. The authors should measure pseudo-label accuracy (e.g., by running the pseudo-labeling model on a held-out subset of human subtitles and computing WER/SubER) or compare against a control that trains on the same amount of audio with a fixed teacher. At the very least, this confound should be discussed explicitly.","section":"§4.4, §4.6"},{"comment":"The selection of GPT-4o as the post-editing LLM is based on SubER scores computed on the test set ('Table 1 shows the SubER results on test data'). The same test set is then used to report final results, making the reported test-time LLM gains optimistic. The authors should reserve a development set for such model-selection decisions or otherwise qualify the reported gains. Similarly, the Optuna-based choice of λ in §4.4 should state which data split was used for optimization.","section":"§4.5, Table 1"},{"comment":"The abstract's statement that the approach 'holds promise for creating subtitle quality close to human standard' and the §4.6 assertion that 'minimal manual post-editing would be required to achieve error-free subtitles' are based only on the authors' subjective assessment; no human evaluation was conducted. This is not a minor omission but a missing piece of evidence for a quality claim directed at human viewers. A small-scale human rating study or an explicit removal of this claim from the abstract and discussion is needed.","section":"§4.6, Abstract"}],"minor_comments":[{"comment":"The meaning of the '!' symbol in the two LLM-post-editing columns is not defined in the table caption. Also, the column headers 'Pseudo-label LLM-post-editing?' and 'Test-time LLM-post-editing?' are easy to confuse; consider renaming them 'LLM post-editing of pseudo-labels (training time)' and 'LLM post-editing of test hypotheses'.","section":"Table 2"},{"comment":"Report p-values as '<0.001' rather than '0.000', and specify whether the Wilcoxon test was one-sided or two-sided and that it was computed over 17 test recordings.","section":"§4.6"},{"comment":"There is a typo in the introduction: 'it’s use at training time' should be 'its use at training time'.","section":"§1"},{"comment":"The phrase 'BLEURT outputs scores that usually in the range of 0..1' is awkward and vague; rephrase to state the actual score range and that higher is better.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution to subtitle generation for a low-resource language, and the public model release is commendable. However, the central claim is internally contradicted by Table 2, and the statistical tests do not isolate the claimed effect. The test-set reuse for LLM selection is a also serious methodology concern that the authors should address. With careful revision, the paper could become publishable, but as it stands the claims outrun the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, practical paper about building a same-language subtitle system for Estonian TV, and the negative result about LLM post-editing during training is genuinely useful. But the claim in the introduction and §4.6 that iterative pseudo-labeling improves 'across all metrics' is directly contradicted by Table 2: going from the supervised subtitle model (A) to the two-iteration pseudo-labeling model without LLM post-editing, SubER improves (35.1→33.4) and AS-BLEURT improves (.799→.853), but t-BLEURT drops (.545→.529). The same pattern already appears at iteration 1 (.545→.526). So the paper's own central statement is false for one of its three metrics. The Wilcoxon test in §4.6 only compares A, B, C (the LLM post-edited variants), not the raw pseudo-labeling comparison, so it doesn't fix this.\n\nWhat's actually new: a concrete recipe for Estonian subtitles—fine-tune Whisper large-v3 on 778h of human subtitles, do two rounds of pseudo-labeling on 3,923h of unlabeled broadcast audio, and post-edit with GPT-4o at test time. That gets SubER from 35.1 to 33.1 and t-BLEURT from .545 to .598. The observation that using GPT-4o to clean pseudo-labels during training does not help, contrary to Xi et al. (2024), is a useful negative result. They also release the fine-tuned model and a demo, which is more than most papers do.\n\nSoft spots besides the overclaim: the test set is 17 recordings (about 1.7% of the supervised set), there are no error bars or per-recording breakdowns, and there is no human evaluation despite the abstract saying the approach 'holds promise for creating subtitle quality close to human standard'—that's speculative. The pseudo-labeling step is confounded with simply training on more in-domain audio; there is no control that adds a comparable amount of supervised audio or selects subsets of pseudo-labels based on confidence. The reader's weakest_assumption is right: without measuring pseudo-label accuracy on the unlabeled set, we can't know whether the gain comes from self-training or from more data. Also, only GPT-4o helped at test time; all the open-source LLMs made SubER worse, so the result is tied to a paid API.\n\nThe citation pattern looks fine—related work is credited, including Xi et al. and the earlier pseudo-labeling ASR papers.\n\nBottom line: this is a useful engineering paper for a low-resource language, with a clear and honest (if overclaimed) empirical setup. A serious referee should engage with it, but the authors need to correct the 'across all metrics' claim, add error bars or a larger test set, and ideally run a human evaluation or at least an error analysis before publication. I'd cite it if I worked on subtitling or semi-supervised ASR.","headline":"Useful engineering result for Estonian subtitles, but the paper overclaims pseudo-labeling gains across all metrics when its own Table 2 shows t-BLEURT dropping.","tokens_in":6974,"tokens_out":2916,"would_cite":true,"duration_ms":24502,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning Whisper on human Estonian subtitles, then retraining on two rounds of model-generated pseudo-labels, improves subtitle quality across all metrics; test-time LLM editing adds further gains.","keywords":["Estonian","subtitling","Whisper","semi-supervised learning","pseudo-labeling","LLM post-editing","SubER","BLEURT"],"falsifier":"Take a held-out subset of the 3,923-hour unlabeled set, transcribe it with the baseline model, and compare the pseudo-labels against human subtitles for the same broadcasts; then train a control model on an equally sized randomly sampled subset of that unlabeled audio without pseudo-labels—if the pseudo-label-trained model does not beat this control on SubER, the iterative gain claim collapses.","tokens_in":6044,"feed_emoji":"🎬","tokens_out":5592,"duration_ms":46551,"temperature":0.7,"pith_summary":"This paper tries to show that a practical recipe can generate near-human-quality Estonian TV subtitles from audio alone. The recipe is: fine-tune the Whisper speech model on human-created subtitles, retrain it twice on its own pseudo-labels for nearly 4,000 hours of unlabeled broadcast audio, and post-edit the output with GPT-4o at inference time. The authors report that each ingredient that works is measurable: pseudo-labeling improves every evaluation metric, and test-time LLM editing helps further, while LLM editing during training does not. If true, this gives broadcasters and streaming services a way to produce same-language subtitles for low-resource languages without collecting new human transcripts.","feed_headline":"Two rounds of pseudo-labels cut Estonian subtitle errors","feed_subtitle":"Retraining Whisper on its own captions improved SubER from 35.1 to 33.4; GPT-4o post-editing helped further.","key_machinery":"The load-bearing mechanism is iterative pseudo-labeling: the current model transcribes unlabeled audio, those transcriptions are added to the supervised training set, and the model is retrained for two rounds. Training uses speed perturbation, SpecAugment, and a weighted loss $L_{\\text{total}} = (1-\\lambda)L_{\\text{supervised}} + \\lambda L_{\\text{pseudo}}$ with $\\lambda = 0.35$ chosen by Optuna. Quality is measured with SubER, an edit-distance metric that accounts for subtitle timing and segmentation shifts, plus two BLEURT-based semantic similarity variants (t-BLEURT and AS-BLEURT). LLM post-editing operates on chunks of 40 subtitle blocks, with a verification step that re-requests the LLM when block numbers or timestamps are corrupted.","core_discovery":"The central discovery is that iterative pseudo-labeling of a large unlabeled dataset improves subtitle quality across all tested metrics, and that test-time LLM post-editing adds a further gain, in the specific setting of Estonian TV subtitles. Starting from a Whisper large-v3 model fine-tuned on 778 hours of human subtitles (SubER 35.1), two training iterations on model-generated pseudo-labels from 3,923 hours of unlabeled audio lower SubER to 33.4 and raise AS-BLEURT from 0.799 to 0.853. Applying GPT-4o post-editing at test time improves the best combined system to SubER 33.1 and AS-BLEURT 0.858, whereas applying GPT-4o to pseudo-labels during training gives no further gain. The authors also show that fine-tuning on subtitles rather than verbatim transcripts matters greatly, cutting SubER from 51.5 to 35.1.","pith_inferences":["The SubER gains from pseudo-labeling may partly reflect simply training on more in-domain audio; a control experiment that trains on an equal number of hours of unlabeled audio without pseudo-labels would separate the two effects.","The recipe should transfer to other languages that have professionally subtitled broadcast corpora but limited verbatim transcripts, since it uses only unlabeled audio plus existing subtitles.","Real-time subtitling would require replacing GPT-4o with a faster distilled or on-device editor; the paper's chunked 40-block post-editing is designed for parallelization and could feed a streaming pipeline."],"forward_implications":["The full pipeline—supervised fine-tuning, two rounds of pseudo-labeling, and test-time GPT-4o editing—yields the best subtitle quality, with SubER 33.1 and AS-BLEURT 0.858.","Subtitle-specific fine-tuning matters: switching from verbatim transcripts to human subtitles cuts SubER from 51.5 to 35.1.","Pseudo-labeling gains accumulate over iterations: iteration 1 gives SubER 34.5, and iteration 2 gives 33.4 without LLM editing.","LLM post-editing is only useful at inference: editing pseudo-labels during training does not improve the final model."],"supporting_citations":[{"why":"Supplies the Whisper model architecture and pretrained weights that the paper fine-tunes on subtitles.","marker":"Radford et al., 2022"},{"why":"Establishes the iterative pseudo-labeling training method that the paper adapts to subtitle data.","marker":"Xu et al., 2020"},{"why":"Defines the SubER metric, the paper's primary segmentation-aware evaluation of subtitle quality.","marker":"Wilken et al., 2022"},{"why":"Provides the BLEURT learned metric used for semantic similarity evaluation of generated subtitles.","marker":"Sellam et al., 2020"},{"why":"Supplies the verbatim Estonian speech dataset used to train the verbatim-transcript baseline for comparison.","marker":"Alum ¨ae et al., 2023"},{"why":"Previous work applying LLM filtering to pseudo-labels in code-switched ASR, which the paper contrasts with its finding that training-time LLM editing does not help.","marker":"Xi et al., 2024"},{"why":"First demonstration of zero-shot and few-shot LLM-based ASR error correction, the basis for the paper's test-time LLM post-editing approach.","marker":"Ma et al., 2023"}],"fun_headline_variants":["Pseudo-labeling lifts Estonian subtitles, LLM post-edit adds more","Retraining Whisper on its own captions cuts SubER to 33.4","Test-time GPT-4o editing boosts Estonian subtitle accuracy","From SubER 51.5 to 33.1: fine-tuning on subtitles, not transcripts","Estonian subtitles: pseudo-labels + LLM editing approach human quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model's own transcriptions of thousands of hours of unlabeled Estonian broadcasts are accurate enough to serve as training targets, and retraining on them without any confidence filtering will not reinforce systematic errors.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-labeling lifts Estonian subtitles, LLM post-edit adds more","Retraining Whisper on its own captions cuts SubER to 33.4","Test-time GPT-4o editing boosts Estonian subtitle accuracy","From SubER 51.5 to 33.1: fine-tuning on subtitles, not transcripts","Estonian subtitles: pseudo-labels + LLM editing approach human quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000538,"raw_usage":{"total_tokens":2533,"prompt_tokens":847,"completion_tokens":1686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1577}},"tokens_in":463,"tokens_out":1686,"duration_ms":10697,"temperature":1.0,"reasoning_tokens":1577,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:09.259876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out subset of the 3,923-hour unlabeled set, transcribe it with the baseline model, and compare the pseudo-labels against human subtitles for the same broadcasts; then train a control model on an equally sized randomly sampled subset of that unlabeled audio without pseudo-labels—if the pseudo-label-trained model does not beat this control on SubER, the iterative gain claim collapses.","supporting_citations":[{"cited_title":"SubER: A Metric for Automatic Evaluation of Subtitle Quality","cited_arxiv_id":"2205.05805","evidence_quote":"Defines the SubER metric, the paper's primary segmentation-aware evaluation of subtitle quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BLEURT learned metric used for semantic similarity evaluation of generated subtitles."},{"cited_title":"a e, Joonas Kalda, K \\","cited_arxiv_id":null,"evidence_quote":"Supplies the verbatim Estonian speech dataset used to train the verbatim-transcript baseline for comparison."}],"review_version":1}