{"id":"a75c99af-357b-41f6-b042-b7865838366c","arxiv_id":"2509.04491","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using subtitles as prompts and pseudo transcripts as targets, with a Gini-based attention weighting, refines Whisper's transcripts on low-resource Flemish TV speech without verbatim labels.","lead":"This paper shows that TV subtitles, normally too noisy to serve as ASR training text, can be used as prompts to guide a Whisper speech model in refining its own transcripts without manual labels. The method improves word error rates on Flemish broadcast TV from 13.07% to 11.37% with prompt-based fine-tuning, and to 10.34% with iterative refinement and a Gini-based attention weighting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 10.34% WER may reflect test-set selection: WA layer choice and iteration count are tuned on subs-annot, the same set used for evaluation; no held-out dev split is used.","rationale":"The paper's contribution is empirical: the central claim is that subtitle prompts plus Gini-weighted attention produce progressively better transcripts without verbatim labels. The most load-bearing weakness is not the absence of a mechanistic analysis of how the model learns to treat prompts as guidance (the reader's weakest_assumption), but the use of the only verbatim-labeled set both for evaluation and for choosing the WA configuration and the number of iterations. This is explicitly admitted in Section V-B and V-C. The five-fold consistency reported for the WA layer choice is a mitigating factor, as is the rare/OOV word analysis in Table II, which supports the claim that prompts help on low-frequency content. However, the reported absolute gains are small, and without a held-out split one cannot rule out that part of the apparent iterative improvement is selection artifact. The conditionality of the reader's verdict is therefore appropriate, and no verdict change is needed. The proposed nested hold-out test would settle whether the 10.34% figure survives honest model selection.","tokens_in":7475,"tokens_out":6508,"duration_ms":78064,"concrete_test":"Construct a proper held-out split: hold out one of the five subs-annot folds as a final test. On the remaining four folds only, select (i) the WA configuration (individual layer vs all layers) and (ii) the stopping iteration from {1,2,3}. Then evaluate the selected configuration on the held-out fold, and also evaluate the NP self-trained baseline on the same fold. Repeat across folds (nested CV). If the average held-out WER is no longer significantly below the NP baseline, or exceeds 10.34% by more than ~0.3-0.5% absolute, the headline improvement should be reported as conditional on test-set selection rather than as a fixed property of the method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The numerical claim that matters—SP fine-tuning plus WA inference reaches 10.34% WER (Table IV) and that WERs 'consistently decrease across iterations'—rests on choices made with access to the evaluation set. Section V-B explicitly says: 'only the test set (subs-annot) contains verbatim transcripts, we divide this set into five folds to simulate different test scenarios and ensure robust layer selection.' The all-layers WA configuration is then applied to the full subs-annot set, and the iteration count in Table IV is chosen after observing the test WER ('training is halted after three updates', Section V-C). Because there is no held-out development split, the reported improvements—11.54→10.34, with iter2→iter3 only 0.32% absolute—could be inflated by selection on the test set. The method might still work, but the central empirical evidence is not cleanly identified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a weakly supervised ASR fine-tuning method that uses TV subtitles as decoder prompts rather than as direct training targets. Pseudo-transcripts generated by pre-trained Whisper serve as the training labels, and the model is fine-tuned with a decoder prompt of the form <|sop|>subtitles<|sot|><|nl|><|transcribe|><|notimestamps|>. The training is iterated, with pseudo-labels refreshed each round. At inference, a weighted-attention (WA) mechanism based on Gini coefficients computed from cross-attention weights re-weights the subtitle prompt keys/values in self-attention. Experiments on 760 hours of Flemish broadcast audio with a 6-hour manually annotated test set (subs-annot) report WER improvements from 18.75%/13.07% (pre-trained medium/large) to 11.54%/11.37% after one SP fine-tuning iteration, 11.02% with WA inference, and 10.34% after three iterative updates. The central claim is that subtitles as prompts, combined with pseudo-transcript targets, enable iterative transcript refinement without verbatim labels.","tokens_in":7722,"tokens_out":4651,"duration_ms":55089,"significance":"If valid, the method is practically useful for low-resource and under-represented domains where verbatim transcripts are unavailable but TV subtitles exist. The idea of converting subtitle mismatch into a prompt-conditioning problem is novel and clearly presented, and the reported rare-word and OOV error reductions are plausible and potentially valuable. The paper also explicitly discusses the relation to and difference from self-training, and it provides a concrete inference-time mechanism (Gini-weighted attention) that is inexpensive and model-agnostic. However, the empirical evidence for the headline numbers is weakened by the use of the evaluation set for configuration and stopping decisions, as detailed in the major comments. Because that issue is fixable with a held-out development split, the contribution is defensible pending revision.","major_comments":[{"comment":"The headline WERs are obtained after using the evaluation set for model selection. Section V-B states that because \"only the test set (subs-annot) contains verbatim transcripts, we divide this set into five folds to simulate different test scenarios and ensure robust layer selection\"; the all-layers WA configuration is then applied to the full subs-annot set. The number of iterations is also chosen after observing test WER (\"training is halted after three updates\", Section V-C), and Table IV reports the resulting 10.34% WER. Since WA layer choice and iteration count are selected on the same set used to report the final accuracy, the monotonic decrease 11.54→10.82→10.52 and the inference WERs 11.02→10.66→10.34 are optimistically biased; the iter2→iter3 gain of 0.32% absolute may be noise. Fix: hold out a development portion of subs-annot (or use nested cross-validation) for layer/iteratio","section":"V-B/V-C, Table IV"},{"comment":"The mechanism underlying SP is not tested. The paper argues that pre-trained Whisper truncates with subtitle prompts, while SP fine-tuning learns to use subtitles as context; however, no analysis is provided to show that the model actually conditions on subtitle content rather than exploiting the prefix structure or copying subtitle tokens. The rWER/oWER improvements (Table II) are suggestive, but a control with shuffled or corrupted subtitle prompts (e.g., random tokens from the same distribution, or reversed subtitles) would test whether the content of the prompt matters. Without such a control, the claim that \"subtitles provide additional information to guide generation\" (Section V-A) remains an interpretation of the WER difference rather than a demonstrated property of the trained model.","section":"V-A, III-A"}],"minor_comments":[{"comment":"The sentence \"6 hours of speech (approximately 2,600 utterances) from each genre\" is ambiguous: is it 6 hours per genre (24 hours total) or 6 hours overall divided across genres? Please clarify.","section":"IV-A"},{"comment":"In Eq. (2), G is described as the Gini coefficient over the prompt sequence, but if G is a vector, the product G Kp should be defined as element-wise (per-token) scaling of the key vectors, not standard matrix multiplication. Please clarify the notation and the exact dimension of G.","section":"III-B, Eq. (2)"},{"comment":"Training details are incomplete: the number of epochs, early stopping criterion, and the pseudo-transcript length filter threshold are not specified. These are needed for reproducibility.","section":"IV-B"},{"comment":"No confidence intervals or significance tests are reported. Given the small differences in Table IV (e.g., 0.32% between iter2 and iter3), error bars or a paired test would help assess whether the improvements are reliable.","section":"V-A, V-B"},{"comment":"Typos and formatting issues: \"frame work\" in the Introduction, \"W A\" and \"SP fine-tuning\" spacing inconsistencies, \"I NTRODUCTION\" heading, and \"subs-annot set\" redundancy. Figure 2 caption should describe the colors and the meaning of the folds.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The principal obstacle is the selection-on-test issue. The authors appear to have used the five-fold heuristic as a robustness check, but it ends up leaking information from the test set into both the WA configuration and the stopping decision. If the authors re-run with a proper development/test split and the iterative improvement still holds, I would be willing to support acceptance. I do not see evidence of bad faith; the issue is an experimental-design oversight."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely different way to use TV subtitles for ASR—as decoder prompts instead of training targets—plus a Gini-weighted attention trick at inference. The core result, that prompt-conditioned self-training beats plain self-training on Flemish broadcast data, is plausible and probably real. But the headline 10.34% WER is not cleanly identified because the evaluation folds were used to choose both the layer configuration and the stopping iteration.\n\nWhat's new and good: the idea that subtitles can constrain decoding without being verbatim labels is well motivated, and the authors' explanation of self-training failure—deletions at utterance starts, reinforced by the large model—is sensible. Using first-layer cross-attention concentration (Gini) to reweight prompt keys and values during inference is original, and the ablation against max- and entropy-based weights gives it some credibility. They also honestly report that directly prompting a pre-trained Whisper with subtitles produces truncated output, and they don't pretend the baseline is strong.\n\nThe biggest problem is experimental hygiene. Section V-B explicitly says they split the only verbatim test set (subs-annot) into folds to \"ensure robust layer selection,\" and the all-layers configuration is then applied to the full set. Table IV then stops at three iterations after observing the test WER. That makes the improvement from 11.54% to 10.34% partly a selection artifact. The honest claim is \"we can reach low 10s on this test set with test-set-informed choices,\" not \"the method robustly improves WER by 1.2 absolute.\" An independent dev split or a pre-specified configuration would fix this.\n\nOther soft spots: there is no baseline that simply uses subtitles as training targets, which is the natural alternative given the literature they cite; training details are thin (epochs, early stopping); no code or data is released; and the WA gain itself is modest (about 0.5 absolute WER) and only demonstrated on one model size.\n\nWho this is for: ASR practitioners working on low-resource or under-resourced languages that have subtitle corpora. The paper deserves a serious referee—the idea is fresh, the flaws are fixable with better evaluation discipline, and the problem is practically important.\n\nRecommendation: send it out, but the reviewers should push for a held-out dev set and a subtitle-as-target baseline before acceptance.","headline":"Subtitle-prompted self-training is a genuinely fresh idea worth refereeing, but the headline WER is tainted by test-set-selected hyperparameters.","tokens_in":8170,"tokens_out":1715,"would_cite":true,"duration_ms":21646,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TV subtitles, used as prompts, let Whisper refine its own transcripts with no verbatim labels.","keywords":["weakly supervised ASR","subtitle prompting","Whisper fine-tuning","pseudo-transcripts","Gini coefficient","weighted attention","Flemish broadcast","rare words"],"falsifier":"Train the same model on the same data with the subtitle prompt format but replace subtitle text with random or scrambled word sequences; if WER does not degrade relative to no-prompt fine-tuning, the claimed informative role of subtitles as prompts is not supported. Conversely, if verbatim transcripts used as prompts do not outperform subtitle prompts beyond the noise level of the subtitles themselves, the contextual value attributed to subtitles would be cast in doubt.","tokens_in":7398,"feed_emoji":"🎙️","tokens_out":1427,"duration_ms":16935,"temperature":0.7,"pith_summary":"The paper tries to establish that misaligned TV subtitles, which are too noisy to serve as training targets, can instead be used as contextual prompts while pseudo-transcripts generated by Whisper serve as targets, letting the model iteratively refine its own output. On 760 hours of Flemish broadcast audio, this recipe lowers word error rate from 18.75% to 10.34% on a verbatim-annotated test set, with no manual transcriptions used in training. The authors also introduce an inference-time weighted-attention mechanism that uses Gini coefficients of cross-attention to emphasize subtitle tokens relevant to the speech. If the method is right, it turns an abundant, cheaply available resource — subtitles — into a lever for improving ASR in low-resource domains.","feed_headline":"Subtitles as prompts cut ASR error rate to 10.34% without verbatim labels","feed_subtitle":"Whisper fine-tuned on 760 hours of Flemish TV leverages subtitle context to refine its own transcripts, no manual transcriptions needed.","key_machinery":"Subtitle prompting (SP) training: the decoder input is structured as <|sop|>subtitles<|sot|><|nl|><|transcribe|><|notimestamps|>, with pseudo-transcripts (generated by Whisper-large-v3) as targets, enabling weakly supervised fine-tuning. The inference-time weighted attention (WA) mechanism computes a Gini coefficient for each prompt token from the first cross-attention layer's attention distribution over speech frames; concentrated attention yields high Gini values, and these weights scale the prompt's key and value matrices before self-attention, emphasizing speech-relevant subtitle tokens and down-weighting distractors.","core_discovery":"The paper claims that a Whisper ASR model can be fine-tuned without verbatim transcripts by prepending subtitle text to the decoder input as a prompt (format <|sop|>subtitles<|sot|><|nl|><|transcribe|><|notimestamps|>), using the model's own pseudo-transcripts as targets, and iterating. Directly prompting a pre-trained Whisper with subtitles produces truncated or empty output, but fine-tuning with this prompt format teaches the model to treat subtitles as guidance rather than text to repeat or truncate against. Adding a weighted attention scheme at inference — scaling prompt key/value vectors by Gini coefficients computed from first-layer cross-attention weights — further improves WER. Acros","pith_inferences":["The method likely generalizes beyond Flemish: any language with available subtitles and a pre-trained generative ASR could use the same prompt-based weak supervision recipe.","The Gini-weighted attention mechanism could be applied to other prompt-conditioned generation tasks where prompts are noisy or loosely aligned with inputs, not just ASR.","A direct comparison against verbatim-supervised fine-tuning on the same data would clarify how close subtitle-prompting gets to fully supervised performance.","The mechanism of why fine-tuning teaches the model to separate prompt content from generation targets remains unexplained; ablation studies isolating prompt-format effects would sharpen the claim."],"forward_implications":["Subtitle-prompted fine-tuning turns abundant, loosely aligned subtitles into a usable signal for ASR improvement without verbatim labels.","Iterative retraining with updated pseudo-transcripts converges: WER decreases from 11.54% to 10.52% over three SP iterations, and further to 10.34% with WA inference.","The method especially helps rare and out-of-vocabulary words: rWER drops from 31.04% to 24.63% and oWER from 74.94% to 70.22% for the medium model with SP.","The Gini-weighted attention scheme consistently outperforms max- and entropy-based weighting, avoiding hallucinations and refining named-entity spelling.","Self-training without prompts harms the large model (WER up to 21.49%), while adding subtitle prompts reverses that degradation, showing prompts mitigate error propagation."],"supporting_citations":[{"why":"Whisper is the foundation model being fine-tuned; supplies the architecture, decoder prompting format, and initial zero-shot baselines.","marker":"[5]"},{"why":"Establishes that Whisper's decoder can be prompted for zero-shot tasks, motivating the use of prefix prompts.","marker":"[13]"},{"why":"Shows prompt-based adaptation of Whisper for spoken language understanding, supporting the prompt-conditioning approach.","marker":"[14]"},{"why":"Demonstrates prompt-conditioning fine-tuning of Whisper for domain-sensitive ASR, the closest prior work this method extends.","marker":"[15]"},{"why":"Uses prompt tuning to adapt Whisper to target-speaker ASR, supporting the feasibility of conditioning on extra text.","marker":"[16]"},{"why":"Documents Whisper hallucination behavior, which motivates the filtering of pseudo-transcripts and the WA mechanism design.","marker":"[18]"},{"why":"Provides the self-training survey that frames the error-propagation problem the subtitle prompts are designed to mitigate.","marker":"[19]"},{"why":"Supplies the Gini coefficient formula used to quantify attention concentration in the WA mechanism.","marker":"[20]"}],"fun_headline_variants":["Subtitles as smart prompts cut ASR error to 10.34%","Whisper fine-tuned with subtitle prompts drops WER to 10.34%","TV subtitles guide ASR self-refinement to 10.34% WER","Subtitle-prompted ASR beats verbatim labeling at 10.34% error"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Fine-tuning the decoder with the subtitle-prompt format teaches the model to treat subtitle text as guidance rather than text to copy or truncate, even though the paper gives no analysis of how this separation is learned.","fun_headline_variants_meta":{"raw":{"variants":["Subtitles as smart prompts cut ASR error to 10.34%","Whisper fine-tuned with subtitle prompts drops WER to 10.34%","TV subtitles guide ASR self-refinement to 10.34% WER","Subtitle-prompted ASR beats verbatim labeling at 10.34% error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3518,"prompt_tokens":700,"completion_tokens":2818,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":2727}},"tokens_in":444,"tokens_out":2818,"duration_ms":23111,"temperature":1.0,"reasoning_tokens":2727,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:34:35.277718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on the same data with the subtitle prompt format but replace subtitle text with random or scrambled word sequences; if WER does not degrade relative to no-prompt fine-tuning, the claimed informative role of subtitles as prompts is not supported. Conversely, if verbatim transcripts used as prompts do not outperform subtitle prompts beyond the noise level of the subtitles themselves, the contextual value attributed to subtitles would be cast in doubt.","supporting_citations":[{"cited_title":"Robust speech recognition via large- scale weak supervision,","cited_arxiv_id":null,"evidence_quote":"Whisper is the foundation model being fine-tuned; supplies the architecture, decoder prompting format, and initial zero-shot baselines."},{"cited_title":"Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization","cited_arxiv_id":"2305.11095","evidence_quote":"Establishes that Whisper's decoder can be prompted for zero-shot tasks, motivating the use of prefix prompts."},{"cited_title":"Prompting whisper for qa-driven zero-shot end-to-end spoken language understanding,","cited_arxiv_id":null,"evidence_quote":"Shows prompt-based adaptation of Whisper for spoken language understanding, supporting the prompt-conditioning approach."},{"cited_title":"Zero-shot domain-sensitive speech recognition with prompt-conditioning fine-tuning,","cited_arxiv_id":null,"evidence_quote":"Demonstrates prompt-conditioning fine-tuning of Whisper for domain-sensitive ASR, the closest prior work this method extends."},{"cited_title":"Extending whisper with prompt tuning to target-speaker asr,","cited_arxiv_id":null,"evidence_quote":"Uses prompt tuning to adapt Whisper to target-speaker ASR, supporting the feasibility of conditioning on extra text."},{"cited_title":"Careless whisper: Speech-to-text hallucination harms,","cited_arxiv_id":null,"evidence_quote":"Documents Whisper hallucination behavior, which motivates the filtering of pseudo-transcripts and the WA mechanism design."},{"cited_title":"Self-training: A survey,","cited_arxiv_id":null,"evidence_quote":"Provides the self-training survey that frames the error-propagation problem the subtitle prompts are designed to mitigate."},{"cited_title":"A formula for the gini coefficient,","cited_arxiv_id":null,"evidence_quote":"Supplies the Gini coefficient formula used to quantify attention concentration in the WA mechanism."}],"review_version":1}