{"id":"e315f009-a6fd-49c0-8445-30117c2c7114","arxiv_id":"2412.16474","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A weighted sum of Whisper's language embeddings, optionally refined by a small MLP, improves ASR on unseen languages in zero-shot and fine-tuning settings.","lead":"This paper proposes a method to help Whisper, a multilingual speech recognition model, understand languages it was never trained on. It blends the model's existing language embeddings according to predicted probabilities and shows improved transcription accuracy on rare languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No leave-one-out ASR validation on seen languages means the core embedding-substitution assumption is untested.","rationale":"The reader's weakest assumption was that Whisper's predicted language probabilities are informative enough for the weighted sum to be useful. My concern is closely related but more specific: the paper never validates the weighted-sum embedding as a substitute for a known language embedding, despite having a natural validation protocol available. The leave-one-out training of the predictor (Section III-C) already creates the inputs needed for such a control; reporting only the final ASR on unnamed unseen languages leaves open the possibility that the gains are due to the specific test languages or to fine-tuning effects. This is a genuine gap in the argument, not a disagreement with consensus. The paper's internal logic is coherent, the idea is novel, and the reported gains are substantial, which is why I do not recommend rejecting or moving to unverified. However, the central mechanism should be confirmed by the leave-one-out ASR check before the results are relied upon. The reader's CONDITIONAL verdict remains appropriate; my read does not move it, but it sharpens the required condition: the authors should supply this control and name the unseen languages. No ad hominem is intended; the critique is about evidence completeness, not about the authors' integrity.","tokens_in":6602,"tokens_out":7552,"duration_ms":73046,"concrete_test":"Implement a leave-one-out control on Whisper large-v2 with a test set covering all 99 seen languages (e.g., ML-SUPERB or FLEURS): for each language L, mask out L's embedding, compute the corpus-wise weighted-sum embedding from the other 98 languages (and the predictor output), then decode with (a) the true L embedding, (b) the weighted-sum embedding, and (c) the predicted embedding. Report per-language CER/WER and the median relative change from (a) to (b). If the median CER increase is large, the core substitution assumption fails and the unseen-language gains are not explained by the mechanism. Also compare the predictor on 98- vs 99-language inputs to quantify the train/test distribution shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that a probability-weighted sum of Whisper's 99 language-token embeddings, possibly refined by an MLP, can stand in for the missing embedding of a truly unseen language. This assumption is never tested where it can be tested directly. The predictor is trained with a leave-one-out scheme on seen languages (Section III-C), but the paper does not report the corresponding leave-one-out ASR result: for each seen language, replace its embedding with the weighted-sum embedding of the other 98 languages and measure CER/WER against the true-embedding condition. Without this control, the zero-shot and fine-tuning gains on the unlisted ML-SUPERB 'rare languages' cannot be attributed to the proposed mechanism; the improvement could come from the particular choice of test languages, from LoRA fine-tuning dynamics, or from initialization effects (the parameterized corpus-wise method improves over baseline largely through initialization). The absence of named unseen languages makes the evaluation impossible to reproduce. Additionally, the predictor sees 98-language weighted sums during training but 99-language sums at inference, a distribution shift that is itself unexamined. The paper has useful components, but the key empirical link—that the weighted-sum embedding is a faithful substitute for a true language embedding—is unsupported by direct evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses Whisper's poor ASR performance on languages absent from its 99-language training set. It proposes replacing the language-tag embedding with a probability-weighted sum of all 99 language embeddings, where the probabilities come from Whisper's own language-tag prediction. A second variant trains a small MLP to map such weighted-sum embeddings to the true embedding of a masked seen language, then applies the predicted embedding to unseen languages. Experiments on ML-SUPERB 'rare languages' with Whisper large-v2 compare zero-shot and LoRA fine-tuning settings, reporting that the best fine-tuned predictor method reduces CER from 28.54% to 22.91% and WER from 69.97% to 61.70% relative to the new-tag baseline.","tokens_in":6839,"tokens_out":4247,"duration_ms":36063,"significance":"Assuming the reported gains survive closer scrutiny, the paper offers a lightweight, parameter-efficient alternative to LLM-based conditioning for unseen-language ASR, and the predictor idea provides a way to leverage seen-language structure without additional training data. The method's simplicity and the use of a standard publicly available model (Whisper large-v2) and benchmark (ML-SUPERB) are strengths, and the paper includes an explicit ablation separating fine-tuning-stage and inference-stage application. However, the current evaluation lacks the transparency and controls needed to establish the central substitution claim.","major_comments":[{"comment":"The unseen languages are never named, and no data statistics are provided. The paper says 'We selected Whisper-unseen languages from this dataset and combined them into Whisper-unseen training and test sets' but does not list the languages, the per-language utterance or hour counts, or the train/test split. This prevents reproduction and leaves open the possibility that the gains are driven by a particular language or by the test-set size. Please provide the complete list of unseen languages and per-language data statistics.","section":"Section III (Data) and Table I"},{"comment":"No leave-one-out ASR validation is reported. The predictor is trained by masking one seen language and predicting its embedding from a weighted sum of the remaining seen languages, yet no ASR result is given for seen languages under this masking. A direct control would replace a seen language's embedding with the weighted-sum embedding (with that language masked) and measure CER/WER against the true-embedding condition. Without this, the claim that the weighted-sum embedding is a faithful substitute for a missing language embedding is not tested, and the observed fine-tuning gains cannot be cleanly attributed to the proposed mechanism.","section":"Section III-C and Table I"},{"comment":"There is a train/inference distribution shift in the predictor input. The predictor is trained on weighted sums over 98 languages (one masked language excluded), but at inference it is applied to weighted sums over all 99 seen languages for an unseen language. This shift is not discussed or evaluated. Please either analyze its effect or train and validate the predictor under the same input distribution used at inference.","section":"Section III-C"},{"comment":"The baseline method is described as adding a new language tag and 'fine-tuning the entire model,' but all fine-tuning experiments use LoRA with rank 32 and alpha 64. If the baseline is not actually fully fine-tuned, the comparison may understate the baseline; if the baseline is LoRA-tuned, the description should be corrected. This inconsistency directly affects the headline fine-tuning comparison.","section":"Section II-A versus Section III-B"},{"comment":"The zero-shot CER/WER numbers are presented without error bars or significance information, and the paper does not state whether they come from a single pass or multiple runs. Given the large variance typical of zero-shot ASR on low-resource languages, the claimed 22% CER reduction in Section IV-A needs uncertainty quantification before it can be evaluated.","section":"Table I, zero-shot rows"}],"minor_comments":[{"comment":"WER values above 100% (e.g., 139.33% in the zero-shot default row) are possible with insertion-heavy outputs, but they should be explained so readers do not doubt the metric.","section":"Table I"},{"comment":"The softmax formula for the language-tag probability is typeset incorrectly: the denominator appears as 'P99 j=1 exp(f (lj|x, sot))' separated from the fraction. Please fix the equation.","section":"Section II-A"},{"comment":"The figure caption says yellow boxes indicate individual language embeddings, but the text does not clearly connect the left and right parts of the figure when describing the predictor input. Adding explicit labels for WsEmb and PredEmb would improve clarity.","section":"Figure 1"},{"comment":"The statement that Whisper large-v3 is 'almost identical to large-v2, differing only by the addition of a new language and a change in spectrogram input size' is imprecise and unnecessary for the paper; please either substantiate it or remove it.","section":"Section III (Model)"},{"comment":"The predictor configuration is said to be 'determined through extensive hyperparameter selection,' but the search space and the selection criterion (e.g., validation MSE versus downstream CER) are not reported. Please provide these details.","section":"Section III-C"},{"comment":"The term 'rare languages' is used without definition. Because ML-SUPERB contains many languages, please clarify what makes these languages rare and unseen for Whisper.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the central idea is plausible, but the missing language list, the absent leave-one-out ASR control, and the unquantified zero-shot results are load-bearing issues that require author response. I would not reject if the authors can supply the missing transparency and controls; the current form is not yet reproducible enough for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper proposes a lightweight way to adapt Whisper to languages it was not trained on, and the reported gains are large enough to matter. The core idea is to replace the missing language-tag embedding with a probability-weighted average of the 99 existing embeddings, using Whisper's own predicted language distribution, and optionally refine that average with a small MLP trained on seen languages under leave-one-out masking. That is a fresh mechanism, not just a rehashed feature interpolation, and the paper does a decent job of comparing several variants with ablations.\n\nWhat the paper does well: the method is genuinely lightweight, the experiments cover zero-shot and fine-tuning with LoRA, and the fine-tuning results come with error bars. The best predictor-based method brings CER down from 28.54% to 22.91% on the ML-SUPERB rare-language set, which is a substantial improvement if it holds up.\n\nThe soft spots are mostly about evidence, not about the idea. The most important missing piece is the direct control experiment that would test the central assumption: for seen languages, replace the true language embedding with the weighted-sum or predictor-based embedding and measure ASR. The predictor is trained with leave-one-out masking, but the paper never reports the corresponding ASR result. Without that, the gains on unseen languages could come from the particular choice of test languages or from LoRA dynamics rather than from the embedding substitution itself. Second, the unseen languages are never named, which makes the evaluation impossible to reproduce. Third, the zero-shot numbers have no error bars, and the paper does not describe a validation protocol for selecting among the many variants, even though it mentions 'extensive hyperparameter selection.' The 98-language-versus-99-language distribution shift for the predictor input is a minor issue but also goes unexamined. No code or data are released, which adds to the reproducibility concern.\n\nBottom line: the paper is for the multilingual ASR and low-resource speech crowd, and it deserves a serious referee. It is not ready as is, though; it should go to review with a request for the leave-one-out ASR experiment, named languages, and a clearer model-selection story. I would be moderately optimistic that the core finding survives that check.","headline":"A genuinely new way to condition Whisper on unseen languages, with large reported gains but a missing control experiment that the paper should run before the central claim is fully trusted.","tokens_in":7332,"tokens_out":4216,"would_cite":true,"duration_ms":34797,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an unseen language can be added to Whisper by feeding its decoder a probability-weighted sum of the embeddings of the 99 known language tags, and that a small MLP refinement of that embedding yields the largest gains.","keywords":["multilingual ASR","unseen languages","Whisper","language embeddings","zero-shot speech recognition","low-resource languages","language identification","fine-tuning"],"falsifier":"For a held-out seen language, mask its embedding, compute the corpus-wise weighted-sum embedding from the other 98 languages, and measure its cosine similarity to the true embedding; if this similarity is near zero or no better than random across many languages, the weighted sum carries no recoverable language identity and the predictor has no signal. A second check: pick an unseen language for which Whisper's predicted language distribution is near-uniform and verify whether the method fails to reduce CER relative to the default argmax tag.","tokens_in":6415,"feed_emoji":"🎙️","tokens_out":9070,"duration_ms":67242,"temperature":0.7,"pith_summary":"This paper asks how to make Whisper, a multilingual ASR model trained on 99 languages, recognize an additional language it has never seen. The proposed answer is to avoid inventing a new language tag: instead, take Whisper's own predicted probabilities over its 99 known language tags and use them to compute a weighted sum of the corresponding language embeddings, which then stands in for the unseen language's embedding. A second, predictor-based variant trains a small two-layer MLP to turn that weighted-sum embedding into a closer approximation of a true language embedding. On rare languages from the ML-SUPERB benchmark, both approaches reduce character and word error rates in zero-shot and fine-tuned settings; the best fine-tuned configuration lowers CER from 28.54% to 22.91%. If the claim holds, extending an ASR system to a new language becomes a lightweight embedding computation instead of a costly retraining effort.","feed_headline":"Blended language embeddings cut errors on unseen languages","feed_subtitle":"Weighting Whisper's 99 known language tags by predicted probabilities lowers character error rate from 28.5% to 22.9%.","key_machinery":"The central object is the weighted-sum language embedding, $\\mathrm{WsEmb} = \\sum_{j=1}^{99} P(l_j \\mid x, \\mathrm{sot})\\, \\mathrm{Emb}_j$, formed from the softmax of Whisper's language-tag logits and the embedding table of its 99 known languages. It converts the discrete choice of a language tag into a continuous interpolation, so the decoder can draw on whatever subset of known languages the audio actually resembles. The second piece of machinery is a two-layer MLP predictor, with hidden size 1280 and MSE loss, trained by masking one seen language at a time and recovering its true embedding from the weighted sum of the remaining languages; at test time it refines the weighted-sum embedding for an unseen language. These objects carry the argument because they put all of the adaptation into the language-conditioning pathway rather than into new parameters or an external model.","core_discovery":"The paper's central claim is that an unseen language can be represented, for Whisper's decoder, as a probability-weighted blend of the 99 embeddings Whisper already has. Given an utterance $x$ of an unseen language, Whisper's own next-token predictor supplies a distribution $P(l_i \\mid x, \\mathrm{sot})$ over language tags, and the method sets $\\mathrm{WsEmb} = \\sum_{j=1}^{99} P(l_j \\mid x, \\mathrm{sot})\\, \\mathrm{Emb}_j$, using this vector in place of the usual language-tag embedding. A corpus-wise variant averages the per-utterance distributions to give one embedding per language, and a parameterized variant initializes a trainable embedding with that average. The predictor-based variant trains an MLP, on seen languages, to map a weighted-sum embedding (computed with the target language masked out) to the target's true embedding; the authors interpret its success as evidence that the weighted sum captures real linguistic relationships. In their experiments, the predictor with the corpus-wise embedding performs best, and applying the weighted embedding during both fine-tuning and inference helps more than applying it only at inference.","pith_inferences":["If the masked-language reconstruction experiment generalizes, the 99 language embeddings form a space in which unseen languages sit approximately inside the convex hull of seen ones; one could test this directly by checking whether the weighted-sum embeddings of related languages cluster by language family.","A testable extension is to apply the same recipe to other token-conditioned speech models and to Whisper large-v3; consistent gains would indicate that embedding interpolation is a general property of language-conditioned decoders, while failures would bound the effect to the specific model tested here.","The method's usefulness should track the quality of Whisper's language identification: an unseen language whose utterances yield near-uniform language probabilities should show smaller gains, so measuring the gain against the entropy of the predicted distribution would be a useful diagnostic.","Since the utterance-wise variant produces a distinct embedding per utterance, it may extend naturally to code-switched speech, where the effective language changes within one recording, though the paper does not test this."],"forward_implications":["A new language can be added to Whisper without training a new language-tag embedding: the weighted sum over existing tags works in zero-shot inference, cutting CER by up to roughly 22% relative in the paper's rare-language evaluation.","Fine-tuning with the weighted embedding, whether fixed, trainable, or MLP-refined, beats fine-tuning a newly added baseline tag, and using the replacement at both fine-tuning and inference gives larger gains than inference-only application.","The predictor results imply that a minimal MLP, trained only on the 99 seen languages, can produce useful embeddings for unseen languages, so no large language model is required for this adaptation.","Because the method changes only the language embedding, it composes with standard efficient fine-tuning such as LoRA, which the paper uses throughout its fine-tuning experiments."],"supporting_citations":[{"why":"Supplies the Whisper model and its 99 language-tag embeddings, the object the paper modifies.","marker":"[7]"},{"why":"Provides the ML-SUPERB rare-language data from which the Whisper-unseen training and test sets are built.","marker":"[13]"},{"why":"Supplies LoRA, the efficient fine-tuning method used in all fine-tuning experiments.","marker":"[14]"},{"why":"Recent work that tackles unseen languages by training an additional LLM, the costly baseline the paper contrasts with its lightweight embeddings.","marker":"[12]"},{"why":"Supports the premise that language tags act as prompts that guide Whisper's decoder, the mechanism the weighted-sum method operates on.","marker":"[8]"}],"fun_headline_variants":["Whisper learns unseen languages by blending known ones","Blend 99 language tags to teach Whisper new tongues","Weighted language embeddings unlock new tongues for Whisper","Whisper's next-token blend lets it hear never-trained languages","Teaching Whisper new languages via weighted tag blends"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that Whisper's predicted language-tag probabilities for an unseen language carry real signal; if those probabilities are near-uniform or dominated by irrelevant languages, the weighted-sum embedding is a blur that does not help recognition.","fun_headline_variants_meta":{"raw":{"variants":["Whisper learns unseen languages by blending known ones","Blend 99 language tags to teach Whisper new tongues","Weighted language embeddings unlock new tongues for Whisper","Whisper's next-token blend lets it hear never-trained languages","Teaching Whisper new languages via weighted tag blends"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2950,"prompt_tokens":972,"completion_tokens":1978,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1897}},"tokens_in":588,"tokens_out":1978,"duration_ms":11889,"temperature":1.0,"reasoning_tokens":1897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:32:27.219555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a held-out seen language, mask its embedding, compute the corpus-wise weighted-sum embedding from the other 98 languages, and measure its cosine similarity to the true embedding; if this similarity is near zero or no better than random across many languages, the weighted sum carries no recoverable language identity and the predictor has no signal. A second check: pick an unseen language for which Whisper's predicted language distribution is near-uniform and verify whether the method fails to reduce CER relative to the default argmax tag.","supporting_citations":[{"cited_title":"ML- SUPERB: Multilingual Speech Universal PERformance Benchmark,","cited_arxiv_id":null,"evidence_quote":"Provides the ML-SUPERB rare-language data from which the Whisper-unseen training and test sets are built."}],"review_version":1}