{"id":"a1794094-0019-4b87-a2b3-3b8985ec6b8f","arxiv_id":"2501.07875","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Embedding Layer Surgery and Task-wise Beam Search reduce catastrophic forgetting when adding new languages to Whisper, lowering old-language AWER from 14.2% to 11.9% versus Experience Replay.","lead":"A team at NTU applied a continual-learning trick to Whisper: give each new language its own copy of the decoder's token embeddings, and pick the right copy during decoding using a language-aware beam search. On 10 unseen languages from Common Voice, this cut average word error rate for old languages from 14.2% to 11.9% versus experience replay without hurting new-language accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inference behavior for tokens absent from the per-language embedding table is unspecified; reported new-language WERs depend on an unstated OOV/fallback mechanism.","rationale":"The reader's weakest assumption identifies the same gap: the per-language lookup table contains only tokens seen in the 10-hour training set, and the paper never defines inference behavior for tokens outside that set. This is the most load-bearing concern because it directly affects the reproducibility of the reported new-language WERs and the claimed isolation of old-language embeddings. If the implementation falls back to the original shared embeddings for OOV tokens, then old-language semantics are still partially updated or shared, undermining the Embedding Layer Surgery rationale; if it does not, decoding would fail on any OOV token, and the reported WERs would be difficult to explain. A single instrumented decoding run on the released code would settle the question. The concern is addressable rather than fatal, so the existing CONDITIONAL verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":9129,"tokens_out":8643,"duration_ms":93603,"concrete_test":"Instrument the decoding implementation for the ten new-language test sets: for each decoded sequence, record how many generated token IDs are not present in that language's A_hat column set, and compute the fraction of reference tokens in the 1h Common Voice test sets that are OOV relative to the 10h training vocabulary. If both fractions are near zero, the concern is vacuous; if they are non-negligible, the fallback path must be identified and rerun with the fallback disabled to measure its effect on old-language AWER.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 defines a separate per-language lookup table A_hat ∈ R^{E×J} with J < U, and Table 1 shows J/U ranges from 0.4% to 7.9% across the ten new languages, but the paper never states what happens during inference when a new-language utterance contains a token outside that set. This is load-bearing for the central claim because the abstract promises that the method does not compromise AWER on the unseen languages, and Section 3 reports new-language WERs comparable to Experience Replay, which uses the full U-token table. For those numbers to be reproducible, one of three things must be true: (1) the test token sets are nearly fully contained in the respective A_hat sets, which is not reported; (2) out-of-set tokens are mapped to a fallback embedding, likely from the original shared table, in which case the claimed isolation of old-language semantics is broken for those tokens; or (3) such tokens are suppressed during decoding, which would inflate WER and contradict the 'without compromising' claim. The paper also leaves unspecified how new-language ID tokens are added to the shared special-token block while keeping old-language embeddings frozen. The central result is therefore conditional on an undeclared engineering choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses catastrophic forgetting in language-agnostic massively multilingual ASR when continually adapting Whisper to unseen languages. It proposes Embedding Layer Surgery, where separate per-language token embedding lookup tables are created and split into a language-shared special-token part and a language-specific vocabulary part, and Task-wise Beam Search, which keeps the top-N language hypotheses during decoding and selects by ASR score. Experiments on Common Voice subsets with 10 unseen and 10 seen languages report that the method reduces the average WER of pre-trained languages from 14.2% to 11.9% compared with Experience Replay, without degrading WER on the newly added languages.","tokens_in":9404,"tokens_out":4347,"duration_ms":44408,"significance":"If the reported results are reliable, the paper makes a useful contribution to continual learning for multilingual ASR: it identifies the token embedding lookup table as a source of catastrophic forgetting in language-agnostic models, proposes a simple architectural remedy, and augments decoding to correct language-identification errors. The use of the CL-MASR benchmark, comparison with several standard CL baselines, and ablation studies are strengths. However, the central quantitative claim is currently conditional on an unspecified out-of-vocabulary mechanism and on the absence of statistical verification, so the significance cannot be fully assessed without revision.","major_comments":[{"comment":"The per-language embedding table A_hat in R^{E x J} is constructed from tokens appearing in the 10-hour training set, and Table 1 reports J/U as low as 0.4% and at most 7.9% of Whisper's vocabulary for the ten new languages. The paper never states what happens during inference when a new-language utterance contains a token outside A_hat. This is load-bearing for the 'without compromising the AWER of the unseen languages' claim: the new-language WERs in Table 2 and Figure 4 depend on whether such tokens are covered by the test set, mapped to a fallback embedding from the original shared table, or suppressed during decoding. Each option changes the meaning of the result and its effect on old-language isolation. Please report token coverage on the test sets, specify the fallback mechanism, or justify that out-of-table tokens cannot occur.","section":"Section 2.3, Table 1"},{"comment":"All experimental results appear to be single runs: no repeated seeds, no error bars, and no significance tests are reported. The headline improvement in Figure 4, reducing AWER from 14.2% to 11.9% for whisper-large-v2, is a 2.3-point difference over Experience Replay, which could plausibly lie within run-to-run variance. In addition, the paper states that hyper-parameters are swept 'for all methods' but does not report the sweep ranges, the selection criterion, or whether selection used the validation set rather than the test set. The task-wise beam search thresholds N=2, Mlen=5, and Moverlap=3 are fixed without sensitivity analysis. Please provide seeds, variance or significance tests, the complete hyperparameter search protocol, and an ablation of the decoding thresholds.","section":"Section 3.1, Section 3.2, Figure 4"},{"comment":"The stability condition for Task-wise Beam Search is underspecified. The text says that decoding is disabled when a path has fewer than Mlen words or more than Moverlap overlapped words with other paths, but it does not say what decoding is performed instead: does the system fall back to the single best LID path, use the original beam search, or suppress the hypothesis? This affects the reproducibility of all language-agnostic results, including Figure 4 and Table 3, and should be stated precisely.","section":"Section 2.5"},{"comment":"The construction of the split lookup table is not fully specified. It is not explained how new language ID tokens are inserted into the language-shared special token block while the original language ID embeddings for the pre-trained languages are kept frozen, nor how the model selects which per-language vocabulary table A_hat to use during the language-agnostic LID phase. Since the language-agnostic claim depends on this mechanism, please provide the exact table construction, the embedding initializations, and the selection rule.","section":"Section 2.4"}],"minor_comments":[{"comment":"The index terms include 'speech emotion recognition (SER)' and 'instruction-tuning', which are not topics of this paper and should be corrected.","section":"Abstract and Section 1, Index Terms"},{"comment":"The entries in the 'avg' column, such as '41.0−00.0%', appear malformed and are not explained; please define whether the column reports average WER or relative improvement, and clarify the notation.","section":"Table 2"},{"comment":"Figure 4 plots AWER over time for whisper-large-v2 but does not give numerical values in the text; please report the exact AWER numbers for all ten languages at the end of sequential adaptation.","section":"Section 3.2, Figure 4"},{"comment":"The ablations in Tables 3 and 4 are performed only for whisper-small on the language pair Esperanto/English, so it is unclear whether the conclusions transfer to whisper-large-v2 and to the full ten-language setting; please add at least one large-v2 ablation or discuss the limitation.","section":"Section 3.3"},{"comment":"There are several typos and formatting issues, including 'V oice' in the abstract, 'langauge' and 'becuase' in Section 2.4, 'forgeting' in Table 2, 'Germen' in Table 2 header, and 'topN = 2scoring' in Figure 3; these should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The proposed method is interesting and the experimental setup follows a standard benchmark, but the central quantitative claim is not yet supported: the unspecified out-of-vocabulary handling and the lack of statistical verification are substantial. For a journal venue, I would expect either repeated-seed experiments with significance tests or a clear explanation of why single-run results are sufficient in this setting, plus a precise specification of the inference-time embedding selection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful idea that deserves a proper review, but the evidence in the current draft is not strong enough to take the headline 14.2% to 11.9% at face value.\n\nWhat is new: most MMASR continual learning work tunes weights without touching the decoder's token embeddings. The paper makes a good case that the shared embedding table is a major source of forgetting, then proposes per-language embedding copies, split into language-shared special tokens and language-specific vocabulary, plus a task-wise beam search that generates hypotheses in the top-N languages and picks the best ASR score. That is a sensible, well-motivated mechanism. The ablation study separates the two contributions, and the LID confusion matrix supports the claimed mechanism for the beam-search gain. Comparisons to standard CL baselines on the CL-MASR setup are clear and consistent.\n\nSoft spots. Biggest: tokens outside the per-language table. Table 1 shows J/U between 0.4% and 7.9%. Since the decoder still has a full vocabulary, any new-language utterance during inference may contain tokens not in A_hat. The paper never says what happens then. Options: fall back to the original shared embedding (which breaks the isolation claim), suppress those tokens (which would inflate WER), or hope test token sets are nearly contained in the training token sets (not reported). This is load-bearing, not a corner case. Second: no error bars, no repeated seeds, no significance test. A 2.3-point AWER gap on a single run is well within noise for this kind of benchmark. Hyperparameters are swept without saying whether validation or test drove the choice. No code release. The thresholds N=2, Mlen=5, Moverlap=3 look hand-tuned as well. Third, minor: ER-E-part in Table 4 performs almost as well as full ER-E, so the \"separate table is necessary\" story is weaker than the text implies.\n\nBottom line: the method is plausible and the paper is readable. With multiple seeds, explicit OOV handling, and open code, this would be a solid contribution. As is, a serious referee should ask for those before accepting. I'd send it to review, but not with a clean acceptance.","headline":"The embedding-surgery idea is timely and the experiments look honest, but the headline gain rests on a single run and an unstated OOV/fallback mechanism for the per-language embedding tables.","tokens_in":9927,"tokens_out":2609,"would_cite":true,"duration_ms":29123,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-language embedding copies reduce catastrophic forgetting when Whisper learns new languages.","keywords":["continual learning","catastrophic forgetting","multilingual ASR","Whisper","token embeddings","language identification","beam search","low-resource speech"],"falsifier":"Decode a new-language test set deliberately constructed to be rich in tokens absent from that language's 10-hour training set, then measure old-language word error rate before and after adaptation; if out-of-vocabulary tokens are routed through the shared embedding and old-language WER degrades, the surgery does not fully isolate old languages.","tokens_in":8932,"feed_emoji":"🎙️","tokens_out":6828,"duration_ms":59359,"temperature":0.7,"pith_summary":"The paper argues that the main hidden source of catastrophic forgetting when a multilingual ASR model like Whisper learns new languages is the decoder's token embedding lookup table, which gets overwritten with new-language semantics. It proposes Embedding Layer Surgery: give each new language its own copy of the token embeddings, keep the original embeddings for old languages, and split each copy into a shared special-token part (used for language identification) and a language-specific vocabulary part. Because language ID errors would then select the wrong embedding table, it adds task-wise beam search that decodes in the top two candidate languages and picks the hypothesis with the highest ASR score. In experiments adapting Whisper-small and Whisper-large-v2 to ten unseen Common Voice languages, the method lowers the average WER on pre-trained languages from 14.2% to 11.9% relative to Experience Replay, with no loss on the new languages. A sympathetic reader would take this as evidence that continual expansion of language-agnostic speech models can be made cheaper and safer by isolating the embedding layer rather than only replaying old data.","feed_headline":"Per-language embeddings cut Whisper's forgetting by 2.3% WER","feed_subtitle":"New method keeps old languages intact while adding 10 low-resource languages.","key_machinery":"Embedding Layer Surgery: for each new language $\\ell$, create a separate token-embedding lookup table $\\hat{A}_\\ell \\in \\mathbb{R}^{E \\times J_\\ell}$ with $J_\\ell < U$, containing only the tokens used by that language, and keep the original matrix $A$ for old languages; split the table into a language-shared special-token part and a language-specific vocabulary part. Task-wise beam search: after the model emits a language ID, decode in the top $N=2$ scoring languages, score each path by summed log probabilities, and choose the best hypothesis, disabling the search when a path has fewer than $M_{\\text{len}}$ words or more than $M_{\\text{overlap}}$ overlaps. The embedding surgery does the work of preventing overwriting; the beam search does the work of preventing LID errors from selecting the wrong embedding table.","core_discovery":"The central claim is that catastrophic forgetting in language-agnostic multilingual ASR is substantially caused by updating the shared token embedding matrix at the decoder, and that this can be prevented by maintaining a separate embedding lookup table for each newly added language. Each new table stores only the subset of tokens that appeared in that language's 10-hour training set; the original table remains untouched for old languages. The lookup table is partitioned so that special tokens such as language IDs stay shared, which keeps the model able to identify the input language and then automatically route decoding to the correct language-specific vocabulary embeddings. Task-wise beam search then fixes LID mistakes by keeping the top-N language hypotheses and selecting the final transcription by ASR score, with pruning rules for blank or overlapping hypotheses. The result is a language-agnostic continual learning system in which adding ten low-resource languages improves average WER on old languages from 14.2% to 11.9% compared with Experience Replay, without hurting new-language WER.","pith_inferences":["Editorial inference: the per-language tables cover only 0.4% to 7.9% of Whisper's vocabulary, so the reported results rely on an unstated policy for out-of-vocabulary tokens; specifying one is the next necessary step.","Editorial inference: if embedding surgery keeps old semantics intact without large replay buffers, it may allow the replay size to be reduced below one hour per language, which the paper does not test.","Editorial inference: the surgery component should transfer to any encoder-decoder ASR with a shared subword embedding, such as other multilingual speech models, where it could be tested independently of the beam-search component.","Editorial inference: task-wise beam search is a general LID-error correction scheme and could improve any language-agnostic decoder, even without per-language embedding surgery."],"forward_implications":["The embedding layer is a primary site of forgetting, so isolating it should be part of any continual learning recipe for multilingual ASR.","The model stays language-agnostic because special tokens, including language IDs, remain shared, so no manual language selection is required.","Task-wise beam search reduces LID confusion errors by more than 40% for ER and 60% for ER-E in the ablation, which directly improves language-agnostic WER.","Adding ten unseen languages one after another improves average WER on old languages from 14.2% to 11.9% compared with Experience Replay while new-language WER is unchanged.","The method applies to both Whisper-small and Whisper-large-v2, suggesting the benefit does not depend on a specific model scale."],"supporting_citations":[{"why":"Supplies the pre-trained Whisper encoder-decoder model and its built-in language identification, which the method adapts.","marker":"[2]"},{"why":"Provides the continual learning benchmark, Common Voice data subsets, and implementation base used for evaluation.","marker":"[23]"},{"why":"Gives the Experience Replay baseline and replay-loss formulation that the proposed method extends.","marker":"[11]"},{"why":"Provides the weight-averaging baseline ('AVG') that the experiments compare against.","marker":"[22]"},{"why":"Provides the diverse beam search idea that task-wise beam search adapts to language hypotheses.","marker":"[38]"},{"why":"Motivates treating the decoder token embedding lookup table as a distinct component to adapt and split.","marker":"[24]"},{"why":"Is the source of the multilingual speech data used for adaptation and evaluation.","marker":"[40]"}],"fun_headline_variants":["Embedding surgery keeps old languages while Whisper learns new ones","Whisper learns 10 new languages without losing old ones","Per-language embedding tables prevent catastrophic forgetting in ASR","Task-wise beam search fixes LID errors in Whisper continual learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The per-language embedding table contains only the tokens that happened to appear in that language's 10-hour training set, and the paper does not state how tokens outside that set are handled during inference; if such tokens fall back to the shared old embeddings, old-language semantics could still be overwritten.","fun_headline_variants_meta":{"raw":{"variants":["Embedding surgery keeps old languages while Whisper learns new ones","Whisper learns 10 new languages without losing old ones","Per-language embedding tables prevent catastrophic forgetting in ASR","Task-wise beam search fixes LID errors in Whisper continual learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001407,"raw_usage":{"total_tokens":5679,"prompt_tokens":932,"completion_tokens":4747,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":4677}},"tokens_in":548,"tokens_out":4747,"duration_ms":31365,"temperature":1.0,"reasoning_tokens":4677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:40.562452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decode a new-language test set deliberately constructed to be rich in tokens absent from that language's 10-hour training set, then measure old-language word error rate before and after adaptation; if out-of-vocabulary tokens are routed through the shared embedding and old-language WER degrades, the surgery does not fully isolate old languages.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained Whisper encoder-decoder model and its built-in language identification, which the method adapts."},{"cited_title":"Clrl-tuning: A novel continual learning approach for automatic speech recog- nition","cited_arxiv_id":null,"evidence_quote":"Provides the continual learning benchmark, Common Voice data subsets, and implementation base used for evaluation."},{"cited_title":"Dual-memory multi- modal learning for continual spoken keyword spotting with confidence selection and diversity enhancement,","cited_arxiv_id":null,"evidence_quote":"Provides the weight-averaging baseline ('AVG') that the experiments compare against."},{"cited_title":"End to end spoken language diarization with wav2vec embeddings","cited_arxiv_id":null,"evidence_quote":"Provides the diverse beam search idea that task-wise beam search adapts to language hypotheses."},{"cited_title":"Massively multilingual asr: A lifelong learning solu- tion,","cited_arxiv_id":null,"evidence_quote":"Motivates treating the decoder token embedding lookup table as a distinct component to adapt and split."},{"cited_title":"Multi- resolution approach to identification of spoken lan- guages and to improve overall language diarization sys- tem using whisper model,","cited_arxiv_id":null,"evidence_quote":"Is the source of the multilingual speech data used for adaptation and evaluation."}],"review_version":1}