Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling

Andy T. Liu; Hung-yi Lee; Kuan-Po Huang; Shao-Syuan Huang

REVIEW 5 major objections 6 minor 14 references

Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that an unseen language can be added to Whisper by feeding its decoder a probability-weighted sum of the embeddings of the 99 known language tags, and that a small MLP refinement of that embedding yields the largest gains.

desk verdict A genuinely new way to condition Whisper on unseen languages, with large reported gains but a missing control experiment that the paper should run before the central claim is fully trusted. read the letter →

arxiv 2412.16474 v1 pith:RF4SEKJJ submitted 2024-12-21 eess.AS cs.CL

Shao-Syuan Huang , Kuan-Po Huang , Andy T. Liu , Hung-yi Lee This is my paper · ORCID

classification eess.AScs.CL

keywords multilingualASRunseenlanguagesWhisperlanguageembeddingszero-shotspeechrecognitionlow-resourceidentificationfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to make Whisper, a multilingual ASR model trained on 99 languages, recognize an additional language it has never seen. The proposed answer is to avoid inventing a new language tag: instead, take Whisper's own predicted probabilities over its 99 known language tags and use them to compute a weighted sum of the corresponding language embeddings, which then stands in for the unseen language's embedding. A second, predictor-based variant trains a small two-layer MLP to turn that weighted-sum embedding into a closer approximation of a true language embedding. On rare languages from the ML-SUPERB benchmark, both approaches reduce character and word error rates in zero-shot and fine-tuned settings; the best fine-tuned configuration lowers CER from 28.54% to 22.91%. If the claim holds, extending an ASR system to a new language becomes a lightweight embedding computation instead of a costly retraining effort.

What carries the argument

The central object is the weighted-sum language embedding, $\mathrm{WsEmb} = \sum_{j=1}^{99} P(l_j \mid x, \mathrm{sot})\, \mathrm{Emb}_j$, formed from the softmax of Whisper's language-tag logits and the embedding table of its 99 known languages. It converts the discrete choice of a language tag into a continuous interpolation, so the decoder can draw on whatever subset of known languages the audio actually resembles. The second piece of machinery is a two-layer MLP predictor, with hidden size 1280 and MSE loss, trained by masking one seen language at a time and recovering its true embedding from the weighted sum of the remaining languages; at test time it refines the weighted-sum embedding for an unseen language. These objects carry the argument because they put all of the adaptation into the language-conditioning pathway rather than into new parameters or an external model.

What would settle it

For a held-out seen language, mask its embedding, compute the corpus-wise weighted-sum embedding from the other 98 languages, and measure its cosine similarity to the true embedding; if this similarity is near zero or no better than random across many languages, the weighted sum carries no recoverable language identity and the predictor has no signal. A second check: pick an unseen language for which Whisper's predicted language distribution is near-uniform and verify whether the method fails to reduce CER relative to the default argmax tag.

Watch

Extended reading notes

Core claim

The paper's central claim is that an unseen language can be represented, for Whisper's decoder, as a probability-weighted blend of the 99 embeddings Whisper already has. Given an utterance $x$ of an unseen language, Whisper's own next-token predictor supplies a distribution $P(l_i \mid x, \mathrm{sot})$ over language tags, and the method sets $\mathrm{WsEmb} = \sum_{j=1}^{99} P(l_j \mid x, \mathrm{sot})\, \mathrm{Emb}_j$, using this vector in place of the usual language-tag embedding. A corpus-wise variant averages the per-utterance distributions to give one embedding per language, and a parameterized variant initializes a trainable embedding with that average. The predictor-based variant trains an MLP, on seen languages, to map a weighted-sum embedding (computed with the target language masked out) to the target's true embedding; the authors interpret its success as evidence that the weighted sum captures real linguistic relationships. In their experiments, the predictor with the corpus-wise embedding performs best, and applying the weighted embedding during both fine-tuning and inference helps more than applying it only at inference.

Load-bearing premise

The load-bearing assumption is that Whisper's predicted language-tag probabilities for an unseen language carry real signal; if those probabilities are near-uniform or dominated by irrelevant languages, the weighted-sum embedding is a blur that does not help recognition.

Editorial extensions

If this is right

A new language can be added to Whisper without training a new language-tag embedding: the weighted sum over existing tags works in zero-shot inference, cutting CER by up to roughly 22% relative in the paper's rare-language evaluation.
Fine-tuning with the weighted embedding, whether fixed, trainable, or MLP-refined, beats fine-tuning a newly added baseline tag, and using the replacement at both fine-tuning and inference gives larger gains than inference-only application.
The predictor results imply that a minimal MLP, trained only on the 99 seen languages, can produce useful embeddings for unseen languages, so no large language model is required for this adaptation.
Because the method changes only the language embedding, it composes with standard efficient fine-tuning such as LoRA, which the paper uses throughout its fine-tuning experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the masked-language reconstruction experiment generalizes, the 99 language embeddings form a space in which unseen languages sit approximately inside the convex hull of seen ones; one could test this directly by checking whether the weighted-sum embeddings of related languages cluster by language family.
A testable extension is to apply the same recipe to other token-conditioned speech models and to Whisper large-v3; consistent gains would indicate that embedding interpolation is a general property of language-conditioned decoders, while failures would bound the effect to the specific model tested here.
The method's usefulness should track the quality of Whisper's language identification: an unseen language whose utterances yield near-uniform language probabilities should show smaller gains, so measuring the gain against the entropy of the predicted distribution would be a useful diagnostic.
Since the utterance-wise variant produces a distinct embedding per utterance, it may extend naturally to code-switched speech, where the effective language changes within one recording, though the paper does not test this.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper proposes a lightweight way to adapt Whisper to languages it was not trained on, and the reported gains are large enough to matter. The core idea is to replace the missing language-tag embedding with a probability-weighted average of the 99 existing embeddings, using Whisper's own predicted language distribution, and optionally refine that average with a small MLP trained on seen languages under leave-one-out masking. That is a fresh mechanism, not just a rehashed feature interpolation, and the paper does a decent job of comparing several variants with ablations.

What the paper does well: the method is genuinely lightweight, the experiments cover zero-shot and fine-tuning with LoRA, and the fine-tuning results come with error bars. The best predictor-based method brings CER down from 28.54% to 22.91% on the ML-SUPERB rare-language set, which is a substantial improvement if it holds up.

The soft spots are mostly about evidence, not about the idea. The most important missing piece is the direct control experiment that would test the central assumption: for seen languages, replace the true language embedding with the weighted-sum or predictor-based embedding and measure ASR. The predictor is trained with leave-one-out masking, but the paper never reports the corresponding ASR result. Without that, the gains on unseen languages could come from the particular choice of test languages or from LoRA dynamics rather than from the embedding substitution itself. Second, the unseen languages are never named, which makes the evaluation impossible to reproduce. Third, the zero-shot numbers have no error bars, and the paper does not describe a validation protocol for selecting among the many variants, even though it mentions 'extensive hyperparameter selection.' The 98-language-versus-99-language distribution shift for the predictor input is a minor issue but also goes unexamined. No code or data are released, which adds to the reproducibility concern.

Bottom line: the paper is for the multilingual ASR and low-resource speech crowd, and it deserves a serious referee. It is not ready as is, though; it should go to review with a request for the leave-one-out ASR experiment, named languages, and a clearer model-selection story. I would be moderately optimistic that the core finding survives that check.

Referee Report

5 major / 6 minor

Summary. The paper addresses Whisper's poor ASR performance on languages absent from its 99-language training set. It proposes replacing the language-tag embedding with a probability-weighted sum of all 99 language embeddings, where the probabilities come from Whisper's own language-tag prediction. A second variant trains a small MLP to map such weighted-sum embeddings to the true embedding of a masked seen language, then applies the predicted embedding to unseen languages. Experiments on ML-SUPERB 'rare languages' with Whisper large-v2 compare zero-shot and LoRA fine-tuning settings, reporting that the best fine-tuned predictor method reduces CER from 28.54% to 22.91% and WER from 69.97% to 61.70% relative to the new-tag baseline.

Significance. Assuming the reported gains survive closer scrutiny, the paper offers a lightweight, parameter-efficient alternative to LLM-based conditioning for unseen-language ASR, and the predictor idea provides a way to leverage seen-language structure without additional training data. The method's simplicity and the use of a standard publicly available model (Whisper large-v2) and benchmark (ML-SUPERB) are strengths, and the paper includes an explicit ablation separating fine-tuning-stage and inference-stage application. However, the current evaluation lacks the transparency and controls needed to establish the central substitution claim.

major comments (5)

[Section III (Data) and Table I] The unseen languages are never named, and no data statistics are provided. The paper says 'We selected Whisper-unseen languages from this dataset and combined them into Whisper-unseen training and test sets' but does not list the languages, the per-language utterance or hour counts, or the train/test split. This prevents reproduction and leaves open the possibility that the gains are driven by a particular language or by the test-set size. Please provide the complete list of unseen languages and per-language data statistics.
[Section III-C and Table I] No leave-one-out ASR validation is reported. The predictor is trained by masking one seen language and predicting its embedding from a weighted sum of the remaining seen languages, yet no ASR result is given for seen languages under this masking. A direct control would replace a seen language's embedding with the weighted-sum embedding (with that language masked) and measure CER/WER against the true-embedding condition. Without this, the claim that the weighted-sum embedding is a faithful substitute for a missing language embedding is not tested, and the observed fine-tuning gains cannot be cleanly attributed to the proposed mechanism.
[Section III-C] There is a train/inference distribution shift in the predictor input. The predictor is trained on weighted sums over 98 languages (one masked language excluded), but at inference it is applied to weighted sums over all 99 seen languages for an unseen language. This shift is not discussed or evaluated. Please either analyze its effect or train and validate the predictor under the same input distribution used at inference.
[Section II-A versus Section III-B] The baseline method is described as adding a new language tag and 'fine-tuning the entire model,' but all fine-tuning experiments use LoRA with rank 32 and alpha 64. If the baseline is not actually fully fine-tuned, the comparison may understate the baseline; if the baseline is LoRA-tuned, the description should be corrected. This inconsistency directly affects the headline fine-tuning comparison.
[Table I, zero-shot rows] The zero-shot CER/WER numbers are presented without error bars or significance information, and the paper does not state whether they come from a single pass or multiple runs. Given the large variance typical of zero-shot ASR on low-resource languages, the claimed 22% CER reduction in Section IV-A needs uncertainty quantification before it can be evaluated.

minor comments (6)

[Table I] WER values above 100% (e.g., 139.33% in the zero-shot default row) are possible with insertion-heavy outputs, but they should be explained so readers do not doubt the metric.
[Section II-A] The softmax formula for the language-tag probability is typeset incorrectly: the denominator appears as 'P99 j=1 exp(f (lj|x, sot))' separated from the fraction. Please fix the equation.
[Figure 1] The figure caption says yellow boxes indicate individual language embeddings, but the text does not clearly connect the left and right parts of the figure when describing the predictor input. Adding explicit labels for WsEmb and PredEmb would improve clarity.
[Section III (Model)] The statement that Whisper large-v3 is 'almost identical to large-v2, differing only by the addition of a new language and a change in spectrogram input size' is imprecise and unnecessary for the paper; please either substantiate it or remove it.
[Section III-C] The predictor configuration is said to be 'determined through extensive hyperparameter selection,' but the search space and the selection criterion (e.g., validation MSE versus downstream CER) are not reported. Please provide these details.
[Abstract and Introduction] The term 'rare languages' is used without definition. Because ML-SUPERB contains many languages, please clarify what makes these languages rare and unseen for Whisper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed embedding blending and predictor are empirical constructions evaluated on held-out unseen languages, not derivations that reduce to their own inputs.

full rationale

I find no circular step in the paper's derivation chain. The weighted-sum method is an explicit construction (WsEmb = sum of 99 language embeddings weighted by Whisper's predicted language probabilities), and the resulting ASR performance on unseen languages is an empirical outcome, not an input to the construction. The predictor is trained on Whisper-seen languages with a leave-one-out masking scheme: the input is the weighted-sum embedding computed from the remaining seen languages and the target is the masked language's true embedding. This supervised training task is distinct from the evaluation metric (CER/WER on unseen-language speech), so the reported improvements are generalization claims rather than fitted predictions. The predictor's MSE objective is not the ASR objective, and no parameter is fitted to the unseen-language CER/WER numbers. The self-citations in the paper ([8], [9]) are used only to support the uncontroversial observation that Whisper uses language-tag prefixes; they are not load-bearing for the proposed method. Concerns about the absence of a leave-one-out ASR control or the train/inference distribution shift in the predictor are validity or evidence-quality issues, not circularity. The paper is self-contained against the external ML-SUPERB benchmark, and the central claims rest on direct experiments rather than on a self-citation chain or definitional equivalence.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim does not introduce any new physical or linguistic entities. It assumes certain properties of Whisper's embedding space and of predicted probabilities.

free parameters (6)

predictor_hidden_size = 1280
Hidden size of the 2-layer MLP predictor; chosen by hand through 'extensive hyperparameter selection' (Section III-C).
predictor_learning_rate = 5e-4
Learning rate for the predictor, set by hyperparameter search (Section III-C).
lora_rank = 32
Rank of LoRA adapters used in fine-tuning (Section III-B).
lora_alpha = 64
LoRA alpha parameter (Section III-B).
fine_tune_learning_rate = 4.7e-5
Learning rate for Whisper fine-tuning (Section III-B).
weight_decay = 0.02
Weight decay for AdamW in fine-tuning (Section III-B).

assumptions (3)

domain assumption Whisper's language embeddings are semantically meaningful and linear combinations reflect linguistic similarity.
The weighted sum method assumes that interpolating embeddings in Euclidean space produces a meaningful conditioning vector (Section II-B).
domain assumption Whisper's predicted language-tag probabilities for unseen languages are informative.
The weights in the weighted sum come from Whisper's softmax over seen language tags for unseen-language audio; if this distribution is uninformative, the method cannot work (Section II-B).
domain assumption The mapping from weighted-sum embeddings to true embeddings learned on seen languages transfers to unseen languages.
The predictor is trained only on seen languages and applied to unseen languages without additional adaptation (Section III-C).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling." pith.science (2026). https://pith.science/paper/RF4SEKJJ

@misc{pith2026241216474,
  author       = {Pith},
  title        = {Pith review of: Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RF4SEKJJ}},
  note         = {Machine review of arXiv:2412.16474}
}

read the original abstract

Multilingual Automatic Speech Recognition (ASR) aims to recognize and transcribe speech from multiple languages within a single system. Whisper, one of the most advanced ASR models, excels in this domain by handling 99 languages effectively, leveraging a vast amount of data and incorporating language tags as prefixes to guide the recognition process. However, despite its success, Whisper struggles with unseen languages, those not included in its pre-training. Motivated by the observation that many languages share linguistic characteristics, we propose methods that exploit these relationships to enhance ASR performance on unseen languages. Specifically, we introduce a weighted sum method, which computes a weighted sum of the embeddings of language tags, using Whisper's predicted language probabilities. In addition, we develop a predictor-based approach that refines the weighted sum embedding to more closely approximate the true embedding for unseen languages. Experimental results demonstrate substantial improvements in ASR performance, both in zero-shot and fine-tuning settings. Our proposed methods outperform baseline approaches, providing an effective solution for addressing unseen languages in multilingual ASR.

Figures

Figures reproduced from arXiv: 2412.16474 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 8 canonical work pages

[1]

Deep speech 2: End-to-end speech recognition in english and mandarin,

D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al. , “Deep speech 2: End-to-end speech recognition in english and mandarin,” in International conference on machine learning . PMLR, 2016, pp. 173– 182

work page 2016
[2]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12 449– 12 460, 2020

2020
[3]

Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,

Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7829–7833

work page 2020
[4]

Multilingual speech recognition with a single end-to-end model,

S. Toshniwal, T. N. Sainath, R. J. Weiss, B. Li, P. Moreno, E. Weinstein, and K. Rao, “Multilingual speech recognition with a single end-to-end model,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2018, pp. 4904–4908

2018
[5]

Un- supervised cross-lingual representation learning for speech recognition,

A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Un- supervised cross-lingual representation learning for speech recognition,” in Interspeech 2021, 2021, pp. 2426–2430

work page 2021
[6]

Xls-r: Self-supervised cross-lingual speech representation learning at scale,

A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” in Interspeech 2022 , 2022, pp. 2278–2282

work page 2022
[7]

Robust speech recognition via large-scale weak supervi- sion,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

2023
[8]

Yang, K.-P

C.-K. Yang, K.-P. Huang, K.-H. Lu, C.-Y . Kuan, C.-Y . Hsiao, and H.-Y . Lee, “Investigating zero-shot generalizability on mandarin-english code- switched asr and speech-to-text translation of recent foundation models with self-supervision and weak supervision,” in 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (I...

2024

Show all 14 references

[9]

Do prompts really prompt? ex- ploring the prompt understanding capability of whisper,

C.-K. Yang, K.-P. Huang, and H.-y. Lee, “Do prompts really prompt? ex- ploring the prompt understanding capability of whisper,” arXiv preprint arXiv:2406.05806, 2024

2024 arXiv
[10]

Prompting the hidden talent of web-scale speech models for zero-shot task generalization,

P. Peng, B. Yan, S. Watanabe, and D. Harwath, “Prompting the hidden talent of web-scale speech models for zero-shot task generalization,” in INTERSPEECH 2023 , 2023, pp. 396–400

2023
[11]

Zero- shot domain-sensitive speech recognition with prompt-conditioning fine- tuning,

F.-T. Liao, Y .-C. Chan, Y .-C. Chen, C.-J. Hsu, and D.-S. Shiu, “Zero- shot domain-sensitive speech recognition with prompt-conditioning fine- tuning,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1–8

2023
[12]

Improving whisper’s recognition performance for under-represented language kazakh leveraging unpaired speech and text,

J. Li, Y . Pu, Q. Sun, and W.-Q. Zhang, “Improving whisper’s recognition performance for under-represented language kazakh leveraging unpaired speech and text,” arXiv preprint arXiv:2408.05554 , 2024

2024 arXiv
[13]

ML- SUPERB: Multilingual Speech Universal PERformance Benchmark,

J. Shi, D. Berrebbi, W. Chen, E.-P. Hu, W.-P. Huang, H.-L. Chung, X. Chang, S.-W. Li, A. Mohamed, H. yi Lee, and S. Watanabe, “ML- SUPERB: Multilingual Speech Universal PERformance Benchmark,” in Proc. INTERSPEECH 2023 , 2023, pp. 884–888

2023
[14]

LoRA: Low-rank adaptation of large language models,

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations , 2022

2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Deep speech 2: End-to-end speech recognition in english and mandarin,

D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al. , “Deep speech 2: End-to-end speech recognition in english and mandarin,” in International conference on machine learning . PMLR, 2016, pp. 173– 182

work page 2016

[2] [2]

wav2vec 2.0: A framework for self-supervised learning of speech representations,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12 449– 12 460, 2020

2020

[3] [3]

Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,

Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7829–7833

work page 2020

[4] [4]

Multilingual speech recognition with a single end-to-end model,

S. Toshniwal, T. N. Sainath, R. J. Weiss, B. Li, P. Moreno, E. Weinstein, and K. Rao, “Multilingual speech recognition with a single end-to-end model,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2018, pp. 4904–4908

2018

[5] [5]

Un- supervised cross-lingual representation learning for speech recognition,

A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Un- supervised cross-lingual representation learning for speech recognition,” in Interspeech 2021, 2021, pp. 2426–2430

work page 2021

[6] [6]

Xls-r: Self-supervised cross-lingual speech representation learning at scale,

A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y . Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” in Interspeech 2022 , 2022, pp. 2278–2282

work page 2022

[7] [7]

Robust speech recognition via large-scale weak supervi- sion,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

2023

[8] [8]

Yang, K.-P

C.-K. Yang, K.-P. Huang, K.-H. Lu, C.-Y . Kuan, C.-Y . Hsiao, and H.-Y . Lee, “Investigating zero-shot generalizability on mandarin-english code- switched asr and speech-to-text translation of recent foundation models with self-supervision and weak supervision,” in 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (I...

2024