Pith. sign in

REVIEW 3 major objections 8 minor 5 cited by

A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data

T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a closed self-refinement loop—ASR pseudo-labels train a TTS, whose synthetic speech fine-tunes the same ASR—adapts Whisper to Taiwanese Mandarin, cutting error rates by up to 20% on Mandarin and 50% on…

desk verdict Useful TTS-based ASR recipe undercut by a self-refining claim the experiments never actually test. read the letter →

arxiv 2506.11130 v2 pith:LBBNJN3F submitted 2025-06-10 cs.CL cs.AIcs.SDeess.AS

Cheng-Kang Chou , Chan-Jan Hsu , Ho-Lam Chung , Liang-Hsuan Tseng , Hsi-Chun Cheng , Yu-Kuan Fu , Kuan Po Huang , Hung-Yi Lee This is my paper · ORCID
classification cs.CLcs.AIcs.SDeess.AS
keywords automaticspeechrecognitiontext-to-speechpseudo-labelingself-refinementcode-switchingTaiwaneseMandarinsyntheticWhisperfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic speech recognition usually improves by adding transcribed audio, which is scarce for low-resource languages. This paper claims a closed self-refinement loop can substitute synthetic speech for most of that real data: an existing ASR model writes pseudo-labels (its own transcriptions) for unlabeled audio, a text-to-speech system learns from those pairs, and the TTS then turns cheap text into massive amounts of paired synthetic speech on which the same ASR is fine-tuned. Applying the loop to Taiwanese Mandarin, the authors adapt Whisper-large-v2 into a model they call Twister, trained with 6,000 hours of unlabeled speech and less than 1GB of text, and report error-rate reductions up to 20% on Mandarin and 50% on Mandarin-English code-switching benchmarks relative to Whisper. They further claim the loop needs about 10 times less real speech data than pseudo-labeling self-distillation baselines while matching or exceeding their accuracy. If correct, the framework offers a path to specialized ASR in low-resource settings using data that is abundant—unlabeled audio and text—rather than transcribed speech.

What carries the argument

The load-bearing object is the self-refining loop itself: ASR F_theta generates pseudo-labels T_hat from unlabeled speech S; TTS G_phi is trained on (S, T_hat); G_phi then synthesizes (S_hat, T) from a text corpus T; and the ASR is fine-tuned on filtered, aligned (S_hat, T). The supporting machinery that makes the loop usable is a validator that computes phoneme error rate between the target text and a lightweight ASR's transcription to drop hallucinated synthetic clips, Montreal Forced Aligner for fine-grained segmentation, concatenation of utterances to create long-form and code-switching audio up to the model's 30-second window, random perturbations for acoustic robustness, and a mixed language embedding initialized as the average of the Mandarin and English token embeddings.

What would settle it

Rebuild the pipeline from scratch: train a TTS solely on Whisper-large-v2 pseudo-labels for the 6,000 hours of unlabeled Taiwanese Mandarin audio, synthesize the 10,000-hour corpus, apply the same filtering, alignment, and augmentation, then fine-tune and evaluate on ASCEND, CommonVoice16-zh-TW, CSZS, and ML-Lecture-2021; if the reported error reductions (up to 55.88% on CSZS) do not reproduce, the gains came from the real-data-trained TTS and real paired data rather than the self-refining loop.

Watch

Extended reading notes

Core claim

The central discovery the authors are trying to establish is that TTS-synthesized speech, filtered and aligned, can be bootstrapped into the original ASR model to form a positive reinforcement loop, so that an ASR system improves on a target language without handcrafted paired data. In the reported demonstration, pseudo-labels from Whisper-large-v2 on unlabeled Mandarin audio, the BreezyVoice TTS, text from FineWeb2, and real English and code-switching audio from CommonVoice and NTUML2021 are combined into a 10,000-hour synthetic corpus; after filtering by phoneme error rate, forced alignment, utterance concatenation, and random perturbation, the corpus is used to fine-tune Whisper-large-v2 into Twister. The authors report that Twister outperforms the base model on nearly every benchmark, with the largest relative error reduction of 55.88% on the CSZS code-switching set, and that this result is achieved with at least 10 times less real speech data than pseudo-label distillation baselines such as COOL-Whisper's 60,000 hours.

Load-bearing premise

The load-bearing premise is that the text-to-speech model in the loop was trained on pseudo-labels from the ASR model's unlabeled speech; the experiments use an existing TTS and some real transcribed audio, so the 'only unlabeled data' claim rests on a component the paper did not test.

Editorial extensions

If this is right

  • If the loop works as claimed, a Whisper-scale ASR can be specialized to a target language using only unlabeled audio and an arbitrary text corpus, so transcribed speech stops being the bottleneck.
  • Code-switching recognition can be improved without large natural code-switching corpora by concatenating synthetic and real monolingual utterances into bilingual clips.
  • Long-form ASR stays intact when training on short synthetic clips because forced-alignment boundaries are used to construct 30-second training windows with continuation tags.
  • The framework scales along two axes the authors name: adding more text to synthesize more speech, and iterating the refinement loop with the improved model.
  • Homophone errors that plague pseudo-labeling in Mandarin are partially avoided because the synthetic training text is ground-truth typed text, not another ASR guess.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As reported, the TTS is adopted off-the-shelf rather than trained on pseudo-labels from the unlabeled speech, so the strongest reading of the paper's 'only unlabeled data' claim is an extrapolation; a direct test is to train the TTS from scratch on pseudo-labels and rerun the pipeline.
  • The 10x data-efficiency comparison is against baselines with different model sizes and training recipes; a cleaner test would hold the base model, compute, and data budget fixed and vary only whether the TTS is in the loop.
  • If the loop is iterated, the method may converge to an equilibrium where the ASR and TTS mutually reinforce each other's biases, so an empirical study of second- and third-generation Twister would show whether gains compound or saturate.
  • The code-switching gains are obtained by sentence-level concatenation; this suggests word-level code-switching from a mixed-text TTS could push the same benchmarks further, but that is beyond the paper's reported experiments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes a self-refining framework for ASR in which an existing ASR model generates pseudo-labels on unlabeled speech, these pseudo-labeled pairs train a TTS model, and the TTS then synthesizes large amounts of speech from text. The ASR model is retrained on the filtered and augmented synthetic pairs, closing a loop that the authors claim requires only unlabeled audio and text. The method is instantiated for Taiwanese Mandarin and Mandarin-English code-switching, producing a Whisper-large-v2-derived model called Twister. The authors report relative MER reductions of up to 19-20% on Mandarin benchmarks and up to 55.88% on a code-switching benchmark relative to Whisper-large-v2, and state that the TTS-in-the-loop approach lowers real speech data requirements by 10x compared to pseudo-labeling self-distillation baselines.

Significance. If the closed-loop, unlabeled-only claims held, this would be a significant contribution to low-resource ASR: it would show a scalable path from abundant text and unlabeled audio to large ASR gains without transcribed target-language speech. The paper reports clear gains on several held-out public benchmarks (ASCEND, CommonVoice, CSZS, ML-lecture), and the authors state they will open-source the model and synthetic datasets, which are concrete strengths. However, the central claims are not supported by the experiments as described: the TTS is adopted pretrained rather than trained on pseudo-labels, and real paired English and code-switching data are included in the final training corpus. The reported results are still relevant for TTS-based data augmentation, but the headline 'self-refining' and 'only unlabeled data' claims need substantial revision or additional experiments.

major comments (3)
  1. [Section III-A and Section IV-A] The method description specifies that the TTS model G_phi is trained on D_pseudo = {(S_i, T-hat_i)} generated by the ASR model F_theta from unlabeled speech. In contrast, Section IV-A states 'we adopt BreezyVoice directly as G_phi' and refers the reader to the original BreezyVoice training procedure. BreezyVoice is a pretrained TTS from prior work, and no evidence is given that it was trained on pseudo-labels from Whisper-large-v2 or on the 6,000 hours of unlabeled speech. The closed self-refinement loop described in Fig. 1 and the abstract is therefore not implemented in the experiments. This is an internal inconsistency between the method and the experimental setup, and it undermines the 'self-refining' and 'only unlabeled datasets' claims. The authors should either retrain or fine-tune the TTS on pseudo-labels and demonstrate the loop, or revise the claims to describe the actual setup, which is fine-tuning an ASR model on synthetic speech from a pretrained TTS.
  2. [Section IV-B and Table II] The final training corpus includes real paired data: CommonVoice English (1,738 hours), NTUML2021 code-switching (11 hours), and a code-switching set of 1,715 hours that is built by mixing English clips from CommonVoice with Mandarin clips from ODC-Synth, i.e., it contains real English and real code-switching audio. The claim that the framework works 'with only unlabeled datasets' is therefore contradicted by the presence of labeled real speech in the training mix. The code-switching gains reported in Table IV, in particular the 55.88% improvement on CSZS-zh-en, may be driven substantially by these real paired English and code-switching data. A proper test of the unlabeled-only claim requires ablations that remove all real paired data from the training corpus, or a reformulation of the contribution as a hybrid method that combines synthetic Mandarin speech with a small amount of real labeled English and code-switching data.
  3. [Section V-B and Abstract] The claim that the framework 'lowers the real speech data required by 10x' compared to pseudo-labeling self-distillation approaches is not supported by the reported experiments. The comparison with Whisper-large-v3 and COOL-Whisper is confounded: the baselines differ in base architecture (Whisper-large-v3 is a different model version; COOL-Whisper is roughly Whisper-medium size), training data composition, and training procedure. No controlled experiment is provided, such as fine-tuning the same Whisper-large-v2 on 6,000 hours of unlabeled speech with pseudo-labeling self-distillation but without TTS synthesis, or the same TTS-pipeline without the real paired data. The 10x factor appears to be an arithmetic comparison of total raw speech hours (60,000 vs. ~6,000), which does not isolate the effect of the TTS loop. The data-efficiency claim should be substantiated with ablations that vary the amount of real and synthetic data while holding the base model and training protocol fixed.
minor comments (8)
  1. [Section III-B] The sentence 'If the validator’s transcription significantly deviates from the original input text.' is an incomplete fragment; it should be joined to the preceding sentence or removed.
  2. [Section IV-B] The description of the Mandarin set says it is 'formed exclusively from concatenating Mandarin clips sampled from ODC-Synth, and some short-form English clips are preserved for replay purpose.' This is internally inconsistent: if the set is Mandarin-only, the inclusion of English clips should be explained as a separate category or the wording should be revised.
  3. [Section IV-B and Table II] The final training corpus lists 'English Real Short En 10' hours, but the text does not specify how these 10 hours are selected from CommonVoice or why this specific amount is used. Please clarify the selection procedure and the rationale.
  4. [Section III-D1] The long-form augmentation procedure is underspecified: the 'special tag' appended to indicate continuation should be explicitly identified (e.g., a Whisper timestamp token or a custom token), and the backtracking rule for L_bound should be described precisely.
  5. [Section IV-C] The initialization of the shared language embedding as 'the element-wise average of the language-token embeddings' should clarify whether this is an average over the token dimension or over the embedding dimension, and whether the resulting vector is used to replace both the |zh| and |en| token embeddings or to create a new one.
  6. [Section V and Table IV] The acronym WERR is used in the text and table without definition. Since the evaluation metric is MER, the relationship between relative WERR and MER should be stated explicitly.
  7. [Section I] The paper states that the model and synthetic datasets are open-sourced, but no URLs or repository identifiers are provided in the manuscript. Please include the links.
  8. [Throughout] Several minor typos and formatting issues arise, including 'enhencement' in Section II-D, 'Twisterreduces' in the Abstract, and 'what we need initially are three objects framed by red dashed lines' in Section III. A careful proofread is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline gains are measured on external benchmarks, and the unverified closed-loop claim is a support/consistency issue rather than a reduction of inputs into outputs.

full rationale

The paper does not derive its headline error-rate reductions from its own fitted inputs. Synthetic Mandarin speech is generated from an independent text corpus (FineWeb2, Section IV-B), and evaluation is carried out on external benchmarks (ASCEND, CommonVoice16, CSZS, ML-Lecture-2021, FormosaSpeech, Table IV). No equation in Section III reduces a predicted MER to a fitted parameter: the filtering threshold alpha is a fixed hyperparameter, forced alignment and augmentation transform the synthetic data without encoding test-set labels, and the Whisper-large-v2 fine-tuning uses a standard supervised objective. The main caveat is that Section III-A defines the self-refining loop as training the TTS on pseudo-labels D_pseudo = {(S_i, hat{T}_i)}, whereas Section IV-A adopts the pretrained BreezyVoice directly as G_phi, so the 'only unlabeled datasets' claim is not actually demonstrated. That is an internal-consistency or missing-support problem, not a circular reduction: the measured gains over Whisper-large-v2 are not forced by construction, and the BreezyVoice self-citation is not used to forbid alternatives or to define the target metric.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the hand-chosen filtering threshold and data mix, and on several unstated assumptions about TTS fidelity and pseudo-label quality that the experiments do not test. No free parameter is fitted to the evaluation benchmarks, so the headline gains are not circular in that sense.

free parameters (2)
  • PER filtering threshold alpha = 0.6
    Section III-B sets maximum phoneme error rate to filter synthesized pairs; no sensitivity analysis is reported, and the retained 4,000 of 10,000 hours depends on this choice.
  • Final data mix volumes = 4,000 h Mandarin long + 70 h Mandarin short + 10 h English + 1,715 h code-switching
    Section IV-B and Table II define the training corpus by hand-selected volumes; no ablation shows how each component contributes.
assumptions (3)
  • domain assumption BreezyVoice can synthesize intelligible, acoustically diverse Taiwanese Mandarin speech from arbitrary text.
    Section IV-A adopts BreezyVoice directly; the paper does not measure synthetic audio quality or TTS error rates in this work, relying on the prior BreezyVoice MOS.
  • domain assumption Pseudo-labels generated by Whisper-large-v2 on unlabeled speech are accurate enough to train the TTS in the proposed loop.
    Section III-A relies on this for the closed loop, but Section IV-A never runs this step, so the assumption is untested in the reported experiments.
  • domain assumption Synthetic speech distribution is close enough to real Taiwanese Mandarin that models trained on it transfer to real benchmarks.
    The entire method depends on this transfer; the paper supports it only indirectly via benchmark improvements, not by a controlled synthetic-to-real analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data." pith.science (2026). https://pith.science/paper/LBBNJN3F

@misc{pith2026250611130,
  author       = {Pith},
  title        = {Pith review of: A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LBBNJN3F}},
  note         = {Machine review of arXiv:2506.11130}
}
read the original abstract

We propose a self-refining framework that enhances ASR performance with only unlabeled datasets. The process starts with an existing ASR model generating pseudo-labels on unannotated speech, which are then used to train a high-fidelity text-to-speech (TTS) system. Then, synthesized speech text pairs are bootstrapped into the original ASR system, completing the closed-loop self-improvement cycle. We demonstrated the effectiveness of the framework on Taiwanese Mandarin speech. Leveraging 6,000 hours of unlabeled speech, a moderate amount of text data, and synthetic content from the AI models, we adapt Whisper-large-v2 into a specialized model, Twister. Twister reduces error rates by up to 20% on Mandarin and 50% on Mandarin-English code-switching benchmarks compared to Whisper. Results highlight the framework as a compelling alternative to pseudo-labeling self-distillation approaches and provides a practical pathway for improving ASR performance in low-resource or domain-specific settings.

Figures

Figures reproduced from arXiv: 2506.11130 by the authors.

Figure 1
Figure 1. Overview of the self-refining framework. The process begins by generating pseudo-labeled speech–text pairs from a pre-existing ASR model(framed [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing

    cs.CL 2026-07 conditional novelty 7.0 of 10

    REDDIT corrects non-speech-induced timestamp drift in autoregressive ASR by editing timestamp targets under cached replay context while anchoring non-timestamp behavior to the frozen base distribution.

  2. When Synthetic Speech Is All You Have: Better Call GRPO

    cs.CL 2026-07 conditional novelty 6.0 of 10

    On synthetic banking speech alone, GRPO cuts ASR WER 40% relative to SFT (36.71%→22.09%) by improving stopping calibration and attention anchoring to audio.

  3. BlueMagpie-TTS: A Token-Efficient Tokenizer, Language Model, and TTS for Taiwanese-Accent Code-Switching Speech

    cs.SD 2026-07 conditional novelty 6.0 of 10

    A Taiwan-specific tokenizer, language model, and bridge to a reused acoustic stack cut code-switching TTS CER from 11.45% to 4.81%, with 65.6% listener preference.

  4. Context-Aware ASR for Mandarin Technical Lectures

    cs.SD 2026-07 conditional novelty 6.0 of 10

    Self-built lecture glossaries from first-pass ASR raise technical-term recall across five backbones while holding or lowering CER on a new Mandarin AI/ML lecture benchmark.

  5. How to Leverage Synthetic Speech for LLM-Based ASR Systems?

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Layer-wise pooling plus RIR-augmented synthetic speech matches a 100%-real ASR baseline with only 25% real data (13.6 h) and beats it at higher real fractions.

Reference graph

Works this paper leans on

40 extracted references · 30 canonical work pages · cited by 5 Pith papers

  1. [1]

    Canary-1b,

    NVIDIA, “Canary-1b,” https://huggingface.co/nvidia/canary-1b, 2023

    work page 2023

  2. [2]

    Robust speech recognition via large- scale weak supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large- scale weak supervision,”OpenAI, 2022

    work page 2022

  3. [3]

    Phi-4-mini technical report: Compact yet pow- erful multimodal language and multimodal models,

    Microsoft Research, “Phi-4-mini technical report: Compact yet pow- erful multimodal language and multimodal models,”arXiv preprint arXiv:2503.01743, 2024

    arXiv 2024

  4. [4]

    Usm: Scaling auto- matic speech recognition beyond 100 languages,

    Yu Zhang, Wei Chen, Zelin Wang, et al., “Usm: Scaling auto- matic speech recognition beyond 100 languages,”arXiv preprint arXiv:2303.01037, 2023

    arXiv 2023

  5. [5]

    Llama: Open and efficient foundation language models,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample, “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  6. [6]

    Olmo: Accelerating the science of language models,

    Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al., “Olmo: Accelerating the science of language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15789–15809

    work page 2024

  7. [7]

    Using synthetic audio to improve the recognition of out-of-vocabulary words in end-to-end asr systems,

    Xianrui Zheng, Yulan Liu, Deniz Gunceler, and Daniel Willett, “Using synthetic audio to improve the recognition of out-of-vocabulary words in end-to-end asr systems,” inICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 5674–5678

    work page 2021

  8. [8]

    Text-only domain adaptation for end-to-end asr using integrated text-to-mel-spectrogram generator,

    Vladimir Bataev, Roman Korostik, Evgeny Shabalin, Vitaly Lavrukhin, and Boris Ginsburg, “Text-only domain adaptation for end-to-end asr using integrated text-to-mel-spectrogram generator,” inProc. Interspeech 2023, 2023, pp. 2928–2932

    work page 2023

Show all 40 references
  1. [9]

    Text is all you need: Personalizing asr models using controllable speech synthesis,

    Karren Yang, Ting-Yao Hu, Jen-Hao Rick Chang, Hema Swetha Kop- pula, and Oncel Tuzel, “Text is all you need: Personalizing asr models using controllable speech synthesis,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEE...

    2023

  2. [10]

    Corpus synthesis for zero-shot asr domain adaptation using large language models,

    Hsuan Su, Ting-Yao Hu, Hema Swetha Koppula, Raviteja Vemulapalli, Jen-Hao Rick Chang, Karren Yang, Gautam Varma Mantena, and Oncel Tuzel, “Corpus synthesis for zero-shot asr domain adaptation using large language models,” inICASSP 2024-2024 IEEE International Conference on Aco...

    2024

  3. [11]

    Task arithmetic can mitigate synthetic-to-real gap in automatic speech recognition,

    Hsuan Su, Hua Farn, Fan-Yun Sun, Shang-Tse Chen, and Hung-Yi Lee, “Task arithmetic can mitigate synthetic-to-real gap in automatic speech recognition,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 8905–8915

    2024

  4. [12]

    Enhancing low-resource asr through versatile tts: Bridging the data gap,

    Guanrou Yang, Fan Yu, Ziyang Ma, Zhihao Du, Zhifu Gao, Shiliang Zhang, and Xie Chen, “Enhancing low-resource asr through versatile tts: Bridging the data gap,” inICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

  5. [13]

    Coqui tts,

    Eren G ¨olge and The Coqui TTS Team, “Coqui tts,” https://github.com/ coqui-ai/TTS, 2021, If you want to cite [name], feel free to use this (but only if you loved it [emoji])

    2021

  6. [14]

    Matcha-tts: A fast tts architecture with conditional flow matching,

    Shivam Mehta, Ruibo Tu, Jonas Beskow, ´Eva Sz ´ekely, and Gustav Eje Henter, “Matcha-tts: A fast tts architecture with conditional flow matching,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11341–11345

    2024

  7. [15]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al., “Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  8. [16]

    Breezyvoice: Adapting tts for taiwanese mandarin with enhanced polyphone disambiguation–challenges and insights,

    Chan-Jan Hsu, Yi-Cheng Lin, Chia-Chun Lin, Wei-Chih Chen, Ho Lam Chung, Chen-An Li, Yi-Chang Chen, Chien-Yu Yu, Ming-Ji Lee, Chien- Cheng Chen, et al., “Breezyvoice: Adapting tts for taiwanese mandarin with enhanced polyphone disambiguation–challenges and insights,” arXiv prep...

    2025 arXiv

  9. [17]

    Leave no knowledge behind during knowledge distillation: Towards practical and effective knowledge distillation for code-switching asr using realistic data,

    Liang-Hsuan Tseng, Zih-Ching Chen, Wei-Shun Chang, Cheng-Kuang Lee, Tsung-Ren Huang, and Hung-yi Lee, “Leave no knowledge behind during knowledge distillation: Towards practical and effective knowledge distillation for code-switching asr using realistic data,” in2024 IEEE Spok...

    2024

  10. [18]

    Whispering in amharic: Fine-tuning whisper for low-resource language,

    Dawit Ketema Gete, Bedru Yimam Ahmed, Tadesse Destaw Belay, Yohannes Ayana Ejigu, Sukairaj Hafiz Imam, Alemu Belay Tessema, Mohammed Oumer Adem, Tadesse Amare Belay, Robert Geislinger, Umma Aliyu Musa, et al., “Whispering in amharic: Fine-tuning whisper for low-resource langua...

    2025 arXiv

  11. [19]

    Whispering in norwegian: Navigating orthographic and dialectic challenges,

    Per E Kummervold, Javier de la Rosa, Freddy Wetjen, Rolv-Arild Braaten, and Per Erik Solberg, “Whispering in norwegian: Navigating orthographic and dialectic challenges,” inProc. Interspeech 2024, 2024, pp. 3984–3988

    2024

  12. [20]

    Improving the inclusivity of dutch speech recognition by fine-tuning whisper on the jasmin-cgn corpus,

    Golshid Shekoufandeh, Paul Boersma, and Antal van den Bosch, “Improving the inclusivity of dutch speech recognition by fine-tuning whisper on the jasmin-cgn corpus,”arXiv preprint arXiv:2502.17284, 2025

    2025 arXiv

  13. [21]

    Whisper finetuning on nepali language,

    Sanjay Rijal, Shital Adhikari, Manish Dahal, Manish Awale, and Vaghawan Ojha, “Whisper finetuning on nepali language,”arXiv preprint arXiv:2411.12587, 2024

    2024 arXiv

  14. [22]

    Efficient adaptation of multilingual models for japanese asr,

    Mark Bajo, Haruka Fukukawa, Ryuji Morita, and Yuma Ogasawara, “Efficient adaptation of multilingual models for japanese asr,”arXiv preprint arXiv:2412.10705, 2024

    2024 arXiv

  15. [23]

    Fine-tuning whisper on low-resource languages for real-world applications,

    Vincenzo Timmel, Claudio Paonessa, Reza Kakooee, Manfred V ogel, and Daniel Perruchoud, “Fine-tuning whisper on low-resource languages for real-world applications,”arXiv preprint arXiv:2412.15726, 2024

    2024 arXiv

  16. [24]

    The NTNU ASR system for Formosa speech recognition challenge 2023,

    Hao-Chien Lu, Chung-Chun Wang, Jhen-Ke Lin, and Tien-Hong Lo, “The NTNU ASR system for Formosa speech recognition challenge 2023,” inProceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), Jheng-Long Wu and Ming-Hsiang Su, Eds., Ta...

    2023

  17. [25]

    Zero resource code-switched speech benchmark using speech utterance pairs for multiple spoken languages,

    Kuan-Po Huang, Chih-Kai Yang, Yu-Kuan Fu, Ewan Dunbar, and Hung- yi Lee, “Zero resource code-switched speech benchmark using speech utterance pairs for multiple spoken languages,” inICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICA...

    2024

  18. [26]

    IEEE, 2024, pp

    Chih-Kai Yang, Kuan-Po Huang, Ke-Han Lu, Chun-Yi Kuan, Chi-Yuan Hsiao, and Hung-yi Lee, “Investigating zero-shot generalizability on mandarin-english code-switched asr and speech-to-text translation of recent foundation models with self-supervision and weak supervision,” in202...

    2024

  19. [27]

    Zero-shot domain-sensitive speech recognition with prompt-conditioning fine-tuning,

    Feng-Ting Liao, Yung-Chieh Chan, Yi-Chang Chen, Chan-Jan Hsu, and Da-shan Shiu, “Zero-shot domain-sensitive speech recognition with prompt-conditioning fine-tuning,” in2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  20. [28]

    Prompting the hidden talent of web-scale speech models for zero-shot task generalization,

    Puyuan Peng, Brian Yan, Shinji Watanabe, and David Harwath, “Prompting the hidden talent of web-scale speech models for zero-shot task generalization,” inProc. Interspeech 2023, 2023, pp. 396–400

    2023

  21. [29]

    Can whisper perform speech-based in-context learning?,

    Siyin Wang, Chao-Han Yang, Ji Wu, and Chao Zhang, “Can whisper perform speech-based in-context learning?,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 13421–13425

    2024

  22. [30]

    High fidelity neural audio compression,

    Alexandre D ´efossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi, “High fidelity neural audio compression,”Transactions on Machine Learning Research

  23. [31]

    Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,

    Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng, “Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 591–595

    2024

  24. [32]

    Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms,

    Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al., “Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms,”CoRR, 2024

    2024

  25. [33]

    Denes and E

    P.B. Denes and E. Pinson,The Speech Chain, Anchor books. Worth Publishers, 1993

    1993

  26. [34]

    Listening while speaking: Speech chain by deep learning,

    Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura, “Listening while speaking: Speech chain by deep learning,” in2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 301–308

    2017

  27. [35]

    Machine speech chain with one-shot speaker adaptation,

    Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura, “Machine speech chain with one-shot speaker adaptation,”Interspeech 2018, 2018

    2018

  28. [36]

    Speech chain for semi-supervised learning of japanese-english code-switching asr and tts,

    Sahoko Nakayama, Andros Tjandra, Sakriani Sakti, and Satoshi Naka- mura, “Speech chain for semi-supervised learning of japanese-english code-switching asr and tts,” in2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 182–189

    2018

  29. [37]

    Montreal forced aligner [computer program],

    Michael McAuliffe, Michaela Socolof, Elias Stengel-Eskin, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger, “Montreal forced aligner [computer program],” http://montrealcorpustools.github. io/Montreal-Forced-Aligner/, 2017, Retrieved 05 May 2017

    2017

  30. [38]

    Fineweb2: A sparkling update with 1000s of languages,

    Guilherme Penedo, Hynek Kydl ´ıˇcek, Vinko Sabolˇcec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf, “Fineweb2: A sparkling update with 1000s of languages,” Dec. 2024

    2024

  31. [39]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” inProceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020, pp. 4211–4215

    2020

  32. [40]

    Ascend: A spontaneous chinese-english dataset for code- switching in multi-turn conversation,

    Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J Barezi, et al., “Ascend: A spontaneous chinese-english dataset for code- switching in multi-turn conversation,” inProceedings of the Thirteenth Language...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.