Pith. sign in

REVIEW 3 major objections 6 minor 52 references

A short spoken sample of any word can fix its pronunciation in zero-shot TTS while the rest of the sentence stays in a chosen target voice.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 08:15 UTC pith:36QIVER2

load-bearing objection Clean engineering fix for a real TTS failure mode: per-word audio grafting plus VC training, with solid ablations and a human study that actually separates systems. the 3 major comments →

arxiv 2607.02633 v1 pith:36QIVER2 submitted 2026-07-02 cs.LG cs.CL

GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech

Antonis Asonitis , Francesco Verdini , Aref Farhadipour , Vijeta Avijeet , Pierre-Edouard Honnet , Marzieh Razavi , Juan Pablo Zuluaga Gomez This is my paper
classification cs.LG cs.CL
keywords text-to-speechpronunciation controlaudio promptingvoice conversionzero-shot synthesisneural codec language modelsper-word conditioning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Text-to-speech systems that rely on spelling or even phonemes still mispronounce rare names, loanwords and technical terms because writing underdetermines how a word should sound. GRAFT solves this by letting a user supply a short recording of the difficult word; the model encodes that clip with its own speech tokenizer and splices the resulting tokens into the prompt exactly where the word belongs. Voice-conversion training forces the model to copy only pronunciation and prosody from the hint, so the hint can come from any speaker while the full sentence is still produced in the desired target voice. Human listeners rank GRAFT first for phonetic and prosodic match to a reference recording, and objective scores across five languages show a 22–39 % drop in target-word phoneme error rate over the identical text-only backbone, with speaker similarity and naturalness preserved. The practical upshot is that non-experts can correct a single word simply by saying it.

Core claim

GRAFT shows that a neural codec language model for text-to-speech can be given direct, per-word acoustic control of pronunciation without new parameters or architecture: a short spoken example of a chosen word is tokenized by the model’s own codec and bound to that word’s position in the prompt, and voice-conversion training data teaches the model to transfer only the pronunciation while rendering the whole utterance in a separately supplied target voice.

What carries the argument

GRAFT’s per-word graft slot: the target word’s text tokens are replaced by codec tokens of an isolated spoken hint (delimited by special tokens), combined with voice-converted training pairs that force speaker–content disentanglement so any-voice hints render in the cloned target voice.

Load-bearing premise

The method assumes that voice conversion applied while building the training data fully separates a word’s pronunciation from the speaker who said it; without that step the model simply pastes the hint’s voice through.

What would settle it

Train an otherwise identical model on the same data without any voice conversion of the per-word hints; if speaker similarity of the grafted word to the target voice remains high and D-PER stays low, the claimed need for disentanglement is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Non-experts can correct rare proper nouns, brand names or loanwords simply by speaking them once, without writing phonetic symbols or consulting a lexicon.
  • The same audio-conditioning slot can be stacked for several difficult words inside one sentence with only modest fluency cost.
  • Existing neural-codec TTS backbones can gain fine-grained pronunciation control at inference cost of only a few extra tokens and no architectural change.
  • Symbolic phoneme or dictionary interfaces become unnecessary for the hardest zero-shot cases once an acoustic example is available.
  • Multilingual systems can inherit the same control mechanism because the interface never depends on language-specific grapheme-to-phoneme rules.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same graft idea could later be extended to multi-word phrases or full prosodic contours once longer isolated references are supplied.
  • Because the control signal is raw audio tokens rather than phones, the method is a natural candidate for tonal languages where symbolic phonemizers discard tone diacritics.
  • If the disentanglement training generalizes, consumer-grade phone recordings of names could become a practical user interface for personalised TTS dictionaries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. GRAFT introduces a per-word pronunciation conditioning mechanism for neural codec language model TTS: a short spoken sample of a chosen word is encoded with the model’s own speech tokenizer and spliced into the prompt in place of that word’s text tokens, while a separate style reference supplies the target voice. Voice-conversion training data (Seed-VC) is used so that the model learns to copy pronunciation/prosody from the graft while ignoring the hint speaker. The method requires no new parameters. On a five-language difficult-word benchmark, GRAFT reduces target-word D-PER by 22–39% relative to the identical text-only Qwen3-TTS backbone and beats open-source phoneme- and text-conditioned zero-shot systems on pronunciation metrics, with speaker similarity and naturalness largely preserved. A 600-comparison English Bradley-Terry listening study ranks GRAFT first for phonetic/prosodic match to a reference recording of the difficult word. Ablations isolate voice-conversion training as necessary for speaker preservation.

Significance. If the results hold, GRAFT provides a practical, architecture-preserving interface for fine-grained pronunciation control that non-experts can use by simply saying a word—addressing a real failure mode of text- and phoneme-conditioned zero-shot TTS on rare names, loanwords and technical terms. Strengths include: (i) a controlled no-VC ablation that cleanly isolates the load-bearing role of voice-conversion training; (ii) consistent gains across five languages; (iii) an oracle-phoneme control ruling out transcription error as the sole cause of baseline deficits; (iv) a well-powered human pairwise study with non-overlapping confidence intervals; and (v) planned release of checkpoints, code and an openly licensed multilingual difficult-word benchmark. These make the contribution both usable and falsifiable.

major comments (3)
  1. [Table II / §V-A] Table II and §V-A: the no-VC ablation is decisive for the disentanglement claim, but the paper reports ECAPA-TDNN speaker scores for the grafted word (target 0.47 vs hint 0.12 with VC; reverse without) only in prose. These numbers should appear in Table II (or a dedicated small table) with the same item set as D-PER/SSIM, so the speaker-identity transfer claim is as auditable as the D-PER numbers.
  2. [Table IV / §VI] Table IV: stacking 4–5 grafts raises carrier WER from ~0.05 to ~0.22–0.25 while D-PER degrades only mildly. The Discussion presents multi-hint stacking as supported, but the fluency cost is large enough that the practical operating regime (how many simultaneous grafts remain usable) needs an explicit recommendation and, ideally, a short qualitative failure analysis of the seams. Without that, the multi-hint claim overreaches the reported numbers.
  3. [§IV-C / Table III] §IV-C / Table III: D-PER is the primary metric and is computed against the same audio hint GRAFT receives, while phoneme baselines receive a symbolic transcription of that hint. The paper correctly adds WS, the human study, and the CMUdict oracle (Table V) as cross-checks, but the main table still leads with D-PER alone. Either promote WS (or a joint ranking) into the primary pronunciation column set, or state more prominently in the table caption that D-PER is an audio-matched metric and that human/WS results are the decisive corroboration.
minor comments (6)
  1. [§III-C / Eq. (1)] Eq. (1) and the surrounding paragraph: p_s, p_a and p_hint are introduced without sensitivity analysis. A one-sentence note on whether results are stable under modest changes (or a pointer to a short appendix sweep) would help reproducibility.
  2. [Fig. 1 / §III-B] Fig. 1 caption and §III-B: the placeholder □ and the replacement of text embeddings by summed residual-codebook embeddings are clear in the text but dense; a short schematic of the composite prompt token sequence would help readers less familiar with Qwen3-TTS.
  3. [§VI / Abstract] §VI Limitations: the non-tonal benchmark and single-word interface are acknowledged; consider elevating the tonal-language caveat earlier (e.g., in the abstract or contributions) since the introduction motivates GRAFT partly against phonemizers that discard diacritics/tone.
  4. [Table III] Table III: single-speaker systems (Matcha, Piper) are correctly excluded from the SSIM ranking via †, but the caption could also note that their SSIM is not comparable rather than merely low.
  5. [Throughout] Minor typos/consistency: “V oice” appears with a space in several headings (e.g., §II-D, Table II caption); “CosyV oice2” likewise. Standardize to “Voice” / “CosyVoice2”.
  6. [References] References: arXiv IDs for contemporaneous systems (Qwen3-TTS, SonoEdit, etc.) are fine for a preprint, but journal version should update to published venues where available.

Circularity Check

0 steps flagged

No significant circularity: empirical TTS method evaluated on held-out human recordings, external systems, and a blind listening study.

full rationale

GRAFT is an engineering method paper, not a first-principles derivation. The central claims (22–39% D-PER reduction vs. the identical text-only backbone, outperformance of open-source zero-shot systems, first place in a Bradley-Terry human study) are measured against external held-out Lingua Libre isolated-word recordings, ZIPA/Whisper/WavLM metrics, and paid human raters, none of which are fitted parameters of the model. Training constructs voice-converted per-word grafts via Seed-VC and fine-tunes with standard next-token CE; evaluation is on a separate difficult-word benchmark. The no-VC ablation (Table II) and ECAPA-TDNN speaker checks explicitly test the load-bearing disentanglement assumption rather than assuming it. Use of the base model’s own codec tokenizer and aligner for data prep is standard architectural reuse, not a self-definitional or self-citation loop that forces the reported gains. No uniqueness theorems, fitted-then-predicted quantities, or renamed known results appear. Score 0 is the correct honest finding.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 1 invented entities

Empirical ML paper whose central claim rests on a small set of training-data construction choices and standard neural-codec assumptions rather than on free physical constants or invented particles. The free parameters are the usual hyper-parameters that control how often and how noisily grafts appear; the domain assumptions are those of the base Qwen3-TTS codec LM and of the Seed-VC model used only for data prep.

free parameters (5)
  • p_hint (probability a word is grafted) = 0.5
    Set to 0.5; controls how frequently the model sees per-word audio slots during training and therefore how strongly it learns the grafting behaviour.
  • p_s (cross-utterance swap probability) = 0.3
    Set to 0.3; forces the model to see isolated rather than coarticulated grafts, matching inference.
  • p_a (augmentation probability) = 0.5
    Set to 0.5; broadens robustness to noisy or loosely trimmed consumer recordings.
  • lambda_sub (sub-talker loss weight) = 0.3
    Set to 0.3; balances the auxiliary codec loss.
  • learning rate / schedule = 5e-6
    5e-6 with cosine decay and 0.02 warmup; standard but chosen by hand for the 0.6 B backbone.
axioms (3)
  • domain assumption Discrete residual codec tokens produced by the base model’s own tokenizer already contain sufficient information to specify a word’s pronunciation and stress.
    Invoked throughout Section III-B; without it the graft slot would be empty of phonetic content.
  • domain assumption Seed-VC (D=20) produces voice-converted copies that preserve phonetic content while changing speaker identity sufficiently for the disentanglement objective.
    Central to the training-data construction of Section III-C and validated only by the no-VC ablation.
  • domain assumption ZIPA phone recogniser yields a reliable phoneme error rate between a human reference clip and a synthesised target word.
    Used as the primary D-PER metric in Table III; partially cross-checked by Whisper similarity and human ratings.
invented entities (1)
  • GRAFT per-word graft slot (<B> … </B>) no independent evidence
    purpose: Binds an arbitrary-length sequence of codec tokens to a specific word position in the text prompt without adding new model parameters.
    The slot delimiters and the replacement of text embeddings by summed residual-codebook embeddings are introduced by the paper; they have no independent existence outside this architecture.

pith-pipeline@v1.1.0-grok45 · 20283 in / 2662 out tokens · 30474 ms · 2026-07-12T08:15:35.888070+00:00 · methodology

0 comments
read the original abstract

We present GRAFT, a per-word pronunciation conditioning mechanism for text-to-speech neural codec language modeling. Existing systems reach high intelligibility and naturalness but inherit the ambiguity of text and mispronounce rare proper nouns, loanwords and technical terms. Even phoneme-conditioned models offer no direct acoustic handle for per-word pronunciation. GRAFT controls the pronunciation of a chosen word from a short spoken sample of it, encoded with the model's own speech tokenizer and bound to the word's position in the prompt. Voice conversion during training-data construction disentangles the hint speaker from the target speaker, so the hint may come from any voice while the output stays in the target voice. In a blind English listening study, human raters rank GRAFT first by a clear margin, judging its rendering of the difficult word closest to a reference recording of that word. On a five-language objective benchmark, GRAFT reduces target-word phoneme error rate by 22-39% over the identical text-only backbone and outperforms competitive open-source zero-shot systems, both phoneme- and text-conditioned, on target-word pronunciation, while preserving speaker similarity and naturalness.

Figures

Figures reproduced from arXiv: 2607.02633 by Antonis Asonitis, Aref Farhadipour, Francesco Verdini, Juan Pablo Zuluaga Gomez, Marzieh Razavi, Pierre-Edouard Honnet, Vijeta Avijeet.

Figure 1
Figure 1. Figure 1: GRAFT at inference. A spoken example of the word (“GRAFT”), [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training-data construction. From a Speaker A utterance we make [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Blind pairwise human listening study on English (ratings initialised [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 3 canonical work pages

  1. [1]

    Natural TTS synthesis by conditioning wavenet on MEL spectrogram predictions,

    J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Ryan, R. A. Saurous, Y . Agiomyrgiannakis, and Y . Wu, “Natural TTS synthesis by conditioning wavenet on MEL spectrogram predictions,” in2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-2...

    doi:10.1109/icassp.2018.8461368 2018

  2. [2]

    Fastspeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” in9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=piLPYqxtWuA

    2021

  3. [4]

    Available: https://arxiv.org/abs/2301.02111

    [Online]. Available: https://arxiv.org/abs/2301.02111

    Pith/arXiv arXiv

  4. [5]

    Llasa: Scaling train-time and inference-time compute for Llama-based speech synthesis,

    Z. Ye, X. Zhu, C.-M. Chanet al., “Llasa: Scaling train-time and inference-time compute for Llama-based speech synthesis,” ArXiv preprint, vol. abs/2502.04128, 2025. [Online]. Available: https://arxiv.org/abs/2502.04128

    Pith/arXiv arXiv 2025

  5. [6]

    Qwen3-TTS technical report,

    Qwen Team, “Qwen3-TTS technical report,”ArXiv preprint, vol. abs/2601.15621, 2026. [Online]. Available: https://arxiv.org/abs/2601. 15621

    Pith/arXiv arXiv 2026

  6. [7]

    CosyV oice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Zhanget al., “CosyV oice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”ArXiv preprint, vol. abs/2407.05407, 2024. [Online]. Available: https://arxiv.org/abs/2407.05407

    Pith/arXiv arXiv 2024

  7. [8]

    Phonemizer: Text to phones transcription for multiple languages in Python,

    M. Bernard and H. Titeux, “Phonemizer: Text to phones transcription for multiple languages in Python,”Journal of Open Source Software, vol. 6, no. 68, p. 3958, 2021

    2021

  8. [9]

    eSpeak NG text-to-speech,

    J. Duddingtonet al., “eSpeak NG text-to-speech,” https://github.com/ espeak-ng/espeak-ng, accessed 2025

    2025

  9. [10]

    ZIPA: A family of efficient models for multilingual phone recognition,

    J. Zhu, F. Samir, E. Chodroff, and D. R. Mortensen, “ZIPA: A family of efficient models for multilingual phone recognition,” inProc. ACL, 2025, pp. 19 568–19 585

    2025

  10. [11]

    Simple and effective zero-shot cross-lingual phoneme recognition,

    Q. Xu, A. Baevski, and M. Auli, “Simple and effective zero-shot cross-lingual phoneme recognition,” in23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, H. Ko and J. H. L. Hansen, Eds. ISCA, 2022, pp. 2113–2117. [Online]. Available: https://doi.org/10.21437/Interspeech.2022-60

    doi:10.21437/interspeech.2022-60 2022

  11. [12]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,

    K. Shen, Z. Ju, X. Tan, E. Liu, Y . Leng, L. He, T. Qin, S. Zhao, and J. Bian, “Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers,” inThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. [Online]. Available: https://openreview.ne...

    2024

  12. [13]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, E. Liu, Y . Leng, K. Song, S. Tang, Z. Wu, T. Qin, X. Li, W. Ye, S. Zhang, J. Bian, L. He, J. Li, and S. Zhao, “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” inForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024....

    2024

  13. [14]

    V oicebox: Text- guided multilingual universal speech generation at scale,

    M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V . Manohar, Y . Adi, J. Mahadeokar, and W. Hsu, “V oicebox: Text- guided multilingual universal speech generation at scale,” inAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Dece...

    2023

  14. [15]

    MaskGCT: Zero-shot text-to-speech with masked generative codec transformer,

    Y . Wang, H. Zhan, L. Liuet al., “MaskGCT: Zero-shot text-to-speech with masked generative codec transformer,” inProc. ICLR, 2025

    2025

  15. [16]

    Speak, read and prompt: High-fidelity text-to-speech with minimal supervision,

    E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, “Speak, read and prompt: High-fidelity text-to-speech with minimal supervision,”Transactions of the Association for Computational Linguistics, vol. 11, pp. 1703–1718,

  16. [17]

    Available: https://aclanthology.org/2023.tacl-1.95

    [Online]. Available: https://aclanthology.org/2023.tacl-1.95

    2023

  17. [18]

    Word- level emotional expression control in zero-shot text-to-speech synthesis,

    T. Wang, H. Wang, M. Ge, C. Gong, C. Qiang, Z. Ma, Z. Huang, G. Yang, X. Wang, E. S. Chng, X. Chen, L. Wang, and J. Dang, “Word- level emotional expression control in zero-shot text-to-speech synthesis,” inAdvances in Neural Information Processing Systems 38 (NeurIPS 2025), 2025

    2025

  18. [19]

    Glow-tts: A generative flow for text-to-speech via monotonic alignment search,

    J. Kim, S. Kim, J. Kong, and S. Yoon, “Glow-tts: A generative flow for text-to-speech via monotonic alignment search,” inAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020....

    2020

  19. [20]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,

    J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” inProceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 5530–5540. [Onli...

    2021

  20. [21]

    Png BERT: augmented BERT on phonemes and graphemes for neural TTS,

    Y . Jia, H. Zen, J. Shen, Y . Zhang, and Y . Wu, “Png BERT: augmented BERT on phonemes and graphemes for neural TTS,” inInterspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021, H. Hermansky, H. Cernock ´y, L. Burget, L. Lamel, O. Scharenborg, and P. Motl ´ıcek, Eds. ISCA, ...

    work page doi:10.21437/interspeech.2021-1757 2021

  21. [22]

    Byt5 model for massively multilingual grapheme-to-phoneme conversion,

    J. Zhu, C. Zhang, and D. Jurgens, “Byt5 model for massively multilingual grapheme-to-phoneme conversion,” in23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, H. Ko and J. H. L. Hansen, Eds. ISCA, 2022, pp. 446–450. [Online]. Available: https://doi.org/10.21437/Interspeech...

    work page doi:10.21437/interspeech.2022-538 2022

  22. [23]

    Neural lexicon reader: Reduce pronunciation errors in end-to-end TTS by leveraging external textual knowledge,

    M. He, J. Yang, L. He, and F. K. Soong, “Neural lexicon reader: Reduce pronunciation errors in end-to-end TTS by leveraging external textual knowledge,” in23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, H. Ko and J. H. L. Hansen, Eds. ISCA, 2022, pp. 441–445. [Online]. ...

    work page doi:10.21437/interspeech.2022-420 2022

  23. [24]

    SonoEdit: Null-space constrained knowledge editing for pronunciation correction in LLM-based TTS,

    A. Singhet al., “SonoEdit: Null-space constrained knowledge editing for pronunciation correction in LLM-based TTS,”ArXiv preprint, vol. abs/2601.17086, 2026. [Online]. Available: https: //arxiv.org/abs/2601.17086

    arXiv 2026

  24. [25]

    Montreal forced aligner: Trainable text-speech alignment using kaldi,

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi,” inInterspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, F. Lacerda, Ed. ISCA, 2017, pp. 498–502. [Online]. Available: http: //www.isca-sp...

    2017

  25. [26]

    CTC- segmentation of large corpora for German end-to-end speech recogni- tion,

    L. K ¨urzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll, “CTC- segmentation of large corpora for German end-to-end speech recogni- tion,” inProc. SPECOM, 2020, pp. 267–278

    2020

  26. [27]

    whisper-timestamped,

    J. Louradour, “whisper-timestamped,” https://github.com/linto-ai/ whisper-timestamped, accessed 2026

    2026

  27. [28]

    stable-ts: Transcription, forced alignment, and audio indexing with OpenAI’s Whisper,

    F. C. Jian, “stable-ts: Transcription, forced alignment, and audio indexing with OpenAI’s Whisper,” https://github.com/jianfch/stable-ts, accessed 2026

    2026

  28. [29]

    Whisperx: Time-accurate speech transcription of long-form audio,

    M. Bain, J. Huh, T. Han, and A. Zisserman, “Whisperx: Time-accurate speech transcription of long-form audio,” in24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, N. Harte, J. Carson-Berndsen, and G. Jones, Eds. ISCA, 2023, pp. 4489–4493. [Online]. Available: https://doi.org...

    doi:10.21437/interspeech.2023-78 2023

  29. [30]

    V oice conversion with just nearest neighbors,

    M. Baas, B. van Niekerk, and H. Kamper, “V oice conversion with just nearest neighbors,” in24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, August 20-24, 2023, N. Harte, J. Carson-Berndsen, and G. Jones, Eds. ISCA, 2023, pp. 2053–2057. [Online]. Available: https://doi.org/10.21437/Interspeech...

    doi:10.21437/interspeech.2023-419 2023

  30. [31]

    Diffusion-based voice conversion with fast maximum likelihood sampling scheme,

    V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, M. S. Kudinov, and J. Wei, “Diffusion-based voice conversion with fast maximum likelihood sampling scheme,” inThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. [Online]. Available: https: //openreview.net/forum?id=8c50f-DoW Au

    2022

  31. [32]

    Zero-shot voice conversion with diffusion transformers,

    S. Liu, “Zero-shot voice conversion with diffusion transformers,” ArXiv preprint, vol. abs/2411.09943, 2024. [Online]. Available: https://arxiv.org/abs/2411.09943

    Pith/arXiv arXiv 2024

  32. [33]

    Autovc: Zero-shot voice style transfer with only autoencoder loss,

    K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “Autovc: Zero-shot voice style transfer with only autoencoder loss,” inProceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97....

    2019

  33. [34]

    Contentvec: An improved self-supervised speech representation by disentangling speakers,

    K. Qian, Y . Zhang, H. Gao, J. Ni, C. Lai, D. D. Cox, M. Hasegawa- Johnson, and S. Chang, “Contentvec: An improved self-supervised speech representation by disentangling speakers,” inInternational Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L...

    2022

  34. [35]

    Neural analysis and synthesis: Reconstructing speech from self-supervised representations,

    H. Choi, J. Lee, W. Kim, J. Lee, H. Heo, and K. Lee, “Neural analysis and synthesis: Reconstructing speech from self-supervised representations,” inAdvances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, M. Ranzato, A. Beygelzimer, Y . N. Dauphin, P....

    arXiv 2021

  35. [36]

    CSTR VCTK corpus: En- glish multi-speaker corpus for CSTR voice cloning toolkit,

    J. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTK corpus: En- glish multi-speaker corpus for CSTR voice cloning toolkit,” University of Edinburgh, CSTR, Tech. Rep., 2019

    2019

  36. [37]

    Libritts: A corpus derived from librispeech for text-to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text-to-speech,” inInterspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, G. Kubin and Z. Kacic, Eds. ISCA, 2019, pp. 1526–1530. [Online]. Available...

    doi:10.21437/interspeech.2019-2441 2019

  37. [38]

    MLS: A large-scale multilingual dataset for speech research,

    V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A large-scale multilingual dataset for speech research,” in Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 2757–2761. [Online]. Availab...

    2020

  38. [39]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” inProceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, F. B ´echet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. ...

    2020

  39. [40]

    Evaluating multilingual speech translation under realistic conditions with resegmentation and terminology,

    E. Salesky, K. Darwish, M. Al-Badrashiny, M. Diab, and J. Niehues, “Evaluating multilingual speech translation under realistic conditions with resegmentation and terminology,” inProceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), E. Salesky, M. Federico, and M. Carpuat, Eds. Toronto, Canada (in-person and online):...

    2023

  40. [41]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  41. [42]

    Silero V AD: pre-trained enterprise-grade voice activity detector,

    Silero Team, “Silero V AD: pre-trained enterprise-grade voice activity detector,” https://github.com/snakers4/silero-vad, accessed 2025

    2025

  42. [43]

    Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,

    Y . A. Li, C. Han, V . S. Raghavan, G. Mischler, and N. Mesgarani, “Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA,...

    2023

  43. [44]

    Available: http://papers.nips.cc/paper\ files/paper/2023/ hash/3eaad2a0b62b5ed7a2e66c2188bb1449-Abstract-Conference.html

    [Online]. Available: http://papers.nips.cc/paper\ files/paper/2023/ hash/3eaad2a0b62b5ed7a2e66c2188bb1449-Abstract-Conference.html

    2023

  44. [45]

    Matcha-tts: A fast TTS architecture with conditional flow matching,

    S. Mehta, R. Tu, J. Beskow, ´E. Sz ´ekely, and G. E. Henter, “Matcha-tts: A fast TTS architecture with conditional flow matching,” inIEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14- 19, 2024. IEEE, 2024, pp. 11 341–11 345. [Online]. Available: https://doi.org/10.1109/ICASSP48485.202...

    doi:10.1109/icassp48485.2024.10448291 2024

  45. [46]

    Piper: A fast, local neural text-to-speech system,

    M. Hansenet al., “Piper: A fast, local neural text-to-speech system,” https://github.com/rhasspy/piper, accessed 2025

    2025

  46. [47]

    XTTS: A massively multilingual zero-shot text-to-speech model,

    E. Casanova, K. Davis, E. G ¨olge, G. G ¨oknar, I. Gulea, L. Hart, A. Alja- fari, J. Meyer, R. Morais, S. Olayemi, and J. Weber, “XTTS: A massively multilingual zero-shot text-to-speech model,” inProc. INTERSPEECH, 2024

    2024

  47. [48]

    F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Maet al., “F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching,”ArXiv preprint, vol. abs/2410.06885, 2024. [Online]. Available: https: //arxiv.org/abs/2410.06885

    Pith/arXiv arXiv 2024

  48. [49]

    CosyV oice 2: Scalable streaming speech synthesis with large language models,

    Z. Du, Y . Wang, Q. Chenet al., “CosyV oice 2: Scalable streaming speech synthesis with large language models,”ArXiv preprint, vol. abs/2412.10117, 2024. [Online]. Available: https: //arxiv.org/abs/2412.10117

    Pith/arXiv arXiv 2024

  49. [50]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett,...

    2023

  50. [51]

    UTMOS: utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: utokyo-sarulab system for voicemos challenge 2022,” in23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, H. Ko and J. H. L. Hansen, Eds. ISCA, 2022, pp. 4521–4525. [Online]. Available: https...

    doi:10.21437/interspeech.2022-439 2022

  51. [52]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chenet al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  52. [53]

    ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” inInterspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, p...

    doi:10.21437/interspeech.2020-2650 2020