Pith. sign in

REVIEW 4 major objections 5 minor 75 references

CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CrossSpeech++ claims that splitting speech generation into language- and speaker-dependent branches, trained with self-supervised linguistic targets, yields more natural cross-lingual speech.

desk verdict Competent incremental extension of CrossSpeech with solid ablations, but the headline claims outrun the evidence and the disentanglement story is only indirectly supported. read the letter →

arxiv 2412.20048 v1 pith:74LOCNPQ submitted 2024-12-28 eess.AS cs.AIcs.SDeess.SP

Ji-Hoon Kim , Hong-Sun Yang , Yoon-Cheol Ju , Il-Hwan Kim , Byeong-Yeol Kim , Joon Son Chung This is my paper · ORCID
classification eess.AScs.AIcs.SDeess.SP
keywords cross-lingualspeechsynthesislanguage-speakerdisentanglementtext-to-speechself-supervisedrepresentationsspeakergeneralizationprosodymodelingmultilingualTTSMixDynamicLayerNormalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CrossSpeech++ tries to close the gap between cross-lingual text-to-speech (speaking a language the target speaker never used in training) and ordinary same-language text-to-speech by attacking the language-speaker entanglement problem at the output stage rather than only at the input stage. It splits speech generation into a language-dependent generator, which produces pronunciation and intonation patterns free of speaker bias, and a speaker-dependent generator, which colors those patterns with timbre and pitch. The two branches are trained together so that their summed mel-spectrogram matches the target, with extra losses steering each branch toward only its own kind of information. On a four-language corpus of monolingual speakers, the paper reports large gains in naturalness and intelligibility over prior systems, including its own earlier CrossSpeech model, in both cross-lingual and same-language settings. If these results hold, separating generation this way could make high-quality multilingual voices practical from ordinary monolingual data.

What carries the argument

The central object is the additive decomposition of the mel-spectrogram into language-dependent and speaker-dependent representations, each produced by its own branch of a FastPitch-style non-autoregressive TTS model. The load-bearing components are MDSLN, a MixStyle-inspired layer normalization that perturbs the LD encoder with statistics from randomly shuffled speaker embeddings (mixing weight drawn from Beta(2,2)); the LDV adaptor, which turns ground-truth pitch and energy into binary up/down sequences per text token so the language branch learns direction of intonation but not absolute pitch; the linguistic adaptor, which trains an LD decoder to predict the last-hidden-layer features of MMS computed on information-perturbed waveforms (formant shifting, pitch randomization, frequency shaping) and reinforces them with a CTC text predictor; and, in the SDG, DSLN plus the SDV adaptor, which predict continuous pitch and energy to restore speaker-specific prosody. The argument is that by making the language branch predict only speaker-shifted, binary, and perturbed targets, speaker information cannot be encoded in it, while the speaker branch must carry the residual identity.

What would settle it

A direct test is to train a speaker classifier on the language-dependent features produced by the LDG for a held-out set of speakers and languages: if it can predict speaker identity above chance, the LD branch has not actually removed speaker information. A complementary check is to swap only the language embedding while holding the speaker fixed and measure speaker-embedding cosine similarity of the output; a large drop in similarity would indicate speaker leakage into the language branch.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that disentangling language and speaker in the output acoustic feature space, by assigning each to a separate generator, is more effective than disentangling only in the input token space. The language-dependent generator is trained to be speaker-generalizable through Mix Dynamic Speaker Layer Normalization, which mixes randomly shuffled speaker statistics during training, plus a language-dependent variance adaptor that predicts binary pitch and energy directions and a linguistic adaptor that regresses self-supervised features extracted from a perturbed waveform by a frozen wav2vec2.0 model (MMS, last hidden layer). The speaker-dependent generator uses Dynamic Speaker Layer Normalization and a speaker-dependent variance adaptor to add timbre and pitch contours. The final mel-spectrogram is the sum of the two branches' outputs, and the full model is trained with a multi-term loss including mel reconstruction, duration, LD and SD variance, linguistic feature L1, and CTC text prediction. The paper reports that this scheme achieves the best naturalness scores (MOS and UTMOS) and the lowest character error rate (CER) among compared cross-lingual systems, with only a small dip in speaker-similarity scores relative to CrossSpeech, which it attributes to more accurate text-driven pronunciation.

Load-bearing premise

The load-bearing premise is that the last hidden layer of the MMS model, applied to waveforms after formant shifting, pitch randomization, and frequency shaping, contains the linguistic content with speaker identity largely removed; if those features still carry residual speaker or language artifacts, the language-dependent generator will encode entangled information and the disentanglement claim collapses.

Editorial extensions

If this is right

  • Cross-lingual synthesis on monolingual corpora can approach intra-lingual quality: the paper reports its cross-lingual MOS (4.06) exceeding all baselines and its CER (13.35) below them.
  • Adding a new language does not require a pre-computed aligner, because the online duration aligner produces ground-truth durations inside training.
  • The SSL-based linguistic adaptor is the single most important module: removing it drops cross-lingual UTMOS from 3.791 to 3.443 in the paper's ablation.
  • The system also improves intra-lingual synthesis, so the decoupled generators are not just a cross-lingual fix but a general TTS design.
  • Compared with zero-shot systems, the paper's model yields higher naturalness (UTMOS 3.863 vs 3.450 for XTTS-v2) and speaker similarity (SECS 0.767 vs 0.763) when both are evaluated on English, Chinese, and Japanese.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the output-space disentanglement is real, the same two-branch additive design could be applied to voice conversion or accent transfer, where one wants to keep linguistic content fixed while swapping only speaker identity.
  • Because the linguistic targets come from MMS, the method's benefit likely depends on how well MMS covers the target language; a testable prediction is that CER and UTMOS degrade for low-resource languages under-represented in MMS's 1,400-language pretraining.
  • The binary pitch and energy targets discard the magnitude of prosodic movement; using continuous, speaker-normalized targets might preserve the disentanglement while capturing more expressive contours.
  • The slight drop in speaker similarity relative to CrossSpeech suggests a controllable trade-off: weighting LD versus SD outputs could let a user tune between accent fidelity and voice identity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CrossSpeech++, a non-autoregressive cross-lingual TTS system built on FastPitch with conformer blocks. The architecture splits generation into a Language-dependent Generator (LDG) and a Speaker-dependent Generator (SDG), whose outputs are summed to obtain the mel-spectrogram. The LDG uses Mix-Dynamic Speaker Layer Normalization (MDSLN), a language-dependent variance (LDV) adaptor, and a linguistic adaptor that regresses SSL features extracted from perturbed waveforms using a frozen MMS model; the SDG uses Dynamic Speaker Layer Normalization (DSLN) and a speaker-dependent variance (SDV) adaptor. Experiments cover four languages with 22 speakers and compare against FastPitch variants, CrossSpeech, and zero-shot models, reporting MOS, SMOS, UTMOS, SECS, and CER. The central claim is that this decoupling effectively disentangles language and speaker information in the output acoustic feature space, yielding significant cross-lingual quality improvements.

Significance. If the claims are substantiated, the contribution is a practical architectural recipe for cross-lingual TTS that moves disentanglement from input representations into the output acoustic feature space, building on the authors' prior CrossSpeech. The paper includes a useful set of ablations, an online aligner that avoids per-language forced alignment, and a comparison across several MMS layers; the demo page and use of public corpora and component models substantially support reproducibility of the system itself. The evaluation is broader than in many TTS papers (MOS, SMOS, UTMOS, SECS, CER, plus ablations), which is a strength. However, the statistical support for the headline claims and the quantitative evidence for the disentanglement mechanism are weaker than the abstract's language suggests, and the zero-shot comparison is not a matched experimental protocol.

major comments (4)
  1. [Table II, Abstract] The abstract claims 'significant improvements' and 'outperforming existing methods by a large margin,' but the numbers in Table II do not consistently support this. Cross-lingual MOS for CrossSpeech++ is 4.06 ± 0.09 vs. 3.93 ± 0.08 for CrossSpeech, which at face value is a 0.13 difference with overlapping 95% confidence intervals and no significance test; SMOS is actually lower (3.82 vs. 3.87) and SECS is lower (0.761 vs. 0.776). Objective metrics are reported as point estimates with no error bars or significance tests. Please provide significance testing for MOS/SMOS (e.g., pairwise tests with multiple-comparison correction) and error estimates for UTMOS, SECS, and CER, or temper the wording of the claims so that they do not overstate the evidence.
  2. [Sec. IV-C, Fig. 6, Table IV] The central disentanglement claim rests on the assertion that the LDG output is essentially free of speaker information, but the only direct evidence is the t-SNE in Fig. 6(a), which is qualitative and can hide clustering structure. The paper's own ablation in Table IV shows that removing the linguistic adaptor (w/oLA) raises SECS in both cross-lingual (0.772 vs. 0.761) and intra-lingual (0.783 vs. 0.777) conditions, and Sec. VII-C attributes this to 'residual speaker information entangled within the text representations.' This is an explicit acknowledgment that the language pathway still carries speaker-dependent information. Please add a quantitative speaker-leakage metric computed directly on LD features (e.g., speaker classification accuracy, or the average cosine similarity of LD features from the same speaker vs. different speakers) and report it for the full model and for ablations. Without such a measurement, the claim that the architecture 'effectively disentangles' language and speaker is not established.
  3. [Sec. VII-E, Table V] The comparison with VALL-E X and XTTS-v2 is not a matched apples-to-apples evaluation. CrossSpeech++ is trained and evaluated on utterances from the same 22-speaker corpus, including the test speakers (the 80/10/10 split in Sec. VI-A is within-speaker), whereas VALL-E X and XTTS-v2 are zero-shot systems that have not been adapted to the target speakers. In addition, Table V excludes Korean because VALL-E X does not support it, changing the test set relative to the main evaluation. This protocol difference biases the comparison in favor of CrossSpeech++. Please either (a) retrain or fine-tune CrossSpeech++ in a strict zero-shot protocol with reference utterances from unseen speakers, or (b) clearly relabel the comparison as 'trained speaker-dependent vs. zero-shot' and add a speaker-dependent baseline such as adapted FastPitch so that the effect of the proposed modules can be isolated.
  4. [Sec. VII-A, Table II] The cross-lingual and intra-lingual conditions are not directly comparable because cross-lingual evaluation uses only four representative speakers per language while intra-lingual uses all speakers. This is visible in the results: for every system, UTMOS is higher in the cross-lingual condition than in the intra-lingual condition (e.g., CrossSpeech++ 3.791 vs. 3.343; CrossSpeech 3.279 vs. 3.039), which is surprising given the usual finding that cross-lingual synthesis is harder. Please clarify the test-set composition (number of utterances per speaker, speaker overlap across languages) and explain this pattern. If the metric is sensitive to speaker identity or language, report per-language and per-condition results so that the reader can assess whether the cross-lingual gains are driven by a small set of favorable speakers.
minor comments (5)
  1. [Fig. 2 caption] The caption contains the typo 'architecutre'; please correct it.
  2. [Eq. (5)] In Eq. (5), the index i-1 at i=1 references an out-of-range value; please specify how the sequence is padded (e.g., p_0 defined as the first value or treated as equal).
  3. [Sec. IV-B] Binarizing pitch and energy discards the magnitude of rises and falls; a sentence of justification or a reference showing that binary contours suffice for the intended prosody modeling would help the reader evaluate this design choice.
  4. [Table II] The row for Ground Truth reports only intra-lingual scores; please also report cross-lingual ground-truth scores (or state that these are unavailable), as the current format makes it hard to judge the absolute gap to natural speech.
  5. [Sec. VI-E] Please report the number of utterances used in the subjective tests (the text says 50 clips, but it is unclear whether that is the total or per condition) and the inter-rater agreement, so that the confidence intervals can be interpreted.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found; self-citations are contextual and the central evidence rests on external metrics and ablations.

full rationale

Walking the derivation chain, no load-bearing step reduces to its own inputs. The linguistic adaptor targets are extracted from frozen MMS features on perturbed waveforms (Sec. IV-C); MMS is an externally pre-trained wav2vec2.0 model, and the targets are not recomputed from the model's own outputs or from the evaluation metrics. The pitch and energy targets in Sec. IV-B are derived from ground-truth speech via averaging and binarization (Eq. 5), but the LD predictors are trained to predict those derived codes and are then evaluated with external MOS/UTMOS/SECS/CER; the derivation is not the evaluation. The choice of MMS layer (Sec. VII-B) is a validation-set hyperparameter choice, not a fitted constant relabeled as a prediction. The self-citations (CrossSpeech [38], DSLN [45], Fre-GAN [66]) are contextual or supply standard components and baselines; they do not carry the central evidence, which rests on external metrics and ablations. The paper's disentanglement claim does rely on an unvalidated assumption that perturbed MMS features are sufficiently speaker-independent, and the w/oLA SECS increase in Table IV suggests possible residual speaker leakage, but this is an empirical validity concern rather than a circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical assumptions about SSL features, additivity of acoustic representations, and speaker-invariance of binary prosodic targets. The hand-fitted choices that directly affect the reported quality are the MMS layer index (24), the MixStyle alpha (2), and the auxiliary loss weights (0.1).

free parameters (3)
  • MMS layer index for linguistic features = 24
    Selected based on validation UTMOS/SECS/CER in Table III; Section VII-B states layer 24 is chosen because it provides improved naturalness and speaker similarity with slight reduction in intelligibility.
  • MixStyle Beta distribution shape alpha = 2
    Chosen by hand for MDSLN speaker mixing in Section IV-A.
  • Auxiliary loss weights lambda = 0.1 for all auxiliary losses
    Fixed to 0.1 in Section V for duration, LD pitch/energy, linguistic, CTC, and SD pitch/energy losses.
assumptions (4)
  • domain assumption SSL features from the last layer of MMS, after information perturbation, encode language-dependent content with speaker information removed.
    Central to the linguistic adaptor target in Section IV-C and Fig. 4.
  • domain assumption The speech generation pipeline can be decomposed into additive language-dependent and speaker-dependent acoustic representations, with the mel-spectrogram as their sum.
    Architecture in Fig. 1 and Section V assume linear additivity in mel space.
  • domain assumption Binary rise/fall patterns of pitch and energy are speaker-independent linguistic variations.
    LD targets in Section IV-B are binarized pitch and energy, discarding speaker-specific magnitude information.
  • domain assumption The pre-trained vocoder Fre-GAN adds no artifact that changes comparative conclusions.
    All models use the same vocoder (Section VI-D), so differences are attributed to the acoustic model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation." pith.science (2026). https://pith.science/paper/74LOCNPQ

@misc{pith2026241220048,
  author       = {Pith},
  title        = {Pith review of: CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/74LOCNPQ}},
  note         = {Machine review of arXiv:2412.20048}
}
read the original abstract

The goal of this work is to generate natural speech in multiple languages while maintaining the same speaker identity, a task known as cross-lingual speech synthesis. A key challenge of cross-lingual speech synthesis is the language-speaker entanglement problem, which causes the quality of cross-lingual systems to lag behind that of intra-lingual systems. In this paper, we propose CrossSpeech++, which effectively disentangles language and speaker information and significantly improves the quality of cross-lingual speech synthesis. To this end, we break the complex speech generation pipeline into two simple components: language-dependent and speaker-dependent generators. The language-dependent generator produces linguistic variations that are not biased by specific speaker attributes. The speaker-dependent generator models acoustic variations that characterize speaker identity. By handling each type of information in separate modules, our method can effectively disentangle language and speaker representation. We conduct extensive experiments using various metrics, and demonstrate that CrossSpeech++ achieves significant improvements in cross-lingual speech synthesis, outperforming existing methods by a large margin.

Figures

Figures reproduced from arXiv: 2412.20048 by the authors.

Figure 1
Figure 1. CrossSpeech++ operates as follows: From text inputs, the language-dependent generator produces language-dependent representations that capture [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall architecutre of CrossSpeech++. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Batch-wise shuffle operation. es is the speaker embeddings and e˜s denotes the shuffled speaker embeddings. concatenation. Given hidden representations h and speaker embeddings es , the speaker-conditioned representations are derived as follows: DSLN(h, es) = W(es) ∗ LN(h) + b(es), (1) where ∗ denotes 1D convolution, and LN refers to layer normalization. The normalized hidden feature space is then shifted according … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: A pipeline for extracting the target linguistic features from waveform. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Visualization of language-dependent (LD) and speaker-dependent (SD) features. We visualize LD and SD features based on two different languages [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: t-SNE plots of speaker feature space of (a) LD features and (b) the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 70 canonical work pages

  1. [1]

    The amazing benefits of being bilingual,

    G. Vince, “The amazing benefits of being bilingual,” BBC, 2016. 1

    work page 2016

  2. [2]

    Disentangled representation learning for multilingual speaker recogni- tion,

    K. Nam, Y . Kim, J. Huh, H.-S. Heo, J. weon Jung, and J. S. Chung, “Disentangled representation learning for multilingual speaker recogni- tion,” in Proc. Interspeech, 2023, pp. 5316–5320. 1

    work page 2023

  3. [3]

    Crosslingual and multilingual speech recognition based on the speech manifold,

    R. Sahraeian and D. Van Compernolle, “Crosslingual and multilingual speech recognition based on the speech manifold,” IEEE/ACM Trans. on Audio, Speech, and Language Processing , vol. 25, pp. 2301–2312,

    work page

  4. [4]

    Distilling a pretrained language model to a multilingual asr model,

    K. Choi and H.-M. Park, “Distilling a pretrained language model to a multilingual asr model,” in Proc. Interspeech, 2022, pp. 2203–2207. 1

    work page 2022

  5. [5]

    Joint ASR and language identification using RNN-T: An efficient approach to dynamic language switching,

    S. Punjabi, H. Arsikere, Z. Raeesy, C. Chandak, N. Bhave, A. Bansal, M. M ¨uller, S. Murillo, A. Rastrow, A. Stolcke et al. , “Joint ASR and language identification using RNN-T: An efficient approach to dynamic language switching,” in Proc. ICASSP, 2021, pp. 7218–7222. 1

    work page 2021

  6. [6]

    Joint unsupervised and supervised learning for context-aware language iden- tification,

    J. Park, H. Y . Kim, J. Park, B.-Y . Kim, S. Choi, and Y . Lim, “Joint unsupervised and supervised learning for context-aware language iden- tification,” in Proc. ICASSP, 2023, pp. 1–5. 1

    work page 2023

  7. [7]

    Fastpitch: Parallel text-to-speech with pitch prediction,

    A. Ła ´ncucki, “Fastpitch: Parallel text-to-speech with pitch prediction,” in Proc. ICASSP, 2021, pp. 6588–6592. 1, 3, 6, 7

    work page 2021

  8. [8]

    Speaker adaptive text-to-speech with timbre-normalized vector-quantized feature,

    C. Du, Y . Guo, X. Chen, and K. Yu, “Speaker adaptive text-to-speech with timbre-normalized vector-quantized feature,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, vol. 31, pp. 3446–3456, 2023. 1

    work page 2023

Show all 75 references
  1. [9]

    EfficientTTS 2: Variational end-to-end text-to-speech synthesis and voice conversion,

    C. Miao, Q. Zhu, M. Chen, J. Ma, S. Wang, and J. Xiao, “EfficientTTS 2: Variational end-to-end text-to-speech synthesis and voice conversion,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, vol. 32, pp. 1650–1661, 2024. 1

    2024

  2. [10]

    Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes,

    B. Li, Y . Zhang, T. Sainath, Y . Wu, and W. Chan, “Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes,” in Proc. ICASSP, 2019, pp. 5621–5625. 1, 2

    2019

  3. [11]

    Improve cross-lingual text-to- speech synthesis on monolingual corpora with pitch contour informa- tion,

    H. Zhan, H. Zhang, W. Ou, and Y . Lin, “Improve cross-lingual text-to- speech synthesis on monolingual corpora with pitch contour informa- tion,” in Proc. Interspeech, 2021, pp. 1599–1603. 1, 2, 4

    2021

  4. [12]

    Language-agnostic meta-learning for low-resource text-to-speech with articulatory features,

    F. Lux and N. T. Vu, “Language-agnostic meta-learning for low-resource text-to-speech with articulatory features,” in Proc. ACL, 2022, pp. 6858–

    2022

  5. [13]

    Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-language voice cloning,

    Y . Zhang, R. J. Weiss, H. Zen, Y . Wu, Z. Chen, R. Skerry-Ryan, Y . Jia, A. Rosenberg, and B. Ramabhadran, “Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-language voice cloning,” in Proc. Interspeech, 2019, pp. 2080–2084. 1, 2, 6, 7

    2019

  6. [14]

    Disentan- gled speaker and language representations using mutual information minimization and domain adaptation for cross-lingual TTS,

    D. Xin, T. Komatsu, S. Takamichi, and H. Saruwatari, “Disentan- gled speaker and language representations using mutual information minimization and domain adaptation for cross-lingual TTS,” in Proc. ICASSP, 2021, pp. 6608–6612. 1, 2

    2021

  7. [15]

    GenerTTS: Pronunciation disentanglement for timbre and style generalization in cross-lingual text-to-speech,

    Y . Cong, H. Zhang, H. Lin, S. Liu, C. Wang, Y . Ren, X. Yin, and Z. Ma, “GenerTTS: Pronunciation disentanglement for timbre and style generalization in cross-lingual text-to-speech,” in Proc. Interspeech , 2023, pp. 5486–5490. 1, 2

    2023

  8. [16]

    DSE-TTS: Dual speaker embedding for cross-lingual text-to-speech,

    S. Liu, Y . Guo, C. Du, X. Chen, and K. Yu, “DSE-TTS: Dual speaker embedding for cross-lingual text-to-speech,” in Proc. Interspeech, 2023, pp. 616–620. 1, 2

    2023

  9. [17]

    ZMM-TTS: Zero-shot multilingual and multi- speaker speech synthesis conditioned on self-supervised discrete speech representations,

    C. Gong, X. Wang, E. Cooper, D. Wells, L. Wang, J. Dang, K. Rich- mond, and J. Yamagishi, “ZMM-TTS: Zero-shot multilingual and multi- speaker speech synthesis conditioned on self-supervised discrete speech representations,” arXiv:2312.14398, 2023. 1, 2

    2023 arXiv

  10. [18]

    Unit selection in a concatenative speech synthesis system using a large speech database,

    A. J. Hunt and A. W. Black, “Unit selection in a concatenative speech synthesis system using a large speech database,” in Proc. ICASSP, 1996, pp. 373–376. 2

    1996

  11. [19]

    Statistical parametric speech synthesis,

    A. W. Black, H. Zen, and K. Tokuda, “Statistical parametric speech synthesis,” in Proc. ICASSP, 2007, pp. 1229–1232. 2

    2007

  12. [20]

    Naturalspeech: End-to-end text-to-speech synthesis with human-level quality,

    X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y . Liu, X. Wang, Y . Leng, Y . Yi, L. Heet al., “Naturalspeech: End-to-end text-to-speech synthesis with human-level quality,”IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 46, no. 6, pp. 1–12, 2024. 2

    2024

  13. [21]

    Harmonic-net: Fundamental frequency and speech rate controllable fast neural vocoder,

    K. Matsubara, T. Okamoto, R. Takashima, T. Takiguchi, T. Toda, and H. Kawai, “Harmonic-net: Fundamental frequency and speech rate controllable fast neural vocoder,” IEEE/ACM Trans. on Audio, Speech, and Language Processing , vol. 31, pp. 1902–1915, 2023. 2

    1902

  14. [22]

    Fregrad: Lightweight and fast frequency-aware diffusion vocoder,

    T. D. Nguyen, J.-H. Kim, Y . Jang, J. Kim, and J. S. Chung, “Fregrad: Lightweight and fast frequency-aware diffusion vocoder,” in Proc. ICASSP, 2024, pp. 10 736–10 740. 2

    2024

  15. [23]

    TriniTTS: Pitch-controllable end-to-end TTS without external aligner

    Y . Ju, I. Kim, H. Yang, J.-H. Kim, B. Kim, S. Maiti, and S. Watanabe, “TriniTTS: Pitch-controllable end-to-end TTS without external aligner.” in Proc. Interspeech, 2022, pp. 16–20. 2

    2022

  16. [24]

    Hierspeech: Bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis,

    S.-H. Lee, S.-B. Kim, J.-H. Lee, E. Song, M.-J. Hwang, and S.-W. Lee, “Hierspeech: Bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis,” in Proc. NeurIPS, 2022, pp. 16 624–16 636. 2, 5

    2022

  17. [25]

    Wavenet: A gener- ative model for raw audio,

    A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A gener- ative model for raw audio,” arXiv:1609.03499, 2016. 2

    2016 arXiv

  18. [26]

    Deep voice: Real-time neural text-to-speech,

    S. ¨O. Arık, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y . Kang, X. Li, J. Miller, A. Ng, J. Raimanet al., “Deep voice: Real-time neural text-to-speech,” in Proc. ICML, 2017, pp. 195–204. 2

    2017

  19. [27]

    Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions,

    J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerry-Ryan et al. , “Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions,” inProc. ICASSP, 2018, pp. 4779–4783. 2

    2018

  20. [28]

    Fastspeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” in Proc. ICLR, 2021. 2, 6

    2021

  21. [29]

    Matcha- TTS: A fast TTS architecture with conditional flow matching,

    S. Mehta, R. Tu, J. Beskow, ´E. Sz ´ekely, and G. E. Henter, “Matcha- TTS: A fast TTS architecture with conditional flow matching,” in Proc. ICASSP, 2024, pp. 11 341–11 345. 2

    2024

  22. [30]

    Multispeech: Multi-speaker text to speech with transformer,

    M. Chen, X. Tan, Y . Ren, J. Xu, H. Sun, S. Zhao, and T. Qin, “Multispeech: Multi-speaker text to speech with transformer,” in Proc. Interspeech, 2020, pp. 4024–4028. 2

    2020

  23. [31]

    Lightspeech: Lightweight and fast text to speech with neural architecture search,

    R. Luo, X. Tan, R. Wang, T. Qin, J. Li, S. Zhao, E. Chen, and T.- Y . Liu, “Lightspeech: Lightweight and fast text to speech with neural architecture search,” in Proc. ICASSP, 2021, pp. 5699–5703. 2

    2021

  24. [32]

    Phonological features for 0-shot multilingual speech synthesis,

    M. Staib, T. H. Teh, A. Torresquintero, D. S. R. Mohan, L. Foglianti, R. Lenain, and J. Gao, “Phonological features for 0-shot multilingual speech synthesis,” in Proc. Interspeech, 2020, pp. 2942–2946. 2

    2020

  25. [33]

    Text-inductive graphone-based language adaptation for low-resource speech synthesis,

    T. Saeki, S. Maiti, X. Li, S. Watanabe, S. Takamichi, and H. Saruwatari, “Text-inductive graphone-based language adaptation for low-resource speech synthesis,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, vol. 32, pp. 1829–1844, 2024. 2

    2024

  26. [34]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL, 2019, pp. 4171–4186. 2 JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10

    2019

  27. [35]

    Domain-adversarial training of neural networks,

    Y . Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Lavi- olette, M. March, and V . Lempitsky, “Domain-adversarial training of neural networks,” J. Mach. Learn. Res., vol. 17, no. 59, pp. 1–35, 2016. 2, 3

    2016

  28. [36]

    Learning disentangled representations via mutual information estimation,

    E. H. Sanchez, M. Serrurier, and M. Ortner, “Learning disentangled representations via mutual information estimation,” in Proc. ECCV , 2020, pp. 205–221. 2

    2020

  29. [37]

    SANE-TTS: Stable and natural end-to-end multilingual text-to-speech,

    H. Cho, W. Jung, J. Lee, and S. H. Woo, “SANE-TTS: Stable and natural end-to-end multilingual text-to-speech,” in Proc. Interspeech, 2022, pp. 1–5. 2, 4, 6, 7

    2022

  30. [38]

    Crossspeech: Speaker-independent acoustic representation for cross- lingual speech synthesis,

    J.-H. Kim, H.-S. Yang, Y .-C. Ju, I.-H. Kim, and B.-Y . Kim, “Crossspeech: Speaker-independent acoustic representation for cross- lingual speech synthesis,” in Proc. ICASSP, 2023, pp. 1–5. 2, 6, 7

    2023

  31. [39]

    Invariant risk minimization,

    M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” in Proc. ICLR, 2019. 3

    2019

  32. [40]

    Domain generalization with mixstyle,

    K. Zhou, Y . Yang, Y . Qiao, and T. Xiang, “Domain generalization with mixstyle,” in Proc. ICLR, 2021. 3, 4, 6

    2021

  33. [41]

    One TTS alignment to rule them all,

    R. Badlani, A. Ła ´ncucki, K. J. Shih, R. Valle, W. Ping, and B. Catanzaro, “One TTS alignment to rule them all,” in Proc. ICASSP, 2022, pp. 3915–

    2022

  34. [42]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” in Proc. Interspeech , 2020, pp. 5036–5040. 3

    2020

  35. [43]

    Feature-critic networks for heterogeneous domain generalization,

    Y . Li, Y . Yang, W. Zhou, and T. Hospedales, “Feature-critic networks for heterogeneous domain generalization,” in Proc. ICML , 2019, pp. 3915–3924. 3

    2019

  36. [44]

    Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech,

    R. Huang, Y . Ren, J. Liu, C. Cui, and Z. Zhao, “Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech,” in Proc. NeurIPS, 2022, pp. 10 970–10 983. 3, 4

    2022

  37. [45]

    PV AE-TTS: Adaptive text-to-speech via progressive style adaptation,

    J.-H. Lee, S.-H. Lee, J.-H. Kim, and S.-W. Lee, “PV AE-TTS: Adaptive text-to-speech via progressive style adaptation,” in Proc. ICASSP, 2022, pp. 6312–6316. 3, 5

    2022

  38. [46]

    Style normalization and restitution for domain generalization and adaptation,

    X. Jin, C. Lan, W. Zeng, and Z. Chen, “Style normalization and restitution for domain generalization and adaptation,” IEEE Trans. on Multimedia, vol. 24, pp. 3636–3651, 2021. 4

    2021

  39. [47]

    Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech,

    Y . Ren, M. Lei, Z. Huang, S. Zhang, Q. Chen, Z. Yan, and Z. Zhao, “Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech,” in Proc. ICASSP, 2022, pp. 7577–7581. 4

    2022

  40. [48]

    Diffprosody: Diffusion-based latent prosody generation for expressive speech synthesis with prosody conditional adversarial training,

    H.-S. Oh, S.-H. Lee, and S.-W. Lee, “Diffprosody: Diffusion-based latent prosody generation for expressive speech synthesis with prosody conditional adversarial training,” IEEE/ACM Trans. on Audio, Speech, and Language Processing , vol. 32, pp. 2654–2666, 2024. 4

    2024

  41. [49]

    pYIN: A fundamental frequency estimator using probabilistic threshold distributions,

    M. Mauch and S. Dixon, “pYIN: A fundamental frequency estimator using probabilistic threshold distributions,” in Proc. ICASSP, 2014, pp. 659–663. 4

    2014

  42. [50]

    Neural analysis and synthesis: Reconstructing speech from self-supervised representations,

    H.-S. Choi, J. Lee, W. Kim, J. Lee, H. Heo, and K. Lee, “Neural analysis and synthesis: Reconstructing speech from self-supervised representations,” in Proc. NeurIPS, 2021, pp. 16 251–16 265. 4, 5

    2021

  43. [51]

    Exploring wav2vec 2.0 on speaker verification and language identification,

    Z. Fan, M. Li, S. Zhou, and B. Xu, “Exploring wav2vec 2.0 on speaker verification and language identification,” in Proc. Interspeech, 2020, pp. 1509–1513. 5

    2020

  44. [52]

    Let there be sound: Reconstructing high quality speech from silent videos,

    J.-H. Kim, J. Kim, and J. S. Chung, “Let there be sound: Reconstructing high quality speech from silent videos,” in Proc. AAAI, 2024, pp. 2759–

    2024

  45. [53]

    Scaling speech technology to 1,000+ languages,

    V . Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi et al. , “Scaling speech technology to 1,000+ languages,” J. Mach. Learn. Res. , vol. 25, no. 97, pp. 1–52, 2024. 5, 7

    2024

  46. [54]

    Wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proc. NeurIPS, 2020, pp. 12 449–12 460. 5

    2020

  47. [55]

    Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,

    A. Graves, S. Fern ´andez, F. Gomez, and J. Schmidhuber, “Connection- ist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, 2006, pp. 369–376. 5

    2006

  48. [56]

    The LJ speech dataset,

    K. Ito and L. Johnson, “The LJ speech dataset,” https://keithito.com/ LJ-Speech-Dataset/, 2017. 5

    2017

  49. [57]

    CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit,

    J. Yamagishi, C. Veaux, and K. MacDonald, “CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit,” https: //doi.org/10.7488/ds/2645, 2019. 5

    2019 doi

  50. [58]

    The BIAOBEI dataset,

    D. T. Co., “The BIAOBEI dataset,” https://en.data-baker.com/datasets/ freeDatasets/, 2022. 5

    2022

  51. [59]

    AISHELL-3: A multi- speaker Mandarin TTS corpus,

    Y . Shi, H. Bu, X. Xu, S. Zhang, and M. Li, “AISHELL-3: A multi- speaker Mandarin TTS corpus,” in Proc. Interspeech, 2021, pp. 2756–

    2021

  52. [60]

    CSS10: A collection of single speaker speech datasets for 10 languages,

    K. Park and T. Mulc, “CSS10: A collection of single speaker speech datasets for 10 languages,” in Proc. Interspeech, 2019, pp. 1566–1570. 5

    2019

  53. [61]

    JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis,

    R. Sonobe, S. Takamichi, and H. Saruwatari, “JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis,” arXiv:1711.00354, 2017. 5

    2017 arXiv

  54. [62]

    Multi-speaker TTS data,

    M. U. Inc., “Multi-speaker TTS data,” https://www.aihub.or.kr/ aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe= realm&dataSetSn=542, 2021. 5

    2021

  55. [63]

    Phonemizer: Text to phones transcription for multiple languages in python,

    M. Bernard and H. Titeux, “Phonemizer: Text to phones transcription for multiple languages in python,” J. Open Source Softw., vol. 6, no. 68, p. 3958, 2021. 5

    2021

  56. [64]

    NANSY++: Unified voice synthesis with neural analysis and synthesis,

    H.-S. Choi, J. Yang, J. Lee, and H. Kim, “NANSY++: Unified voice synthesis with neural analysis and synthesis,” in Proc. ICLR, 2022. 6

    2022

  57. [65]

    Language modeling with gated convolutional networks,

    Y . N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in Proc. ICML, 2017, pp. 933–941. 6

    2017

  58. [66]

    Fre-GAN: Adversarial frequency-consistent audio synthesis,

    J.-H. Kim, S.-H. Lee, J.-H. Lee, and S.-W. Lee, “Fre-GAN: Adversarial frequency-consistent audio synthesis,” in Proc. Interspeech, 2021, pp. 2197–2201. 6

    2021

  59. [67]

    UTMOS: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: Utokyo-sarulab system for voicemos challenge 2022,” in Proc. Interspeech, 2022, pp. 4521–4525. 6

    2022

  60. [68]

    The blizzard challenge 2005: Evaluating corpus-based speech synthesis on common databases,

    A. Black and K. Tokuda, “The blizzard challenge 2005: Evaluating corpus-based speech synthesis on common databases,” in Proc. Inter- speech, 2005, pp. 77–80. 6

    2005

  61. [69]

    USAT: A universal speaker-adaptive text-to-speech approach,

    W. Wang, Y . Song, and S. Jha, “USAT: A universal speaker-adaptive text-to-speech approach,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, vol. 32, pp. 2590–2604, 2024. 6

    2024

  62. [70]

    Dual-branch modeling based on state-space model for speech enhancement,

    L. Sun, S. Yuan, A. Gong, L. Ye, and E. S. Chng, “Dual-branch modeling based on state-space model for speech enhancement,” IEEE/ACM Trans. on Audio, Speech, and Language Processing , 2024. 6

    2024

  63. [71]

    V oicegrad: Non-parallel any-to-many voice conversion with annealed langevin dy- namics,

    H. Kameoka, T. Kaneko, K. Tanaka, N. Hojo, and S. Seki, “V oicegrad: Non-parallel any-to-many voice conversion with annealed langevin dy- namics,” IEEE/ACM Trans. on Audio, Speech, and Language Processing, vol. 32, pp. 1457–1467, 2024. 6

    2024

  64. [72]

    Robust speech recognition via large-scale weak super- vision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak super- vision,” in Proc. ICML, 2023, pp. 28 492–28 518. 6

    2023

  65. [73]

    Visualizing data using t-SNE,

    L. Van der Maaten and G. Hinton, “Visualizing data using t-SNE,” J. Mach. Learn. Res. , vol. 9, no. 11, pp. 2579–2605, 2008. 8

    2008

  66. [74]

    Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,

    Z. Zhang, L. Zhou, C. Wang, S. Chen, Y . Wu, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,” arXiv:2303.03926,

    arXiv

  67. [75]

    XTTS: a massively multilingual zero-shot text-to-speech model,

    E. Casanova, K. Davis, E. G ¨olge, G. G ¨oknar, I. Gulea, L. Hart, A. Aljafari, J. Meyer, R. Morais, S. Olayemi et al., “XTTS: a massively multilingual zero-shot text-to-speech model,” arXiv:2406.04904, 2024. 8 Ji-Hoon Kim is a Ph.D. student in Electrical Engineering at the Ko...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.