Pith. sign in

REVIEW 4 major objections 7 minor 41 references

CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CookVoice claims a single non-autoregressive model with frame-level alignment can generate both speech and singing voice, matching larger baselines on quality while exceeding them on style and prosody control.

desk verdict A practical unified TTS/singing framework with real efficiency gains, but its style-controllability lead rests on an in-house metric that needs independent verification. read the letter →

arxiv 2608.11590 v2 pith:ZPYKLRHR submitted 2026-08-12 cs.SD cs.LG

Haowei Lou , Hye-Young Paik , Dai Jia , Kai Li , Lina Yao This is my paper · ORCID
classification cs.SDcs.LG
keywords voicegenerationtext-to-speechtext-to-singingstylecontrolprosodyflowmatchingdiffusiontransformernon-autoregressive
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CookVoice claims that a single non-autoregressive (parallel) model can generate both speech and singing voice by decomposing every voice into content, prosody, and style, and by aligning all control signals to individual acoustic frames. If this is right, one 43.51-million-parameter system replaces task-specific text-to-speech, singing, style-transfer, and voice-editing models without sacrificing quality and with finer-grained control than larger autoregressive systems. The paper reports style similarity of 91.65% for speech and 95.00% for singing, pitch-following correlation of 0.7102 and 0.8425, and real-time inference with 4–8 ODE steps. The unification matters because a user can decide what is said, how it sounds, and where the pitch comes from within one model rather than switching between isolated task systems.

What carries the argument

The central mechanism is a flexible frame-level alignment strategy. Phoneme durations, discrete prosody tokens, style embeddings, and a continuous F0 contour are each expanded or projected so that every control signal is represented as a sequence of length $T$, the same number of frames as the target latent spectrogram; the expanded style, content, and prosody tensors are concatenated along the feature dimension into a conditioning tensor $C \in \mathbb{R}^{3D \times T}$ that a flow-matching DiT (a transformer trained to regress the vector field carrying Gaussian noise to the target latent audio) uses as cross-attention conditioning to regress the vector field $Y_1 - Y_0$. Task differences reduce to duration-expansion rules, learned or predicted for speech and score-derived for singing, and to a training-time condition-switching scheme that randomly samples style from text or voice and prosody from discrete tokens or continuous F0, so no task-specific heads or objectives are required. The prosody encoder removes the voice-level mean from F0, so the explicit contour carries only relative melody, avoiding entanglement with the style embedding.

What would settle it

Re-score the same generated samples with human style-similarity ratings or with an independently trained style encoder; if CookVoice's roughly 16-percentage-point TTS and 7-percentage-point TTSV S-SIM lead over the strongest baselines shrinks or reverses, the frame-level controllability advantage is not established.

Watch

Extended reading notes

Core claim

CookVoice's central claim is that frame-level alignment, not model scale or autoregressive decoding, is what enables precise style and prosody control across speech and singing. The model decomposes a voice into content (phonemes), prosody (lexical tones, stresses, MIDI notes, or a reference F0 contour), and style (text prompt or reference voice), expands each control signal to the acoustic frame length, concatenates them, and feeds the result to a flow-matching diffusion transformer that generates a latent acoustic embedding. Because the same conditioning machinery serves every task, text-to-speech, text-to-singing, style-controllable generation, voice mimicry, voice conversion, and voice editing are all just different combinations of the same three factors. In the reported evaluations, the best configurations reach style similarity of 91.65% for TTS and 95.00% for TTSV, F0 correlation of 0.7102 and 0.8425, and MOS of 3.98 for TTS and 3.40 for TTSV, with 43.51M parameters and real-time inference.

Load-bearing premise

The controllability claims rest on a style-similarity metric computed by a single pre-trained style encoder; if that encoder is biased toward this model's own conditioning design, the reported advantage over the baselines would not hold under an independent measure.

Editorial extensions

If this is right

  • A single 43.51M-parameter model can cover TTS, text-to-singing, style-controllable generation, voice mimicry, voice conversion, and voice editing by toggling which of content, style, and prosody conditions are active.
  • Users can combine style from a text prompt or reference voice with prosody from discrete tokens (tones, stresses, MIDI notes) or a continuous reference F0 contour in any pairing.
  • At 4–8 ODE steps the system generates audio faster than real time, using about 5% of the parameters, about 20% of the GPU memory, and less than 1% of the inference time of the closest unified speech-and-singing baseline, while reporting higher style and prosody controllability.
  • Voice-based style plus continuous F0 gives the strongest reported controllability (S-SIM 91.65/95.00, F0-CORR 0.7102/0.8425 for TTS/TTSV), while text-style plus discrete prosody still improves on the baselines by 41.48% and 13.11% S-SIM respectively.
  • Inference step count is a real trade-off in the reported results: style and prosody fidelity improve up to roughly 8 steps, while intelligibility peaks near 4 steps, so the model's operating point depends on whether content clarity or style adherence matters more.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper reports experiments for TTS and TTSV; a direct test of the unified claim would be to measure whether the same frame-level conditioning also improves the listed but unevaluated tasks of voice conversion, speech/singing editing, sketch-to-voice, and humming-to-voice.
  • Because the prosody encoder deliberately strips the voice-level mean from F0 and leaves absolute pitch to the style embedding, the design predicts that shifting a reference contour's mean while keeping style fixed should not change the generated pitch range; supplying a plain reference F0 with a different mean would test this decoupling directly.
  • The reported optimum of 4–8 ODE steps is task-dependent: intelligibility is best near 4 steps while style and prosody fidelity keep improving to 8, so a deployment could pick the step count by which objective matters more.
  • The authors note the model has not been scaled; if the architecture's ceiling is capacity, training the same design with a larger DiT backbone and more data could plausibly close the remaining MOS gap to large baselines while keeping the efficiency and controllability profile. This is an untested scaling prediction, not a claim of the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents CookVoice, a non-autoregressive flow-matching DiT framework that unifies speech and singing voice generation by decomposing the voice into content, prosody, and style and expanding these control signals to the acoustic frame level. Style can be provided as a text prompt or a reference voice, and prosody can be provided as discrete lexical or note tokens or as a continuous F0 contour. A random condition-switching strategy is used during training so that a single model can support multiple task combinations. Experiments compare CookVoice against CosyVoice, F5-TTS, IndexTTS, ParaStyleTTS, Vevo2, DiffSinger, StyleSinger, TCSinger, and Vevo1.5. The paper claims comparable generation quality to baselines while providing stronger style similarity (S-SIM) and prosody controllability (F0-CORR), with only 43.51M parameters and inference using as few as 4 ODE steps.

Significance. The proposed design is clean and practically motivated: frame-level alignment of content, style, and prosody is an interpretable way to obtain fine-grained control, and random condition switching is an elegant mechanism for multi-task training. Objective prosody metrics such as F0-CORR and F0-RMSE are grounded in acoustic signals, and the efficiency analysis addresses a real deployment concern. However, the headline style-controllability claim rests almost entirely on S-SIM computed with an in-house style encoder from the same research group, subjective quality claims are not supported by statistical methodology, and the multi-task capability is not empirically demonstrated beyond TTS and TTSV. If these evaluation concerns are addressed, CookVoice would be a solid contribution to unified voice generation; as presented, the strongest claims are not yet adequately supported.

major comments (4)
  1. [§4.4, Eq. (9), Table 2]
  2. [§5.1, Table 3] The subjective MOS and MC-MOS evaluations lack essential methodology: the number of listeners, the number of utterances per system, and any significance testing are not reported. In the TTS condition, CookVoice's best MOS (3.98) is numerically below several baselines (IndexTTS 4.42, F5-TTS 4.35, CosyVoice 4.30, Vevo2 4.30), so the abstract's claim of 'comparable generation quality' is not supported by the table as it stands. In the TTSV condition, the difference between CookVoice (3.40) and Vevo2 (3.42) is too small to interpret without significance tests. Please report listener counts, confidence intervals, and paired significance tests for the MOS and MC-MOS comparisons.
  3. [Table 5, §5] The paper's central multi-task claim (contribution 1 and Section 3.5) lists 13 tasks including voice conversion, singing voice conversion, speech and singing editing, sketch-to-voice, and humming-to-voice, but the experiments in Section 5 only evaluate TTS and TTSV. As written, support for these additional tasks is purely conceptual, resting on the claim that condition-switching allows the same model to handle them. Please provide at least a small proof-of-concept evaluation (e.g., objective speaker-similarity for VC, or qualitative demos with clear task definitions) for the principal non-TTS tasks, or explicitly delimit the empirical claims to TTS, TTSV, and style/prosody-controllable speech and singing. Without this, the unified multi-task contribution is overstated.
  4. [§2.2, §5.5, Table 4] The claimed parameter efficiency ('only 43.51M parameters') appears to count only the trainable DiT-S generation backbone and not the frozen HiFi-GAN-style autoencoder or the style encoder that are part of the full generation pipeline. Since the parameter counts for baselines in Table 4 may use a different accounting convention, the efficiency comparison may be inconsistent. Please report a parameter breakdown for CookVoice (generation model, AE encoder and decoder, style encoder) and state whether the same accounting convention is applied to all baselines. In addition, report the real-time factor for the full pipeline, including latent encoding, ODE integration, and waveform decoding.
minor comments (7)
  1. [Table 2] The relative-percentage annotations for F0-RMSE are ambiguous because lower is better, yet the table mixes downward and upward arrows without stating which direction indicates an improvement.
  2. [§4.4, Appendix F] The main text mentions WER, PhoER, and ProER, but the full per-language results appear only in Appendix F; please add a pointer to that table at the relevant point in Section 5.
  3. [Appendix B vs. Tables 3–4] The metric is called M-CMOS in Appendix B but MC-MOS in Tables 3 and 4; please use one consistent name throughout the paper.
  4. [§4.2, Eq. (5)] The voiced-frame set U is used in Eq. (5) without a formal definition; please define U as the set of voiced frames and state how voiced frames are detected.
  5. [§5.3] The sentence beginning 'Which leads to an observable style-sim degradation...' is a grammatical fragment; please merge it with the preceding sentence.
  6. [Figure 2] The caption reads 'Redder indicate stronger performance'; this should be 'Redder colors indicate stronger performance'.
  7. [Table 4] Vevo2 appears twice in the table (once for TTS and once for TTSV); consider merging the rows or adding a footnote to explain that the same model is listed in both task groups.

Circularity Check

1 steps flagged · score 4.0 of 10

Style-controllability claim relies on the authors' own AutoSIFT style encoder [29]; the generation pipeline and prosody metrics are otherwise self-contained.

  1. self citation load bearing [Section 4.4 'Evaluation Metrics', Table 2 (RQ2), Appendix B Eq. (9), reference [29]]
    "To evaluate style controllability, we compute Style Similarity (S-SIM) between the generated voice and the target style reference. Specifically, we extract style embeddings from both audio samples using a pre-trained style encoder [29] and calculate their cosine similarity, where a higher value indicates better style preservation."

    The central claim of stronger style controllability is operationally the S-SIM column of Table 2 (91.65% vs. 75.11% for the best TTS baseline; 95.00% vs. 88.09% for the best TTSV baseline). Eq. (9) defines S-SIM as cosine similarity in the embedding space of a pre-trained style encoder [29], and [29] is the authors' own AutoSIFT work, sharing the same style-representation lineage as ParaMeta [10] and LanStyleTTS [19]. The paper reports neither AutoSIFT's training data nor any evidence that its embedding space is impartial with respect to CookVoice's voice-style conditioning scheme, so the claimed style advantage reduces to a comparison inside a same-group artifact rather than an externally validated metric.

full rationale

No full derivation-level circularity was found in the core pipeline. The flow-matching objective (Eq. 1) and Euler ODE solver (Eq. 2) are standard and train v_theta to regress Y1-Y0 from the conditioning C; the duration expansion and beat-relative frame assignment are deterministic computations, not fitted parameters renamed as predictions. The multi-task training switch (Eq. 3) is a task specification rather than an evaluation target. The prosody claim is supported by F0-RMSE and F0-CORR computed directly on acoustic contours, and the efficiency claim is arithmetic from Table 4, so those parts are self-contained against external baselines. The one load-bearing same-group element is the S-SIM evaluator: Section 4.4 defines style similarity through the authors' own pre-trained style encoder [29], and Table 2's style-controllability advantage is the main RQ2 result. Because the paper gives no evidence that AutoSIFT's embedding space is independent of CookVoice's conditioning scheme, the headline style-controllability superiority is partly carried by a self-citation. CookVoice's own voice-style encoder is trained with the flow-matching objective rather than directly to maximize AutoSIFT S-SIM, so this is self-citation load-bearing rather than full definitional circularity. Same-group duration and G2P modules [3, 19] are preprocessing utilities and do not further raise the score.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on four external tool modules (HiFi-GAN AE, ParaStyleTTS durations, LanStyleTTS G2P, flow-matching DiT) and on two ad hoc design choices: relative-F0 normalization and 50/50 random condition switching. The F0 normalization bounds and unvoiced placeholder are hand-chosen numbers that affect the conditioning signal. No new physical or conceptual entities are introduced.

free parameters (4)
  • F0 normalization bounds = f_min=50 Hz; f_max unspecified
    Eq. (4) scales F0 using f_min and f_max; only f_min is stated. The choice affects the conditioning signal and the F0-RMSE and F0-CORR evaluation, and the missing f_max blocks exact replication.
  • Unvoiced-frame placeholder = -2
    Eq. (5) sets unvoiced frames to a constant -2 in the relative pitch contour; this hand-chosen value affects how the model treats unvoiced regions.
  • Style and prosody source sampling probability = p=0.5
    Eq. (3) randomly selects text versus voice style and discrete versus continuous prosody with probability 0.5 per sample; this is a hand-chosen training schedule that determines multi-task behavior.
  • Singing-to-speech training ratio = 1:9 on 110k samples
    Section 4.3 trains on 110k samples with a 1:9 singing-to-speech ratio; the data mix affects the unified model's performance balance and is not derived from a principled criterion.
assumptions (6)
  • standard math Flow-matching objective and Euler solver are valid generative tools for latent acoustic generation.
    The OT-FM objective (Eq. 1) and Euler integration (Eq. 2) follow from Lipman et al. [12]; no new theory is introduced.
  • domain assumption HiFi-GAN-style autoencoder provides a high-fidelity latent acoustic representation that can serve as both training target and conditioning space.
    Section 2.2 relies on a HiFi-GAN-style AE for encoding and decoding; fidelity and invertibility of the latent space are inherited from prior work [9].
  • domain assumption ParaStyleTTS alignments and duration predictor provide correct ground-truth phoneme durations for speech.
    Section 3.2 uses ParaStyleTTS [3] for alignments during training and duration prediction at inference; errors in these durations propagate directly to frame alignment and generated audio.
  • domain assumption LanStyleTTS G2P provides accurate phoneme and lexical prosody tokens for English and Chinese.
    Section 3.2 relies on [19] for phoneme and tone or stress tokens; inaccurate G2P degrades content and prosody conditioning.
  • ad hoc to paper Relative F0 after mean removal is a sufficient prosody condition that does not conflict with style encoding.
    Eq. (5) subtracts voice-level mean F0 to avoid entanglement with style; this is a design assumption introduced specifically for this model and is not validated independently.
  • ad hoc to paper Random 50/50 condition switching during training is sufficient for the model to learn all task combinations.
    Eq. (3) assumes that sample-level random selection yields a unified multi-task model; no ablation is shown to verify that all combinations are learned equally well.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation." pith.science (2026). https://pith.science/paper/ZPYKLRHR

@misc{pith2026260811590,
  author       = {Pith},
  title        = {Pith review of: CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZPYKLRHR}},
  note         = {Machine review of arXiv:2608.11590}
}
read the original abstract

Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves comparable performance to large-scale baselines with only 43.51 million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications. Demo page is available at https://haoweilou.github.io/CookVoice/.

Figures

Figures reproduced from arXiv: 2608.11590 by the authors.

Figure 1
Figure 1. Architecture of the CookVoice 3.4. Flow-Matching DiT Given the style Se, content Xe, and prosody Pe satisfy Se, Xe, Pe ∈ R D×T , we concatenate them along the feature dimension to construct a conditioning embedding C ∈ R 3D×T : C = [Se; Xe; Pe] In the final stage of CookVoice, our objective is to translate the condition embedding C ∈ R 3D×T into high-fidelity latent embedding. To achieve this, we use a Diffusion Tra… view at source ↗
Figure 2
Figure 2. Marginal effects of different style and prosody control signals on CookVoice’s performance. Redder indicate stronger performance, bluer indicate weaker performance. 5.3. RQ3: Effects of Different Control Signals In this section, we analyze how different control signals af￾fect the performance of CookVoice [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Style similarity under different numbers of ODE steps for TTS and TTSV. (a) F0-RMSE on TTS (b) F0-RMSE on TTSV (c) F0-Correlation on TTS (d) F0-Correlation on TTSV [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Prosody fidelity under different numbers of ODE steps. F0-RMSE measures absolute pitch deviation, while F0- Correlation measures the consistency of the generated F0 con￾tour with the target contour. D. Task Definition CookVoice is designed as a unified framework for sp…
Figure 5
Figure 5. Figure 5: Intelligibility under different numbers of ODE steps for TTS and TTSV. WER, PhoER, and ProER are used to eval￾uate word-level, phoneme-level, and prosody-token-level error rates, respectively. (a) MOS on TTS (b) MOS on TTSV (c) MC-MOS on TTSV [PITH_FULL_IMAGE:figures/…
Figure 6
Figure 6. Figure 6: Human subjective listening test evaluation under dif￾ferent numbers of ODE steps. MOS measures overall natural￾ness, MC-MOS measures the melody accuracy of the generated singing voice. Specifically, X represents the phoneme derived from text or lyric to be generated, w…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 23 canonical work pages

  1. [29]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  2. [1]

    Introduction Recent years have witnessed an emergence of human voice generative AI, with models capable of high-quality text-to- speech (TTS) [1, 2, 3, 4], singing voice generation [5, 6, 7], zero-shot voice cloning [2, 4, 1], and voice editing [8]. Driven by advances in acoustic representation learning [9, 10], and gen- erative artificial intelligence [1...

    work page Pith review arXiv 2026

  3. [2]

    By decomposing human voice into style, content, and prosody, CookV oice successfully unifying multiple voice generation task within a unified model

    We proposeCookVoice, a unified framework for multi-task human voice generation. By decomposing human voice into style, content, and prosody, CookV oice successfully unifying multiple voice generation task within a unified model

    work page

  4. [3]

    We design a flexible alignment mechanism for fine-grained controllable human voice generation. By aligning different control signals at frame level, CookV oice can flexibly support multimodal style and prosody control, enabling more pre- cise control over paralinguistic style, melody, duration, and prosodic expression

    work page

  5. [4]

    Our experiment demonstrate CookV oice achieves higher style and prosody controllability than existing baselines, in- cluding models trained with larger-scale data or larger model sizes. Despite using a lightweight architecture and a much smaller training data, CookV oice achieves superior style sim- ilarity and F0 controllability while maintaining efficie...

    work page

  6. [5]

    Problem Formulation Human V oice Generation (HVG) is to generate waveform from different modality of control signal

    Preliminary 2.1. Problem Formulation Human V oice Generation (HVG) is to generate waveform from different modality of control signal. In this study, we define the human voice as a combination of human speech and the singing voice. Control signals includetext,lyric,note, andreference voice. Full details covered tasks are presented in Appendix D. For reader...

    work page

  7. [6]

    This section will present the architecture of CookV oice in detail

    CookVoice The overall architecture of CookV oice is presented in Figure 1. This section will present the architecture of CookV oice in detail. 3.1. Style Encoder Style Encoder is designed to generate a style embeddingS, which dictates the stylistic characteristics of the generated voice. In CookV oice, style conditioning can be derived from two modalities...

    work page

  8. [7]

    to find optimal alignments between the speech audio and the phoneme sequence, yielding ground-truth durationsD X. For singing voice, where score durations are denoted in rela- tive beats rather than absolute time (e.g., seconds), we compute the relative temporal proportion of each phoneme based on its assigned beats. This relative proportion is then multi...

    work page

Show all 41 references
  1. [8]

    Experiments 4.1. Dataset We combine multiple open-sourced bilingual (english and chi- nese ) speech and singing voice dataset to conduct the experi- ment includingBaker[21],LJSpeech[22],ESD[23],CREMA- D[24],CommonPhone[25],Genshin Voicedataset [26], GTSinger [27]. The combined...

  2. [9]

    Results & Discussion In this section, we provide an analysis of experimental results and organize the discussion to answer the following five re- search questions (RQs): •RQ1:How does CookV oice perform compared with existing TTS and TTSV baselines in term of audio? •RQ2:How c...

  3. [10]

    Conclusion In this paper, we presentCookVoice, a unified framework for multimodal, multi-style, and multi-task human voice genera- tion. By decomposing human voice into content, prosody, and style, CookV oice supports a wide range of speech and singing voice generation tasks w...

  4. [11]

    First, due to re- source constraints, CookV oice has not yet been scaled up

    Limitation CookV oice shows potential for unified and controllable human voice generation, several limitations remain. First, due to re- source constraints, CookV oice has not yet been scaled up. The current model only use the DiT-S version of diffusion trans- former (43.51M p...

  5. [12]

    Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Maet al., “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  6. [13]

    Indextts: An industrial-level controllable and efficient zero-shot text-to-speech system,

    W. Deng, S. Zhou, J. Shu, J. Wang, and L. Wang, “Indextts: An industrial-level controllable and efficient zero-shot text-to-speech system,”arXiv preprint arXiv:2502.05512, 2025

    2025 arXiv

  7. [14]

    Parastyletts: Toward effi- cient and robust paralinguistic style control for expressive text-to- speech generation,

    H. Lou, H.-Y . Paik, W. Hu, and L. Yao, “Parastyletts: Toward effi- cient and robust paralinguistic style control for expressive text-to- speech generation,” inProceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025, pp. 1979–1988

    2025

  8. [15]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. JianZhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025,...

    2025

  9. [16]

    Diffsinger: Singing voice synthesis via shallow diffusion mechanism,

    J. Liu, C. Li, Y . Ren, F. Chen, and Z. Zhao, “Diffsinger: Singing voice synthesis via shallow diffusion mechanism,” inProceedings of the AAAI conference on artificial intelligence, vol. 36, no. 10, 2022, pp. 11 020–11 028

    2022

  10. [17]

    Stylesinger: Style transfer for out-of- domain singing voice synthesis,

    Y . Zhang, R. Huang, R. Li, J. He, Y . Xia, F. Chen, X. Duan, B. Huai, and Z. Zhao, “Stylesinger: Style transfer for out-of- domain singing voice synthesis,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, pp. 19 597–19 605

    2024

  11. [18]

    Tcsinger: Zero-shot singing voice synthesis with style transfer and multi-level style control,

    Y . Zhang, Z. Jiang, R. Li, C. Pan, J. He, R. Huang, C. Wang, and Z. Zhao, “Tcsinger: Zero-shot singing voice synthesis with style transfer and multi-level style control,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, 2024, pp. 1960–1975

    2024

  12. [19]

    Vevo2: A unified and controllable framework for speech and singing voice generation,

    X. Zhang, J. Zhang, Y . Wang, C. Wang, Y . Chen, D. Jia, Z. Chen, and Z. Wu, “Vevo2: A unified and controllable framework for speech and singing voice generation,”IEEE Transactions on Au- dio, Speech and Language Processing, 2026

    2026

  13. [20]

    Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial net- works for efficient and high fidelity speech synthesis,”Advances in neural information processing systems, vol. 33, pp. 17 022– 17 033, 2020

    2020

  14. [21]

    Param- eta: Towards learning disentangled paralinguistic speak- ing styles representations from speech,

    H. Lou, H. young Paik, W. Hu, and L. Yao, “Param- eta: Towards learning disentangled paralinguistic speak- ing styles representations from speech,” inAAAI Confer- ence on Artificial Intelligence, 2026. [Online]. Available: https://api.semanticscholar.org/CorpusID:284910281

    2026

  15. [22]

    Scalable diffusion models with transform- ers,

    W. Peebles and S. Xie, “Scalable diffusion models with transform- ers,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4195–4205

    2023

  16. [23]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,”arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  17. [24]

    Fastspeech: Fast, robust and controllable text to speech,

    Y . Ren, Y . Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech: Fast, robust and controllable text to speech,” Advances in neural information processing systems, vol. 32, 2019

    2019

  18. [25]

    M4singer: A multi-style, multi- singer and musical score provided mandarin singing corpus,

    L. Zhang, R. Li, S. Wang, L. Deng, J. Liu, Y . Ren, J. He, R. Huang, J. Zhu, X. Chenet al., “M4singer: A multi-style, multi- singer and musical score provided mandarin singing corpus,”Ad- vances in Neural Information Processing Systems, vol. 35, pp. 6914–6926, 2022

    2022

  19. [26]

    Vevo: Controllable zero-shot voice imitation with self- supervised disentanglement,

    X. Zhang, X. Zhang, K. Peng, Z. Tang, V . Manohar, Y . Liu, J. Hwang, D. Li, Y . Wang, J. Chan, Y . Huang, Z. Wu, and M. Ma, “Vevo: Controllable zero-shot voice imitation with self- supervised disentanglement,” inICLR. OpenReview.net, 2025

    2025

  20. [27]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,

    J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” inInter- national conference on machine learning. PMLR, 2021, pp. 5530–5540

    2021

  21. [28]

    Mpnet: Masked and permuted pre-training for language understanding,

    K. Song, X. Tan, T. Qin, J. Lu, and T.-Y . Liu, “Mpnet: Masked and permuted pre-training for language understanding,”Advances in neural information processing systems, vol. 33, pp. 16 857– 16 867, 2020

    2020

  22. [30]

    Generalized multi- lingual text-to-speech generation with language-aware style adap- tation,

    H. Lou, H.-y. Paik, S. Li, W. Hu, and L. Yao, “Generalized multi- lingual text-to-speech generation with language-aware style adap- tation,”arXiv preprint arXiv:2504.08274, 2025

    2025 arXiv

  23. [31]

    Tacotron: Towards end-to-end speech synthesis,

    Y . Wang, R. Skerry-Ryan, D. Stanton, Y . Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y . Xiao, Z. Chen, S. Bengioet al., “Tacotron: Towards end-to-end speech synthesis,”arXiv preprint arXiv:1703.10135, 2017

    2017 arXiv

  24. [32]

    Chinese mandarin female corpus,

    Databaker, “Chinese mandarin female corpus,” https://en. data-baker.com/datasets/freeDatasets/, 2020, accessed: 2023-04- 20

    2020

  25. [33]

    The lj speech dataset,

    K. Ito and L. Johnson, “The lj speech dataset,” https://keithito. com/LJ-Speech-Dataset/, 2017

    2017

  26. [34]

    Seen and unseen emo- tional style transfer for voice conversion with a new emotional speech dataset,

    K. Zhou, B. Sisman, R. Liu, and H. Li, “Seen and unseen emo- tional style transfer for voice conversion with a new emotional speech dataset,” inICASSP 2021-2021 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 920–924

    2021

  27. [35]

    Crema-d: Crowd-sourced emotional multimodal actors dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,”IEEE transactions on affective computing, vol. 5, no. 4, pp. 377–390, 2014

    2014

  28. [36]

    Common phone: A multilingual dataset for robust acoustic modelling,

    P. Klumpp, T. Arias, P. A. P ´erez-Toro, E. Noeth, and J. Orozco- Arroyave, “Common phone: A multilingual dataset for robust acoustic modelling,” inProceedings of the Thirteenth Language Resources and Evaluation Conference, 2022, pp. 763–768

    2022

  29. [37]

    Genshin voice: A multi-lingual voice dataset from Genshin Impact,

    Simon3000, “Genshin voice: A multi-lingual voice dataset from Genshin Impact,” https://huggingface.co/datasets/simon3000/ genshin-voice, 2025, hugging Face Datasets

    2025

  30. [38]

    Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks,

    Y . Zhang, C. Pan, W. Guo, R. Li, Z. Zhu, J. Wang, W. Xu, J. Lu, Z. Hong, C. Wanget al., “Gtsinger: A global multi-technique singing corpus with realistic music scores for all singing tasks,” Advances in Neural Information Processing Systems, vol. 37, pp. 1117–1140, 2024

    2024

  31. [39]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  32. [40]

    Autosift: Automatic style sifting for controllable speech generation with arbitrary style infilling,

    H. Lou, J. Wu, C. Huang, T. Yu, H.-y. Paik, W. Hu, and L. Yao, “Autosift: Automatic style sifting for controllable speech generation with arbitrary style infilling,”arXiv preprint arXiv:2607.12706, 2026

    2026 arXiv

  33. [41]

    Denoising diffusion probabilis- tic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilis- tic models,”Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020. A. Related Work Autoregressive voice generation.Recent advances in neural voice generation have significantly improved t...

    2020

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.