Pith. sign in

REVIEW 5 major objections 5 minor 45 references

CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CtrlSpeech adds phone-level pitch, loudness, and duration control to zero-shot TTS without sacrificing voice cloning.

desk verdict Solid integration of phone-level prosody control into DiTAR, but the controllability test is oracle-reconstruction and needs perturbation evidence before the main claim is fully supported. read the letter →

arxiv 2608.08362 v1 pith:ZXC6SZJP submitted 2026-08-08 eess.AS cs.HCcs.SD

Zhisheng Zheng , Xiaohang Sun , Zhu Liu , Caren Chen , Rohith Kumar , Manoj Aggarwal , Gerard Medioni , David Harwath This is my paper · ORCID
classification eess.AScs.HCcs.SD
keywords text-to-speechspeechsynthesisprosodycontrolzero-shotvoicecloningautoregressivediffusionphone-alignedconditioningpitchflowmatching
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CtrlSpeech is a text-to-speech system that aims to show that fine-grained prosody control and zero-shot voice cloning can live in the same model. It augments the DiTAR autoregressive-diffusion backbone with phone-aligned pitch, loudness, and duration signals, while a global speaker embedding plus prompt audio carries timbre. At 0.6B scale the system reaches 2.46% WER on LibriSpeech-PC test-clean and 2.58% WER on Seed-TTS test-en, matching or slightly beating the authors' DiTAR reproduction. When the phone-aligned signals are supplied, pitch RMSE falls from 67.86 to 38.39 Hz, loudness RMSE from 6.35 to 4.56 dB, and phoneme duration MAE from 28.08 to 11.86 frames. The intended payoff is an editing workflow where a user first generates an utterance and then iteratively refines local prosodic events without losing the target voice.

What carries the argument

The mechanism is the patch-wise diffusion-autoregressive factorization of DiTAR, where a causal autoregressive transformer emits a context representation for each patch of four continuous speech tokens and a local diffusion transformer denoises the next patch conditioned on that context. CtrlSpeech inserts control by augmenting phone embeddings with phone-aligned pitch, loudness, and duration tokens, and by adding a speaker embedding and prompt-speech condition to the autoregressive stream. The model is optimized with a flow-matching velocity objective plus an auxiliary loss that classifies each acoustic patch as first, middle, or last. The phone-aligned control stream is what lets the decoder attend to local prosodic targets, while the autoregressive context preserves long-range coherence across patches.

What would settle it

Perturb the supplied control signals during inference—shift pitch up one bin, stretch all durations by 20%, or replace the ground-truth contours with user-drawn sketches—and measure pitch, loudness, and duration RMSE against the edited targets; if the output does not track the edits, or if the oracle-conditioned gains vanish once the signals are not the same values used in training, the controllability claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that local prosodic control does not have to come at the cost of zero-shot synthesis quality. CtrlSpeech concatenates each phone embedding with quantized control tokens—pitch in 128 Mel-scale bins over 65–650 Hz, loudness in 64 bins over -60 to 0 dB, and duration in acoustic frames per phone—and conditions the utterance on a global speaker embedding and optional prompt speech. On LibriSpeech-PC test-clean the 0.6B model obtains 2.46% WER and 0.65 SIM-o, and on Seed-TTS test-en 2.58% WER and 0.63 SIM-o, both slightly ahead of the reproduced DiTAR backbone. With ground-truth control signals provided, pitch RMSE on LJSpeech drops from 67.86 to 38.39 Hz and loudness RMSE from 6.35 to 4.56 dB, while phoneme duration MAE on LibriSpeech-PC drops from 28.08 to 11.86 frames. The authors interpret these numbers as evidence that aligned low-level prosodic conditions give users direct local control while the global condition preserves timbre.

Load-bearing premise

The load-bearing premise is that feeding the model ground-truth phone-level pitch, loudness, and duration and then measuring error against those same ground-truth values is a faithful test of how well a user can control prosody; if the extracted control signals are noisy or the model is simply copying strong oracle inputs, the reported control gains may not carry over to real manual editing.

Editorial extensions

If this is right

  • At 0.6B scale, CtrlSpeech matches or slightly beats a reproduced DiTAR on zero-shot WER and speaker similarity on both LibriSpeech-PC test-clean and Seed-TTS test-en.
  • Providing phone-aligned pitch and loudness reduces frame-level RMSE substantially on LJSpeech, from 67.86 to 38.39 Hz and from 6.35 to 4.56 dB.
  • Providing phone-level target durations cuts phoneme duration MAE on LibriSpeech-PC from 28.08 to 11.86 frames, showing effective pacing control.
  • Combining speaker embedding and prompt speech outperforms either alone for speaker similarity (SIM-o 0.63 vs 0.53 for prompt-only), so the two coarse conditions are complementary.
  • The system's design supports iterative coarse-to-fine editing: generate with global conditions, then refine local pitch, loudness, and duration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that the quantized control tokens could be transplanted from one utterance to another, turning the model into a prosody-transfer tool: extract pitch, loudness, and duration contours from a reference recording and apply them to new text with a different speaker embedding.
  • Because the reported controllability metrics are measured against the same ground-truth signals fed into the model, a stricter test would have human users draw or tweak contours and check whether the audio follows the edited curves rather than the original recording; the paper does not run that experiment.
  • The coarse-to-fine conditioning scheme suggests a natural extension to additional phone-aligned attributes such as emotion tags, voice quality, or articulation rate, provided those attributes can be extracted or annotated at phone granularity.
  • One testable robustness extension is to corrupt or jitter the input control signals during inference and measure how gracefully pitch and duration RMSE degrade; the paper only reports the oracle-conditioned case.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CtrlSpeech, a controllable expressive TTS system built on the DiTAR autoregressive-diffusion backbone. It augments phone embeddings with phone-aligned pitch, loudness, and duration signals, and combines these with a global speaker embedding for zero-shot voice cloning. The authors report competitive zero-shot TTS results on LibriSpeech-PC test-clean and Seed-TTS test-en, and claim fine-grained controllability, supported by reduced pitch/loudness RMSE and phoneme-level duration MAE when ground-truth control signals are provided. The paper also describes a coarse-to-fine interactive refinement interface.

Significance. If the controllability claims hold, the paper makes a practically useful contribution: it shows that phone-level prosodic control and zero-shot speaker conditioning can be combined in a continuous autoregressive diffusion TTS system, with code and model weights promised. The zero-shot results at 0.6B are respectable, and the speaker-conditioning ablation in Table 2 clearly shows the complementary value of speaker embeddings and prompt speech. However, the central fine-grained controllability claim currently rests on oracle-conditioned reconstruction metrics (feeding ground-truth prosody and measuring error to the same values), which is necessary but not sufficient evidence for user-facing control. The paper would be strengthened by perturbation-based controllability tests, listening tests on edited controls, and statistical significance assessment.

major comments (5)
  1. [§4.4, Tables 3-4] The controllability evaluation is an oracle-reconstruction test. Pitch, loudness, and duration are extracted from the target utterance and fed as inputs, and RMSE/MAE are measured against those same target values. This demonstrates that the model can exploit strong ground-truth conditions to reproduce reference prosody, but it does not show that arbitrary user edits are realized locally. The paper should add perturbation experiments in which non-ground-truth controls are applied (e.g., raising pitch on a specific phone, stretching a single duration, or changing loudness) and verify that the output changes in the intended direction on that phone, leaves neighboring phones and other attributes approximately unchanged, and remains natural. A listening test for edited, non-oracle controls would directly support the coarse-to-fine control claim in §5.3–5.4.
  2. [§5.3, Table 4] The duration MAE reduction may largely reflect the oracle duration condition combined with the stop-prediction loss (§3.4), which classifies each patch as first, middle, or last and can act as a frame-counting mechanism. The paper does not isolate whether the model produces naturally paced, editable speech or simply counts frames to the supplied length. The authors should analyze responses to non-oracle duration edits, for example by feeding a uniformly stretched or locally modified duration vector and reporting whether the output follows the edit, and by checking naturalness of the resulting pacing.
  3. [§5.1, Tables 1-2] All objective and subjective results are single point estimates without confidence intervals or significance tests. The key zero-shot comparisons are close (WER 2.46 vs 2.55 on LibriSpeech-PC; CMOS -0.16 vs -0.22), so it is not clear whether CtrlSpeech is actually competitive with the reproduced DiTAR baseline beyond random variation. Per-utterance or bootstrap confidence intervals for WER/SIM-o and appropriate statistical tests for CMOS/SMOS are needed to support the 'competitive' claim.
  4. [§5.4, Figure 1] The coarse-to-fine iterative refinement workflow is a core claimed contribution but is never evaluated. The paper should present at least one experiment or case study showing a user making a coarse selection, refining local pitch/loudness/duration, and the output changing accordingly across rounds while preserving speaker identity. Without this, the interface description in §5.4 is unsupported by evidence.
  5. [Table 3] The pitch controllability result is not compared to DrawSpeech's 'With Sketch' condition on equal footing. DrawSpeech with sketch achieves 27.78 Hz pitch RMSE, which is lower than CtrlSpeech's 38.39 Hz with explicit control signals, although CtrlSpeech is better on loudness RMSE (4.56 vs 13.15 dB). Since DrawSpeech is the most relevant controllability baseline, the authors should either discuss this discrepancy or provide a matched comparison, rather than only contrasting CtrlSpeech with its own text-only setting.
minor comments (5)
  1. [§3.4 vs Eq. (4)] The text says 'L1 flow-matching objective' but Eq. (4) is a squared L2 norm. Please clarify which loss is actually used and fix the inconsistency.
  2. [§4.1] Please specify the forced-alignment tool and the exact data filtering criteria for the Emilia and GigaSpeech subsets, including how the 20,000-hour total is distributed between the two corpora.
  3. [§3.3.2] The bin assignment for f0 values above 650 Hz is described as 'clipped to bin 127', while voiced frames map to bins 1–126. Clarify the boundary between bin 126 and bin 127, and whether 650 Hz itself maps to bin 126 or 127.
  4. [§3.1] The VAE encoder produces a posterior from which latent tokens are sampled, but it is not stated whether sampling is also used at inference or whether the mean is used. This affects reconstruction fidelity and should be clarified.
  5. [§4.3] The classifier-free guidance description says 'the time and speaker embeddings are treated as a unified conditioning signal'; please specify whether the prosodic control signals are also dropped during training for CFG or whether they are always provided.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: CtrlSpeech is an empirical system paper; the oracle-conditioned controllability metric is a validity concern, not a circular derivation.

full rationale

CtrlSpeech does not present a formal derivation whose conclusion is an input by construction. Its central claims are empirical comparisons (WER, SIM-o, RMSE, MAE) against baselines on held-out test sets. The DiTAR backbone is an external architecture; the control-signal conditioning (pitch, loudness, duration) is not fitted to the test metrics. The one place where the evaluation is close to an oracle loop is Section 4.4 / Tables 3–4: the model is fed ground-truth phone-aligned pitch, loudness, and duration and then scored by RMSE/MAE against those same ground-truth values, so the reported reductions (e.g., pitch RMSE 67.86 to 38.39 Hz, duration MAE 28.08 to 11.86 frames) show that the model can exploit strong oracle conditions to reconstruct reference prosody, not that arbitrary user edits are realized. However, this is a measurement-validity limitation, not circularity: the input does not by construction determine the output metric, and the paper also reports text-only ablations that degrade markedly, indicating the conditioning carries real information. No load-bearing argument reduces to a self-citation; citations to prior work by the same authors (e.g., [6], [13]) are contextual, not load-bearing. The limitations section candidly notes dependence on pitch-extraction and forced-alignment quality, which further supports treating the oracle-loop as an acknowledged weakness rather than a hidden circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; control signals are standard speech features. The free parameters are design choices in quantization, patching, and inference, and the axioms are standard math plus domain assumptions about the acoustic features and the baseline reproduction.

free parameters (6)
  • Pitch quantization range and bin count = 65-650 Hz, 128 bins
    Section 3.3.2; hand-chosen range and resolution bound the pitch control resolution and clip values above 650 Hz.
  • Loudness quantization range and bin count = -60 to 0 dB, 64 bins
    Section 3.3.2; hand-chosen clipping and ~1 dB resolution; affects loudness RMSE.
  • Patch size P = 4 latent tokens
    Section 3.4; chosen as a trade-off between local coherence and efficiency; central to autoregressive factorization.
  • CFG guidance scale = 1.5
    Section 4.3; affects naturalness and speaker similarity; no sensitivity analysis.
  • Diffusion sampling steps = 32
    Section 4.3; fixed inference budget; no ablation.
  • Stop-loss weight lambda = unspecified
    Section 3.4, Eq. (7); balancing coefficient for auxiliary loss is not reported.
assumptions (5)
  • standard math Autoregressive patch factorization (Eq. 2) approximates the full sequence distribution.
    Section 3.2.1; standard AR assumption, not derived from data.
  • domain assumption The 40 Hz continuous latent tokens from Semantic-VAE preserve sufficient fine-grained prosody for phone-level control.
    Section 3.1 and 4.2; if the codec loses f0/loudness detail, control cannot be realized.
  • domain assumption WORLD f0 extraction and forced alignment yield accurate phone-aligned prosodic targets.
    Sections 3.3.2 and 7; paper acknowledges alignment errors can degrade control.
  • domain assumption A-weighted frame RMS approximates perceived loudness for control purposes.
    Section 3.3.2; standard acoustic approximation.
  • domain assumption The authors' reproduction of DiTAR is a faithful baseline despite official code being unavailable.
    Table 1 footnote; load-bearing for the zero-shot comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis." pith.science (2026). https://pith.science/paper/ZXC6SZJP

@misc{pith2026260808362,
  author       = {Pith},
  title        = {Pith review of: CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXC6SZJP}},
  note         = {Machine review of arXiv:2608.08362}
}
read the original abstract

Recent Text-To-Speech (TTS) systems have achieved strong naturalness and zero-shot voice cloning performance, but fine-grained control of expressive speech at the word or phoneme level remains challenging. We propose CtrlSpeech, a controllable, expressive TTS framework with coarse-to-fine control. Built on the DiTAR architecture, CtrlSpeech combines global speaker conditioning with phone-aligned pitch, loudness, and duration signals, enabling localized prosodic control while preserving the target speaker's timbre. This design allows users to adjust expressive attributes at a fine temporal granularity, making speech refinement more flexible and controllable. Experimental results show that CtrlSpeech achieves competitive zero-shot TTS performance and improves controllability over expressive attributes, demonstrating its effectiveness for flexible and practical expressive speech synthesis.

Figures

Figures reproduced from arXiv: 2608.08362 by the authors.

Figure 1
Figure 1. Coarse-to-fine speech control pipeline. 1. We propose CTRLSPEECH, a controllable, expressive speech synthesis framework that enables unified coarse-to-fine con￾trol over both global and local attributes. 2. We design an explicit control pipeline that supports iterative speech refinement through aligned pitch, loudness, and du￾ration signals, allowing users to flexibly adjust synthesized speech. 3. Extensive experime… view at source ↗
Figure 2
Figure 2. The architecture of CTRLSPEECH. systems [11–15] have further extended this paradigm, enabling more unified and expressive instruction-following speech syn￾thesis. 3. Methodology 3.1. Continuous Speech Representations We model speech in a continuous latent space rather than with discrete codec tokens, as continuous representations avoid the information loss introduced by quantization [1, 16, 17]. Prior work also note… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 21 canonical work pages

  1. [1]

    Introduction Recent text-to-speech (TTS) systems have achieved remarkable progress in naturalness and zero-shot voice cloning. Reference- conditioned methods can synthesize speech for unseen speak- ers from a short prompt waveform, while text-prompted and instruction-based systems provide a more intuitive interface for controlling speaking style. However,...

    work page

  2. [2]

    We propose CTRLSPEECH, a controllable, expressive speech synthesis framework that enables unified coarse-to-fine con- trol over both global and local attributes

    work page

  3. [3]

    We design an explicit control pipeline that supports iterative speech refinement through aligned pitch, loudness, and du- ration signals, allowing users to flexibly adjust synthesized speech

    work page

  4. [4]

    Extensive experiments show that CTRLSPEECHachieves competitive zero-shot TTS performance and significantly im- proves fine-grained controllability over expressive attributes

    work page

  5. [5]

    CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis

    Related Work 2.1. Reference-Conditioned Speech Synthesis Reference-conditioned speech synthesis, also known as voice- cloning or zero-shot TTS, aims to generate target speech by conditioning the model on a short reference utterance from an unseen speaker. The input text specifies the linguistic content and the reference speech provides speaker-dependent c...

    work page Pith review arXiv 2026

  6. [6]

    outpainting

    Methodology 3.1. Continuous Speech Representations We model speech in a continuous latent space rather than with discrete codec tokens, as continuous representations avoid the information loss introduced by quantization [1, 16, 17]. Prior work also notes that discrete-token pipelines often rely on multi-stage coarse-to-fine generation, which can increase ...

    work page

  7. [7]

    Experimental Setup 4.1. Datasets During the pretraining stage, CTRLSPEECHis trained on a sub- set of the English split of Emilia [26] and a subset of Gi- gaSpeech [27], yielding approximately 20,000 hours of English speech in total. This pretraining corpus provides broad acous- tic and linguistic coverage, enabling the model to acquire ro- bust speech gen...

    work page

  8. [8]

    Results and Analysis Table 1:Zero-Shot TTS performance on LibriSpeech-PC test- clean and Seed-TTS test-en. Model WER(%)↓SIM-o↑CMOS↑SMOS↑ LibriSpeech-PCtest-clean Ground Truth 2.40 0.69 0.00 3.90 V ocoder Reconstructed 2.45 0.68 - - DiTAR† 2.55 0.61 -0.22 3.74 CtrlSpeech (0.1B) 4.36 0.61 -0.65 3.66 CtrlSpeech (0.6B)2.46 0.65-0.16 3.81 Seed-TTStest-en Groun...

    work page

Show all 45 references
  1. [9]

    Conclusion In this paper, we proposed CTRLSPEECH, a controllable ex- pressive TTS framework with coarse-to-fine control. By com- bining global speaker conditioning with phone-aligned prosodic signals, CTRLSPEECHenables flexible control over expressive speech while preserving z...

  2. [10]

    First, our experiments are mainly conducted on English speech, so the effectiveness of CTRLSPEECHfor multilingual or code-switching synthesis re- mains unexplored

    Limitations This work has several limitations. First, our experiments are mainly conducted on English speech, so the effectiveness of CTRLSPEECHfor multilingual or code-switching synthesis re- mains unexplored. Second, the proposed fine-grained control depends on phone-aligned...

  3. [11]

    2D- 16003984 through the Amazon-UT Austin HUB

    Acknowledgments This work was supported by Amazon.com, PO No. 2D- 16003984 through the Amazon-UT Austin HUB. We thank Guanrou Yang and Zhikang Niu for their incredible help

  4. [12]

    All authors remain fully responsible for the con- tent of this manuscript

    Generative AI Use Disclosure Generative AI tools were used only for language editing and proofreading. All authors remain fully responsible for the con- tent of this manuscript

  5. [13]

    Ditar: Diffusion transformer autoregressive modeling for speech generation,

    D. Jia, Z. Chen, J. Chen, C. Du, J. Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y . Wanget al., “Ditar: Diffusion transformer autoregressive modeling for speech generation,”arXiv preprint arXiv:2502.03930, 2025

    2025

  6. [14]

    Neural codec language mod- els are zero-shot text to speech synthesizers,

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Liet al., “Neural codec language mod- els are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  7. [15]

    Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tanget al., “Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,”arXiv preprint arXiv:2403.03100, 2024

    2024 arXiv

  8. [16]

    Maskgct: Zero-shot text-to- speech with masked generative codec transformer,

    Y . Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “Maskgct: Zero-shot text-to- speech with masked generative codec transformer,”arXiv preprint arXiv:2409.00750, 2024

    2024 arXiv

  9. [17]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. JianZhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025,...

    2025

  10. [18]

    V oicecraft-x: Unifying multilingual, voice- cloning speech synthesis and speech editing,

    Z. Zheng, P. Peng, A. Diwan, C. P. Huynh, X. Sun, Z. Liu, V . Bhat, and D. Harwath, “V oicecraft-x: Unifying multilingual, voice- cloning speech synthesis and speech editing,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 2737–2756

    2025

  11. [19]

    Prompttts: Control- lable text-to-speech with text descriptions,

    Z. Guo, Y . Leng, Y . Wu, S. Zhao, and X. Tan, “Prompttts: Control- lable text-to-speech with text descriptions,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  12. [20]

    Prompttts 2: Describing and generating voices with text prompt,

    Y . Leng, Z. Guo, K. Shen, X. Tan, Z. Ju, Y . Liu, Y . Liu, D. Yang, L. Zhang, K. Songet al., “Prompttts 2: Describing and generating voices with text prompt,”arXiv preprint arXiv:2309.02285, 2023

    2023 arXiv

  13. [21]

    Instructtts: Modelling expressive tts in discrete latent space with natural lan- guage style prompt,

    D. Yang, S. Liu, R. Huang, C. Weng, and H. Meng, “Instructtts: Modelling expressive tts in discrete latent space with natural lan- guage style prompt,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2913–2925, 2024

    2024

  14. [22]

    Natural language guidance of high- fidelity text-to-speech with synthetic annotations,

    D. Lyth and S. King, “Natural language guidance of high- fidelity text-to-speech with synthetic annotations,”arXiv preprint arXiv:2402.01912, 2024

    2024 arXiv

  15. [23]

    V oxinstruct: Expressive human instruction-to-speech generation with unified multilingual codec language modelling,

    Y . Zhou, X. Qin, Z. Jin, S. Zhou, S. Lei, S. Zhou, Z. Wu, and J. Jia, “V oxinstruct: Expressive human instruction-to-speech generation with unified multilingual codec language modelling,” inProceed- ings of the 32nd ACM International Conference on Multimedia, 2024, pp. 554–563

    2024

  16. [24]

    Emovoice: Llm-based emotional text-to-speech model with freestyle text prompting,

    G. Yang, C. Yang, Q. Chen, Z. Ma, W. Chen, W. Wang, T. Wang, Y . Yang, Z. Niu, W. Liuet al., “Emovoice: Llm-based emotional text-to-speech model with freestyle text prompting,” inProceed- ings of the 33rd ACM International Conference on Multimedia, 2025, pp. 10 748–10 757

    2025

  17. [25]

    Scaling rich style- prompted text-to-speech datasets,

    A. Diwan, Z. Zheng, D. Harwath, and E. Choi, “Scaling rich style- prompted text-to-speech datasets,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Process- ing, 2025, pp. 3639–3659

    2025

  18. [26]

    Cosyvoice 3: Towards in-the- wild speech generation via scaling-up and post-training,

    Z. Du, C. Gao, Y . Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, C. Ni, X. Shiet al., “Cosyvoice 3: Towards in-the- wild speech generation via scaling-up and post-training,”arXiv preprint arXiv:2505.17589, 2025

    2025 arXiv

  19. [27]

    Vevo2: A unified and controllable frame- work for speech and singing voice generation,

    X. Zhang, J. Zhang, Y . Wang, C. Wang, Y . Chen, D. Jia, Z. Chen, and Z. Wu, “Vevo2: A unified and controllable frame- work for speech and singing voice generation,”arXiv preprint arXiv:2508.16332, 2025

    2025

  20. [28]

    Mela-tts: Joint transformer-diffusion model with representation alignment for speech synthesis,

    K. An, Z. Zhang, C. Gao, Y . Li, Z. Peng, H. Wang, Z. Du, H. Zhao, Z. Gao, and X. Li, “Mela-tts: Joint transformer-diffusion model with representation alignment for speech synthesis,”arXiv preprint arXiv:2509.14784, 2025

    2025

  21. [29]

    Streammel: Real-time zero-shot text-to-speech via interleaved continuous autoregressive model- ing,

    H. Wang, Y . Yang, S. Liu, J. Li, L. Meng, Y . Liu, J. Zhou, H. Sun, Y . Lu, and Y . Qin, “Streammel: Real-time zero-shot text-to-speech via interleaved continuous autoregressive model- ing,”IEEE Signal Processing Letters, 2025

    2025

  22. [30]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,”Ad- vances in Neural Information Processing Systems, vol. 36, pp. 27 980–27 993, 2023

    2023

  23. [31]

    Bigvgan: A universal neural vocoder with large-scale training,

    S.-g. Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, “Bigvgan: A universal neural vocoder with large-scale training,” arXiv preprint arXiv:2206.04658, 2022

    2022 arXiv

  24. [32]

    Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Maet al., “Cosyvoice: A scalable multi- lingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,”arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  25. [33]

    Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,

    Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wanget al., “Cosyvoice 2: Scalable stream- ing speech synthesis with large language models,”arXiv preprint arXiv:2412.10117, 2024

    2024 arXiv

  26. [34]

    World: a vocoder-based high-quality speech synthesis system for real-time applications,

    M. Morise, F. Yokomori, and K. Ozawa, “World: a vocoder-based high-quality speech synthesis system for real-time applications,” IEICE TRANSACTIONS on Information and Systems, vol. 99, no. 7, pp. 1877–1884, 2016

    2016

  27. [35]

    Fast and reliable f0 estimation method based on the period extraction of vocal fold vi- bration of singing voice and speech,

    M. Morise, H. Kawahara, and H. Katayose, “Fast and reliable f0 estimation method based on the period extraction of vocal fold vi- bration of singing voice and speech,” inAudio Engineering Soci- ety Conference: 35th International Conference: Audio for Games. Audio Engineering S...

    2009

  28. [36]

    Continuous-token diffu- sion for speaker-referenced tts in multimodal llms,

    X. He, S. N. Ray, H. Mallidi, J.-H. Huang, A. Bellur, C. Chan- dak, M. Maruf, and V . Ravichandran, “Continuous-token diffu- sion for speaker-referenced tts in multimodal llms,”arXiv preprint arXiv:2510.12995, 2025

    2025

  29. [37]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

    H. He, Z. Shang, C. Wang, X. Li, Y . Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shiet al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” in2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 885–890

    2024

  30. [38]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,

    G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhanget al., “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021

    2021 arXiv

  31. [39]

    Seed-tts: A family of high-quality versatile speech generation models,

    P. Anastassiou, J. Chen, J. Chen, Y . Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gaoet al., “Seed-tts: A family of high-quality versatile speech generation models,”arXiv preprint arXiv:2406.02430, 2024

    2024 arXiv

  32. [40]

    The lj speech dataset,

    K. Ito and L. Johnson, “The lj speech dataset,” 2017

    2017

  33. [41]

    Semantic-vae: Semantic- alignment latent representation for better speech synthesis,

    Z. Niu, S. Hu, J. Choi, Y . Chen, P. Chen, P. Zhu, Y . Yang, B. Zhang, J. Zhao, C. Wanget al., “Semantic-vae: Semantic- alignment latent representation for better speech synthesis,”arXiv preprint arXiv:2509.22167, 2025

    2025

  34. [42]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,”arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  35. [43]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

    2023

  36. [44]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  37. [45]

    Drawspeech: Expressive speech synthesis using prosodic sketches as control conditions,

    W. Chen, S. Yang, G. Li, and X. Wu, “Drawspeech: Expressive speech synthesis using prosodic sketches as control conditions,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5

    2025

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.