Pith

open record

sign in

arxiv: 2607.06461 · v1 · pith:3R5AFDRK · submitted 2026-07-07 · eess.AS · cs.CL· cs.SD

WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

Sihang Nie , Jinxin Ji , Xiaofen Xing , Deyi Tuo , Chengbin Jin , Jialong Mai , Xiangmin Xu This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-07-08 04:54 UTCglm-5.2pith:3R5AFDRKrecord.jsonopen to challenge →

Figure 3
Figure 3. Figure 3: Built upon the CosyVoice3 [30] backbone, our frame- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] reproduced from arXiv: 2607.06461
classification eess.AS cs.CLcs.SD
keywords text-to-speechword-level controlprosody planningbound tokenacoustic attributesflow matchingspeech synthesisLLM-based TTS
0
0 comments X

The pith

Word-level acoustic control for LLM-based speech synthesis

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that current LLM-based text-to-speech systems, while natural-sounding, cannot give users explicit, per-word control over acoustic attributes like duration, pitch, energy, prosodic boundary, and tone. The authors claim that by introducing a special boundary token (⟨b⟩) into the autoregressive LLM's generation sequence, the model can be forced to explicitly plan five acoustic attributes for each word before generating the corresponding speech tokens. This transforms speech generation from an implicit end-to-end mapping into an explicit, interpretable acoustic planning process. They pair this with a fine-grained style modulation module in the waveform synthesis stage to ensure that the word-level attributes planned by the LLM survive the lossy token-to-waveform conversion. The paper also introduces WordVoice-5A, a 4.7k-hour bilingual dataset annotated with these five word-level attributes via a linguistically-guided pipeline. The central claim is that this dual-stage architecture achieves decoupled, precise control across all five dimensions simultaneously, while maintaining competitive zero-shot synthesis quality.

Core claim

The paper discovers that inserting a bound-token mechanism into an LLM-based TTS pipeline restructures the causal generation process so that acoustic attribute prediction becomes an explicit planning step rather than a byproduct of end-to-end generation. When the LLM emits the boundary token ⟨b⟩ for a given word, it is routed to predict five discrete attributes (duration, boundary level, energy, pitch, tone) that form a style token, which then conditions the generation of that word's speech chunk. This enables a dual-mode system: the model can autonomously plan prosody (free mode) or users can override specific attributes for any word (control mode) without retraining. The ablation confirms:

What carries the argument

The bound-token mechanism (⟨b⟩): a special token inserted into the LLM's autoregressive sequence that triggers a lightweight Word Decoder to predict five acoustic attributes per word, forming a style token that conditions both the LLM's speech chunk generation and the downstream Flow Matching waveform synthesizer's frame-level modulation.

If this is right

  • Audiobook and video dubbing workflows could gain deterministic, per-word control over timing and emphasis, enabling precise lip-sync alignment and expressive narration without retraining models.
  • The bound-token planning architecture could generalize to other modalities where fine-grained, per-unit control is needed in autoregressive generation, such as music generation or video frame synthesis.
  • The five-dimensional word-level annotation pipeline could become a standard for speech corpora, enabling broader prosodic research beyond what utterance-level datasets support.
  • The dual-mode (free/control) paradigm suggests a path toward interactive TTS where users iteratively refine specific words' prosody while the model handles the rest autonomously.

Load-bearing premise

The objective control-precision metrics (MAE and ER) are computed by extracting timestamps from generated audio using the same forced aligner (Qwen3-FA) that was used to extract the ground-truth annotations in the training data pipeline. Any systematic bias in this aligner would inflate apparent control precision, since both the target labels and the measurement share the same tool's biases. No human-validated gold-standard subset is used to verify annotation accuracy.

What would settle it

If independent human annotation of word-level acoustic attributes on a held-out subset showed that the Qwen3-FA-derived labels diverge substantially from perceptual ground truth, the reported control-precision gains would be called into question.

read the original abstract

While recent Large Language Model (LLM)-based Text-to-Speech (TTS) systems have achieved remarkable naturalness, they predominantly rely on implicit end-to-end generation paradigms, resulting in coarse-grained control. In scenarios demanding precise stylistic interventions and strict temporal alignment, such as audiobook narration and video dubbing, the inability to explicitly manipulate word-level acoustic attributes remains a critical bottleneck. This limitation is primarily amplified by the severe scarcity of fine-grained annotated datasets and the architectural challenge of integrating multi-dimensional control signals into discrete autoregressive generation. To address this, we propose a unified framework for highly precise word-level control. First, we construct WordVoice-5A, a massive 4.7k-hour bilingual dataset featuring five-dimensional word-level annotations (duration, boundary, energy, pitch and tone) developed through a rigorous linguistically-guided pipeline. Second, we introduce WordVoice to transform the implicit generation process into an explicit, highly controllable paradigm. Specifically, we introduce a bound-token mechanism within the LLM to formulate an explicit ``acoustic planning'' process, enabling adaptive multi-task prosodic planning and flexible manual intervention. Furthermore, we augment the token-to-waveform stage with a fine-grained acoustic modulation module, bridging the resolution gap to strictly align word-level attributes between highly compressed discrete tokens and continuous waveforms. Extensive experiments demonstrate that WordVoice achieves superior, decoupled control over multiple acoustic dimensions while maintaining competitive zero-shot synthesis stability. The code and audio samples are publicly available at https://xxh333.github.io/wordvoice-demo/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 9 minor

Summary. The paper proposes WordVoice, a framework for explicit, multi-dimensional word-level control in LLM-based TTS. The work has two main contributions: (1) WordVoice-5A, a 4.7k-hour bilingual dataset with five-dimensional word-level annotations (duration, boundary, energy, pitch, tone) produced via a linguistically-guided pipeline, and (2) a dual-stage architecture combining a bound-token mechanism for macro-prosodic planning in the LLM with a fine-grained style modulation module in the Flow Matching stage. The system supports both autonomous (free) and user-specified (control) inference modes. Experiments include subjective MOS evaluations with significance testing, objective MAE/ER metrics, decoupled single-attribute control verification, ablation studies, and comparison with MagicTTS. The core architectural design is sound, and the decoupled control claim is supported by the diagonal pattern in Table 6. However, the objective evaluation methodology has a measurement circularity concern that needs to be addressed.

Significance. The paper addresses a genuine gap: existing LLM-based TTS systems lack explicit, multi-dimensional word-level control, and no large-scale word-level annotated dataset with five acoustic dimensions currently exists. The bound-token mechanism (§3.2) is a clean architectural solution that restructures causal generation to interleave acoustic planning with speech token generation, and the dual-mode inference design is practical. The decoupled single-attribute control experiment (Table 6, Part 1) provides a falsifiable test of the independence claim, and the ablation study (Table 6, Part 2) cleanly isolates the contribution of the WV-FM module. The open-sourced dataset and code, along with the public demo page, are notable strengths that enhance reproducibility. The work is positioned clearly against existing paradigms (Table 1) and the comparison with MagicTTS (Table 5) demonstrates multi-attribute control without sacrificing precision relative to single-attribute systems.

major comments (3)
  1. §4.1: The objective control-precision metrics (MAE, ER in Table 4) are computed by extracting timestamps from generated audio using Qwen3-FA, which is also a core component of the WordVoice-5A annotation pipeline (§2.2.1). While the annotation pipeline applies additional MFA cross-validation and loudness optimization that the evaluation does not, any systematic alignment bias in Qwen3-FA would be shared between the target labels and the measurement, potentially deflating WordVoice-Control's errors relative to a truly independent measurement. The Ground-Truth row in Table 4 (e.g., Dur-MAE=0.0142 for zh) confirms measurement noise but does not rule out shared systematic bias. This is load-bearing for the claim of 'superior, decoupled control' because the magnitude of improvement in Table 4 is a primary quantitative pillar. The authors should either (a) re-evaluate a subset using an独立 align
  2. §2.2.1: The consistency check retains 'only the top 8% highest-quality aligned data.' This aggressive filtering raises two concerns: (1) it may bias the dataset toward utterances where MFA and Qwen3-FA already agree, which would further amplify the measurement circularity concern above; and (2) the resulting dataset distribution may not be representative of general speech, limiting the generalizability claims. The paper should report what fraction of the original LEMAS data survives this filter and discuss whether the retained subset is distributionally representative.
  3. Table 4: WordVoice-Control shows higher WER than CosyVoice3 (2.86% vs 2.31% for zh; 1.57% vs 1.06% for en), which the authors attribute to structural conditioning perturbing linguistic robustness. However, WordVoice-Free also shows higher WER than CosyVoice3 (2.58% vs 2.31% for zh). Since WordVoice-Free does not use explicit control, the WER increase in free mode suggests the bound-token architecture itself introduces a robustness cost independent of the control mechanism. The paper should clarify whether this is an inherent trade-off of the interleaved generation structure or a training artifact, as it affects the claim of 'maintaining competitive zero-shot synthesis stability.'
minor comments (9)
  1. §2.2.1: The thresholds for the consistency check conditions (No Overlap, Duration Divergence, Edge Divergence) are described as 'calibrated to retain only the top 8%' but the specific threshold values are not provided. These should be specified or included in a supplementary document.
  2. §3.4: The uncertainty weights σ_k in Eq. (4) are listed as free parameters. The paper should report the learned values of σ_k for each attribute task, as these reveal the relative difficulty the model assigns to each attribute and would strengthen the analysis.
  3. Table 3: The Spk-MOS for WordVoice-Control (3.324 for zh) is lower than CosyVoice3 (3.527 for zh), and the difference does not appear to be tested for significance. Given that the paper claims 'competitive zero-shot synthesis stability,' a significance test on Spk-MOS would clarify whether this drop is meaningful.
  4. §4.2.3, Fig. 4: The 2D density heatmaps are described but the figure quality in the manuscript makes it difficult to verify the 'pronounced diagonal alignment' claim for WordVoice-Control. Consider using clearer colormap annotations or providing numerical diagonal-concentration statistics.
  5. §2.2.3: The decision tree thresholds for mapping continuous F0 contours to the seven tone categories are described as 'designed by phonetic experts' but not specified. Providing these thresholds (or the tree structure) would improve reproducibility of the annotation pipeline.
  6. §3.3: The masking ratio of 30% for speech tokens during FM training is stated without justification. A brief rationale or sensitivity analysis would strengthen this design choice.
  7. Table 5: The comparison with MagicTTS is on a 100-utterance subset, which is small. Confidence intervals or bootstrap analysis would help assess the reliability of the reported differences.
  8. §4.1: The test set composition (2,000 Chinese and 1,500 English utterances) should specify whether these are held-out speakers or held-out utterances from seen speakers, as this affects the zero-shot evaluation protocol.
  9. Reference [8] (MagicTTS) and several other 2026 references appear to be very recent; ensure all citations are to published or stable versions.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee correctly identifies the core architectural contributions and the falsifiable decoupled control evidence. We address each major comment below. For the measurement circularity concern, we will conduct an independent re-evaluation using MMS-FA. For the dataset filtering concern, we will report survival rates and add distributional analysis. For the WER concern, we will add discussion of the bound-token architecture's inherent trade-offs.

read point-by-point responses

  1. Referee: §4.1: Measurement circularity — Qwen3-FA used in both annotation pipeline and evaluation, potentially deflating WordVoice-Control's errors. The Ground-Truth row confirms measurement noise but not absence of shared systematic bias.

    Authors: The referee raises a valid and important methodological concern. We acknowledge that using Qwen3-FA in both the annotation pipeline (§2.2.1) and the objective evaluation (§4.1) creates a potential for shared systematic bias, even though the annotation pipeline additionally applies MFA cross-validation and loudness optimization that the evaluation does not. The Ground-Truth row in Table 4 (Dur-MAE=0.0142 for zh) characterizes measurement noise but, as the referee correctly notes, does not rule out correlated biases between label generation and measurement. We agree this is load-bearing for the 'superior, decoupled control' claim since Table 4 is a primary quantitative pillar. In the revision, we will re-evaluate a representative subset (approximately 200 utterances per language) using MMS-FA as a fully independent aligner — MMS-FA is already used in our inference pipeline (§3.4.1) but is architecturally independent of Qwen3-FA and was not used in the WordVoice-5A annotation pipeline. We will report these supplementary results alongside the original Table 4 metrics and explicitly discuss the measurement circularity limitation. If the independent evaluation confirms the same trend, this strengthens our claims; if it reveals bias inflation, we will report the corrected numbers honestly. revision: yes

  2. Referee: §2.2.1: The 8% retention filter may bias the dataset toward MFA–Qwen3-FA agreement, amplifying measurement circularity, and may limit distributional representativeness. Report survival fraction and discuss representativeness.

    Authors: This is a fair concern. The 8% retention rate is indeed aggressive, and we agree that it could bias the dataset toward utterances where MFA and Qwen3-FA already agree, which would compound the measurement circularity issue identified above. In the revision, we will: (1) report the fraction of the original LEMAS data that survives each stage of the filtering pipeline (dual-model extraction, loudness optimization, and consistency check), making the 8% figure transparent and contextualized; (2) provide a distributional analysis comparing the retained subset against the full LEMAS corpus on key dimensions — speaker distribution, utterance duration, speaking rate, and language balance — to assess representativeness; and (3) add an explicit discussion of the trade-off between annotation reliability and dataset coverage, noting that the aggressive filtering prioritizes label quality for training a controllable system but may limit generalization to noisier or more conversational speech. We acknowledge this is a genuine limitation of the current dataset construction and will state it as such. revision: yes

  3. Referee: Table 4: WordVoice-Free shows higher WER than CosyVoice3 (2.58% vs 2.31% for zh; 1.31% vs 1.06% for en), suggesting the bound-token architecture itself introduces a robustness cost independent of the control mechanism. Clarify whether this is an inherent trade-off or a training artifact.

    Authors: The referee's observation is accurate and well-taken. WordVoice-Free does not use explicit user-specified control, yet it still shows a WER increase over CosyVoice3. This indicates that the bound-token interleaved generation structure itself introduces a modest robustness cost, not just the control-mode conditioning. We believe this is primarily an inherent architectural trade-off rather than a pure training artifact, for the following reason: the bound-token mechanism restructures the causal generation sequence to interleave acoustic planning tokens (⟨b⟩, v_i) with speech chunks (s_i), which increases the effective sequence length and introduces additional decision points (five attribute predictions per word) before each speech chunk. This expanded autoregressive dependency chain creates more opportunities for error propagation compared to CosyVoice3's direct text-to-token generation. However, we cannot fully rule out a training artifact contribution — for instance, the multi-task loss balancing (Eq. 4) may allocate capacity away from linguistic fidelity. In the revision, we will: (1) clarify this as an inherent trade-off of the interleaved structure, with the caveat that training-time factors may also contribute; (2) note that the WER increase is modest (0.27% absolute for zh, 0.25% for en) relative to the substantial control-precision gains; and (3) soften the claim of 'maintaining competitive zero-shot synthesis stability' to more precisely state that WordVoice maintains competitive quality with a minor, quantified trade-off in linguistic robustness that is inherent to the explicit acoustic planning paradigm. revision: yes

Circularity Check

0 steps flagged

No structural circularity found; the shared Qwen3-FA between annotation and evaluation is a measurement-validity concern, not a derivation that reduces to its inputs by construction.

full rationale

The paper's central claims rest on two independent pillars: (1) an architectural innovation (bound-token mechanism + Flow Matching modulation) that restructures the generation sequence, and (2) experimental evaluation comparing against baselines (CosyVoice3, MagicTTS) and ablations. Neither pillar reduces to its inputs by construction. The reader's concern about Qwen3-FA being shared between the annotation pipeline (§2.2.1) and evaluation (§4.1) is a legitimate measurement-validity risk, but it does not constitute structural circularity in the sense of the patterns checked here: (a) the training labels are cross-validated between MFA and Qwen3-FA with strict consistency filtering, so they are not purely Qwen3-FA outputs; (b) in WordVoice-Control mode, the model must still generate speech tokens and waveforms that, when re-extracted by Qwen3-FA, match the target attributes—this is not guaranteed by definition, as evidenced by CosyVoice3's much higher MAE using the same evaluation tool; (c) the Ground-Truth row in Table 4 showing non-zero errors confirms the measurement introduces independent noise rather than being tautologically zero. The subjective Ctrl-MOS results (Table 3) provide partially independent support via human listeners. Self-citations exist (refs [6], [8], [31] share authors) but are cited as related work, not as load-bearing premises for any derivation step. No equation or prediction in the paper is equivalent to its input by construction. The measurement concern is best classified as correctness risk, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The paper introduces one genuinely new architectural entity (bound token) whose effectiveness is tested, and one new dataset whose quality is not independently validated. The free parameters are mostly standard hyperparameter choices (quantization bins, masking ratio) that are typical for TTS systems but under-justified. The domain assumptions about forced alignment accuracy and truncation strategies are load-bearing for the dataset quality, which in turn is load-bearing for all downstream claims.

free parameters (6)
  • Uncertainty weights σ_k = learned during training
    Multi-task loss weights for the five attribute CE losses are learned via the uncertainty weighting scheme (Eq. 4), not set by theory.
  • Quantization bins for pitch/energy = 20 uniform bins
    Pitch and energy are uniformly quantized into 20 discrete bins (§3.4); this bin count is chosen ad hoc without justification.
  • Duration frame rate = 40ms
    Duration quantization frame rate of 40ms is stated without derivation or sensitivity analysis.
  • Speech token masking ratio = 30%
    30% of input speech tokens are randomly masked during FM training (§3.4) to enforce reliance on style tokens; this ratio is not tuned or justified.
  • Timestamp search region = 10% shift of adjacent word durations
    The valid search region for timestamp optimization is restricted to 10% of adjacent word durations (§2.2.1) without sensitivity analysis.
  • Tone resampling points = 16 points
    F0 contour is resampled to 16 points for polynomial regression in tone classification (§2.2.3); this number is not justified.
axioms (5)
  • domain assumption Forced alignment tools (MFA, Qwen3-FA) produce timestamps accurate enough to serve as ground truth for word-level acoustic attribute extraction.
    The entire annotation pipeline (§2.2.1) depends on this. The consistency check filters divergent alignments but does not validate against human annotations.
  • domain assumption Averaging the top 50% of frame-level RMS energy and central 80% of F0 contour yields meaningful word-level acoustic representations.
    These truncation choices (§2.2.3) are motivated by phonetic practice but not validated against perceptual or expert annotations in this paper.
  • domain assumption Seven prosodic morphologies (flat, rise, strong rise, fall, strong fall, peak, valley) are sufficient to represent word-level tone for both tonal and intonational languages.
    The tone classification system (§2.2.3) assumes these 7 categories capture meaningful prosodic variation; no comparison with alternative category counts is provided.
  • domain assumption Discrete quantization of continuous acoustic attributes into classification targets is sufficient for LLM-based prediction.
    The paper follows pGSLM's approach (§3.4) of discretizing all continuous values, assuming the quantization resolution does not lose critical information.
  • domain assumption CosyVoice3 is a representative baseline for LLM-based TTS quality.
    The paper compares primarily against CosyVoice3 (§4.1), assuming it represents the current quality frontier; other 2026 SOTAs (IndexTTS2, Qwen3-TTS, Moss-TTS) are cited but not compared.
invented entities (3)
  • Bound token ⟨b⟩ independent evidence
    purpose: Triggers word-level acoustic attribute prediction within the LLM's autoregressive sequence, restructuring generation into explicit acoustic planning followed by speech token chunks.
    The bound token is a structural mechanism whose effectiveness is tested via ablation (Table 6) and whose control precision is measured objectively (Table 4). It is falsifiable: if it did not enable control, the control-mode MAE would not decrease.
  • Word-level style token c_i independent evidence
    purpose: Dense representation of five acoustic attributes that conditions both LLM speech token generation and FM waveform synthesis.
    The style token's contribution is ablated in Table 6 Part 2, where removing WV-FM style modulation degrades Eng-MAE and Pit-MAE significantly.
  • WordVoice-5A dataset no independent evidence
    purpose: 4.7k-hour bilingual dataset with five-dimensional word-level annotations for training and evaluation.
    The dataset is constructed by the authors' own pipeline and evaluated using the same tools (Qwen3-FA) used for annotation. No external human-validated gold standard is used to verify annotation quality.

pith-pipeline@v1.1.0-glm · 16270 in / 3625 out tokens · 380695 ms · 2026-07-08T04:54:00.283340+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages · 7 internal anchors

  1. [1]

    WordVoice: Explicit and Decoupled Multi-Dimensional Word-Level Control for LLM-Based TTS

    INTRODUCTION Recent advancements in controllable Text-to-Speech (TTS) have enabled expressive style manipulation. Existing ap- proaches primarily fall into two paradigms. The first relies on global embedding control [1, 2, 3], which achieves sat- isfactory results in specific domains but suffers from coarse control granularity and limited generalization. ...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Overview of WordV oice-5A Several large-scale open-source speech corpora have sig- nificantly advanced the development of zero-shot TTS [14, 15, 16]

    WORD-LEVEL ANNOTA TION PIPELINE FOR WORDVOICE-5A 2.1. Overview of WordV oice-5A Several large-scale open-source speech corpora have sig- nificantly advanced the development of zero-shot TTS [14, 15, 16]. However, these datasets are typically limited to utterance-level transcripts and instructions. Although the re- cent LEMAS dataset [17] provides word-lev...

    work page

  3. [3]

    acoustic planning

    PROPOSED METHOD: WORDVOICE 3.1. Method Overview Mainstream LLM-based TTS models [27, 28, 29] typically cascade an autoregressive LLM for discrete speech token gen- eration with a non-autoregressive model (e.g., Flow Match- ing) for waveform synthesis. While yielding impressive zero- shot performance, this paradigm treats speech as an undif- ferentiated st...

    work page

  4. [4]

    Experimental Setup We evaluate our method on the WordV oice-5A-test set, com- prising about 2,000 Chinese and 1,500 English unseen ut- terances

    EXPERIMENTS 4.1. Experimental Setup We evaluate our method on the WordV oice-5A-test set, com- prising about 2,000 Chinese and 1,500 English unseen ut- terances. For zero-shot synthesis, the audio corresponding to the first 30% of the text in each utterance is extracted as the prompt audio to guide the generation of the remaining content. While recent SOT...

    work page arXiv 2030

  5. [5]

    CONCLUSION In this paper, we present WordV oice to overcome the coarse- grained limitations of current LLM-based TTS, achieving precise, multi-dimensional word-level control. This advance- ment is driven by two core contributions: WordV oice-5A, a 4.7k-hour dataset with linguistically-guided acoustic annota- tions; and a dual-stage architecture integratin...

    work page

  6. [6]

    Emosphere++: Emotion-controllable zero- shot text-to-speech via emotion-adaptive spherical vector,

    Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, and Seong-Whan Lee, “Emosphere++: Emotion-controllable zero- shot text-to-speech via emotion-adaptive spherical vector,” IEEE Transactions on Affective Computing, 2025

    work page 2025

  7. [7]

    Facespeak: Expressive and high-quality speech synthesis from human portraits of different styles,

    Tian-Hao Zhang, Jiawei Zhang, Jun Wang, Xinyuan Qian, and Xu-Cheng Yin, “Facespeak: Expressive and high-quality speech synthesis from human portraits of different styles,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2025, vol. 39, pp. 25922–25930

    work page 2025

  8. [8]

    Scalable controllable accented tts,

    Henry Li Xinyuan, Zexin Cai, Ashi Garg, Kevin Duh, Leibny Paola Garc´ıa-Perera, Sanjeev Khudanpur, Nicholas An- drews, and Matthew Wiesner, “Scalable controllable accented tts,” in2025 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), 2025, pp. 1–8

    work page 2025

  9. [9]

    Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt,

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng, “Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 2913–2925, 2024

    work page 2024

  10. [10]

    CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,”arXiv preprint arXiv:2412.10117, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    Hd-ppt: Hierarchical decoding of content- and prompt-preference tokens for instruction-based tts,

    Sihang Nie, Xiaofen Xing, Jingyuan Xing, Baiji Liu, and Xiangmin Xu, “Hd-ppt: Hierarchical decoding of content- and prompt-preference tokens for instruction-based tts,” in ICASSP 2026-2026 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2026, pp. 16487–16491

    work page 2026

  12. [12]

    Deep dubbing: End-to-end auto-audiobook system with text-to-timbre and context-aware instruct-tts,

    Ziqi Dai, Yiting Chen, Jiacheng Xu, Liufei Xie, Yuchen Wang, Zhenchuan Yang, Bingsong Bai, Yangsheng Gao, Wenjiang Zhou, Weifeng Zhao, et al., “Deep dubbing: End-to-end auto-audiobook system with text-to-timbre and context-aware instruct-tts,” inICASSP 2026-2026 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 202...

    work page 2026

  13. [13]

    MAGIC-TTS: Fine-Grained Controllable Speech Synthesis with Explicit Local Duration and Pause Control

    Jialong Mai, Xiaofen Xing, and Xiangmin Xu, “Magic-tts: Fine-grained controllable speech synthesis with explicit local duration and pause control,”arXiv preprint arXiv:2604.21164, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  14. [14]

    Synctalk: The devil is in the synchronization for talking head synthesis,

    Ziqiao Peng, Wentao Hu, Yue Shi, Xiangyu Zhu, Xiaomei Zhang, Hao Zhao, Jun He, Hongyan Liu, and Zhaoxin Fan, “Synctalk: The devil is in the synchronization for talking head synthesis,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 666–676

    work page 2024

  15. [15]

    Di- flowdubber: Discrete flow matching for automated video dub- bing via cross-modal alignment and synchronization,

    Ngoc-Son Nguyen, Thanh VT Tran, Jeongsoo Choi, Hieu- Nghia Huynh-Nguyen, Truong-Son Hy, and Van Nguyen, “Di- flowdubber: Discrete flow matching for automated video dub- bing via cross-modal alignment and synchronization,” inPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 5838–5848

    work page 2026

  16. [16]

    Fastspeech 2: Fast and high-quality end-to- end text to speech,

    Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu, “Fastspeech 2: Fast and high-quality end-to- end text to speech,” inInternational Conference on Learning Representations, 2021

    work page 2021

  17. [17]

    Conditional vari- ational autoencoder with adversarial learning for end-to-end text-to-speech,

    Jaehyeon Kim, Jungil Kong, and Juhee Son, “Conditional vari- ational autoencoder with adversarial learning for end-to-end text-to-speech,” inInternational conference on machine learn- ing. PMLR, 2021, pp. 5530–5540

    work page 2021

  18. [18]

    Word-level emotional expres- sion control in zero-shot text-to-speech synthesis,

    Tianrui Wang, Haoyu Wang, Meng Ge, Cheng Gong, Chunyu Qiang, Ziyang Ma, Zikang Huang, Guanrou Yang, Xiaobao Wang, Eng-Siong Chng, et al., “Word-level emotional expres- sion control in zero-shot text-to-speech synthesis,”Advances in Neural Information Processing Systems, vol. 38, pp. 147377– 147405, 2026

    work page 2026

  19. [19]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” in2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 885–890

    work page 2024

  20. [20]

    Textrolspeech: A text style control speech corpus with codec language text-to-speech models,

    Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao, “Textrolspeech: A text style control speech corpus with codec language text-to-speech models,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). IEEE, 2024, pp. 10301–10305

    work page 2024

  21. [21]

    Speechcraft: A fine-grained expressive speech dataset with natural language description,

    Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Song- tao Zhou, Xiaoyu Qin, and Zhiyong Wu, “Speechcraft: A fine-grained expressive speech dataset with natural language description,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 1255–1264

    work page 2024

  22. [22]

    Lemas: Large a 150k-hour large-scale extensible mul- tilingual audio suite with generative speech models,

    Zhiyuan Zhao, Lijian Lin, Ye Zhu, Kai Xie, Yunfei Liu, and Yu Li, “Lemas: Large a 150k-hour large-scale extensible mul- tilingual audio suite with generative speech models,”arXiv preprint arXiv:2601.04233, 2026

    work page arXiv 2026

  23. [23]

    Praat script to detect syl- lable nuclei and measure speech rate automatically,

    Nivja H. de Jong and Ton Wempe, “Praat script to detect syl- lable nuclei and measure speech rate automatically,”Behavior Research Methods, vol. 41, no. 2, pp. 385–390, May 2009

    work page 2009

  24. [24]

    Tobi: A standard for labeling english prosody,

    Kim Silverman, Mary Beckman, John Pitrelli, Mari Ostendorf, Colin Wightman, Patti Price, Janet Pierrehumbert, and Julia Hirschberg, “Tobi: A standard for labeling english prosody,” in Proceedings of the Second International Conference on Spoken Language Processing, 1992, pp. 867–870

    work page 1992

  25. [25]

    Chinese prosody and prosodic labeling of sponta- neous speech,

    Aijun Li, “Chinese prosody and prosodic labeling of sponta- neous speech,” inSpeech Prosody 2002, 2002, pp. 39–46

    work page 2002

  26. [26]

    Smoothing and dif- ferentiation of data by simplified least squares procedures,

    Abraham Savitzky and Marcel JE Golay, “Smoothing and dif- ferentiation of data by simplified least squares procedures,” Analytical chemistry, vol. 36, no. 8, pp. 1627–1639, 1964

    work page 1964

  27. [27]

    Monotone piece- wise cubic interpolation,

    Frederick N Fritsch and Ralph E Carlson, “Monotone piece- wise cubic interpolation,”SIAM Journal on Numerical Analy- sis, vol. 17, no. 2, pp. 238–246, 1980

    work page 1980

  28. [28]

    Sound level protrusions as physical correlates of sonority,

    Steve Parker, “Sound level protrusions as physical correlates of sonority,”Journal of Phonetics, vol. 36, no. 1, pp. 55–90, 2008

    work page 2008

  29. [29]

    Considerations in the normalisation of the funda- mental frequency of linguistic tone,

    Phil Rose, “Considerations in the normalisation of the funda- mental frequency of linguistic tone,”Speech Communication, vol. 6, no. 4, pp. 343–352, 1987

    work page 1987

  30. [30]

    Acoustic features of mandarin tone production in noise: A comparison between chinese native speakers and korean l2 learners,

    Jinxin Ji, Yiying Hu, Xiaohu Yang, and Gang Peng, “Acoustic features of mandarin tone production in noise: A comparison between chinese native speakers and korean l2 learners,” in Interspeech 2025, 2025, pp. 4448–4452

    work page 2025

  31. [31]

    Modeling tone and intonation in mandarin and english as a process of target approximation,

    Santitham Prom-on, Yi Xu, and Bundit Thipakorn, “Modeling tone and intonation in mandarin and english as a process of target approximation,”The Journal of the Acoustical Society of America, vol. 125, no. 1, pp. 405–424, 2009

    work page 2009

  32. [32]

    Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech,

    Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, and Jingchen Shu, “Indextts2: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text-to-speech,” inProceedings of the AAAI Confer- ence on Artificial Intelligence, 2026, vol. 40, pp. 35139–35148

    work page 2026

  33. [33]

    Qwen3-TTS Technical Report

    Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zis- han Guo, et al., “Qwen3-tts technical report,”arXiv preprint arXiv:2601.15621, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  34. [34]

    Moss-tts technical report,

    Yitian Gong, Botian Jiang, Yiwei Zhao, Yucheng Yuan, Kuangwei Chen, Yaozhou Jiang, Cheng Chang, Dong Hong, Mingshu Chen, Ruixiao Li, et al., “Moss-tts technical report,” arXiv preprint arXiv:2603.18090, 2026

    work page arXiv 2026

  35. [35]

    CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training

    Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al., “Cosyvoice 3: Towards in-the-wild speech gen- eration via scaling-up and post-training,”arXiv preprint arXiv:2505.17589, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech

    Sihang Nie, Xiaofen Xing, Rui Xing, Haoming Li, Ruitong Xiao, Jingyuan Xing, Baiji Liu, and Xiangmin Xu, “Hpro: Hierarchical progressive reward optimization via preference extraction for emotional text-to-speech,”arXiv preprint arXiv:2606.28249, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  37. [37]

    Qwen2.5 Technical Report

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al., “Qwen2.5 technical report,”arXiv preprint arXiv:2412.15115, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    Text- free prosody-aware generative spoken language modeling,

    Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, et al., “Text- free prosody-aware generative spoken language modeling,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8666–8681

    work page 2022

  39. [39]

    Multi-task learning using uncertainty to weigh losses for scene geome- try and semantics,

    Alex Kendall, Yarin Gal, and Roberto Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geome- try and semantics,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7482– 7491

    work page 2018

  40. [40]

    Scaling speech tech- nology to 1,000+ languages,

    Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al., “Scaling speech tech- nology to 1,000+ languages,”Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024

    work page 2024

  41. [41]

    An overview of voice conversion and its challenges: From statistical modeling to deep learning,

    Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li, “An overview of voice conversion and its challenges: From statistical modeling to deep learning,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 132– 157, 2020

    work page 2020