FleSpeech: Flexibly Controllable Speech Generation with Various Prompts

Hanzhao Li; Jingbin Hu; Lei Xie; Qicong Xie; Shan Yang; Xinsheng Wang; Yuke Li

REVIEW 3 major objections 4 minor 3 cited by

FleSpeech: Flexibly Controllable Speech Generation with Various Prompts

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FleSpeech lets users control a generated voice with any mix of text, audio, and face prompts.

desk verdict A capable controllable-TTS system paper whose headline flexibility claim is only indirectly tested; worth reviewing but needs a direct conflict-prompt experiment and cleaner internal metrics. read the letter →

arxiv 2501.04644 v2 pith:Z6PGG7JM submitted 2025-01-08 eess.AS cs.SD

Hanzhao Li , Yuke Li , Xinsheng Wang , Jingbin Hu , Qicong Xie , Shan Yang , Lei Xie This is my paper · ORCID

classification eess.AScs.SD

keywords controllablespeechsynthesismultimodalpromptencodertext-to-speechvoiceconversionspeakingstyleeditingflowmatchingface-driventransfer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FleSpeech is a multi-stage speech generation framework that aims to make voice synthesis controllable by any combination of text, audio, and face prompts. The central claim is that because speech style and timbre are modeled at two different stages, a user can assign each attribute to a different prompt—for example, style from a text description and timbre from a reference recording. The paper proposes a multimodal prompt encoder that maps text and face representations into the space of reference audio embeddings, and a three-stage training strategy that keeps the generation backbone stable while learning these mappings. If the claim holds, users could request a voice like 'that woman's timbre, but angry' without recording anything new, and systems could generate a voice that plausibly matches a character's face.

What carries the argument

The multimodal prompt encoder (MPE) is the carrying mechanism. It is a query-based transformer, following the IP-Adapter design, that consumes embeddings from BERT-encoded text, CLIP-encoded face images, and a six-block reference audio encoder whose last block is average-pooled into a global style-timbre vector. The encoder runs a diffusion process that maps the prompt embeddings to the reference audio embedding space, and the resulting condition vector is fed separately into the language model stage (style-oriented) and the flow-matching stage (acoustic/timbre-oriented), with separate parameters for each stage.

What would settle it

Take a face image of a speaker whose voice has a distinctive accent or vocal quality not represented in the training labels, use it as the only prompt, and measure whether the synthesized voice matches that accent or quality in a forced-choice listening test. If listeners cannot identify the intended speaker from the generated voice, the face-to-embedding mapping is incomplete for attributes outside the annotated set.

Watch

Extended reading notes

Core claim

The paper's central claim is that speech synthesis can be flexibly controlled by arbitrary combinations of text, audio, and visual prompts, with each stage of a cascaded language-model-plus-flow-matching generator taking its own condition embedding. To make this work, FleSpeech trains a multimodal prompt encoder that uses learnable queries to extract speech-relevant information from each input modality, then runs a diffusion process that maps the query outputs to the same embedding space as a global, average-pooled reference audio embedding. The audio embedding serves as the anchor for all modalities: text and face prompts are only accepted insofar as they can be diffused into this space. The paper reports that this design lets the model separate style control (first stage) from timbre control (second stage), and shows objective and subjective results across single-prompt and multi-prompt conditions, plus extensions to speaking-style editing and voice conversion.

Load-bearing premise

The load-bearing premise is that a single average-pooled vector from a reference audio clip holds all the time-invariant style and timbre information a voice prompt needs, and that text or face prompts can be faithfully translated into that same vector space by the diffusion training.

Editorial extensions

If this is right

FleSpeech accepts prompts from any subset of modalities at inference, masking missing ones, so a user can go from text-only style control to full text+audio+face control in the same framework.
Style and timbre are modeled in separate stages, so a style text prompt can be combined with a reference audio prompt to adjust emotion while preserving a chosen speaker's voice.
Face prompts, when augmented with static and dynamic captions, generate speech whose timbre listeners match to the face, and can edit voice attributes such as age, BMI, and race at above 70% accuracy.
FleSpeech extends to speaking-style editing and voice conversion, modifying style or timbre from the same trained system without task-specific fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The two-stage design suggests a general recipe: any new speech attribute pair could be split across stages and conditioned by the same multimodal encoder, so the framework may scale to finer-grained control (e.g., rhythm in a third stage) without rearchitecting the generator.
Because text and face prompts are only defined through the reference audio embedding space, attributes that the audio prompt does not carry — like accent — should remain uncontrollable by text or face, a prediction consistent with the paper's stated limitation on face-derived information.
The released data collection pipeline could be reused to add new prompt modalities (e.g., gesture or scene captions) by generating descriptions and diffusing them into the same audio embedding space, effectively extending FleSpeech's input vocabulary at zero generator cost.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful thing here is the architecture: a two-stage semantic-LM plus flow-matching generator, with a multimodal prompt encoder that maps text and face prompts into reference-audio embedding space via a diffusion objective. That combination is genuinely new, and the paper gives it a serious workout: 50k hours of training data, ablations on face captions, and a broad evaluation suite using external predictors (Whisper, WavLM, UTMOS, emotion2vec, human MOS). The data-collection pipeline, while not yet released, is a real contribution for a field that lacks multimodal TTS data. Credit where due: this is a solid systems paper with competitive numbers on single-prompt tasks.

Now the soft spots, in proportion. The stress-test concern about stage-specific control is partly valid. The multi-prompt experiments in Section 6.2 deliberately pair emotional/neutral first-stage prompts with only neutral second-stage prompts, so they never put text style in direct conflict with audio style. That dodges the flagship "same timbre, different style" case. However, the speaking-style editing task in Section 6.3.1 does exactly that — text description for style, source audio for timbre — and FleSpeech preserves speaker similarity while getting 63.6% emotion accuracy, just below Audiobox. So the claim is not untested, but it is under-tested, and the paper should have included a direct conflicting-prompt experiment (e.g., angry text + calm reference) and reported whether the text style wins, loses, or blends.

Two smaller evaluation issues deserve referee attention. First, the face-attribute accuracy in Section D.1 uses an MLP trained on MPE speaker embeddings to predict BMI/age from those same embeddings; that is internally coupled and tells you little about true face-to-voice mapping. Second, objective metrics have no error bars or significance tests, and several baseline numbers are taken from other papers, which makes the comparisons hard to weigh. Code and data are promised only on acceptance, which is frustrating for a method that hinges on training details.

The central claim — flexible control across text, audio, and face prompts with stage-specific conditioning — is plausible and backed by enough independent evidence to take seriously. It deserves peer review, but the authors should be pushed to make the conflict test explicit, fix the circular face-attribute metric, and release the pipeline or at least the evaluation code. A clear revise-and-resubmit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FleSpeech, a multi-stage speech generation framework in which a language model predicts semantic tokens from phonemes plus a global condition embedding, and a flow-matching module predicts acoustic latents from semantic tokens plus another condition embedding. A multimodal prompt encoder (MPE) maps text descriptions, reference audio, and face images into a unified embedding space using a query-based encoder and a diffusion objective, with the reference audio embedding as the training target. The model is trained in three stages on LibriHeavy plus a newly collected 616-hour multimodal dataset, and is evaluated on single-prompt controllable TTS, multi-prompt combinations, speaking style editing, and face-based voice conversion, using WER, speaker similarity, UTMOS, emotion/gender/attribute accuracy, and subjective MOS. The authors claim that FleSpeech supports any combination of prompts and enables stage-specific control, such as adjusting style while preserving a target speaker's timbre.

Significance. If the central claims hold, FleSpeech would be a meaningful step beyond single-prompt controllable TTS: it provides a concrete recipe for unify text, audio, and face prompts in one generative framework and demonstrates competitive objective and subjective results on several tasks. The paper also contributes a multimodal data collection pipeline and releases audio samples. Strengths of the evaluation include the use of external predictors (Whisper, WavLM, emotion2vec, UTMOS) for the main intelligibility, similarity, and quality metrics, subjective MOS with confidence intervals, and ablations on face captions and face embeddings. However, the signature capability of stage-specific control is asserted rather than demonstrated, because the experiments never place the text style prompt and the audio prompt in conflict, and at least one secondary evaluation metric is trained on the model's own embeddings. These gaps are load-bearing for the paper's flagship examples, so the manuscript needs additional evidence before the central claim can be accepted.

major comments (3)

[§3.3, §6.2, Abstract] The stage-specific control claim is not tested in the reported experiments. Section 3.3 states that the reference audio embedding 'captures all time-invariant information, such as style and timbre' and is obtained by average-pooling the reference audio encoder output, with no disentanglement loss, attribute masking, or separate style/timbre encoders. In the multi-prompt experiments of Section 6.2, the second-stage audio prompt is always neutral or the first-stage text prompt is neutral, so the audio prompt never conflicts with the style specified by the text prompt. The abstract's headline scenarios—'Like this voice, but slower' and 'that woman's voice, but more angry'—are exactly the conflicting cases. The paper should either add experiments in which the text style prompt and the reference audio prompt specify different styles and report whether the generated speech follows the text style, or explicitly qualify the claim that FleSpeech can independently control style and timbre when both are specified in different stages.
[Appendix D.1, Table 4] The face-attribute accuracy used to support the voice conversion claim is internally coupled to the proposed model. Appendix D.1 says that speaker embeddings are extracted from MPE and an MLP-based predictor is trained to infer facial attributes (BMI, age, race) from those embeddings. If the MLP is trained on the same MPE embedding space used to generate the speech, high accuracy can reflect the predictor learning the model's own biases rather than an independent face-to-voice correspondence. The paper should report how the MLP is trained and evaluated, use an external or human evaluation of the edited attributes, or otherwise demonstrate that the accuracy is not an artifact of the coupling. This is load-bearing for the 'voice conversion by face caption' result in Table 4.
[Tables 1-3] Objective results are reported without error bars, confidence intervals, or significance tests, while several baselines numbers are taken from the original papers or reproduced by the authors (marked ♢ and †). Differences between FleSpeech and baselines are often small (e.g., Text-prompt emotion accuracy 66.7 vs. 63.5, Audio-prompt WER 5.8 vs. 7.6), and the claims of 'significantly better' or 'optimal performance' are not supported by statistics. The authors should provide bootstrap confidence intervals or significance tests for the objective metrics, and report the number of test utterances used for each metric so the reader can assess the reliability of the comparisons.

minor comments (4)

[Table 1] The header 'V olum' contains a typo and should read 'Volume'.
[§1 and §6.1.1] 'Comparsion' in the Section 6.1.1 heading and 'Langauge' in Section 3.2 are misspelled; please correct them.
[Appendix B] The description of masking says the audio modality prompt 'remains consistently present' while the main text says MPE accepts inputs from any modality and masks non-input modalities; please clarify whether audio is always required at training time and how inference with only text or face prompts is handled.
[Figure 3] The similarity matrices in Figure 3 are discussed as supporting the superiority of the full FleSpeech model, but no quantitative comparison of the matrices is given; a scalar summary (e.g., mean off-diagonal vs. diagonal margin) would make the claim more concrete.

Circularity Check

1 steps flagged · score 4.0 of 10

Core controllable-TTS results are independently evaluated, but the face-driven 'voice conversion' accuracy is read back from the model's own condition embedding through a fitted MLP, making that secondary capability claim partly circular.

fitted input called prediction [Appendix D.1 (Objective Metrics); result claimed in Section 6.3.2 and Table 4]
"For face attribute evaluation, we extract speaker embeddings from MPE and use a face classifier to predict Body Mass Index (BMI). Additionally, we apply the DeepFace (Serengil and Ozpinar, 2021) model to determine gender, race, and age. We then train an MLP-based predictor to infer facial attributes from the speaker embeddings, comparing the predicted attributes against the provided facial descriptions to compute the accuracy."

The 'editing accuracy' used in Section 6.3.2 to demonstrate FleSpeech's ability to edit speaker-timbre-related attributes is computed by training an MLP to decode facial attributes from the MPE's own condition embedding, not from the synthesized audio. Appendix B shows the MPE is trained to map text/face prompts into the reference-audio embedding with L1 and generation losses, so the condition embedding is built to contain the prompt's attributes; a classifier fitted to that embedding can recover those attributes even if the generated voice is unchanged. The reported accuracy is therefore a readout of the conditioning signal, not an independent measurement of the generated speech, so this particular 'prediction' reduces to the model's input by construction.

full rationale

FleSpeech's main controllable-TTS claims are evaluated with external predictors (Whisper for WER, WavLM for speaker similarity, emotion2vec for emotion, UTMOS and human MOS for quality/naturalness), so the core derivation is not circular and has independent grounding. The single genuinely circular component is the face-attribute/voice-conversion accuracy in Table 4, which is derived from the MPE condition embedding rather than from generated audio, with an MLP fitted to that same embedding. The stage-specific disentanglement concern (audio-prompt embedding contains both style and timbre, Section 3.3) is a real validity gap because the multi-prompt experiments only use neutral second-stage prompts, but it is an untested assumption, not a reduction of the derivation to its inputs, so it does not add a circularity step. Self-citations (Glow-WaveGAN, VecTok, AnyoneNet, etc.) are used as standard components rather than as load-bearing uniqueness or justification for the central claim, so they do not raise the score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on assumptions that a single audio embedding can capture all time-invariant style and timbre information, that discrete WavLM tokens preserve linguistic content while leaving style to the condition, and that text and face prompts can be mapped into that audio embedding space by a learned diffusion process. The paper provides internal validation (t-SNE, generation quality) but no external proof for these premises. The listed free parameters are design choices introduced without sensitivity analysis.

free parameters (4)

K-means vocabulary size = 300
Number of semantic tokens chosen for tokenization; affects the granularity of linguistic representation.
Masking probability in MPE training = 1/3 per non-audio modality
Prompts masked with one-third probability each for text and face modalities to enable any combination at inference.
Data filtering thresholds = SNR > 0.6, DNSMOS > 2.6, min 3 words
Thresholds chosen to retain clean talking-head videos; no sensitivity analysis provided.
Attribute interval split = mean +/- one standard deviation
Used to label pitch, speed, and volume as low/normal/high for both captioning and evaluation.

assumptions (5)

domain assumption Reference audio embedding captures all time-invariant information such as style and timbre.
Section 3.3 states this directly; the MPE maps text and visual prompts to this embedding space, so its completeness is required for the control signal.
domain assumption K-means discretization of WavLM features into 300 tokens primarily associates with linguistic information.
Section 3.2; if these tokens miss prosodic or paralinguistic information, flow matching must recover it all from the condition embedding.
domain assumption Text and visual prompts can be mapped to the reference audio embedding space through a diffusion process.
Section 3.3; this learnability premise underpins the MPE; it is validated only internally via t-SNE and generation quality.
domain assumption Pretrained models (BERT, CLIP, ShareGPT4V, Video-LLaVA, Whisper) provide faithful representations for the prompt modalities.
The pipeline relies on these external encoders for captioning and embedding; failures propagate to prompt conditioning.
domain assumption The collected multimodal dataset and its automatic captions are clean and attribute-faithful after the SNR, DNSMOS, and Whisper filters.
Section 4; the quality of text and face descriptions determines the upper bound of control accuracy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FleSpeech: Flexibly Controllable Speech Generation with Various Prompts." pith.science (2026). https://pith.science/paper/Z6PGG7JM

@misc{pith2026250104644,
  author       = {Pith},
  title        = {Pith review of: FleSpeech: Flexibly Controllable Speech Generation with Various Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z6PGG7JM}},
  note         = {Machine review of arXiv:2501.04644}
}

read the original abstract

Controllable speech generation methods typically rely on single or fixed prompts, hindering creativity and flexibility. These limitations make it difficult to meet specific user needs in certain scenarios, such as adjusting the style while preserving a selected speaker's timbre, or choosing a style and generating a voice that matches a character's visual appearance. To overcome these challenges, we propose \textit{FleSpeech}, a novel multi-stage speech generation framework that allows for more flexible manipulation of speech attributes by integrating various forms of control. FleSpeech employs a multimodal prompt encoder that processes and unifies different text, audio, and visual prompts into a cohesive representation. This approach enhances the adaptability of speech synthesis and supports creative and precise control over the generated speech. Additionally, we develop a data collection pipeline for multimodal datasets to facilitate further research and applications in this field. Comprehensive subjective and objective experiments demonstrate the effectiveness of FleSpeech. Audio samples are available at https://kkksuper.github.io/FleSpeech/

Figures

Figures reproduced from arXiv: 2501.04644 by the authors.

**Figure 2.** The model architecture of FleSpeech. when handling large-scale speech data. To address this challenge, language model-based voice conversion methods have begun to emerge (Wang et al., 2024a,b). Instead of changing timbre, style editing focuses on modifying the speech style while preserving linguistic content and timbre. VoxEditor (Sheng et al., 2024) introduces a voice attribute editing model that facilitates the … view at source ↗

**Figure 3.** Cosine similarity matrix of speaker embed [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Fundamental Frequency (F0) curve of the speech at different ages and BMI levels groups by gender F Visualizing the Relationship between Facial Attributes and Voice To further validate that FleSpeech can establish associations between facial attributes and voice charac…

**Figure 5.** Figure 5: TSNE visualization of MPE output embedding [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

JIS: A Speech Corpus of Japanese Idol Speakers with Various Speaking Styles
cs.SD 2025-06 conditional novelty 7.0 of 10

JIS is a new 169-speaker Japanese speech corpus of live idols, built to support listener-familiarity-based evaluation of TTS and VC speaker similarity.
IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
cs.CL 2025-06 conditional novelty 5.0 of 10

IndexTTS2 achieves precise token-count-based duration control and emotion/speaker disentanglement in an autoregressive zero-shot TTS, reporting SOTA WER, speaker similarity, and emotional fidelity.
MultiActor-Audiobook: Zero-Shot Audiobook Generation with Faces and Voices of Multiple Speakers
cs.SD 2025-05 conditional novelty 5.0 of 10

A zero-shot pipeline that creates character voices from AI-generated faces and LLM-written prosody instructions can produce expressive audiobooks without extra training or manual annotation, though human quality score...

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages · cited by 3 Pith papers

[3]

In Pro- ceedings of the 32nd ACM International Conference on Multimedia, pages 1255–1264

Speechcraft: A fine-grained expressive speech dataset with natural language description. In Pro- ceedings of the 32nd ACM International Conference on Multimedia, pages 1255–1264. Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. 2024. Libriheavy: A 50, 000 hours ASR corpus with punctuation casing and co...

arXiv 2024
[8]

arXiv preprint arXiv:2312.15185

emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185. Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. 2021. Meta-stylespeech: Multi- speaker adaptive text-to-speech generation. In In- ternational Conference on Machine Learning, pages 7748–7759. PMLR. Ziqian Ning, Qicong Xie, Pengcheng Zhu,...

arXiv 2021
[9]

In ICASSP 2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP)

Dnsmos p.835: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors. In ICASSP 2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab sys- tem for ...

arXiv 2022
[11]

CoRR, abs/2312.15821

Audiobox: Unified audio generation with nat- ural language prompts. CoRR, abs/2312.15821. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. Neural codec language models are zero-shot text to speech synthesizers. CoRR, abs/2301.02111. Disong Wang,...

arXiv 2023
[12]

Kazuki Yamauchi, Yusuke Ijima, and Yuki Saito

Towards controllable speech synthesis in the era of large language models: A survey. Kazuki Yamauchi, Yusuke Ijima, and Yuki Saito. 2024. Stylecap: Automatic speaking-style captioning from speech based on speech and language self-supervised learning models. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICA...

work page 2024
[13]

CoRR, abs/2308.06721

Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. CoRR, abs/2308.06721. Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. 2021. Flow-guided one-shot talking face gener- ation with a high-resolution audio-visual dataset. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 20...

arXiv 2021
[14]

In ACM Multimedia, pages 7513–7522

Unistyle: Unified style modeling for speaking style captioning and stylistic speech synthesis. In ACM Multimedia, pages 7513–7522. A Model Configurations The language model for semantic prediction adopts the LLaMA architecture with 16 layers and 16 attention heads. The hidden size and intermedi- ate size are 1024 and 4096, respectively. The flow matching ...

work page 2021
[15]

fast speaking rate

8 model. 8https://huggingface.co/openai/whisper-large-v3 SPK-Sim is used to evaluate the similarity be- tween the generated audio and the reference au- dio in terms of speaker characteristics. A higher SPK-Sim value indicates greater similarity between the synthesized speech and the reference audio in terms of the speaker’s identity. Here, we use WavLM-la...

work page 2023

Show all 15 references

[2017]

CoRR, abs/1711.05447

Emotional end-to-end neural speech synthe- sizer. CoRR, abs/1711.05447. Yi Lei, Shan Yang, Xinsheng Wang, and Lei Xie

arXiv
[2021]

Score-based generative modeling through stochastic differential equations. In ICLR. Lourdes Bernadete Rocha de Souza and Marquiony Mar- ques dos Santos. 2018. Body mass index and acoustic voice parameters: is there a relationship? Brazilian journal of otorhinolaryngology, 84(4...

2018
[2022]

IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 30:853–864

Msemotts: Multi-scale emotion transfer, pre- diction, and control for emotional speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 30:853–864. Yichong Leng, Zhifang Guo, Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yufei Liu, Dongchao Yang, Leying Z...

2024 arXiv
[2023]

In ICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5

Imaginary voice: Face-styled diffusion model for text-to-speech. In ICASSP 2023-2023 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. Minyoung Lee, Eunil Park, and Sungeun Hong. 2024. Fvtts : Face based voice synthesis for t...

2023
[2024]

CoRR, abs/2404.15275

Id-animator: Zero-shot identity-preserving human video generation. CoRR, abs/2404.15275. Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao. 2024a. Textrolspeech: A text style control speech corpus with codec language te...

2024 arXiv
[2025]

In European Confer- ence on Computer Vision, pages 370–387

Sharegpt4v: Improving large multi-modal models with better captions. In European Confer- ence on Computer Vision, pages 370–387. Springer. Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, and Tao Qin. 2020. Multispeech: Multi- speaker text to speech with transformer...

2020 arXiv
[7771]

Dan Lyth and Simon King

IEEE. Dan Lyth and Simon King. 2024. Natural language guid- ance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01912. Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen

2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [3]

In Pro- ceedings of the 32nd ACM International Conference on Multimedia, pages 1255–1264

Speechcraft: A fine-grained expressive speech dataset with natural language description. In Pro- ceedings of the 32nd ACM International Conference on Multimedia, pages 1255–1264. Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. 2024. Libriheavy: A 50, 000 hours ASR corpus with punctuation casing and co...

arXiv 2024

[2] [8]

arXiv preprint arXiv:2312.15185

emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185. Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. 2021. Meta-stylespeech: Multi- speaker adaptive text-to-speech generation. In In- ternational Conference on Machine Learning, pages 7748–7759. PMLR. Ziqian Ning, Qicong Xie, Pengcheng Zhu,...

arXiv 2021

[3] [9]

In ICASSP 2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP)

Dnsmos p.835: A non-intrusive perceptual objective speech quality metric to evaluate noise sup- pressors. In ICASSP 2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab sys- tem for ...

arXiv 2022

[4] [11]

CoRR, abs/2312.15821

Audiobox: Unified audio generation with nat- ural language prompts. CoRR, abs/2312.15821. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. Neural codec language models are zero-shot text to speech synthesizers. CoRR, abs/2301.02111. Disong Wang,...

arXiv 2023

[5] [12]

Kazuki Yamauchi, Yusuke Ijima, and Yuki Saito

Towards controllable speech synthesis in the era of large language models: A survey. Kazuki Yamauchi, Yusuke Ijima, and Yuki Saito. 2024. Stylecap: Automatic speaking-style captioning from speech based on speech and language self-supervised learning models. In ICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICA...

work page 2024

[6] [13]

CoRR, abs/2308.06721

Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. CoRR, abs/2308.06721. Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan. 2021. Flow-guided one-shot talking face gener- ation with a high-resolution audio-visual dataset. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 20...

arXiv 2021

[7] [14]

In ACM Multimedia, pages 7513–7522

Unistyle: Unified style modeling for speaking style captioning and stylistic speech synthesis. In ACM Multimedia, pages 7513–7522. A Model Configurations The language model for semantic prediction adopts the LLaMA architecture with 16 layers and 16 attention heads. The hidden size and intermedi- ate size are 1024 and 4096, respectively. The flow matching ...

work page 2021

[8] [15]

fast speaking rate

8 model. 8https://huggingface.co/openai/whisper-large-v3 SPK-Sim is used to evaluate the similarity be- tween the generated audio and the reference au- dio in terms of speaker characteristics. A higher SPK-Sim value indicates greater similarity between the synthesized speech and the reference audio in terms of the speaker’s identity. Here, we use WavLM-la...

work page 2023