Pith. sign in

REVIEW 3 major objections 7 minor 45 references

Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Chain-Talker claims that chaining emotion understanding, semantic understanding, and empathetic rendering yields more expressive and empathetic conversational speech than existing CSS models.

desk verdict A well-engineered three-stage chain for empathetic CSS with a genuinely useful captioning dataset, but the headline emotion metric (ACC_m) is circular and the human edge over emotion-label variants is modest. read the letter →

arxiv 2505.12597 v1 pith:R5DI43VF submitted 2025-05-19 cs.SD eess.AS

Yifan Hu , Rui Liu , Yi Ren , Xiang Yin , Haizhou Li This is my paper · ORCID
classification cs.SDeess.AS
keywords conversationalspeechsynthesisempatheticemotionunderstandingcaptioningchainmodelingsemantictokensdialoguecontextexpressivetext-to-speech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conversational speech synthesis is supposed to make a voice agent respond with the right emotion for the dialogue. Chain-Talker's central claim is that this is best done not by predicting speech tokens directly from context, but by following a chain: first infer an empathetic caption for the target utterance from the dialogue history, then generate compact semantic speech codes, and only then render speech conditioned on both. To make that first step trainable at scale, the paper introduces CSS-EmCap, a pipeline that uses an LLM to annotate about 384 hours of dialogue from three datasets with natural-language emotion captions. On those datasets, Chain-Talker reports higher naturalness and expressiveness ratings, higher emotion accuracy, lower pitch-distance to ground truth, and higher speaker similarity than the CSS baselines. The point of caring is that the caption becomes an interpretable, controllable intermediate representation between conversation context and acoustic expression.

What carries the argument

The machinery is the cognitive chain encoded as a single autoregressive model, EmGPT, whose token sequence is ordered so that emotion comes before semantics before acoustics. A supervised ASR model with a vector quantizer supplies compact semantic speech codes, distinguishing them from unsupervised codes like HuBERT that mix acoustic and semantic information. The Synthesizer uses an optimal-transport conditional flow matching model, conditioned on the predicted caption, semantic codes, speaker identity, and masked mel-spectrogram, with HiFi-GAN as vocoder. The CSS-EmCap pipeline generates the empathetic captions by extracting sentence-level style factors (gender, pitch, energy, tempo) and dialogue-level emotion, then prompting an LLM to write and diversify basic descriptions while checking consistency with the audio.

What would settle it

Pick a random sample of NCSSD-EmCap utterances and have independent human annotators write emotion captions and label emotion category and style factors; compute agreement with the CSS-EmCap captions. If agreement is low, or if a CSS model trained on human-written captions outperforms Chain-Talker in DMOS-E and ACC_m, the reported empathy gains would not be attributable to reliable emotion understanding.

Watch

Extended reading notes

Core claim

The paper's discovery is that a sequential chain of two understanding steps plus a rendering step beats direct context-to-token prediction for empathetic speech. In the paper's own terms, EmGPT first predicts the empathetic caption from dialogue history and target text, then serially predicts the semantic code sequence, and the Synthesizer turns caption, codes, speaker embedding, and masked mel-spectrogram into waveform via optimal-transport conditional flow matching. The contribution is two-sided: chain modeling provides interpretability, and the CSS-EmCap annotation pipeline provides the training signal. Experiments show Chain-Talker outperforming GPT-Talker and other CSS baselines on all reported dialogue-level metrics, e.g., DMOS-E 4.239 vs 3.913, ACC_m 0.612 vs 0.562, DDTW 38.784 vs 44.625, and that removing the caption loss or the context degrades results.

Load-bearing premise

The load-bearing premise is that the automatically generated empathetic captions accurately and completely describe the true emotion and style of each conversation turn; if this fails, the training signal and the main evaluation metrics become unreliable.

Editorial extensions

If this is right

  • If the chain ordering is right, adding an explicit emotion-caption prediction stage to any CSS model should improve empathetic expressiveness without extra human annotation.
  • Semantic-only speech codes should replace mixed acoustic-semantic codes in conversational speech synthesis, since Chain-Talker's ablations tie its gains partly to cleaner semantic tokens.
  • The caption loss is load-bearing: removing it drops naturalness and expressiveness scores by about 0.2 to 0.28, so the emotion-understanding objective is not decorative.
  • Large-scale single-sentence pretraining remains necessary before dialogue fine-tuning; skipping first-stage pretraining drops all reported metrics.
  • The model generalizes to dialogue lengths beyond training, suggesting further gains from training on even longer conversational contexts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the caption is an editable control surface—rewriting an empathetic caption should change the emotion and style of the synthesized speech without retraining, since rendering is conditioned directly on it.
  • Editorial inference: because CSS-EmCap is fully automatic, the same annotation recipe could be applied to new languages or domains; whether caption quality and downstream gains survive that transfer is an open empirical question.
  • Editorial inference: the reported diversity measures count distinct word n-grams, not distinct acoustic renditions; a stricter test would ask whether different captions for the same style combination actually produce perceptibly different prosody.
  • Editorial inference: the paper's own latency note points to streaming or faster decoding as the natural next development, but Chain-Talker itself is not real-time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Chain-Talker, a three-stage framework for conversational speech synthesis (CSS) that separates emotion understanding, semantic understanding, and empathetic rendering. To obtain training targets for emotion understanding, the authors propose CSS-EmCap, a pipeline that extracts sentence-level style factors (gender, pitch, energy, tempo) and dialog-level emotion labels, and uses Gemini to generate free-form empathetic captions. They annotate three CSS datasets (NCSSD, DailyTalk, MultiDialog) totaling 384 hours, and compare Chain-Talker against several CSS baselines and ablations using subjective MOS and objective metrics including caption semantic similarity (SIM*), emotion accuracy (ACC_m), DDTW, and SSIM. The main claims are that Chain-Talker produces more expressive and empathetic speech than existing methods, and that CSS-EmCap contributes reliable emotion modeling.

Significance. If the claims hold, Chain-Talker provides a useful step toward interpretable CSS by decoupling emotion and semantic understanding, and CSS-EmCap offers a scalable alternative to manual emotion annotation for dialogue speech. The paper is reproducible in spirit: the authors release code and demos, the pipeline is described in enough detail to re-implement, and the dataset statistics are reported. However, the empirical evidence for the central claim is weakened by circularity in two of the main objective metrics, so the significance currently rests more on the architecture and resource contribution than on the demonstrated empathy gains.

major comments (3)
  1. [Section 6.3 (SIM*) and Section 5 (CSS-EmCap)] The SIM* metric as defined compares generated captions to "descriptions composed of all Ground Truth style factors (e.g., 'gender is female, pitch is high...')" using cosine similarity. CSS-EmCap's caption generation in Section 5.1 and Appendix A.1 explicitly feeds these exact style factors (gender, pitch, energy, tempo) plus dialog-level emotion into the LLM prompt. Therefore the high SIM* scores for CSS-EmCap in Table 1 (0.568/0.694) are to a large degree by construction: the captions are generated from the very factors used as the similarity reference. This makes SIM* unsuitable as evidence that the pipeline "more accurately reflect[s] the real style." The independent human DMOS-C result in Table 1 is meaningful, but the SIM* claims should be either removed or reframed as a check of prompt adherence rather than caption quality.
  2. [Section 6.3 (ACC_m) and Section 5 (CSS-EmCap)] The emotion accuracy metric ACC_m is computed by prompting Gemini to classify the emotion of synthesized speech. The same model family (Gemini 1.5 Pro) was used in CSS-EmCap to extract dialog-level emotion labels (Section 5.1, Appendix A.1) and to generate the empathetic captions that serve as the training target for the Emotion Understanding stage (Section 5.2). Consequently, ACC_m does not measure agreement with an independent ground truth; it measures agreement with the same model that defined the emotional categories and generated the training captions. A model trained to mimic Gemini's stylistic and emotional judgments could score higher on this metric without producing speech that human listeners perceive as more empathetic. The paper should replace or supplement ACC_m with a human-annotated emotion test set or an independently trained speech emotion recognition model, and report the correlation with human judgments.
  3. [Section 6.5, Table 2, and Appendix B.4] The specific evidence that free-form empathetic captions, rather than simple emotion labels, improve empathetic speech synthesis is limited. In Table 2, Chain-Talker's DMOS-E advantage over Chain-Talker_e (the emotion-label variant) is 0.112 (4.239 vs 4.127), and the advantage over GPT-Talker_c is 0.137. These differences are statistically non-overlapping given the reported confidence intervals, but the evaluation is based on 50 utterances rated by 30 non-native English speakers (Appendix B.4). Since ACC_m is circular for the reasons given above, the subjective DMOS-E margin is the only non-circular evidence specifically supporting empathetic captions over simpler conditioning. Given the small absolute margin and limited test size, the paper should report effect sizes, per-utterance agreement, or an additional human evaluation with more diverse listeners and a larger sample before claiming that CSS-EmCap provides "reliable emotion modeling."
minor comments (7)
  1. [Section 4 heading] The heading reads "Methodlogy" and should be "Methodology."
  2. [Throughout] The name "CosyVoice" is inconsistently typeset as "CosyV oice" in several places (e.g., Sections 4.1, 4.4, 4.5, and the Limitations section).
  3. [Section 6.4] The sentence "the observation that DMOS-C values surpass those of the Ground Truth demonstrates that empathetic captions described in natural language are superior to style and emotion labels" is confusing. The text should clarify what "Ground Truth" refers to in Table 1 (e.g., human-written captions, style labels, or emotion labels) so that the comparison is interpretable.
  4. [Table 3] Some style-factor counts are extremely imbalanced (e.g., DailyTalk Energy Low=2 and Normal=52; MultiDialog Energy Normal=0). The authors should comment on whether these imbalances reflect the underlying data or a thresholding artifact, as they may affect caption diversity and model robustness.
  5. [Section 6.8 and Figure 3(b)] The caption of Figure 3(b) contains "=3, =0.61", which appears to be missing variable names (presumably N=3 and ACC_m=0.61). The text also uses the notation "N=1→3" without defining that it means the number of dialogue turns.
  6. [Appendix B.3, Table 5] The text claims Chain-Talker "excels in emotional expression by 0.092%" over the second-best model. This should be stated as an absolute percentage-point difference (e.g., 0.092 percentage points), not a relative percentage.
  7. [Table 6] The table title contains a typo: "Emapthetic Caption" should be "Empathetic Caption."

Circularity Check

2 steps flagged · score 6.0 of 10

ACC_m and SIM* are self-referential: Gemini grades the model trained on Gemini labels, and SIM* compares captions to the style factors that generated them.

  1. self definitional [Section 6.3 (Metrics, SIM*); Sections 5.1–5.2 (CSS-EmCap caption generation)]
    ""Semantic Similarity (SIM*): We utilize RoBERTa ... and mGTE ... to encode captions generated with different methods and descriptions composed of all Ground Truth style factors (e.g., 'gender is female, pitch is high...'). The semantic similarity is then calculated using cosine similarity. Higher values mean that the captions more accurately reflect the real style." — §6.3. "we pre-extract two types of key expressive attributes (style factors and emotion) from conversational speech before generation" — §5.1."

    The SIM* reference is a description composed of exactly the style factors (gender, pitch, energy, tempo) that CSS-EmCap itself extracts and feeds into the LLM when generating captions. The captions are therefore paraphrases (with LLM embellishment) of those same factors plus Gemini emotion labels; comparing them to a bag of the same factors measures self-similarity, not independent accuracy. High SIM* is expected by construction even if the captions misdescribe the speech, because the generative pipeline starts from these exact factors. This makes the claim that CSS-EmCap captions 'more accurately reflect the real style' circular with respect to the pipeline's own inputs.

  2. fitted input called prediction [Section 6.3 (Metrics, ACC_m); Sections 5.1–5.2 and Appendix A.1 (CSS-EmCap emotion labeling)]
    ""Emotion Accuracy (ACC m): We calculate the emotion accuracy of synthesized speech using Gemini." — §6.3. "we use multimodal information such as speech, text, and speaker data to prompt LLM ... to accurately distinguish the emotional category of each sentence within the dialogue context" — §5.1. "we employ Gemini 4 to generate diverse natural language descriptions for each speech" — §5.2. Appendix A.1: "we carefully design prompts to enable the state-of-the-art Gemini 1.5 pro model ... to return results accurately.""

    The empathetic captions that define the Emotion Understanding training target are generated by Gemini from Gemini-extracted dialog-level emotion labels. ACC_m then asks Gemini to classify the emotion of synthesized speech and checks agreement with those same Gemini labels. Chain-Talker is therefore trained to imitate Gemini's stylistic and emotional judgments and evaluated by Gemini's own classifier. High ACC_m reflects mimicry of the annotator/evaluator rather than an external ground truth. Since the same model family (Gemini 1.5 Pro) both produced the labels embedded in the training captions and grades the output, ACC_m is, to a significant degree, measuring agreement with the system's own annotation, not independently verified empathy.

full rationale

The three-stage Chain-Talker architecture itself is not circular: Emotion Understanding predicts empathetic captions, Semantic Understanding predicts semantic codes, and Empathetic Rendering synthesizes speech; the losses and the OT-CFM rendering are standard and self-contained. No load-bearing self-citation chain or imported uniqueness theorem is present. However, two of the headline evaluation supports are self-referential. First, SIM* validates CSS-EmCap captions by comparing them to descriptions composed of the same style factors that CSS-EmCap extracts and feeds into the LLM caption generator, so the reported caption-quality advantage is in part a self-similarity artifact. Second, ACC_m uses Gemini as the emotion classifier; the same Gemini model family was used to extract the dialog-level emotion labels and to generate the empathetic captions that serve as the Chain-Talker training target. The model is thus optimized to match Gemini's annotations and then scored by Gemini's own judgment, making the objective empathy gain partly circular. The human DMOS-E result is genuinely external evidence, but the margin over the emotion-label variant is only 0.112 and over GPT-Talker_c is 0.137, so once ACC_m and SIM* are discounted, the remaining independent support for free-form empathetic captions is thin rather than definitive.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The main free parameters are the annotation thresholds and model hyperparameters, which are fitted to the datasets or tuned during development. The axioms are domain assumptions about the reliability of the LLM-generated captions and the semantic tokenizer. There are no new physical entities.

free parameters (3)
  • Style factor thresholds = pitch [136.577, 196.098], tempo [0.252, 0.386], energy [0.033, 0.0505]
    Uniform thresholds used in CSS-EmCap to categorize low/normal/high for pitch, tempo, and energy. These are derived from the statistics of the annotated datasets (Appendix A.1) and affect the style factors that condition caption generation.
  • Number of dialogue turns N during training = 1 to 3 (with inference up to 4)
    The context window length is a hyperparameter. The best results occur at N=3 (Section 6.8), so this is a tuned parameter that affects model performance.
  • Top-k sampling parameters = top_k=25, tau_r=0.1, win_size=10
    Decoding hyperparameters for EmGPT (Table 4) that influence caption and code generation. They are chosen by the authors and affect the reported quality.
assumptions (3)
  • domain assumption The supervised semantic speech tokens from the ASR-based tokenizer capture purely semantic information and are an effective conditioning signal for expressive speech synthesis.
    Chain-Talker relies on CosyVoice's semantic tokenizer (Section 4.1 and 4.3). If these tokens also carry style/emotion information or lose prosodic detail, the decoupling of emotion and semantics is incomplete.
  • domain assumption Gemini's emotion classification and natural language caption generation are reliable, consistent, and reflect the true expressive content of the speech.
    CSS-EmCap uses Gemini 1.5 Pro to assign dialog-level emotion labels and to generate/rewrite captions (Section 5). All training targets and several evaluation metrics depend on these outputs.
  • domain assumption The pretrained CosyVoice-300M-25Hz model provides a strong prior for natural single-sentence TTS.
    The First-Stage training directly uses CosyVoice's pretrained weights (Section 4.5). The final system inherits its strengths and weaknesses, so the contribution is an adaptation rather than a from-scratch model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis." pith.science (2026). https://pith.science/paper/R5DI43VF

@misc{pith2026250512597,
  author       = {Pith},
  title        = {Pith review of: Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5DI43VF}},
  note         = {Machine review of arXiv:2505.12597}
}
read the original abstract

Conversational Speech Synthesis (CSS) aims to align synthesized speech with the emotional and stylistic context of user-agent interactions to achieve empathy. Current generative CSS models face interpretability limitations due to insufficient emotional perception and redundant discrete speech coding. To address the above issues, we present Chain-Talker, a three-stage framework mimicking human cognition: Emotion Understanding derives context-aware emotion descriptors from dialogue history; Semantic Understanding generates compact semantic codes via serialized prediction; and Empathetic Rendering synthesizes expressive speech by integrating both components. To support emotion modeling, we develop CSS-EmCap, an LLM-driven automated pipeline for generating precise conversational speech emotion captions. Experiments on three benchmark datasets demonstrate that Chain-Talker produces more expressive and empathetic speech than existing methods, with CSS-EmCap contributing to reliable emotion modeling. The code and demos are available at: https://github.com/AI-S2-Lab/Chain-Talker.

Figures

Figures reproduced from arXiv: 2505.12597 by the authors.

Figure 1
Figure 1. (a) Previous methods predict speech tokens [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall architecture of Chain-Talker. Chain-Talker comprises two main components: EmGPT and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) The ability of Chain-Talker and various baseline models to synthesize target speech with different [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The overall process of CSS-EmCap, It includes extracting Sentence-level style factors and Dialog-level [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: A sample set of conversational data annotated using the CSS-EmCap pipeline. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 24 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Xiuying Chen, Zhi Cui, Jiayi Zhang, Chen Wei, Jianwei Cui, Bin Wang, Dongyan Zhao, and Rui Yan. 2021. Reasoning in dialog: Improving response generation by context reading comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12683--12691

    work page 2021

  4. [4]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. https://doi.org/10.48550/ARXIV.2407.10759 Qwen2-audio technical report . CoRR, abs/2407.10759

    arXiv doi:10.48550/arxiv.2407.10759 2024

  5. [5]

    Yayue Deng, Jinlong Xue, Fengping Wang, Yingming Gao, and Ya Li. 2023. CMCU-CSS: enhancing naturalness via commonsense-based multi-modal context understanding in conversational speech synthesis. In Proceedings of the 31st ACM International Conference on Multimedia , pages 6081--6089

    work page 2023

  6. [6]

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407

    arXiv 2024

  7. [7]

    Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23--38

    1994

  8. [8]

    Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, and Shiliang Zhang. 2023. https://doi.org/10.21437/INTERSPEECH.2023-1428 Funasr: A fundamental end-to-end speech recognition toolkit . In 24th Annual Conference of the International Speech Communication Association, Interspeech 2023, Dublin, Ireland, ...

    work page doi:10.21437/interspeech.2023-1428 2023

Show all 45 references
  1. [9]

    GPT-SoVITS. 2024. https://github.com/RVC-Boss/GPT-SoVITS

    2024

  2. [10]

    Haohan Guo, Shaofei Zhang, Frank K Soong, Lei He, and Lei Xie. 2021. Conversational end-to-end tts for voice agents. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 403--409. IEEE

    2021

  3. [11]

    Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. 2023. Prompttts: Controllable text-to-speech with text descriptions. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1--5

    2023

  4. [12]

    Wei - Ning Hsu, Benjamin Bolte, Yao - Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process. , 29:3451--3460

    2021

  5. [13]

    Zhaopei Huang, Jinming Zhao, and Qin Jin. 2024. Ecr-chain: Advancing generative language models to better emotion-cause reasoners through reasoning chains. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 , pages 6288--6296...

    2024

  6. [14]

    Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics, pages 37--42

    2023

  7. [15]

    Razan Jaber, Sabrina Zhong, Sanna Kuoppam \"a ki, Aida Hosseini, Iona Gessinger, Duncan P Brumby, Benjamin R Cowan, and Donald McMillan. 2024. Cooking with agents: Designing context-aware voice interaction. In Proceedings of the CHI Conference on Human Factors in Computing Sys...

    2024

  8. [16]

    Garima Jain, Amita Shukla, Nitesh Kumar Bairwa, Anamika Chaudhary, Ashish Patel, and Ankush Jain. 2024. Spear: Design and implementation of an advanced virtual assistant. In 2024 4th International Conference on Sustainable Expert Systems (ICSES), pages 1715--1720. IEEE

    2024

  9. [17]

    Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao. 2024. Textrolspeech: A text style control speech corpus with codec language text-to-speech models. In 2024 IEEE International Conference on Acoustics, Speech and Signal...

    2024

  10. [18]

    Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Chen Zhang, Zhenhui Ye, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun Ma, et al. 2023. Mega-tts 2: Zero-shot text-to-speech with arbitrary length speech prompts. arXiv preprint arXiv:2307.07218

    2023 arXiv

  11. [19]

    Masaya Kawamura, Ryuichi Yamamoto, Yuma Shirahata, Takuya Hasumi, and Kentaro Tachibana. 2024. Libritts-p: A corpus with speaking style and speaker identity prompts for text-to-speech and style captioning. CoRR, abs/2406.07969

    2024 arXiv

  12. [20]

    Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Sungroh Yoon, and Kang Min Yoo. 2024. Unified speech-text pretraining for spoken dialog modeling. arXiv preprint arXiv:2402.05706

    2024 arXiv

  13. [21]

    Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530--5540. PMLR

    2021

  14. [22]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. https://proceedings.neurips.cc/paper/2020/hash/c5d736809766d46260d816d8dbc9eb44-Abstract.html Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis . In Advances in Neural Information Proc...

    2020

  15. [23]

    Keon Lee, Kyumin Park, and Daeyoung Kim. 2023. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1--5

    2023

  16. [24]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055

    2015 arXiv

  17. [25]

    Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, et al. 2024. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark. arXiv preprint arXiv:2410.01495

    2024 arXiv

  18. [26]

    Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko. 2024. Paralinguistics-enhanced large language modeling of spoken dialogue. In 2024 IEEE International Conference on Acoustics,...

    2024

  19. [27]

    Guanghou Liu, Yongmao Zhang, Yi Lei, Yunlin Chen, Rui Wang, Lei Xie, and Zhifei Li. 2023 a . Promptstyle: Controllable style transfer for text-to-speech with natural language descriptions. In INTERSPEECH, pages 4888--4892

    2023

  20. [28]

    Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, and Haizhou Li. 2024 a . Emotion rendering for conversational speech synthesis with heterogeneous graph-based context modeling. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):18698--18706

    2024

  21. [29]

    Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, and Haizhou Li. 2024 b . Generative expressive conversational speech synthesis. In ACM Multimedia 2024

    2024

  22. [30]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  23. [31]

    Yuchen Liu, Haoyu Zhang, Shichao Liu, Xiang Yin, Zejun Ma, and Qin Jin. 2023 b . https://doi.org/10.1145/3581783.3613823 Emotionally situated text-to-speech synthesis in user-agent conversation . In Proceedings of the 31st ACM International Conference on Multimedia , pages 5966--5974

    2023

  24. [32]

    Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. 2016. World: a vocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, pages 1877--1884

    2016

  25. [33]

    Meinard M \"u ller. 2007. Information retrieval for music and motion, volume 2. Springer

    2007

  26. [34]

    Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. 2024. Spoken question answering and speech continuation using spectrogram-powered LLM . In The Twelfth Intern...

    2024

  27. [35]

    Yuto Nishimura, Yuki Saito, Shinnosuke Takamichi, Kentaro Tachibana, and Hiroshi Saruwatari. 2022. Acoustic Modeling for End-to-End Empathetic Dialogue Speech Synthesis Using Linguistic and Prosodic Contexts of Dialogue History . In Interspeech 2022, pages 3373--3377

    2022

  28. [36]

    Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.860 Let's go real talk: Spoken dialogue model for face-to-face conversation . In Proceedings of the 62nd Annual Meeting of the Ass...

    2024 doi

  29. [37]

    Alec Radford. 2018. Improving language understanding by generative pre-training

    2018

  30. [38]

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. Neural codec language models are zero-shot text to speech synthesizers. CoRR

    2023

  31. [39]

    Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi - Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. 2024. https://doi.org/10.1609/AAAI.V38I17.29902 Secap: Speech emotion captioning with large language model . In Thirty-Eighth AAAI Conference on Artificial ...

    2024 doi

  32. [40]

    Jinlong Xue, Yayue Deng, Fengping Wang, Ya Li, Yingming Gao, Jianhua Tao, Jianqing Sun, and Jiaen Liang. 2023. M 2-ctts: End-to-end multi-scale multi-modal conversational text-to-speech synthesis. In 2023 IEEE International Conference on Acoustics, Speech and Signal Processing...

    2023

  33. [41]

    Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng. 2024. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 2913--2925

    2024

  34. [42]

    Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2022. https://doi.org/10.1109/TASLP.2021.3129994 Soundstream: An end-to-end neural audio codec . IEEE ACM Trans. Audio Speech Lang. Process. , 30:495--507

    2022

  35. [43]

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.1055 Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities . In Findings of the Association for...

    2023 doi

  36. [44]

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407.19669

    2024 arXiv

  37. [45]

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. 2023. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5168--5191

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.