REVIEW 5 major objections 6 minor 7 references
JoyTTS: LLM-based Spoken Chatbot With Voice Cloning
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read JoyTTS claims that a spoken chatbot can clone a voice by feeding the chat LLM's hidden states into a CosyVoice2 TTS module, and reports a speaker similarity of 0.73 with a word error rate of 5.09 on seed-tts-zh.
desk verdict Open-source integration of MiniCPM-o and CosyVoice2 whose central hidden-state claim is contradicted by its own Table 1. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the TTS embed, the sum of an LLM token embedding and an MLP-projected hidden state: TTS_embed = Emb(y_i) + MLP(h_i). The MLP maps the chat model's 3584-dimensional hidden states down to 768 dimensions so that they can be added directly to the text embeddings consumed by the CosyVoice2-based LLM-TTS module; at inference the same conditioning is built from the prompt text and prompt wave. This one equation is what lets a single spoken-chatbot pipeline carry speaker identity and conversational context into speech-token generation. Training proceeds in two stages, with the LLM-Chat and LLM-TTS modules first specialized separately and then jointly fine-tuned through a unified loss.
What would settle it
Record short utterances from a set of speakers who never appear in the training corpora, run JoyTTS with those recordings as the prompt wave, and compare SS and WER with the same prompts passed through CosyVoice2; if JoyTTS's speaker similarity drops well below its reported 0.73 while CosyVoice2 stays near 0.748, the synthetic training signal is the cause. Separately, run inference with TTS_embed set to Emb(y_i)+MLP(h_i) and to Emb(y_i) alone; if SS, WER, and latency are unchanged, the paper's hidden-state claim is not supported.
Extended reading notes
Core claim
The paper's central claim is that voice cloning in a spoken chatbot can be achieved by the fusion defined in Eq. (1): TTS_embed = Emb(y_i) + MLP(h_i), where h_i are the chat LLM's 3584-dimensional hidden states projected to 768 dimensions and added to the text-token embeddings before they enter the LLM-TTS module. With this conditioning, the prompt text and prompt wave are both passed through the LLM-Chat module at inference, supplying the TTS module with semantic context and speaker identity. Trained in two stages—separate LLM-Chat and LLM-TTS training followed by joint fine-tuning with a combined loss—JoyTTS reports SS 0.73 and WER 5.09 on seed-tts-zh, comparable to CosyVoice2's SS 0.748 and better than gpt-sovits's SS 0.55 at the same WER scale. The authors take this as evidence that hidden-state integration enhances both quality and speed for cloned-voice dialogue.
Load-bearing premise
The decisive assumption is that 2000 hours of dialogue audio created by CosyVoice2—the very model family used as the speech generator—teaches the model to clone real human voices rather than merely to imitate CosyVoice2's synthetic voice distribution.
Editorial extensions
If this is right
- A released training recipe now exists for an end-to-end spoken chatbot with voice cloning, so other groups can reproduce and modify the pipeline rather than starting from scratch.
- Because cloning is conditioned through the chat LLM's hidden states, the same trick could be carried to larger or smaller chat backbones without changing the TTS module.
- At 1.8 seconds of latency on one consumer GPU, the architecture is close enough to conversational turn-taking to be tested in real interactive settings.
- The SEED results anchor a three-way comparison among gpt-sovits, CosyVoice2, and JoyTTS, giving practitioners a concrete reference point for cloned-voice spoken dialogue.
- The paper's proposed next step, emotion-control inputs into the LLM, has a natural insertion point through the same TTS-embed conditioning path.
Reading between the lines
- Beyond the paper: because all 2000 hours of training audio were synthesized by CosyVoice2 itself, the reported SS may measure how well JoyTTS clones voices as rendered by CosyVoice2, not how well it clones arbitrary real speakers; testing on real out-of-corpus recordings is needed.
- Beyond the paper: the claim that hidden states improve performance and reduce latency is not isolated by an ablation in Table 1; a direct A/B test with and without the MLP(h_i) term would confirm or refute that causal story.
- Beyond the paper: the same hidden-state conditioning could be applied to other LLM-TTS pairs, or extended to carry explicit emotion or style embeddings, which would give the architecture a cheap route to expressive control.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents JoyTTS, an end-to-end spoken chatbot that combines the MiniCPM-o LLM with a CosyVoice2-based TTS module through a hidden-state injection pathway (Eq. 1). The system is trained on about 2000 hours of multi-turn dialogue audio synthesized by CosyVoice2, with random reference speakers from WenetSpeech4TTS. On the SEED test-zh set, the authors report a speaker similarity (SS) of 0.73 and a word error rate (WER) of 5.09, as well as a 1.8-second latency on a single NVIDIA 4090D. The central claim is that injecting hidden states into the TTS embeddings simultaneously enhances performance and reduces latency, and the paper provides open-sourced training and inference code.
Significance. If the central claim were validated, JoyTTS would be a valuable open-source contribution: a spoken chatbot with voice cloning, built on two well-known public models, with training code provided. The proposed hidden-state pathway is a plausible mechanism for transferring semantic context from the LLM to the TTS module, and the two-stage training recipe is worth reporting. However, the evidence in the manuscript does not support the claim. The only quantitative comparison, Table 1, shows that JoyTTS underperforms its own CosyVoice2 backbone on both SS and WER, and no ablation is provided to isolate the effect of the hidden-state pathway. The self-referential training data (synthesized by CosyVoice2) introduces an additional confound. The strengths are the open-sourced code and the clear system description, but the evaluation is too thin to substantiate the stated benefits.
major comments (5)
- [Section 5 and Table 1] The conclusion that hidden-state integration 'enhances the model's performance' is directly contradicted by the paper's own baseline comparison. In Table 1, JoyTTS achieves SS 0.73 versus CosyVoice2's 0.748, and WER 5.09 versus CosyVoice2's 1.45. Since CosyVoice2 is the actual LLM-TTS module that JoyTTS is built upon, the proposed modification (the MLP-hidden-state pathway of Eq. 1) is associated with worse speaker similarity and substantially worse content consistency. No ablation is reported that removes the MLP path or that uses text embeddings alone, so the paper's principal claimed contribution is not merely unproven; the available data point in the opposite direction.
- [Section 2] The entire training corpus is generated by CosyVoice2, the same model family used as the LLM-TTS module, with reference voices drawn from WenetSpeech4TTS. This means the model is trained on its own component's outputs. The manuscript does not acknowledge this self-referential training setup or discuss its implications. If the synthetic distribution differs systematically from real conversational speech, the SEED benchmark scores will overestimate true voice-cloning quality in practical use. The authors should either provide an analysis of the distribution shift (e.g., evaluation on real recorded speech) or explicitly frame the results as applying only to CosyVoice2-like synthetic inputs.
- [Section 4] The latency claim of 1.8 seconds on a single NVIDIA 4090D is unsupported. No definition of latency is given (e.g., end-to-end response time, time to first token, per-utterance generation time), and there is no comparator without the hidden-state pathway. The conclusion that hidden states 'reduce the latency' therefore has no empirical basis. The authors should specify the measurement protocol and report latency for a configuration without the hidden-state injection, ideally as a function of input length.
- [Section 4 and Table 1] The evaluation is missing essential statistical and procedural details: the number of test utterances, error bars or confidence intervals, the definition of the SS/SM metric, and the protocol for selecting reference speakers and prompt texts. Furthermore, the paper cites Qwen2.5-Omni and LLaMA-Omni 2 as related spoken chatbots in Section 1.1 but does not compare JoyTTS against them. Without these details and comparisons, the reported numbers cannot be interpreted as meaningful evidence for the system's overall quality.
- [Section 3 and Eq. (2)] The two-stage training process is described only qualitatively. Equation (2) defines the loss as a sum of the LLM-Chat and LLM-TTS losses, but no weighting scheme, learning rates, batch sizes, or training durations are given. This makes the training recipe irreproducible from the paper alone, despite the claim of providing complete training code. The authors should specify these hyperparameters and clarify how the 'balanced' objective is achieved.
minor comments (6)
- [References] Reference [3] is listed as 'Minicpm-v: A gpt-4v level mllm on your phone', which appears to be MiniCPM-V, not MiniCPM-o as cited in the text. Please verify and correct the citation.
- [Throughout] There are several typos: 'Lama-Omni2' should be 'LLaMA-Omni 2' in Section 1.1, and the title has an extra space in 'V oice'.
- [Section 4 vs Table 1] The metric is called 'SM (Similarity Measure)' in the text but 'SS' in Table 1 and the abstract. Please use one consistent name and define it clearly.
- [Abstract and Table 1] The abstract refers to 'the testing machine seed-tts-zh', while Table 1 says 'SEED test-zh'. Use a consistent naming convention.
- [Section 2] No references are given for the SEED test set or WenetSpeech4TTS. These should be cited.
- [Eq. (1)] Please clarify the dimensions and operation in Eq. (1): the text embedding Emb(y_i) is said to be 768-dimensional; does MLP(h_i) also produce 768 dimensions and is the sum an elementwise addition? The sentence 'These mapped features are then combined with text embeddings' is ambiguous about whether a concatenation or a residual sum is used.
Circularity Check
No significant circularity: the paper's claims are empirical, externally benchmarked, and contain no derivation that reduces to its own inputs.
full rationale
JoyTTS makes no formal derivation; its central quantitative claims are SS and WER scores measured on the external seed-tts-zh benchmark (Table 1) and a reported latency of 1.8 seconds. There is no fitted parameter that is then renamed a prediction, and no uniqueness theorem or load-bearing citation that forces the architecture. The authors cite MiniCPM-o and CosyVoice2 as external prior work, and the present author list (Fangru Zhou, Jun Zhao, Guoxin Wang) does not overlap with the cited lists, so this is not a self-citation chain. The most notable in-scope concern is that the training audio was generated by CosyVoice2 ('To convert text dialogues into audio, we employed CosyVoice2') while CosyVoice2 is also the LLM-TTS module; this is a training-distribution overlap that may affect generalization, but it is not circular because the evaluation is held-out natural speech and the scores could in principle falsify the approach. Similarly, the conclusion that hidden states 'not only enhances the model's performance but also reduces the latency' is unsupported by any ablation and is actually undermined by Table 1, where JoyTTS has worse WER (5.09) than CosyVoice2 (1.45); however, a missing ablation or a contradicted empirical claim is a correctness/evidence problem, not a circular derivation. No step in the paper reduces by construction to its own inputs, so the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- MLP output dimension =
768
- Joint loss weighting =
1 and 1 in Eq. (2)
assumptions (4)
- domain assumption Pretrained MiniCPM-o and CosyVoice2 provide suitable base features.
- domain assumption CosyVoice2-synthesized conversational audio is a valid training signal for real voice cloning.
- domain assumption SS and WER on seed-tts-zh measure spoken-chatbot quality.
- ad hoc to paper Equation (1) sufficiently conditions the TTS module for voice cloning.
Cite this review
Pith. "Pith review of JoyTTS: LLM-based Spoken Chatbot With Voice Cloning." pith.science (2026). https://pith.science/paper/EADPR5OK
@misc{pith2026250702380,
author = {Pith},
title = {Pith review of: JoyTTS: LLM-based Spoken Chatbot With Voice Cloning},
year = {2026},
howpublished = {\url{https://pith.science/paper/EADPR5OK}},
note = {Machine review of arXiv:2507.02380}
}
read the original abstract
JoyTTS is an end-to-end spoken chatbot that combines large language models (LLM) with text-to-speech (TTS) technology, featuring voice cloning capabilities. This project is built upon the open-source MiniCPM-o and CosyVoice2 models and trained on 2000 hours of conversational data. We have also provided the complete training code to facilitate further development and optimization by the community. On the testing machine seed-tts-zh, it achieves a SS (speaker similarity) score of 0.73 and a WER (Word Error Rate) of 5.09. The code and models, along with training and inference scripts, are available at https://github.com/jdh-algo/JoyTTS.git.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Shuai Bai Ting He, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215 , 2025
arXiv 2025
-
[3]
LL a MA - O mni 2: Llm-based real-time spoken chatbot with autoregressive streaming speech synthesis
Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. LL a MA - O mni 2: Llm-based real-time spoken chatbot with autoregressive streaming speech synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics , 2025
work page 2025
-
[4]
Minicpm-v: A gpt-4v level mllm on your phone
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 , 2024
arXiv 2024
-
[5]
Cosyvoice 2: Scalable streaming speech synthesis with large language models
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117 , 2024
arXiv 2024
-
[6]
Redgpt: Reference-enlightened-dialogue by gpt and for gpt
Dongjie Yang, Ruifeng Yuan, Yuantao Fan, Yifei Yang, Zili Wang, and Shusen Wang. Redgpt: Reference-enlightened-dialogue by gpt and for gpt. GitHub repository , 2023
work page 2023
-
[7]
Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases. arXiv preprint arXiv:2303.14742 , 2023
arXiv 2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.