{"id":"8dc46f2e-2152-4b8b-9f89-c6fcc43c9b28","arxiv_id":"2507.02380","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"JoyTTS is an end-to-end spoken chatbot with voice cloning, built by feeding LLM hidden states into CosyVoice2 and releasing the training code.","lead":"JoyTTS is an open-source spoken chatbot that joins the MiniCPM-o large language model with the CosyVoice2 text-to-speech engine to enable voice cloning. It reports a speaker-similarity score of 0.73 and a word-error rate of 5.09 on the SEED Chinese test set, though the word-error rate is worse than CosyVoice2 alone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation supports the hidden-state benefit, and Table 1 shows JoyTTS underperforms its own CosyVoice2 backbone on both SS and WER, undermining the central claim.","rationale":"The reader's weakest_assumption focuses on the synthetic training corpus generated by CosyVoice2. That is a valid generalizability concern, but it is not the most load-bearing issue for the paper's central claim. The central claim is that the hidden-state integration improves performance and latency. The paper's own Table 1 shows that JoyTTS underperforms CosyVoice2, the very TTS backbone it extends, on both reported metrics (SS 0.73 vs 0.748; WER 5.09 vs 1.45). Since the only architectural novelty described is the hidden-state conditioning in Eq. 1, and no ablation isolates this component, the paper provides no positive evidence for its main claimed benefit. In fact, the comparison suggests the hidden-state pathway may be detrimental, at least for WER. This internal inconsistency is more directly relevant than the synthetic-data issue, which concerns extrapolation to real speech rather than the validity of the reported comparison. My proposed ablation would settle whether the hidden-state pathway helps, hurts, or has no effect. This does not change the reader's conditional verdict; it strengthens the reasons behind it, so I recommend keeping the verdict unchanged while adding this ablation as an explicit acceptance condition.","tokens_in":3409,"tokens_out":3032,"duration_ms":32344,"concrete_test":"Run an ablation on seed-tts-zh with JoyTTS hidden-state pathway enabled versus disabled: set TTS_embed = Emb(y_i) + MLP(h_i) per Eq. 1 versus TTS_embed = Emb(y_i) only, keeping all other components, training, and inference settings identical. Report SS, WER, and end-to-end latency on the same single NVIDIA 4090D for both variants. If the disabled variant does not show worse SS or WER, or does not show higher latency, then the Section 5 claim that hidden states 'enhance performance and reduce latency' is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 5) is that hidden-state integration 'enhances the model's performance but also reduces the latency'. The only quantitative evidence is Table 1, where JoyTTS achieves SS 0.73 and WER 5.09, while CosyVoice2, the LLM-TTS module that JoyTTS is built upon, achieves SS 0.748 and WER 1.45. Since the proposed innovation is the hidden-state pathway of Eq. 1 (TTS_embed = Emb(y_i) + MLP(h_i)), and no ablation removing this pathway is reported, the available data point in the opposite direction: adding hidden states is associated with worse speaker similarity and substantially worse word error rate. The latency claim is similarly unsupported, because the reported 1.8 s figure has no hidden-state-free comparator. Without an ablation, the paper's principal claimed contribution is not merely unproven; it is contradicted by the paper's own baseline comparison. This concern is more load-bearing than the synthetic-data issue because it attacks the central contribution directly, independent of training-data generalizability.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents JoyTTS, an end-to-end spoken chatbot that combines the MiniCPM-o LLM with a CosyVoice2-based TTS module through a hidden-state injection pathway (Eq. 1). The system is trained on about 2000 hours of multi-turn dialogue audio synthesized by CosyVoice2, with random reference speakers from WenetSpeech4TTS. On the SEED test-zh set, the authors report a speaker similarity (SS) of 0.73 and a word error rate (WER) of 5.09, as well as a 1.8-second latency on a single NVIDIA 4090D. The central claim is that injecting hidden states into the TTS embeddings simultaneously enhances performance and reduces latency, and the paper provides open-sourced training and inference code.","tokens_in":3761,"tokens_out":4265,"duration_ms":46547,"significance":"If the central claim were validated, JoyTTS would be a valuable open-source contribution: a spoken chatbot with voice cloning, built on two well-known public models, with training code provided. The proposed hidden-state pathway is a plausible mechanism for transferring semantic context from the LLM to the TTS module, and the two-stage training recipe is worth reporting. However, the evidence in the manuscript does not support the claim. The only quantitative comparison, Table 1, shows that JoyTTS underperforms its own CosyVoice2 backbone on both SS and WER, and no ablation is provided to isolate the effect of the hidden-state pathway. The self-referential training data (synthesized by CosyVoice2) introduces an additional confound. The strengths are the open-sourced code and the clear system description, but the evaluation is too thin to substantiate the stated benefits.","major_comments":[{"comment":"The conclusion that hidden-state integration 'enhances the model's performance' is directly contradicted by the paper's own baseline comparison. In Table 1, JoyTTS achieves SS 0.73 versus CosyVoice2's 0.748, and WER 5.09 versus CosyVoice2's 1.45. Since CosyVoice2 is the actual LLM-TTS module that JoyTTS is built upon, the proposed modification (the MLP-hidden-state pathway of Eq. 1) is associated with worse speaker similarity and substantially worse content consistency. No ablation is reported that removes the MLP path or that uses text embeddings alone, so the paper's principal claimed contribution is not merely unproven; the available data point in the opposite direction.","section":"Section 5 and Table 1"},{"comment":"The entire training corpus is generated by CosyVoice2, the same model family used as the LLM-TTS module, with reference voices drawn from WenetSpeech4TTS. This means the model is trained on its own component's outputs. The manuscript does not acknowledge this self-referential training setup or discuss its implications. If the synthetic distribution differs systematically from real conversational speech, the SEED benchmark scores will overestimate true voice-cloning quality in practical use. The authors should either provide an analysis of the distribution shift (e.g., evaluation on real recorded speech) or explicitly frame the results as applying only to CosyVoice2-like synthetic inputs.","section":"Section 2"},{"comment":"The latency claim of 1.8 seconds on a single NVIDIA 4090D is unsupported. No definition of latency is given (e.g., end-to-end response time, time to first token, per-utterance generation time), and there is no comparator without the hidden-state pathway. The conclusion that hidden states 'reduce the latency' therefore has no empirical basis. The authors should specify the measurement protocol and report latency for a configuration without the hidden-state injection, ideally as a function of input length.","section":"Section 4"},{"comment":"The evaluation is missing essential statistical and procedural details: the number of test utterances, error bars or confidence intervals, the definition of the SS/SM metric, and the protocol for selecting reference speakers and prompt texts. Furthermore, the paper cites Qwen2.5-Omni and LLaMA-Omni 2 as related spoken chatbots in Section 1.1 but does not compare JoyTTS against them. Without these details and comparisons, the reported numbers cannot be interpreted as meaningful evidence for the system's overall quality.","section":"Section 4 and Table 1"},{"comment":"The two-stage training process is described only qualitatively. Equation (2) defines the loss as a sum of the LLM-Chat and LLM-TTS losses, but no weighting scheme, learning rates, batch sizes, or training durations are given. This makes the training recipe irreproducible from the paper alone, despite the claim of providing complete training code. The authors should specify these hyperparameters and clarify how the 'balanced' objective is achieved.","section":"Section 3 and Eq. (2)"}],"minor_comments":[{"comment":"Reference [3] is listed as 'Minicpm-v: A gpt-4v level mllm on your phone', which appears to be MiniCPM-V, not MiniCPM-o as cited in the text. Please verify and correct the citation.","section":"References"},{"comment":"There are several typos: 'Lama-Omni2' should be 'LLaMA-Omni 2' in Section 1.1, and the title has an extra space in 'V oice'.","section":"Throughout"},{"comment":"The metric is called 'SM (Similarity Measure)' in the text but 'SS' in Table 1 and the abstract. Please use one consistent name and define it clearly.","section":"Section 4 vs Table 1"},{"comment":"The abstract refers to 'the testing machine seed-tts-zh', while Table 1 says 'SEED test-zh'. Use a consistent naming convention.","section":"Abstract and Table 1"},{"comment":"No references are given for the SEED test set or WenetSpeech4TTS. These should be cited.","section":"Section 2"},{"comment":"Please clarify the dimensions and operation in Eq. (1): the text embedding Emb(y_i) is said to be 768-dimensional; does MLP(h_i) also produce 768 dimensions and is the sum an elementwise addition? The sentence 'These mapped features are then combined with text embeddings' is ambiguous about whether a concatenation or a residual sum is used.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"This is a very preliminary system-description manuscript. The central claim is contradicted by the paper's own table, and the self-referential training data concern is not addressed. The paper might be more suitable for a workshop or demo track after substantial revision. The citation mismatch for MiniCPM-o/V and the thin evaluation suggest the authors should carefully rework the paper before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: JoyTTS is a genuinely open integration of MiniCPM-o's language model with CosyVoice2's TTS, using an MLP to map chat hidden states into the TTS embedding. That specific combination is not in the cited papers, so as a systems recipe it's new. The paper also provides training code, model weights, a latency figure, and a clear two-stage training scheme. That is real value for practitioners.\n\nWhat it doesn't give is evidence for its central claim. Section 5 says hidden-state integration 'enhances the model's performance,' but Table 1 shows JoyTTS at SS 0.73 and WER 5.09 against CosyVoice2's 0.748 and 1.45 on the same benchmark. The baseline is stronger on both metrics. There's no ablation without the MLP path, so the only quantitative comparison available argues the opposite of the prose. The latency claim (1.8 s on a 4090D) is likewise reported without a hidden-state-free comparator, so we can't know what the pathway contributes.\n\nThe evaluation supporting the system is thin: one test set, two metrics, no error bars, no significance, and no comparison against the spoken-chatbot systems named in the introduction. The training data is entirely synthesized by CosyVoice2 itself—the same TTS used as the LLM-TTS module—with random reference voices pulled from WenetSpeech4TTS. That's a self-referential training set that may systematically differ from real conversational audio, and the paper doesn't acknowledge it as a limitation.\n\nThere are also minor citation problems: MiniCPM-o is cited as the MiniCPM-V paper, and the text calls LLaMA-Omni 2 'Lama-Omni2.' These are fixable but sloppy.\n\nIs this a serious research contribution? Not on the evidence: the main innovation is unsupported, and the baseline comparison actively undermines it. But the system is open, documented, and probably useful to people building voice-cloning chatbots. If JoyTTS comes through peer review, I'd send it to reviewers with clear instructions to ask for an ablation, proper baselines, error bars, and an honest discussion of the synthetic-data pipeline. I would not accept the current claims as they stand.","headline":"Open-source integration of MiniCPM-o and CosyVoice2 whose central hidden-state claim is contradicted by its own Table 1.","tokens_in":4147,"tokens_out":2482,"would_cite":false,"duration_ms":27355,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JoyTTS claims that a spoken chatbot can clone a voice by feeding the chat LLM's hidden states into a CosyVoice2 TTS module, and reports a speaker similarity of 0.73 with a word error rate of 5.09 on seed-tts-zh.","keywords":["spoken chatbot","voice cloning","text-to-speech","large language model","hidden-state conditioning","speaker similarity","SEED benchmark","low-latency speech synthesis"],"falsifier":"Record short utterances from a set of speakers who never appear in the training corpora, run JoyTTS with those recordings as the prompt wave, and compare SS and WER with the same prompts passed through CosyVoice2; if JoyTTS's speaker similarity drops well below its reported 0.73 while CosyVoice2 stays near 0.748, the synthetic training signal is the cause. Separately, run inference with TTS_embed set to Emb(y_i)+MLP(h_i) and to Emb(y_i) alone; if SS, WER, and latency are unchanged, the paper's hidden-state claim is not supported.","tokens_in":3205,"feed_emoji":"🎙️","tokens_out":7207,"duration_ms":73907,"temperature":0.7,"pith_summary":"JoyTTS aims to show that an open, end-to-end spoken chatbot can add useful voice cloning by replacing the TTS component of an existing LLM-based chatbot with CosyVoice2 and by conditioning that TTS module on the chat LLM's hidden states. The authors report that, on the SEED Chinese test set, the model reaches a speaker-similarity score of 0.73 and a word error rate of 5.09, with about 1.8 seconds of latency on a single NVIDIA 4090D before engineering optimization. If these numbers hold, JoyTTS would be a reproducible foundation for conversational agents that speak in a chosen voice, since the full training and inference code is released. The paper's distinctive claim is that the hidden-state bridge between text generation and speech synthesis both improves cloning quality and lowers latency.","feed_headline":"Voice-cloning chatbot posts 0.73 speaker similarity at 1.8s latency","feed_subtitle":"JoyTTS pairs a 7B chat LLM with CosyVoice2, hitting WER 5.09 on Chinese SEED test prompts.","key_machinery":"The load-bearing object is the TTS embed, the sum of an LLM token embedding and an MLP-projected hidden state: TTS_embed = Emb(y_i) + MLP(h_i). The MLP maps the chat model's 3584-dimensional hidden states down to 768 dimensions so that they can be added directly to the text embeddings consumed by the CosyVoice2-based LLM-TTS module; at inference the same conditioning is built from the prompt text and prompt wave. This one equation is what lets a single spoken-chatbot pipeline carry speaker identity and conversational context into speech-token generation. Training proceeds in two stages, with the LLM-Chat and LLM-TTS modules first specialized separately and then jointly fine-tuned through a unified loss.","core_discovery":"The paper's central claim is that voice cloning in a spoken chatbot can be achieved by the fusion defined in Eq. (1): TTS_embed = Emb(y_i) + MLP(h_i), where h_i are the chat LLM's 3584-dimensional hidden states projected to 768 dimensions and added to the text-token embeddings before they enter the LLM-TTS module. With this conditioning, the prompt text and prompt wave are both passed through the LLM-Chat module at inference, supplying the TTS module with semantic context and speaker identity. Trained in two stages—separate LLM-Chat and LLM-TTS training followed by joint fine-tuning with a combined loss—JoyTTS reports SS 0.73 and WER 5.09 on seed-tts-zh, comparable to CosyVoice2's SS 0.748 and better than gpt-sovits's SS 0.55 at the same WER scale. The authors take this as evidence that hidden-state integration enhances both quality and speed for cloned-voice dialogue.","pith_inferences":["Beyond the paper: because all 2000 hours of training audio were synthesized by CosyVoice2 itself, the reported SS may measure how well JoyTTS clones voices as rendered by CosyVoice2, not how well it clones arbitrary real speakers; testing on real out-of-corpus recordings is needed.","Beyond the paper: the claim that hidden states improve performance and reduce latency is not isolated by an ablation in Table 1; a direct A/B test with and without the MLP(h_i) term would confirm or refute that causal story.","Beyond the paper: the same hidden-state conditioning could be applied to other LLM-TTS pairs, or extended to carry explicit emotion or style embeddings, which would give the architecture a cheap route to expressive control."],"forward_implications":["A released training recipe now exists for an end-to-end spoken chatbot with voice cloning, so other groups can reproduce and modify the pipeline rather than starting from scratch.","Because cloning is conditioned through the chat LLM's hidden states, the same trick could be carried to larger or smaller chat backbones without changing the TTS module.","At 1.8 seconds of latency on one consumer GPU, the architecture is close enough to conversational turn-taking to be tested in real interactive settings.","The SEED results anchor a three-way comparison among gpt-sovits, CosyVoice2, and JoyTTS, giving practitioners a concrete reference point for cloned-voice spoken dialogue.","The paper's proposed next step, emotion-control inputs into the LLM, has a natural insertion point through the same TTS-embed conditioning path."],"supporting_citations":[{"why":"Supplies the MiniCPM-o backbone and the GPT-Sovits-based TTS module that JoyTTS replaces, and defines the baseline the paper improves on.","marker":"[3]"},{"why":"Supplies both the LLM-TTS module and the synthesizer used to turn the 400K dialogues into the 2000-hour training corpus.","marker":"[4]"},{"why":"One of two open-source multi-turn dialogue datasets that provide the text training samples.","marker":"[5]"},{"why":"The other dialogue dataset; together with [5] it yields the 400K multi-turn text samples.","marker":"[6]"}],"fun_headline_variants":["LLM chatbot clones voice via hidden-state fusion, SS 0.73","Open-source JoyTTS: spoken chatbot with voice cloning via MLP","Voice-cloning LLM hits SS 0.73, WER 5.09 on Chinese prompts","JoyTTS: 2000-hr training enables LLM-TTS voice cloning","Hidden-state conditioning powers voice cloning in spoken chatbot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The decisive assumption is that 2000 hours of dialogue audio created by CosyVoice2—the very model family used as the speech generator—teaches the model to clone real human voices rather than merely to imitate CosyVoice2's synthetic voice distribution.","fun_headline_variants_meta":{"raw":{"variants":["LLM chatbot clones voice via hidden-state fusion, SS 0.73","Open-source JoyTTS: spoken chatbot with voice cloning via MLP","Voice-cloning LLM hits SS 0.73, WER 5.09 on Chinese prompts","JoyTTS: 2000-hr training enables LLM-TTS voice cloning","Hidden-state conditioning powers voice cloning in spoken chatbot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1327,"prompt_tokens":900,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":325}},"tokens_in":516,"tokens_out":427,"duration_ms":4772,"temperature":1.0,"reasoning_tokens":325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:30:32.617135+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record short utterances from a set of speakers who never appear in the training corpora, run JoyTTS with those recordings as the prompt wave, and compare SS and WER with the same prompts passed through CosyVoice2; if JoyTTS's speaker similarity drops well below its reported 0.73 while CosyVoice2 stays near 0.748, the synthetic training signal is the cause. Separately, run inference with TTS_embed set to Emb(y_i)+MLP(h_i) and to Emb(y_i) alone; if SS, WER, and latency are unchanged, the paper's hidden-state claim is not supported.","supporting_citations":[{"cited_title":"LL a MA - O mni 2: Llm-based real-time spoken chatbot with autoregressive streaming speech synthesis","cited_arxiv_id":null,"evidence_quote":"Supplies the MiniCPM-o backbone and the GPT-Sovits-based TTS module that JoyTTS replaces, and defines the baseline the paper improves on."},{"cited_title":"Redgpt: Reference-enlightened-dialogue by gpt and for gpt","cited_arxiv_id":null,"evidence_quote":"The other dialogue dataset; together with [5] it yields the 400K multi-turn text samples."}],"review_version":1}