{"id":"5b05e32b-0db0-4a0f-b19d-c07423eec420","arxiv_id":"2501.04904","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"JELLY fine-tunes an LLM with partial LoRA adapters and an emotion-aware Q-former to predict and synthesize emotionally appropriate conversational speech from speech alone.","lead":"JELLY is a new speech system that predicts the emotion of the next sentence in a conversation from the sounds of previous sentences, then uses that emotion to make the voice sound natural. It combines a speech model, an emotion sensor, and a large language model, but its headline results may be inflated because the training and test conversations may overlap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DailyTalk test dialogues likely overlap with DailyDialog stage-2 pre-training, so the Table II emotion/intensity gains may reflect memorized labels rather than context reasoning.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: the absence of any stated exclusion of DailyTalk-test-origin dialogues from the 13,118-dialogue DailyDialog pre-training set. This concern directly threatens Table II, the cleanest evidence for the paper's core claim of superior emotional context reasoning. The architecture, ablations, and subjective listening results are otherwise plausible, and the speech-only variant is a meaningful contribution even if the pre-training overlap is confirmed, because its E-DMOS and ECA still beat several baselines. I do not find additional load-bearing objections beyond this data-hygiene issue: the objective metrics without confidence intervals are a weakness, but not as decisive as the potential memorization of the evaluation dialogues. The recommended verdict remains the reader's CONDITIONAL: the authors should be required to document the split or rerun the affected experiments, after which the central empirical claim can be assessed. I therefore keep the verdict unchanged rather than moving to rejection, because the paper may well have used a proper split that it simply failed to describe.","tokens_in":9905,"tokens_out":4261,"duration_ms":45052,"concrete_test":"Perform a contamination audit: retrieve the official DailyTalk train/validation/test dialogue IDs, map each test dialogue to its originating DailyDialog dialogue ID using the DailyTalk metadata, and check whether any mapped ID appears in the 13,118-dialogue DailyDialog pre-training set. Report the exact overlap count. If overlap is nonzero, rerun stage-2 pre-training with all overlapping DailyDialog dialogues removed (keeping the DailyTalk train/validation split intact) and recompute Table II emotion and intensity WA/UA/F1. If JELLY remains near 78.54/59.09 while ECSS stays near 43.51/13.66, the concern is resolved; if the numbers drop substantially, the central claim must be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the DailyTalk test split is disjoint from the DailyDialog text used in stage-2 pre-training. Section II-C.2 says PLoRA-T is pre-trained on 'DailyDialog [30] text dataset with 13,118 dialogues' along with DailyTalk, using text-form emotion labels; Section III-A says DailyTalk is a corpus of 2,541 dialogues 'derived from DailyDialog.' Because DailyTalk is built from DailyDialog dialogues, any DailyTalk test conversation will almost certainly also appear in the 13,118-dialogue DailyDialog set unless explicitly excluded. Since DailyDialog provides emotion labels and those labels are used as text during pre-training, the LLM may have seen the exact test transcripts and their emotional labels before Table II is computed. Table II is the principal quantitative support for the central claim that JELLY 'excels in emotional context modeling': emotion weighted accuracy is 78.54 for JELLY versus 43.51 for ECSS. The 'w/o PT in stage 2' ablation still reaches 67.15, but it does not resolve the concern because all variants fine-tune on the same DailyTalk data and the overlap question concerns the pre-corpus itself. If overlap exists, the headline emotion and intensity improvements are inflated by memorization rather than by proposed EQ-former and PLoRA reasoning. This is not an internal inconsistency, but a missing data-hygiene guarantee that the paper must document.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JELLY, a three-stage framework for conversational speech synthesis that jointly performs emotion recognition and emotional context reasoning using a frozen large language model with multiple partial LoRA adapters, an EQ-former speech-emotion alignment module, and a FastSpeech 2 backbone. Stage 1 aligns speech emotion with text; stage 2 pre-trains text-only emotional context reasoning on DailyDialog and DailyTalk; and stage 3 synthesizes target speech conditioned on the predicted emotion and intensity. The authors report improvements over baselines in MOS, emotion classification accuracy (ECA), MCD, DDUR, and emotion/intensity prediction accuracy, including a speech-only variant that requires no transcripts or emotion labels at inference.","tokens_in":10078,"tokens_out":5009,"duration_ms":48602,"significance":"If the reported results survive a correction of the data-hygiene issue described below, the paper would be a valuable contribution to conversational speech synthesis: it addresses the real gap of emotional context modeling, proposes a practical architecture that can run from speech alone at inference, and makes code and audio samples available. The three-stage design is well motivated, and the ablation study gives useful evidence for the roles of the EQ-former, TLTR, and PLoRA modules. However, the central quantitative claim of 'excelling in emotional context modeling' rests on Table II, whose validity currently depends on an unverified assumption about train/test separation between the DailyDialog pre-training text and the DailyTalk test dialogues.","major_comments":[{"comment":"Stage-2 pre-training uses the full DailyDialog text dataset (13,118 dialogues), and the DailyTalk corpus used for fine-tuning and evaluation is explicitly described as derived from DailyDialog. The paper never states that the DailyTalk test conversations were excluded from the DailyDialog pre-training set, nor does it specify which DailyTalk split is used in the stage-2 fine-tuning and in the Table II evaluation. If the test dialogues appear in either the pre-training data or the fine-tuning data, the large gains in emotion weighted accuracy (78.54 vs. 43.51) and in ECA may be inflated by memorized transcripts and labels rather than by the proposed context-reasoning mechanism. Please document the exact data splits, provide an explicit overlap check between the DailyDialog pre-training dialogues and the DailyTalk test set, and, if overlap exists, re-run the experiments with the test dialogues removed from all pre-training and fine-tuning data.","section":"Section II-C.2 and Section III-A"},{"comment":"The paper states 'We added three intensity levels from ECSS [21] to the DailyTalk dataset' but does not explain how these intensity labels were obtained. If they are outputs of the ECSS system, then the intensity evaluation in Table II uses those same predictions as ground truth while comparing against ECSS as a baseline; this would be circular and would also make the intensity labels potentially noisy. Please specify the annotation procedure, report inter-annotator agreement if humans were used, and, if the labels were machine-generated, treat the intensity results as a separate analysis rather than as a ground-truth evaluation.","section":"Section III-A, Table II"}],"minor_comments":[{"comment":"The text says JELLY 'outperforms' baselines on N-DMOS, but the 95% confidence intervals for JELLY (3.847 ± 0.042) and several baselines (e.g., GRU-based 3.792 ± 0.043, ECSS 3.802 ± 0.045) overlap substantially; a paired significance test or per-listener variance reporting would support the subjective claims more rigorously.","section":"Section IV-A, Table I"},{"comment":"Figure 2 is dense and difficult to read; the text refers to three panels in the left, middle, and right, but the panel boundaries are not clearly labeled in the figure, and the font size is very small.","section":"Figure 2"},{"comment":"The notation for emotion embeddings is ambiguous: the text uses both e_k (ground-truth label) and E_k / \\hat{E}_k (embedding), but the distinction between the text-derived E_k used in pre-training and the EQ-former-derived \\hat{E}_k used in inference is not defined clearly; please clarify this in the text.","section":"Section II-C.2"},{"comment":"The ECA metric is said to be computed with emotion2vec, but the paper does not specify whether ECA is per-utterance classification accuracy, how the seven emotion categories and three intensity levels are mapped, or how ties are broken; please define the protocol.","section":"Section III-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution, but the data-hygiene issue is load-bearing and must be resolved before publication. In particular, the overlap between DailyDialog pre-training text and the DailyTalk test set needs to be either explicitly ruled out or corrected by re-running the experiments. If the authors can provide a cleaned evaluation, the paper could be acceptable; if the overlap is confirmed and cannot be fixed, the central claims would be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid engineering contribution to conversational TTS: the three-stage pipeline, the EQ-former with TLTR and Q-former for emotion-text alignment, and the multiple PLoRA adapters are all sensible, and the speech-only inference variant is a genuine practical advantage. The ablations are informative, and the audio samples make the work easy to assess qualitatively.\n\nThe soft spot is load-bearing. Stage-2 pre-training uses all 13,118 DailyDialog dialogues plus DailyTalk, and DailyTalk is stated to be derived from DailyDialog. If the DailyTalk test dialogues appear in that DailyDialog text—which they almost certainly do unless explicitly excluded—the LLM has already seen the exact transcripts and their emotion labels before Table II is computed. That would make the 78.54 weighted accuracy for emotion prediction (vs. 43.51 for ECSS) a memorization artifact rather than evidence of context reasoning. The paper never states that test dialogues were removed from pre-training, and the 'w/o PT in stage 2' ablation does not resolve it because it removes the entire pre-training corpus, not just the overlapping part.\n\nOther concerns are minor by comparison: the objective metrics in Table I lack confidence intervals, and intensity labels come from ECSS while comparing against ECSS, which introduces a mild circularity. The citation pattern looks fair, and the method description is clear enough to reproduce.\n\nIf the authors can document a non-overlapping split or rerun the evaluation with cleaned data, the paper would be a respectable contribution to the CSS literature. As it stands, the central claim is unverified.\n\nThis paper deserves a serious referee, but the referee should make the data hygiene question a condition of acceptance. I would bring it to a reading group for the methodology and the leak discussion, but I would not cite it in my own work until the overlap is addressed.","headline":"JELLY is a coherent LLM-based conversational TTS system with a real speech-only inference advantage, but the headline emotional context gains rest on a possible DailyDialog/DailyTalk overlap that the authors must rule out.","tokens_in":10677,"tokens_out":2588,"would_cite":false,"duration_ms":25535,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JELLY is a conversational speech synthesis framework that infers the emotional state of the next utterance from the audio of the conversation history alone, and uses that inference to synthesize emotionally matched speech.","keywords":["conversational speech synthesis","emotion recognition","context reasoning","large language model","LoRA","Q-former","emotion-aware encoder","emotion intensity prediction"],"falsifier":"Check whether the DailyTalk test dialogues appear in the 13,118 DailyDialog dialogues used for stage-2 pre-training. If any overlap exists, hold out the overlapping dialogues and re-measure JELLY's emotion and intensity weighted accuracy against ECSS; the gap should persist on non-overlapping dialogues if the central claim is correct.","tokens_in":9632,"feed_emoji":"🎙️","tokens_out":10471,"duration_ms":82571,"temperature":0.7,"pith_summary":"This paper is about making conversational speech synthesis (CSS) more emotionally aware. It claims that the emotion and intensity of the next utterance in a dialogue can be inferred from the conversation history by a large language model, provided the speech of earlier turns is converted into emotion embeddings aligned with text. To show this, the authors build JELLY, a three-stage pipeline that first aligns speech emotions with text using a new Emotion-aware Q-former encoder, then fine-tunes the LLM with separate LoRA adapters for emotion and text tokens to predict the target utterance's emotional state, and finally conditions a FastSpeech 2 synthesizer on that prediction. On the DailyTalk corpus they report a weighted emotion-prediction accuracy of 78.54% versus 43.51% for the ECSS baseline, and better naturalness and emotional-alignment scores in both subjective listening tests and objective metrics; a speech-only variant achieves this without transcripts or emotion labels at inference. The significance of the claim is that it shows emotional context can be extracted from audio alone, which would make CSS usable in real-world settings where transcripts and labels are unavailable.","feed_headline":"JELLY predicts a speaker's emotion from conversation audio alone","feed_subtitle":"LLM with separate LoRA adapters predicts conversational emotion and synthesizes matching speech, no labels at inference.","key_machinery":"The machinery is the pair of modality-specific partial LoRA adapters inside a frozen Vicuna-7B LLM, used with the Emotion-aware Q-former (EQ-former) encoder. The EQ-former passes the 32 intermediate layers of Whisper's encoder through a Time and Layer-Wise Transformer that attends over layers to pick emotionally relevant representations, then a Q-former with 25 learnable queries aligns those features with text through cross-attention; during inference Whisper's decoder supplies transcripts from audio. The LLM reasons over a dialogue history in which each utterance is represented by a concatenation of a speaker prefix, the EQ-former emotion embedding, and the transcript embedding, with the emotion and text embeddings routed through separate PLoRA adapters. Stage-1 pre-training on emotional speech datasets teaches the EQ-former to align emotion with text, and stage-2 pre-training of only the text adapter on DailyDialog teaches conversational emotion reasoning without requiring a large emotion-annotated speech corpus.","core_discovery":"The central discovery is that emotion and intensity for the next utterance can be jointly inferred from the emotional states of previous utterances, and that this inference improves downstream speech synthesis when the emotion representations are aligned with the LLM's text space. The authors demonstrate this by training the EQ-former to turn Whisper encoder features into emotion tokens through a Q-former, feeding those tokens through a dedicated LoRA adapter while text tokens pass through another, and pre-training the text-side adapter on the 13,118-dialogue DailyDialog corpus before fine-tuning on DailyTalk. In their experiments, JELLY outperforms ECSS and a GRU-based baseline on most synthesis metrics, with the largest gains in emotional context reasoning rather than low-level prosody, and the speech-only variant remains competitive, showing that the model can stand on audio alone.","pith_inferences":["A natural stress test is to verify that the DailyTalk test split is disjoint from the 13,118 DailyDialog dialogues used in stage-2 pre-training; if any test dialogue appears verbatim in pre-training, the reported accuracy gains would need to be re-measured on non-overlapping dialogues.","The EQ-former's emotion embeddings could be probed for speaker-independence; if they encode emotion rather than speaker identity, they should transfer to unseen speakers and languages without re-alignment, which the paper does not test.","The speech-only variant relies on Whisper's transcripts, so its emotional-context accuracy is upper-bounded by Whisper's ASR quality; perturbing transcripts would show how much of the reasoning is lexical versus paralinguistic.","Because the emotion-reasoning and synthesis stages are trained separately, an end-to-end fine-tune could allow synthesis losses to reshape the emotion representations, although the paper intentionally keeps the stages distinct."],"forward_implications":["A conversational speech system built this way can operate without ground-truth emotion labels or transcripts at inference: the speech-only variant of JELLY matches most baselines and beats them on MCD, DDUR, and F1v/uv using only audio history.","The three-stage pipeline is claimed to reduce the data bottleneck: stage-1 alignment uses widely available emotional speech datasets, stage-2 reasoning pre-trains on text-only dialogues, and only the final synthesis stage needs the smaller emotional conversational corpus.","JELLY's emotion and intensity prediction outperforms ECSS on weighted accuracy, unweighted accuracy, and macro F1, indicating that LLM-based joint reasoning captures conversational emotional context better than the heterogeneous-graph baseline.","The ablation results attribute the gains to the TLTR layer-selection mechanism, the Q-former alignment, the separation of emotion and text into distinct PLoRA adapters, and the two pre-training stages; removing any one of these lowers accuracy.","If the claim holds, JELLY's speech-only mode is a practical route to emotion-aware conversational agents in real-world settings, where transcripts and emotion labels are typically unavailable."],"supporting_citations":[{"why":"ECSS provides the graph-based baseline and the emotion/intensity predictors that JELLY is compared against, and supplies the intensity labels added to DailyTalk.","marker":"[21]"},{"why":"DailyTalk is the spoken dialogue dataset used for stage-1 fine-tuning, stage-2 fine-tuning, and the emotion/intensity prediction evaluation.","marker":"[29]"},{"why":"DailyDialog's 13,118 dialogues are used to pre-train the PLoRA-T adapter in stage 2, the step claimed to mitigate the scarcity of emotional conversational data.","marker":"[30]"},{"why":"BLIP-2's Q-former is the cross-modal alignment architecture adapted into the EQ-former to align speech emotion with text.","marker":"[25]"},{"why":"Whisper's frozen encoder supplies the 32 intermediate layer representations that the TLTR pools, and its decoder generates transcripts at inference.","marker":"[26]"},{"why":"FastSpeech 2 is the TTS backbone extended with emotion and intensity encoders in the synthesis stage.","marker":"[2]"},{"why":"InternLM-XComposer2 introduces the partial LoRA (PLoRA) approach that routes emotion and text embeddings through separate adapters.","marker":"[22]"},{"why":"Vicuna-7B is the instruction-tuned LLM that is frozen and adapted with the PLoRA modules for emotional context reasoning.","marker":"[36]"}],"fun_headline_variants":["JELLY reads conversational emotion from audio, then speaks it","LLM aligns speech emotion to text for context-aware synthetic speech","Audio-only emotion cues drive context-aware speech synthesis in JELLY","JELLY fuses emotion recognition and context reasoning to synthesize conversation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the DailyTalk test conversations were not among the DailyDialog dialogues used to pre-train the text-side LoRA adapter, so the emotion predictions reflect reasoning about new conversations rather than memorized transcripts.","fun_headline_variants_meta":{"raw":{"variants":["JELLY reads conversational emotion from audio, then speaks it","LLM aligns speech emotion to text for context-aware synthetic speech","Audio-only emotion cues drive context-aware speech synthesis in JELLY","JELLY fuses emotion recognition and context reasoning to synthesize conversation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001076,"raw_usage":{"total_tokens":4457,"prompt_tokens":850,"completion_tokens":3607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":3534}},"tokens_in":466,"tokens_out":3607,"duration_ms":24334,"temperature":1.0,"reasoning_tokens":3534,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:23:38.547140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether the DailyTalk test dialogues appear in the 13,118 DailyDialog dialogues used for stage-2 pre-training. If any overlap exists, hold out the overlapping dialogues and re-measure JELLY's emotion and intensity weighted accuracy against ECSS; the gap should persist on non-overlapping dialogues if the central claim is correct.","supporting_citations":[{"cited_title":"Emotion rendering for conversational speech synthesis with heterogeneous graph- based context modeling,","cited_arxiv_id":null,"evidence_quote":"ECSS provides the graph-based baseline and the emotion/intensity predictors that JELLY is compared against, and supplies the intensity labels added to DailyTalk."},{"cited_title":"DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech,","cited_arxiv_id":null,"evidence_quote":"DailyTalk is the spoken dialogue dataset used for stage-1 fine-tuning, stage-2 fine-tuning, and the emotion/intensity prediction evaluation."},{"cited_title":"DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset,","cited_arxiv_id":null,"evidence_quote":"DailyDialog's 13,118 dialogues are used to pre-train the PLoRA-T adapter in stage 2, the step claimed to mitigate the scarcity of emotional conversational data."},{"cited_title":"BLIP-2: Boot- strapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,","cited_arxiv_id":null,"evidence_quote":"BLIP-2's Q-former is the cross-modal alignment architecture adapted into the EQ-former to align speech emotion with text."},{"cited_title":"Robust Speech Recognition via Large-Scale Weak Supervision,","cited_arxiv_id":null,"evidence_quote":"Whisper's frozen encoder supplies the 32 intermediate layer representations that the TLTR pools, and its decoder generates transcripts at inference."},{"cited_title":"FastSpeech 2: Fast and High-Quality End-to-End Text to Speech,","cited_arxiv_id":null,"evidence_quote":"FastSpeech 2 is the TTS backbone extended with emotion and intensity encoders in the synthesis stage."},{"cited_title":"Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,","cited_arxiv_id":null,"evidence_quote":"Vicuna-7B is the instruction-tuned LLM that is frozen and adapted with the PLoRA modules for emotional context reasoning."}],"review_version":1}