Pith. sign in

REVIEW 2 major objections 4 minor 42 references

JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read JELLY is a conversational speech synthesis framework that infers the emotional state of the next utterance from the audio of the conversation history alone, and uses that inference to synthesize emotionally matched speech.

desk verdict JELLY is a coherent LLM-based conversational TTS system with a real speech-only inference advantage, but the headline emotional context gains rest on a possible DailyDialog/DailyTalk overlap that the authors must rule out. read the letter →

arxiv 2501.04904 v1 pith:ZAMWZGAW submitted 2025-01-09 cs.CL cs.SDeess.AS

Jun-Hyeok Cha , Seung-Bin Kim , Hyung-Seok Oh , Seong-Whan Lee This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords conversationalspeechsynthesisemotionrecognitioncontextreasoninglargelanguagemodelLoRAQ-formeremotion-awareencoderintensityprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is about making conversational speech synthesis (CSS) more emotionally aware. It claims that the emotion and intensity of the next utterance in a dialogue can be inferred from the conversation history by a large language model, provided the speech of earlier turns is converted into emotion embeddings aligned with text. To show this, the authors build JELLY, a three-stage pipeline that first aligns speech emotions with text using a new Emotion-aware Q-former encoder, then fine-tunes the LLM with separate LoRA adapters for emotion and text tokens to predict the target utterance's emotional state, and finally conditions a FastSpeech 2 synthesizer on that prediction. On the DailyTalk corpus they report a weighted emotion-prediction accuracy of 78.54% versus 43.51% for the ECSS baseline, and better naturalness and emotional-alignment scores in both subjective listening tests and objective metrics; a speech-only variant achieves this without transcripts or emotion labels at inference. The significance of the claim is that it shows emotional context can be extracted from audio alone, which would make CSS usable in real-world settings where transcripts and labels are unavailable.

What carries the argument

The machinery is the pair of modality-specific partial LoRA adapters inside a frozen Vicuna-7B LLM, used with the Emotion-aware Q-former (EQ-former) encoder. The EQ-former passes the 32 intermediate layers of Whisper's encoder through a Time and Layer-Wise Transformer that attends over layers to pick emotionally relevant representations, then a Q-former with 25 learnable queries aligns those features with text through cross-attention; during inference Whisper's decoder supplies transcripts from audio. The LLM reasons over a dialogue history in which each utterance is represented by a concatenation of a speaker prefix, the EQ-former emotion embedding, and the transcript embedding, with the emotion and text embeddings routed through separate PLoRA adapters. Stage-1 pre-training on emotional speech datasets teaches the EQ-former to align emotion with text, and stage-2 pre-training of only the text adapter on DailyDialog teaches conversational emotion reasoning without requiring a large emotion-annotated speech corpus.

What would settle it

Check whether the DailyTalk test dialogues appear in the 13,118 DailyDialog dialogues used for stage-2 pre-training. If any overlap exists, hold out the overlapping dialogues and re-measure JELLY's emotion and intensity weighted accuracy against ECSS; the gap should persist on non-overlapping dialogues if the central claim is correct.

Watch

Extended reading notes

Core claim

The central discovery is that emotion and intensity for the next utterance can be jointly inferred from the emotional states of previous utterances, and that this inference improves downstream speech synthesis when the emotion representations are aligned with the LLM's text space. The authors demonstrate this by training the EQ-former to turn Whisper encoder features into emotion tokens through a Q-former, feeding those tokens through a dedicated LoRA adapter while text tokens pass through another, and pre-training the text-side adapter on the 13,118-dialogue DailyDialog corpus before fine-tuning on DailyTalk. In their experiments, JELLY outperforms ECSS and a GRU-based baseline on most synthesis metrics, with the largest gains in emotional context reasoning rather than low-level prosody, and the speech-only variant remains competitive, showing that the model can stand on audio alone.

Load-bearing premise

The evaluation assumes that the DailyTalk test conversations were not among the DailyDialog dialogues used to pre-train the text-side LoRA adapter, so the emotion predictions reflect reasoning about new conversations rather than memorized transcripts.

Editorial extensions

If this is right

  • A conversational speech system built this way can operate without ground-truth emotion labels or transcripts at inference: the speech-only variant of JELLY matches most baselines and beats them on MCD, DDUR, and F1v/uv using only audio history.
  • The three-stage pipeline is claimed to reduce the data bottleneck: stage-1 alignment uses widely available emotional speech datasets, stage-2 reasoning pre-trains on text-only dialogues, and only the final synthesis stage needs the smaller emotional conversational corpus.
  • JELLY's emotion and intensity prediction outperforms ECSS on weighted accuracy, unweighted accuracy, and macro F1, indicating that LLM-based joint reasoning captures conversational emotional context better than the heterogeneous-graph baseline.
  • The ablation results attribute the gains to the TLTR layer-selection mechanism, the Q-former alignment, the separation of emotion and text into distinct PLoRA adapters, and the two pre-training stages; removing any one of these lowers accuracy.
  • If the claim holds, JELLY's speech-only mode is a practical route to emotion-aware conversational agents in real-world settings, where transcripts and emotion labels are typically unavailable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test is to verify that the DailyTalk test split is disjoint from the 13,118 DailyDialog dialogues used in stage-2 pre-training; if any test dialogue appears verbatim in pre-training, the reported accuracy gains would need to be re-measured on non-overlapping dialogues.
  • The EQ-former's emotion embeddings could be probed for speaker-independence; if they encode emotion rather than speaker identity, they should transfer to unseen speakers and languages without re-alignment, which the paper does not test.
  • The speech-only variant relies on Whisper's transcripts, so its emotional-context accuracy is upper-bounded by Whisper's ASR quality; perturbing transcripts would show how much of the reasoning is lexical versus paralinguistic.
  • Because the emotion-reasoning and synthesis stages are trained separately, an end-to-end fine-tune could allow synthesis losses to reshape the emotion representations, although the paper intentionally keeps the stages distinct.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes JELLY, a three-stage framework for conversational speech synthesis that jointly performs emotion recognition and emotional context reasoning using a frozen large language model with multiple partial LoRA adapters, an EQ-former speech-emotion alignment module, and a FastSpeech 2 backbone. Stage 1 aligns speech emotion with text; stage 2 pre-trains text-only emotional context reasoning on DailyDialog and DailyTalk; and stage 3 synthesizes target speech conditioned on the predicted emotion and intensity. The authors report improvements over baselines in MOS, emotion classification accuracy (ECA), MCD, DDUR, and emotion/intensity prediction accuracy, including a speech-only variant that requires no transcripts or emotion labels at inference.

Significance. If the reported results survive a correction of the data-hygiene issue described below, the paper would be a valuable contribution to conversational speech synthesis: it addresses the real gap of emotional context modeling, proposes a practical architecture that can run from speech alone at inference, and makes code and audio samples available. The three-stage design is well motivated, and the ablation study gives useful evidence for the roles of the EQ-former, TLTR, and PLoRA modules. However, the central quantitative claim of 'excelling in emotional context modeling' rests on Table II, whose validity currently depends on an unverified assumption about train/test separation between the DailyDialog pre-training text and the DailyTalk test dialogues.

major comments (2)
  1. [Section II-C.2 and Section III-A] Stage-2 pre-training uses the full DailyDialog text dataset (13,118 dialogues), and the DailyTalk corpus used for fine-tuning and evaluation is explicitly described as derived from DailyDialog. The paper never states that the DailyTalk test conversations were excluded from the DailyDialog pre-training set, nor does it specify which DailyTalk split is used in the stage-2 fine-tuning and in the Table II evaluation. If the test dialogues appear in either the pre-training data or the fine-tuning data, the large gains in emotion weighted accuracy (78.54 vs. 43.51) and in ECA may be inflated by memorized transcripts and labels rather than by the proposed context-reasoning mechanism. Please document the exact data splits, provide an explicit overlap check between the DailyDialog pre-training dialogues and the DailyTalk test set, and, if overlap exists, re-run the experiments with the test dialogues removed from all pre-training and fine-tuning data.
  2. [Section III-A, Table II] The paper states 'We added three intensity levels from ECSS [21] to the DailyTalk dataset' but does not explain how these intensity labels were obtained. If they are outputs of the ECSS system, then the intensity evaluation in Table II uses those same predictions as ground truth while comparing against ECSS as a baseline; this would be circular and would also make the intensity labels potentially noisy. Please specify the annotation procedure, report inter-annotator agreement if humans were used, and, if the labels were machine-generated, treat the intensity results as a separate analysis rather than as a ground-truth evaluation.
minor comments (4)
  1. [Section IV-A, Table I] The text says JELLY 'outperforms' baselines on N-DMOS, but the 95% confidence intervals for JELLY (3.847 ± 0.042) and several baselines (e.g., GRU-based 3.792 ± 0.043, ECSS 3.802 ± 0.045) overlap substantially; a paired significance test or per-listener variance reporting would support the subjective claims more rigorously.
  2. [Figure 2] Figure 2 is dense and difficult to read; the text refers to three panels in the left, middle, and right, but the panel boundaries are not clearly labeled in the figure, and the font size is very small.
  3. [Section II-C.2] The notation for emotion embeddings is ambiguous: the text uses both e_k (ground-truth label) and E_k / \hat{E}_k (embedding), but the distinction between the text-derived E_k used in pre-training and the EQ-former-derived \hat{E}_k used in inference is not defined clearly; please clarify this in the text.
  4. [Section III-C] The ECA metric is said to be computed with emotion2vec, but the paper does not specify whether ECA is per-utterance classification accuracy, how the seven emotion categories and three intensity levels are mapped, or how ties are broken; please define the protocol.

Circularity Check

1 steps flagged · score 6.0 of 10

DailyTalk test dialogues are contained in the 13,118-dialogue DailyDialog stage-2 pre-training corpus, so the Table II emotion/intensity 'predictions' are partly memorization rather than independent context reasoning.

  1. fitted input called prediction [Section II-C.2 (Emotional Context Reasoning) and Section III-A (Experimental Setup)]
    "For the second stage, we use the DailyDialog [30] text dataset with 13,118 dialogues and the DailyTalk [29] dataset, which contains 20 hours of speech across 2,541 dialogues derived from DailyDialog. ... This strategy focuses on training the PLoRA-T module using only textual data, including emotion labels ek, from DailyTalk [29] and DailyDialog [30]."

    DailyTalk is a subset of DailyDialog, so the stage-2 pre-training corpus includes the exact transcripts and DailyDialog emotion labels of every DailyTalk dialogue, including the test split used for Table II. The LLM is therefore trained on the test inputs and outputs before 'predicting' them. The reported 78.54 weighted emotion accuracy (vs ECSS's 43.51) therefore measures memorization of a seen dialogue-to-label mapping rather than emotional context reasoning. The paper never states that DailyTalk test dialogues were excluded from the 13,118-dialogue DailyDialog pre-training, and because the split into training/validation/test is specified only for the third stage, the default reading is that all DailyDialog dialogues, including the DailyTalk test transcripts, were used.

full rationale

The paper is an empirical systems paper, not a theoretical derivation, so most of its pipeline is standard supervised training rather than a derivation chain. The central empirical claim ('JELLY excels in emotional context modeling') rests on Table II, and that table is compromised by a data-hygiene gap: DailyTalk is derived from DailyDialog, and stage-2 pre-training uses the entire DailyDialog text corpus with emotion labels. Unless the DailyTalk test dialogues were explicitly removed, which the paper does not state, the LLM has already seen the test transcripts and their labels, making the reported prediction accuracy a memorization score. This fits the fitted-input-called-prediction circularity pattern. The 'w/o PT in stage 2' and 'w/o stage 1' ablations each remove only one of the two possible contamination routes (DailyDialog text pre-training and DailyTalk audio in stage 1), so neither provides a fully clean independent estimate. The synthesis metrics in Table I (N-DMOS, E-DMOS, MCD, DDUR) provide partial independent support for output quality, but the emotional-context reasoning component is not cleanly evaluated. A milder provenance concern is that intensity labels were added to DailyTalk from ECSS, the baseline, which weakens the independence of the intensity ground truth but is not a by-construction reduction because the model is trained against those annotations and the baseline is scored on the same labels. Self-citations are limited to related emotional-TTS work and are not load-bearing.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical systems paper, not a derivation, so there are no fitted constants of the type the ledger tracks. The load-bearing assumptions are data-validity assumptions: Whisper representations contain enough paralinguistic information for emotion extraction, DailyTalk labels plus ECSS-added intensity labels are valid ground truth, DailyDialog pre-training and DailyTalk evaluation are disjoint, and LLM next-token prediction is a sufficient objective. The independence assumption is the most fragile.

assumptions (4)
  • domain assumption Whisper encoder intermediate representations capture paralinguistic cues sufficient for emotion recognition.
    Invoked in Section II-A: the EQ-former uses Whisper encoder representations and TLTR attention because they 'encompass various types of speech information, such as paralinguistic cues.' No ablation on Whisper layer choice is given other than TLTR versus last-layer.
  • domain assumption The emotion and intensity labels in DailyTalk and DailyDialog, including the ECSS-added intensity levels, are valid ground truth for training and evaluation.
    Section III-A adds three intensity levels from ECSS to DailyTalk, which originally had none; the procedure and validation are not described, so the paper assumes these labels are correct.
  • ad hoc to paper The DailyTalk test dialogues are independent of the DailyDialog pre-training text.
    Section II-C2 pre-trains PLoRA-T on DailyDialog text, and Section III-A states DailyTalk is derived from DailyDialog, but no overlap exclusion is described. If false, Table II results are inflated.
  • domain assumption LLM next-token prediction with an instruction prompt is a sufficient training signal to align emotion embeddings with text and to learn emotional context reasoning.
    The emotion-text alignment stage uses next-token prediction on the LLM in Section II-C1, but no evidence is provided that this objective maximizes emotion reasoning quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis." pith.science (2026). https://pith.science/paper/ZAMWZGAW

@misc{pith2026250104904,
  author       = {Pith},
  title        = {Pith review of: JELLY: Joint Emotion Recognition and Context Reasoning with LLMs for Conversational Speech Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAMWZGAW}},
  note         = {Machine review of arXiv:2501.04904}
}
read the original abstract

Recently, there has been a growing demand for conversational speech synthesis (CSS) that generates more natural speech by considering the conversational context. To address this, we introduce JELLY, a novel CSS framework that integrates emotion recognition and context reasoning for generating appropriate speech in conversation by fine-tuning a large language model (LLM) with multiple partial LoRA modules. We propose an Emotion-aware Q-former encoder, which enables the LLM to perceive emotions in speech. The encoder is trained to align speech emotions with text, utilizing datasets of emotional speech. The entire model is then fine-tuned with conversational speech data to infer emotional context for generating emotionally appropriate speech in conversation. Our experimental results demonstrate that JELLY excels in emotional context modeling, synthesizing speech that naturally aligns with conversation, while mitigating the scarcity of emotional conversational speech datasets.

Figures

Figures reproduced from arXiv: 2501.04904 by the authors.

Figure 1
Figure 1. Examples of different emotional contexts that can arise from the same [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overview of JELLY framework. II. METHODS A. Emotion-aware Q-former Encoder Comprehending the emotional context of a conversation requires recognizing the emotional state of the speaker in each utterance. Ac￾cordingly, we propose EQ-former as an emotion perception module for each utterance, as described in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 2
Figure 2. The models for the emotional context reasoning stage and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 36 canonical work pages

  1. [21]

    Emotion rendering for conversational speech synthesis with heterogeneous graph- based context modeling,

    Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, and Haizhou Li, “Emotion rendering for conversational speech synthesis with heterogeneous graph- based context modeling,” in Proc. AAAI Conf. Artif. Intell. (AAAI), 2024

    work page 2024

  2. [1]

    Natural TTS Synthesis by Conditioning Wavenet on Mel Spectrogram Predictions,

    Jonatha Shen et al., “Natural TTS Synthesis by Conditioning Wavenet on Mel Spectrogram Predictions,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2018

    work page 2018

  3. [2]

    FastSpeech 2: Fast and High-Quality End-to-End Text to Speech,

    Yi Ren et al., “FastSpeech 2: Fast and High-Quality End-to-End Text to Speech,” in Proc. Int. Conf. Learn. Represent. (ICLR) , 2021

    work page 2021

  4. [3]

    HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis,

    Sang-Hoon Lee et al., “HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2022

    work page 2022

  5. [4]

    Matcha-TTS: A fast TTS Architecture with Conditional Flow Matching,

    Shivam Mehta, Ruibo Tu, Jonas Beskow, ´Eva Sz ´ekely, and Gustav Eje Henter, “Matcha-TTS: A fast TTS Architecture with Conditional Flow Matching,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2024

    work page 2024

  6. [5]

    V oice- Flow: Efficient Text-To-Speech with Rectified Flow Matching,

    Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu, “V oice- Flow: Efficient Text-To-Speech with Rectified Flow Matching,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2024

    work page 2024

  7. [6]

    Conversational End-to-End TTS for V oice Agents,

    Haohan Guo, Shaofei Zhang, Frank K. Soong, Lei He, and Lei Xie, “Conversational End-to-End TTS for V oice Agents,” in IEEE Workshop Spok. Lang. Technol. (SLT) , 2021

    work page 2021

  8. [7]

    Enhancing Speaking Styles in Conversational Text-to- Speech Synthesis with Graph-Based Multi-Modal Context Modeling,

    Jingbei Li et al., “Enhancing Speaking Styles in Conversational Text-to- Speech Synthesis with Graph-Based Multi-Modal Context Modeling,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2022

    work page 2022

Show all 42 references
  1. [8]

    M2-CTTS: End-to-End Multi-Scale Multi-Modal Conversational Text-to-Speech Synthesis,

    Jinlong Xue et al., “M2-CTTS: End-to-End Multi-Scale Multi-Modal Conversational Text-to-Speech Synthesis,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2023

    2023

  2. [9]

    Concss: Contrastive-based Context Comprehension for Dialogue-Appropriate Prosody in Conversational Speech Synthesis,

    Yayue Deng et al., “Concss: Contrastive-based Context Comprehension for Dialogue-Appropriate Prosody in Conversational Speech Synthesis,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2024

    2024

  3. [10]

    Considering Temporal Connection between Turns for Conversational Speech Synthesis,

    Kangdi Mei et al., “Considering Temporal Connection between Turns for Conversational Speech Synthesis,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2024

    2024

  4. [11]

    A new recurrent neural-network architecture for visual pattern recognition,

    Seong-Whan Lee and Hee-Heon Song, “A new recurrent neural-network architecture for visual pattern recognition,” IEEE Transactions on Neural Networks, vol. 8, pp. 331–340, 1997

    1997

  5. [12]

    Classification of drowsiness levels based on a deep spatio-temporal convolutional bidirectional LSTM network using electroencephalogra- phy signals,

    Ji-Hoon Jeong, Baek-Woon Yu, Dae-Hyeok Lee, and Seong-Whan Lee, “Classification of drowsiness levels based on a deep spatio-temporal convolutional bidirectional LSTM network using electroencephalogra- phy signals,” Brain Sci., vol. 9, pp. 348, 2019

    2019

  6. [13]

    Towards an EEG-based intuitive BCI communication system using imagined speech and visual imagery,

    Seo-Hyun Lee, Minji Lee, Ji-Hoon Jeong, and Seong-Whan Lee, “Towards an EEG-based intuitive BCI communication system using imagined speech and visual imagery,” in Proc. IEEE Int. Conf. Syst. Man Cybern. (SMC) , 2019

    2019

  7. [14]

    A multi-view cnn with novel variance layer for motor imagery brain computer interface,

    Ravikiran Mane, Neethu Robinson, A Prasad Vinod, Seong-Whan Lee, and Cuntai Guan, “A multi-view cnn with novel variance layer for motor imagery brain computer interface,” in Proc. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC) , 2020

    2020

  8. [15]

    An adaptive deep reinforcement learning framework enables curling robots with human-like performance in real-world conditions,

    Dong-Ok Won, Klaus-Robert M ¨uller, and Seong-Whan Lee, “An adaptive deep reinforcement learning framework enables curling robots with human-like performance in real-world conditions,” Sci. Robot., vol. 5, pp. eabb9764, 2020

    2020

  9. [16]

    Periodwave: Multi-period flow matching for high-fidelity waveform generation,

    Sang-Hoon Lee, Ha-Yeong Choi, and Seong-Whan Lee, “Periodwave: Multi-period flow matching for high-fidelity waveform generation,” arXiv preprint arXiv:2408.07547 , 2024

    2024 arXiv

  10. [17]

    Emoq-tts: Emotion intensity quantization for fine-grained controllable emotional text-to-speech,

    Chae-Bin Im, Sang-Hoon Lee, Seung-Bin Kim, and Seong-Whan Lee, “Emoq-tts: Emotion intensity quantization for fine-grained controllable emotional text-to-speech,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2022, pp. 6317–6321

    2022

  11. [18]

    Diffprosody: Diffusion-based latent prosody generation for expressive speech synthe- sis with prosody conditional adversarial training,

    Hyung-Seok Oh, Sang-Hoon Lee, and Seong-Whan Lee, “Diffprosody: Diffusion-based latent prosody generation for expressive speech synthe- sis with prosody conditional adversarial training,” IEEE/ACM Trans. Audio, Speech, Lang. Process. , vol. 32, pp. 2654–2666, 2024

    2024

  12. [19]

    EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech,

    Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, Sang-Hoon Lee, and Seong-Whan Lee, “EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech,” in Ann. Conf. Int. Speech Commun. Assoc. (INTER- SPEECH), 2024

    2024

  13. [20]

    DurFlex-EVC: Duration-Flexible Emotional V oice Conversion with Parallel Generation,

    Hyoung-Seok Oh, Sang-Hoon Lee, Deok-Hyun Cho, and Seong-Whan Lee, “DurFlex-EVC: Duration-Flexible Emotional V oice Conversion with Parallel Generation,” arXiv preprint arXiv:2401.08095 , 2024

    2024 arXiv

  14. [22]

    Internlm-xcomposer2: Mastering free-form text- image composition and comprehension in vision-language large model,

    Xiaoyi Dong et al., “Internlm-xcomposer2: Mastering free-form text- image composition and comprehension in vision-language large model,” arXiv preprint arXiv:2401.16420 , 2024

    2024 arXiv

  15. [23]

    BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowl- edge Distillation,

    Chen Wang, Minpeng Liao, Zhongqiang Huang, and Jiajun Zhang, “BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowl- edge Distillation,” arXiv preprint arXiv:2405.19041 , 2024

    2024 arXiv

  16. [24]

    Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers,

    Yuan Gong, Sameer Khurana, Leonid Karlinsky, and James Glass, “Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers,” in Ann. Conf. Int. Speech Commun. Assoc. (INTERSPEECH) , 2023

    2023

  17. [25]

    BLIP-2: Boot- strapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “BLIP-2: Boot- strapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2023

    2023

  18. [26]

    Robust Speech Recognition via Large-Scale Weak Supervision,

    Alec Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2023

    2023

  19. [27]

    Joint Audio and Speech Understanding,

    Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass, “Joint Audio and Speech Understanding,” in IEEE Autom. Speech Recognit. Underst. Workshop (ASRU) , 2023

    2023

  20. [28]

    HiFi-GAN: Gen- erative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN: Gen- erative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” in Adv. Neural Inf. Process. Syst. (NeurIPS) , 2020

    2020

  21. [29]

    DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech,

    Keon Lee, Kyumin Park, and Daeyoung Kim, “DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech,” in IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP) , 2023

    2023

  22. [30]

    DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset,

    Yanran Li et al., “DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset,” in Int. Jt. Conf. Nat. Lang. Process. (IJCNLP), 2017

    2017

  23. [31]

    CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset,

    Houwei Cao et al., “CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset,” IEEE Trans. Affect. Comput., vol. 5, pp. 377–390, 2014

    2014

  24. [32]

    The emotional voices database: Towards controlling the emotion dimension in voice generation systems,

    Adaeze Adigwe, No ´e Tits, Kevin El Haddad, Sarah Ostadabbas, and Thierry Dutoit, “The emotional voices database: Towards controlling the emotion dimension in voice generation systems,” arXiv preprint arXiv:1806.09514, 2018

    2018 arXiv

  25. [33]

    IEMOCAP: Interactive emotional dyadic motion capture database,

    Carlos Busso et al., “IEMOCAP: Interactive emotional dyadic motion capture database,” Lang. Resour. Eval., vol. 42, pp. 335–359, 2008

    2008

  26. [34]

    MEAD: A Large-scale Audio-visual Dataset for Emotional Talking-face Generation,

    Kaisiyuan Wang et al., “MEAD: A Large-scale Audio-visual Dataset for Emotional Talking-face Generation,” in Eur. Conf. Comput. Vis. (ECCV), 2020

    2020

  27. [35]

    Toronto emotional speech set (tess)-younger talker happy,

    Kate Dupuis and M Kathleen Pichora-Fuller, “Toronto emotional speech set (tess)-younger talker happy,” 2010

    2010

  28. [36]

    Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,

    Wei-Lin Chiang et al., “Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,” 2023

    2023

  29. [37]

    LLama: Open and efficient foundation language models,

    Hugo Touvron et al., “LLama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  30. [38]

    Decoupled Weight Decay Regular- ization,

    Ilya Loshchilov and Frank Hutter, “Decoupled Weight Decay Regular- ization,” in Int. Conf. Learn. Represent. (ICLR) , 2019

    2019

  31. [39]

    emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation,

    Ziyang Ma et al., “emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation,” in Annu. Meet. Assoc. Comput. Linguist. (ACL), 2024

    2024

  32. [40]

    wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” in Adv. Neural Inf. Process. Syst. (NeurIPS) , 2020

    2020

  33. [41]

    Sequence-to-Sequence Acoustic Modeling for V oice Conversion,

    Jing-Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, and Li-Rong Dai, “Sequence-to-Sequence Acoustic Modeling for V oice Conversion,” IEEE/ACM Trans. Audio Speech Lang. Process. , vol. 27, pp. 631–644, 2019

    2019

  34. [42]

    LoRA: Low-Rank Adaptation of Large Language Models,

    Edward J Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” in Proc. Int. Conf. Learn. Represent. (ICLR) , 2022

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.